System

Abstract

A system is provided.SOLUTION: A system includes a means for analyzing a question of a user by using a generation AI and generating an appropriate answer, a means for AR-displaying an operation procedure to a home electric appliance by using a camera of a smartphone, a means for receiving an input of the user and displaying the answer on the basis of an analysis result, and a means for providing a function retrieval system of the home electric appliance.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As modern home appliances become more and more functional, it is becoming more difficult to understand how to operate them and how to troubleshoot them. Furthermore, paper instruction manuals are often thick and it takes time to find the information you need, making them less convenient for users. Furthermore, inquiries to support centers can sometimes have long wait times, meaning that a prompt response cannot be expected. To resolve these issues, a more intuitive and responsive support system is needed. [Means for solving the problem]

[0005] This invention provides a system that uses generative AI to analyze user questions and generate appropriate answers, allowing users to instantly obtain troubleshooting information for home appliances. It also has a function that uses a smartphone camera to display AR operating procedures for actual home appliances, allowing users to intuitively confirm the correct operating method. Furthermore, it has a function that displays the contents of PDF manuals in simplified form in natural language, allowing users to quickly obtain the information they need. This system also provides a function search system for home appliances, allowing users to easily search for and check the information they need about specific functions. This greatly improves user convenience, making it easier to operate home appliances and troubleshoot problems.

[0006] "Generative AI" is a technology or system that uses artificial intelligence to generate optimal answers to user questions.

[0007] "Question analysis" is the process of using natural language processing technology to understand the text entered by the user and extract appropriate response information.

[0008] An "appropriate response" is information or procedures that respond accurately and quickly to a user's question or request.

[0009] A "smartphone camera" is a photographic device installed in a portable information communication terminal.

[0010] "Displaying operating procedures in AR" refers to using augmented reality technology to visually display operating procedures overlaid on actual images.

[0011] "User input" refers to information, such as text or voice, that a user provides through an application.

[0012] "Analysis results" are data regarding the meaning and intent extracted from user input using natural language processing technology.

[0013] "Displaying the answer" refers to visually presenting information generated based on the analysis results on the user's device screen.

[0014] "Home appliances" refers to any electrical or electronic device used in the home.

[0015] A "function search system" is a system that searches for and displays information about specific home appliances and their functions.

[0016] An "AI support chatbot" is a program that uses natural language processing and artificial intelligence to provide answers to users' questions in an interactive format.

[0017] "Troubleshooting" is the process of identifying the cause of a problem or malfunction in a home appliance and providing procedures and methods for resolving it.

[0018] A "PDF instruction manual" is a digital document format that contains instructions and specifications for a home appliance.

[0019] "Natural language" refers to the language form used by humans on a daily basis, and is text based on grammar and vocabulary.

[0020] "Simplification" refers to converting complex information or procedures into a simple, easy-to-understand form.

[0021] "Display" refers to providing information visually on a device screen or the like.

[0022] A "database" is a system that systematically organizes and stores information so that it can be quickly searched and retrieved as needed. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] This invention is a system that revolutionizes home appliance instruction manuals by utilizing generative AI and augmented reality (AR) technology. This system allows users to easily access operation and troubleshooting information for home appliances through a smartphone app, enabling them to operate and repair appliances intuitively and quickly.

[0045] System program flow

[0046] 1. Launch the app

[0047] Users simply launch a dedicated app on their smartphone to enable the camera function.

[0048] The device prepares to display the camera image in real time.

[0049] 2. Providing AR operation instructions

[0050] The user points the camera at a home appliance.

[0051] The device analyzes camera footage and recognizes home appliances.

[0052] Based on the recognized home appliances, the device will provide the user with operating procedures and methods using AR displays.

[0053] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[0054] 3. Use AI support chatbots

[0055] Users type their questions into the chatbot about how to use it or troubleshooting.

[0056] The terminal receives the user's question and sends it to the server.

[0057] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[0058] The device receives the generated answer and displays it to the user.

[0059] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[0060] 4. Simplified instructions

[0061] The user opens a PDF instruction manual within the app to check the operating instructions.

[0062] The device sends the PDF contents to the server.

[0063] The server uses generative AI to convert the content into a concise description in natural language.

[0064] The terminal will display the generated brief description to the user.

[0065] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[0066] 5. New product proposals and e-commerce site referrals

[0067] Users ask for information about specific features or new products.

[0068] The terminal sends this information to the server.

[0069] The server analyzes the information and generates relevant new product suggestions and e-commerce site purchase links.

[0070] The device displays this information to the user.

[0071] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[0072] This system allows users to quickly and intuitively understand and respond to operating and troubleshooting methods for home appliances. For manufacturers, this system configuration not only improves the quality of user support, but also helps promote new products and sales.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] The user launches the dedicated app on their smartphone, which then begins preparations to enable the camera function.

[0076] Step 2:

[0077] The device displays camera footage in real time and displays the footage the user takes on the screen.

[0078] Step 3:

[0079] Users point their camera at their appliances and use the system to get information on how to operate them or if they need to troubleshoot.

[0080] Step 4:

[0081] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[0082] Step 5:

[0083] The device sends information about the identified home appliances to a server, which then retrieves corresponding operating procedures and troubleshooting data from a database.

[0084] Step 6:

[0085] The server then sends the acquired operation instructions to the device, which then prepares the device for displaying the correct operation guide in AR (Augmented Reality).

[0086] Step 7:

[0087] The device overlays the received operating instructions and troubleshooting methods on the camera image in real time, allowing users to intuitively check how to operate their home appliances through the camera screen.

[0088] Step 8:

[0089] To get more detailed information, users can use the chatbot function installed in the app to input questions, such as "I don't know how to use XX" or "XX doesn't work."

[0090] Step 9:

[0091] The device sends the user's input question in text format to the server, which then analyzes the question using natural language processing (NLP) technology.

[0092] Step 10:

[0093] The server generates appropriate answers from a database based on the analysis results, and this process may involve the use of generative AI.

[0094] Step 11:

[0095] The server generates an appropriate response and sends it to the terminal, which then displays the received response to the user.

[0096] Step 12:

[0097] The user can then review the displayed answers and take any necessary action or troubleshooting steps, such as cleaning the filter or pressing the reset button.

[0098] Step 13:

[0099] If a user opens a PDF manual for a particular home appliance within the app, that's possible. If the PDF is difficult to understand, the user will ask for a simplified explanation.

[0100] Step 14:

[0101] The device sends the PDF content to the server, which then uses generative AI to convert the PDF's detailed content into simplified, natural language.

[0102] Step 15:

[0103] The server then sends the generated natural language brief instructions to the terminal, which then displays them to the user, allowing the user to easily understand complex operating procedures.

[0104] Step 16:

[0105] When users have questions about new product information or specific features, they enter that information through the chatbot.

[0106] Step 17:

[0107] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[0108] Step 18:

[0109] The server sends the new product information and a purchase link to the device, which displays it to the user. The user can check the new product information and purchase it directly from the purchase link if necessary.

[0110] These process steps allow users to quickly and intuitively operate home appliances, troubleshoot, explore new products, and more.

[0111] Example 1

[0112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] Traditionally, instruction manuals and troubleshooting guides for home appliances have mainly been in paper or static PDF format, making it difficult to operate and repair products intuitively and quickly. Furthermore, when users searched for information on new products, they had limited means of quickly accessing the appropriate information. This often resulted in users spending time using or repairing the product, causing inconvenience. Furthermore, manufacturers faced challenges in streamlining user support and promoting new products.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0115] In this invention, the server includes means for analyzing a user's question using a generation AI and generating an appropriate answer, means for displaying operating procedures for the device in augmented reality using a video display of the mobile device, and means for receiving user input and displaying the answer based on the analysis results. This allows users to quickly and intuitively understand how to operate and troubleshoot home appliances and easily obtain information on new products.

[0116] "Generative AI" is a technology that uses advanced artificial intelligence algorithms to analyze input data and automatically generate appropriate information and answers.

[0117] A "video device for a mobile device" is a device that has the function of acquiring and displaying images of the real world using a camera or display device installed on a smartphone, tablet, etc.

[0118] "Augmented reality display (AR display)" is a technology that overlays digital information on images of the real world, and is a means of providing users with intuitive information.

[0119] "Operating instructions" means a set of instructions or steps for the proper operation of a home appliance, explaining how to access the product's features.

[0120] "User input" refers to the text data and commands provided by the user through the application, and serves as the basis for the system to respond appropriately based on the analysis results.

[0121] A "function search system" is an information search technology that provides related information in response to a user's request when searching for specific function or product information.

[0122] "PDF Content" means the text and graphics data contained within a Portable Document Format (PDF) file, developed by Adobe Systems Incorporated, a document format that can be stored and viewed electronically.

[0123] "Natural language" refers to the language used by humans on a daily basis, along with the technology to convert it into a format that is easy for machines to understand and process the data.

[0124] "New product suggestions" are a way to introduce relevant new products based on a user's specific needs and interests, providing information and encouraging them to purchase.

[0125] A "purchase link" refers to a URL on the Internet that a user can access to purchase a product, and is a link that directs the user to an e-commerce site.

[0126] This invention is a system that revolutionizes home appliance instruction manuals by utilizing generative AI and augmented reality (AR) technology. Users can easily access operating instructions and troubleshooting information for home appliances through a smartphone app, enabling them to operate and repair appliances intuitively and quickly. The following describes the system's program processing and the hardware and software used.

[0127] Hardware and Software Use

[0128] Device: Use a smartphone or tablet with its built-in camera and display device.

[0129] Server: A server with high-performance computing resources is used to perform natural language processing and data analysis.

[0130] Generative AI model: A natural language processing model using artificial intelligence algorithms (e.g., GPT).

[0131] AR technology: Augmented reality software frameworks such as ARKit (Apple) and ARCore (Google).

[0132] Explanation of program processing

[0133] 1. Launch the app:

[0134] The user launches a dedicated app on their smartphone.

[0135] The device detects that the app has been launched and prepares to enable the camera function.

[0136] Example: When a user taps an app icon, the app launches and displays a camera view.

[0137] 2. AR operation instructions provided:

[0138] Users point their smartphone camera at home appliances.

[0139] The device analyzes camera footage in real time and recognizes home appliances.

[0140] Based on the recognized home appliances, the device will display operating procedures and methods on the screen using AR.

[0141] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes are displayed on the screen.

[0142] 3. Use AI-supported chatbots:

[0143] Users type questions into the chatbot about how to use it and troubleshooting.

[0144] The terminal sends the user's question to the server.

[0145] The server uses generative AI to analyze the user's question and generate an appropriate answer.

[0146] The device receives the generated answer and displays it to the user.

[0147] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[0148] Example prompt: "What should I do if my washing machine won't start?"

[0149] 4. Simplified instructions:

[0150] Users can open the instruction PDF within the app to check the operating instructions.

[0151] The device sends the PDF contents to the server.

[0152] The server uses generative AI to convert the content into a concise description in natural language.

[0153] The terminal will display the generated brief description to the user.

[0154] Example: When you open a PDF containing detailed operating instructions for an oven range, simple instructions such as "Press the timer setting button, enter the time, and press the start button" appear on the screen.

[0155] Example prompt: "Please briefly explain the oven operating procedure described in this PDF."

[0156] 5. New product proposals and e-commerce site referrals:

[0157] Users ask for information about specific features or new products.

[0158] The terminal sends this information to the server.

[0159] The server uses generative AI to analyze the information and generate relevant new product suggestions and purchase links.

[0160] The device displays this information to the user.

[0161] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[0162] Example prompt: "What are the features of the latest smart TVs?"

[0163] This system allows users to quickly and intuitively access information on how to operate and troubleshoot home appliances and new product information, and is also useful for manufacturers to streamline user support and promote new products.

[0164] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0165] Step 1: Launch the app

[0166] User: Launch the dedicated app on your smartphone.

[0167] What happens: The user taps on the app icon.

[0168] Input: The user's touch.

[0169] Output: The app starts.

[0170] Device: Detects that the app has been launched and prepares to enable the camera function.

[0171] What happens: The camera initialization process happens in the background and the camera view appears in the app.

[0172] Input: App launch signal.

[0173] Output: Camera view display.

[0174] Step 2: Provide AR operation instructions

[0175] User: Point your smartphone camera at a home appliance.

[0176] Specific action: To point the camera at a home appliance, hold the smartphone in your hand and frame the product.

[0177] Input: Home appliance footage.

[0178] Output: Acquisition of video data.

[0179] Terminal: Analyzes camera footage in real time and recognizes home appliances.

[0180] Specific operation: An image recognition algorithm is activated, analyzing the video data to detect the shape and characteristics of the home appliance.

[0181] Input: Video data.

[0182] Output: Recognition results of home appliances.

[0183] Terminal: Based on the recognized home appliance, the device displays operating procedures and methods on the screen using AR.

[0184] Specific behavior: Operation guides related to home appliances are overlaid on the screen.

[0185] Input: Recognition results for home appliances.

[0186] Output: AR display operation procedure.

[0187] Step 3: Use an AI-powered chatbot

[0188] Users: Type questions into the chatbot for instructions or troubleshooting.

[0189] Specific actions: Tap the chat button in the app, enter the question text and send it.

[0190] Input: Question text.

[0191] Output: The question is sent.

[0192] Terminal: Receives the user's question and sends it to the server.

[0193] Specific operation: Sends the question text to the server via an HTTP request.

[0194] Input: Question text.

[0195] Output: Sending question data.

[0196] Server: Uses generative AI to analyze the user's question and generate an appropriate answer.

[0197] What it does: The input text is parsed, answers are extracted from relevant databases, and formatted in the appropriate natural language format.

[0198] Input: Question data.

[0199] Output: Response data.

[0200] Terminal: Receives the generated answer and displays it to the user.

[0201] Specific behavior: The reply text received from the server is displayed in the chat window.

[0202] Input: Response data.

[0203] Output: Display the answer.

[0204] Step 4: Simplify the instructions

[0205] User: Open the instruction PDF within the app to view instructions.

[0206] Specific operation: To view an instruction PDF, select the PDF you want from the list within the app.

[0207] Input: Select PDF file.

[0208] Output: PDF content display.

[0209] Device: Send the PDF content to the server.

[0210] Specific operation: Extract the contents of the PDF file as text data and upload it to the server.

[0211] Input: PDF content.

[0212] Output: Sending text data.

[0213] Server: Uses generative AI to convert the content into concise descriptions in natural language.

[0214] What it does: A generative AI model is activated, analyzing the text data and converting it into a concise description.

[0215] Input: Text data.

[0216] Output: A short description.

[0217] Terminal: Display the generated short description to the user.

[0218] Specific behavior: The generated short description is displayed in the PDF viewer.

[0219] Input: A short description.

[0220] Output: Display a short description.

[0221] Step 5: Propose new products and direct them to the EC site

[0222] Users: Ask for information about specific features or new products.

[0223] What it does: Use the search box within the app to enter specific product information.

[0224] Input: Your search query.

[0225] Output: Submitting a search query.

[0226] Device: Sends this information to the server.

[0227] Specific operation: The entered search query is sent to the server via an HTTP request.

[0228] Input: Your search query.

[0229] Output: Submitting a search query.

[0230] Server: Analyzes the information using generative AI and generates relevant new product suggestions and purchase links.

[0231] What it does: The server parses the search query and generates relevant product information and purchase links.

[0232] Input: Your search query.

[0233] Output: New product suggestions and purchase links.

[0234] Terminal: Display this information to the user.

[0235] What happens: The generated suggestions and links are displayed in the results area within the app.

[0236] Input: New product suggestions and purchase links.

[0237] Output: Display information.

[0238] (Application example 1)

[0239] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0240] There is a wide range of information on the operation, maintenance, and troubleshooting of factory machinery, and conventional manuals alone make it difficult for users to respond quickly and accurately.In addition, there are limited means to quickly provide information on new tools and parts, and these issues need to be improved.

[0241] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0242] In this invention, the server includes a means for analyzing a user's question using a generative AI and generating an appropriate answer, a means for displaying operating procedures for the machine in AR using a camera, a means for receiving user input and displaying answers based on the analysis results, and a means for providing a machine function search system. This allows users to quickly and intuitively understand and respond to operating methods and troubleshooting of the machine. It also includes a means for suggesting new products and necessary parts related to the analyzed machine and providing links to e-commerce sites, allowing users to quickly obtain the necessary information.

[0243] "Generative AI" refers to an artificial intelligence model that analyzes user input and generates appropriate answers or information.

[0244] A "camera" is an optical device for visually capturing mechanical equipment.

[0245] "Machinery" refers to mechanical equipment and robots used in factories.

[0246] An "operating procedure" refers to a series of actions or procedures for operating a machine or device correctly.

[0247] "AR display" refers to the use of augmented reality technology to overlay virtual information onto real-world images.

[0248] "User input" refers to information such as operations or questions that a user makes to the system.

[0249] "Analysis results" refer to the answers and information derived by the generating AI based on user input.

[0250] A "function search system" is a system that allows you to search and obtain information about the functions and characteristics of mechanical devices.

[0251] An "AI support chatbot" is an interactive program that uses artificial intelligence to answer users' questions and provide support.

[0252] "Troubleshooting information" refers to information that provides solutions or methods of dealing with problems or malfunctions that occur in mechanical equipment.

[0253] "PDF instruction manual content" refers to an electronic document that contains detailed instructions for operating and maintaining a machine or device.

[0254] "Natural language simplification" refers to the process of expressing technical content in simple, everyday language so that it is easy for users to understand.

[0255] "New products" are new tools or parts that are newly introduced to the market.

[0256] An "e-commerce site" is a website that sells goods and services over the Internet.

[0257] This invention provides a system for intuitive and rapid operation, maintenance, and troubleshooting of factory machinery. The system utilizes generative AI models and augmented reality (AR) technology to allow users to access machinery information using their smartphones.

[0258] Program processing

[0259] Hardware and software used

[0260] Smartphone: High-resolution camera, AR-enabled

[0261] Server: Equipped with high-performance CPU / GPU, natural language processing (NLP) software, generative AI models

[0262] Frameworks: TensorFlow, PyTorch

[0263] API: Google Vision API, OpenAI API

[0264] Data processing and calculation

[0265] 1. Camera image analysis:

[0266] The server uses the Google Vision API to recognize mechanical devices from the smartphone camera image.

[0267] Example: When a user points their smartphone camera at a factory robot, the robot's model number and type are automatically recognized.

[0268] 2. Natural Language Processing:

[0269] The server analyzes the user's question using the OpenAI API and generates an appropriate solution.

[0270] Example: When a user asks, "What should I do when error code E02 appears?", the server generates a solution such as, "Error code E02 indicates that the machine is overheating. Make sure the cooling system is working properly and, if possible, turn the power off and restart."

[0271] 3. Manual Simplification Using Generative AI:

[0272] The server uses a generation AI (GPT-3) to simplify the PDF of the detailed manual in natural language.

[0273] Example: Simplifying a PDF operating manual for a machine generates simple instructions such as "Press the emergency stop button and press the restart button after an interruption."

[0274] 4. New product proposal:

[0275] The server suggests new products and necessary parts related to the analyzed machinery and provides links to e-commerce sites.

[0276] Example: The server analyzes a user's question, "I want to know the features of the latest automation tool," and provides information such as, "This automation tool is equipped with advanced sensor functions and a real-time monitoring system. You can purchase it here (link)."

[0277] Specific examples (prompt examples)

[0278] Example prompt sentence:

[0279] "What should I do when error code E02 appears?"

[0280] In this way, this invention provides a system that streamlines the provision of information on how to operate and troubleshoot machinery in a factory, allowing users to respond intuitively. It also quickly provides information on necessary new products and parts, and easily guides users to e-commerce sites.

[0281] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0282] Step 1:

[0283] The user launches a dedicated app on their smartphone and enables the camera function. The device prepares to display camera footage in real time, allowing the user to capture machinery and equipment with the camera.

[0284] Step 2:

[0285] The user points their smartphone camera at a machine. The device analyzes the camera image and recognizes the machine. The Google Vision API is used to do this, and the model number and type of the machine are sent to the device as analysis results.

[0286] Step 3:

[0287] The device uses AR to display operation procedures and maintenance methods for the recognized machinery. This allows the user to proceed with the work while visually checking the operation procedures. The displayed content is based on data provided in advance by the server.

[0288] Step 4:

[0289] When a user has a question about a machine, they can input it using the chatbot function within the app. The device then sends the question to the server. A specific example of input is a prompt such as, "Please tell me what to do when error code E02 appears."

[0290] Step 5:

[0291] The server uses the OpenAI API to analyze the user's question and generate an appropriate solution. The generated answer is sent from the server to the device and displayed to the user. For example, the user may be provided with information such as, "Error code E02 indicates that the machine is overheating. Please make sure the cooling system is operating properly and, if possible, turn the power off and restart."

[0292] Step 6:

[0293] When a user opens a PDF of a detailed operating manual within the app, the device sends the contents of the PDF to the server. The server uses generative AI (GPT-3) to simplify the PDF contents in natural language and sends the result to the device. The device then displays the simplified instructions to the user. For example, a simple instruction such as "Press the emergency stop button, then press the restart button after interruption" is displayed.

[0294] Step 7:

[0295] When a user requests information about a new tool or part, the device sends the request to the server. Based on the analysis results, the server generates relevant new product information and links to e-commerce sites, and sends them to the device. The device then displays this information to the user. For example, the device may provide information such as, "This automation tool is equipped with advanced sensor functions and a real-time monitoring system. Purchase here (link)."

[0296] Through these processing steps, the present invention provides a system that streamlines the operation and troubleshooting of machinery in a factory, allowing users to respond quickly and intuitively.

[0297] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0298] This invention is a system that revolutionizes home appliance instruction manuals by combining generative AI, augmented reality (AR) technology, and an emotion engine that recognizes user emotions. This system allows users to easily access instructions and troubleshooting information for home appliances through a smartphone app, and also provides support based on the user's emotional state, creating a more personalized user experience.

[0299] System program flow

[0300] 1. Launch the app

[0301] The user launches the dedicated app on their smartphone, which activates the camera function and simultaneously activates the emotion engine.

[0302] The device displays the camera footage in real time and prepares the footage the user captures to be displayed on the screen.

[0303] 2. Providing AR operation instructions

[0304] The user points the camera at a home appliance.

[0305] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[0306] The device sends information about the recognized home appliances to the server, and the server retrieves corresponding operating procedures and troubleshooting data from a database.

[0307] The server sends the acquired operating procedures to the device, and the device displays the operating procedures and troubleshooting methods to the user in AR.

[0308] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[0309] 3. Use AI support chatbots

[0310] Users type their questions into the chatbot about how to use it or troubleshooting.

[0311] The terminal receives the user's question and sends it to the server.

[0312] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[0313] The device receives the generated answer and displays it to the user.

[0314] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[0315] 4. Utilizing the Emotion Engine

[0316] When a user enters a question via text or voice, the emotion engine analyzes the user's emotions.

[0317] The device sends the analysis results to the server, and the server adjusts the response content based on the emotional data.

[0318] Example: If the user displays emotions that indicate confusion or annoyance, the server might respond with a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[0319] 5. Simplified instructions

[0320] The user opens a PDF instruction manual within the app to check the operating instructions.

[0321] The device sends the PDF content to the server, which then uses generative AI to convert the PDF's detailed content into simplified, natural language.

[0322] The terminal will display the generated brief description to the user.

[0323] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[0324] 6. New product proposals and e-commerce site referrals

[0325] Users ask for information about specific features or new products.

[0326] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[0327] The device displays this information to the user.

[0328] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[0329] This system allows users to quickly and intuitively understand and respond to how to operate and troubleshoot home appliances. In addition, responses are given that take the user's emotions into consideration, providing more personalized support. For manufacturers, this system configuration not only improves the quality of user support, but is also useful for promoting new products and sales.

[0330] The processing flow will be explained below.

[0331] Step 1:

[0332] The user launches the dedicated app on their smartphone, which activates the camera function and simultaneously activates the emotion engine.

[0333] Step 2:

[0334] The device displays the camera footage in real time and prepares the footage the user captures to be displayed on the screen.

[0335] Step 3:

[0336] The user points the camera at a home appliance.

[0337] Step 4:

[0338] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[0339] Step 5:

[0340] The device sends information about the recognized home appliances to the server.

[0341] Step 6:

[0342] The server retrieves the corresponding operating procedures and troubleshooting data from the database.

[0343] Step 7:

[0344] The server sends the acquired operating procedures to the device, and the device displays the operating procedures and troubleshooting methods to the user in AR.

[0345] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[0346] Step 8:

[0347] Users type their questions into the chatbot about how to use it or troubleshooting.

[0348] Step 9:

[0349] The terminal sends the user's question in text format to the server.

[0350] Step 10:

[0351] The server uses natural language processing techniques to analyze the user's question.

[0352] Step 11:

[0353] Based on the analysis results, the server generates appropriate answers from the database and also analyzes the user's emotions using an emotion engine.

[0354] Step 12:

[0355] The device sends the user's voice and text input data to the emotion engine in real time and analyzes emotions.

[0356] Step 13:

[0357] The emotion engine determines the user's emotional state based on the user's emotional data.

[0358] Step 14:

[0359] The server uses the analysis results of the emotion engine to generate a response message that is tailored to the user's emotional state.

[0360] Example: If a user expresses confusion or frustration, the server tailors and generates a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[0361] Step 15:

[0362] The server sends the generated response message to the terminal, which displays it to the user.

[0363] Step 16:

[0364] The user can refer to the displayed answers to perform the necessary operations or troubleshoot.

[0365] Step 17:

[0366] When a user opens a PDF manual for a particular home appliance within the app, the device sends the contents of the PDF to the server.

[0367] Step 18:

[0368] The server uses generative AI to convert the detailed content of the PDF into simplified, natural language.

[0369] Step 19:

[0370] The server sends the generated natural language short description to the terminal, which displays it to the user.

[0371] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[0372] Step 20:

[0373] When users have questions about new product information or specific features, they enter that information through the chatbot.

[0374] Step 21:

[0375] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[0376] Step 22:

[0377] The server sends the new product information and a purchase link to the device, which displays it to the user. The user can check the new product information and purchase it directly from the purchase link if necessary.

[0378] In this way, users can quickly and intuitively understand and respond to how to operate and troubleshoot their home appliances, and also receive personalized, emotionally sensitive support.

[0379] Example 2

[0380] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0381] Conventional instruction manuals for home appliances have scattered information, making it difficult for users to quickly obtain the information they need. Furthermore, they lack a mechanism for reading emotions and responding appropriately, making it difficult to alleviate user confusion and frustration. Furthermore, they lack a means for providing new product information and links to related e-commerce sites, making it difficult to stimulate users' desire to purchase. The purpose of this invention is to solve these problems.

[0382] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing a user's question using a generation AI and generating an appropriate answer, a means for displaying operating procedures for the device in augmented reality using a camera of a mobile terminal, and a means for analyzing the user's emotions and adjusting the answer content based on the analysis results. This allows the user to quickly and intuitively understand the operating procedures and receive personalized support that takes emotions into consideration. Furthermore, it also makes it easy to promote new products and direct users to related online shopping sites.

[0383] "Generative AI" is an artificial intelligence technology that analyzes questions asked in natural language by users and automatically generates appropriate answers.

[0384] "Mobile terminal" refers to a portable computing device such as a smartphone or tablet.

[0385] A "camera" is a device that converts optical images into digital data for display or analysis in real time.

[0386] "Augmented reality display" is a technology that displays digital information overlaid on real-world images.

[0387] "Equipment" broadly refers to electronic devices such as home appliances.

[0388] "Sentiment analysis" refers to the process of inferring a user's emotional state from their text or voice.

[0389] "Adjusting the content of the response based on the analysis results" means appropriately changing the tone and content of the generated response depending on the user's emotional state.

[0390] "Promotional Information" means information intended to promote new products or services.

[0391] "Related Shopping Sites" means websites where users can purchase products online.

[0392] This invention is a system that combines generative AI, augmented reality (AR) technology, and an emotion engine that recognizes user emotions, and provides operation instructions and troubleshooting information for home appliances to improve the user experience. This system mainly operates through the collaboration of three parties: a server, a device, and a user.

[0393] First, the user launches the dedicated app on their smartphone or tablet, which activates the camera function and activates the emotion engine. The device displays the camera footage in real time and shows the footage the user takes on the screen. At this time, the footage captured by the device's camera is analyzed using an object detection algorithm (e.g., YOLO or ResNet).

[0394] Next, when the user points the camera at a home appliance, the device recognizes the appliance and sends the recognition results to the server. The server retrieves the relevant operating procedures and troubleshooting information from a database (e.g., MySQL, PostgreSQL) and sends the information back to the device. This allows the device to display the operating procedures or troubleshooting methods on the camera image in the form of augmented reality (AR). For example, when the user points the camera at an air conditioner, the screen will show how to use each button on the remote control and how to set a specific mode.

[0395] Additionally, users can input questions about operation methods and troubleshooting into a dedicated chatbot. The device sends these questions as text data to the server, which then uses natural language processing technology (e.g., GPT-3, BERT) to analyze the user's question and generate an appropriate answer. The generated answer is then sent back to the device and displayed to the user in chat format. For example, if a user asks, "My washing machine isn't working," the generating AI will respond with specific instructions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[0396] The emotion engine can also be used to analyze the emotions expressed when a user types a question. The device also sends the emotion analysis results to the server, which then uses the results to tailor the answer to the user's emotional state. For example, if the user expresses confusion or irritation, the server can generate a considerate response such as, "We apologize for the inconvenience. Could you please try the following steps?"

[0397] The contents of PDF manuals can also be simplified and displayed. When a user opens a PDF manual within the app, the device sends the contents of the PDF as text data to the server. The server then uses generative AI to simplify this text data, converts it into natural language, and sends it back to the device. The device then displays the simplified instructions to the user. For example, when a user opens a PDF containing detailed operating instructions for a microwave oven, simple instructions such as "Press the timer setting button, enter the time, and press the start button" are displayed.

[0398] Furthermore, when a user requests information about a specific feature or new product, the device sends this input information to the server. The server analyzes the information, generates new product suggestions and links to related online shopping sites, and sends them to the device. The device then displays this information to the user. For example, if a user asks, "I want to know the features of the latest smart TV," the server might respond, "This smart TV has high resolution and voice recognition capabilities. You can purchase it here (link)."

[0399] This system allows users to quickly and intuitively understand and respond to how to operate and troubleshoot their home appliances, and also provides personalized support that takes into account the user's emotions, resulting in a more satisfying user experience.

[0400] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0401] Step 1:

[0402] The user launches a dedicated app on their smartphone.

[0403] Specific action: The user taps the app icon on the smartphone's home screen.

[0404] Input: Tap input from the user

[0405] Output: Launching the app and displaying the initial screen

[0406] Step 2:

[0407] The device will enable the app's camera function and emotion engine.

[0408] Specific operation: The device will launch the camera function and ask the user for permission to use the camera. At the same time, the emotion engine will also launch in the background.

[0409] Input: App launch signal, camera access permission from user

[0410] Output: Streaming camera footage and emotion engine status

[0411] Step 3:

[0412] The user points the camera at a home appliance.

[0413] Specific action: The user places a home appliance within the camera frame.

[0414] Input: Camera video data

[0415] Output: Video frame showing home appliances

[0416] Step 4:

[0417] The device analyzes camera footage in real time and runs object detection algorithms to recognize home appliances.

[0418] How it works: An object detection algorithm (e.g., YOLO or ResNet) extracts features from video frames and matches them with known home appliances in a database.

[0419] Input: Camera image frame

[0420] Output: ID or feature information of the recognized home appliance

[0421] Step 5:

[0422] The device sends information about the recognized home appliances to the server.

[0423] Specific operation: The device sends a request including the home appliance ID and image data to the server.

[0424] Input: ID or characteristic information of home appliance

[0425] Output: Request message to the server

[0426] Step 6:

[0427] The server retrieves the corresponding operating procedures and troubleshooting data from the database.

[0428] Specific operation: The server executes an SQL query based on the ID of the home appliance to obtain related information.

[0429] Input: Home appliance ID

[0430] Output: Records of operating procedures and troubleshooting information

[0431] Step 7:

[0432] The server transmits the acquired operating procedure to the terminal.

[0433] Specific operation: The server generates a JSON format response and sends it to the terminal.

[0434] Input: Records of operating procedures and troubleshooting information

[0435] Output: Response message to the terminal

[0436] Step 8:

[0437] The device provides users with operating instructions and troubleshooting methods using AR displays.

[0438] Specific operation: The device uses an AR library (e.g., ARKit, ARCore) to overlay operating instructions on the camera image.

[0439] Input: Operation procedure information of response message

[0440] Output: AR-displayed operating procedures and troubleshooting information

[0441] Example: When a user points the camera at an air conditioner, the screen will show how to use each button on the remote control and how to set a specific mode.

[0442] Step 9:

[0443] Users type their questions into the chatbot about how to use it or troubleshooting.

[0444] Specific behavior: The user types a question into the chat box within the app and presses the send button.

[0445] Input: User's question text

[0446] Output: Message to send to the chatbot

[0447] Step 10:

[0448] The terminal sends the user's question to the server.

[0449] Specific operation: The terminal sends text data to the server.

[0450] Input: User's question text

[0451] Output: Request message to the server

[0452] Step 11:

[0453] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[0454] How it works: The server calls the generative AI (e.g., GPT-3), passes the user's question as a prompt, and retrieves the generated answer.

[0455] Input: User's question text

[0456] Output: Generated answer text

[0457] Step 12:

[0458] The terminal displays the generated answer to the user.

[0459] Specific behavior: The device displays the generated text in the chat box and notifies the user.

[0460] Input: Generated answer text

[0461] Output: The answer text that is displayed to the user

[0462] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[0463] Step 13:

[0464] When a user types a question, the emotion engine analyzes the user's emotions.

[0465] Specific operation: The emotion engine analyzes input text and voice data and estimates the emotional state (e.g., confusion, irritation).

[0466] Input: User question text or voice data

[0467] Output: Emotion analysis results

[0468] Step 14:

[0469] The terminal transmits the analysis results to the server.

[0470] Specific operation: The device sends a request including emotion data to the server.

[0471] Input: Sentiment analysis results

[0472] Output: Request message to the server

[0473] Step 15:

[0474] The server adjusts the response content based on the emotional data.

[0475] Specific behavior: The server adjusts the tone and content of the response text depending on the user's emotional state (e.g., using emojis and softer language).

[0476] Input: Sentiment analysis results, generated answer text

[0477] Output: Adjusted answer text

[0478] Example: If a user expresses confusion or frustration, the server responds with a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[0479] Step 16:

[0480] The user opens a PDF instruction manual within the app to check the operating instructions.

[0481] Specific behavior: The user taps the instruction icon, which launches the PDF viewer.

[0482] Input: User tap input

[0483] Output: Launch PDF viewer and display PDF

[0484] Step 17:

[0485] The device sends the PDF contents to the server.

[0486] Specific operation: The device extracts the text data from the PDF and sends it to the server.

[0487] Input: PDF text data

[0488] Output: Request message to the server

[0489] Step 18:

[0490] The server uses generative AI to simplify the detailed content of the PDF.

[0491] Specific behavior: The server sends a prompt to the generation AI, instructing it to generate simplified text.

[0492] Input: PDF text data

[0493] Output: Simplified descriptive text

[0494] Step 19:

[0495] The terminal will display the generated brief description to the user.

[0496] What happens: The device retrieves the abbreviated text and displays it as an overlay on the PDF viewer.

[0497] Input: Simplified description text

[0498] Output: A simplified description that is displayed to the user.

[0499] Example: When you open a PDF containing detailed operating instructions for an oven range, you see a simple explanation: "Press the timer setting button, enter the time, and press the start button."

[0500] Step 20:

[0501] Users ask for information about specific features or new products.

[0502] Specific behavior: A user types a question such as "I want to know the features of the latest smart TVs."

[0503] Input: User's question text

[0504] Output: A message with the question text

[0505] Step 21:

[0506] The terminal sends this information to the server.

[0507] Specific operation: The terminal sends the question text to the server.

[0508] Input: User's question text

[0509] Output: Request message to the server

[0510] Step 22:

[0511] The server analyzes the data and generates new product suggestions and links to related online shopping sites.

[0512] Specific operation: The server analyzes the content of the question, retrieves related information from a product information database, and generates links to online shopping sites.

[0513] Input: User's question text

[0514] Output: New product proposals and online shopping site links

[0515] Step 23:

[0516] The device displays this information to the user.

[0517] Specific operation: The device displays the generated information and links in a chat box or dedicated display area.

[0518] Input: New product proposal and online shopping site link

[0519] Output: New product suggestions and online shopping links displayed to users

[0520] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[0521] (Application example 2)

[0522] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0523] Today's consumers often seek detailed information and reviews before purchasing a product, especially when shopping in a virtual store. However, no system exists that allows consumers to obtain detailed information while browsing products in a virtual store and receive appropriate support tailored to their emotions. Furthermore, conventional systems have difficulty responding appropriately when users become confused or frustrated, and lack means to improve the quality of the consumer experience. To address these issues, the present invention aims to dramatically improve the consumer experience in a virtual store.

[0524] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0525] In this invention, the server includes means for analyzing a user's question using a generation AI and generating an appropriate answer, means for displaying operating procedures for home appliances in AR using a smartphone camera, means for receiving user input and displaying an answer based on the analysis results, means for providing detailed product information, usage instructions, and reviews in a virtual store, and means for analyzing the user's emotional state and providing appropriate assistance according to the user's emotion. This enables consumers to intuitively obtain detailed information while browsing products in a virtual store and receive appropriate support according to their emotional state.

[0526] "Generative AI" is an artificial intelligence technology that analyzes user input and questions and automatically generates appropriate information and answers.

[0527] "AR display" is a technology that uses augmented reality technology to overlay digital information on actual images.

[0528] "User input" refers to the act of a user giving instructions or questions to the system via text or voice.

[0529] "Analysis results" are information obtained as a result of the generative AI analyzing the user's input and questions.

[0530] A "virtual store" is an online store that is a virtual shopping space set up on the Internet where users can browse and purchase items.

[0531] "Detailed product information" refers to information that users want to know before purchasing, such as product specifications, functions, instructions for use, and price.

[0532] "Reviews" refer to ratings and opinions of consumers who have already used a product.

[0533] "Emotional state" indicates the user's current psychological state or emotion, and is a category of emotions such as confusion, irritation, and excitement.

[0534] "Appropriate assistance" is specific and useful support and advice provided according to the user's emotional state.

[0535] This invention realizes a system that provides product information and enhances user support in a virtual store by combining generative AI, augmented reality (AR) technology, and an emotion recognition engine. Specific embodiments of the system are described below.

[0536] First, users launch a dedicated application on their smartphone or smart glasses. This application includes a camera function and displays camera images in real time, allowing users to take photos of products in the virtual store and obtain detailed information.

[0537] Hardware and software used

[0538] 1. Hardware

[0539] Smartphone or smart glasses camera: Captures video in real time and is used to recognize the product being photographed.

[0540] Server: Hosts the generative AI model, sentiment analysis engine, and database, and performs analytical processing.

[0541] 2. Software

[0542] OpenCV: Captures camera images and uses them for product recognition.

[0543] Transformers (transformers package): Used to perform generative AI and sentiment analysis.

[0544] HTTP request: Communicate with external APIs or product databases to obtain product information.

[0545] Data processing and calculation

[0546] Camera video analysis:

[0547] The device analyzes the real-time video captured by the user and recognizes the products. To do this, it runs an object detection algorithm and, once the product is identified, sends its name to the server.

[0548] Get product information:

[0549] The server receives the product name and retrieves the product's details, usage instructions, and reviews from a product database or external API. The retrieved information is sent to the user's device and displayed on the screen.

[0550] Sentiment analysis and providing appropriate support:

[0551] When a user enters information via text or voice, an emotion recognition engine analyzes their emotional state. If the input indicates frustration or confusion, the system generates a thoughtful response that reflects that emotion. The response is then sent to the user's device and displayed on the screen.

[0552] Specific examples

[0553] Suppose a user is walking around a virtual store and looking at products using smart glasses. For example, if the user points the camera at a smart TV and wants to get more information about it, they might enter the following prompt:

[0554] What features does this smart TV have?

[0555] Actions based on sentiment analysis results:

[0556] If the user's input indicates frustration, the system generates a considerate response such as:

[0557] We apologize for any inconvenience. This smart TV has high resolution and voice recognition. Purchase here (link).

[0558] Under normal circumstances, this will produce a response like this:

[0559] This smart TV has high resolution and voice recognition. Buy it here.

[0560] This allows users to intuitively obtain product information in a virtual store and receive appropriate support based on their emotions. This system not only dramatically improves the consumer experience, but also encourages purchases in the virtual store.

[0561] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0562] Step 1:

[0563] The user launches the dedicated application on their smartphone or smart glasses, which activates the camera function and prepares to display images in real time, allowing the images captured by the user to be displayed on the screen.

[0564] Input: User application launch

[0565] Output: Real-time video display

[0566] Step 2:

[0567] The user points the camera at a product in a virtual store, and the device analyzes the camera image and uses an object detection algorithm to recognize the product. The recognized product information is identified and the product name is extracted.

[0568] Input: Camera image

[0569] Output: Recognized product name

[0570] Step 3:

[0571] The device sends the extracted product name to the server, which retrieves detailed information, usage instructions, and reviews from a product database or external API based on the product name. The retrieved information is formatted on the server side and sent to the device.

[0572] Input: Product name

[0573] Output: Product details (description, usage, reviews, etc.)

[0574] Step 4:

[0575] The terminal displays the acquired product details to the user, who can then check the product details on the screen and request more information if necessary.

[0576] Input: Product details information

[0577] Output: Display detailed product information

[0578] Step 5:

[0579] The user inputs a question via text or voice. The emotion recognition engine analyzes this input and determines the user's emotional state (e.g., irritation, confusion). The analyzed emotion information and the user's question are sent to the server.

[0580] Input: User text and voice input

[0581] Output: Parsed sentiment information and question

[0582] Step 6:

[0583] The server uses a generative AI model to generate an appropriate response based on the user's emotional state and question. If the emotional state is irritation, it is converted into a considerate sentence. The generated response is sent to the device.

[0584] Input: Parsed emotion information and question

[0585] Output: The generated answer

[0586] Step 7:

[0587] The device displays the generated answer to the user, who can then check the answer on the screen and receive appropriate support.

[0588] Input: Generated Answer

[0589] Output: Show answer

[0590] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0591] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0592] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0593] [Second embodiment]

[0594] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0595] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0596] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0597] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0598] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0599] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0600] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0601] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0602] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0603] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0604] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0605] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0606] This invention is a system that revolutionizes home appliance instruction manuals by utilizing generative AI and augmented reality (AR) technology. This system allows users to easily access operation and troubleshooting information for home appliances through a smartphone app, enabling them to operate and repair appliances intuitively and quickly.

[0607] System program flow

[0608] 1. Launch the app

[0609] Users simply launch a dedicated app on their smartphone to enable the camera function.

[0610] The device prepares to display the camera image in real time.

[0611] 2. Providing AR operation instructions

[0612] The user points the camera at a home appliance.

[0613] The device analyzes camera footage and recognizes home appliances.

[0614] Based on the recognized home appliances, the device will provide the user with operating procedures and methods using AR displays.

[0615] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[0616] 3. Use AI support chatbots

[0617] Users type their questions into the chatbot about how to use it or troubleshooting.

[0618] The terminal receives the user's question and sends it to the server.

[0619] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[0620] The device receives the generated answer and displays it to the user.

[0621] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[0622] 4. Simplified instructions

[0623] The user opens a PDF instruction manual within the app to check the operating instructions.

[0624] The device sends the PDF contents to the server.

[0625] The server uses generative AI to convert the content into a concise description in natural language.

[0626] The terminal will display the generated brief description to the user.

[0627] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[0628] 5. New product proposals and e-commerce site referrals

[0629] Users ask for information about specific features or new products.

[0630] The terminal sends this information to the server.

[0631] The server analyzes the information and generates relevant new product suggestions and e-commerce site purchase links.

[0632] The device displays this information to the user.

[0633] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[0634] This system allows users to quickly and intuitively understand and respond to operating and troubleshooting methods for home appliances. For manufacturers, this system configuration not only improves the quality of user support, but also helps promote new products and sales.

[0635] The processing flow will be explained below.

[0636] Step 1:

[0637] The user launches the dedicated app on their smartphone, which then begins preparations to enable the camera function.

[0638] Step 2:

[0639] The device displays camera footage in real time and displays the footage the user takes on the screen.

[0640] Step 3:

[0641] Users point their camera at their appliances and use the system to get information on how to operate them or if they need to troubleshoot.

[0642] Step 4:

[0643] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[0644] Step 5:

[0645] The device sends information about the identified home appliances to a server, which then retrieves corresponding operating procedures and troubleshooting data from a database.

[0646] Step 6:

[0647] The server then sends the acquired operation instructions to the device, which then prepares the device for displaying the correct operation guide in AR (Augmented Reality).

[0648] Step 7:

[0649] The device overlays the received operating instructions and troubleshooting methods on the camera image in real time, allowing users to intuitively check how to operate their home appliances through the camera screen.

[0650] Step 8:

[0651] To get more detailed information, users can use the chatbot function installed in the app to input questions, such as "I don't know how to use XX" or "XX doesn't work."

[0652] Step 9:

[0653] The device sends the user's input question in text format to the server, which then analyzes the question using natural language processing (NLP) technology.

[0654] Step 10:

[0655] The server generates appropriate answers from a database based on the analysis results, and this process may involve the use of generative AI.

[0656] Step 11:

[0657] The server generates an appropriate response and sends it to the terminal, which then displays the received response to the user.

[0658] Step 12:

[0659] The user can then review the displayed answers and take any necessary action or troubleshooting steps, such as cleaning the filter or pressing the reset button.

[0660] Step 13:

[0661] If a user opens a PDF manual for a particular home appliance within the app, that's possible. If the PDF is difficult to understand, the user will ask for a simplified explanation.

[0662] Step 14:

[0663] The device sends the PDF content to the server, which then uses generative AI to convert the PDF's detailed content into simplified, natural language.

[0664] Step 15:

[0665] The server then sends the generated natural language brief instructions to the terminal, which then displays them to the user, allowing the user to easily understand complex operating procedures.

[0666] Step 16:

[0667] When users have questions about new product information or specific features, they enter that information through the chatbot.

[0668] Step 17:

[0669] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[0670] Step 18:

[0671] The server sends the new product information and a purchase link to the device, which displays it to the user. The user can check the new product information and purchase it directly from the purchase link if necessary.

[0672] These process steps allow users to quickly and intuitively operate home appliances, troubleshoot, explore new products, and more.

[0673] Example 1

[0674] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0675] Traditionally, instruction manuals and troubleshooting guides for home appliances have mainly been in paper or static PDF format, making it difficult to operate and repair products intuitively and quickly. Furthermore, when users searched for information on new products, they had limited means of quickly accessing the appropriate information. This often resulted in users spending time using or repairing the product, causing inconvenience. Furthermore, manufacturers faced challenges in streamlining user support and promoting new products.

[0676] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0677] In this invention, the server includes means for analyzing a user's question using a generation AI and generating an appropriate answer, means for displaying operating procedures for the device in augmented reality using a video display of the mobile device, and means for receiving user input and displaying the answer based on the analysis results. This allows users to quickly and intuitively understand how to operate and troubleshoot home appliances and easily obtain information on new products.

[0678] "Generative AI" is a technology that uses advanced artificial intelligence algorithms to analyze input data and automatically generate appropriate information and answers.

[0679] A "video device for a mobile device" is a device that has the function of acquiring and displaying images of the real world using a camera or display device installed on a smartphone, tablet, etc.

[0680] "Augmented reality display (AR display)" is a technology that overlays digital information on images of the real world, and is a means of providing users with intuitive information.

[0681] "Operating instructions" means a set of instructions or steps for the proper operation of a home appliance, explaining how to access the product's features.

[0682] "User input" refers to the text data and commands provided by the user through the application, and serves as the basis for the system to respond appropriately based on the analysis results.

[0683] A "function search system" is an information search technology that provides related information in response to a user's request when searching for specific function or product information.

[0684] "PDF Content" means the text and graphics data contained within a Portable Document Format (PDF) file, developed by Adobe Systems Incorporated, a document format that can be stored and viewed electronically.

[0685] "Natural language" refers to the language used by humans on a daily basis, along with the technology to convert it into a format that is easy for machines to understand and process the data.

[0686] "New product suggestions" are a way to introduce relevant new products based on a user's specific needs and interests, providing information and encouraging them to purchase.

[0687] A "purchase link" refers to a URL on the Internet that a user can access to purchase a product, and is a link that directs the user to an e-commerce site.

[0688] This invention is a system that revolutionizes home appliance instruction manuals by utilizing generative AI and augmented reality (AR) technology. Users can easily access operating instructions and troubleshooting information for home appliances through a smartphone app, enabling them to operate and repair appliances intuitively and quickly. The following describes the system's program processing and the hardware and software used.

[0689] Hardware and Software Use

[0690] Device: Use a smartphone or tablet with its built-in camera and display device.

[0691] Server: A server with high-performance computing resources is used to perform natural language processing and data analysis.

[0692] Generative AI model: A natural language processing model using artificial intelligence algorithms (e.g., GPT).

[0693] AR technology: Augmented reality software frameworks such as ARKit (Apple) and ARCore (Google).

[0694] Explanation of program processing

[0695] 1. Launch the app:

[0696] The user launches a dedicated app on their smartphone.

[0697] The device detects that the app has been launched and prepares to enable the camera function.

[0698] Example: When a user taps an app icon, the app launches and displays a camera view.

[0699] 2. AR operation instructions provided:

[0700] Users point their smartphone camera at home appliances.

[0701] The device analyzes camera footage in real time and recognizes home appliances.

[0702] Based on the recognized home appliances, the device will display operating procedures and methods on the screen using AR.

[0703] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes are displayed on the screen.

[0704] 3. Use AI-supported chatbots:

[0705] Users type questions into the chatbot about how to use it and troubleshooting.

[0706] The terminal sends the user's question to the server.

[0707] The server uses generative AI to analyze the user's question and generate an appropriate answer.

[0708] The device receives the generated answer and displays it to the user.

[0709] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[0710] Example prompt: "What should I do if my washing machine won't start?"

[0711] 4. Simplified instructions:

[0712] Users can open the instruction PDF within the app to check the operating instructions.

[0713] The device sends the PDF contents to the server.

[0714] The server uses generative AI to convert the content into a concise description in natural language.

[0715] The terminal will display the generated brief description to the user.

[0716] Example: When you open a PDF containing detailed operating instructions for an oven range, simple instructions such as "Press the timer setting button, enter the time, and press the start button" appear on the screen.

[0717] Example prompt: "Please briefly explain the oven operating procedure described in this PDF."

[0718] 5. New product proposals and e-commerce site referrals:

[0719] Users ask for information about specific features or new products.

[0720] The terminal sends this information to the server.

[0721] The server uses generative AI to analyze the information and generate relevant new product suggestions and purchase links.

[0722] The device displays this information to the user.

[0723] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[0724] Example prompt: "What are the features of the latest smart TVs?"

[0725] This system allows users to quickly and intuitively access information on how to operate and troubleshoot home appliances and new product information, and is also useful for manufacturers to streamline user support and promote new products.

[0726] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0727] Step 1: Launch the app

[0728] User: Launch the dedicated app on your smartphone.

[0729] What happens: The user taps on the app icon.

[0730] Input: The user's touch.

[0731] Output: The app starts.

[0732] Device: Detects that the app has been launched and prepares to enable the camera function.

[0733] What happens: The camera initialization process happens in the background and the camera view appears in the app.

[0734] Input: App launch signal.

[0735] Output: Camera view display.

[0736] Step 2: Provide AR operation instructions

[0737] User: Point your smartphone camera at a home appliance.

[0738] Specific action: To point the camera at a home appliance, hold the smartphone in your hand and frame the product.

[0739] Input: Home appliance footage.

[0740] Output: Acquisition of video data.

[0741] Terminal: Analyzes camera footage in real time and recognizes home appliances.

[0742] Specific operation: An image recognition algorithm is activated, analyzing the video data to detect the shape and characteristics of the home appliance.

[0743] Input: Video data.

[0744] Output: Recognition results of home appliances.

[0745] Terminal: Based on the recognized home appliance, the device displays operating procedures and methods on the screen using AR.

[0746] Specific behavior: Operation guides related to home appliances are overlaid on the screen.

[0747] Input: Recognition results for home appliances.

[0748] Output: AR display operation procedure.

[0749] Step 3: Use an AI-powered chatbot

[0750] Users: Type questions into the chatbot for instructions or troubleshooting.

[0751] Specific actions: Tap the chat button in the app, enter the question text and send it.

[0752] Input: Question text.

[0753] Output: The question is sent.

[0754] Terminal: Receives the user's question and sends it to the server.

[0755] Specific operation: Sends the question text to the server via an HTTP request.

[0756] Input: Question text.

[0757] Output: Sending question data.

[0758] Server: Uses generative AI to analyze the user's question and generate an appropriate answer.

[0759] What it does: The input text is parsed, answers are extracted from relevant databases, and formatted in the appropriate natural language format.

[0760] Input: Question data.

[0761] Output: Response data.

[0762] Terminal: Receives the generated answer and displays it to the user.

[0763] Specific behavior: The reply text received from the server is displayed in the chat window.

[0764] Input: Response data.

[0765] Output: Display the answer.

[0766] Step 4: Simplify the instructions

[0767] User: Open the instruction PDF within the app to view instructions.

[0768] Specific operation: To view an instruction PDF, select the PDF you want from the list within the app.

[0769] Input: Select PDF file.

[0770] Output: PDF content display.

[0771] Device: Send the PDF content to the server.

[0772] Specific operation: Extract the contents of the PDF file as text data and upload it to the server.

[0773] Input: PDF content.

[0774] Output: Sending text data.

[0775] Server: Uses generative AI to convert the content into concise descriptions in natural language.

[0776] What it does: A generative AI model is activated, analyzing the text data and converting it into a concise description.

[0777] Input: Text data.

[0778] Output: A short description.

[0779] Terminal: Display the generated short description to the user.

[0780] Specific behavior: The generated short description is displayed in the PDF viewer.

[0781] Input: A short description.

[0782] Output: Display a short description.

[0783] Step 5: Propose new products and direct them to the EC site

[0784] Users: Ask for information about specific features or new products.

[0785] What it does: Use the search box within the app to enter specific product information.

[0786] Input: Your search query.

[0787] Output: Submitting a search query.

[0788] Device: Sends this information to the server.

[0789] Specific operation: The entered search query is sent to the server via an HTTP request.

[0790] Input: Your search query.

[0791] Output: Submitting a search query.

[0792] Server: Analyzes the information using generative AI and generates relevant new product suggestions and purchase links.

[0793] What it does: The server parses the search query and generates relevant product information and purchase links.

[0794] Input: Your search query.

[0795] Output: New product suggestions and purchase links.

[0796] Terminal: Display this information to the user.

[0797] What happens: The generated suggestions and links are displayed in the results area within the app.

[0798] Input: New product suggestions and purchase links.

[0799] Output: Display information.

[0800] (Application example 1)

[0801] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0802] There is a wide range of information on the operation, maintenance, and troubleshooting of factory machinery, and conventional manuals alone make it difficult for users to respond quickly and accurately.In addition, there are limited means to quickly provide information on new tools and parts, and these issues need to be improved.

[0803] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0804] In this invention, the server includes a means for analyzing a user's question using a generative AI and generating an appropriate answer, a means for displaying operating procedures for the machine in AR using a camera, a means for receiving user input and displaying answers based on the analysis results, and a means for providing a machine function search system. This allows users to quickly and intuitively understand and respond to operating methods and troubleshooting of the machine. It also includes a means for suggesting new products and necessary parts related to the analyzed machine and providing links to e-commerce sites, allowing users to quickly obtain the necessary information.

[0805] "Generative AI" refers to an artificial intelligence model that analyzes user input and generates appropriate answers or information.

[0806] A "camera" is an optical device for visually capturing mechanical equipment.

[0807] "Machinery" refers to mechanical equipment and robots used in factories.

[0808] An "operating procedure" refers to a series of actions or procedures for operating a machine or device correctly.

[0809] "AR display" refers to the use of augmented reality technology to overlay virtual information onto real-world images.

[0810] "User input" refers to information such as operations or questions that a user makes to the system.

[0811] "Analysis results" refer to the answers and information derived by the generating AI based on user input.

[0812] A "function search system" is a system that allows you to search and obtain information about the functions and characteristics of mechanical devices.

[0813] An "AI support chatbot" is an interactive program that uses artificial intelligence to answer users' questions and provide support.

[0814] "Troubleshooting information" refers to information that provides solutions or methods of dealing with problems or malfunctions that occur in mechanical equipment.

[0815] "PDF instruction manual content" refers to an electronic document that contains detailed instructions for operating and maintaining a machine or device.

[0816] "Natural language simplification" refers to the process of expressing technical content in simple, everyday language so that it is easy for users to understand.

[0817] "New products" are new tools or parts that are newly introduced to the market.

[0818] An "e-commerce site" is a website that sells goods and services over the Internet.

[0819] This invention provides a system for intuitive and rapid operation, maintenance, and troubleshooting of factory machinery. The system utilizes generative AI models and augmented reality (AR) technology to allow users to access machinery information using their smartphones.

[0820] Program processing

[0821] Hardware and software used

[0822] Smartphone: High-resolution camera, AR-enabled

[0823] Server: Equipped with high-performance CPU / GPU, natural language processing (NLP) software, generative AI models

[0824] Frameworks: TensorFlow, PyTorch

[0825] API: Google Vision API, OpenAI API

[0826] Data processing and calculation

[0827] 1. Camera image analysis:

[0828] The server uses the Google Vision API to recognize mechanical devices from the smartphone camera image.

[0829] Example: When a user points their smartphone camera at a factory robot, the robot's model number and type are automatically recognized.

[0830] 2. Natural Language Processing:

[0831] The server analyzes the user's question using the OpenAI API and generates an appropriate solution.

[0832] Example: When a user asks, "What should I do when error code E02 appears?", the server generates a solution such as, "Error code E02 indicates that the machine is overheating. Make sure the cooling system is working properly and, if possible, turn the power off and restart."

[0833] 3. Manual Simplification Using Generative AI:

[0834] The server uses a generation AI (GPT-3) to simplify the PDF of the detailed manual in natural language.

[0835] Example: Simplifying a PDF operating manual for a machine generates simple instructions such as "Press the emergency stop button and press the restart button after an interruption."

[0836] 4. New product proposal:

[0837] The server suggests new products and necessary parts related to the analyzed machinery and provides links to e-commerce sites.

[0838] Example: The server analyzes a user's question, "I want to know the features of the latest automation tool," and provides information such as, "This automation tool is equipped with advanced sensor functions and a real-time monitoring system. You can purchase it here (link)."

[0839] Specific examples (prompt examples)

[0840] Example prompt sentence:

[0841] "What should I do when error code E02 appears?"

[0842] In this way, this invention provides a system that streamlines the provision of information on how to operate and troubleshoot machinery in a factory, allowing users to respond intuitively. It also quickly provides information on necessary new products and parts, and easily guides users to e-commerce sites.

[0843] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0844] Step 1:

[0845] The user launches a dedicated app on their smartphone and enables the camera function. The device prepares to display camera footage in real time, allowing the user to capture machinery and equipment with the camera.

[0846] Step 2:

[0847] The user points their smartphone camera at a machine. The device analyzes the camera image and recognizes the machine. The Google Vision API is used to do this, and the model number and type of the machine are sent to the device as analysis results.

[0848] Step 3:

[0849] The device uses AR to display operation procedures and maintenance methods for the recognized machinery. This allows the user to proceed with the work while visually checking the operation procedures. The displayed content is based on data provided in advance by the server.

[0850] Step 4:

[0851] When a user has a question about a machine, they can input it using the chatbot function within the app. The device then sends the question to the server. A specific example of input is a prompt such as, "Please tell me what to do when error code E02 appears."

[0852] Step 5:

[0853] The server uses the OpenAI API to analyze the user's question and generate an appropriate solution. The generated answer is sent from the server to the device and displayed to the user. For example, the user may be provided with information such as, "Error code E02 indicates that the machine is overheating. Please make sure the cooling system is operating properly and, if possible, turn the power off and restart."

[0854] Step 6:

[0855] When a user opens a PDF of a detailed operating manual within the app, the device sends the contents of the PDF to the server. The server uses generative AI (GPT-3) to simplify the PDF contents in natural language and sends the result to the device. The device then displays the simplified instructions to the user. For example, a simple instruction such as "Press the emergency stop button, then press the restart button after interruption" is displayed.

[0856] Step 7:

[0857] When a user requests information about a new tool or part, the device sends the request to the server. Based on the analysis results, the server generates relevant new product information and links to e-commerce sites, and sends them to the device. The device then displays this information to the user. For example, the device may provide information such as, "This automation tool is equipped with advanced sensor functions and a real-time monitoring system. Purchase here (link)."

[0858] Through these processing steps, the present invention provides a system that streamlines the operation and troubleshooting of machinery in a factory, allowing users to respond quickly and intuitively.

[0859] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0860] This invention is a system that revolutionizes home appliance instruction manuals by combining generative AI, augmented reality (AR) technology, and an emotion engine that recognizes user emotions. This system allows users to easily access instructions and troubleshooting information for home appliances through a smartphone app, and also provides support based on the user's emotional state, creating a more personalized user experience.

[0861] System program flow

[0862] 1. Launch the app

[0863] The user launches the dedicated app on their smartphone, which activates the camera function and simultaneously activates the emotion engine.

[0864] The device displays the camera footage in real time and prepares the footage the user captures to be displayed on the screen.

[0865] 2. Providing AR operation instructions

[0866] The user points the camera at a home appliance.

[0867] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[0868] The device sends information about the recognized home appliances to the server, and the server retrieves corresponding operating procedures and troubleshooting data from a database.

[0869] The server sends the acquired operating procedures to the device, and the device displays the operating procedures and troubleshooting methods to the user in AR.

[0870] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[0871] 3. Use AI support chatbots

[0872] Users type their questions into the chatbot about how to use it or troubleshooting.

[0873] The terminal receives the user's question and sends it to the server.

[0874] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[0875] The device receives the generated answer and displays it to the user.

[0876] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[0877] 4. Utilizing the Emotion Engine

[0878] When a user enters a question via text or voice, the emotion engine analyzes the user's emotions.

[0879] The device sends the analysis results to the server, and the server adjusts the response content based on the emotional data.

[0880] Example: If the user displays emotions that indicate confusion or annoyance, the server might respond with a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[0881] 5. Simplified instructions

[0882] The user opens a PDF instruction manual within the app to check the operating instructions.

[0883] The device sends the PDF content to the server, which then uses generative AI to convert the PDF's detailed content into simplified, natural language.

[0884] The terminal will display the generated brief description to the user.

[0885] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[0886] 6. New product proposals and e-commerce site referrals

[0887] Users ask for information about specific features or new products.

[0888] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[0889] The device displays this information to the user.

[0890] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[0891] This system allows users to quickly and intuitively understand and respond to how to operate and troubleshoot home appliances. In addition, responses are given that take the user's emotions into consideration, providing more personalized support. For manufacturers, this system configuration not only improves the quality of user support, but is also useful for promoting new products and sales.

[0892] The processing flow will be explained below.

[0893] Step 1:

[0894] The user launches the dedicated app on their smartphone, which activates the camera function and simultaneously activates the emotion engine.

[0895] Step 2:

[0896] The device displays the camera footage in real time and prepares the footage the user captures to be displayed on the screen.

[0897] Step 3:

[0898] The user points the camera at a home appliance.

[0899] Step 4:

[0900] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[0901] Step 5:

[0902] The device sends information about the recognized home appliances to the server.

[0903] Step 6:

[0904] The server retrieves the corresponding operating procedures and troubleshooting data from the database.

[0905] Step 7:

[0906] The server sends the acquired operating procedures to the device, and the device displays the operating procedures and troubleshooting methods to the user in AR.

[0907] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[0908] Step 8:

[0909] Users type their questions into the chatbot about how to use it or troubleshooting.

[0910] Step 9:

[0911] The terminal sends the user's question in text format to the server.

[0912] Step 10:

[0913] The server uses natural language processing techniques to analyze the user's question.

[0914] Step 11:

[0915] Based on the analysis results, the server generates appropriate answers from the database and also analyzes the user's emotions using an emotion engine.

[0916] Step 12:

[0917] The device sends the user's voice and text input data to the emotion engine in real time and analyzes emotions.

[0918] Step 13:

[0919] The emotion engine determines the user's emotional state based on the user's emotional data.

[0920] Step 14:

[0921] The server uses the analysis results of the emotion engine to generate a response message that is tailored to the user's emotional state.

[0922] Example: If a user expresses confusion or frustration, the server tailors and generates a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[0923] Step 15:

[0924] The server sends the generated response message to the terminal, which displays it to the user.

[0925] Step 16:

[0926] The user can refer to the displayed answers to perform the necessary operations or troubleshoot.

[0927] Step 17:

[0928] When a user opens a PDF manual for a particular home appliance within the app, the device sends the contents of the PDF to the server.

[0929] Step 18:

[0930] The server uses generative AI to convert the detailed content of the PDF into simplified, natural language.

[0931] Step 19:

[0932] The server sends the generated natural language short description to the terminal, which displays it to the user.

[0933] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[0934] Step 20:

[0935] When users have questions about new product information or specific features, they enter that information through the chatbot.

[0936] Step 21:

[0937] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[0938] Step 22:

[0939] The server sends the new product information and a purchase link to the device, which displays it to the user. The user can check the new product information and purchase it directly from the purchase link if necessary.

[0940] In this way, users can quickly and intuitively understand and respond to how to operate and troubleshoot their home appliances, and also receive personalized, emotionally sensitive support.

[0941] Example 2

[0942] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0943] Conventional instruction manuals for home appliances have scattered information, making it difficult for users to quickly obtain the information they need. Furthermore, they lack a mechanism for reading emotions and responding appropriately, making it difficult to alleviate user confusion and frustration. Furthermore, they lack a means for providing new product information and links to related e-commerce sites, making it difficult to stimulate users' desire to purchase. The purpose of this invention is to solve these problems.

[0944] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing a user's question using a generation AI and generating an appropriate answer, a means for displaying operating procedures for the device in augmented reality using a camera of a mobile terminal, and a means for analyzing the user's emotions and adjusting the answer content based on the analysis results. This allows the user to quickly and intuitively understand the operating procedures and receive personalized support that takes emotions into consideration. Furthermore, it also makes it easy to promote new products and direct users to related online shopping sites.

[0945] "Generative AI" is an artificial intelligence technology that analyzes questions asked in natural language by users and automatically generates appropriate answers.

[0946] "Mobile terminal" refers to a portable computing device such as a smartphone or tablet.

[0947] A "camera" is a device that converts optical images into digital data for display or analysis in real time.

[0948] "Augmented reality display" is a technology that displays digital information overlaid on real-world images.

[0949] "Equipment" broadly refers to electronic devices such as home appliances.

[0950] "Sentiment analysis" refers to the process of inferring a user's emotional state from their text or voice.

[0951] "Adjusting the content of the response based on the analysis results" means appropriately changing the tone and content of the generated response depending on the user's emotional state.

[0952] "Promotional Information" means information intended to promote new products or services.

[0953] "Related Shopping Sites" means websites where users can purchase products online.

[0954] This invention is a system that combines generative AI, augmented reality (AR) technology, and an emotion engine that recognizes user emotions, and provides operation instructions and troubleshooting information for home appliances to improve the user experience. This system mainly operates through the collaboration of three parties: a server, a device, and a user.

[0955] First, the user launches the dedicated app on their smartphone or tablet, which activates the camera function and activates the emotion engine. The device displays the camera footage in real time and shows the footage the user takes on the screen. At this time, the footage captured by the device's camera is analyzed using an object detection algorithm (e.g., YOLO or ResNet).

[0956] Next, when the user points the camera at a home appliance, the device recognizes the appliance and sends the recognition results to the server. The server retrieves the relevant operating procedures and troubleshooting information from a database (e.g., MySQL, PostgreSQL) and sends the information back to the device. This allows the device to display the operating procedures or troubleshooting methods on the camera image in the form of augmented reality (AR). For example, when the user points the camera at an air conditioner, the screen will show how to use each button on the remote control and how to set a specific mode.

[0957] Additionally, users can input questions about operation methods and troubleshooting into a dedicated chatbot. The device sends these questions as text data to the server, which then uses natural language processing technology (e.g., GPT-3, BERT) to analyze the user's question and generate an appropriate answer. The generated answer is then sent back to the device and displayed to the user in chat format. For example, if a user asks, "My washing machine isn't working," the generating AI will respond with specific instructions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[0958] The emotion engine can also be used to analyze the emotions expressed when a user types a question. The device also sends the emotion analysis results to the server, which then uses the results to tailor the answer to the user's emotional state. For example, if the user expresses confusion or irritation, the server can generate a considerate response such as, "We apologize for the inconvenience. Could you please try the following steps?"

[0959] The contents of PDF manuals can also be simplified and displayed. When a user opens a PDF manual within the app, the device sends the contents of the PDF as text data to the server. The server then uses generative AI to simplify this text data, converts it into natural language, and sends it back to the device. The device then displays the simplified instructions to the user. For example, when a user opens a PDF containing detailed operating instructions for a microwave oven, simple instructions such as "Press the timer setting button, enter the time, and press the start button" are displayed.

[0960] Furthermore, when a user requests information about a specific feature or new product, the device sends this input information to the server. The server analyzes the information, generates new product suggestions and links to related online shopping sites, and sends them to the device. The device then displays this information to the user. For example, if a user asks, "I want to know the features of the latest smart TV," the server might respond, "This smart TV has high resolution and voice recognition capabilities. You can purchase it here (link)."

[0961] This system allows users to quickly and intuitively understand and respond to how to operate and troubleshoot their home appliances, and also provides personalized support that takes into account the user's emotions, resulting in a more satisfying user experience.

[0962] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0963] Step 1:

[0964] The user launches a dedicated app on their smartphone.

[0965] Specific action: The user taps the app icon on the smartphone's home screen.

[0966] Input: Tap input from the user

[0967] Output: Launching the app and displaying the initial screen

[0968] Step 2:

[0969] The device will enable the app's camera function and emotion engine.

[0970] Specific operation: The device will launch the camera function and ask the user for permission to use the camera. At the same time, the emotion engine will also launch in the background.

[0971] Input: App launch signal, camera access permission from user

[0972] Output: Streaming camera footage and emotion engine status

[0973] Step 3:

[0974] The user points the camera at a home appliance.

[0975] Specific action: The user places a home appliance within the camera frame.

[0976] Input: Camera video data

[0977] Output: Video frame showing home appliances

[0978] Step 4:

[0979] The device analyzes camera footage in real time and runs object detection algorithms to recognize home appliances.

[0980] How it works: An object detection algorithm (e.g., YOLO or ResNet) extracts features from video frames and matches them with known home appliances in a database.

[0981] Input: Camera image frame

[0982] Output: ID or feature information of the recognized home appliance

[0983] Step 5:

[0984] The device sends information about the recognized home appliances to the server.

[0985] Specific operation: The device sends a request including the home appliance ID and image data to the server.

[0986] Input: ID or characteristic information of home appliance

[0987] Output: Request message to the server

[0988] Step 6:

[0989] The server retrieves the corresponding operating procedures and troubleshooting data from the database.

[0990] Specific operation: The server executes an SQL query based on the ID of the home appliance to obtain related information.

[0991] Input: Home appliance ID

[0992] Output: Records of operating procedures and troubleshooting information

[0993] Step 7:

[0994] The server transmits the acquired operating procedure to the terminal.

[0995] Specific operation: The server generates a JSON format response and sends it to the terminal.

[0996] Input: Records of operating procedures and troubleshooting information

[0997] Output: Response message to the terminal

[0998] Step 8:

[0999] The device provides users with operating instructions and troubleshooting methods using AR displays.

[1000] Specific operation: The device uses an AR library (e.g., ARKit, ARCore) to overlay operating instructions on the camera image.

[1001] Input: Operation procedure information of response message

[1002] Output: AR-displayed operating procedures and troubleshooting information

[1003] Example: When a user points the camera at an air conditioner, the screen will show how to use each button on the remote control and how to set a specific mode.

[1004] Step 9:

[1005] Users type their questions into the chatbot about how to use it or troubleshooting.

[1006] Specific behavior: The user types a question into the chat box within the app and presses the send button.

[1007] Input: User's question text

[1008] Output: Message to send to the chatbot

[1009] Step 10:

[1010] The terminal sends the user's question to the server.

[1011] Specific operation: The terminal sends text data to the server.

[1012] Input: User's question text

[1013] Output: Request message to the server

[1014] Step 11:

[1015] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[1016] How it works: The server calls the generative AI (e.g., GPT-3), passes the user's question as a prompt, and retrieves the generated answer.

[1017] Input: User's question text

[1018] Output: Generated answer text

[1019] Step 12:

[1020] The terminal displays the generated answer to the user.

[1021] Specific behavior: The device displays the generated text in the chat box and notifies the user.

[1022] Input: Generated answer text

[1023] Output: The answer text that is displayed to the user

[1024] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[1025] Step 13:

[1026] When a user types a question, the emotion engine analyzes the user's emotions.

[1027] Specific operation: The emotion engine analyzes input text and voice data and estimates the emotional state (e.g., confusion, irritation).

[1028] Input: User question text or voice data

[1029] Output: Emotion analysis results

[1030] Step 14:

[1031] The terminal transmits the analysis results to the server.

[1032] Specific operation: The device sends a request including emotion data to the server.

[1033] Input: Sentiment analysis results

[1034] Output: Request message to the server

[1035] Step 15:

[1036] The server adjusts the response content based on the emotional data.

[1037] Specific behavior: The server adjusts the tone and content of the response text depending on the user's emotional state (e.g., using emojis and softer language).

[1038] Input: Sentiment analysis results, generated answer text

[1039] Output: Adjusted answer text

[1040] Example: If a user expresses confusion or frustration, the server responds with a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[1041] Step 16:

[1042] The user opens a PDF instruction manual within the app to check the operating instructions.

[1043] Specific behavior: The user taps the instruction icon, which launches the PDF viewer.

[1044] Input: User tap input

[1045] Output: Launch PDF viewer and display PDF

[1046] Step 17:

[1047] The device sends the PDF contents to the server.

[1048] Specific operation: The device extracts the text data from the PDF and sends it to the server.

[1049] Input: PDF text data

[1050] Output: Request message to the server

[1051] Step 18:

[1052] The server uses generative AI to simplify the detailed content of the PDF.

[1053] Specific behavior: The server sends a prompt to the generation AI, instructing it to generate simplified text.

[1054] Input: PDF text data

[1055] Output: Simplified descriptive text

[1056] Step 19:

[1057] The terminal will display the generated brief description to the user.

[1058] What happens: The device retrieves the abbreviated text and displays it as an overlay on the PDF viewer.

[1059] Input: Simplified description text

[1060] Output: A simplified description that is displayed to the user.

[1061] Example: When you open a PDF containing detailed operating instructions for an oven range, you see a simple explanation: "Press the timer setting button, enter the time, and press the start button."

[1062] Step 20:

[1063] Users ask for information about specific features or new products.

[1064] Specific behavior: A user types a question such as "I want to know the features of the latest smart TVs."

[1065] Input: User's question text

[1066] Output: A message with the question text

[1067] Step 21:

[1068] The terminal sends this information to the server.

[1069] Specific operation: The terminal sends the question text to the server.

[1070] Input: User's question text

[1071] Output: Request message to the server

[1072] Step 22:

[1073] The server analyzes the data and generates new product suggestions and links to related online shopping sites.

[1074] Specific operation: The server analyzes the content of the question, retrieves related information from a product information database, and generates links to online shopping sites.

[1075] Input: User's question text

[1076] Output: New product proposals and online shopping site links

[1077] Step 23:

[1078] The device displays this information to the user.

[1079] Specific operation: The device displays the generated information and links in a chat box or dedicated display area.

[1080] Input: New product proposal and online shopping site link

[1081] Output: New product suggestions and online shopping links displayed to users

[1082] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[1083] (Application example 2)

[1084] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1085] Today's consumers often seek detailed information and reviews before purchasing a product, especially when shopping in a virtual store. However, no system exists that allows consumers to obtain detailed information while browsing products in a virtual store and receive appropriate support tailored to their emotions. Furthermore, conventional systems have difficulty responding appropriately when users become confused or frustrated, and lack means to improve the quality of the consumer experience. To address these issues, the present invention aims to dramatically improve the consumer experience in a virtual store.

[1086] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1087] In this invention, the server includes means for analyzing a user's question using a generation AI and generating an appropriate answer, means for displaying operating procedures for home appliances in AR using a smartphone camera, means for receiving user input and displaying an answer based on the analysis results, means for providing detailed product information, usage instructions, and reviews in a virtual store, and means for analyzing the user's emotional state and providing appropriate assistance according to the user's emotion. This enables consumers to intuitively obtain detailed information while browsing products in a virtual store and receive appropriate support according to their emotional state.

[1088] "Generative AI" is an artificial intelligence technology that analyzes user input and questions and automatically generates appropriate information and answers.

[1089] "AR display" is a technology that uses augmented reality technology to overlay digital information on actual images.

[1090] "User input" refers to the act of a user giving instructions or questions to the system via text or voice.

[1091] "Analysis results" are information obtained as a result of the generative AI analyzing the user's input and questions.

[1092] A "virtual store" is an online store that is a virtual shopping space set up on the Internet where users can browse and purchase items.

[1093] "Detailed product information" refers to information that users want to know before purchasing, such as product specifications, functions, instructions for use, and price.

[1094] "Reviews" refer to ratings and opinions of consumers who have already used a product.

[1095] "Emotional state" indicates the user's current psychological state or emotion, and is a category of emotions such as confusion, irritation, and excitement.

[1096] "Appropriate assistance" is specific and useful support and advice provided according to the user's emotional state.

[1097] This invention realizes a system that provides product information and enhances user support in a virtual store by combining generative AI, augmented reality (AR) technology, and an emotion recognition engine. Specific embodiments of the system are described below.

[1098] First, users launch a dedicated application on their smartphone or smart glasses. This application includes a camera function and displays camera images in real time, allowing users to take photos of products in the virtual store and obtain detailed information.

[1099] Hardware and software used

[1100] 1. Hardware

[1101] Smartphone or smart glasses camera: Captures video in real time and is used to recognize the product being photographed.

[1102] Server: Hosts the generative AI model, sentiment analysis engine, and database, and performs analytical processing.

[1103] 2. Software

[1104] OpenCV: Captures camera images and uses them for product recognition.

[1105] Transformers (transformers package): Used to perform generative AI and sentiment analysis.

[1106] HTTP request: Communicate with external APIs or product databases to obtain product information.

[1107] Data processing and calculation

[1108] Camera video analysis:

[1109] The device analyzes the real-time video captured by the user and recognizes the products. To do this, it runs an object detection algorithm and, once the product is identified, sends its name to the server.

[1110] Get product information:

[1111] The server receives the product name and retrieves the product's details, usage instructions, and reviews from a product database or external API. The retrieved information is sent to the user's device and displayed on the screen.

[1112] Sentiment analysis and providing appropriate support:

[1113] When a user enters information via text or voice, an emotion recognition engine analyzes their emotional state. If the input indicates frustration or confusion, the system generates a thoughtful response that reflects that emotion. The response is then sent to the user's device and displayed on the screen.

[1114] Specific examples

[1115] Suppose a user is walking around a virtual store and looking at products using smart glasses. For example, if the user points the camera at a smart TV and wants to get more information about it, they might enter the following prompt:

[1116] What features does this smart TV have?

[1117] Actions based on sentiment analysis results:

[1118] If the user's input indicates frustration, the system generates a considerate response such as:

[1119] We apologize for any inconvenience. This smart TV has high resolution and voice recognition. Purchase here (link).

[1120] Under normal circumstances, this will produce a response like this:

[1121] This smart TV has high resolution and voice recognition. Buy it here.

[1122] This allows users to intuitively obtain product information in a virtual store and receive appropriate support based on their emotions. This system not only dramatically improves the consumer experience, but also encourages purchases in the virtual store.

[1123] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1124] Step 1:

[1125] The user launches the dedicated application on their smartphone or smart glasses, which activates the camera function and prepares to display images in real time, allowing the images captured by the user to be displayed on the screen.

[1126] Input: User application launch

[1127] Output: Real-time video display

[1128] Step 2:

[1129] The user points the camera at a product in a virtual store, and the device analyzes the camera image and uses an object detection algorithm to recognize the product. The recognized product information is identified and the product name is extracted.

[1130] Input: Camera image

[1131] Output: Recognized product name

[1132] Step 3:

[1133] The device sends the extracted product name to the server, which retrieves detailed information, usage instructions, and reviews from a product database or external API based on the product name. The retrieved information is formatted on the server side and sent to the device.

[1134] Input: Product name

[1135] Output: Product details (description, usage, reviews, etc.)

[1136] Step 4:

[1137] The terminal displays the acquired product details to the user, who can then check the product details on the screen and request more information if necessary.

[1138] Input: Product details information

[1139] Output: Display detailed product information

[1140] Step 5:

[1141] The user inputs a question via text or voice. The emotion recognition engine analyzes this input and determines the user's emotional state (e.g., irritation, confusion). The analyzed emotion information and the user's question are sent to the server.

[1142] Input: User text and voice input

[1143] Output: Parsed sentiment information and question

[1144] Step 6:

[1145] The server uses a generative AI model to generate an appropriate response based on the user's emotional state and question. If the emotional state is irritation, it is converted into a considerate sentence. The generated response is sent to the device.

[1146] Input: Parsed emotion information and question

[1147] Output: The generated answer

[1148] Step 7:

[1149] The device displays the generated answer to the user, who can then check the answer on the screen and receive appropriate support.

[1150] Input: Generated Answer

[1151] Output: Show answer

[1152] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1153] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1154] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1155] [Third embodiment]

[1156] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1157] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1158] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1159] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1160] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1161] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1162] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1163] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1164] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1165] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1166] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1167] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1168] This invention is a system that revolutionizes home appliance instruction manuals by utilizing generative AI and augmented reality (AR) technology. This system allows users to easily access operation and troubleshooting information for home appliances through a smartphone app, enabling them to operate and repair appliances intuitively and quickly.

[1169] System program flow

[1170] 1. Launch the app

[1171] Users simply launch a dedicated app on their smartphone to enable the camera function.

[1172] The device prepares to display the camera image in real time.

[1173] 2. Providing AR operation instructions

[1174] The user points the camera at a home appliance.

[1175] The device analyzes camera footage and recognizes home appliances.

[1176] Based on the recognized home appliances, the device will provide the user with operating procedures and methods using AR displays.

[1177] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[1178] 3. Use AI support chatbots

[1179] Users type their questions into the chatbot about how to use it or troubleshooting.

[1180] The terminal receives the user's question and sends it to the server.

[1181] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[1182] The device receives the generated answer and displays it to the user.

[1183] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[1184] 4. Simplified instructions

[1185] The user opens a PDF instruction manual within the app to check the operating instructions.

[1186] The device sends the PDF contents to the server.

[1187] The server uses generative AI to convert the content into a concise description in natural language.

[1188] The terminal will display the generated brief description to the user.

[1189] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[1190] 5. New product proposals and e-commerce site referrals

[1191] Users ask for information about specific features or new products.

[1192] The terminal sends this information to the server.

[1193] The server analyzes the information and generates relevant new product suggestions and e-commerce site purchase links.

[1194] The device displays this information to the user.

[1195] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[1196] This system allows users to quickly and intuitively understand and respond to operating and troubleshooting methods for home appliances. For manufacturers, this system configuration not only improves the quality of user support, but also helps promote new products and sales.

[1197] The processing flow will be explained below.

[1198] Step 1:

[1199] The user launches the dedicated app on their smartphone, which then begins preparations to enable the camera function.

[1200] Step 2:

[1201] The device displays camera footage in real time and displays the footage the user takes on the screen.

[1202] Step 3:

[1203] Users point their camera at their appliances and use the system to get information on how to operate them or if they need to troubleshoot.

[1204] Step 4:

[1205] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[1206] Step 5:

[1207] The device sends information about the identified home appliances to a server, which then retrieves corresponding operating procedures and troubleshooting data from a database.

[1208] Step 6:

[1209] The server then sends the acquired operation instructions to the device, which then prepares the device for displaying the correct operation guide in AR (Augmented Reality).

[1210] Step 7:

[1211] The device overlays the received operating instructions and troubleshooting methods on the camera image in real time, allowing users to intuitively check how to operate their home appliances through the camera screen.

[1212] Step 8:

[1213] To get more detailed information, users can use the chatbot function installed in the app to input questions, such as "I don't know how to use XX" or "XX doesn't work."

[1214] Step 9:

[1215] The device sends the user's input question in text format to the server, which then analyzes the question using natural language processing (NLP) technology.

[1216] Step 10:

[1217] The server generates appropriate answers from a database based on the analysis results, and this process may involve the use of generative AI.

[1218] Step 11:

[1219] The server generates an appropriate response and sends it to the terminal, which then displays the received response to the user.

[1220] Step 12:

[1221] The user can then review the displayed answers and take any necessary action or troubleshooting steps, such as cleaning the filter or pressing the reset button.

[1222] Step 13:

[1223] If a user opens a PDF manual for a particular home appliance within the app, that's possible. If the PDF is difficult to understand, the user will ask for a simplified explanation.

[1224] Step 14:

[1225] The device sends the PDF content to the server, which then uses generative AI to convert the PDF's detailed content into simplified, natural language.

[1226] Step 15:

[1227] The server then sends the generated natural language brief instructions to the terminal, which then displays them to the user, allowing the user to easily understand complex operating procedures.

[1228] Step 16:

[1229] When users have questions about new product information or specific features, they enter that information through the chatbot.

[1230] Step 17:

[1231] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[1232] Step 18:

[1233] The server sends the new product information and a purchase link to the device, which displays it to the user. The user can check the new product information and purchase it directly from the purchase link if necessary.

[1234] These process steps allow users to quickly and intuitively operate home appliances, troubleshoot, explore new products, and more.

[1235] Example 1

[1236] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1237] Traditionally, instruction manuals and troubleshooting guides for home appliances have mainly been in paper or static PDF format, making it difficult to operate and repair products intuitively and quickly. Furthermore, when users searched for information on new products, they had limited means of quickly accessing the appropriate information. This often resulted in users spending time using or repairing the product, causing inconvenience. Furthermore, manufacturers faced challenges in streamlining user support and promoting new products.

[1238] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1239] In this invention, the server includes means for analyzing a user's question using a generation AI and generating an appropriate answer, means for displaying operating procedures for the device in augmented reality using a video display of the mobile device, and means for receiving user input and displaying the answer based on the analysis results. This allows users to quickly and intuitively understand how to operate and troubleshoot home appliances and easily obtain information on new products.

[1240] "Generative AI" is a technology that uses advanced artificial intelligence algorithms to analyze input data and automatically generate appropriate information and answers.

[1241] A "video device for a mobile device" is a device that has the function of acquiring and displaying images of the real world using a camera or display device installed on a smartphone, tablet, etc.

[1242] "Augmented reality display (AR display)" is a technology that overlays digital information on images of the real world, and is a means of providing users with intuitive information.

[1243] "Operating instructions" means a set of instructions or steps for the proper operation of a home appliance, explaining how to access the product's features.

[1244] "User input" refers to the text data and commands provided by the user through the application, and serves as the basis for the system to respond appropriately based on the analysis results.

[1245] A "function search system" is an information search technology that provides related information in response to a user's request when searching for specific function or product information.

[1246] "PDF Content" means the text and graphics data contained within a Portable Document Format (PDF) file, developed by Adobe Systems Incorporated, a document format that can be stored and viewed electronically.

[1247] "Natural language" refers to the language used by humans on a daily basis, along with the technology to convert it into a format that is easy for machines to understand and process the data.

[1248] "New product suggestions" are a way to introduce relevant new products based on a user's specific needs and interests, providing information and encouraging them to purchase.

[1249] A "purchase link" refers to a URL on the Internet that a user can access to purchase a product, and is a link that directs the user to an e-commerce site.

[1250] This invention is a system that revolutionizes home appliance instruction manuals by utilizing generative AI and augmented reality (AR) technology. Users can easily access operating instructions and troubleshooting information for home appliances through a smartphone app, enabling them to operate and repair appliances intuitively and quickly. The following describes the system's program processing and the hardware and software used.

[1251] Hardware and Software Use

[1252] Device: Use a smartphone or tablet with its built-in camera and display device.

[1253] Server: A server with high-performance computing resources is used to perform natural language processing and data analysis.

[1254] Generative AI model: A natural language processing model using artificial intelligence algorithms (e.g., GPT).

[1255] AR technology: Augmented reality software frameworks such as ARKit (Apple) and ARCore (Google).

[1256] Explanation of program processing

[1257] 1. Launch the app:

[1258] The user launches a dedicated app on their smartphone.

[1259] The device detects that the app has been launched and prepares to enable the camera function.

[1260] Example: When a user taps an app icon, the app launches and displays a camera view.

[1261] 2. AR operation instructions provided:

[1262] Users point their smartphone camera at home appliances.

[1263] The device analyzes camera footage in real time and recognizes home appliances.

[1264] Based on the recognized home appliances, the device will display operating procedures and methods on the screen using AR.

[1265] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes are displayed on the screen.

[1266] 3. Use AI-supported chatbots:

[1267] Users type questions into the chatbot about how to use it and troubleshooting.

[1268] The terminal sends the user's question to the server.

[1269] The server uses generative AI to analyze the user's question and generate an appropriate answer.

[1270] The device receives the generated answer and displays it to the user.

[1271] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[1272] Example prompt: "What should I do if my washing machine won't start?"

[1273] 4. Simplified instructions:

[1274] Users can open the instruction PDF within the app to check the operating instructions.

[1275] The device sends the PDF contents to the server.

[1276] The server uses generative AI to convert the content into a concise description in natural language.

[1277] The terminal will display the generated brief description to the user.

[1278] Example: When you open a PDF containing detailed operating instructions for an oven range, simple instructions such as "Press the timer setting button, enter the time, and press the start button" appear on the screen.

[1279] Example prompt: "Please briefly explain the oven operating procedure described in this PDF."

[1280] 5. New product proposals and e-commerce site referrals:

[1281] Users ask for information about specific features or new products.

[1282] The terminal sends this information to the server.

[1283] The server uses generative AI to analyze the information and generate relevant new product suggestions and purchase links.

[1284] The device displays this information to the user.

[1285] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[1286] Example prompt: "What are the features of the latest smart TVs?"

[1287] This system allows users to quickly and intuitively access information on how to operate and troubleshoot home appliances and new product information, and is also useful for manufacturers to streamline user support and promote new products.

[1288] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1289] Step 1: Launch the app

[1290] User: Launch the dedicated app on your smartphone.

[1291] What happens: The user taps on the app icon.

[1292] Input: The user's touch.

[1293] Output: The app starts.

[1294] Device: Detects that the app has been launched and prepares to enable the camera function.

[1295] What happens: The camera initialization process happens in the background and the camera view appears in the app.

[1296] Input: App launch signal.

[1297] Output: Camera view display.

[1298] Step 2: Provide AR operation instructions

[1299] User: Point your smartphone camera at a home appliance.

[1300] Specific action: To point the camera at a home appliance, hold the smartphone in your hand and frame the product.

[1301] Input: Home appliance footage.

[1302] Output: Acquisition of video data.

[1303] Terminal: Analyzes camera footage in real time and recognizes home appliances.

[1304] Specific operation: An image recognition algorithm is activated, analyzing the video data to detect the shape and characteristics of the home appliance.

[1305] Input: Video data.

[1306] Output: Recognition results of home appliances.

[1307] Terminal: Based on the recognized home appliance, the device displays operating procedures and methods on the screen using AR.

[1308] Specific behavior: Operation guides related to home appliances are overlaid on the screen.

[1309] Input: Recognition results for home appliances.

[1310] Output: AR display operation procedure.

[1311] Step 3: Use an AI-powered chatbot

[1312] Users: Type questions into the chatbot for instructions or troubleshooting.

[1313] Specific actions: Tap the chat button in the app, enter the question text and send it.

[1314] Input: Question text.

[1315] Output: The question is sent.

[1316] Terminal: Receives the user's question and sends it to the server.

[1317] Specific operation: Sends the question text to the server via an HTTP request.

[1318] Input: Question text.

[1319] Output: Sending question data.

[1320] Server: Uses generative AI to analyze the user's question and generate an appropriate answer.

[1321] What it does: The input text is parsed, answers are extracted from relevant databases, and formatted in the appropriate natural language format.

[1322] Input: Question data.

[1323] Output: Response data.

[1324] Terminal: Receives the generated answer and displays it to the user.

[1325] Specific behavior: The reply text received from the server is displayed in the chat window.

[1326] Input: Response data.

[1327] Output: Display the answer.

[1328] Step 4: Simplify the instructions

[1329] User: Open the instruction PDF within the app to view instructions.

[1330] Specific operation: To view an instruction PDF, select the PDF you want from the list within the app.

[1331] Input: Select PDF file.

[1332] Output: PDF content display.

[1333] Device: Send the PDF content to the server.

[1334] Specific operation: Extract the contents of the PDF file as text data and upload it to the server.

[1335] Input: PDF content.

[1336] Output: Sending text data.

[1337] Server: Uses generative AI to convert the content into concise descriptions in natural language.

[1338] What it does: A generative AI model is activated, analyzing the text data and converting it into a concise description.

[1339] Input: Text data.

[1340] Output: A short description.

[1341] Terminal: Display the generated short description to the user.

[1342] Specific behavior: The generated short description is displayed in the PDF viewer.

[1343] Input: A short description.

[1344] Output: Display a short description.

[1345] Step 5: Propose new products and direct them to the EC site

[1346] Users: Ask for information about specific features or new products.

[1347] What it does: Use the search box within the app to enter specific product information.

[1348] Input: Your search query.

[1349] Output: Submitting a search query.

[1350] Device: Sends this information to the server.

[1351] Specific operation: The entered search query is sent to the server via an HTTP request.

[1352] Input: Your search query.

[1353] Output: Submitting a search query.

[1354] Server: Analyzes the information using generative AI and generates relevant new product suggestions and purchase links.

[1355] What it does: The server parses the search query and generates relevant product information and purchase links.

[1356] Input: Your search query.

[1357] Output: New product suggestions and purchase links.

[1358] Terminal: Display this information to the user.

[1359] What happens: The generated suggestions and links are displayed in the results area within the app.

[1360] Input: New product suggestions and purchase links.

[1361] Output: Display information.

[1362] (Application example 1)

[1363] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1364] There is a wide range of information on the operation, maintenance, and troubleshooting of factory machinery, and conventional manuals alone make it difficult for users to respond quickly and accurately.In addition, there are limited means to quickly provide information on new tools and parts, and these issues need to be improved.

[1365] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1366] In this invention, the server includes a means for analyzing a user's question using a generative AI and generating an appropriate answer, a means for displaying operating procedures for the machine in AR using a camera, a means for receiving user input and displaying answers based on the analysis results, and a means for providing a machine function search system. This allows users to quickly and intuitively understand and respond to operating methods and troubleshooting of the machine. It also includes a means for suggesting new products and necessary parts related to the analyzed machine and providing links to e-commerce sites, allowing users to quickly obtain the necessary information.

[1367] "Generative AI" refers to an artificial intelligence model that analyzes user input and generates appropriate answers or information.

[1368] A "camera" is an optical device for visually capturing mechanical equipment.

[1369] "Machinery" refers to mechanical equipment and robots used in factories.

[1370] An "operating procedure" refers to a series of actions or procedures for operating a machine or device correctly.

[1371] "AR display" refers to the use of augmented reality technology to overlay virtual information onto real-world images.

[1372] "User input" refers to information such as operations or questions that a user makes to the system.

[1373] "Analysis results" refer to the answers and information derived by the generating AI based on user input.

[1374] A "function search system" is a system that allows you to search and obtain information about the functions and characteristics of mechanical devices.

[1375] An "AI support chatbot" is an interactive program that uses artificial intelligence to answer users' questions and provide support.

[1376] "Troubleshooting information" refers to information that provides solutions or methods of dealing with problems or malfunctions that occur in mechanical equipment.

[1377] "PDF instruction manual content" refers to an electronic document that contains detailed instructions for operating and maintaining a machine or device.

[1378] "Natural language simplification" refers to the process of expressing technical content in simple, everyday language so that it is easy for users to understand.

[1379] "New products" are new tools or parts that are newly introduced to the market.

[1380] An "e-commerce site" is a website that sells goods and services over the Internet.

[1381] This invention provides a system for intuitive and rapid operation, maintenance, and troubleshooting of factory machinery. The system utilizes generative AI models and augmented reality (AR) technology to allow users to access machinery information using their smartphones.

[1382] Program processing

[1383] Hardware and software used

[1384] Smartphone: High-resolution camera, AR-enabled

[1385] Server: Equipped with high-performance CPU / GPU, natural language processing (NLP) software, generative AI models

[1386] Frameworks: TensorFlow, PyTorch

[1387] API: Google Vision API, OpenAI API

[1388] Data processing and calculation

[1389] 1. Camera image analysis:

[1390] The server uses the Google Vision API to recognize mechanical devices from the smartphone camera image.

[1391] Example: When a user points their smartphone camera at a factory robot, the robot's model number and type are automatically recognized.

[1392] 2. Natural Language Processing:

[1393] The server analyzes the user's question using the OpenAI API and generates an appropriate solution.

[1394] Example: When a user asks, "What should I do when error code E02 appears?", the server generates a solution such as, "Error code E02 indicates that the machine is overheating. Make sure the cooling system is working properly and, if possible, turn the power off and restart."

[1395] 3. Manual Simplification Using Generative AI:

[1396] The server uses a generation AI (GPT-3) to simplify the PDF of the detailed manual in natural language.

[1397] Example: Simplifying a PDF operating manual for a machine generates simple instructions such as "Press the emergency stop button and press the restart button after an interruption."

[1398] 4. New product proposal:

[1399] The server suggests new products and necessary parts related to the analyzed machinery and provides links to e-commerce sites.

[1400] Example: The server analyzes a user's question, "I want to know the features of the latest automation tool," and provides information such as, "This automation tool is equipped with advanced sensor functions and a real-time monitoring system. You can purchase it here (link)."

[1401] Specific examples (prompt examples)

[1402] Example prompt sentence:

[1403] "What should I do when error code E02 appears?"

[1404] In this way, this invention provides a system that streamlines the provision of information on how to operate and troubleshoot machinery in a factory, allowing users to respond intuitively. It also quickly provides information on necessary new products and parts, and easily guides users to e-commerce sites.

[1405] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1406] Step 1:

[1407] The user launches a dedicated app on their smartphone and enables the camera function. The device prepares to display camera footage in real time, allowing the user to capture machinery and equipment with the camera.

[1408] Step 2:

[1409] The user points their smartphone camera at a machine. The device analyzes the camera image and recognizes the machine. The Google Vision API is used to do this, and the model number and type of the machine are sent to the device as analysis results.

[1410] Step 3:

[1411] The device uses AR to display operation procedures and maintenance methods for the recognized machinery. This allows the user to proceed with the work while visually checking the operation procedures. The displayed content is based on data provided in advance by the server.

[1412] Step 4:

[1413] When a user has a question about a machine, they can input it using the chatbot function within the app. The device then sends the question to the server. A specific example of input is a prompt such as, "Please tell me what to do when error code E02 appears."

[1414] Step 5:

[1415] The server uses the OpenAI API to analyze the user's question and generate an appropriate solution. The generated answer is sent from the server to the device and displayed to the user. For example, the user may be provided with information such as, "Error code E02 indicates that the machine is overheating. Please make sure the cooling system is operating properly and, if possible, turn the power off and restart."

[1416] Step 6:

[1417] When a user opens a PDF of a detailed operating manual within the app, the device sends the contents of the PDF to the server. The server uses generative AI (GPT-3) to simplify the PDF contents in natural language and sends the result to the device. The device then displays the simplified instructions to the user. For example, a simple instruction such as "Press the emergency stop button, then press the restart button after interruption" is displayed.

[1418] Step 7:

[1419] When a user requests information about a new tool or part, the device sends the request to the server. Based on the analysis results, the server generates relevant new product information and links to e-commerce sites, and sends them to the device. The device then displays this information to the user. For example, the device may provide information such as, "This automation tool is equipped with advanced sensor functions and a real-time monitoring system. Purchase here (link)."

[1420] Through these processing steps, the present invention provides a system that streamlines the operation and troubleshooting of machinery in a factory, allowing users to respond quickly and intuitively.

[1421] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1422] This invention is a system that revolutionizes home appliance instruction manuals by combining generative AI, augmented reality (AR) technology, and an emotion engine that recognizes user emotions. This system allows users to easily access instructions and troubleshooting information for home appliances through a smartphone app, and also provides support based on the user's emotional state, creating a more personalized user experience.

[1423] System program flow

[1424] 1. Launch the app

[1425] The user launches the dedicated app on their smartphone, which activates the camera function and simultaneously activates the emotion engine.

[1426] The device displays the camera footage in real time and prepares the footage the user captures to be displayed on the screen.

[1427] 2. Providing AR operation instructions

[1428] The user points the camera at a home appliance.

[1429] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[1430] The device sends information about the recognized home appliances to the server, and the server retrieves corresponding operating procedures and troubleshooting data from a database.

[1431] The server sends the acquired operating procedures to the device, and the device displays the operating procedures and troubleshooting methods to the user in AR.

[1432] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[1433] 3. Use AI support chatbots

[1434] Users type their questions into the chatbot about how to use it or troubleshooting.

[1435] The terminal receives the user's question and sends it to the server.

[1436] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[1437] The device receives the generated answer and displays it to the user.

[1438] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[1439] 4. Utilizing the Emotion Engine

[1440] When a user enters a question via text or voice, the emotion engine analyzes the user's emotions.

[1441] The device sends the analysis results to the server, and the server adjusts the response content based on the emotional data.

[1442] Example: If the user displays emotions that indicate confusion or annoyance, the server might respond with a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[1443] 5. Simplified instructions

[1444] The user opens a PDF instruction manual within the app to check the operating instructions.

[1445] The device sends the PDF content to the server, which then uses generative AI to convert the PDF's detailed content into simplified, natural language.

[1446] The terminal will display the generated brief description to the user.

[1447] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[1448] 6. New product proposals and e-commerce site referrals

[1449] Users ask for information about specific features or new products.

[1450] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[1451] The device displays this information to the user.

[1452] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[1453] This system allows users to quickly and intuitively understand and respond to how to operate and troubleshoot home appliances. In addition, responses are given that take the user's emotions into consideration, providing more personalized support. For manufacturers, this system configuration not only improves the quality of user support, but is also useful for promoting new products and sales.

[1454] The processing flow will be explained below.

[1455] Step 1:

[1456] The user launches the dedicated app on their smartphone, which activates the camera function and simultaneously activates the emotion engine.

[1457] Step 2:

[1458] The device displays the camera footage in real time and prepares the footage the user captures to be displayed on the screen.

[1459] Step 3:

[1460] The user points the camera at a home appliance.

[1461] Step 4:

[1462] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[1463] Step 5:

[1464] The device sends information about the recognized home appliances to the server.

[1465] Step 6:

[1466] The server retrieves the corresponding operating procedures and troubleshooting data from the database.

[1467] Step 7:

[1468] The server sends the acquired operating procedures to the device, and the device displays the operating procedures and troubleshooting methods to the user in AR.

[1469] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[1470] Step 8:

[1471] Users type their questions into the chatbot about how to use it or troubleshooting.

[1472] Step 9:

[1473] The terminal sends the user's question in text format to the server.

[1474] Step 10:

[1475] The server uses natural language processing techniques to analyze the user's question.

[1476] Step 11:

[1477] Based on the analysis results, the server generates appropriate answers from the database and also analyzes the user's emotions using an emotion engine.

[1478] Step 12:

[1479] The device sends the user's voice and text input data to the emotion engine in real time and analyzes emotions.

[1480] Step 13:

[1481] The emotion engine determines the user's emotional state based on the user's emotional data.

[1482] Step 14:

[1483] The server uses the analysis results of the emotion engine to generate a response message that is tailored to the user's emotional state.

[1484] Example: If a user expresses confusion or frustration, the server tailors and generates a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[1485] Step 15:

[1486] The server sends the generated response message to the terminal, which displays it to the user.

[1487] Step 16:

[1488] The user can refer to the displayed answers to perform the necessary operations or troubleshoot.

[1489] Step 17:

[1490] When a user opens a PDF manual for a particular home appliance within the app, the device sends the contents of the PDF to the server.

[1491] Step 18:

[1492] The server uses generative AI to convert the detailed content of the PDF into simplified, natural language.

[1493] Step 19:

[1494] The server sends the generated natural language short description to the terminal, which displays it to the user.

[1495] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[1496] Step 20:

[1497] When users have questions about new product information or specific features, they enter that information through the chatbot.

[1498] Step 21:

[1499] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[1500] Step 22:

[1501] The server sends the new product information and a purchase link to the device, which displays it to the user. The user can check the new product information and purchase it directly from the purchase link if necessary.

[1502] In this way, users can quickly and intuitively understand and respond to how to operate and troubleshoot their home appliances, and also receive personalized, emotionally sensitive support.

[1503] Example 2

[1504] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1505] Conventional instruction manuals for home appliances have scattered information, making it difficult for users to quickly obtain the information they need. Furthermore, they lack a mechanism for reading emotions and responding appropriately, making it difficult to alleviate user confusion and frustration. Furthermore, they lack a means for providing new product information and links to related e-commerce sites, making it difficult to stimulate users' desire to purchase. The purpose of this invention is to solve these problems.

[1506] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing a user's question using a generation AI and generating an appropriate answer, a means for displaying operating procedures for the device in augmented reality using a camera of a mobile terminal, and a means for analyzing the user's emotions and adjusting the answer content based on the analysis results. This allows the user to quickly and intuitively understand the operating procedures and receive personalized support that takes emotions into consideration. Furthermore, it also makes it easy to promote new products and direct users to related online shopping sites.

[1507] "Generative AI" is an artificial intelligence technology that analyzes questions asked in natural language by users and automatically generates appropriate answers.

[1508] "Mobile terminal" refers to a portable computing device such as a smartphone or tablet.

[1509] A "camera" is a device that converts optical images into digital data for display or analysis in real time.

[1510] "Augmented reality display" is a technology that displays digital information overlaid on real-world images.

[1511] "Equipment" broadly refers to electronic devices such as home appliances.

[1512] "Sentiment analysis" refers to the process of inferring a user's emotional state from their text or voice.

[1513] "Adjusting the content of the response based on the analysis results" means appropriately changing the tone and content of the generated response depending on the user's emotional state.

[1514] "Promotional Information" means information intended to promote new products or services.

[1515] "Related Shopping Sites" means websites where users can purchase products online.

[1516] This invention is a system that combines generative AI, augmented reality (AR) technology, and an emotion engine that recognizes user emotions, and provides operation instructions and troubleshooting information for home appliances to improve the user experience. This system mainly operates through the collaboration of three parties: a server, a device, and a user.

[1517] First, the user launches the dedicated app on their smartphone or tablet, which activates the camera function and activates the emotion engine. The device displays the camera footage in real time and shows the footage the user takes on the screen. At this time, the footage captured by the device's camera is analyzed using an object detection algorithm (e.g., YOLO or ResNet).

[1518] Next, when the user points the camera at a home appliance, the device recognizes the appliance and sends the recognition results to the server. The server retrieves the relevant operating procedures and troubleshooting information from a database (e.g., MySQL, PostgreSQL) and sends the information back to the device. This allows the device to display the operating procedures or troubleshooting methods on the camera image in the form of augmented reality (AR). For example, when the user points the camera at an air conditioner, the screen will show how to use each button on the remote control and how to set a specific mode.

[1519] Additionally, users can input questions about operation methods and troubleshooting into a dedicated chatbot. The device sends these questions as text data to the server, which then uses natural language processing technology (e.g., GPT-3, BERT) to analyze the user's question and generate an appropriate answer. The generated answer is then sent back to the device and displayed to the user in chat format. For example, if a user asks, "My washing machine isn't working," the generating AI will respond with specific instructions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[1520] The emotion engine can also be used to analyze the emotions expressed when a user types a question. The device also sends the emotion analysis results to the server, which then uses the results to tailor the answer to the user's emotional state. For example, if the user expresses confusion or irritation, the server can generate a considerate response such as, "We apologize for the inconvenience. Could you please try the following steps?"

[1521] The contents of PDF manuals can also be simplified and displayed. When a user opens a PDF manual within the app, the device sends the contents of the PDF as text data to the server. The server then uses generative AI to simplify this text data, converts it into natural language, and sends it back to the device. The device then displays the simplified instructions to the user. For example, when a user opens a PDF containing detailed operating instructions for a microwave oven, simple instructions such as "Press the timer setting button, enter the time, and press the start button" are displayed.

[1522] Furthermore, when a user requests information about a specific feature or new product, the device sends this input information to the server. The server analyzes the information, generates new product suggestions and links to related online shopping sites, and sends them to the device. The device then displays this information to the user. For example, if a user asks, "I want to know the features of the latest smart TV," the server might respond, "This smart TV has high resolution and voice recognition capabilities. You can purchase it here (link)."

[1523] This system allows users to quickly and intuitively understand and respond to how to operate and troubleshoot their home appliances, and also provides personalized support that takes into account the user's emotions, resulting in a more satisfying user experience.

[1524] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1525] Step 1:

[1526] The user launches a dedicated app on their smartphone.

[1527] Specific action: The user taps the app icon on the smartphone's home screen.

[1528] Input: Tap input from the user

[1529] Output: Launching the app and displaying the initial screen

[1530] Step 2:

[1531] The device will enable the app's camera function and emotion engine.

[1532] Specific operation: The device will launch the camera function and ask the user for permission to use the camera. At the same time, the emotion engine will also launch in the background.

[1533] Input: App launch signal, camera access permission from user

[1534] Output: Streaming camera footage and emotion engine status

[1535] Step 3:

[1536] The user points the camera at a home appliance.

[1537] Specific action: The user places a home appliance within the camera frame.

[1538] Input: Camera video data

[1539] Output: Video frame showing home appliances

[1540] Step 4:

[1541] The device analyzes camera footage in real time and runs object detection algorithms to recognize home appliances.

[1542] How it works: An object detection algorithm (e.g., YOLO or ResNet) extracts features from video frames and matches them with known home appliances in a database.

[1543] Input: Camera image frame

[1544] Output: ID or feature information of the recognized home appliance

[1545] Step 5:

[1546] The device sends information about the recognized home appliances to the server.

[1547] Specific operation: The device sends a request including the home appliance ID and image data to the server.

[1548] Input: ID or characteristic information of home appliance

[1549] Output: Request message to the server

[1550] Step 6:

[1551] The server retrieves the corresponding operating procedures and troubleshooting data from the database.

[1552] Specific operation: The server executes an SQL query based on the ID of the home appliance to obtain related information.

[1553] Input: Home appliance ID

[1554] Output: Records of operating procedures and troubleshooting information

[1555] Step 7:

[1556] The server transmits the acquired operating procedure to the terminal.

[1557] Specific operation: The server generates a JSON format response and sends it to the terminal.

[1558] Input: Records of operating procedures and troubleshooting information

[1559] Output: Response message to the terminal

[1560] Step 8:

[1561] The device provides users with operating instructions and troubleshooting methods using AR displays.

[1562] Specific operation: The device uses an AR library (e.g., ARKit, ARCore) to overlay operating instructions on the camera image.

[1563] Input: Operation procedure information of response message

[1564] Output: AR-displayed operating procedures and troubleshooting information

[1565] Example: When a user points the camera at an air conditioner, the screen will show how to use each button on the remote control and how to set a specific mode.

[1566] Step 9:

[1567] Users type their questions into the chatbot about how to use it or troubleshooting.

[1568] Specific behavior: The user types a question into the chat box within the app and presses the send button.

[1569] Input: User's question text

[1570] Output: Message to send to the chatbot

[1571] Step 10:

[1572] The terminal sends the user's question to the server.

[1573] Specific operation: The terminal sends text data to the server.

[1574] Input: User's question text

[1575] Output: Request message to the server

[1576] Step 11:

[1577] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[1578] How it works: The server calls the generative AI (e.g., GPT-3), passes the user's question as a prompt, and retrieves the generated answer.

[1579] Input: User's question text

[1580] Output: Generated answer text

[1581] Step 12:

[1582] The terminal displays the generated answer to the user.

[1583] Specific behavior: The device displays the generated text in the chat box and notifies the user.

[1584] Input: Generated answer text

[1585] Output: The answer text that is displayed to the user

[1586] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[1587] Step 13:

[1588] When a user types a question, the emotion engine analyzes the user's emotions.

[1589] Specific operation: The emotion engine analyzes input text and voice data and estimates the emotional state (e.g., confusion, irritation).

[1590] Input: User question text or voice data

[1591] Output: Emotion analysis results

[1592] Step 14:

[1593] The terminal transmits the analysis results to the server.

[1594] Specific operation: The device sends a request including emotion data to the server.

[1595] Input: Sentiment analysis results

[1596] Output: Request message to the server

[1597] Step 15:

[1598] The server adjusts the response content based on the emotional data.

[1599] Specific behavior: The server adjusts the tone and content of the response text depending on the user's emotional state (e.g., using emojis and softer language).

[1600] Input: Sentiment analysis results, generated answer text

[1601] Output: Adjusted answer text

[1602] Example: If a user expresses confusion or frustration, the server responds with a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[1603] Step 16:

[1604] The user opens a PDF instruction manual within the app to check the operating instructions.

[1605] Specific behavior: The user taps the instruction icon, which launches the PDF viewer.

[1606] Input: User tap input

[1607] Output: Launch PDF viewer and display PDF

[1608] Step 17:

[1609] The device sends the PDF contents to the server.

[1610] Specific operation: The device extracts the text data from the PDF and sends it to the server.

[1611] Input: PDF text data

[1612] Output: Request message to the server

[1613] Step 18:

[1614] The server uses generative AI to simplify the detailed content of the PDF.

[1615] Specific behavior: The server sends a prompt to the generation AI, instructing it to generate simplified text.

[1616] Input: PDF text data

[1617] Output: Simplified descriptive text

[1618] Step 19:

[1619] The terminal will display the generated brief description to the user.

[1620] What happens: The device retrieves the abbreviated text and displays it as an overlay on the PDF viewer.

[1621] Input: Simplified description text

[1622] Output: A simplified description that is displayed to the user.

[1623] Example: When you open a PDF containing detailed operating instructions for an oven range, you see a simple explanation: "Press the timer setting button, enter the time, and press the start button."

[1624] Step 20:

[1625] Users ask for information about specific features or new products.

[1626] Specific behavior: A user types a question such as "I want to know the features of the latest smart TVs."

[1627] Input: User's question text

[1628] Output: A message with the question text

[1629] Step 21:

[1630] The terminal sends this information to the server.

[1631] Specific operation: The terminal sends the question text to the server.

[1632] Input: User's question text

[1633] Output: Request message to the server

[1634] Step 22:

[1635] The server analyzes the data and generates new product suggestions and links to related online shopping sites.

[1636] Specific operation: The server analyzes the content of the question, retrieves related information from a product information database, and generates links to online shopping sites.

[1637] Input: User's question text

[1638] Output: New product proposals and online shopping site links

[1639] Step 23:

[1640] The device displays this information to the user.

[1641] Specific operation: The device displays the generated information and links in a chat box or dedicated display area.

[1642] Input: New product proposal and online shopping site link

[1643] Output: New product suggestions and online shopping links displayed to users

[1644] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[1645] (Application example 2)

[1646] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1647] Today's consumers often seek detailed information and reviews before purchasing a product, especially when shopping in a virtual store. However, no system exists that allows consumers to obtain detailed information while browsing products in a virtual store and receive appropriate support tailored to their emotions. Furthermore, conventional systems have difficulty responding appropriately when users become confused or frustrated, and lack means to improve the quality of the consumer experience. To address these issues, the present invention aims to dramatically improve the consumer experience in a virtual store.

[1648] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1649] In this invention, the server includes means for analyzing a user's question using a generation AI and generating an appropriate answer, means for displaying operating procedures for home appliances in AR using a smartphone camera, means for receiving user input and displaying an answer based on the analysis results, means for providing detailed product information, usage instructions, and reviews in a virtual store, and means for analyzing the user's emotional state and providing appropriate assistance according to the user's emotion. This enables consumers to intuitively obtain detailed information while browsing products in a virtual store and receive appropriate support according to their emotional state.

[1650] "Generative AI" is an artificial intelligence technology that analyzes user input and questions and automatically generates appropriate information and answers.

[1651] "AR display" is a technology that uses augmented reality technology to overlay digital information on actual images.

[1652] "User input" refers to the act of a user giving instructions or questions to the system via text or voice.

[1653] "Analysis results" are information obtained as a result of the generative AI analyzing the user's input and questions.

[1654] A "virtual store" is an online store that is a virtual shopping space set up on the Internet where users can browse and purchase items.

[1655] "Detailed product information" refers to information that users want to know before purchasing, such as product specifications, functions, instructions for use, and price.

[1656] "Reviews" refer to ratings and opinions of consumers who have already used a product.

[1657] "Emotional state" indicates the user's current psychological state or emotion, and is a category of emotions such as confusion, irritation, and excitement.

[1658] "Appropriate assistance" is specific and useful support and advice provided according to the user's emotional state.

[1659] This invention realizes a system that provides product information and enhances user support in a virtual store by combining generative AI, augmented reality (AR) technology, and an emotion recognition engine. Specific embodiments of the system are described below.

[1660] First, users launch a dedicated application on their smartphone or smart glasses. This application includes a camera function and displays camera images in real time, allowing users to take photos of products in the virtual store and obtain detailed information.

[1661] Hardware and software used

[1662] 1. Hardware

[1663] Smartphone or smart glasses camera: Captures video in real time and is used to recognize the product being photographed.

[1664] Server: Hosts the generative AI model, sentiment analysis engine, and database, and performs analytical processing.

[1665] 2. Software

[1666] OpenCV: Captures camera images and uses them for product recognition.

[1667] Transformers (transformers package): Used to perform generative AI and sentiment analysis.

[1668] HTTP request: Communicate with external APIs or product databases to obtain product information.

[1669] Data processing and calculation

[1670] Camera video analysis:

[1671] The device analyzes the real-time video captured by the user and recognizes the products. To do this, it runs an object detection algorithm and, once the product is identified, sends its name to the server.

[1672] Get product information:

[1673] The server receives the product name and retrieves the product's details, usage instructions, and reviews from a product database or external API. The retrieved information is sent to the user's device and displayed on the screen.

[1674] Sentiment analysis and providing appropriate support:

[1675] When a user enters information via text or voice, an emotion recognition engine analyzes their emotional state. If the input indicates frustration or confusion, the system generates a thoughtful response that reflects that emotion. The response is then sent to the user's device and displayed on the screen.

[1676] Specific examples

[1677] Suppose a user is walking around a virtual store and looking at products using smart glasses. For example, if the user points the camera at a smart TV and wants to get more information about it, they might enter the following prompt:

[1678] What features does this smart TV have?

[1679] Actions based on sentiment analysis results:

[1680] If the user's input indicates frustration, the system generates a considerate response such as:

[1681] We apologize for any inconvenience. This smart TV has high resolution and voice recognition. Purchase here (link).

[1682] Under normal circumstances, this will produce a response like this:

[1683] This smart TV has high resolution and voice recognition. Buy it here.

[1684] This allows users to intuitively obtain product information in a virtual store and receive appropriate support based on their emotions. This system not only dramatically improves the consumer experience, but also encourages purchases in the virtual store.

[1685] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1686] Step 1:

[1687] The user launches the dedicated application on their smartphone or smart glasses, which activates the camera function and prepares to display images in real time, allowing the images captured by the user to be displayed on the screen.

[1688] Input: User application launch

[1689] Output: Real-time video display

[1690] Step 2:

[1691] The user points the camera at a product in a virtual store, and the device analyzes the camera image and uses an object detection algorithm to recognize the product. The recognized product information is identified and the product name is extracted.

[1692] Input: Camera image

[1693] Output: Recognized product name

[1694] Step 3:

[1695] The device sends the extracted product name to the server, which retrieves detailed information, usage instructions, and reviews from a product database or external API based on the product name. The retrieved information is formatted on the server side and sent to the device.

[1696] Input: Product name

[1697] Output: Product details (description, usage, reviews, etc.)

[1698] Step 4:

[1699] The terminal displays the acquired product details to the user, who can then check the product details on the screen and request more information if necessary.

[1700] Input: Product details information

[1701] Output: Display detailed product information

[1702] Step 5:

[1703] The user inputs a question via text or voice. The emotion recognition engine analyzes this input and determines the user's emotional state (e.g., irritation, confusion). The analyzed emotion information and the user's question are sent to the server.

[1704] Input: User text and voice input

[1705] Output: Parsed sentiment information and question

[1706] Step 6:

[1707] The server uses a generative AI model to generate an appropriate response based on the user's emotional state and question. If the emotional state is irritation, it is converted into a considerate sentence. The generated response is sent to the device.

[1708] Input: Parsed emotion information and question

[1709] Output: The generated answer

[1710] Step 7:

[1711] The device displays the generated answer to the user, who can then check the answer on the screen and receive appropriate support.

[1712] Input: Generated Answer

[1713] Output: Show answer

[1714] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1715] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1716] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1717] [Fourth embodiment]

[1718] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1719] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1720] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1721] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1722] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1723] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1724] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1725] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1726] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1727] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1728] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1729] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1730] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1731] This invention is a system that revolutionizes home appliance instruction manuals by utilizing generative AI and augmented reality (AR) technology. This system allows users to easily access operation and troubleshooting information for home appliances through a smartphone app, enabling them to operate and repair appliances intuitively and quickly.

[1732] System program flow

[1733] 1. Launch the app

[1734] Users simply launch a dedicated app on their smartphone to enable the camera function.

[1735] The device prepares to display the camera image in real time.

[1736] 2. Providing AR operation instructions

[1737] The user points the camera at a home appliance.

[1738] The device analyzes camera footage and recognizes home appliances.

[1739] Based on the recognized home appliances, the device will provide the user with operating procedures and methods using AR displays.

[1740] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[1741] 3. Use AI support chatbots

[1742] Users type their questions into the chatbot about how to use it or troubleshooting.

[1743] The terminal receives the user's question and sends it to the server.

[1744] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[1745] The device receives the generated answer and displays it to the user.

[1746] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[1747] 4. Simplified instructions

[1748] The user opens a PDF instruction manual within the app to check the operating instructions.

[1749] The device sends the PDF contents to the server.

[1750] The server uses generative AI to convert the content into a concise description in natural language.

[1751] The terminal will display the generated brief description to the user.

[1752] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[1753] 5. New product proposals and e-commerce site referrals

[1754] Users ask for information about specific features or new products.

[1755] The terminal sends this information to the server.

[1756] The server analyzes the information and generates relevant new product suggestions and e-commerce site purchase links.

[1757] The device displays this information to the user.

[1758] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[1759] This system allows users to quickly and intuitively understand and respond to operating and troubleshooting methods for home appliances. For manufacturers, this system configuration not only improves the quality of user support, but also helps promote new products and sales.

[1760] The processing flow will be explained below.

[1761] Step 1:

[1762] The user launches the dedicated app on their smartphone, which then begins preparations to enable the camera function.

[1763] Step 2:

[1764] The device displays camera footage in real time and displays the footage the user takes on the screen.

[1765] Step 3:

[1766] Users point their camera at their appliances and use the system to get information on how to operate them or if they need to troubleshoot.

[1767] Step 4:

[1768] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[1769] Step 5:

[1770] The device sends information about the identified home appliances to a server, which then retrieves corresponding operating procedures and troubleshooting data from a database.

[1771] Step 6:

[1772] The server then sends the acquired operation instructions to the device, which then prepares the device for displaying the correct operation guide in AR (Augmented Reality).

[1773] Step 7:

[1774] The device overlays the received operating instructions and troubleshooting methods on the camera image in real time, allowing users to intuitively check how to operate their home appliances through the camera screen.

[1775] Step 8:

[1776] To get more detailed information, users can use the chatbot function installed in the app to input questions, such as "I don't know how to use XX" or "XX doesn't work."

[1777] Step 9:

[1778] The device sends the user's input question in text format to the server, which then analyzes the question using natural language processing (NLP) technology.

[1779] Step 10:

[1780] The server generates appropriate answers from a database based on the analysis results, and this process may involve the use of generative AI.

[1781] Step 11:

[1782] The server generates an appropriate response and sends it to the terminal, which then displays the received response to the user.

[1783] Step 12:

[1784] The user can then review the displayed answers and take any necessary action or troubleshooting steps, such as cleaning the filter or pressing the reset button.

[1785] Step 13:

[1786] If a user opens a PDF manual for a particular home appliance within the app, that's possible. If the PDF is difficult to understand, the user will ask for a simplified explanation.

[1787] Step 14:

[1788] The device sends the PDF content to the server, which then uses generative AI to convert the PDF's detailed content into simplified, natural language.

[1789] Step 15:

[1790] The server then sends the generated natural language brief instructions to the terminal, which then displays them to the user, allowing the user to easily understand complex operating procedures.

[1791] Step 16:

[1792] When users have questions about new product information or specific features, they enter that information through the chatbot.

[1793] Step 17:

[1794] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[1795] Step 18:

[1796] The server sends the new product information and a purchase link to the device, which displays it to the user. The user can check the new product information and purchase it directly from the purchase link if necessary.

[1797] These process steps allow users to quickly and intuitively operate home appliances, troubleshoot, explore new products, and more.

[1798] Example 1

[1799] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1800] Traditionally, instruction manuals and troubleshooting guides for home appliances have mainly been in paper or static PDF format, making it difficult to operate and repair products intuitively and quickly. Furthermore, when users searched for information on new products, they had limited means of quickly accessing the appropriate information. This often resulted in users spending time using or repairing the product, causing inconvenience. Furthermore, manufacturers faced challenges in streamlining user support and promoting new products.

[1801] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1802] In this invention, the server includes means for analyzing a user's question using a generation AI and generating an appropriate answer, means for displaying operating procedures for the device in augmented reality using a video display of the mobile device, and means for receiving user input and displaying the answer based on the analysis results. This allows users to quickly and intuitively understand how to operate and troubleshoot home appliances and easily obtain information on new products.

[1803] "Generative AI" is a technology that uses advanced artificial intelligence algorithms to analyze input data and automatically generate appropriate information and answers.

[1804] A "video device for a mobile device" is a device that has the function of acquiring and displaying images of the real world using a camera or display device installed on a smartphone, tablet, etc.

[1805] "Augmented reality display (AR display)" is a technology that overlays digital information on images of the real world, and is a means of providing users with intuitive information.

[1806] "Operating instructions" means a set of instructions or steps for the proper operation of a home appliance, explaining how to access the product's features.

[1807] "User input" refers to the text data and commands provided by the user through the application, and serves as the basis for the system to respond appropriately based on the analysis results.

[1808] A "function search system" is an information search technology that provides related information in response to a user's request when searching for specific function or product information.

[1809] "PDF Content" means the text and graphics data contained within a Portable Document Format (PDF) file, developed by Adobe Systems Incorporated, a document format that can be stored and viewed electronically.

[1810] "Natural language" refers to the language used by humans on a daily basis, along with the technology to convert it into a format that is easy for machines to understand and process the data.

[1811] "New product suggestions" are a way to introduce relevant new products based on a user's specific needs and interests, providing information and encouraging them to purchase.

[1812] A "purchase link" refers to a URL on the Internet that a user can access to purchase a product, and is a link that directs the user to an e-commerce site.

[1813] This invention is a system that revolutionizes home appliance instruction manuals by utilizing generative AI and augmented reality (AR) technology. Users can easily access operating instructions and troubleshooting information for home appliances through a smartphone app, enabling them to operate and repair appliances intuitively and quickly. The following describes the system's program processing and the hardware and software used.

[1814] Hardware and Software Use

[1815] Device: Use a smartphone or tablet with its built-in camera and display device.

[1816] Server: A server with high-performance computing resources is used to perform natural language processing and data analysis.

[1817] Generative AI model: A natural language processing model using artificial intelligence algorithms (e.g., GPT).

[1818] AR technology: Augmented reality software frameworks such as ARKit (Apple) and ARCore (Google).

[1819] Explanation of program processing

[1820] 1. Launch the app:

[1821] The user launches a dedicated app on their smartphone.

[1822] The device detects that the app has been launched and prepares to enable the camera function.

[1823] Example: When a user taps an app icon, the app launches and displays a camera view.

[1824] 2. AR operation instructions provided:

[1825] Users point their smartphone camera at home appliances.

[1826] The device analyzes camera footage in real time and recognizes home appliances.

[1827] Based on the recognized home appliances, the device will display operating procedures and methods on the screen using AR.

[1828] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes are displayed on the screen.

[1829] 3. Use AI-supported chatbots:

[1830] Users type questions into the chatbot about how to use it and troubleshooting.

[1831] The terminal sends the user's question to the server.

[1832] The server uses generative AI to analyze the user's question and generate an appropriate answer.

[1833] The device receives the generated answer and displays it to the user.

[1834] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[1835] Example prompt: "What should I do if my washing machine won't start?"

[1836] 4. Simplified instructions:

[1837] Users can open the instruction PDF within the app to check the operating instructions.

[1838] The device sends the PDF contents to the server.

[1839] The server uses generative AI to convert the content into a concise description in natural language.

[1840] The terminal will display the generated brief description to the user.

[1841] Example: When you open a PDF containing detailed operating instructions for an oven range, simple instructions such as "Press the timer setting button, enter the time, and press the start button" appear on the screen.

[1842] Example prompt: "Please briefly explain the oven operating procedure described in this PDF."

[1843] 5. New product proposals and e-commerce site referrals:

[1844] Users ask for information about specific features or new products.

[1845] The terminal sends this information to the server.

[1846] The server uses generative AI to analyze the information and generate relevant new product suggestions and purchase links.

[1847] The device displays this information to the user.

[1848] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[1849] Example prompt: "What are the features of the latest smart TVs?"

[1850] This system allows users to quickly and intuitively access information on how to operate and troubleshoot home appliances and new product information, and is also useful for manufacturers to streamline user support and promote new products.

[1851] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1852] Step 1: Launch the app

[1853] User: Launch the dedicated app on your smartphone.

[1854] What happens: The user taps on the app icon.

[1855] Input: The user's touch.

[1856] Output: The app starts.

[1857] Device: Detects that the app has been launched and prepares to enable the camera function.

[1858] What happens: The camera initialization process happens in the background and the camera view appears in the app.

[1859] Input: App launch signal.

[1860] Output: Camera view display.

[1861] Step 2: Provide AR operation instructions

[1862] User: Point your smartphone camera at a home appliance.

[1863] Specific action: To point the camera at a home appliance, hold the smartphone in your hand and frame the product.

[1864] Input: Home appliance footage.

[1865] Output: Acquisition of video data.

[1866] Terminal: Analyzes camera footage in real time and recognizes home appliances.

[1867] Specific operation: An image recognition algorithm is activated, analyzing the video data to detect the shape and characteristics of the home appliance.

[1868] Input: Video data.

[1869] Output: Recognition results of home appliances.

[1870] Terminal: Based on the recognized home appliance, the device displays operating procedures and methods on the screen using AR.

[1871] Specific behavior: Operation guides related to home appliances are overlaid on the screen.

[1872] Input: Recognition results for home appliances.

[1873] Output: AR display operation procedure.

[1874] Step 3: Use an AI-powered chatbot

[1875] Users: Type questions into the chatbot for instructions or troubleshooting.

[1876] Specific actions: Tap the chat button in the app, enter the question text and send it.

[1877] Input: Question text.

[1878] Output: The question is sent.

[1879] Terminal: Receives the user's question and sends it to the server.

[1880] Specific operation: Sends the question text to the server via an HTTP request.

[1881] Input: Question text.

[1882] Output: Sending question data.

[1883] Server: Uses generative AI to analyze the user's question and generate an appropriate answer.

[1884] What it does: The input text is parsed, answers are extracted from relevant databases, and formatted in the appropriate natural language format.

[1885] Input: Question data.

[1886] Output: Response data.

[1887] Terminal: Receives the generated answer and displays it to the user.

[1888] Specific behavior: The reply text received from the server is displayed in the chat window.

[1889] Input: Response data.

[1890] Output: Display the answer.

[1891] Step 4: Simplify the instructions

[1892] User: Open the instruction PDF within the app to view instructions.

[1893] Specific operation: To view an instruction PDF, select the PDF you want from the list within the app.

[1894] Input: Select PDF file.

[1895] Output: PDF content display.

[1896] Device: Send the PDF content to the server.

[1897] Specific operation: Extract the contents of the PDF file as text data and upload it to the server.

[1898] Input: PDF content.

[1899] Output: Sending text data.

[1900] Server: Uses generative AI to convert the content into concise descriptions in natural language.

[1901] What it does: A generative AI model is activated, analyzing the text data and converting it into a concise description.

[1902] Input: Text data.

[1903] Output: A short description.

[1904] Terminal: Display the generated short description to the user.

[1905] Specific behavior: The generated short description is displayed in the PDF viewer.

[1906] Input: A short description.

[1907] Output: Display a short description.

[1908] Step 5: Propose new products and direct them to the EC site

[1909] Users: Ask for information about specific features or new products.

[1910] What it does: Use the search box within the app to enter specific product information.

[1911] Input: Your search query.

[1912] Output: Submitting a search query.

[1913] Device: Sends this information to the server.

[1914] Specific operation: The entered search query is sent to the server via an HTTP request.

[1915] Input: Your search query.

[1916] Output: Submitting a search query.

[1917] Server: Analyzes the information using generative AI and generates relevant new product suggestions and purchase links.

[1918] What it does: The server parses the search query and generates relevant product information and purchase links.

[1919] Input: Your search query.

[1920] Output: New product suggestions and purchase links.

[1921] Terminal: Display this information to the user.

[1922] What happens: The generated suggestions and links are displayed in the results area within the app.

[1923] Input: New product suggestions and purchase links.

[1924] Output: Display information.

[1925] (Application example 1)

[1926] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1927] There is a wide range of information on the operation, maintenance, and troubleshooting of factory machinery, and conventional manuals alone make it difficult for users to respond quickly and accurately.In addition, there are limited means to quickly provide information on new tools and parts, and these issues need to be improved.

[1928] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1929] In this invention, the server includes a means for analyzing a user's question using a generative AI and generating an appropriate answer, a means for displaying operating procedures for the machine in AR using a camera, a means for receiving user input and displaying answers based on the analysis results, and a means for providing a machine function search system. This allows users to quickly and intuitively understand and respond to operating methods and troubleshooting of the machine. It also includes a means for suggesting new products and necessary parts related to the analyzed machine and providing links to e-commerce sites, allowing users to quickly obtain the necessary information.

[1930] "Generative AI" refers to an artificial intelligence model that analyzes user input and generates appropriate answers or information.

[1931] A "camera" is an optical device for visually capturing mechanical equipment.

[1932] "Machinery" refers to mechanical equipment and robots used in factories.

[1933] An "operating procedure" refers to a series of actions or procedures for operating a machine or device correctly.

[1934] "AR display" refers to the use of augmented reality technology to overlay virtual information onto real-world images.

[1935] "User input" refers to information such as operations or questions that a user makes to the system.

[1936] "Analysis results" refer to the answers and information derived by the generating AI based on user input.

[1937] A "function search system" is a system that allows you to search and obtain information about the functions and characteristics of mechanical devices.

[1938] An "AI support chatbot" is an interactive program that uses artificial intelligence to answer users' questions and provide support.

[1939] "Troubleshooting information" refers to information that provides solutions or methods of dealing with problems or malfunctions that occur in mechanical equipment.

[1940] "PDF instruction manual content" refers to an electronic document that contains detailed instructions for operating and maintaining a machine or device.

[1941] "Natural language simplification" refers to the process of expressing technical content in simple, everyday language so that it is easy for users to understand.

[1942] "New products" are new tools or parts that are newly introduced to the market.

[1943] An "e-commerce site" is a website that sells goods and services over the Internet.

[1944] This invention provides a system for intuitive and rapid operation, maintenance, and troubleshooting of factory machinery. The system utilizes generative AI models and augmented reality (AR) technology to allow users to access machinery information using their smartphones.

[1945] Program processing

[1946] Hardware and software used

[1947] Smartphone: High-resolution camera, AR-enabled

[1948] Server: Equipped with high-performance CPU / GPU, natural language processing (NLP) software, generative AI models

[1949] Frameworks: TensorFlow, PyTorch

[1950] API: Google Vision API, OpenAI API

[1951] Data processing and calculation

[1952] 1. Camera image analysis:

[1953] The server uses the Google Vision API to recognize mechanical devices from the smartphone camera image.

[1954] Example: When a user points their smartphone camera at a factory robot, the robot's model number and type are automatically recognized.

[1955] 2. Natural Language Processing:

[1956] The server analyzes the user's question using the OpenAI API and generates an appropriate solution.

[1957] Example: When a user asks, "What should I do when error code E02 appears?", the server generates a solution such as, "Error code E02 indicates that the machine is overheating. Make sure the cooling system is working properly and, if possible, turn the power off and restart."

[1958] 3. Manual Simplification Using Generative AI:

[1959] The server uses a generation AI (GPT-3) to simplify the PDF of the detailed manual in natural language.

[1960] Example: Simplifying a PDF operating manual for a machine generates simple instructions such as "Press the emergency stop button and press the restart button after an interruption."

[1961] 4. New product proposal:

[1962] The server suggests new products and necessary parts related to the analyzed machinery and provides links to e-commerce sites.

[1963] Example: The server analyzes a user's question, "I want to know the features of the latest automation tool," and provides information such as, "This automation tool is equipped with advanced sensor functions and a real-time monitoring system. You can purchase it here (link)."

[1964] Specific examples (prompt examples)

[1965] Example prompt sentence:

[1966] "What should I do when error code E02 appears?"

[1967] In this way, this invention provides a system that streamlines the provision of information on how to operate and troubleshoot machinery in a factory, allowing users to respond intuitively. It also quickly provides information on necessary new products and parts, and easily guides users to e-commerce sites.

[1968] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1969] Step 1:

[1970] The user launches a dedicated app on their smartphone and enables the camera function. The device prepares to display camera footage in real time, allowing the user to capture machinery and equipment with the camera.

[1971] Step 2:

[1972] The user points their smartphone camera at a machine. The device analyzes the camera image and recognizes the machine. The Google Vision API is used to do this, and the model number and type of the machine are sent to the device as analysis results.

[1973] Step 3:

[1974] The device uses AR to display operation procedures and maintenance methods for the recognized machinery. This allows the user to proceed with the work while visually checking the operation procedures. The displayed content is based on data provided in advance by the server.

[1975] Step 4:

[1976] When a user has a question about a machine, they can input it using the chatbot function within the app. The device then sends the question to the server. A specific example of input is a prompt such as, "Please tell me what to do when error code E02 appears."

[1977] Step 5:

[1978] The server uses the OpenAI API to analyze the user's question and generate an appropriate solution. The generated answer is sent from the server to the device and displayed to the user. For example, the user may be provided with information such as, "Error code E02 indicates that the machine is overheating. Please make sure the cooling system is operating properly and, if possible, turn the power off and restart."

[1979] Step 6:

[1980] When a user opens a PDF of a detailed operating manual within the app, the device sends the contents of the PDF to the server. The server uses generative AI (GPT-3) to simplify the PDF contents in natural language and sends the result to the device. The device then displays the simplified instructions to the user. For example, a simple instruction such as "Press the emergency stop button, then press the restart button after interruption" is displayed.

[1981] Step 7:

[1982] When a user requests information about a new tool or part, the device sends the request to the server. Based on the analysis results, the server generates relevant new product information and links to e-commerce sites, and sends them to the device. The device then displays this information to the user. For example, the device may provide information such as, "This automation tool is equipped with advanced sensor functions and a real-time monitoring system. Purchase here (link)."

[1983] Through these processing steps, the present invention provides a system that streamlines the operation and troubleshooting of machinery in a factory, allowing users to respond quickly and intuitively.

[1984] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1985] This invention is a system that revolutionizes home appliance instruction manuals by combining generative AI, augmented reality (AR) technology, and an emotion engine that recognizes user emotions. This system allows users to easily access instructions and troubleshooting information for home appliances through a smartphone app, and also provides support based on the user's emotional state, creating a more personalized user experience.

[1986] System program flow

[1987] 1. Launch the app

[1988] The user launches the dedicated app on their smartphone, which activates the camera function and simultaneously activates the emotion engine.

[1989] The device displays the camera footage in real time and prepares the footage the user captures to be displayed on the screen.

[1990] 2. Providing AR operation instructions

[1991] The user points the camera at a home appliance.

[1992] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[1993] The device sends information about the recognized home appliances to the server, and the server retrieves corresponding operating procedures and troubleshooting data from a database.

[1994] The server sends the acquired operating procedures to the device, and the device displays the operating procedures and troubleshooting methods to the user in AR.

[1995] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[1996] 3. Use AI support chatbots

[1997] Users type their questions into the chatbot about how to use it or troubleshooting.

[1998] The terminal receives the user's question and sends it to the server.

[1999] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[2000] The device receives the generated answer and displays it to the user.

[2001] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[2002] 4. Utilizing the Emotion Engine

[2003] When a user enters a question via text or voice, the emotion engine analyzes the user's emotions.

[2004] The device sends the analysis results to the server, and the server adjusts the response content based on the emotional data.

[2005] Example: If the user displays emotions that indicate confusion or annoyance, the server might respond with a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[2006] 5. Simplified instructions

[2007] The user opens a PDF instruction manual within the app to check the operating instructions.

[2008] The device sends the PDF content to the server, which then uses generative AI to convert the PDF's detailed content into simplified, natural language.

[2009] The terminal will display the generated brief description to the user.

[2010] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[2011] 6. New product proposals and e-commerce site referrals

[2012] Users ask for information about specific features or new products.

[2013] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[2014] The device displays this information to the user.

[2015] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[2016] This system allows users to quickly and intuitively understand and respond to how to operate and troubleshoot home appliances. In addition, responses are given that take the user's emotions into consideration, providing more personalized support. For manufacturers, this system configuration not only improves the quality of user support, but is also useful for promoting new products and sales.

[2017] The processing flow will be explained below.

[2018] Step 1:

[2019] The user launches the dedicated app on their smartphone, which activates the camera function and simultaneously activates the emotion engine.

[2020] Step 2:

[2021] The device displays the camera footage in real time and prepares the footage the user captures to be displayed on the screen.

[2022] Step 3:

[2023] The user points the camera at a home appliance.

[2024] Step 4:

[2025] The device analyzes the camera footage in real time and runs object detection algorithms to recognize home appliances in the footage.

[2026] Step 5:

[2027] The device sends information about the recognized home appliances to the server.

[2028] Step 6:

[2029] The server retrieves the corresponding operating procedures and troubleshooting data from the database.

[2030] Step 7:

[2031] The server sends the acquired operating procedures to the device, and the device displays the operating procedures and troubleshooting methods to the user in AR.

[2032] Example: When a user points the camera at an air conditioner, instructions on how to use the air conditioner remote control and how to set specific modes will be displayed on the screen.

[2033] Step 8:

[2034] Users type their questions into the chatbot about how to use it or troubleshooting.

[2035] Step 9:

[2036] The terminal sends the user's question in text format to the server.

[2037] Step 10:

[2038] The server uses natural language processing techniques to analyze the user's question.

[2039] Step 11:

[2040] Based on the analysis results, the server generates appropriate answers from the database and also analyzes the user's emotions using an emotion engine.

[2041] Step 12:

[2042] The device sends the user's voice and text input data to the emotion engine in real time and analyzes emotions.

[2043] Step 13:

[2044] The emotion engine determines the user's emotional state based on the user's emotional data.

[2045] Step 14:

[2046] The server uses the analysis results of the emotion engine to generate a response message that is tailored to the user's emotional state.

[2047] Example: If a user expresses confusion or frustration, the server tailors and generates a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[2048] Step 15:

[2049] The server sends the generated response message to the terminal, which displays it to the user.

[2050] Step 16:

[2051] The user can refer to the displayed answers to perform the necessary operations or troubleshoot.

[2052] Step 17:

[2053] When a user opens a PDF manual for a particular home appliance within the app, the device sends the contents of the PDF to the server.

[2054] Step 18:

[2055] The server uses generative AI to convert the detailed content of the PDF into simplified, natural language.

[2056] Step 19:

[2057] The server sends the generated natural language short description to the terminal, which displays it to the user.

[2058] Example: When you open a PDF containing detailed operating instructions for an oven range, a simple explanation sent from the server appears: "Press the timer setting button, enter the time, and press the start button."

[2059] Step 20:

[2060] When users have questions about new product information or specific features, they enter that information through the chatbot.

[2061] Step 21:

[2062] The device sends this information to a server, which analyzes it and generates new product suggestions and related e-commerce site purchase links.

[2063] Step 22:

[2064] The server sends the new product information and a purchase link to the device, which displays it to the user. The user can check the new product information and purchase it directly from the purchase link if necessary.

[2065] In this way, users can quickly and intuitively understand and respond to how to operate and troubleshoot their home appliances, and also receive personalized, emotionally sensitive support.

[2066] Example 2

[2067] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2068] Conventional instruction manuals for home appliances have scattered information, making it difficult for users to quickly obtain the information they need. Furthermore, they lack a mechanism for reading emotions and responding appropriately, making it difficult to alleviate user confusion and frustration. Furthermore, they lack a means for providing new product information and links to related e-commerce sites, making it difficult to stimulate users' desire to purchase. The purpose of this invention is to solve these problems.

[2069] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing a user's question using a generation AI and generating an appropriate answer, a means for displaying operating procedures for the device in augmented reality using a camera of a mobile terminal, and a means for analyzing the user's emotions and adjusting the answer content based on the analysis results. This allows the user to quickly and intuitively understand the operating procedures and receive personalized support that takes emotions into consideration. Furthermore, it also makes it easy to promote new products and direct users to related online shopping sites.

[2070] "Generative AI" is an artificial intelligence technology that analyzes questions asked in natural language by users and automatically generates appropriate answers.

[2071] "Mobile terminal" refers to a portable computing device such as a smartphone or tablet.

[2072] A "camera" is a device that converts optical images into digital data for display or analysis in real time.

[2073] "Augmented reality display" is a technology that displays digital information overlaid on real-world images.

[2074] "Equipment" broadly refers to electronic devices such as home appliances.

[2075] "Sentiment analysis" refers to the process of inferring a user's emotional state from their text or voice.

[2076] "Adjusting the content of the response based on the analysis results" means appropriately changing the tone and content of the generated response depending on the user's emotional state.

[2077] "Promotional Information" means information intended to promote new products or services.

[2078] "Related Shopping Sites" means websites where users can purchase products online.

[2079] This invention is a system that combines generative AI, augmented reality (AR) technology, and an emotion engine that recognizes user emotions, and provides operation instructions and troubleshooting information for home appliances to improve the user experience. This system mainly operates through the collaboration of three parties: a server, a device, and a user.

[2080] First, the user launches the dedicated app on their smartphone or tablet, which activates the camera function and activates the emotion engine. The device displays the camera footage in real time and shows the footage the user takes on the screen. At this time, the footage captured by the device's camera is analyzed using an object detection algorithm (e.g., YOLO or ResNet).

[2081] Next, when the user points the camera at a home appliance, the device recognizes the appliance and sends the recognition results to the server. The server retrieves the relevant operating procedures and troubleshooting information from a database (e.g., MySQL, PostgreSQL) and sends the information back to the device. This allows the device to display the operating procedures or troubleshooting methods on the camera image in the form of augmented reality (AR). For example, when the user points the camera at an air conditioner, the screen will show how to use each button on the remote control and how to set a specific mode.

[2082] Additionally, users can input questions about operation methods and troubleshooting into a dedicated chatbot. The device sends these questions as text data to the server, which then uses natural language processing technology (e.g., GPT-3, BERT) to analyze the user's question and generate an appropriate answer. The generated answer is then sent back to the device and displayed to the user in chat format. For example, if a user asks, "My washing machine isn't working," the generating AI will respond with specific instructions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[2083] The emotion engine can also be used to analyze the emotions expressed when a user types a question. The device also sends the emotion analysis results to the server, which then uses the results to tailor the answer to the user's emotional state. For example, if the user expresses confusion or irritation, the server can generate a considerate response such as, "We apologize for the inconvenience. Could you please try the following steps?"

[2084] The contents of PDF manuals can also be simplified and displayed. When a user opens a PDF manual within the app, the device sends the contents of the PDF as text data to the server. The server then uses generative AI to simplify this text data, converts it into natural language, and sends it back to the device. The device then displays the simplified instructions to the user. For example, when a user opens a PDF containing detailed operating instructions for a microwave oven, simple instructions such as "Press the timer setting button, enter the time, and press the start button" are displayed.

[2085] Furthermore, when a user requests information about a specific feature or new product, the device sends this input information to the server. The server analyzes the information, generates new product suggestions and links to related online shopping sites, and sends them to the device. The device then displays this information to the user. For example, if a user asks, "I want to know the features of the latest smart TV," the server might respond, "This smart TV has high resolution and voice recognition capabilities. You can purchase it here (link)."

[2086] This system allows users to quickly and intuitively understand and respond to how to operate and troubleshoot their home appliances, and also provides personalized support that takes into account the user's emotions, resulting in a more satisfying user experience.

[2087] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2088] Step 1:

[2089] The user launches a dedicated app on their smartphone.

[2090] Specific action: The user taps the app icon on the smartphone's home screen.

[2091] Input: Tap input from the user

[2092] Output: Launching the app and displaying the initial screen

[2093] Step 2:

[2094] The device will enable the app's camera function and emotion engine.

[2095] Specific operation: The device will launch the camera function and ask the user for permission to use the camera. At the same time, the emotion engine will also launch in the background.

[2096] Input: App launch signal, camera access permission from user

[2097] Output: Streaming camera footage and emotion engine status

[2098] Step 3:

[2099] The user points the camera at a home appliance.

[2100] Specific action: The user places a home appliance within the camera frame.

[2101] Input: Camera video data

[2102] Output: Video frame showing home appliances

[2103] Step 4:

[2104] The device analyzes camera footage in real time and runs object detection algorithms to recognize home appliances.

[2105] How it works: An object detection algorithm (e.g., YOLO or ResNet) extracts features from video frames and matches them with known home appliances in a database.

[2106] Input: Camera image frame

[2107] Output: ID or feature information of the recognized home appliance

[2108] Step 5:

[2109] The device sends information about the recognized home appliances to the server.

[2110] Specific operation: The device sends a request including the home appliance ID and image data to the server.

[2111] Input: ID or characteristic information of home appliance

[2112] Output: Request message to the server

[2113] Step 6:

[2114] The server retrieves the corresponding operating procedures and troubleshooting data from the database.

[2115] Specific operation: The server executes an SQL query based on the ID of the home appliance to obtain related information.

[2116] Input: Home appliance ID

[2117] Output: Records of operating procedures and troubleshooting information

[2118] Step 7:

[2119] The server transmits the acquired operating procedure to the terminal.

[2120] Specific operation: The server generates a JSON format response and sends it to the terminal.

[2121] Input: Records of operating procedures and troubleshooting information

[2122] Output: Response message to the terminal

[2123] Step 8:

[2124] The device provides users with operating instructions and troubleshooting methods using AR displays.

[2125] Specific operation: The device uses an AR library (e.g., ARKit, ARCore) to overlay operating instructions on the camera image.

[2126] Input: Operation procedure information of response message

[2127] Output: AR-displayed operating procedures and troubleshooting information

[2128] Example: When a user points the camera at an air conditioner, the screen will show how to use each button on the remote control and how to set a specific mode.

[2129] Step 9:

[2130] Users type their questions into the chatbot about how to use it or troubleshooting.

[2131] Specific behavior: The user types a question into the chat box within the app and presses the send button.

[2132] Input: User's question text

[2133] Output: Message to send to the chatbot

[2134] Step 10:

[2135] The terminal sends the user's question to the server.

[2136] Specific operation: The terminal sends text data to the server.

[2137] Input: User's question text

[2138] Output: Request message to the server

[2139] Step 11:

[2140] The server uses natural language processing technology to analyze the user's question and generate an appropriate answer.

[2141] How it works: The server calls the generative AI (e.g., GPT-3), passes the user's question as a prompt, and retrieves the generated answer.

[2142] Input: User's question text

[2143] Output: Generated answer text

[2144] Step 12:

[2145] The terminal displays the generated answer to the user.

[2146] Specific behavior: The device displays the generated text in the chat box and notifies the user.

[2147] Input: Generated answer text

[2148] Output: The answer text that is displayed to the user

[2149] Example: When a user asks, "My washing machine isn't working," the server responds with specific solutions, such as, "Make sure the power is on and that the drain hose isn't clogged."

[2150] Step 13:

[2151] When a user types a question, the emotion engine analyzes the user's emotions.

[2152] Specific operation: The emotion engine analyzes input text and voice data and estimates the emotional state (e.g., confusion, irritation).

[2153] Input: User question text or voice data

[2154] Output: Emotion analysis results

[2155] Step 14:

[2156] The terminal transmits the analysis results to the server.

[2157] Specific operation: The device sends a request including emotion data to the server.

[2158] Input: Sentiment analysis results

[2159] Output: Request message to the server

[2160] Step 15:

[2161] The server adjusts the response content based on the emotional data.

[2162] Specific behavior: The server adjusts the tone and content of the response text depending on the user's emotional state (e.g., using emojis and softer language).

[2163] Input: Sentiment analysis results, generated answer text

[2164] Output: Adjusted answer text

[2165] Example: If a user expresses confusion or frustration, the server responds with a considerate response such as, "We're sorry for the inconvenience. Would you like to try the steps below?"

[2166] Step 16:

[2167] The user opens a PDF instruction manual within the app to check the operating instructions.

[2168] Specific behavior: The user taps the instruction icon, which launches the PDF viewer.

[2169] Input: User tap input

[2170] Output: Launch PDF viewer and display PDF

[2171] Step 17:

[2172] The device sends the PDF contents to the server.

[2173] Specific operation: The device extracts the text data from the PDF and sends it to the server.

[2174] Input: PDF text data

[2175] Output: Request message to the server

[2176] Step 18:

[2177] The server uses generative AI to simplify the detailed content of the PDF.

[2178] Specific behavior: The server sends a prompt to the generation AI, instructing it to generate simplified text.

[2179] Input: PDF text data

[2180] Output: Simplified descriptive text

[2181] Step 19:

[2182] The terminal will display the generated brief description to the user.

[2183] What happens: The device retrieves the abbreviated text and displays it as an overlay on the PDF viewer.

[2184] Input: Simplified description text

[2185] Output: A simplified description that is displayed to the user.

[2186] Example: When you open a PDF containing detailed operating instructions for an oven range, you see a simple explanation: "Press the timer setting button, enter the time, and press the start button."

[2187] Step 20:

[2188] Users ask for information about specific features or new products.

[2189] Specific behavior: A user types a question such as "I want to know the features of the latest smart TVs."

[2190] Input: User's question text

[2191] Output: A message with the question text

[2192] Step 21:

[2193] The terminal sends this information to the server.

[2194] Specific operation: The terminal sends the question text to the server.

[2195] Input: User's question text

[2196] Output: Request message to the server

[2197] Step 22:

[2198] The server analyzes the data and generates new product suggestions and links to related online shopping sites.

[2199] Specific operation: The server analyzes the content of the question, retrieves related information from a product information database, and generates links to online shopping sites.

[2200] Input: User's question text

[2201] Output: New product proposals and online shopping site links

[2202] Step 23:

[2203] The device displays this information to the user.

[2204] Specific operation: The device displays the generated information and links in a chat box or dedicated display area.

[2205] Input: New product proposal and online shopping site link

[2206] Output: New product suggestions and online shopping links displayed to users

[2207] Example: A user asks, "What are the features of the latest smart TV?" The server responds, "This smart TV has high resolution and voice recognition. Buy it here (link)."

[2208] (Application example 2)

[2209] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2210] Today's consumers often seek detailed information and reviews before purchasing a product, especially when shopping in a virtual store. However, no system exists that allows consumers to obtain detailed information while browsing products in a virtual store and receive appropriate support tailored to their emotions. Furthermore, conventional systems have difficulty responding appropriately when users become confused or frustrated, and lack means to improve the quality of the consumer experience. To address these issues, the present invention aims to dramatically improve the consumer experience in a virtual store.

[2211] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2212] In this invention, the server includes means for analyzing a user's question using a generation AI and generating an appropriate answer, means for displaying operating procedures for home appliances in AR using a smartphone camera, means for receiving user input and displaying an answer based on the analysis results, means for providing detailed product information, usage instructions, and reviews in a virtual store, and means for analyzing the user's emotional state and providing appropriate assistance according to the user's emotion. This enables consumers to intuitively obtain detailed information while browsing products in a virtual store and receive appropriate support according to their emotional state.

[2213] "Generative AI" is an artificial intelligence technology that analyzes user input and questions and automatically generates appropriate information and answers.

[2214] "AR display" is a technology that uses augmented reality technology to overlay digital information on actual images.

[2215] "User input" refers to the act of a user giving instructions or questions to the system via text or voice.

[2216] "Analysis results" are information obtained as a result of the generative AI analyzing the user's input and questions.

[2217] A "virtual store" is an online store that is a virtual shopping space set up on the Internet where users can browse and purchase items.

[2218] "Detailed product information" refers to information that users want to know before purchasing, such as product specifications, functions, instructions for use, and price.

[2219] "Reviews" refer to ratings and opinions of consumers who have already used a product.

[2220] "Emotional state" indicates the user's current psychological state or emotion, and is a category of emotions such as confusion, irritation, and excitement.

[2221] "Appropriate assistance" is specific and useful support and advice provided according to the user's emotional state.

[2222] This invention realizes a system that provides product information and enhances user support in a virtual store by combining generative AI, augmented reality (AR) technology, and an emotion recognition engine. Specific embodiments of the system are described below.

[2223] First, users launch a dedicated application on their smartphone or smart glasses. This application includes a camera function and displays camera images in real time, allowing users to take photos of products in the virtual store and obtain detailed information.

[2224] Hardware and software used

[2225] 1. Hardware

[2226] Smartphone or smart glasses camera: Captures video in real time and is used to recognize the product being photographed.

[2227] Server: Hosts the generative AI model, sentiment analysis engine, and database, and performs analytical processing.

[2228] 2. Software

[2229] OpenCV: Captures camera images and uses them for product recognition.

[2230] Transformers (transformers package): Used to perform generative AI and sentiment analysis.

[2231] HTTP request: Communicate with external APIs or product databases to obtain product information.

[2232] Data processing and calculation

[2233] Camera video analysis:

[2234] The device analyzes the real-time video captured by the user and recognizes the products. To do this, it runs an object detection algorithm and, once the product is identified, sends its name to the server.

[2235] Get product information:

[2236] The server receives the product name and retrieves the product's details, usage instructions, and reviews from a product database or external API. The retrieved information is sent to the user's device and displayed on the screen.

[2237] Sentiment analysis and providing appropriate support:

[2238] When a user enters information via text or voice, an emotion recognition engine analyzes their emotional state. If the input indicates frustration or confusion, the system generates a thoughtful response that reflects that emotion. The response is then sent to the user's device and displayed on the screen.

[2239] Specific examples

[2240] Suppose a user is walking around a virtual store and looking at products using smart glasses. For example, if the user points the camera at a smart TV and wants to get more information about it, they might enter the following prompt:

[2241] What features does this smart TV have?

[2242] Actions based on sentiment analysis results:

[2243] If the user's input indicates frustration, the system generates a considerate response such as:

[2244] We apologize for any inconvenience. This smart TV has high resolution and voice recognition. Purchase here (link).

[2245] Under normal circumstances, this will produce a response like this:

[2246] This smart TV has high resolution and voice recognition. Buy it here.

[2247] This allows users to intuitively obtain product information in a virtual store and receive appropriate support based on their emotions. This system not only dramatically improves the consumer experience, but also encourages purchases in the virtual store.

[2248] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2249] Step 1:

[2250] The user launches the dedicated application on their smartphone or smart glasses, which activates the camera function and prepares to display images in real time, allowing the images captured by the user to be displayed on the screen.

[2251] Input: User application launch

[2252] Output: Real-time video display

[2253] Step 2:

[2254] The user points the camera at a product in a virtual store, and the device analyzes the camera image and uses an object detection algorithm to recognize the product. The recognized product information is identified and the product name is extracted.

[2255] Input: Camera image

[2256] Output: Recognized product name

[2257] Step 3:

[2258] The device sends the extracted product name to the server, which retrieves detailed information, usage instructions, and reviews from a product database or external API based on the product name. The retrieved information is formatted on the server side and sent to the device.

[2259] Input: Product name

[2260] Output: Product details (description, usage, reviews, etc.)

[2261] Step 4:

[2262] The terminal displays the acquired product details to the user, who can then check the product details on the screen and request more information if necessary.

[2263] Input: Product details information

[2264] Output: Display detailed product information

[2265] Step 5:

[2266] The user inputs a question via text or voice. The emotion recognition engine analyzes this input and determines the user's emotional state (e.g., irritation, confusion). The analyzed emotion information and the user's question are sent to the server.

[2267] Input: User text and voice input

[2268] Output: Parsed sentiment information and question

[2269] Step 6:

[2270] The server uses a generative AI model to generate an appropriate response based on the user's emotional state and question. If the emotional state is irritation, it is converted into a considerate sentence. The generated response is sent to the device.

[2271] Input: Parsed emotion information and question

[2272] Output: The generated answer

[2273] Step 7:

[2274] The device displays the generated answer to the user, who can then check the answer on the screen and receive appropriate support.

[2275] Input: Generated Answer

[2276] Output: Show answer

[2277] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2278] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2279] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2280] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2281] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2282] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2283] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2284] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2285] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2286] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2287] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2288] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2289] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2290] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2291] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2292] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2293] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2294] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2295] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2296] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2297] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2298] The following is further disclosed regarding the above embodiment.

[2299] (Claim 1)

[2300] A means of using generative AI to analyze user questions and generate appropriate answers;

[2301] A means for displaying operating procedures for home appliances in AR using a smartphone camera;

[2302] A means of receiving user input and displaying an answer based on the analysis results;

[2303] A means for providing a function search system for a home appliance;

[2304] A system including:

[2305] (Claim 2)

[2306] 10. The system of claim 1, further comprising means for using an AI support chatbot to provide troubleshooting information in response to a user's question.

[2307] (Claim 3)

[2308] 2. The system according to claim 1, further comprising means for displaying the PDF contents of the instruction manual in a simplified form in natural language.

[2309] "Example 1"

[2310] (Claim 1)

[2311] A means of using generative AI to analyze user questions and generate appropriate answers;

[2312] a means for augmented reality display of operating procedures for the device using a video device of the mobile device;

[2313] A means of receiving user input and displaying an answer based on the analysis results;

[2314] means for providing a device function search system;

[2315] A means to simplify and display the contents of the PDF manual in natural language,

[2316] A means of using generative AI to analyze the information and generate relevant new product suggestions and purchase links;

[2317] A system including:

[2318] (Claim 2)

[2319] 10. The system of claim 1, further comprising means for using an AI support chatbot to provide troubleshooting information in response to a user's question.

[2320] (Claim 3)

[2321] The system according to claim 1, further comprising means for, when a camera is pointed at a device within the app, displaying an AR display of operating procedures based on the recognition of the device.

[2322] "Application Example 1"

[2323] (Claim 1)

[2324] A means of using generative AI to analyze user questions and generate appropriate answers;

[2325] a means for displaying an AR display of an operating procedure for a machine using a camera;

[2326] A means of receiving user input and displaying an answer based on the analysis results;

[2327] means for providing a function search system for a machine;

[2328] A system including:

[2329] (Claim 2)

[2330] 10. The system of claim 1, further comprising means for using an AI support chatbot to provide troubleshooting information in response to a user's question.

[2331] (Claim 3)

[2332] 2. The system according to claim 1, further comprising means for displaying the PDF contents of the instruction manual in a simplified form in natural language.

[2333] (Claim 4)

[2334] 10. The system of claim 1, further comprising means for suggesting new products and required parts related to the analyzed machinery and providing links to e-commerce sites.

[2335] "Example 2: Combining Emotion Engines"

[2336] (Claim 1)

[2337] A means of using generative AI to analyze user questions and generate appropriate answers;

[2338] a means for augmented reality display of operation procedures for the device using a camera of the mobile terminal;

[2339] A means of receiving u...

Claims

1. A means of using generative AI to analyze user questions and generate appropriate answers; A means for displaying operating procedures for home appliances in AR using a smartphone camera; A means of receiving user input and displaying an answer based on the analysis results; A means for providing a function search system for a home appliance; A system including:

2. The system of claim 1, further comprising means for providing troubleshooting information in response to a user's question using an AI support chatbot.

3. 2. The system according to claim 1, further comprising means for displaying the PDF contents of the manual in a simplified form in natural language.

Images