system

Abstract

Provide a system. The present invention provides a means for a user to send an inquiry for confirming an operation method through a message application; a server receiving a query from the user and analyzing the query using a natural language processing model; A server acquires appropriate manual information from a database based on the analysis results; a means for generating a response message based on the acquired manual information by the server; means for the server to send the response message to the user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern life and business environments, many users need information such as technical support, operation instructions, and confirmation of pricing plans, but responding to these requests quickly and appropriately is not easy. In particular, providing real-time support through messaging applications is time-consuming and laborious. Similar challenges exist when communicating with and operating devices with foreigners. A means to efficiently solve these problems is needed. [Means for solving the problem]

[0005] The system of the present invention allows users to quickly and appropriately receive confirmation of operation methods, image-based advice, and pricing plan information via a messaging application. Specifically, when a user sends an inquiry or image via the messaging application, the server receives it and analyzes it using a natural language processing model and image analysis technology. Based on the analysis results, the server retrieves appropriate manual information and pricing plan information from a database, generates a response message, and sends it to the user. This system allows users to receive efficient support and respond quickly to a variety of requests.

[0006] A "messaging application" is software that allows users to send and receive data such as text messages and images to each other.

[0007] A "user" is a person or organization that uses a messaging application to make an inquiry or obtain information.

[0008] A "server" is a computer system that has the function of receiving inquiries and images from users, performing the necessary analysis and retrieving information from a database, and generating and sending a response message.

[0009] A "natural language processing model" is an artificial intelligence technology used to analyze text messages from users and understand their meaning and intent.

[0010] "Image analysis technology" is a technology for analyzing image data and identifying and classifying information in the image.

[0011] "Database" means a system or location where the Server stores, searches, and retrieves necessary manual information and pricing plan information.

[0012] "Manual information" is information that contains detailed explanations of specific operating methods and procedures.

[0013] A "response message" is a response that a server generates and sends in response to a user's inquiry.

[0014] A "fee plan" is a plan that indicates the structure and details of the fee to be paid by a user for a service that the user uses. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, advice using images, and information on rate plans through a message application.

[0037] 1. Checking the operation method

[0038] When a user uses a messaging application to make a query such as "How do I back up my new smartphone?", the server receives the message. The server uses a natural language processing model to analyze the user's message and extract queries related to "how to back up." The server retrieves appropriate manual information from a database based on this query. Based on the retrieved manual information, the server generates a response message for the user and sends it to the user's device. The user then backs up their smartphone based on the received information.

[0039] 2. Photo advice

[0040] When a user sends a screenshot of the part of the operation they are having trouble with via a messaging application, the server receives the image. The server then analyzes the image using image analysis technology to identify the relevant operating procedure. Based on the analysis results, the server retrieves relevant manual information from a database and generates a response message including specific operating procedures and reference images. The generated response message is then sent from the server to the user's device. The user then refers to the received information and continues operating the device.

[0041] 3. Providing information on pricing plans

[0042] When a user sends a query requesting information about pricing plans through a messaging application, the server receives the message. The server uses a natural language processing model to analyze the user message and extract queries related to "pricing plans." The server then retrieves the user's usage data from a database. The server uses a generative AI model to calculate the optimal pricing plan based on the retrieved usage data. Based on the calculated pricing plan details, the server generates a response message for the user and sends it to the user's device. The user compares and considers the proposed pricing plans based on the received information.

[0043] Specific examples

[0044] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives and analyzes the message, then retrieves the relevant manual information from the database. Based on the retrieved information, the server generates and sends a response message containing concise and easy-to-understand backup instructions. The user can then use the received information to successfully back up their smartphone.

[0045] Furthermore, if a user has trouble with the smartphone's settings screen, they can send a screenshot. The server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant manual information from a database and sends the user a response message containing detailed instructions and reference images. This allows the user to understand the specific operation method and solve the problem.

[0046] When providing information on pricing plans, if the user sends a message saying "Tell me the best pricing plan," the server will analyze the user's usage data, calculate the best pricing plan, and propose it to them. The user can then check the details of the plans provided and select the one that best suits them.

[0047] The processing flow will be explained below.

[0048] Check operation method

[0049] Step 1:

[0050] A user uses a messaging application to send a message saying, "How do I back up my new phone?"

[0051] Step 2:

[0052] A server receives a message from the user.

[0053] Step 3:

[0054] The server analyzes the user message using a natural language processing model and extracts the query "How to back up."

[0055] Step 4:

[0056] The server retrieves relevant manual information from a database based on the query.

[0057] Step 5:

[0058] The server generates a response message based on the acquired manual information.

[0059] Step 6:

[0060] The server generates a response message and sends it to the user's terminal.

[0061] Step 7:

[0062] The smartphone is backed up based on the response message received by the user.

[0063] Photo-based advice

[0064] Step 1:

[0065] The user sends a screenshot of the part of the operation they are having trouble with via a messaging application.

[0066] Step 2:

[0067] A server receives the image from the user.

[0068] Step 3:

[0069] The server uses image analysis technology to analyze the received images and identify the problem areas.

[0070] Step 4:

[0071] The server retrieves related manual information from a database based on the analysis results.

[0072] Step 5:

[0073] The server generates a response message based on the acquired manual information and reference image.

[0074] Step 6:

[0075] The server generates a response message and sends it to the user's terminal.

[0076] Step 7:

[0077] The user continues the operation while referring to the response message received.

[0078] Providing information on pricing plans

[0079] Step 1:

[0080] A user uses a messaging app to send a message saying, "What's the best rate plan?"

[0081] Step 2:

[0082] A server receives a message from the user.

[0083] Step 3:

[0084] The server analyzes the user message using a natural language processing model and extracts the query "price plan."

[0085] Step 4:

[0086] The server retrieves the user usage data from the database based on the query.

[0087] Step 5:

[0088] The server uses the generated AI model to calculate the optimal pricing plan based on the acquired usage data.

[0089] Step 6:

[0090] The server generates a response message based on the calculated details of the pricing plan.

[0091] Step 7:

[0092] The server generates a response message and sends it to the user's terminal.

[0093] Step 8:

[0094] Based on the response message received by the user, the user compares and considers the proposed pricing plans.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] The challenge is to effectively utilize messaging applications, a modern means of communication, to quickly and appropriately confirm operation procedures, resolve problems, and recommend optimal pricing plans. Conventional systems require users to search multiple sources of information and obtain the necessary information on their own, which can be a cumbersome process. In addition, selecting a pricing plan requires users to perform complex calculations, making it difficult to choose the optimal plan.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes: means for sending an inquiry for the user to confirm an operation method through a message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain appropriate information from a database based on the analysis result; means for the server to generate a response message based on the obtained information; means for the server to send the response message to the user; means for the user to perform an operation based on the response message displayed on the terminal; means for the user to send an image through the message application; means for the server to receive an image from the user and analyze it using image analysis technology; means for the server to obtain related information from a database based on the analysis result; means for the server to generate a response message based on the obtained information and a reference image; means for the server to send the response message to the user; means for the user to perform an operation based on the response message displayed on the terminal; means for the user to send an inquiry requesting information on a rate plan through the message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain usage data of the user from a database; and means for the server to calculate an optimal rate plan based on the obtained usage data. The system includes a means for the server to generate a response message based on the calculated details of the pricing plan, a means for the server to send the response message to the user, and a means for the user to consider plans based on the response message displayed on the terminal, thereby enabling the user to quickly and appropriately obtain the necessary information, efficiently solve the problems they are facing, and select the most suitable pricing plan without going through complicated processes.

[0100] "User" refers to an individual or corporation that uses the system.

[0101] "Messaging application" refers to software or applications for sending and receiving information such as text messages and images.

[0102] An "inquiry" refers to a question or request made by a user to the system.

[0103] "Server" refers to a computer system that receives and processes requests from users.

[0104] A "natural language processing model" refers to algorithms and tools for analyzing human language.

[0105] "Analysis" refers to the act of understanding and processing received data using natural language processing models and image analysis techniques.

[0106] A "database" refers to a system for organizing and storing data and efficiently retrieving necessary information.

[0107] "Manual information" refers to documents and data that explain operation methods and procedures.

[0108] A "response message" refers to a message that a server generates and sends in response to a user's query.

[0109] "Image analysis technology" refers to technology for analyzing image data and understanding its contents.

[0110] A "reference image" refers to a reference image provided for the purpose of explaining or instructing a user.

[0111] "Usage status data" refers to data related to a user's service usage status and history.

[0112] "Price plan" refers to various plans that determine the fee structure for the services that a user uses.

[0113] The "optimal pricing plan" refers to the pricing plan that is most profitable for the user based on the user's usage data.

[0114] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a message application. This system performs various data processing and data calculations using the following hardware and software.

[0115] 1. Hardware and Software Used

[0116] Server: A high-performance server computer is required. In this system, it stores the database and AI models and performs calculations.

[0117] Device: A mobile device operated by a user, such as a smartphone or tablet, that must have a messaging application installed.

[0118] Message application: Software for sending and receiving messages between users and servers.

[0119] Natural language processing model: A model for analyzing messages from users, such as BERT or OpenAI's GPT-3 (registered trademark).

[0120] Database: This stores manual information and user usage data. For example, a database management system such as MySQL (registered trademark) is used.

[0121] Image analysis technology: Image analysis tools such as Google® Cloud Vision API are used.

[0122] Generative AI model: For example, OpenAI's GPT-3 is used as an AI model to calculate the optimal pricing plan based on usage data.

[0123] 2. Implementing the Invention

[0124] Check operation method

[0125] A user uses a messaging application to send a message saying, "Tell me how to back up my new smartphone." The server receives this message and analyzes it using a natural language processing model (e.g., BERT) to extract queries related to "how to back up." Based on the extracted query, the server retrieves the appropriate manual information from a MySQL database, generates a response message, and sends it to the user's device. The user then backs up their smartphone based on the received information.

[0126] Photo-based advice

[0127] The user sends a screenshot of the part of the operation where they are having trouble to the server via a messaging application. The server receives the image and analyzes it using the Google Cloud Vision API. Based on the analysis results, the server retrieves relevant manual information from a MySQL database, generates a response message containing specific operating procedures and reference images, and sends it to the user's device. The user can continue operating the device by referring to the received information.

[0128] Providing information on pricing plans

[0129] The user sends a message via a messaging application saying, "What is the best pricing plan?" The server receives this message, analyzes it using a natural language processing model (for example, OpenAI's GPT-3), and extracts queries related to "pricing plans." The server retrieves the user's usage data from a MySQL database and calculates the best pricing plan using GPT-3 based on the retrieved data. A response message is generated based on the details of the calculated pricing plan and sent to the user's device. The user then checks the details of the provided plans and selects the most suitable plan.

[0130] Specific examples

[0131] Check operation method

[0132] Example user question: "How do I back up my new phone?"

[0133] Example prompt: "Generate a message to return when a user is asked how to back up their new phone."

[0134] Photo-based advice

[0135] Example user question: "Send a screenshot of your smartphone's settings screen"

[0136] Example prompt: "Generate a response message when a user sends you a screenshot of their settings screen."

[0137] Providing information on pricing plans

[0138] Example user question: "What is the best pricing plan?"

[0139] Example prompt: "Generate a response message when a user asks what the best pricing plan is."

[0140] This allows users to quickly and appropriately obtain the necessary information, efficiently resolve problems, and select the most suitable rate plan without having to go through complicated processes.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Check operation method

[0143] Step 1:

[0144] A query is sent to the user to confirm the operation method.

[0145] Type: A user uses the Messages app to type the text message "How do I back up my new phone?"

[0146] Output: The message is sent to the server.

[0147] Step 2:

[0148] A server receives the user query and analyzes it using a natural language processing model.

[0149] Input: The server receives a message sent by the user.

[0150] Data processing: The server analyzes the message using a natural language processing model (e.g., BERT) and extracts keywords related to "backup methods."

[0151] Output: Queries related to "Backup Method" are generated.

[0152] Step 3:

[0153] The server retrieves the appropriate information from the database.

[0154] Input: Queries related to "how to back up".

[0155] Data processing: The server searches and retrieves the relevant manual information from the MySQL database.

[0156] Output: The retrieved manual information.

[0157] Step 4:

[0158] The server generates a response message based on the acquired information.

[0159] Input: The retrieved manual information.

[0160] Data processing: Based on the obtained manual information, the server generates a response message in a format that is easy for the user to understand.

[0161] Output: A response message is generated.

[0162] Step 5:

[0163] The server sends the response message to the user.

[0164] Input: The response message.

[0165] Output: A reply message is sent to the user's device via the Messages application.

[0166] Step 6:

[0167] The user performs an operation based on the response message displayed on the terminal.

[0168] Input: The response message.

[0169] What happens: The user follows the instructions in the response message to start backing up their smartphone.

[0170] Output: Your phone is backed up.

[0171] Photo-based advice

[0172] Step 1:

[0173] A user sends an image through a messaging application.

[0174] Input: The user attaches a screenshot of the problem and sends it via messaging.

[0175] Output: The screenshot is sent to the server.

[0176] Step 2:

[0177] A server receives the image from the user and analyzes it using image analysis techniques.

[0178] Input: A screenshot submitted by the user.

[0179] Data processing: The server uses the Google Cloud Vision API to analyze the screenshots and identify specific problem areas and operational steps.

[0180] Output: Analysis results.

[0181] Step 3:

[0182] The server retrieves the relevant information from the database.

[0183] Input: Analysis results.

[0184] Data processing: The server searches and retrieves the relevant manual information and reference images from the MySQL database.

[0185] Output: Retrieved information and reference images.

[0186] Step 4:

[0187] The server generates a response message based on the acquired information and the reference image.

[0188] Input: Retrieved information and reference image.

[0189] Data processing: Based on this information, the server generates a response message in a format that is easy for the user to understand.

[0190] Output: A response message is generated.

[0191] Step 5:

[0192] The server sends the response message to the user.

[0193] Input: The response message.

[0194] Output: A reply message is sent to the user's device via the Messages application.

[0195] Step 6:

[0196] The user performs the operation by referring to the response message displayed on the terminal.

[0197] Input: The response message.

[0198] Specific actions: The user follows the response message and uses the displayed steps and reference images to solve the problem.

[0199] Output: Problem solved.

[0200] Providing information on pricing plans

[0201] Step 1:

[0202] A user sends a request for pricing plan information through a messaging application.

[0203] Input: A user sends a message asking, "What is the best pricing plan?"

[0204] Output: The query is sent to the server.

[0205] Step 2:

[0206] A server receives the user query and analyzes it using a natural language processing model.

[0207] Input: A query message from the user.

[0208] Data processing: The server analyzes the message using a natural language processing model such as OpenAI's GPT-3 to extract queries related to "price plans."

[0209] Output: Queries related to "price plan" are generated.

[0210] Step 3:

[0211] The server retrieves user usage data from the database.

[0212] Input: Queries related to "price plans".

[0213] Data processing: The server retrieves user usage data from the MySQL database.

[0214] Output: The captured usage data.

[0215] Step 4:

[0216] The optimal pricing plan is calculated based on the usage data acquired by the server.

[0217] Input: Captured usage data.

[0218] Data processing: The server uses a generative AI model (e.g., OpenAI's GPT-3) to calculate the optimal pricing plan.

[0219] Output: Best pricing plan.

[0220] Step 5:

[0221] The server generates a response message based on the calculated details of the rate plan.

[0222] Input: Your best rate plan information.

[0223] Data processing: The server generates a response message in a format that is easy for the user to understand.

[0224] Output: A response message is generated.

[0225] Step 6:

[0226] The server sends the response message to the user.

[0227] Input: The response message.

[0228] Output: A reply message is sent to the user's device via the Messages application.

[0229] Step 7:

[0230] The user considers a plan based on the response message displayed on the terminal.

[0231] Input: The response message.

[0232] Specific behavior: The user reviews the details of the offered pricing plans and selects the most suitable plan.

[0233] Output: The user selects the best pricing plan.

[0234] (Application example 1)

[0235] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0236] Modern users face numerous configuration and security issues related to digital devices and security measures. While a support system is needed to quickly and appropriately resolve these issues, conventional methods often require complex operation procedures and responses to anomaly detection alerts, making them difficult for users to understand. While it is also important to propose optimal security and pricing plans, there is a lack of systems that can effectively accomplish this. The objective of this invention is to solve these problems by providing a system that allows users to quickly and appropriately confirm operation procedures, respond to anomaly detection, and receive recommendations for optimal plans.

[0237] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0238] In this invention, the server includes: means for a user to send an inquiry about operation instructions or how to respond to an anomaly detection alert via a message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain appropriate manual information or response information from a database based on the analysis results; means for the server to generate a response message based on the obtained information; means for the server to send the response message to the user; means for the server to receive an anomaly detection alert image from the user and analyze it using image analysis technology; and means for the server to obtain manual information about the cause of the anomaly and how to deal with it based on the analysis results. This allows the user to easily and quickly confirm operation instructions, respond to an anomaly detection, and receive suggestions for optimal security and pricing plans.

[0239] "User" refers to an individual or company that uses the system or service.

[0240] A "messaging application" is software for sending and receiving text messages and images.

[0241] "Method of operation" refers to the steps or techniques for using a digital device or software.

[0242] An "anomaly detection alert" is a warning message that notifies the user when the system detects an anomaly.

[0243] A "query" is a message sent by a user to the system requesting information or instructions.

[0244] A "server" is a computer system that receives and analyzes messages from users and provides the necessary information.

[0245] A "natural language processing model" refers to algorithms and technologies used to analyze and understand the meaning of text messages entered by users.

[0246] "Analysis results" are the results of analyzing data obtained using natural language processing models and image analysis technology.

[0247] "Manual information" refers to materials that include specific information required by users, such as operation methods and procedures for responding to abnormality detection.

[0248] A "response message" is a message containing a response generated by a server in response to a user's inquiry.

[0249] "Image analysis technology" is a technology for processing transmitted images and analyzing their contents.

[0250] A "reference image" is an image used to explain operating procedures and how to respond to abnormalities.

[0251] "Price Plan" refers to the pricing structure or plan that a User can select to use the Service.

[0252] "Usage status data" refers to data that indicates a user's service usage history and trends.

[0253] "Security Plan" refers to a combination of security measures designed to protect a user's devices and data.

[0254] A "generative AI model" is an artificial intelligence model that generates new information and responses based on data.

[0255] A "prompt" refers to an instruction or question that is input to a model.

[0256] The present invention provides a system that allows a user to quickly and appropriately check operation instructions and respond to an anomaly detection alert through a message application. Specific embodiments of this system will be described below.

[0257] First, when a user wants to check how to operate a device or how to respond to an anomaly detection alert, they send an inquiry from their device via a messaging application. Here, "user" refers to individuals or companies that use the system or service. "Messaging application" is software for sending and receiving text messages and images. "Operation method" refers to the steps and methods for using digital devices and software.

[0258] The server receives the user's inquiry and analyzes it using a natural language processing model (software used: OpenAI GPT-3). Based on the analysis results, the server retrieves "manual information" from a database (software used: MySQL, PostgreSQL). A "natural language processing model" refers to the algorithms and technologies used to analyze and understand the meaning of text messages entered by users. The "database" contains detailed manual information on operating procedures and how to respond to abnormalities.

[0259] Next, the server generates a "response message" based on the retrieved manual information and sends it to the user's terminal. A "response message" is a message that the server generates containing a response to a user's inquiry. For example, if a user sends a message saying "Please tell me how to change the firewall settings," the server generates a response message based on the relevant manual information and sends it to the user.

[0260] We will also explain the case where a user sends an anomaly detection alert with a screenshot attached. The image sent from the user's device is analyzed on the server using "image analysis technology" (software used: OpenCV, TENSORFLOW (registered trademark)). Based on the analysis results, the server retrieves manual information on the "cause of the anomaly" and its "measures" from the database and sends it to the user as a response message.

[0261] Finally, if a user requests information on the optimal pricing plan or security plan, the optimal plan is calculated using the user's usage data and provided to the user as a response message. "Price plan" refers to the pricing structure or plan that a user can select to use the service. "Security plan" refers to a combination of security measures to protect the user's device and data.

[0262] As a concrete example, consider the case where a user sends a message saying, "What is the best security plan for me?" In this case, the "prompt sentence" would be as follows:

[0263] A user sends a message saying, "What security plan is best for me?"

[0264] The server analyzes this message, obtains the user's usage data, calculates the optimal security plan, and generates a response message based on this information.

[0265] As described above, the present invention enables a user to easily and quickly check operation methods, respond to detected abnormalities, and receive proposals for optimal security and fee plans.

[0266] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0267] Step 1:

[0268] A user sends an inquiry about operation methods or how to respond to an anomaly detection alert through a messaging application. In this case, for example, the input is a text message such as "Please tell me how to change the firewall settings." This message is sent to the server. The input is the user's message, and the output is the message sent to the server.

[0269] Step 2:

[0270] The server receives the query from the user. The server takes the message sent from the messaging application and analyzes the text message using a natural language processing model (software used: OpenAI GPT-3). Specifically, the server extracts meaning from the user's message and forms a query such as "How do I change my firewall settings?" The input is the user's message, and the output is the analyzed query.

[0271] Step 3:

[0272] The server retrieves appropriate manual information from a database (software used: MySQL, PostgreSQL) based on the analysis results. The server creates a database query using the analyzed query and searches for related manual information. Specifically, it retrieves manual information containing procedures and explanations on "how to change firewall settings" from the database. The input is the analyzed query, and the output is the retrieved manual information.

[0273] Step 4:

[0274] The server generates a response message based on the acquired information. Based on the acquired manual information, the server assembles the response message in a format that is easy for the user to understand. Specifically, it organizes the procedures by item and creates a message with supplementary explanations. It uses a generative AI model to generate a response message in natural language. The input is the acquired manual information, and the output is the generated response message.

[0275] Step 5:

[0276] The server sends the response message to the user. The server generates a response message and sends it to the user's terminal again through the message application. The user can then view the received message and take action based on its content. The input is the generated response message, and the output is the message sent to the user's terminal.

[0277] Step 6:

[0278] The user sends a screenshot of the anomaly detection alert through a messaging application. For example, the image is sent along with a text message such as "I don't understand the content of the screenshot below." The input is the screenshot image and message from the user, and the output is the data sent to the server.

[0279] Step 7:

[0280] The server receives the image from the user and analyzes it using image analysis technology. The server captures the screenshot image sent and identifies any abnormalities using image analysis software (software used: OpenCV, TensorFlow). Specifically, it extracts error messages and warnings from the screen and analyzes the related information. The input is the screenshot image, and the output is the analyzed abnormalities and their contents.

[0281] Step 8:

[0282] The server retrieves manual information relating to the cause of the anomaly and how to deal with it from a database based on the analysis results. The server creates a database query based on the analyzed anomaly location and searches for related manual information. Specifically, it retrieves manual information including procedures and solutions for the identified anomaly from the database. The input is the analyzed anomaly location, and the output is the retrieved manual information.

[0283] Step 9:

[0284] The server generates a response message based on the acquired information and sends it to the user. The server creates a response message that clearly explains the cause of the abnormality and how to deal with it. The input is the acquired manual information, and the output is the generated response message and its transmission to the user's terminal.

[0285] Example prompt sentence:

[0286] A user sent a message asking, "How do I change my firewall settings?"

[0287] A user has sent a screenshot. Please analyze the screenshot and provide appropriate advice.

[0288] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0289] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a message application, and also aims to recognize the user's emotions and provide a response that corresponds to those emotions.

[0290] 1. Checking the operation method

[0291] When a user uses a messaging application to make a query such as "How do I back up my new smartphone?", the server receives the message. The server uses a natural language processing model to analyze the user's message and extract queries related to "how to back up." The server then retrieves appropriate manual information from a database based on this query. Based on the retrieved manual information, the server generates a response message for the user and sends it to the user's device. Next, the emotion engine analyzes the user's emotion in the response message and adjusts the content of the response message according to the user's emotion. The user then backs up their smartphone based on the received information.

[0292] 2. Photo advice

[0293] When a user sends a screenshot of a part of the operation where they are having trouble via a messaging application, the server receives the image. The server analyzes the image using image analysis technology and identifies the relevant operating procedure. Based on the analysis results, the server retrieves relevant manual information from a database and generates a response message including specific operating procedures and reference images. The generated response message is then sent from the server to the user's device. The emotion engine analyzes the user's emotion in the response message and adjusts the content of the response message according to the user's emotion. The user then continues operating the device, referring to the received information.

[0294] 3. Providing information on pricing plans

[0295] When a user sends a query requesting information about pricing plans through a messaging application, the server receives the message. The server uses a natural language processing model to analyze the user message and extract queries related to "pricing plans." The server then retrieves the user's usage data from a database. The server uses a generative AI model to calculate the optimal pricing plan based on the retrieved usage data. Based on the calculated pricing plan details, the server generates a response message for the user and sends it to the user's device. The emotion engine analyzes the user's emotions in the response message and adjusts the content of the response message according to the user's emotions. The user compares and considers the proposed pricing plans based on the received information.

[0296] Specific examples

[0297] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives and analyzes the message and retrieves relevant manual information from the database. The server then generates a response message with concise and easy-to-understand backup instructions based on the retrieved information. The emotion engine analyzes this response message and, if it determines that the user is confused, adjusts it by adding more detailed explanations and supplementary information. The user can then use the received information to successfully back up their smartphone.

[0298] If a user has trouble with the smartphone's settings screen, they can send a screenshot. The server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant manual information from a database and generates a response message including detailed instructions and reference images. If the emotion engine analyzes the response message and determines that the user is frustrated, it can soften the tone of the response message and add words of encouragement. This allows the user to understand the specific operation method and solve the problem.

[0299] When providing pricing plan information, the user sends a message asking, "What is the best plan based on this month's usage?" The server obtains the user's usage data and calculates the optimal pricing plan based on that data. After compiling the obtained pricing plan details into a response message, the emotion engine analyzes the user's emotions in the response message and, if the user is dissatisfied or has questions about the plan, provides additional information or support options to alleviate those emotions. The user can review the provided plan details and select the plan that best suits them.

[0300] The processing flow will be explained below.

[0301] Check operation method

[0302] Step 1:

[0303] A user uses a messaging application to send a message saying, "How do I back up my new phone?"

[0304] Step 2:

[0305] A server receives a message from the user.

[0306] Step 3:

[0307] The server analyzes the user message using a natural language processing model and extracts the query "How to back up."

[0308] Step 4:

[0309] The server retrieves relevant manual information from a database based on the query.

[0310] Step 5:

[0311] The server generates a response message based on the acquired manual information.

[0312] Step 6:

[0313] The server generates a response message and sends it to the user's terminal.

[0314] Step 7:

[0315] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[0316] Step 8:

[0317] The smartphone is backed up based on the response message received by the user.

[0318] Photo-based advice

[0319] Step 1:

[0320] The user sends a screenshot of the part of the operation they are having trouble with via a messaging application.

[0321] Step 2:

[0322] A server receives the image from the user.

[0323] Step 3:

[0324] The server uses image analysis technology to analyze the received images and identify the problem areas.

[0325] Step 4:

[0326] The server retrieves related manual information from a database based on the analysis results.

[0327] Step 5:

[0328] The server generates a response message based on the acquired manual information and reference image.

[0329] Step 6:

[0330] The server generates a response message and sends it to the user's terminal.

[0331] Step 7:

[0332] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[0333] Step 8:

[0334] The user continues the operation while referring to the response message received.

[0335] Providing information on pricing plans

[0336] Step 1:

[0337] A user uses a messaging app to send a message saying, "What's the best rate plan?"

[0338] Step 2:

[0339] A server receives a message from the user.

[0340] Step 3:

[0341] The server analyzes the user message using a natural language processing model and extracts the query "price plan."

[0342] Step 4:

[0343] The server retrieves the user usage data from the database based on the query.

[0344] Step 5:

[0345] The server uses the generated AI model to calculate the optimal pricing plan based on the acquired usage data.

[0346] Step 6:

[0347] The server generates a response message based on the calculated details of the pricing plan.

[0348] Step 7:

[0349] The server generates a response message and sends it to the user's terminal.

[0350] Step 8:

[0351] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[0352] Step 9:

[0353] Based on the response message received by the user, the user compares and considers the proposed pricing plans.

[0354] Example 2

[0355] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0356] Today's users need to quickly and accurately obtain various information and operate digital devices without hesitation. However, if the information they need, such as confirmation of operation instructions, image-based support, or selection of pricing plans, is not provided promptly, it can cause stress and waste time. Furthermore, if appropriate responses are not provided based on the user's emotions, the user experience can be significantly impaired. Therefore, there is a need to develop a system that can recognize user emotions and provide appropriate responses accordingly.

[0357] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for sending an inquiry for a user to confirm an operation method through a communication application; means for receiving the inquiry from the user and analyzing it using a natural language processing engine; means for acquiring appropriate command information from a data storage means based on the analysis result; means for generating a response message based on the acquired command information; means for sending the response message to the user; and means for analyzing the user's emotion in the response message using an emotion recognition engine and adjusting the content. This allows the user to efficiently acquire information through the communication application and receive an appropriate response that matches their emotion.

[0358] A "user" is someone who uses this system to inquire about information, send images, or check pricing plans.

[0359] A "communication application" is software that allows users to communicate with a server in two-way communication.

[0360] A "server" is a central processing unit that receives inquiries and images from users and analyzes and responds.

[0361] A "natural language processing engine" is a software component that analyzes and understands the meaning of a user's text messages.

[0362] "Command information" refers to instructions and explanations regarding operation methods and procedures, and information desired by the user.

[0363] "Data storage means" refers to a database or storage system that stores information such as manual information and usage data.

[0364] A "response message" is a message that is generated based on the analysis results and command information and sent to the user.

[0365] An "emotion recognition engine" is a software component that analyzes the user's emotions in a response message and adjusts the response content accordingly.

[0366] "Image analysis technology" is a technology for analyzing images sent by users and identifying the information and operation locations contained therein.

[0367] A "generative AI model" is an artificial intelligence model that calculates the optimal pricing plan based on user usage data.

[0368] "Usage Data" refers to the usage history, patterns, and consumption data of the services you use.

[0369] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a communication application. This system aims to recognize the user's emotions and provide a response that corresponds to those emotions. Specific embodiments of this invention are described below.

[0370] First, the user communicates with the server using a communication application, which provides an interface for sending user queries and images to the server in text or image format.

[0371] The server has an API for receiving inquiries and images from users. The received messages and images are analyzed using a natural language processing engine (e.g., Google NLU) and image analysis technology (e.g., Google Vision API). The natural language processing engine analyzes the user's text message and understands its meaning. Meanwhile, the image analysis technology analyzes the image sent by the user and identifies specific operation points and problems.

[0372] Based on the analysis results, the server retrieves appropriate instruction information from a data storage means (e.g., SQL database), including information such as operating procedures, reference images, and pricing plans. Based on the retrieved information, the server generates a response message and sends it to the user.

[0373] Furthermore, the server is equipped with an emotion recognition engine (e.g., Microsoft® Text Analytics API) that analyzes the user's emotions in the generated response message. Depending on the analysis results, the content and tone of the response message are adjusted. This function allows appropriate support to be provided according to the user's situation and emotions.

[0374] As a concrete example, consider a case where a user sends a query through a communication application asking, "How do I back up my new smartphone?" In this case, the server receives and analyzes the message and retrieves relevant instruction information from a database. The server then uses the retrieved information to generate a response message containing concise and easy-to-understand backup instructions. This response message is then analyzed by an emotion recognition engine, and detailed explanations and supplementary information are added as needed. Finally, the user can successfully back up their smartphone using the received information.

[0375] Furthermore, if a user encounters a problem with the smartphone's settings screen, they can send a screenshot via a communication application. In this case, the server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant instruction information from a database and generates a response message containing detailed instructions and reference images. This response message is also analyzed by an emotion recognition engine, and the tone and content are adjusted according to the user's emotions. This allows the user to understand the specific operation method and solve the problem.

[0376] In providing pricing plan information, consider the case where a user asks, "What is the best plan based on my usage this month?" In this case, the server retrieves the user's usage data from a database and calculates the optimal pricing plan using a generative AI model (e.g., OpenAI GPT-3). The obtained pricing plan details are compiled into a response message, analyzed using an emotion recognition engine, and additional information and support options are provided as needed. This allows the user to review the plans offered and select the one that best suits them.

[0377] This system is expected to improve the user experience by allowing users to efficiently obtain information through communication applications and receive appropriate responses that match their emotions.

[0378] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0379] Step 1:

[0380] A user sends an inquiry through a communication application requesting confirmation of an operation method, advice with an image, or information on a rate plan.

[0381] What happens: A user enters and attaches a text message (e.g., "How do I back up my new phone?" or a screenshot) into the input field in a messaging app and hits the send button.

[0382] Input: A query sent by the user in the form of text or an image.

[0383] Output: The query data received by the server from the user.

[0384] Step 2:

[0385] The server receives the inquiry data (text message or image) received through the communication application.

[0386] Specific operation: The server's API receives the query data and adds it to an internal queue for analysis.

[0387] Input: Query data sent from the communication application to the server.

[0388] Output: Query data added to the queue for analysis inside the server.

[0389] Step 3:

[0390] The server uses a natural language processing engine (e.g., Google NLU) to analyze the text message and understand the user's intent.

[0391] What it does: A natural language processing engine analyzes text messages to identify keywords and intent.

[0392] Input: A text message from the user.

[0393] Output: Parsed keywords and user intent.

[0394] Step 4:

[0395] The server uses image analysis technology (e.g., Google Vision API) to analyze the image sent by the user and identify specific operation points and problems.

[0396] What it does: Image analysis technology analyzes image data and identifies the elements it contains.

[0397] Input: Image data sent by the user.

[0398] Output: Information on the analyzed elements in the image and the specific operation locations.

[0399] Step 5:

[0400] The server retrieves the appropriate instruction information (operational procedures, reference images, pricing plans, etc.) from a data storage means (e.g., SQL database).

[0401] Specific operation: An SQL query is executed and the necessary instruction information is retrieved from the database.

[0402] Input: Analysis results from a natural language processing engine or image analysis technology.

[0403] Output: Instruction information retrieved from the database.

[0404] Step 6:

[0405] The server generates a response message based on the acquired information and sends it to the user.

[0406] Specific operation: The text generation engine creates a response message and delivers it to the user's device via the message sending function.

[0407] Input: Instruction information retrieved from the database.

[0408] Output: The generated response message.

[0409] Step 7:

[0410] The server's emotion recognition engine (e.g., Microsoft Text Analytics API) analyzes the user's emotion in the response message and adjusts the content of the response message as necessary.

[0411] What it does: Emotion-sensing algorithms analyze the user's tone and emotions and adjust the content and tone of the response message accordingly.

[0412] Input: The generated response message.

[0413] Output: The adjusted response message.

[0414] Step 8:

[0415] The user takes appropriate action based on the information received.

[0416] Specific actions: Read the information provided in the messaging app and follow the instructions to change your smartphone settings or check your plan.

[0417] Input: The response message received from the server.

[0418] Output: User action (performing a procedure, resolving a problem, checking pricing plans, etc.).

[0419] (Application example 2)

[0420] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0421] In conventional inquiry systems using messaging applications, it was difficult for users to receive appropriate responses when requesting confirmation of operation procedures or problem resolution. Furthermore, responses did not take into account the user's emotions, resulting in insufficient support for users who felt particularly confused or dissatisfied. This resulted in a poor user experience and a decrease in system usage. Furthermore, in content distribution services, real-time support for the content being viewed was insufficient, making it difficult to improve user satisfaction.

[0422] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0423] In this invention, the server includes: means for sending a query to a user to confirm operation methods through a message application; means for receiving the query from the user and analyzing the query using a natural language processing model; means for retrieving appropriate manual information from a database based on the analysis result; means for generating a response message based on the retrieved manual information; means for adjusting the content of the response message using an emotion engine that analyzes the user's emotions; means for sending the response message to the user; and means for responding in real time to inquiries about content being viewed as part of a content distribution service. This enables users to receive prompt and appropriate support and further allows them to receive flexible responses based on their emotions, which is expected to significantly improve the user experience and increase the frequency of system use.

[0424] A "messaging application" is software that allows users to send and receive messages to and from a server.

[0425] A "server" is a computer system that receives user queries and images, retrieves information from a database, and generates and sends response messages.

[0426] A "natural language processing model" is a machine learning algorithm that analyzes text data and understands its meaning and intent.

[0427] "Manual information" refers to documents or data that contain detailed explanations of operation and setting methods.

[0428] A "response message" is text data that is generated by the server in response to a user's inquiry.

[0429] An "emotion engine" is a technology that analyzes emotions from a user's text message and adjusts the content of the response message accordingly.

[0430] "Image analysis technology" is a technology for analyzing received image data and extracting important information.

[0431] "Usage data" is data about a user's activity and usage history.

[0432] A "generative AI model" is an artificial intelligence model that makes predictions and recommendations based on collected data.

[0433] A "content distribution service" is a platform and service for providing users with digital content such as videos, music, and e-books.

[0434] This invention is a system that allows users to quickly and appropriately receive operation instructions, image advice, and information on pricing plans through a messaging application. This system also has the ability to analyze the user's emotions and provide responses according to them.

[0435] The system is configured as follows:

[0436] 1. User Interface

[0437] Users use the messaging application to ask for instructions, send images, or request information about pricing plans.

[0438] 2. Server Functions

[0439] The server responds to user inquiries using various modules, specifically with the following functions:

[0440] Natural Language Processing Model

[0441] The server analyzes the user's text query using a natural language processing model, which is used to understand the user's intent and generate an appropriate query.

[0442] Database Access

[0443] Based on the analysis results, the server retrieves relevant manual information and usage data from the database.

[0444] Response message generation

[0445] The server generates a response message for the user based on the acquired information, and also uses an emotion engine to take the user's emotions into account.

[0446] Emotion Engine

[0447] The server uses an emotion engine to analyze the user's emotions and adjusts the content of the response message, adding more detailed explanations or encouraging words if the user is confused.

[0448] 3. Image analysis function

[0449] When a user submits an image, the server uses image analysis technology to analyze the image and retrieve relevant information.

[0450] 4. Optimization function

[0451] The server uses a generative AI model to recommend the optimal pricing plan based on the user's usage data.

[0452] Hardware and software used

[0453] Server: High-performance computer

[0454] Natural Language Processing models: Machine learning algorithms such as BERT

[0455] Database: SQL or NoSQL database

[0456] Sentiment Engine: Dedicated sentiment analysis software

[0457] Image analysis technology: OpenCV, etc.

[0458] Generative AI models: GPT-3, etc.

[0459] Specific examples

[0460] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives the message, analyzes it using a natural language processing model, and retrieves the query "how to back up" from the database. After constructing a response message, it analyzes the user's emotions using an emotion engine, adjusts the content as necessary, and sends the response message.

[0461] Prompt Sentence Examples

[0462] When a user sends a message asking, "What is the best plan based on this month's usage?", the server analyzes it using a natural language processing model, retrieves the user's usage data from a database, analyzes it using a generative AI model, calculates the optimal pricing plan, and sends a response message adjusted by an emotion engine to the user.

[0463] This allows users to receive prompt and appropriate support, improving the user experience.

[0464] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0465] Step 1:

[0466] A user sends an inquiry through a messaging application, asking for confirmation of operation instructions, sending an image, or requesting information on a pricing plan. The input is a text message or image data from the user, which is sent to the server. The output is the data received by the server.

[0467] Step 2:

[0468] The server receives the user's inquiry and image and analyzes the text message using a natural language processing model. The input is the user's inquiry message, and based on this input, a natural language processing model (e.g., BERT) is used to analyze and extract meaning and intent. Specifically, the message is input into the model, and the intent and related keywords are obtained. The output is the analysis result (intent or query).

[0469] Step 3:

[0470] Based on the analysis results, the server retrieves related manual information and user usage data from the database. The input is the analysis result from step 2, and based on this analysis result, it queries the appropriate information from an SQL or NoSQL database. Specifically, it executes a database query and retrieves the relevant data. The output is the retrieved manual information and usage data.

[0471] Step 4:

[0472] The server uses image analysis technology to analyze the image received from the user. The input is the image data sent by the user, which is analyzed using image analysis software such as OpenCV. Specifically, the image is input and information related to the operation method is extracted. The output is the analysis result.

[0473] Step 5:

[0474] The server uses a generative AI model to calculate the optimal pricing plan based on the user's usage data. The input is the user's usage data, which is then input into the model to generate the optimal plan. Specifically, the generative AI model (e.g., GPT-3) makes predictions based on the usage data and calculates the pricing plan. The output is details of the calculated pricing plan.

[0475] Step 6:

[0476] The server generates a response message based on the acquired manual information and analysis results. The input is the manual information and analysis results, and a response message is generated for the user based on this. Specifically, the required information is compiled as text. The output is the generated response message.

[0477] Step 7:

[0478] The server adjusts the content of the response message using an emotion engine that analyzes the user's emotion. The input is the user's message and the generated response message, which are analyzed by the emotion engine. Specifically, the emotion in the message is analyzed and the tone and content of the response message are adjusted accordingly. The output is the adjusted response message.

[0479] Step 8:

[0480] The server sends the adjusted response message to the user, the input is the adjusted response message, and the server performs an operation to send it to the user's terminal, and the output is the response message displayed on the user's terminal.

[0481] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0482] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0483] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0484] [Second embodiment]

[0485] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0486] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0487] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0488] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0489] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0490] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0491] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0492] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0493] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0494] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0495] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0496] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0497] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, advice using images, and information on rate plans through a message application.

[0498] 1. Checking the operation method

[0499] When a user uses a messaging application to make a query such as "How do I back up my new smartphone?", the server receives the message. The server uses a natural language processing model to analyze the user's message and extract queries related to "how to back up." The server retrieves appropriate manual information from a database based on this query. Based on the retrieved manual information, the server generates a response message for the user and sends it to the user's device. The user then backs up their smartphone based on the received information.

[0500] 2. Photo advice

[0501] When a user sends a screenshot of the part of the operation they are having trouble with via a messaging application, the server receives the image. The server then analyzes the image using image analysis technology to identify the relevant operating procedure. Based on the analysis results, the server retrieves relevant manual information from a database and generates a response message including specific operating procedures and reference images. The generated response message is then sent from the server to the user's device. The user then refers to the received information and continues operating the device.

[0502] 3. Providing information on pricing plans

[0503] When a user sends a query requesting information about pricing plans through a messaging application, the server receives the message. The server uses a natural language processing model to analyze the user message and extract queries related to "pricing plans." The server then retrieves the user's usage data from a database. The server uses a generative AI model to calculate the optimal pricing plan based on the retrieved usage data. Based on the calculated pricing plan details, the server generates a response message for the user and sends it to the user's device. The user compares and considers the proposed pricing plans based on the received information.

[0504] Specific examples

[0505] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives and analyzes the message, then retrieves the relevant manual information from the database. Based on the retrieved information, the server generates and sends a response message containing concise and easy-to-understand backup instructions. The user can then use the received information to successfully back up their smartphone.

[0506] Furthermore, if a user has trouble with the smartphone's settings screen, they can send a screenshot. The server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant manual information from a database and sends the user a response message containing detailed instructions and reference images. This allows the user to understand the specific operation method and solve the problem.

[0507] When providing information on pricing plans, if the user sends a message saying "Tell me the best pricing plan," the server will analyze the user's usage data, calculate the best pricing plan, and propose it to them. The user can then check the details of the plans provided and select the one that best suits them.

[0508] The processing flow will be explained below.

[0509] Check operation method

[0510] Step 1:

[0511] A user uses a messaging application to send a message saying, "How do I back up my new phone?"

[0512] Step 2:

[0513] A server receives a message from the user.

[0514] Step 3:

[0515] The server analyzes the user message using a natural language processing model and extracts the query "How to back up."

[0516] Step 4:

[0517] The server retrieves relevant manual information from a database based on the query.

[0518] Step 5:

[0519] The server generates a response message based on the acquired manual information.

[0520] Step 6:

[0521] The server generates a response message and sends it to the user's terminal.

[0522] Step 7:

[0523] The smartphone is backed up based on the response message received by the user.

[0524] Photo-based advice

[0525] Step 1:

[0526] The user sends a screenshot of the part of the operation they are having trouble with via a messaging application.

[0527] Step 2:

[0528] A server receives the image from the user.

[0529] Step 3:

[0530] The server uses image analysis technology to analyze the received images and identify the problem areas.

[0531] Step 4:

[0532] The server retrieves related manual information from a database based on the analysis results.

[0533] Step 5:

[0534] The server generates a response message based on the acquired manual information and reference image.

[0535] Step 6:

[0536] The server generates a response message and sends it to the user's terminal.

[0537] Step 7:

[0538] The user continues the operation while referring to the response message received.

[0539] Providing information on pricing plans

[0540] Step 1:

[0541] A user uses a messaging app to send a message saying, "What's the best rate plan?"

[0542] Step 2:

[0543] A server receives a message from the user.

[0544] Step 3:

[0545] The server analyzes the user message using a natural language processing model and extracts the query "price plan."

[0546] Step 4:

[0547] The server retrieves the user usage data from the database based on the query.

[0548] Step 5:

[0549] The server uses the generated AI model to calculate the optimal pricing plan based on the acquired usage data.

[0550] Step 6:

[0551] The server generates a response message based on the calculated details of the pricing plan.

[0552] Step 7:

[0553] The server generates a response message and sends it to the user's terminal.

[0554] Step 8:

[0555] Based on the response message received by the user, the user compares and considers the proposed pricing plans.

[0556] Example 1

[0557] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0558] The challenge is to effectively utilize messaging applications, a modern means of communication, to quickly and appropriately confirm operation procedures, resolve problems, and recommend optimal pricing plans. Conventional systems require users to search multiple sources of information and obtain the necessary information on their own, which can be a cumbersome process. In addition, selecting a pricing plan requires users to perform complex calculations, making it difficult to choose the optimal plan.

[0559] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0560] In this invention, the server includes: means for sending an inquiry for the user to confirm an operation method through a message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain appropriate information from a database based on the analysis result; means for the server to generate a response message based on the obtained information; means for the server to send the response message to the user; means for the user to perform an operation based on the response message displayed on the terminal; means for the user to send an image through the message application; means for the server to receive an image from the user and analyze it using image analysis technology; means for the server to obtain related information from a database based on the analysis result; means for the server to generate a response message based on the obtained information and a reference image; means for the server to send the response message to the user; means for the user to perform an operation based on the response message displayed on the terminal; means for the user to send an inquiry requesting information on a rate plan through the message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain usage data of the user from a database; and means for the server to calculate an optimal rate plan based on the obtained usage data. The system includes a means for the server to generate a response message based on the calculated details of the pricing plan, a means for the server to send the response message to the user, and a means for the user to consider plans based on the response message displayed on the terminal, thereby enabling the user to quickly and appropriately obtain the necessary information, efficiently solve the problems they are facing, and select the most suitable pricing plan without going through complicated processes.

[0561] "User" refers to an individual or corporation that uses the system.

[0562] "Messaging application" refers to software or applications for sending and receiving information such as text messages and images.

[0563] An "inquiry" refers to a question or request made by a user to the system.

[0564] "Server" refers to a computer system that receives and processes requests from users.

[0565] A "natural language processing model" refers to algorithms and tools for analyzing human language.

[0566] "Analysis" refers to the act of understanding and processing received data using natural language processing models and image analysis techniques.

[0567] A "database" refers to a system for organizing and storing data and efficiently retrieving necessary information.

[0568] "Manual information" refers to documents and data that explain operation methods and procedures.

[0569] A "response message" refers to a message that a server generates and sends in response to a user's query.

[0570] "Image analysis technology" refers to technology for analyzing image data and understanding its contents.

[0571] A "reference image" refers to a reference image provided for the purpose of explaining or instructing a user.

[0572] "Usage status data" refers to data related to a user's service usage status and history.

[0573] "Price plan" refers to various plans that determine the fee structure for the services that a user uses.

[0574] The "optimal pricing plan" refers to the pricing plan that is most profitable for the user based on the user's usage data.

[0575] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a message application. This system performs various data processing and data calculations using the following hardware and software.

[0576] 1. Hardware and Software Used

[0577] Server: A high-performance server computer is required. In this system, it stores the database and AI models and performs calculations.

[0578] Device: A mobile device operated by a user, such as a smartphone or tablet, that must have a messaging application installed.

[0579] Message application: Software for sending and receiving messages between users and servers.

[0580] Natural language processing model: A model for analyzing messages from users, such as BERT or OpenAI's GPT-3.

[0581] Database: This stores manual information and user usage data. For example, a database management system such as MySQL is used.

[0582] Image analysis technology: Image analysis tools such as Google Cloud Vision API are used.

[0583] Generative AI model: For example, OpenAI's GPT-3 is used as an AI model to calculate the optimal pricing plan based on usage data.

[0584] 2. Implementing the Invention

[0585] Check operation method

[0586] A user uses a messaging application to send a message saying, "Tell me how to back up my new smartphone." The server receives this message and analyzes it using a natural language processing model (e.g., BERT) to extract queries related to "how to back up." Based on the extracted query, the server retrieves the appropriate manual information from a MySQL database, generates a response message, and sends it to the user's device. The user then backs up their smartphone based on the received information.

[0587] Photo-based advice

[0588] The user sends a screenshot of the part of the operation where they are having trouble to the server via a messaging application. The server receives the image and analyzes it using the Google Cloud Vision API. Based on the analysis results, the server retrieves relevant manual information from a MySQL database, generates a response message containing specific operating procedures and reference images, and sends it to the user's device. The user can continue operating the device by referring to the received information.

[0589] Providing information on pricing plans

[0590] The user sends a message via a messaging application saying, "What is the best pricing plan?" The server receives this message, analyzes it using a natural language processing model (for example, OpenAI's GPT-3), and extracts queries related to "pricing plans." The server retrieves the user's usage data from a MySQL database and calculates the best pricing plan using GPT-3 based on the retrieved data. A response message is generated based on the details of the calculated pricing plan and sent to the user's device. The user then checks the details of the provided plans and selects the most suitable plan.

[0591] Specific examples

[0592] Check operation method

[0593] Example user question: "How do I back up my new phone?"

[0594] Example prompt: "Generate a message to return when a user is asked how to back up their new phone."

[0595] Photo-based advice

[0596] Example user question: "Send a screenshot of your smartphone's settings screen"

[0597] Example prompt: "Generate a response message when a user sends you a screenshot of their settings screen."

[0598] Providing information on pricing plans

[0599] Example user question: "What is the best pricing plan?"

[0600] Example prompt: "Generate a response message when a user asks what the best pricing plan is."

[0601] This allows users to quickly and appropriately obtain the necessary information, efficiently resolve problems, and select the most suitable rate plan without having to go through complicated processes.

[0602] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0603] Check operation method

[0604] Step 1:

[0605] A query is sent to the user to confirm the operation method.

[0606] Type: A user uses the Messages app to type the text message "How do I back up my new phone?"

[0607] Output: The message is sent to the server.

[0608] Step 2:

[0609] A server receives the user query and analyzes it using a natural language processing model.

[0610] Input: The server receives a message sent by the user.

[0611] Data processing: The server analyzes the message using a natural language processing model (e.g., BERT) and extracts keywords related to "backup methods."

[0612] Output: Queries related to "Backup Method" are generated.

[0613] Step 3:

[0614] The server retrieves the appropriate information from the database.

[0615] Input: Queries related to "how to back up".

[0616] Data processing: The server searches and retrieves the relevant manual information from the MySQL database.

[0617] Output: The retrieved manual information.

[0618] Step 4:

[0619] The server generates a response message based on the acquired information.

[0620] Input: The retrieved manual information.

[0621] Data processing: Based on the obtained manual information, the server generates a response message in a format that is easy for the user to understand.

[0622] Output: A response message is generated.

[0623] Step 5:

[0624] The server sends the response message to the user.

[0625] Input: The response message.

[0626] Output: A reply message is sent to the user's device via the Messages application.

[0627] Step 6:

[0628] The user performs an operation based on the response message displayed on the terminal.

[0629] Input: The response message.

[0630] What happens: The user follows the instructions in the response message to start backing up their smartphone.

[0631] Output: Your phone is backed up.

[0632] Photo-based advice

[0633] Step 1:

[0634] A user sends an image through a messaging application.

[0635] Input: The user attaches a screenshot of the problem and sends it via messaging.

[0636] Output: The screenshot is sent to the server.

[0637] Step 2:

[0638] A server receives the image from the user and analyzes it using image analysis techniques.

[0639] Input: A screenshot submitted by the user.

[0640] Data processing: The server uses the Google Cloud Vision API to analyze the screenshots and identify specific problem areas and operational steps.

[0641] Output: Analysis results.

[0642] Step 3:

[0643] The server retrieves the relevant information from the database.

[0644] Input: Analysis results.

[0645] Data processing: The server searches and retrieves the relevant manual information and reference images from the MySQL database.

[0646] Output: Retrieved information and reference images.

[0647] Step 4:

[0648] The server generates a response message based on the acquired information and the reference image.

[0649] Input: Retrieved information and reference image.

[0650] Data processing: Based on this information, the server generates a response message in a format that is easy for the user to understand.

[0651] Output: A response message is generated.

[0652] Step 5:

[0653] The server sends the response message to the user.

[0654] Input: The response message.

[0655] Output: A reply message is sent to the user's device via the Messages application.

[0656] Step 6:

[0657] The user performs the operation by referring to the response message displayed on the terminal.

[0658] Input: The response message.

[0659] Specific actions: The user follows the response message and uses the displayed steps and reference images to solve the problem.

[0660] Output: Problem solved.

[0661] Providing information on pricing plans

[0662] Step 1:

[0663] A user sends a request for pricing plan information through a messaging application.

[0664] Input: A user sends a message asking, "What is the best pricing plan?"

[0665] Output: The query is sent to the server.

[0666] Step 2:

[0667] A server receives the user query and analyzes it using a natural language processing model.

[0668] Input: A query message from the user.

[0669] Data processing: The server analyzes the message using a natural language processing model such as OpenAI's GPT-3 to extract queries related to "price plans."

[0670] Output: Queries related to "price plan" are generated.

[0671] Step 3:

[0672] The server retrieves user usage data from the database.

[0673] Input: Queries related to "price plans".

[0674] Data processing: The server retrieves user usage data from the MySQL database.

[0675] Output: The captured usage data.

[0676] Step 4:

[0677] The optimal pricing plan is calculated based on the usage data acquired by the server.

[0678] Input: Captured usage data.

[0679] Data processing: The server uses a generative AI model (e.g., OpenAI's GPT-3) to calculate the optimal pricing plan.

[0680] Output: Best pricing plan.

[0681] Step 5:

[0682] The server generates a response message based on the calculated details of the rate plan.

[0683] Input: Your best rate plan information.

[0684] Data processing: The server generates a response message in a format that is easy for the user to understand.

[0685] Output: A response message is generated.

[0686] Step 6:

[0687] The server sends the response message to the user.

[0688] Input: The response message.

[0689] Output: A reply message is sent to the user's device via the Messages application.

[0690] Step 7:

[0691] The user considers a plan based on the response message displayed on the terminal.

[0692] Input: The response message.

[0693] Specific behavior: The user reviews the details of the offered pricing plans and selects the most suitable plan.

[0694] Output: The user selects the best pricing plan.

[0695] (Application example 1)

[0696] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0697] Modern users face numerous configuration and security issues related to digital devices and security measures. While a support system is needed to quickly and appropriately resolve these issues, conventional methods often require complex operation procedures and responses to anomaly detection alerts, making them difficult for users to understand. While it is also important to propose optimal security and pricing plans, there is a lack of systems that can effectively accomplish this. The objective of this invention is to solve these problems by providing a system that allows users to quickly and appropriately confirm operation procedures, respond to anomaly detection, and receive recommendations for optimal plans.

[0698] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0699] In this invention, the server includes: means for a user to send an inquiry about operation instructions or how to respond to an anomaly detection alert via a message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain appropriate manual information or response information from a database based on the analysis results; means for the server to generate a response message based on the obtained information; means for the server to send the response message to the user; means for the server to receive an anomaly detection alert image from the user and analyze it using image analysis technology; and means for the server to obtain manual information about the cause of the anomaly and how to deal with it based on the analysis results. This allows the user to easily and quickly confirm operation instructions, respond to an anomaly detection, and receive suggestions for optimal security and pricing plans.

[0700] "User" refers to an individual or company that uses the system or service.

[0701] A "messaging application" is software for sending and receiving text messages and images.

[0702] "Method of operation" refers to the steps or techniques for using a digital device or software.

[0703] An "anomaly detection alert" is a warning message that notifies the user when the system detects an anomaly.

[0704] A "query" is a message sent by a user to the system requesting information or instructions.

[0705] A "server" is a computer system that receives and analyzes messages from users and provides the necessary information.

[0706] A "natural language processing model" refers to algorithms and technologies used to analyze and understand the meaning of text messages entered by users.

[0707] "Analysis results" are the results of analyzing data obtained using natural language processing models and image analysis technology.

[0708] "Manual information" refers to materials that include specific information required by users, such as operation methods and procedures for responding to abnormality detection.

[0709] A "response message" is a message containing a response generated by a server in response to a user's inquiry.

[0710] "Image analysis technology" is a technology for processing transmitted images and analyzing their contents.

[0711] A "reference image" is an image used to explain operating procedures and how to respond to abnormalities.

[0712] "Price Plan" refers to the pricing structure or plan that a User can select to use the Service.

[0713] "Usage status data" refers to data that indicates a user's service usage history and trends.

[0714] "Security Plan" refers to a combination of security measures designed to protect a user's devices and data.

[0715] A "generative AI model" is an artificial intelligence model that generates new information and responses based on data.

[0716] A "prompt" refers to an instruction or question that is input to a model.

[0717] The present invention provides a system that allows a user to quickly and appropriately check operation instructions and respond to an anomaly detection alert through a message application. Specific embodiments of this system will be described below.

[0718] First, when a user wants to check how to operate a device or how to respond to an anomaly detection alert, they send an inquiry from their device via a messaging application. Here, "user" refers to individuals or companies that use the system or service. "Messaging application" is software for sending and receiving text messages and images. "Operation method" refers to the steps and methods for using digital devices and software.

[0719] The server receives the user's inquiry and analyzes it using a natural language processing model (software used: OpenAI GPT-3). Based on the analysis results, the server retrieves "manual information" from a database (software used: MySQL, PostgreSQL). A "natural language processing model" refers to the algorithms and technologies used to analyze and understand the meaning of text messages entered by users. The "database" contains detailed manual information on operating procedures and how to respond to abnormalities.

[0720] Next, the server generates a "response message" based on the retrieved manual information and sends it to the user's terminal. A "response message" is a message that the server generates containing a response to a user's inquiry. For example, if a user sends a message saying "Please tell me how to change the firewall settings," the server generates a response message based on the relevant manual information and sends it to the user.

[0721] We will also explain the case where a user sends an anomaly detection alert with a screenshot attached. The image sent from the user's device is analyzed on the server using "image analysis technology" (software used: OpenCV, TensorFlow). Based on the analysis results, the server retrieves manual information on the "cause of the anomaly" and its "measures" from the database and sends it to the user as a response message.

[0722] Finally, if a user requests information on the optimal pricing plan or security plan, the optimal plan is calculated using the user's usage data and provided to the user as a response message. "Price plan" refers to the pricing structure or plan that a user can select to use the service. "Security plan" refers to a combination of security measures to protect the user's device and data.

[0723] As a concrete example, consider the case where a user sends a message saying, "What is the best security plan for me?" In this case, the "prompt sentence" would be as follows:

[0724] A user sends a message saying, "What security plan is best for me?"

[0725] The server analyzes this message, obtains the user's usage data, calculates the optimal security plan, and generates a response message based on this information.

[0726] As described above, the present invention enables a user to easily and quickly check operation methods, respond to detected abnormalities, and receive proposals for optimal security and fee plans.

[0727] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0728] Step 1:

[0729] A user sends an inquiry about operation methods or how to respond to an anomaly detection alert through a messaging application. In this case, for example, the input is a text message such as "Please tell me how to change the firewall settings." This message is sent to the server. The input is the user's message, and the output is the message sent to the server.

[0730] Step 2:

[0731] The server receives the query from the user. The server takes the message sent from the messaging application and analyzes the text message using a natural language processing model (software used: OpenAI GPT-3). Specifically, the server extracts meaning from the user's message and forms a query such as "How do I change my firewall settings?" The input is the user's message, and the output is the analyzed query.

[0732] Step 3:

[0733] The server retrieves appropriate manual information from a database (software used: MySQL, PostgreSQL) based on the analysis results. The server creates a database query using the analyzed query and searches for related manual information. Specifically, it retrieves manual information containing procedures and explanations on "how to change firewall settings" from the database. The input is the analyzed query, and the output is the retrieved manual information.

[0734] Step 4:

[0735] The server generates a response message based on the acquired information. Based on the acquired manual information, the server assembles the response message in a format that is easy for the user to understand. Specifically, it organizes the procedures by item and creates a message with supplementary explanations. It uses a generative AI model to generate a response message in natural language. The input is the acquired manual information, and the output is the generated response message.

[0736] Step 5:

[0737] The server sends the response message to the user. The server generates a response message and sends it to the user's terminal again through the message application. The user can then view the received message and take action based on its content. The input is the generated response message, and the output is the message sent to the user's terminal.

[0738] Step 6:

[0739] The user sends a screenshot of the anomaly detection alert through a messaging application. For example, the image is sent along with a text message such as "I don't understand the content of the screenshot below." The input is the screenshot image and message from the user, and the output is the data sent to the server.

[0740] Step 7:

[0741] The server receives the image from the user and analyzes it using image analysis technology. The server captures the screenshot image sent and identifies any abnormalities using image analysis software (software used: OpenCV, TensorFlow). Specifically, it extracts error messages and warnings from the screen and analyzes the related information. The input is the screenshot image, and the output is the analyzed abnormalities and their contents.

[0742] Step 8:

[0743] The server retrieves manual information relating to the cause of the anomaly and how to deal with it from a database based on the analysis results. The server creates a database query based on the analyzed anomaly location and searches for related manual information. Specifically, it retrieves manual information including procedures and solutions for the identified anomaly from the database. The input is the analyzed anomaly location, and the output is the retrieved manual information.

[0744] Step 9:

[0745] The server generates a response message based on the acquired information and sends it to the user. The server creates a response message that clearly explains the cause of the abnormality and how to deal with it. The input is the acquired manual information, and the output is the generated response message and its transmission to the user's terminal.

[0746] Example prompt sentence:

[0747] A user sent a message asking, "How do I change my firewall settings?"

[0748] A user has sent a screenshot. Please analyze the screenshot and provide appropriate advice.

[0749] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0750] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a message application, and also aims to recognize the user's emotions and provide a response that corresponds to those emotions.

[0751] 1. Checking the operation method

[0752] When a user uses a messaging application to make a query such as "How do I back up my new smartphone?", the server receives the message. The server uses a natural language processing model to analyze the user's message and extract queries related to "how to back up." The server then retrieves appropriate manual information from a database based on this query. Based on the retrieved manual information, the server generates a response message for the user and sends it to the user's device. Next, the emotion engine analyzes the user's emotion in the response message and adjusts the content of the response message according to the user's emotion. The user then backs up their smartphone based on the received information.

[0753] 2. Photo advice

[0754] When a user sends a screenshot of a part of the operation where they are having trouble via a messaging application, the server receives the image. The server analyzes the image using image analysis technology and identifies the relevant operating procedure. Based on the analysis results, the server retrieves relevant manual information from a database and generates a response message including specific operating procedures and reference images. The generated response message is then sent from the server to the user's device. The emotion engine analyzes the user's emotion in the response message and adjusts the content of the response message according to the user's emotion. The user then continues operating the device, referring to the received information.

[0755] 3. Providing information on pricing plans

[0756] When a user sends a query requesting information about pricing plans through a messaging application, the server receives the message. The server uses a natural language processing model to analyze the user message and extract queries related to "pricing plans." The server then retrieves the user's usage data from a database. The server uses a generative AI model to calculate the optimal pricing plan based on the retrieved usage data. Based on the calculated pricing plan details, the server generates a response message for the user and sends it to the user's device. The emotion engine analyzes the user's emotions in the response message and adjusts the content of the response message according to the user's emotions. The user compares and considers the proposed pricing plans based on the received information.

[0757] Specific examples

[0758] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives and analyzes the message and retrieves relevant manual information from the database. The server then generates a response message with concise and easy-to-understand backup instructions based on the retrieved information. The emotion engine analyzes this response message and, if it determines that the user is confused, adjusts it by adding more detailed explanations and supplementary information. The user can then use the received information to successfully back up their smartphone.

[0759] If a user has trouble with the smartphone's settings screen, they can send a screenshot. The server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant manual information from a database and generates a response message including detailed instructions and reference images. If the emotion engine analyzes the response message and determines that the user is frustrated, it can soften the tone of the response message and add words of encouragement. This allows the user to understand the specific operation method and solve the problem.

[0760] When providing pricing plan information, the user sends a message asking, "What is the best plan based on this month's usage?" The server obtains the user's usage data and calculates the optimal pricing plan based on that data. After compiling the obtained pricing plan details into a response message, the emotion engine analyzes the user's emotions in the response message and, if the user is dissatisfied or has questions about the plan, provides additional information or support options to alleviate those emotions. The user can review the provided plan details and select the plan that best suits them.

[0761] The processing flow will be explained below.

[0762] Check operation method

[0763] Step 1:

[0764] A user uses a messaging application to send a message saying, "How do I back up my new phone?"

[0765] Step 2:

[0766] A server receives a message from the user.

[0767] Step 3:

[0768] The server analyzes the user message using a natural language processing model and extracts the query "How to back up."

[0769] Step 4:

[0770] The server retrieves relevant manual information from a database based on the query.

[0771] Step 5:

[0772] The server generates a response message based on the acquired manual information.

[0773] Step 6:

[0774] The server generates a response message and sends it to the user's terminal.

[0775] Step 7:

[0776] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[0777] Step 8:

[0778] The smartphone is backed up based on the response message received by the user.

[0779] Photo-based advice

[0780] Step 1:

[0781] The user sends a screenshot of the part of the operation they are having trouble with via a messaging application.

[0782] Step 2:

[0783] A server receives the image from the user.

[0784] Step 3:

[0785] The server uses image analysis technology to analyze the received images and identify the problem areas.

[0786] Step 4:

[0787] The server retrieves related manual information from a database based on the analysis results.

[0788] Step 5:

[0789] The server generates a response message based on the acquired manual information and reference image.

[0790] Step 6:

[0791] The server generates a response message and sends it to the user's terminal.

[0792] Step 7:

[0793] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[0794] Step 8:

[0795] The user continues the operation while referring to the response message received.

[0796] Providing information on pricing plans

[0797] Step 1:

[0798] A user uses a messaging app to send a message saying, "What's the best rate plan?"

[0799] Step 2:

[0800] A server receives a message from the user.

[0801] Step 3:

[0802] The server analyzes the user message using a natural language processing model and extracts the query "price plan."

[0803] Step 4:

[0804] The server retrieves the user usage data from the database based on the query.

[0805] Step 5:

[0806] The server uses the generated AI model to calculate the optimal pricing plan based on the acquired usage data.

[0807] Step 6:

[0808] The server generates a response message based on the calculated details of the pricing plan.

[0809] Step 7:

[0810] The server generates a response message and sends it to the user's terminal.

[0811] Step 8:

[0812] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[0813] Step 9:

[0814] Based on the response message received by the user, the user compares and considers the proposed pricing plans.

[0815] Example 2

[0816] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0817] Today's users need to quickly and accurately obtain various information and operate digital devices without hesitation. However, if the information they need, such as confirmation of operation instructions, image-based support, or selection of pricing plans, is not provided promptly, it can cause stress and waste time. Furthermore, if appropriate responses are not provided based on the user's emotions, the user experience can be significantly impaired. Therefore, there is a need to develop a system that can recognize user emotions and provide appropriate responses accordingly.

[0818] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for sending an inquiry for a user to confirm an operation method through a communication application; means for receiving the inquiry from the user and analyzing it using a natural language processing engine; means for acquiring appropriate command information from a data storage means based on the analysis result; means for generating a response message based on the acquired command information; means for sending the response message to the user; and means for analyzing the user's emotion in the response message using an emotion recognition engine and adjusting the content. This allows the user to efficiently acquire information through the communication application and receive an appropriate response that matches their emotion.

[0819] A "user" is someone who uses this system to inquire about information, send images, or check pricing plans.

[0820] A "communication application" is software that allows users to communicate with a server in two-way communication.

[0821] A "server" is a central processing unit that receives inquiries and images from users and analyzes and responds.

[0822] A "natural language processing engine" is a software component that analyzes and understands the meaning of a user's text messages.

[0823] "Command information" refers to instructions and explanations regarding operation methods and procedures, and information desired by the user.

[0824] "Data storage means" refers to a database or storage system that stores information such as manual information and usage data.

[0825] A "response message" is a message that is generated based on the analysis results and command information and sent to the user.

[0826] An "emotion recognition engine" is a software component that analyzes the user's emotions in a response message and adjusts the response content accordingly.

[0827] "Image analysis technology" is a technology for analyzing images sent by users and identifying the information and operation locations contained therein.

[0828] A "generative AI model" is an artificial intelligence model that calculates the optimal pricing plan based on user usage data.

[0829] "Usage Data" refers to the usage history, patterns, and consumption data of the services you use.

[0830] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a communication application. This system aims to recognize the user's emotions and provide a response that corresponds to those emotions. Specific embodiments of this invention are described below.

[0831] First, the user communicates with the server using a communication application, which provides an interface for sending user queries and images to the server in text or image format.

[0832] The server has an API for receiving inquiries and images from users. The received messages and images are analyzed using a natural language processing engine (e.g., Google NLU) and image analysis technology (e.g., Google Vision API). The natural language processing engine analyzes the user's text message and understands its meaning. Meanwhile, the image analysis technology analyzes the image sent by the user and identifies specific operation points and problems.

[0833] Based on the analysis results, the server retrieves appropriate instruction information from a data storage means (e.g., SQL database), including information such as operating procedures, reference images, and pricing plans. Based on the retrieved information, the server generates a response message and sends it to the user.

[0834] Furthermore, the server is equipped with an emotion recognition engine (e.g., Microsoft Text Analytics API) that analyzes the user's emotions in the generated response message. Depending on the analysis results, the content and tone of the response message are adjusted. This function allows appropriate support to be provided according to the user's situation and emotions.

[0835] As a concrete example, consider a case where a user sends a query through a communication application asking, "How do I back up my new smartphone?" In this case, the server receives and analyzes the message and retrieves relevant instruction information from a database. The server then uses the retrieved information to generate a response message containing concise and easy-to-understand backup instructions. This response message is then analyzed by an emotion recognition engine, and detailed explanations and supplementary information are added as needed. Finally, the user can successfully back up their smartphone using the received information.

[0836] Furthermore, if a user encounters a problem with the smartphone's settings screen, they can send a screenshot via a communication application. In this case, the server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant instruction information from a database and generates a response message containing detailed instructions and reference images. This response message is also analyzed by an emotion recognition engine, and the tone and content are adjusted according to the user's emotions. This allows the user to understand the specific operation method and solve the problem.

[0837] In providing pricing plan information, consider the case where a user asks, "What is the best plan based on my usage this month?" In this case, the server retrieves the user's usage data from a database and calculates the optimal pricing plan using a generative AI model (e.g., OpenAI GPT-3). The obtained pricing plan details are compiled into a response message, analyzed using an emotion recognition engine, and additional information and support options are provided as needed. This allows the user to review the plans offered and select the one that best suits them.

[0838] This system is expected to improve the user experience by allowing users to efficiently obtain information through communication applications and receive appropriate responses that match their emotions.

[0839] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0840] Step 1:

[0841] A user sends an inquiry through a communication application requesting confirmation of an operation method, advice with an image, or information on a rate plan.

[0842] What happens: A user enters and attaches a text message (e.g., "How do I back up my new phone?" or a screenshot) into the input field in a messaging app and hits the send button.

[0843] Input: A query sent by the user in the form of text or an image.

[0844] Output: The query data received by the server from the user.

[0845] Step 2:

[0846] The server receives the inquiry data (text message or image) received through the communication application.

[0847] Specific operation: The server's API receives the query data and adds it to an internal queue for analysis.

[0848] Input: Query data sent from the communication application to the server.

[0849] Output: Query data added to the queue for analysis inside the server.

[0850] Step 3:

[0851] The server uses a natural language processing engine (e.g., Google NLU) to analyze the text message and understand the user's intent.

[0852] What it does: A natural language processing engine analyzes text messages to identify keywords and intent.

[0853] Input: A text message from the user.

[0854] Output: Parsed keywords and user intent.

[0855] Step 4:

[0856] The server uses image analysis technology (e.g., Google Vision API) to analyze the image sent by the user and identify specific operation points and problems.

[0857] What it does: Image analysis technology analyzes image data and identifies the elements it contains.

[0858] Input: Image data sent by the user.

[0859] Output: Information on the analyzed elements in the image and the specific operation locations.

[0860] Step 5:

[0861] The server retrieves the appropriate instruction information (operational procedures, reference images, pricing plans, etc.) from a data storage means (e.g., SQL database).

[0862] Specific operation: An SQL query is executed and the necessary instruction information is retrieved from the database.

[0863] Input: Analysis results from a natural language processing engine or image analysis technology.

[0864] Output: Instruction information retrieved from the database.

[0865] Step 6:

[0866] The server generates a response message based on the acquired information and sends it to the user.

[0867] Specific operation: The text generation engine creates a response message and delivers it to the user's device via the message sending function.

[0868] Input: Instruction information retrieved from the database.

[0869] Output: The generated response message.

[0870] Step 7:

[0871] The server's emotion recognition engine (e.g., Microsoft Text Analytics API) analyzes the user's emotion in the response message and adjusts the content of the response message as necessary.

[0872] What it does: Emotion-sensing algorithms analyze the user's tone and emotions and adjust the content and tone of the response message accordingly.

[0873] Input: The generated response message.

[0874] Output: The adjusted response message.

[0875] Step 8:

[0876] The user takes appropriate action based on the information received.

[0877] Specific actions: Read the information provided in the messaging app and follow the instructions to change your smartphone settings or check your plan.

[0878] Input: The response message received from the server.

[0879] Output: User action (performing a procedure, resolving a problem, checking pricing plans, etc.).

[0880] (Application example 2)

[0881] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0882] In conventional inquiry systems using messaging applications, it was difficult for users to receive appropriate responses when requesting confirmation of operation procedures or problem resolution. Furthermore, responses did not take into account the user's emotions, resulting in insufficient support for users who felt particularly confused or dissatisfied. This resulted in a poor user experience and a decrease in system usage. Furthermore, in content distribution services, real-time support for the content being viewed was insufficient, making it difficult to improve user satisfaction.

[0883] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0884] In this invention, the server includes: means for sending a query to a user to confirm operation methods through a message application; means for receiving the query from the user and analyzing the query using a natural language processing model; means for retrieving appropriate manual information from a database based on the analysis result; means for generating a response message based on the retrieved manual information; means for adjusting the content of the response message using an emotion engine that analyzes the user's emotions; means for sending the response message to the user; and means for responding in real time to inquiries about content being viewed as part of a content distribution service. This enables users to receive prompt and appropriate support and further allows them to receive flexible responses based on their emotions, which is expected to significantly improve the user experience and increase the frequency of system use.

[0885] A "messaging application" is software that allows users to send and receive messages to and from a server.

[0886] A "server" is a computer system that receives user queries and images, retrieves information from a database, and generates and sends response messages.

[0887] A "natural language processing model" is a machine learning algorithm that analyzes text data and understands its meaning and intent.

[0888] "Manual information" refers to documents or data that contain detailed explanations of operation and setting methods.

[0889] A "response message" is text data that is generated by the server in response to a user's inquiry.

[0890] An "emotion engine" is a technology that analyzes emotions from a user's text message and adjusts the content of the response message accordingly.

[0891] "Image analysis technology" is a technology for analyzing received image data and extracting important information.

[0892] "Usage data" is data about a user's activity and usage history.

[0893] A "generative AI model" is an artificial intelligence model that makes predictions and recommendations based on collected data.

[0894] A "content distribution service" is a platform and service for providing users with digital content such as videos, music, and e-books.

[0895] This invention is a system that allows users to quickly and appropriately receive operation instructions, image advice, and information on pricing plans through a messaging application. This system also has the ability to analyze the user's emotions and provide responses according to them.

[0896] The system is configured as follows:

[0897] 1. User Interface

[0898] Users use the messaging application to ask for instructions, send images, or request information about pricing plans.

[0899] 2. Server Functions

[0900] The server responds to user inquiries using various modules, specifically with the following functions:

[0901] Natural Language Processing Model

[0902] The server analyzes the user's text query using a natural language processing model, which is used to understand the user's intent and generate an appropriate query.

[0903] Database Access

[0904] Based on the analysis results, the server retrieves relevant manual information and usage data from the database.

[0905] Response message generation

[0906] The server generates a response message for the user based on the acquired information, and also uses an emotion engine to take the user's emotions into account.

[0907] Emotion Engine

[0908] The server uses an emotion engine to analyze the user's emotions and adjusts the content of the response message, adding more detailed explanations or encouraging words if the user is confused.

[0909] 3. Image analysis function

[0910] When a user submits an image, the server uses image analysis technology to analyze the image and retrieve relevant information.

[0911] 4. Optimization function

[0912] The server uses a generative AI model to recommend the optimal pricing plan based on the user's usage data.

[0913] Hardware and software used

[0914] Server: High-performance computer

[0915] Natural Language Processing models: Machine learning algorithms such as BERT

[0916] Database: SQL or NoSQL database

[0917] Sentiment Engine: Dedicated sentiment analysis software

[0918] Image analysis technology: OpenCV, etc.

[0919] Generative AI models: GPT-3, etc.

[0920] Specific examples

[0921] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives the message, analyzes it using a natural language processing model, and retrieves the query "how to back up" from the database. After constructing a response message, it analyzes the user's emotions using an emotion engine, adjusts the content as necessary, and sends the response message.

[0922] Prompt Sentence Examples

[0923] When a user sends a message asking, "What is the best plan based on this month's usage?", the server analyzes it using a natural language processing model, retrieves the user's usage data from a database, analyzes it using a generative AI model, calculates the optimal pricing plan, and sends a response message adjusted by an emotion engine to the user.

[0924] This allows users to receive prompt and appropriate support, improving the user experience.

[0925] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0926] Step 1:

[0927] A user sends an inquiry through a messaging application, asking for confirmation of operation instructions, sending an image, or requesting information on a pricing plan. The input is a text message or image data from the user, which is sent to the server. The output is the data received by the server.

[0928] Step 2:

[0929] The server receives the user's inquiry and image and analyzes the text message using a natural language processing model. The input is the user's inquiry message, and based on this input, a natural language processing model (e.g., BERT) is used to analyze and extract meaning and intent. Specifically, the message is input into the model, and the intent and related keywords are obtained. The output is the analysis result (intent or query).

[0930] Step 3:

[0931] Based on the analysis results, the server retrieves related manual information and user usage data from the database. The input is the analysis result from step 2, and based on this analysis result, it queries the appropriate information from an SQL or NoSQL database. Specifically, it executes a database query and retrieves the relevant data. The output is the retrieved manual information and usage data.

[0932] Step 4:

[0933] The server uses image analysis technology to analyze the image received from the user. The input is the image data sent by the user, which is analyzed using image analysis software such as OpenCV. Specifically, the image is input and information related to the operation method is extracted. The output is the analysis result.

[0934] Step 5:

[0935] The server uses a generative AI model to calculate the optimal pricing plan based on the user's usage data. The input is the user's usage data, which is then input into the model to generate the optimal plan. Specifically, the generative AI model (e.g., GPT-3) makes predictions based on the usage data and calculates the pricing plan. The output is details of the calculated pricing plan.

[0936] Step 6:

[0937] The server generates a response message based on the acquired manual information and analysis results. The input is the manual information and analysis results, and a response message is generated for the user based on this. Specifically, the required information is compiled as text. The output is the generated response message.

[0938] Step 7:

[0939] The server adjusts the content of the response message using an emotion engine that analyzes the user's emotion. The input is the user's message and the generated response message, which are analyzed by the emotion engine. Specifically, the emotion in the message is analyzed and the tone and content of the response message are adjusted accordingly. The output is the adjusted response message.

[0940] Step 8:

[0941] The server sends the adjusted response message to the user, the input is the adjusted response message, and the server performs an operation to send it to the user's terminal, and the output is the response message displayed on the user's terminal.

[0942] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0943] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0944] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0945] [Third embodiment]

[0946] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0947] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0948] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0949] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0950] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0951] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0952] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0953] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0954] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0955] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0956] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0957] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0958] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, advice using images, and information on rate plans through a message application.

[0959] 1. Checking the operation method

[0960] When a user uses a messaging application to make a query such as "How do I back up my new smartphone?", the server receives the message. The server uses a natural language processing model to analyze the user's message and extract queries related to "how to back up." The server retrieves appropriate manual information from a database based on this query. Based on the retrieved manual information, the server generates a response message for the user and sends it to the user's device. The user then backs up their smartphone based on the received information.

[0961] 2. Photo advice

[0962] When a user sends a screenshot of the part of the operation they are having trouble with via a messaging application, the server receives the image. The server then analyzes the image using image analysis technology to identify the relevant operating procedure. Based on the analysis results, the server retrieves relevant manual information from a database and generates a response message including specific operating procedures and reference images. The generated response message is then sent from the server to the user's device. The user then refers to the received information and continues operating the device.

[0963] 3. Providing information on pricing plans

[0964] When a user sends a query requesting information about pricing plans through a messaging application, the server receives the message. The server uses a natural language processing model to analyze the user message and extract queries related to "pricing plans." The server then retrieves the user's usage data from a database. The server uses a generative AI model to calculate the optimal pricing plan based on the retrieved usage data. Based on the calculated pricing plan details, the server generates a response message for the user and sends it to the user's device. The user compares and considers the proposed pricing plans based on the received information.

[0965] Specific examples

[0966] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives and analyzes the message, then retrieves the relevant manual information from the database. Based on the retrieved information, the server generates and sends a response message containing concise and easy-to-understand backup instructions. The user can then use the received information to successfully back up their smartphone.

[0967] Furthermore, if a user has trouble with the smartphone's settings screen, they can send a screenshot. The server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant manual information from a database and sends the user a response message containing detailed instructions and reference images. This allows the user to understand the specific operation method and solve the problem.

[0968] When providing information on pricing plans, if the user sends a message saying "Tell me the best pricing plan," the server will analyze the user's usage data, calculate the best pricing plan, and propose it to them. The user can then check the details of the plans provided and select the one that best suits them.

[0969] The processing flow will be explained below.

[0970] Check operation method

[0971] Step 1:

[0972] A user uses a messaging application to send a message saying, "How do I back up my new phone?"

[0973] Step 2:

[0974] A server receives a message from the user.

[0975] Step 3:

[0976] The server analyzes the user message using a natural language processing model and extracts the query "How to back up."

[0977] Step 4:

[0978] The server retrieves relevant manual information from a database based on the query.

[0979] Step 5:

[0980] The server generates a response message based on the acquired manual information.

[0981] Step 6:

[0982] The server generates a response message and sends it to the user's terminal.

[0983] Step 7:

[0984] The smartphone is backed up based on the response message received by the user.

[0985] Photo-based advice

[0986] Step 1:

[0987] The user sends a screenshot of the part of the operation they are having trouble with via a messaging application.

[0988] Step 2:

[0989] A server receives the image from the user.

[0990] Step 3:

[0991] The server uses image analysis technology to analyze the received images and identify the problem areas.

[0992] Step 4:

[0993] The server retrieves related manual information from a database based on the analysis results.

[0994] Step 5:

[0995] The server generates a response message based on the acquired manual information and reference image.

[0996] Step 6:

[0997] The server generates a response message and sends it to the user's terminal.

[0998] Step 7:

[0999] The user continues the operation while referring to the response message received.

[1000] Providing information on pricing plans

[1001] Step 1:

[1002] A user uses a messaging app to send a message saying, "What's the best rate plan?"

[1003] Step 2:

[1004] A server receives a message from the user.

[1005] Step 3:

[1006] The server analyzes the user message using a natural language processing model and extracts the query "price plan."

[1007] Step 4:

[1008] The server retrieves the user usage data from the database based on the query.

[1009] Step 5:

[1010] The server uses the generated AI model to calculate the optimal pricing plan based on the acquired usage data.

[1011] Step 6:

[1012] The server generates a response message based on the calculated details of the pricing plan.

[1013] Step 7:

[1014] The server generates a response message and sends it to the user's terminal.

[1015] Step 8:

[1016] Based on the response message received by the user, the user compares and considers the proposed pricing plans.

[1017] Example 1

[1018] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1019] The challenge is to effectively utilize messaging applications, a modern means of communication, to quickly and appropriately confirm operation procedures, resolve problems, and recommend optimal pricing plans. Conventional systems require users to search multiple sources of information and obtain the necessary information on their own, which can be a cumbersome process. In addition, selecting a pricing plan requires users to perform complex calculations, making it difficult to choose the optimal plan.

[1020] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1021] In this invention, the server includes: means for sending an inquiry for the user to confirm an operation method through a message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain appropriate information from a database based on the analysis result; means for the server to generate a response message based on the obtained information; means for the server to send the response message to the user; means for the user to perform an operation based on the response message displayed on the terminal; means for the user to send an image through the message application; means for the server to receive an image from the user and analyze it using image analysis technology; means for the server to obtain related information from a database based on the analysis result; means for the server to generate a response message based on the obtained information and a reference image; means for the server to send the response message to the user; means for the user to perform an operation based on the response message displayed on the terminal; means for the user to send an inquiry requesting information on a rate plan through the message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain usage data of the user from a database; and means for the server to calculate an optimal rate plan based on the obtained usage data. The system includes a means for the server to generate a response message based on the calculated details of the pricing plan, a means for the server to send the response message to the user, and a means for the user to consider plans based on the response message displayed on the terminal, thereby enabling the user to quickly and appropriately obtain the necessary information, efficiently solve the problems they are facing, and select the most suitable pricing plan without going through complicated processes.

[1022] "User" refers to an individual or corporation that uses the system.

[1023] "Messaging application" refers to software or applications for sending and receiving information such as text messages and images.

[1024] An "inquiry" refers to a question or request made by a user to the system.

[1025] "Server" refers to a computer system that receives and processes requests from users.

[1026] A "natural language processing model" refers to algorithms and tools for analyzing human language.

[1027] "Analysis" refers to the act of understanding and processing received data using natural language processing models and image analysis techniques.

[1028] A "database" refers to a system for organizing and storing data and efficiently retrieving necessary information.

[1029] "Manual information" refers to documents and data that explain operation methods and procedures.

[1030] A "response message" refers to a message that a server generates and sends in response to a user's query.

[1031] "Image analysis technology" refers to technology for analyzing image data and understanding its contents.

[1032] A "reference image" refers to a reference image provided for the purpose of explaining or instructing a user.

[1033] "Usage status data" refers to data related to a user's service usage status and history.

[1034] "Price plan" refers to various plans that determine the fee structure for the services that a user uses.

[1035] The "optimal pricing plan" refers to the pricing plan that is most profitable for the user based on the user's usage data.

[1036] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a message application. This system performs various data processing and data calculations using the following hardware and software.

[1037] 1. Hardware and Software Used

[1038] Server: A high-performance server computer is required. In this system, it stores the database and AI models and performs calculations.

[1039] Device: A mobile device operated by a user, such as a smartphone or tablet, that must have a messaging application installed.

[1040] Message application: Software for sending and receiving messages between users and servers.

[1041] Natural language processing model: A model for analyzing messages from users, such as BERT or OpenAI's GPT-3.

[1042] Database: This stores manual information and user usage data. For example, a database management system such as MySQL is used.

[1043] Image analysis technology: Image analysis tools such as Google Cloud Vision API are used.

[1044] Generative AI model: For example, OpenAI's GPT-3 is used as an AI model to calculate the optimal pricing plan based on usage data.

[1045] 2. Implementing the Invention

[1046] Check operation method

[1047] A user uses a messaging application to send a message saying, "Tell me how to back up my new smartphone." The server receives this message and analyzes it using a natural language processing model (e.g., BERT) to extract queries related to "how to back up." Based on the extracted query, the server retrieves the appropriate manual information from a MySQL database, generates a response message, and sends it to the user's device. The user then backs up their smartphone based on the received information.

[1048] Photo-based advice

[1049] The user sends a screenshot of the part of the operation where they are having trouble to the server via a messaging application. The server receives the image and analyzes it using the Google Cloud Vision API. Based on the analysis results, the server retrieves relevant manual information from a MySQL database, generates a response message containing specific operating procedures and reference images, and sends it to the user's device. The user can continue operating the device by referring to the received information.

[1050] Providing information on pricing plans

[1051] The user sends a message via a messaging application saying, "What is the best pricing plan?" The server receives this message, analyzes it using a natural language processing model (for example, OpenAI's GPT-3), and extracts queries related to "pricing plans." The server retrieves the user's usage data from a MySQL database and calculates the best pricing plan using GPT-3 based on the retrieved data. A response message is generated based on the details of the calculated pricing plan and sent to the user's device. The user then checks the details of the provided plans and selects the most suitable plan.

[1052] Specific examples

[1053] Check operation method

[1054] Example user question: "How do I back up my new phone?"

[1055] Example prompt: "Generate a message to return when a user is asked how to back up their new phone."

[1056] Photo-based advice

[1057] Example user question: "Send a screenshot of your smartphone's settings screen"

[1058] Example prompt: "Generate a response message when a user sends you a screenshot of their settings screen."

[1059] Providing information on pricing plans

[1060] Example user question: "What is the best pricing plan?"

[1061] Example prompt: "Generate a response message when a user asks what the best pricing plan is."

[1062] This allows users to quickly and appropriately obtain the necessary information, efficiently resolve problems, and select the most suitable rate plan without having to go through complicated processes.

[1063] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1064] Check operation method

[1065] Step 1:

[1066] A query is sent to the user to confirm the operation method.

[1067] Type: A user uses the Messages app to type the text message "How do I back up my new phone?"

[1068] Output: The message is sent to the server.

[1069] Step 2:

[1070] A server receives the user query and analyzes it using a natural language processing model.

[1071] Input: The server receives a message sent by the user.

[1072] Data processing: The server analyzes the message using a natural language processing model (e.g., BERT) and extracts keywords related to "backup methods."

[1073] Output: Queries related to "Backup Method" are generated.

[1074] Step 3:

[1075] The server retrieves the appropriate information from the database.

[1076] Input: Queries related to "how to back up".

[1077] Data processing: The server searches and retrieves the relevant manual information from the MySQL database.

[1078] Output: The retrieved manual information.

[1079] Step 4:

[1080] The server generates a response message based on the acquired information.

[1081] Input: The retrieved manual information.

[1082] Data processing: Based on the obtained manual information, the server generates a response message in a format that is easy for the user to understand.

[1083] Output: A response message is generated.

[1084] Step 5:

[1085] The server sends the response message to the user.

[1086] Input: The response message.

[1087] Output: A reply message is sent to the user's device via the Messages application.

[1088] Step 6:

[1089] The user performs an operation based on the response message displayed on the terminal.

[1090] Input: The response message.

[1091] What happens: The user follows the instructions in the response message to start backing up their smartphone.

[1092] Output: Your phone is backed up.

[1093] Photo-based advice

[1094] Step 1:

[1095] A user sends an image through a messaging application.

[1096] Input: The user attaches a screenshot of the problem and sends it via messaging.

[1097] Output: The screenshot is sent to the server.

[1098] Step 2:

[1099] A server receives the image from the user and analyzes it using image analysis techniques.

[1100] Input: A screenshot submitted by the user.

[1101] Data processing: The server uses the Google Cloud Vision API to analyze the screenshots and identify specific problem areas and operational steps.

[1102] Output: Analysis results.

[1103] Step 3:

[1104] The server retrieves the relevant information from the database.

[1105] Input: Analysis results.

[1106] Data processing: The server searches and retrieves the relevant manual information and reference images from the MySQL database.

[1107] Output: Retrieved information and reference images.

[1108] Step 4:

[1109] The server generates a response message based on the acquired information and the reference image.

[1110] Input: Retrieved information and reference image.

[1111] Data processing: Based on this information, the server generates a response message in a format that is easy for the user to understand.

[1112] Output: A response message is generated.

[1113] Step 5:

[1114] The server sends the response message to the user.

[1115] Input: The response message.

[1116] Output: A reply message is sent to the user's device via the Messages application.

[1117] Step 6:

[1118] The user performs the operation by referring to the response message displayed on the terminal.

[1119] Input: The response message.

[1120] Specific actions: The user follows the response message and uses the displayed steps and reference images to solve the problem.

[1121] Output: Problem solved.

[1122] Providing information on pricing plans

[1123] Step 1:

[1124] A user sends a request for pricing plan information through a messaging application.

[1125] Input: A user sends a message asking, "What is the best pricing plan?"

[1126] Output: The query is sent to the server.

[1127] Step 2:

[1128] A server receives the user query and analyzes it using a natural language processing model.

[1129] Input: A query message from the user.

[1130] Data processing: The server analyzes the message using a natural language processing model such as OpenAI's GPT-3 to extract queries related to "price plans."

[1131] Output: Queries related to "price plan" are generated.

[1132] Step 3:

[1133] The server retrieves user usage data from the database.

[1134] Input: Queries related to "price plans".

[1135] Data processing: The server retrieves user usage data from the MySQL database.

[1136] Output: The captured usage data.

[1137] Step 4:

[1138] The optimal pricing plan is calculated based on the usage data acquired by the server.

[1139] Input: Captured usage data.

[1140] Data processing: The server uses a generative AI model (e.g., OpenAI's GPT-3) to calculate the optimal pricing plan.

[1141] Output: Best pricing plan.

[1142] Step 5:

[1143] The server generates a response message based on the calculated details of the rate plan.

[1144] Input: Your best rate plan information.

[1145] Data processing: The server generates a response message in a format that is easy for the user to understand.

[1146] Output: A response message is generated.

[1147] Step 6:

[1148] The server sends the response message to the user.

[1149] Input: The response message.

[1150] Output: A reply message is sent to the user's device via the Messages application.

[1151] Step 7:

[1152] The user considers a plan based on the response message displayed on the terminal.

[1153] Input: The response message.

[1154] Specific behavior: The user reviews the details of the offered pricing plans and selects the most suitable plan.

[1155] Output: The user selects the best pricing plan.

[1156] (Application example 1)

[1157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1158] Modern users face numerous configuration and security issues related to digital devices and security measures. While a support system is needed to quickly and appropriately resolve these issues, conventional methods often require complex operation procedures and responses to anomaly detection alerts, making them difficult for users to understand. While it is also important to propose optimal security and pricing plans, there is a lack of systems that can effectively accomplish this. The objective of this invention is to solve these problems by providing a system that allows users to quickly and appropriately confirm operation procedures, respond to anomaly detection, and receive recommendations for optimal plans.

[1159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1160] In this invention, the server includes: means for a user to send an inquiry about operation instructions or how to respond to an anomaly detection alert via a message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain appropriate manual information or response information from a database based on the analysis results; means for the server to generate a response message based on the obtained information; means for the server to send the response message to the user; means for the server to receive an anomaly detection alert image from the user and analyze it using image analysis technology; and means for the server to obtain manual information about the cause of the anomaly and how to deal with it based on the analysis results. This allows the user to easily and quickly confirm operation instructions, respond to an anomaly detection, and receive suggestions for optimal security and pricing plans.

[1161] "User" refers to an individual or company that uses the system or service.

[1162] A "messaging application" is software for sending and receiving text messages and images.

[1163] "Method of operation" refers to the steps or techniques for using a digital device or software.

[1164] An "anomaly detection alert" is a warning message that notifies the user when the system detects an anomaly.

[1165] A "query" is a message sent by a user to the system requesting information or instructions.

[1166] A "server" is a computer system that receives and analyzes messages from users and provides the necessary information.

[1167] A "natural language processing model" refers to algorithms and technologies used to analyze and understand the meaning of text messages entered by users.

[1168] "Analysis results" are the results of analyzing data obtained using natural language processing models and image analysis technology.

[1169] "Manual information" refers to materials that include specific information required by users, such as operation methods and procedures for responding to abnormality detection.

[1170] A "response message" is a message containing a response generated by a server in response to a user's inquiry.

[1171] "Image analysis technology" is a technology for processing transmitted images and analyzing their contents.

[1172] A "reference image" is an image used to explain operating procedures and how to respond to abnormalities.

[1173] "Price Plan" refers to the pricing structure or plan that a User can select to use the Service.

[1174] "Usage status data" refers to data that indicates a user's service usage history and trends.

[1175] "Security Plan" refers to a combination of security measures designed to protect a user's devices and data.

[1176] A "generative AI model" is an artificial intelligence model that generates new information and responses based on data.

[1177] A "prompt" refers to an instruction or question that is input to a model.

[1178] The present invention provides a system that allows a user to quickly and appropriately check operation instructions and respond to an anomaly detection alert through a message application. Specific embodiments of this system will be described below.

[1179] First, when a user wants to check how to operate a device or how to respond to an anomaly detection alert, they send an inquiry from their device via a messaging application. Here, "user" refers to individuals or companies that use the system or service. "Messaging application" is software for sending and receiving text messages and images. "Operation method" refers to the steps and methods for using digital devices and software.

[1180] The server receives the user's inquiry and analyzes it using a natural language processing model (software used: OpenAI GPT-3). Based on the analysis results, the server retrieves "manual information" from a database (software used: MySQL, PostgreSQL). A "natural language processing model" refers to the algorithms and technologies used to analyze and understand the meaning of text messages entered by users. The "database" contains detailed manual information on operating procedures and how to respond to abnormalities.

[1181] Next, the server generates a "response message" based on the retrieved manual information and sends it to the user's terminal. A "response message" is a message that the server generates containing a response to a user's inquiry. For example, if a user sends a message saying "Please tell me how to change the firewall settings," the server generates a response message based on the relevant manual information and sends it to the user.

[1182] We will also explain the case where a user sends an anomaly detection alert with a screenshot attached. The image sent from the user's device is analyzed on the server using "image analysis technology" (software used: OpenCV, TensorFlow). Based on the analysis results, the server retrieves manual information on the "cause of the anomaly" and its "measures" from the database and sends it to the user as a response message.

[1183] Finally, if a user requests information on the optimal pricing plan or security plan, the optimal plan is calculated using the user's usage data and provided to the user as a response message. "Price plan" refers to the pricing structure or plan that a user can select to use the service. "Security plan" refers to a combination of security measures to protect the user's device and data.

[1184] As a concrete example, consider the case where a user sends a message saying, "What is the best security plan for me?" In this case, the "prompt sentence" would be as follows:

[1185] A user sends a message saying, "What security plan is best for me?"

[1186] The server analyzes this message, obtains the user's usage data, calculates the optimal security plan, and generates a response message based on this information.

[1187] As described above, the present invention enables a user to easily and quickly check operation methods, respond to detected abnormalities, and receive proposals for optimal security and fee plans.

[1188] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1189] Step 1:

[1190] A user sends an inquiry about operation methods or how to respond to an anomaly detection alert through a messaging application. In this case, for example, the input is a text message such as "Please tell me how to change the firewall settings." This message is sent to the server. The input is the user's message, and the output is the message sent to the server.

[1191] Step 2:

[1192] The server receives the query from the user. The server takes the message sent from the messaging application and analyzes the text message using a natural language processing model (software used: OpenAI GPT-3). Specifically, the server extracts meaning from the user's message and forms a query such as "How do I change my firewall settings?" The input is the user's message, and the output is the analyzed query.

[1193] Step 3:

[1194] The server retrieves appropriate manual information from a database (software used: MySQL, PostgreSQL) based on the analysis results. The server creates a database query using the analyzed query and searches for related manual information. Specifically, it retrieves manual information containing procedures and explanations on "how to change firewall settings" from the database. The input is the analyzed query, and the output is the retrieved manual information.

[1195] Step 4:

[1196] The server generates a response message based on the acquired information. Based on the acquired manual information, the server assembles the response message in a format that is easy for the user to understand. Specifically, it organizes the procedures by item and creates a message with supplementary explanations. It uses a generative AI model to generate a response message in natural language. The input is the acquired manual information, and the output is the generated response message.

[1197] Step 5:

[1198] The server sends the response message to the user. The server generates a response message and sends it to the user's terminal again through the message application. The user can then view the received message and take action based on its content. The input is the generated response message, and the output is the message sent to the user's terminal.

[1199] Step 6:

[1200] The user sends a screenshot of the anomaly detection alert through a messaging application. For example, the image is sent along with a text message such as "I don't understand the content of the screenshot below." The input is the screenshot image and message from the user, and the output is the data sent to the server.

[1201] Step 7:

[1202] The server receives the image from the user and analyzes it using image analysis technology. The server captures the screenshot image sent and identifies any abnormalities using image analysis software (software used: OpenCV, TensorFlow). Specifically, it extracts error messages and warnings from the screen and analyzes the related information. The input is the screenshot image, and the output is the analyzed abnormalities and their contents.

[1203] Step 8:

[1204] The server retrieves manual information relating to the cause of the anomaly and how to deal with it from a database based on the analysis results. The server creates a database query based on the analyzed anomaly location and searches for related manual information. Specifically, it retrieves manual information including procedures and solutions for the identified anomaly from the database. The input is the analyzed anomaly location, and the output is the retrieved manual information.

[1205] Step 9:

[1206] The server generates a response message based on the acquired information and sends it to the user. The server creates a response message that clearly explains the cause of the abnormality and how to deal with it. The input is the acquired manual information, and the output is the generated response message and its transmission to the user's terminal.

[1207] Example prompt sentence:

[1208] A user sent a message asking, "How do I change my firewall settings?"

[1209] A user has sent a screenshot. Please analyze the screenshot and provide appropriate advice.

[1210] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1211] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a message application, and also aims to recognize the user's emotions and provide a response that corresponds to those emotions.

[1212] 1. Checking the operation method

[1213] When a user uses a messaging application to make a query such as "How do I back up my new smartphone?", the server receives the message. The server uses a natural language processing model to analyze the user's message and extract queries related to "how to back up." The server then retrieves appropriate manual information from a database based on this query. Based on the retrieved manual information, the server generates a response message for the user and sends it to the user's device. Next, the emotion engine analyzes the user's emotion in the response message and adjusts the content of the response message according to the user's emotion. The user then backs up their smartphone based on the received information.

[1214] 2. Photo advice

[1215] When a user sends a screenshot of a part of the operation where they are having trouble via a messaging application, the server receives the image. The server analyzes the image using image analysis technology and identifies the relevant operating procedure. Based on the analysis results, the server retrieves relevant manual information from a database and generates a response message including specific operating procedures and reference images. The generated response message is then sent from the server to the user's device. The emotion engine analyzes the user's emotion in the response message and adjusts the content of the response message according to the user's emotion. The user then continues operating the device, referring to the received information.

[1216] 3. Providing information on pricing plans

[1217] When a user sends a query requesting information about pricing plans through a messaging application, the server receives the message. The server uses a natural language processing model to analyze the user message and extract queries related to "pricing plans." The server then retrieves the user's usage data from a database. The server uses a generative AI model to calculate the optimal pricing plan based on the retrieved usage data. Based on the calculated pricing plan details, the server generates a response message for the user and sends it to the user's device. The emotion engine analyzes the user's emotions in the response message and adjusts the content of the response message according to the user's emotions. The user compares and considers the proposed pricing plans based on the received information.

[1218] Specific examples

[1219] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives and analyzes the message and retrieves relevant manual information from the database. The server then generates a response message with concise and easy-to-understand backup instructions based on the retrieved information. The emotion engine analyzes this response message and, if it determines that the user is confused, adjusts it by adding more detailed explanations and supplementary information. The user can then use the received information to successfully back up their smartphone.

[1220] If a user has trouble with the smartphone's settings screen, they can send a screenshot. The server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant manual information from a database and generates a response message including detailed instructions and reference images. If the emotion engine analyzes the response message and determines that the user is frustrated, it can soften the tone of the response message and add words of encouragement. This allows the user to understand the specific operation method and solve the problem.

[1221] When providing pricing plan information, the user sends a message asking, "What is the best plan based on this month's usage?" The server obtains the user's usage data and calculates the optimal pricing plan based on that data. After compiling the obtained pricing plan details into a response message, the emotion engine analyzes the user's emotions in the response message and, if the user is dissatisfied or has questions about the plan, provides additional information or support options to alleviate those emotions. The user can review the provided plan details and select the plan that best suits them.

[1222] The processing flow will be explained below.

[1223] Check operation method

[1224] Step 1:

[1225] A user uses a messaging application to send a message saying, "How do I back up my new phone?"

[1226] Step 2:

[1227] A server receives a message from the user.

[1228] Step 3:

[1229] The server analyzes the user message using a natural language processing model and extracts the query "How to back up."

[1230] Step 4:

[1231] The server retrieves relevant manual information from a database based on the query.

[1232] Step 5:

[1233] The server generates a response message based on the acquired manual information.

[1234] Step 6:

[1235] The server generates a response message and sends it to the user's terminal.

[1236] Step 7:

[1237] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[1238] Step 8:

[1239] The smartphone is backed up based on the response message received by the user.

[1240] Photo-based advice

[1241] Step 1:

[1242] The user sends a screenshot of the part of the operation they are having trouble with via a messaging application.

[1243] Step 2:

[1244] A server receives the image from the user.

[1245] Step 3:

[1246] The server uses image analysis technology to analyze the received images and identify the problem areas.

[1247] Step 4:

[1248] The server retrieves related manual information from a database based on the analysis results.

[1249] Step 5:

[1250] The server generates a response message based on the acquired manual information and reference image.

[1251] Step 6:

[1252] The server generates a response message and sends it to the user's terminal.

[1253] Step 7:

[1254] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[1255] Step 8:

[1256] The user continues the operation while referring to the response message received.

[1257] Providing information on pricing plans

[1258] Step 1:

[1259] A user uses a messaging app to send a message saying, "What's the best rate plan?"

[1260] Step 2:

[1261] A server receives a message from the user.

[1262] Step 3:

[1263] The server analyzes the user message using a natural language processing model and extracts the query "price plan."

[1264] Step 4:

[1265] The server retrieves the user usage data from the database based on the query.

[1266] Step 5:

[1267] The server uses the generated AI model to calculate the optimal pricing plan based on the acquired usage data.

[1268] Step 6:

[1269] The server generates a response message based on the calculated details of the pricing plan.

[1270] Step 7:

[1271] The server generates a response message and sends it to the user's terminal.

[1272] Step 8:

[1273] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[1274] Step 9:

[1275] Based on the response message received by the user, the user compares and considers the proposed pricing plans.

[1276] Example 2

[1277] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1278] Today's users need to quickly and accurately obtain various information and operate digital devices without hesitation. However, if the information they need, such as confirmation of operation instructions, image-based support, or selection of pricing plans, is not provided promptly, it can cause stress and waste time. Furthermore, if appropriate responses are not provided based on the user's emotions, the user experience can be significantly impaired. Therefore, there is a need to develop a system that can recognize user emotions and provide appropriate responses accordingly.

[1279] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for sending an inquiry for a user to confirm an operation method through a communication application; means for receiving the inquiry from the user and analyzing it using a natural language processing engine; means for acquiring appropriate command information from a data storage means based on the analysis result; means for generating a response message based on the acquired command information; means for sending the response message to the user; and means for analyzing the user's emotion in the response message using an emotion recognition engine and adjusting the content. This allows the user to efficiently acquire information through the communication application and receive an appropriate response that matches their emotion.

[1280] A "user" is someone who uses this system to inquire about information, send images, or check pricing plans.

[1281] A "communication application" is software that allows users to communicate with a server in two-way communication.

[1282] A "server" is a central processing unit that receives inquiries and images from users and analyzes and responds.

[1283] A "natural language processing engine" is a software component that analyzes and understands the meaning of a user's text messages.

[1284] "Command information" refers to instructions and explanations regarding operation methods and procedures, and information desired by the user.

[1285] "Data storage means" refers to a database or storage system that stores information such as manual information and usage data.

[1286] A "response message" is a message that is generated based on the analysis results and command information and sent to the user.

[1287] An "emotion recognition engine" is a software component that analyzes the user's emotions in a response message and adjusts the response content accordingly.

[1288] "Image analysis technology" is a technology for analyzing images sent by users and identifying the information and operation locations contained therein.

[1289] A "generative AI model" is an artificial intelligence model that calculates the optimal pricing plan based on user usage data.

[1290] "Usage Data" refers to the usage history, patterns, and consumption data of the services you use.

[1291] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a communication application. This system aims to recognize the user's emotions and provide a response that corresponds to those emotions. Specific embodiments of this invention are described below.

[1292] First, the user communicates with the server using a communication application, which provides an interface for sending user queries and images to the server in text or image format.

[1293] The server has an API for receiving inquiries and images from users. The received messages and images are analyzed using a natural language processing engine (e.g., Google NLU) and image analysis technology (e.g., Google Vision API). The natural language processing engine analyzes the user's text message and understands its meaning. Meanwhile, the image analysis technology analyzes the image sent by the user and identifies specific operation points and problems.

[1294] Based on the analysis results, the server retrieves appropriate instruction information from a data storage means (e.g., SQL database), including information such as operating procedures, reference images, and pricing plans. Based on the retrieved information, the server generates a response message and sends it to the user.

[1295] Furthermore, the server is equipped with an emotion recognition engine (e.g., Microsoft Text Analytics API) that analyzes the user's emotions in the generated response message. Depending on the analysis results, the content and tone of the response message are adjusted. This function allows appropriate support to be provided according to the user's situation and emotions.

[1296] As a concrete example, consider a case where a user sends a query through a communication application asking, "How do I back up my new smartphone?" In this case, the server receives and analyzes the message and retrieves relevant instruction information from a database. The server then uses the retrieved information to generate a response message containing concise and easy-to-understand backup instructions. This response message is then analyzed by an emotion recognition engine, and detailed explanations and supplementary information are added as needed. Finally, the user can successfully back up their smartphone using the received information.

[1297] Furthermore, if a user encounters a problem with the smartphone's settings screen, they can send a screenshot via a communication application. In this case, the server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant instruction information from a database and generates a response message containing detailed instructions and reference images. This response message is also analyzed by an emotion recognition engine, and the tone and content are adjusted according to the user's emotions. This allows the user to understand the specific operation method and solve the problem.

[1298] In providing pricing plan information, consider the case where a user asks, "What is the best plan based on my usage this month?" In this case, the server retrieves the user's usage data from a database and calculates the optimal pricing plan using a generative AI model (e.g., OpenAI GPT-3). The obtained pricing plan details are compiled into a response message, analyzed using an emotion recognition engine, and additional information and support options are provided as needed. This allows the user to review the plans offered and select the one that best suits them.

[1299] This system is expected to improve the user experience by allowing users to efficiently obtain information through communication applications and receive appropriate responses that match their emotions.

[1300] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1301] Step 1:

[1302] A user sends an inquiry through a communication application requesting confirmation of an operation method, advice with an image, or information on a rate plan.

[1303] What happens: A user enters and attaches a text message (e.g., "How do I back up my new phone?" or a screenshot) into the input field in a messaging app and hits the send button.

[1304] Input: A query sent by the user in the form of text or an image.

[1305] Output: The query data received by the server from the user.

[1306] Step 2:

[1307] The server receives the inquiry data (text message or image) received through the communication application.

[1308] Specific operation: The server's API receives the query data and adds it to an internal queue for analysis.

[1309] Input: Query data sent from the communication application to the server.

[1310] Output: Query data added to the queue for analysis inside the server.

[1311] Step 3:

[1312] The server uses a natural language processing engine (e.g., Google NLU) to analyze the text message and understand the user's intent.

[1313] What it does: A natural language processing engine analyzes text messages to identify keywords and intent.

[1314] Input: A text message from the user.

[1315] Output: Parsed keywords and user intent.

[1316] Step 4:

[1317] The server uses image analysis technology (e.g., Google Vision API) to analyze the image sent by the user and identify specific operation points and problems.

[1318] What it does: Image analysis technology analyzes image data and identifies the elements it contains.

[1319] Input: Image data sent by the user.

[1320] Output: Information on the analyzed elements in the image and the specific operation locations.

[1321] Step 5:

[1322] The server retrieves the appropriate instruction information (operational procedures, reference images, pricing plans, etc.) from a data storage means (e.g., SQL database).

[1323] Specific operation: An SQL query is executed and the necessary instruction information is retrieved from the database.

[1324] Input: Analysis results from a natural language processing engine or image analysis technology.

[1325] Output: Instruction information retrieved from the database.

[1326] Step 6:

[1327] The server generates a response message based on the acquired information and sends it to the user.

[1328] Specific operation: The text generation engine creates a response message and delivers it to the user's device via the message sending function.

[1329] Input: Instruction information retrieved from the database.

[1330] Output: The generated response message.

[1331] Step 7:

[1332] The server's emotion recognition engine (e.g., Microsoft Text Analytics API) analyzes the user's emotion in the response message and adjusts the content of the response message as necessary.

[1333] What it does: Emotion-sensing algorithms analyze the user's tone and emotions and adjust the content and tone of the response message accordingly.

[1334] Input: The generated response message.

[1335] Output: The adjusted response message.

[1336] Step 8:

[1337] The user takes appropriate action based on the information received.

[1338] Specific actions: Read the information provided in the messaging app and follow the instructions to change your smartphone settings or check your plan.

[1339] Input: The response message received from the server.

[1340] Output: User action (performing a procedure, resolving a problem, checking pricing plans, etc.).

[1341] (Application example 2)

[1342] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1343] In conventional inquiry systems using messaging applications, it was difficult for users to receive appropriate responses when requesting confirmation of operation procedures or problem resolution. Furthermore, responses did not take into account the user's emotions, resulting in insufficient support for users who felt particularly confused or dissatisfied. This resulted in a poor user experience and a decrease in system usage. Furthermore, in content distribution services, real-time support for the content being viewed was insufficient, making it difficult to improve user satisfaction.

[1344] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1345] In this invention, the server includes: means for sending a query to a user to confirm operation methods through a message application; means for receiving the query from the user and analyzing the query using a natural language processing model; means for retrieving appropriate manual information from a database based on the analysis result; means for generating a response message based on the retrieved manual information; means for adjusting the content of the response message using an emotion engine that analyzes the user's emotions; means for sending the response message to the user; and means for responding in real time to inquiries about content being viewed as part of a content distribution service. This enables users to receive prompt and appropriate support and further allows them to receive flexible responses based on their emotions, which is expected to significantly improve the user experience and increase the frequency of system use.

[1346] A "messaging application" is software that allows users to send and receive messages to and from a server.

[1347] A "server" is a computer system that receives user queries and images, retrieves information from a database, and generates and sends response messages.

[1348] A "natural language processing model" is a machine learning algorithm that analyzes text data and understands its meaning and intent.

[1349] "Manual information" refers to documents or data that contain detailed explanations of operation and setting methods.

[1350] A "response message" is text data that is generated by the server in response to a user's inquiry.

[1351] An "emotion engine" is a technology that analyzes emotions from a user's text message and adjusts the content of the response message accordingly.

[1352] "Image analysis technology" is a technology for analyzing received image data and extracting important information.

[1353] "Usage data" is data about a user's activity and usage history.

[1354] A "generative AI model" is an artificial intelligence model that makes predictions and recommendations based on collected data.

[1355] A "content distribution service" is a platform and service for providing users with digital content such as videos, music, and e-books.

[1356] This invention is a system that allows users to quickly and appropriately receive operation instructions, image advice, and information on pricing plans through a messaging application. This system also has the ability to analyze the user's emotions and provide responses according to them.

[1357] The system is configured as follows:

[1358] 1. User Interface

[1359] Users use the messaging application to ask for instructions, send images, or request information about pricing plans.

[1360] 2. Server Functions

[1361] The server responds to user inquiries using various modules, specifically with the following functions:

[1362] Natural Language Processing Model

[1363] The server analyzes the user's text query using a natural language processing model, which is used to understand the user's intent and generate an appropriate query.

[1364] Database Access

[1365] Based on the analysis results, the server retrieves relevant manual information and usage data from the database.

[1366] Response message generation

[1367] The server generates a response message for the user based on the acquired information, and also uses an emotion engine to take the user's emotions into account.

[1368] Emotion Engine

[1369] The server uses an emotion engine to analyze the user's emotions and adjusts the content of the response message, adding more detailed explanations or encouraging words if the user is confused.

[1370] 3. Image analysis function

[1371] When a user submits an image, the server uses image analysis technology to analyze the image and retrieve relevant information.

[1372] 4. Optimization function

[1373] The server uses a generative AI model to recommend the optimal pricing plan based on the user's usage data.

[1374] Hardware and software used

[1375] Server: High-performance computer

[1376] Natural Language Processing models: Machine learning algorithms such as BERT

[1377] Database: SQL or NoSQL database

[1378] Sentiment Engine: Dedicated sentiment analysis software

[1379] Image analysis technology: OpenCV, etc.

[1380] Generative AI models: GPT-3, etc.

[1381] Specific examples

[1382] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives the message, analyzes it using a natural language processing model, and retrieves the query "how to back up" from the database. After constructing a response message, it analyzes the user's emotions using an emotion engine, adjusts the content as necessary, and sends the response message.

[1383] Prompt Sentence Examples

[1384] When a user sends a message asking, "What is the best plan based on this month's usage?", the server analyzes it using a natural language processing model, retrieves the user's usage data from a database, analyzes it using a generative AI model, calculates the optimal pricing plan, and sends a response message adjusted by an emotion engine to the user.

[1385] This allows users to receive prompt and appropriate support, improving the user experience.

[1386] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1387] Step 1:

[1388] A user sends an inquiry through a messaging application, asking for confirmation of operation instructions, sending an image, or requesting information on a pricing plan. The input is a text message or image data from the user, which is sent to the server. The output is the data received by the server.

[1389] Step 2:

[1390] The server receives the user's inquiry and image and analyzes the text message using a natural language processing model. The input is the user's inquiry message, and based on this input, a natural language processing model (e.g., BERT) is used to analyze and extract meaning and intent. Specifically, the message is input into the model, and the intent and related keywords are obtained. The output is the analysis result (intent or query).

[1391] Step 3:

[1392] Based on the analysis results, the server retrieves related manual information and user usage data from the database. The input is the analysis result from step 2, and based on this analysis result, it queries the appropriate information from an SQL or NoSQL database. Specifically, it executes a database query and retrieves the relevant data. The output is the retrieved manual information and usage data.

[1393] Step 4:

[1394] The server uses image analysis technology to analyze the image received from the user. The input is the image data sent by the user, which is analyzed using image analysis software such as OpenCV. Specifically, the image is input and information related to the operation method is extracted. The output is the analysis result.

[1395] Step 5:

[1396] The server uses a generative AI model to calculate the optimal pricing plan based on the user's usage data. The input is the user's usage data, which is then input into the model to generate the optimal plan. Specifically, the generative AI model (e.g., GPT-3) makes predictions based on the usage data and calculates the pricing plan. The output is details of the calculated pricing plan.

[1397] Step 6:

[1398] The server generates a response message based on the acquired manual information and analysis results. The input is the manual information and analysis results, and a response message is generated for the user based on this. Specifically, the required information is compiled as text. The output is the generated response message.

[1399] Step 7:

[1400] The server adjusts the content of the response message using an emotion engine that analyzes the user's emotion. The input is the user's message and the generated response message, which are analyzed by the emotion engine. Specifically, the emotion in the message is analyzed and the tone and content of the response message are adjusted accordingly. The output is the adjusted response message.

[1401] Step 8:

[1402] The server sends the adjusted response message to the user, the input is the adjusted response message, and the server performs an operation to send it to the user's terminal, and the output is the response message displayed on the user's terminal.

[1403] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1404] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1405] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1406] [Fourth embodiment]

[1407] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1408] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1409] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1410] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1411] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1412] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1413] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1414] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1415] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1416] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1417] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1418] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1419] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1420] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, advice using images, and information on rate plans through a message application.

[1421] 1. Checking the operation method

[1422] When a user uses a messaging application to make a query such as "How do I back up my new smartphone?", the server receives the message. The server uses a natural language processing model to analyze the user's message and extract queries related to "how to back up." The server retrieves appropriate manual information from a database based on this query. Based on the retrieved manual information, the server generates a response message for the user and sends it to the user's device. The user then backs up their smartphone based on the received information.

[1423] 2. Photo advice

[1424] When a user sends a screenshot of the part of the operation they are having trouble with via a messaging application, the server receives the image. The server then analyzes the image using image analysis technology to identify the relevant operating procedure. Based on the analysis results, the server retrieves relevant manual information from a database and generates a response message including specific operating procedures and reference images. The generated response message is then sent from the server to the user's device. The user then refers to the received information and continues operating the device.

[1425] 3. Providing information on pricing plans

[1426] When a user sends a query requesting information about pricing plans through a messaging application, the server receives the message. The server uses a natural language processing model to analyze the user message and extract queries related to "pricing plans." The server then retrieves the user's usage data from a database. The server uses a generative AI model to calculate the optimal pricing plan based on the retrieved usage data. Based on the calculated pricing plan details, the server generates a response message for the user and sends it to the user's device. The user compares and considers the proposed pricing plans based on the received information.

[1427] Specific examples

[1428] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives and analyzes the message, then retrieves the relevant manual information from the database. Based on the retrieved information, the server generates and sends a response message containing concise and easy-to-understand backup instructions. The user can then use the received information to successfully back up their smartphone.

[1429] Furthermore, if a user has trouble with the smartphone's settings screen, they can send a screenshot. The server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant manual information from a database and sends the user a response message containing detailed instructions and reference images. This allows the user to understand the specific operation method and solve the problem.

[1430] When providing information on pricing plans, if the user sends a message saying "Tell me the best pricing plan," the server will analyze the user's usage data, calculate the best pricing plan, and propose it to them. The user can then check the details of the plans provided and select the one that best suits them.

[1431] The processing flow will be explained below.

[1432] Check operation method

[1433] Step 1:

[1434] A user uses a messaging application to send a message saying, "How do I back up my new phone?"

[1435] Step 2:

[1436] A server receives a message from the user.

[1437] Step 3:

[1438] The server analyzes the user message using a natural language processing model and extracts the query "How to back up."

[1439] Step 4:

[1440] The server retrieves relevant manual information from a database based on the query.

[1441] Step 5:

[1442] The server generates a response message based on the acquired manual information.

[1443] Step 6:

[1444] The server generates a response message and sends it to the user's terminal.

[1445] Step 7:

[1446] The smartphone is backed up based on the response message received by the user.

[1447] Photo-based advice

[1448] Step 1:

[1449] The user sends a screenshot of the part of the operation they are having trouble with via a messaging application.

[1450] Step 2:

[1451] A server receives the image from the user.

[1452] Step 3:

[1453] The server uses image analysis technology to analyze the received images and identify the problem areas.

[1454] Step 4:

[1455] The server retrieves related manual information from a database based on the analysis results.

[1456] Step 5:

[1457] The server generates a response message based on the acquired manual information and reference image.

[1458] Step 6:

[1459] The server generates a response message and sends it to the user's terminal.

[1460] Step 7:

[1461] The user continues the operation while referring to the response message received.

[1462] Providing information on pricing plans

[1463] Step 1:

[1464] A user uses a messaging app to send a message saying, "What's the best rate plan?"

[1465] Step 2:

[1466] A server receives a message from the user.

[1467] Step 3:

[1468] The server analyzes the user message using a natural language processing model and extracts the query "price plan."

[1469] Step 4:

[1470] The server retrieves the user usage data from the database based on the query.

[1471] Step 5:

[1472] The server uses the generated AI model to calculate the optimal pricing plan based on the acquired usage data.

[1473] Step 6:

[1474] The server generates a response message based on the calculated details of the pricing plan.

[1475] Step 7:

[1476] The server generates a response message and sends it to the user's terminal.

[1477] Step 8:

[1478] Based on the response message received by the user, the user compares and considers the proposed pricing plans.

[1479] Example 1

[1480] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1481] The challenge is to effectively utilize messaging applications, a modern means of communication, to quickly and appropriately confirm operation procedures, resolve problems, and recommend optimal pricing plans. Conventional systems require users to search multiple sources of information and obtain the necessary information on their own, which can be a cumbersome process. In addition, selecting a pricing plan requires users to perform complex calculations, making it difficult to choose the optimal plan.

[1482] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1483] In this invention, the server includes: means for sending an inquiry for the user to confirm an operation method through a message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain appropriate information from a database based on the analysis result; means for the server to generate a response message based on the obtained information; means for the server to send the response message to the user; means for the user to perform an operation based on the response message displayed on the terminal; means for the user to send an image through the message application; means for the server to receive an image from the user and analyze it using image analysis technology; means for the server to obtain related information from a database based on the analysis result; means for the server to generate a response message based on the obtained information and a reference image; means for the server to send the response message to the user; means for the user to perform an operation based on the response message displayed on the terminal; means for the user to send an inquiry requesting information on a rate plan through the message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain usage data of the user from a database; and means for the server to calculate an optimal rate plan based on the obtained usage data. The system includes a means for the server to generate a response message based on the calculated details of the pricing plan, a means for the server to send the response message to the user, and a means for the user to consider plans based on the response message displayed on the terminal, thereby enabling the user to quickly and appropriately obtain the necessary information, efficiently solve the problems they are facing, and select the most suitable pricing plan without going through complicated processes.

[1484] "User" refers to an individual or corporation that uses the system.

[1485] "Messaging application" refers to software or applications for sending and receiving information such as text messages and images.

[1486] An "inquiry" refers to a question or request made by a user to the system.

[1487] "Server" refers to a computer system that receives and processes requests from users.

[1488] A "natural language processing model" refers to algorithms and tools for analyzing human language.

[1489] "Analysis" refers to the act of understanding and processing received data using natural language processing models and image analysis techniques.

[1490] A "database" refers to a system for organizing and storing data and efficiently retrieving necessary information.

[1491] "Manual information" refers to documents and data that explain operation methods and procedures.

[1492] A "response message" refers to a message that a server generates and sends in response to a user's query.

[1493] "Image analysis technology" refers to technology for analyzing image data and understanding its contents.

[1494] A "reference image" refers to a reference image provided for the purpose of explaining or instructing a user.

[1495] "Usage status data" refers to data related to a user's service usage status and history.

[1496] "Price plan" refers to various plans that determine the fee structure for the services that a user uses.

[1497] The "optimal pricing plan" refers to the pricing plan that is most profitable for the user based on the user's usage data.

[1498] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a message application. This system performs various data processing and data calculations using the following hardware and software.

[1499] 1. Hardware and Software Used

[1500] Server: A high-performance server computer is required. In this system, it stores the database and AI models and performs calculations.

[1501] Device: A mobile device operated by a user, such as a smartphone or tablet, that must have a messaging application installed.

[1502] Message application: Software for sending and receiving messages between users and servers.

[1503] Natural language processing model: A model for analyzing messages from users, such as BERT or OpenAI's GPT-3.

[1504] Database: This stores manual information and user usage data. For example, a database management system such as MySQL is used.

[1505] Image analysis technology: Image analysis tools such as Google Cloud Vision API are used.

[1506] Generative AI model: For example, OpenAI's GPT-3 is used as an AI model to calculate the optimal pricing plan based on usage data.

[1507] 2. Implementing the Invention

[1508] Check operation method

[1509] A user uses a messaging application to send a message saying, "Tell me how to back up my new smartphone." The server receives this message and analyzes it using a natural language processing model (e.g., BERT) to extract queries related to "how to back up." Based on the extracted query, the server retrieves the appropriate manual information from a MySQL database, generates a response message, and sends it to the user's device. The user then backs up their smartphone based on the received information.

[1510] Photo-based advice

[1511] The user sends a screenshot of the part of the operation where they are having trouble to the server via a messaging application. The server receives the image and analyzes it using the Google Cloud Vision API. Based on the analysis results, the server retrieves relevant manual information from a MySQL database, generates a response message containing specific operating procedures and reference images, and sends it to the user's device. The user can continue operating the device by referring to the received information.

[1512] Providing information on pricing plans

[1513] The user sends a message via a messaging application saying, "What is the best pricing plan?" The server receives this message, analyzes it using a natural language processing model (for example, OpenAI's GPT-3), and extracts queries related to "pricing plans." The server retrieves the user's usage data from a MySQL database and calculates the best pricing plan using GPT-3 based on the retrieved data. A response message is generated based on the details of the calculated pricing plan and sent to the user's device. The user then checks the details of the provided plans and selects the most suitable plan.

[1514] Specific examples

[1515] Check operation method

[1516] Example user question: "How do I back up my new phone?"

[1517] Example prompt: "Generate a message to return when a user is asked how to back up their new phone."

[1518] Photo-based advice

[1519] Example user question: "Send a screenshot of your smartphone's settings screen"

[1520] Example prompt: "Generate a response message when a user sends you a screenshot of their settings screen."

[1521] Providing information on pricing plans

[1522] Example user question: "What is the best pricing plan?"

[1523] Example prompt: "Generate a response message when a user asks what the best pricing plan is."

[1524] This allows users to quickly and appropriately obtain the necessary information, efficiently resolve problems, and select the most suitable rate plan without having to go through complicated processes.

[1525] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1526] Check operation method

[1527] Step 1:

[1528] A query is sent to the user to confirm the operation method.

[1529] Type: A user uses the Messages app to type the text message "How do I back up my new phone?"

[1530] Output: The message is sent to the server.

[1531] Step 2:

[1532] A server receives the user query and analyzes it using a natural language processing model.

[1533] Input: The server receives a message sent by the user.

[1534] Data processing: The server analyzes the message using a natural language processing model (e.g., BERT) and extracts keywords related to "backup methods."

[1535] Output: Queries related to "Backup Method" are generated.

[1536] Step 3:

[1537] The server retrieves the appropriate information from the database.

[1538] Input: Queries related to "how to back up".

[1539] Data processing: The server searches and retrieves the relevant manual information from the MySQL database.

[1540] Output: The retrieved manual information.

[1541] Step 4:

[1542] The server generates a response message based on the acquired information.

[1543] Input: The retrieved manual information.

[1544] Data processing: Based on the obtained manual information, the server generates a response message in a format that is easy for the user to understand.

[1545] Output: A response message is generated.

[1546] Step 5:

[1547] The server sends the response message to the user.

[1548] Input: The response message.

[1549] Output: A reply message is sent to the user's device via the Messages application.

[1550] Step 6:

[1551] The user performs an operation based on the response message displayed on the terminal.

[1552] Input: The response message.

[1553] What happens: The user follows the instructions in the response message to start backing up their smartphone.

[1554] Output: Your phone is backed up.

[1555] Photo-based advice

[1556] Step 1:

[1557] A user sends an image through a messaging application.

[1558] Input: The user attaches a screenshot of the problem and sends it via messaging.

[1559] Output: The screenshot is sent to the server.

[1560] Step 2:

[1561] A server receives the image from the user and analyzes it using image analysis techniques.

[1562] Input: A screenshot submitted by the user.

[1563] Data processing: The server uses the Google Cloud Vision API to analyze the screenshots and identify specific problem areas and operational steps.

[1564] Output: Analysis results.

[1565] Step 3:

[1566] The server retrieves the relevant information from the database.

[1567] Input: Analysis results.

[1568] Data processing: The server searches and retrieves the relevant manual information and reference images from the MySQL database.

[1569] Output: Retrieved information and reference images.

[1570] Step 4:

[1571] The server generates a response message based on the acquired information and the reference image.

[1572] Input: Retrieved information and reference image.

[1573] Data processing: Based on this information, the server generates a response message in a format that is easy for the user to understand.

[1574] Output: A response message is generated.

[1575] Step 5:

[1576] The server sends the response message to the user.

[1577] Input: The response message.

[1578] Output: A reply message is sent to the user's device via the Messages application.

[1579] Step 6:

[1580] The user performs the operation by referring to the response message displayed on the terminal.

[1581] Input: The response message.

[1582] Specific actions: The user follows the response message and uses the displayed steps and reference images to solve the problem.

[1583] Output: Problem solved.

[1584] Providing information on pricing plans

[1585] Step 1:

[1586] A user sends a request for pricing plan information through a messaging application.

[1587] Input: A user sends a message asking, "What is the best pricing plan?"

[1588] Output: The query is sent to the server.

[1589] Step 2:

[1590] A server receives the user query and analyzes it using a natural language processing model.

[1591] Input: A query message from the user.

[1592] Data processing: The server analyzes the message using a natural language processing model such as OpenAI's GPT-3 to extract queries related to "price plans."

[1593] Output: Queries related to "price plan" are generated.

[1594] Step 3:

[1595] The server retrieves user usage data from the database.

[1596] Input: Queries related to "price plans".

[1597] Data processing: The server retrieves user usage data from the MySQL database.

[1598] Output: The captured usage data.

[1599] Step 4:

[1600] The optimal pricing plan is calculated based on the usage data acquired by the server.

[1601] Input: Captured usage data.

[1602] Data processing: The server uses a generative AI model (e.g., OpenAI's GPT-3) to calculate the optimal pricing plan.

[1603] Output: Best pricing plan.

[1604] Step 5:

[1605] The server generates a response message based on the calculated details of the rate plan.

[1606] Input: Your best rate plan information.

[1607] Data processing: The server generates a response message in a format that is easy for the user to understand.

[1608] Output: A response message is generated.

[1609] Step 6:

[1610] The server sends the response message to the user.

[1611] Input: The response message.

[1612] Output: A reply message is sent to the user's device via the Messages application.

[1613] Step 7:

[1614] The user considers a plan based on the response message displayed on the terminal.

[1615] Input: The response message.

[1616] Specific behavior: The user reviews the details of the offered pricing plans and selects the most suitable plan.

[1617] Output: The user selects the best pricing plan.

[1618] (Application example 1)

[1619] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1620] Modern users face numerous configuration and security issues related to digital devices and security measures. While a support system is needed to quickly and appropriately resolve these issues, conventional methods often require complex operation procedures and responses to anomaly detection alerts, making them difficult for users to understand. While it is also important to propose optimal security and pricing plans, there is a lack of systems that can effectively accomplish this. The objective of this invention is to solve these problems by providing a system that allows users to quickly and appropriately confirm operation procedures, respond to anomaly detection, and receive recommendations for optimal plans.

[1621] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1622] In this invention, the server includes: means for a user to send an inquiry about operation instructions or how to respond to an anomaly detection alert via a message application; means for the server to receive the inquiry from the user and analyze it using a natural language processing model; means for the server to obtain appropriate manual information or response information from a database based on the analysis results; means for the server to generate a response message based on the obtained information; means for the server to send the response message to the user; means for the server to receive an anomaly detection alert image from the user and analyze it using image analysis technology; and means for the server to obtain manual information about the cause of the anomaly and how to deal with it based on the analysis results. This allows the user to easily and quickly confirm operation instructions, respond to an anomaly detection, and receive suggestions for optimal security and pricing plans.

[1623] "User" refers to an individual or company that uses the system or service.

[1624] A "messaging application" is software for sending and receiving text messages and images.

[1625] "Method of operation" refers to the steps or techniques for using a digital device or software.

[1626] An "anomaly detection alert" is a warning message that notifies the user when the system detects an anomaly.

[1627] A "query" is a message sent by a user to the system requesting information or instructions.

[1628] A "server" is a computer system that receives and analyzes messages from users and provides the necessary information.

[1629] A "natural language processing model" refers to algorithms and technologies used to analyze and understand the meaning of text messages entered by users.

[1630] "Analysis results" are the results of analyzing data obtained using natural language processing models and image analysis technology.

[1631] "Manual information" refers to materials that include specific information required by users, such as operation methods and procedures for responding to abnormality detection.

[1632] A "response message" is a message containing a response generated by a server in response to a user's inquiry.

[1633] "Image analysis technology" is a technology for processing transmitted images and analyzing their contents.

[1634] A "reference image" is an image used to explain operating procedures and how to respond to abnormalities.

[1635] "Price Plan" refers to the pricing structure or plan that a User can select to use the Service.

[1636] "Usage status data" refers to data that indicates a user's service usage history and trends.

[1637] "Security Plan" refers to a combination of security measures designed to protect a user's devices and data.

[1638] A "generative AI model" is an artificial intelligence model that generates new information and responses based on data.

[1639] A "prompt" refers to an instruction or question that is input to a model.

[1640] The present invention provides a system that allows a user to quickly and appropriately check operation instructions and respond to an anomaly detection alert through a message application. Specific embodiments of this system will be described below.

[1641] First, when a user wants to check how to operate a device or how to respond to an anomaly detection alert, they send an inquiry from their device via a messaging application. Here, "user" refers to individuals or companies that use the system or service. "Messaging application" is software for sending and receiving text messages and images. "Operation method" refers to the steps and methods for using digital devices and software.

[1642] The server receives the user's inquiry and analyzes it using a natural language processing model (software used: OpenAI GPT-3). Based on the analysis results, the server retrieves "manual information" from a database (software used: MySQL, PostgreSQL). A "natural language processing model" refers to the algorithms and technologies used to analyze and understand the meaning of text messages entered by users. The "database" contains detailed manual information on operating procedures and how to respond to abnormalities.

[1643] Next, the server generates a "response message" based on the retrieved manual information and sends it to the user's terminal. A "response message" is a message that the server generates containing a response to a user's inquiry. For example, if a user sends a message saying "Please tell me how to change the firewall settings," the server generates a response message based on the relevant manual information and sends it to the user.

[1644] We will also explain the case where a user sends an anomaly detection alert with a screenshot attached. The image sent from the user's device is analyzed on the server using "image analysis technology" (software used: OpenCV, TensorFlow). Based on the analysis results, the server retrieves manual information on the "cause of the anomaly" and its "measures" from the database and sends it to the user as a response message.

[1645] Finally, if a user requests information on the optimal pricing plan or security plan, the optimal plan is calculated using the user's usage data and provided to the user as a response message. "Price plan" refers to the pricing structure or plan that a user can select to use the service. "Security plan" refers to a combination of security measures to protect the user's device and data.

[1646] As a concrete example, consider the case where a user sends a message saying, "What is the best security plan for me?" In this case, the "prompt sentence" would be as follows:

[1647] A user sends a message saying, "What security plan is best for me?"

[1648] The server analyzes this message, obtains the user's usage data, calculates the optimal security plan, and generates a response message based on this information.

[1649] As described above, the present invention enables a user to easily and quickly check operation methods, respond to detected abnormalities, and receive proposals for optimal security and fee plans.

[1650] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1651] Step 1:

[1652] A user sends an inquiry about operation methods or how to respond to an anomaly detection alert through a messaging application. In this case, for example, the input is a text message such as "Please tell me how to change the firewall settings." This message is sent to the server. The input is the user's message, and the output is the message sent to the server.

[1653] Step 2:

[1654] The server receives the query from the user. The server takes the message sent from the messaging application and analyzes the text message using a natural language processing model (software used: OpenAI GPT-3). Specifically, the server extracts meaning from the user's message and forms a query such as "How do I change my firewall settings?" The input is the user's message, and the output is the analyzed query.

[1655] Step 3:

[1656] The server retrieves appropriate manual information from a database (software used: MySQL, PostgreSQL) based on the analysis results. The server creates a database query using the analyzed query and searches for related manual information. Specifically, it retrieves manual information containing procedures and explanations on "how to change firewall settings" from the database. The input is the analyzed query, and the output is the retrieved manual information.

[1657] Step 4:

[1658] The server generates a response message based on the acquired information. Based on the acquired manual information, the server assembles the response message in a format that is easy for the user to understand. Specifically, it organizes the procedures by item and creates a message with supplementary explanations. It uses a generative AI model to generate a response message in natural language. The input is the acquired manual information, and the output is the generated response message.

[1659] Step 5:

[1660] The server sends the response message to the user. The server generates a response message and sends it to the user's terminal again through the message application. The user can then view the received message and take action based on its content. The input is the generated response message, and the output is the message sent to the user's terminal.

[1661] Step 6:

[1662] The user sends a screenshot of the anomaly detection alert through a messaging application. For example, the image is sent along with a text message such as "I don't understand the content of the screenshot below." The input is the screenshot image and message from the user, and the output is the data sent to the server.

[1663] Step 7:

[1664] The server receives the image from the user and analyzes it using image analysis technology. The server captures the screenshot image sent and identifies any abnormalities using image analysis software (software used: OpenCV, TensorFlow). Specifically, it extracts error messages and warnings from the screen and analyzes the related information. The input is the screenshot image, and the output is the analyzed abnormalities and their contents.

[1665] Step 8:

[1666] The server retrieves manual information relating to the cause of the anomaly and how to deal with it from a database based on the analysis results. The server creates a database query based on the analyzed anomaly location and searches for related manual information. Specifically, it retrieves manual information including procedures and solutions for the identified anomaly from the database. The input is the analyzed anomaly location, and the output is the retrieved manual information.

[1667] Step 9:

[1668] The server generates a response message based on the acquired information and sends it to the user. The server creates a response message that clearly explains the cause of the abnormality and how to deal with it. The input is the acquired manual information, and the output is the generated response message and its transmission to the user's terminal.

[1669] Example prompt sentence:

[1670] A user sent a message asking, "How do I change my firewall settings?"

[1671] A user has sent a screenshot. Please analyze the screenshot and provide appropriate advice.

[1672] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1673] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a message application, and also aims to recognize the user's emotions and provide a response that corresponds to those emotions.

[1674] 1. Checking the operation method

[1675] When a user uses a messaging application to make a query such as "How do I back up my new smartphone?", the server receives the message. The server uses a natural language processing model to analyze the user's message and extract queries related to "how to back up." The server then retrieves appropriate manual information from a database based on this query. Based on the retrieved manual information, the server generates a response message for the user and sends it to the user's device. Next, the emotion engine analyzes the user's emotion in the response message and adjusts the content of the response message according to the user's emotion. The user then backs up their smartphone based on the received information.

[1676] 2. Photo advice

[1677] When a user sends a screenshot of a part of the operation where they are having trouble via a messaging application, the server receives the image. The server analyzes the image using image analysis technology and identifies the relevant operating procedure. Based on the analysis results, the server retrieves relevant manual information from a database and generates a response message including specific operating procedures and reference images. The generated response message is then sent from the server to the user's device. The emotion engine analyzes the user's emotion in the response message and adjusts the content of the response message according to the user's emotion. The user then continues operating the device, referring to the received information.

[1678] 3. Providing information on pricing plans

[1679] When a user sends a query requesting information about pricing plans through a messaging application, the server receives the message. The server uses a natural language processing model to analyze the user message and extract queries related to "pricing plans." The server then retrieves the user's usage data from a database. The server uses a generative AI model to calculate the optimal pricing plan based on the retrieved usage data. Based on the calculated pricing plan details, the server generates a response message for the user and sends it to the user's device. The emotion engine analyzes the user's emotions in the response message and adjusts the content of the response message according to the user's emotions. The user compares and considers the proposed pricing plans based on the received information.

[1680] Specific examples

[1681] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives and analyzes the message and retrieves relevant manual information from the database. The server then generates a response message with concise and easy-to-understand backup instructions based on the retrieved information. The emotion engine analyzes this response message and, if it determines that the user is confused, adjusts it by adding more detailed explanations and supplementary information. The user can then use the received information to successfully back up their smartphone.

[1682] If a user has trouble with the smartphone's settings screen, they can send a screenshot. The server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant manual information from a database and generates a response message including detailed instructions and reference images. If the emotion engine analyzes the response message and determines that the user is frustrated, it can soften the tone of the response message and add words of encouragement. This allows the user to understand the specific operation method and solve the problem.

[1683] When providing pricing plan information, the user sends a message asking, "What is the best plan based on this month's usage?" The server obtains the user's usage data and calculates the optimal pricing plan based on that data. After compiling the obtained pricing plan details into a response message, the emotion engine analyzes the user's emotions in the response message and, if the user is dissatisfied or has questions about the plan, provides additional information or support options to alleviate those emotions. The user can review the provided plan details and select the plan that best suits them.

[1684] The processing flow will be explained below.

[1685] Check operation method

[1686] Step 1:

[1687] A user uses a messaging application to send a message saying, "How do I back up my new phone?"

[1688] Step 2:

[1689] A server receives a message from the user.

[1690] Step 3:

[1691] The server analyzes the user message using a natural language processing model and extracts the query "How to back up."

[1692] Step 4:

[1693] The server retrieves relevant manual information from a database based on the query.

[1694] Step 5:

[1695] The server generates a response message based on the acquired manual information.

[1696] Step 6:

[1697] The server generates a response message and sends it to the user's terminal.

[1698] Step 7:

[1699] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[1700] Step 8:

[1701] The smartphone is backed up based on the response message received by the user.

[1702] Photo-based advice

[1703] Step 1:

[1704] The user sends a screenshot of the part of the operation they are having trouble with via a messaging application.

[1705] Step 2:

[1706] A server receives the image from the user.

[1707] Step 3:

[1708] The server uses image analysis technology to analyze the received images and identify the problem areas.

[1709] Step 4:

[1710] The server retrieves related manual information from a database based on the analysis results.

[1711] Step 5:

[1712] The server generates a response message based on the acquired manual information and reference image.

[1713] Step 6:

[1714] The server generates a response message and sends it to the user's terminal.

[1715] Step 7:

[1716] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[1717] Step 8:

[1718] The user continues the operation while referring to the response message received.

[1719] Providing information on pricing plans

[1720] Step 1:

[1721] A user uses a messaging app to send a message saying, "What's the best rate plan?"

[1722] Step 2:

[1723] A server receives a message from the user.

[1724] Step 3:

[1725] The server analyzes the user message using a natural language processing model and extracts the query "price plan."

[1726] Step 4:

[1727] The server retrieves the user usage data from the database based on the query.

[1728] Step 5:

[1729] The server uses the generated AI model to calculate the optimal pricing plan based on the acquired usage data.

[1730] Step 6:

[1731] The server generates a response message based on the calculated details of the pricing plan.

[1732] Step 7:

[1733] The server generates a response message and sends it to the user's terminal.

[1734] Step 8:

[1735] An emotion engine analyzes the response message and adjusts the content and tone of the message depending on the user's emotion.

[1736] Step 9:

[1737] Based on the response message received by the user, the user compares and considers the proposed pricing plans.

[1738] Example 2

[1739] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1740] Today's users need to quickly and accurately obtain various information and operate digital devices without hesitation. However, if the information they need, such as confirmation of operation instructions, image-based support, or selection of pricing plans, is not provided promptly, it can cause stress and waste time. Furthermore, if appropriate responses are not provided based on the user's emotions, the user experience can be significantly impaired. Therefore, there is a need to develop a system that can recognize user emotions and provide appropriate responses accordingly.

[1741] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for sending an inquiry for a user to confirm an operation method through a communication application; means for receiving the inquiry from the user and analyzing it using a natural language processing engine; means for acquiring appropriate command information from a data storage means based on the analysis result; means for generating a response message based on the acquired command information; means for sending the response message to the user; and means for analyzing the user's emotion in the response message using an emotion recognition engine and adjusting the content. This allows the user to efficiently acquire information through the communication application and receive an appropriate response that matches their emotion.

[1742] A "user" is someone who uses this system to inquire about information, send images, or check pricing plans.

[1743] A "communication application" is software that allows users to communicate with a server in two-way communication.

[1744] A "server" is a central processing unit that receives inquiries and images from users and analyzes and responds.

[1745] A "natural language processing engine" is a software component that analyzes and understands the meaning of a user's text messages.

[1746] "Command information" refers to instructions and explanations regarding operation methods and procedures, and information desired by the user.

[1747] "Data storage means" refers to a database or storage system that stores information such as manual information and usage data.

[1748] A "response message" is a message that is generated based on the analysis results and command information and sent to the user.

[1749] An "emotion recognition engine" is a software component that analyzes the user's emotions in a response message and adjusts the response content accordingly.

[1750] "Image analysis technology" is a technology for analyzing images sent by users and identifying the information and operation locations contained therein.

[1751] A "generative AI model" is an artificial intelligence model that calculates the optimal pricing plan based on user usage data.

[1752] "Usage Data" refers to the usage history, patterns, and consumption data of the services you use.

[1753] This invention is a system that allows users to quickly and appropriately receive confirmation of operation methods, image advice, and information on pricing plans through a communication application. This system aims to recognize the user's emotions and provide a response that corresponds to those emotions. Specific embodiments of this invention are described below.

[1754] First, the user communicates with the server using a communication application, which provides an interface for sending user queries and images to the server in text or image format.

[1755] The server has an API for receiving inquiries and images from users. The received messages and images are analyzed using a natural language processing engine (e.g., Google NLU) and image analysis technology (e.g., Google Vision API). The natural language processing engine analyzes the user's text message and understands its meaning. Meanwhile, the image analysis technology analyzes the image sent by the user and identifies specific operation points and problems.

[1756] Based on the analysis results, the server retrieves appropriate instruction information from a data storage means (e.g., SQL database), including information such as operating procedures, reference images, and pricing plans. Based on the retrieved information, the server generates a response message and sends it to the user.

[1757] Furthermore, the server is equipped with an emotion recognition engine (e.g., Microsoft Text Analytics API) that analyzes the user's emotions in the generated response message. Depending on the analysis results, the content and tone of the response message are adjusted. This function allows appropriate support to be provided according to the user's situation and emotions.

[1758] As a concrete example, consider a case where a user sends a query through a communication application asking, "How do I back up my new smartphone?" In this case, the server receives and analyzes the message and retrieves relevant instruction information from a database. The server then uses the retrieved information to generate a response message containing concise and easy-to-understand backup instructions. This response message is then analyzed by an emotion recognition engine, and detailed explanations and supplementary information are added as needed. Finally, the user can successfully back up their smartphone using the received information.

[1759] Furthermore, if a user encounters a problem with the smartphone's settings screen, they can send a screenshot via a communication application. In this case, the server receives the image and uses image analysis technology to identify the problem area. The server then retrieves the relevant instruction information from a database and generates a response message containing detailed instructions and reference images. This response message is also analyzed by an emotion recognition engine, and the tone and content are adjusted according to the user's emotions. This allows the user to understand the specific operation method and solve the problem.

[1760] In providing pricing plan information, consider the case where a user asks, "What is the best plan based on my usage this month?" In this case, the server retrieves the user's usage data from a database and calculates the optimal pricing plan using a generative AI model (e.g., OpenAI GPT-3). The obtained pricing plan details are compiled into a response message, analyzed using an emotion recognition engine, and additional information and support options are provided as needed. This allows the user to review the plans offered and select the one that best suits them.

[1761] This system is expected to improve the user experience by allowing users to efficiently obtain information through communication applications and receive appropriate responses that match their emotions.

[1762] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1763] Step 1:

[1764] A user sends an inquiry through a communication application requesting confirmation of an operation method, advice with an image, or information on a rate plan.

[1765] What happens: A user enters and attaches a text message (e.g., "How do I back up my new phone?" or a screenshot) into the input field in a messaging app and hits the send button.

[1766] Input: A query sent by the user in the form of text or an image.

[1767] Output: The query data received by the server from the user.

[1768] Step 2:

[1769] The server receives the inquiry data (text message or image) received through the communication application.

[1770] Specific operation: The server's API receives the query data and adds it to an internal queue for analysis.

[1771] Input: Query data sent from the communication application to the server.

[1772] Output: Query data added to the queue for analysis inside the server.

[1773] Step 3:

[1774] The server uses a natural language processing engine (e.g., Google NLU) to analyze the text message and understand the user's intent.

[1775] What it does: A natural language processing engine analyzes text messages to identify keywords and intent.

[1776] Input: A text message from the user.

[1777] Output: Parsed keywords and user intent.

[1778] Step 4:

[1779] The server uses image analysis technology (e.g., Google Vision API) to analyze the image sent by the user and identify specific operation points and problems.

[1780] What it does: Image analysis technology analyzes image data and identifies the elements it contains.

[1781] Input: Image data sent by the user.

[1782] Output: Information on the analyzed elements in the image and the specific operation locations.

[1783] Step 5:

[1784] The server retrieves the appropriate instruction information (operational procedures, reference images, pricing plans, etc.) from a data storage means (e.g., SQL database).

[1785] Specific operation: An SQL query is executed and the necessary instruction information is retrieved from the database.

[1786] Input: Analysis results from a natural language processing engine or image analysis technology.

[1787] Output: Instruction information retrieved from the database.

[1788] Step 6:

[1789] The server generates a response message based on the acquired information and sends it to the user.

[1790] Specific operation: The text generation engine creates a response message and delivers it to the user's device via the message sending function.

[1791] Input: Instruction information retrieved from the database.

[1792] Output: The generated response message.

[1793] Step 7:

[1794] The server's emotion recognition engine (e.g., Microsoft Text Analytics API) analyzes the user's emotion in the response message and adjusts the content of the response message as necessary.

[1795] What it does: Emotion-sensing algorithms analyze the user's tone and emotions and adjust the content and tone of the response message accordingly.

[1796] Input: The generated response message.

[1797] Output: The adjusted response message.

[1798] Step 8:

[1799] The user takes appropriate action based on the information received.

[1800] Specific actions: Read the information provided in the messaging app and follow the instructions to change your smartphone settings or check your plan.

[1801] Input: The response message received from the server.

[1802] Output: User action (performing a procedure, resolving a problem, checking pricing plans, etc.).

[1803] (Application example 2)

[1804] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1805] In conventional inquiry systems using messaging applications, it was difficult for users to receive appropriate responses when requesting confirmation of operation procedures or problem resolution. Furthermore, responses did not take into account the user's emotions, resulting in insufficient support for users who felt particularly confused or dissatisfied. This resulted in a poor user experience and a decrease in system usage. Furthermore, in content distribution services, real-time support for the content being viewed was insufficient, making it difficult to improve user satisfaction.

[1806] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1807] In this invention, the server includes: means for sending a query to a user to confirm operation methods through a message application; means for receiving the query from the user and analyzing the query using a natural language processing model; means for retrieving appropriate manual information from a database based on the analysis result; means for generating a response message based on the retrieved manual information; means for adjusting the content of the response message using an emotion engine that analyzes the user's emotions; means for sending the response message to the user; and means for responding in real time to inquiries about content being viewed as part of a content distribution service. This enables users to receive prompt and appropriate support and further allows them to receive flexible responses based on their emotions, which is expected to significantly improve the user experience and increase the frequency of system use.

[1808] A "messaging application" is software that allows users to send and receive messages to and from a server.

[1809] A "server" is a computer system that receives user queries and images, retrieves information from a database, and generates and sends response messages.

[1810] A "natural language processing model" is a machine learning algorithm that analyzes text data and understands its meaning and intent.

[1811] "Manual information" refers to documents or data that contain detailed explanations of operation and setting methods.

[1812] A "response message" is text data that is generated by the server in response to a user's inquiry.

[1813] An "emotion engine" is a technology that analyzes emotions from a user's text message and adjusts the content of the response message accordingly.

[1814] "Image analysis technology" is a technology for analyzing received image data and extracting important information.

[1815] "Usage data" is data about a user's activity and usage history.

[1816] A "generative AI model" is an artificial intelligence model that makes predictions and recommendations based on collected data.

[1817] A "content distribution service" is a platform and service for providing users with digital content such as videos, music, and e-books.

[1818] This invention is a system that allows users to quickly and appropriately receive operation instructions, image advice, and information on pricing plans through a messaging application. This system also has the ability to analyze the user's emotions and provide responses according to them.

[1819] The system is configured as follows:

[1820] 1. User Interface

[1821] Users use the messaging application to ask for instructions, send images, or request information about pricing plans.

[1822] 2. Server Functions

[1823] The server responds to user inquiries using various modules, specifically with the following functions:

[1824] Natural Language Processing Model

[1825] The server analyzes the user's text query using a natural language processing model, which is used to understand the user's intent and generate an appropriate query.

[1826] Database Access

[1827] Based on the analysis results, the server retrieves relevant manual information and usage data from the database.

[1828] Response message generation

[1829] The server generates a response message for the user based on the acquired information, and also uses an emotion engine to take the user's emotions into account.

[1830] Emotion Engine

[1831] The server uses an emotion engine to analyze the user's emotions and adjusts the content of the response message, adding more detailed explanations or encouraging words if the user is confused.

[1832] 3. Image analysis function

[1833] When a user submits an image, the server uses image analysis technology to analyze the image and retrieve relevant information.

[1834] 4. Optimization function

[1835] The server uses a generative AI model to recommend the optimal pricing plan based on the user's usage data.

[1836] Hardware and software used

[1837] Server: High-performance computer

[1838] Natural Language Processing models: Machine learning algorithms such as BERT

[1839] Database: SQL or NoSQL database

[1840] Sentiment Engine: Dedicated sentiment analysis software

[1841] Image analysis technology: OpenCV, etc.

[1842] Generative AI models: GPT-3, etc.

[1843] Specific examples

[1844] For example, if a user sends a message saying, "Tell me how to back up my new smartphone," the server receives the message, analyzes it using a natural language processing model, and retrieves the query "how to back up" from the database. After constructing a response message, it analyzes the user's emotions using an emotion engine, adjusts the content as necessary, and sends the response message.

[1845] Prompt Sentence Examples

[1846] When a user sends a message asking, "What is the best plan based on this month's usage?", the server analyzes it using a natural language processing model, retrieves the user's usage data from a database, analyzes it using a generative AI model, calculates the optimal pricing plan, and sends a response message adjusted by an emotion engine to the user.

[1847] This allows users to receive prompt and appropriate support, improving the user experience.

[1848] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1849] Step 1:

[1850] A user sends an inquiry through a messaging application, asking for confirmation of operation instructions, sending an image, or requesting information on a pricing plan. The input is a text message or image data from the user, which is sent to the server. The output is the data received by the server.

[1851] Step 2:

[1852] The server receives the user's inquiry and image and analyzes the text message using a natural language processing model. The input is the user's inquiry message, and based on this input, a natural language processing model (e.g., BERT) is used to analyze and extract meaning and intent. Specifically, the message is input into the model, and the intent and related keywords are obtained. The output is the analysis result (intent or query).

[1853] Step 3:

[1854] Based on the analysis results, the server retrieves related manual information and user usage data from the database. The input is the analysis result from step 2, and based on this analysis result, it queries the appropriate information from an SQL or NoSQL database. Specifically, it executes a database query and retrieves the relevant data. The output is the retrieved manual information and usage data.

[1855] Step 4:

[1856] The server uses image analysis technology to analyze the image received from the user. The input is the image data sent by the user, which is analyzed using image analysis software such as OpenCV. Specifically, the image is input and information related to the operation method is extracted. The output is the analysis result.

[1857] Step 5:

[1858] The server uses a generative AI model to calculate the optimal pricing plan based on the user's usage data. The input is the user's usage data, which is then input into the model to generate the optimal plan. Specifically, the generative AI model (e.g., GPT-3) makes predictions based on the usage data and calculates the pricing plan. The output is details of the calculated pricing plan.

[1859] Step 6:

[1860] The server generates a response message based on the acquired manual information and analysis results. The input is the manual information and analysis results, and a response message is generated for the user based on this. Specifically, the required information is compiled as text. The output is the generated response message.

[1861] Step 7:

[1862] The server adjusts the content of the response message using an emotion engine that analyzes the user's emotion. The input is the user's message and the generated response message, which are analyzed by the emotion engine. Specifically, the emotion in the message is analyzed and the tone and content of the response message are adjusted accordingly. The output is the adjusted response message.

[1863] Step 8:

[1864] The server sends the adjusted response message to the user, the input is the adjusted response message, and the server performs an operation to send it to the user's terminal, and the output is the response message displayed on the user's terminal.

[1865] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1866] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1867] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1868] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1869] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1870] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1871] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1872] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1873] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1874] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1875] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1876] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1877] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1878] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1879] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1880] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1881] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1882] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1883] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1884] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1885] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1886] The following is further disclosed regarding the above embodiment.

[1887] (Claim 1)

[1888] A means for a user to send an inquiry through a message application to confirm an operation method;

[1889] a server receiving a query from the user and analyzing the query using a natural language processing model;

[1890] A server acquires appropriate manual information from a database based on the analysis results;

[1891] a means for generating a response message based on the acquired manual information by the server;

[1892] means for the server to send the response message to the user;

[1893] A system including:

[1894] (Claim 2)

[1895] a means for a user to send an image through a messaging application;

[1896] a server receiving the image from the user and analyzing it using image analysis technology;

[1897] A server acquires related manual information from a database based on the analysis result;

[1898] a means for generating a response message based on the acquired manual information and reference image by the server;

[1899] means for the server to send the response message to the user;

[1900] 10. The system of claim 1, comprising:

[1901] (Claim 3)

[1902] means for a user to send an inquiry requesting information on a pricing plan via a message application;

[1903] a server receiving a query from the user and analyzing the query using a natural language processing model;

[1904] A server acquires the user's usage data from a database;

[1905] A means for the server to calculate an optimal rate plan based on the acquired usage data;

[1906] a means for generating a response message based on the calculated fee plan details by the server;

[1907] means for the server to send the response message to the user;

[1908] 10. The system of claim 1, comprising:

[1909] "Example 1"

[1910] (Claim 1)

[1911] A means for a user to send an inquiry through a message application to confirm an operation method;

[1912] a server receiving a query from the user and analyzing the query using a natural language processing model;

[1913] A means for the server to acquire appropriate information from a database based on the analysis result;

[1914] a means for generating a response message based on the acquired information by the server;

[1915] means for the server to send the response message to the user;

[1916] A means for the user to perform an operation based on the response message displayed on the terminal;

[1917] A system including:

[1918] (Claim 2)

[1919] a means for a user to send an image through a messaging application;

[1920] a server receiving the image from the user and analyzing it using image analysis technology;

[1921] A means for the server to acquire related information from a database based on the analysis result;

[1922] a means for generating a response message based on the acquired information and a reference image by the server;

[1923] means for the server to send the response message to the user;

[1924] A means for the user to perform an operation by referring to a response message displayed on the terminal;

[1925] 10. The system of claim 1, comprising:

[1926] (Claim 3)

[1927] means for a user to send an inquiry requesting information on a pricing plan via a message application;

[1928] a server receiving a query from the user and analyzing the query using a natural language processing model;

[1929] A server acquires the user's usage data from a database;

[1930] A means for the server to calculate an optimal rate plan based on the acquired usage data;

[1931] a means for generating a response message based on the calculated fee plan details by the server;

[1932] means for the server to send the response message to the user;

[1933] A means for the user to consider a plan based on the response message displayed on the terminal;

[1934] 10. The system of claim 1, comprising:

[1935] "Application Example 1"

[1936] (Claim 1)

[1937] A means for users to send inquiries about operation methods and how to respond to abnormality detection alerts through a messaging application;

[1938] a server receiving a query from the user and analyzing the query using a natural language processing model;

[1939] A server acquires appropriate manual information or response information from a database based on the analysis result;

[1940] a means for generating a response message based on the acquired information by the server;

[1941] means for the server to send the response message to the user;

[1942] A server receives an anomaly detection alert image from the user and analyzes it using image analysis technology;

[1943] a means for the server to acquire manual information on the cause of the abnormality and how to deal with it based on the analysis result;

[1944] A system including:

[1945] (Clai...

Claims

1. A means for a user to send an inquiry through a message application to confirm an operation method; a server receiving a query from the user and analyzing the query using a natural language processing model; A server acquires appropriate manual information from a database based on the analysis results; a means for generating a response message based on the acquired manual information by the server; means for the server to send the response message to the user; A system including:

2. a means for a user to send an image through a messaging application; a server receiving the image from the user and analyzing it using image analysis technology; A server acquires related manual information from a database based on the analysis result; a means for generating a response message based on the acquired manual information and reference image by the server; means for the server to send the response message to the user; The system of claim 1 , comprising:

3. means for a user to send an inquiry requesting information on a pricing plan via a message application; a server receiving a query from the user and analyzing the query using a natural language processing model; A server acquires the user's usage data from a database; A means for the server to calculate an optimal rate plan based on the acquired usage data; a means for generating a response message based on the calculated fee plan details by the server; means for the server to send the response message to the user; The system of claim 1 , comprising:

Images