system

Abstract

We provide the system. [Solution] A means for collecting data from the customer's terminal, A means of sending the collected data to the server, A means of analyzing the data received by the server to identify the customer's problem, A means of generating solutions to problems using images, videos, text, and audio, A means of providing the generated solution to the customer's terminal, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional smartphone support systems relied on one-way communication where customers reported smartphone problems and operators solved them. Therefore, when customers received procedures and explanations that were difficult to understand, problem-solving was often delayed. Also, there was an issue that visual understanding was difficult with only text-based support, and the operation hurdle was high especially for users unfamiliar with technology. Against this background, there has been a demand for a method that can immediately and intuitively solve smartphone troubleshooting.

Means for Solving the Problems

[0005] This invention relates to a system that collects data from a customer's device and transmits that data to a server for analysis. The server analyzes the received data to identify the customer's problem. Furthermore, it generates solutions to the problem using images, videos, text, and audio, and provides these solutions to the customer's device, enabling the customer to resolve the problem intuitively and quickly. In particular, the inclusion of error logs and configuration information in the collected data allows for a detailed analysis of the problem's cause. Additionally, the solutions, which include specific operating procedures generated based on the analysis results, make it easier for the customer to visually understand how to actually perform the operation. In this way, the system reduces the time spent troubleshooting smartphone issues and improves customer satisfaction.

[0006] "Terminal" refers to an electronic device used by a user, and in particular, to portable information processing devices such as smartphones and tablets.

[0007] "Server" refers to a computer system that provides services to other computers via a network, and in this invention, it refers to a system that performs data analysis and solution generation.

[0008] "Data" refers to information collected from a device, and specifically includes information such as error logs, settings information, screenshots, and screen recordings.

[0009] "Data collection" refers to the process by which a device monitors user input and system status to collect information according to its purpose.

[0010] "Transmission" refers to the act of sending data from a terminal to a server, and it is desirable that this data transmission be accompanied by security measures such as encryption.

[0011] "Analysis" refers to the process of deciphering data received by a server to identify the cause of a problem.

[0012] "Problems" refer to malfunctions or operational problems that customers encounter on their devices.

[0013] A "solution" refers to specific steps or methods proposed for an identified problem, and this solution is provided in the form of images, videos, text, or audio.

[0014] "Image" refers to a still image containing visual information, and is used to illustrate explanations or procedures.

[0015] "Video" refers to images that contain moving visual information and is used to show operating procedures or provide visual explanations.

[0016] "Text" refers to written information containing characters and is used to provide detailed explanations or operating procedures.

[0017] "Audio" refers to an audio message containing auditory information and is used to provide verbal explanations or instructions.

[0018] "Providing" refers to the act of sending the solution generated by the server to the terminal, making it accessible to the customer. [Brief explanation of the drawing]

[0019] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7]It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0020] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0021] First, the language used in the following description will be explained.

[0022] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0023] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0024] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0025] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0027] [First Embodiment]

[0028] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0029] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0032] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0035] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0039] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0040] overview

[0041] This invention is a system that assists in troubleshooting customers' smartphones. This system collects data from the customer's device, sends the collected data to a server, the server analyzes the received data to identify the problem, generates a solution to that problem, and provides it to the customer's device.

[0042] System operation

[0043] The present invention's system mainly consists of the following elements:

[0044] 1. Terminal:

[0045] A terminal is a portable information processing device such as a smartphone or tablet used by the user. When a user reports a problem, the terminal collects system data and display information (screenshots, screen recordings).

[0046] 2. Server:

[0047] A server is a computer system that receives data transmitted from terminals over a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[0048] System processing flow

[0049] 1. Data Collection

[0050] A user discovers a problem with their smartphone, launches a support app, and reports the issue.

[0051] The device receives user reports and automatically collects relevant system data (error logs, configuration information). Furthermore, it requests permission from the user to take screenshots or screen recordings to collect display information at the time the problem occurred.

[0052] 2. Sending data

[0053] The collected data is encrypted on the device and sent to the server using a secure channel. The system confirms that data transmission is complete and notifies the user of the successful transmission.

[0054] 3. Data Analysis

[0055] The server decompresses and decodes the received data. It verifies the correct format of the received data and analyzes it using a generative AI model. Based on display information and system data, it identifies the cause of the problem.

[0056] 4. Generating support information

[0057] After the cause of the problem is identified, the server generates a solution using images, videos, text, and audio. This solution includes specific steps and precautions.

[0058] 5. Providing support information

[0059] The generated support information is sent from the server to the terminal. The terminal analyzes the received data and provides it to the user through the user interface. The user can then resolve the problem according to the solution.

[0060] Specific example

[0061] Examples of resolving app crashes

[0062] 1. A user discovers a problem where a specific app repeatedly crashes and launches the support app to report the problem.

[0063] 2. The device receives the problem report and collects the error log and screenshots from the crash. The device encrypts the collected data and sends it to the server.

[0064] 3. The server analyzes the received data and identifies that a specific library needs to be updated.

[0065] 4. The server generates videos and text containing library update procedures, providing users with easy-to-understand instructions on how to perform the specific operations.

[0066] 5. The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[0067] In the form described above, the present invention is a system that enables efficient and intuitive troubleshooting of smartphones. This significantly reduces the time and effort required to resolve customer problems, thereby improving customer satisfaction.

[0068] The following describes the processing flow.

[0069] Step 1: Data Collection

[0070] User:

[0071] Users launch the support app when they encounter problems with their smartphone.

[0072] Tap the "Report a Problem" button in the support app and enter a summary of the problem or describe it verbally.

[0073] Terminal:

[0074] Upon receiving user input, the terminal automatically collects relevant system data (error logs and configuration information).

[0075] The device will display a message asking the user for permission to take screenshots or screen recordings.

[0076] If the user grants permission, the device will take screenshots and screen recordings when the problem occurs.

[0077] The collected data is saved on the device as a temporary file.

[0078] Step 2: Send

[0079] Terminal:

[0080] The collected data is encrypted and sent to the server using a secure channel.

[0081] Confirm that data transmission is complete and notify the user of its success.

[0082] Step 3: Data Analysis

[0083] server:

[0084] The received data is decompressed, and its format is verified.

[0085] The generative AI model is launched, and data analysis begins.

[0086] Image recognition technology is used to analyze display information (screenshots and screen recordings) and identify the area where the problem is displayed.

[0087] Analyze system data (error logs and configuration information) to identify the root cause of the problem.

[0088] Step 4: Generating support information

[0089] server:

[0090] Based on the cause of the problem, generate specific solution steps.

[0091] The troubleshooting steps are created in detail using images, videos, text, and audio. For example, a video of the app update procedure, screenshots of the settings change procedure, text instructions, and an audio guide.

[0092] The generated support information is packaged and prepared for transmission to the device.

[0093] Step 5: Providing support information

[0094] Terminal:

[0095] Receive support information sent from the server.

[0096] Verify the integrity of the received data and confirm that there are no problems.

[0097] Support information is displayed or played to the user through the user interface. For example, a video of the troubleshooting procedure or text explanation may be displayed.

[0098] User:

[0099] Follow the displayed solution steps.

[0100] If necessary, watch the provided explanatory videos and listen to the audio guide.

[0101] Confirm that the problem has been resolved and provide feedback through the support app.

[0102] In this way, the system of the present invention supports users in efficiently troubleshooting their smartphones.

[0103] (Example 1)

[0104] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0105] Modern information processing devices offer numerous functions, but the number of problems and issues users face is also increasing. These problems are often complex and difficult to resolve on their own. Furthermore, troubleshooting requires significant time and effort, leading to decreased user satisfaction. This invention aims to provide a system that efficiently and intuitively resolves problems users encounter with information processing devices, thereby reducing the burden on users.

[0106] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0107] In this invention, the server includes means for collecting data from the user's information processing device, means for transmitting the collected data to the processing device, and means for the processing device to analyze the received data and identify a problem. This enables the user to solve complex problems quickly and efficiently.

[0108] A "user" is an individual or group that uses an information processing device.

[0109] An "information processing device" is a device that includes hardware and software for collecting, transmitting, receiving, analyzing, and displaying data.

[0110] "Data" refers to various types of information and records within an information processing device, such as error logs, configuration information, screenshots, and screen recordings.

[0111] A "processing device" is a computer system that receives data, analyzes it, and generates results.

[0112] "Analysis" is the process of understanding the content of received data and identifying the cause of a problem.

[0113] A "solution" is information that includes steps and methods for a user to resolve a problem, based on the cause of the problem.

[0114] An "image" is a still image used to convey information visually.

[0115] A "video" is a dynamic medium that conveys visual information along a time axis.

[0116] "Text" refers to information expressed using characters.

[0117] "Sound" refers to sound signals that transmit information through hearing.

[0118] "To transmit" means to move information or data from one device or system to another.

[0119] "To collect" means to gather and incorporate necessary data and information.

[0120] Modes for carrying out the invention

[0121] overview

[0122] This invention is a system that assists in troubleshooting a user's information processing device (e.g., a smartphone or tablet). The system collects data from the user's device and sends the collected data to a server. The server analyzes the received data, identifies the problem the user is facing, generates a solution to that problem, and provides it to the user's device.

[0123] System Configuration

[0124] This system mainly consists of the following elements:

[0125] 1. User's device:

[0126] This refers to information processing devices such as smartphones and tablets used by users. When a user reports a problem, the device collects system data (error logs, configuration information) and display information (screenshots, screen recordings).

[0127] 2. Server:

[0128] This computer system receives data transmitted from terminals via a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[0129] Hardware and software to be used

[0130] 1. On the device side:

[0131] OS: ANDROID (registered trademark) or iOS

[0132] Log collection tools: "Logcat" for Android, "Console" for iOS

[0133] Cryptographic library: OpenSSL

[0134] Communication protocol: HTTPS

[0135] 2. Server side:

[0136] Data analysis tools: TENSORFLOW® (registered trademark) (image recognition), OpenCV (image analysis)

[0137] Generative AI models: BERT, GPT-3 (registered trademark)

[0138] Video processing tool: FFMPEG

[0139] System processing flow

[0140] Data collection

[0141] When a user discovers a problem with their smartphone, they launch the support app and report the issue. Upon receiving the report, the device automatically collects relevant system data (error logs, configuration information). It also collects screenshots and screen recordings. This allows for a detailed understanding of the user's specific problem.

[0142] Sending data

[0143] The collected data is encrypted on the device and sent to the server using HTTPS. The user is notified upon successful transmission. This ensures secure data transmission and timely communication with the user.

[0144] Data analysis and solution generation

[0145] The server decompresses the received data and parses the JSON formatted data. TensorFlow and OpenCV are used to analyze screenshots and screen recordings, and a generative AI model (such as GPT-3) is used to analyze error logs. This allows for rapid identification of user problems and generation of appropriate solutions.

[0146] Providing solutions

[0147] The generated solutions are sent from the server to the terminal in text, image, video, and audio formats. The server creates content containing detailed steps and notes for the solution, which the terminal displays in its user interface. The user refers to this content and follows the instructions to resolve the problem.

[0148] Specific examples and prompt statements

[0149] Specific example

[0150] For example, suppose a user reports that a particular app is repeatedly crashing. In this case, the device uses Logcat to collect error logs from the crash and also takes screenshots. This data is encrypted with OpenSSL and sent to the server via HTTPS. The server analyzes the received data and uses a generative AI model to identify that "a specific library needs to be updated." Then, it generates a video using FFMPEG explaining the library update procedure and provides it to the user.

[0151] Example of a prompt

[0152] "Please identify the cause of the problem from this error log."

[0153] In this way, this system can efficiently solve user problems by linking terminals and servers. By automating everything from data collection and analysis to solution generation and delivery, it is expected to significantly reduce the burden on users and improve their satisfaction.

[0154] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0155] Step 1:

[0156] Report the problem

[0157] Input: User reports of problems occurring on their smartphone (e.g., app crashes)

[0158] Action: The user launches the support app and clicks the "Report a problem" button.

[0159] Output: A problem report trigger is sent to the system.

[0160] Step 2:

[0161] Data collection

[0162] Input: User problem report

[0163] Action 1: The terminal automatically collects system data (e.g., error logs, configuration information).

[0164] For Android: Use the "Logcat" tool to collect error logs.

[0165] For iOS: Use the "Console" app to collect error logs.

[0166] Action 2: The device requests permission from the user to take screenshots or screen recordings, and collects them if the user grants permission.

[0167] Output: Collected system data and display information (screenshots, screen recordings) are generated.

[0168] Step 3:

[0169] Sending data

[0170] Input: Collected system data and display information

[0171] Action 1: The device encrypts the collected data.

[0172] We will use OpenSSL as the encryption library.

[0173] Action 2: Send encrypted data to the server via the HTTPS protocol.

[0174] Action 3: Confirm that data transmission is complete and notify the user if successful.

[0175] Output: The encrypted data is sent to the server, and the user receives a notification that "Data transmission complete."

[0176] Step 4:

[0177] Data reception and analysis

[0178] Input: Encrypted data

[0179] Action 1: The server decompresses and decodes the received data.

[0180] Use a decompression tool to extract the ZIP file and read the data file in JSON format.

[0181] Action 2: Verify that the received data is in the correct format.

[0182] Step 3: Analyze the error log using a generative AI model (e.g., GPT-3).

[0183] Enter the prompt message "Identify the cause of the problem from this error log" to obtain the analysis results.

[0184] Step 4: Analyze screenshots and screen recordings using image recognition technology (e.g., TensorFlow, OpenCV).

[0185] Output: The cause of the problem and, if necessary, additional information (image analysis results) are identified.

[0186] Step 5:

[0187] Solution generation

[0188] Input: Analysis results

[0189] Action 1: The server generates a solution using the generated AI model.

[0190] Enter the prompt message "Generate steps and precautions to resolve this issue" to obtain the solution.

[0191] Action 2: Create a video explaining the solution using FFMPEG as needed.

[0192] Output: Solutions in image, video, text, and audio formats are generated.

[0193] Step 6:

[0194] Providing solutions

[0195] Input: Generated solution (image, video, text, audio)

[0196] Action 1: The server encrypts the generated solution and sends it to the terminal via the HTTPS protocol.

[0197] Operation 2: The terminal decompresses the received data and provides it to the user through the user interface.

[0198] Action 3: The user solves the problem by following the solution.

[0199] Output: The user is provided with a concrete solution, and the problem is resolved.

[0200] (Application Example 1)

[0201] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0202] Because autonomous vehicles are controlled by numerous electronic devices and software, rapid and accurate troubleshooting is essential to ensure their proper operation. However, current troubleshooting of autonomous vehicles often requires on-site response by specialized technicians, which can be time-consuming and laborious. Especially in emergencies, a rapid response is crucial, and there is a need for a way to resolve the problem as quickly as possible.

[0203] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0204] In this invention, the server includes means for collecting data from the customer's terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server to identify the customer's problem, means for generating a solution to the problem using images, videos, text, and audio, means for providing the generated solution to the customer's terminal, means for the central processing unit in the autonomous vehicle to collect and manage the data necessary for the execution of these means, and means for providing procedures for resolving problems in the autonomous vehicle. This enables a rapid and accurate response to problems in the autonomous vehicle, and allows the user to resolve the problem themselves.

[0205] A "terminal" refers to a portable information processing device used by a customer, such as a smartphone, tablet, in-car console display, or smart glasses.

[0206] A "server" is a computer system that receives data transmitted from terminals via a network, analyzes it, generates appropriate solutions, and provides them to the terminals.

[0207] A "data collection method" is a function that automatically collects relevant system data, sensor data, and software version information when a terminal receives a problem report.

[0208] "Data transmission means" refers to the function of encrypting collected data and sending it to the server using a secure channel.

[0209] "Data analysis means" refers to the function of a server that decompresses and decodes received data and uses a generated AI model to identify the cause of a problem.

[0210] A "solution generation method" is a function in which, after the cause of a problem has been identified, the server generates a solution that includes specific operating procedures and precautions using images, videos, text, and audio.

[0211] A "solution provisioning mechanism" is a function that sends a generated solution from a server to a terminal, which then receives it and provides it to the user through a user interface.

[0212] The "Central Processing Unit" is the main computing unit within an autonomous vehicle that collects and manages data and effectively executes various functions.

[0213] "Troubleshooting" refers to a series of processes for quickly identifying and addressing various problems that may occur in autonomous vehicles.

[0214] This invention is a system that assists in troubleshooting in autonomous vehicles. This system collects system data from the customer's terminal (e.g., an in-car console display or smart glasses), sends that data to a server for analysis, identifies problems, generates solutions, and provides them to the customer's terminal.

[0215] Specifically, this system works as follows:

[0216] When a user discovers a specific problem in an autonomous vehicle, they report it using the in-car console display or smart glasses. The device receives the report and automatically collects relevant system data (error logs, sensor data, software version information, etc.). Furthermore, it collects screenshots and screen recordings of display information and important messages.

[0217] The collected data is encrypted within the device and sent to the server via a secure communication channel. Once the data transmission is complete, the device notifies the user of the successful transmission.

[0218] The server decompresses and decodes the received data and analyzes it using a generative AI model. Based on the received system data and display information, the server identifies the cause of the problem.

[0219] After the cause of the problem is identified, the server generates a solution using images, videos, text, and audio. The solution includes specific operating procedures and precautions. For example, a solution for a sensor error might provide instructions for reinstalling the sensor driver, using both video and text.

[0220] The generated solution is sent from the server to the terminal, which then provides the received solution to the user through a user interface. The user can then follow the displayed instructions to resolve the problem with the autonomous vehicle.

[0221] Specific examples of hardware and software to be used

[0222] Hardware: Central computer in the autonomous vehicle, in-car console display, smart glasses

[0223] Software: Python program, encryption library (Fernet), HTTP library for data transmission (requests)

[0224] Examples of specific cases and prompt statements

[0225] As a concrete example, consider a scenario where a user discovers a sensor error in an autonomous vehicle and reports the problem via the console display. In this case, the system collects an error log and sends it to a server, which analyzes the error log and generates a procedure for reinstalling the sensor driver. This procedure is then displayed on the console display.

[0226] Example of a prompt:

[0227] text

[0228] A user detects a sensor error in an autonomous vehicle and reports the problem using the in-vehicle console display. The system collects the error log and sends it to the server. As a solution, instructions for reinstalling the sensor driver are generated and displayed on the in-vehicle display.

[0229] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0230] Step 1:

[0231] The user discovers a specific problem in the autonomous vehicle (e.g., a sensor error) and reports the problem using the in-vehicle console display or smart glasses. The input is the problem report information provided by the user. The output is a system confirmation notification indicating that a problem has been reported.

[0232] Step 2:

[0233] The device receives user problem reports and automatically collects relevant system data (error logs, sensor data, software version information, etc.). The input is the user problem report, and the output is the collected system data. The device also collects screenshots and screen recordings.

[0234] Step 3:

[0235] The terminal encrypts the data it collects and sends it to the server using a secure communication channel. The input is the collected system data, and the output is the encrypted data and a notification that the transmission is complete. The terminal uses an encryption library (Fernet) to protect the data and an HTTP library (requests) to send the data.

[0236] Step 4:

[0237] The server decompresses and decodes the received data. The input is encrypted system data, and the output is the decompressed and decoded data. The server verifies the integrity of the data and proceeds with appropriate analysis.

[0238] Step 5:

[0239] The server uses a generated AI model to analyze the decompressed data and identify the root cause of the problem. The input is the decompressed and decoded system data, and the output is the identified cause of the problem. The server uses the AI ​​model to analyze patterns in the data and find the root cause of the error.

[0240] Step 6:

[0241] Based on the cause of the problem identified by the server, a solution is generated using images, videos, text, and audio. The input is the identified cause of the problem, and the output is the content of the generated solution. The solution includes specific operating procedures and precautions.

[0242] Step 7:

[0243] The server sends the generated solution to the terminal. The input is the content of the generated solution, and the output is the data of the solution sent to the terminal. The server uses the HTTP protocol to securely transmit the data.

[0244] Step 8:

[0245] The terminal analyzes the received solution data and provides it to the user through a user interface. The input is the solution data sent from the server, and the output is the content of the solution displayed to the user. The terminal provides the solution to the user in an easy-to-understand format, such as video or text.

[0246] The coordination of the above processing steps enables efficient troubleshooting within autonomous vehicles. This also allows users to quickly obtain appropriate solutions and resolve problems.

[0247] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0248] overview

[0249] This invention is a system that assists in troubleshooting customers' smartphones and incorporates an emotion engine that recognizes user emotions. The system collects data from the customer's device, sends the collected data to a server, the server analyzes the received data to identify the problem, generates a solution to that problem, and provides it to the customer's device. It also features the ability to recognize user emotions and provide emotion-based adaptive support.

[0250] System operation

[0251] The system of the present invention consists of the following elements:

[0252] 1. Terminal:

[0253] A terminal is a portable information processing device such as a smartphone or tablet used by the user. When a user reports a problem, the terminal collects system data and display information (screenshots, screen recordings).

[0254] It is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their emotions.

[0255] 2. Server:

[0256] A server is a computer system that receives data transmitted from terminals over a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[0257] Based on the emotional data transmitted from the emotion engine, the method of providing solutions is adjusted.

[0258] System processing flow

[0259] 1. Data Collection

[0260] The user discovers a problem with their smartphone and launches the support app to report the issue. If the user's voice and camera are enabled, the emotion engine analyzes their voice and facial expressions to recognize their emotional state.

[0261] The device automatically collects relevant system data (error logs, configuration information) and simultaneously takes screenshots and screen recordings. The emotion engine analyzes the user's voice and facial expressions to collect emotion data.

[0262] 2. Sending data

[0263] The device encrypts the collected data and sends it to the server using a secure channel. The transmitted data includes sentiment data. The device confirms that data transmission is complete and notifies the user of its success.

[0264] 3. Data Analysis

[0265] The server decompresses and decodes the received data. It verifies that the format of the received data is correct. It analyzes the data using a generative AI model and identifies the cause of problems based on display information and system data. It also analyzes emotion data from the emotion engine to understand the user's emotional state.

[0266] 4. Generating support information

[0267] The server generates specific troubleshooting steps based on the cause of the problem. These steps are detailed and include images, videos, text, and audio. For example, a video of the app update process, screenshots of settings changes, text instructions, and audio guides.

[0268] Based on emotional data, it generates adaptive support information tailored to the user's emotional state. For example, if a user is feeling frustrated, it provides a more polite and easy-to-understand explanation.

[0269] The generated support information is packaged and prepared for transmission to the device.

[0270] 5. Providing support information

[0271] The terminal receives support information sent from the server. It verifies the integrity of the received data and confirms that there are no problems. The support information is displayed or played back to the user through the user interface.

[0272] We adjust the delivery method based on emotional data. For example, if the emotional state is negative, we add a message of encouragement.

[0273] The user follows the displayed troubleshooting steps. If necessary, they watch the provided explanatory videos and listen to the audio guide. They then confirm that the problem has been resolved and provide feedback through the support app.

[0274] Specific example

[0275] Examples of resolving app crashes

[0276] 1. The user discovers a problem where a specific app repeatedly crashes and launches the support app to report the issue. The emotion engine analyzes the user's voice and facial expressions and recognizes that they are feeling frustrated.

[0277] 2. The device receives the problem report and collects the error log and screenshots from the crash. The device encrypts the collected data and sentiment data and sends it to the server.

[0278] 3. Analyze the data received by the server and identify that a specific library needs updating. Also, since the user is feeling frustrated, provide a more detailed explanation of the solution.

[0279] 4. The server generates videos and texts including the library update procedure and provides them to the user in an easy-to-understand form. Encouraging messages are added to the videos.

[0280] 5. The terminal receives the generated support information and displays it to the user through the user interface. The user updates the app library according to the presented procedure and solves the problem.

[0281] As described above, the system of the present invention combined with the emotion engine supports efficient troubleshooting of the user's smartphone and aims to improve customer satisfaction.

[0282] The following describes the processing flow.

[0283] Step 1: Problem report

[0284] User:

[0285] The user discovers a problem with the smartphone and launches the support app.

[0286] In the support app, tap the "Report Problem" button and enter the problem summary or explain it by voice.

[0287] Step 2: Data collection

[0288] Terminal:

[0289] Upon receiving the user's input, the terminal automatically collects relevant system data (error logs, setting information).

[0290] At the same time, request the user for permission to take screenshots or screen recordings.

[0291] If the user gives permission, the terminal takes screenshots or screen recordings at the time of the problem occurrence.

[0292] The emotion engine analyzes the user's voice and facial expressions to collect emotional data.

[0293] The collected data and sentiment data are saved to a temporary file.

[0294] Step 3: Data transmission

[0295] Terminal:

[0296] The collected data and sentiment data are encrypted and sent to the server using a secure channel.

[0297] Confirm that data transmission is complete and notify the user of its success.

[0298] Step 4: Data Analysis

[0299] server:

[0300] Decompresses and decodes the received data.

[0301] Verify that the format of the received data is correct.

[0302] Start analyzing the data using a generative AI model.

[0303] Image recognition technology is used to analyze display information (screenshots and screen recordings) and identify the area where the problem is displayed.

[0304] Analyze system data (error logs and configuration information) to identify the root cause of the problem.

[0305] We analyze emotional data from the emotion engine to understand the user's emotional state.

[0306] Step 5: Generating support information

[0307] server:

[0308] Generate specific solution procedures based on the cause of the problem.

[0309] The solution procedures are created in detail using images, videos, texts, and voices.

[0310] Generate adaptive support information according to the user's emotional state based on the emotional data.

[0311] For example, when the user feels frustrated, provide a more detailed and understandable explanation.

[0312] Package the generated support information and prepare it for transmission to the terminal.

[0313] Step 6: Provision of Support Information

[0314] Terminal:

[0315] Receive the support information sent from the server.

[0316] Check the integrity of the received data and confirm that there is no problem.

[0317] Display and play the support information to the user through the user interface.

[0318] Adjust the provision method based on the emotional data. For example, when the emotion is negative, add an encouraging message.

[0319]

[0320] [[ID]] User:

[0321] Perform operations according to the displayed solution procedures.

[0322] Watch the provided explanatory video and listen to the voice guide as needed.

[0323] ​Confirm that the problem has been resolved and provide feedback through the support app.

[0324] (Example 2)

[0325] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0326] Conventional smartphone troubleshooting systems often provided only general explanations without considering the user's emotional state when resolving their problems. This could lead to user frustration and dissatisfaction, potentially lowering customer satisfaction. Furthermore, identifying the user's problem and generating solutions could be inefficient, resulting in prolonged problem-solving times. This invention aims to enable efficient and highly satisfying troubleshooting by recognizing the user's emotional state in real time and providing adaptive support based on that emotion.

[0327] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0328] In this invention, the server includes means for collecting system data, display information, and emotional data from the customer's terminal; means for transmitting the collected system data, display information, and emotional data to the server; means for analyzing the system data, display information, and emotional data received by the server to identify the customer's problem and emotional state; means for generating a solution to the problem using adaptive information based on images, videos, text, audio, and emotions; and means for providing the generated solution to the customer's terminal. This enables adaptive support that takes the user's emotions into consideration, resulting in efficient and highly satisfactory troubleshooting.

[0329] "System data" refers to technical information related to the operation of a terminal, including error logs and configuration information.

[0330] "Display information" refers to information captured from the content displayed on the device's screen, including screenshots and screen recordings.

[0331] "Emotional data" refers to digital data that indicates a user's emotional state, obtained by analyzing the user's voice and facial expressions.

[0332] A "server" is a computer system that can receive, analyze, and transmit data over a network.

[0333] "Analysis" is the process of thoroughly investigating and examining received data to identify problems and understand emotional states.

[0334] A "solution" is a proposed plan that includes specific action steps to address an identified problem, and is generated using adaptive information based on images, videos, text, audio, and emotions.

[0335] "Emotion-based adaptive information" refers to information that includes explanations and support messages tailored to the user's emotional state (e.g., frustration or anxiety).

[0336] "To collect" means the act of selecting and gathering necessary data.

[0337] "To transmit" refers to the act of transferring data to another device or system.

[0338] "Generating" is the act of creating new data or information based on specific input.

[0339] "Providing" means the act of handing over information, such as generated solutions, in a form that users can use.

[0340] This invention is a system that assists customers in troubleshooting their smartphones, and by combining it with an emotion engine that recognizes the user's emotions, it provides more efficient and satisfying support. This system consists of the following elements:

[0341] 1. Hardware Configuration

[0342] Terminal:

[0343] These are portable information processing devices such as smartphones and tablets. Users use these devices to report problems and receive support information.

[0344] The device is equipped with sensors and software to collect system data, display information (screenshots, screen recordings), and an emotion engine.

[0345] server:

[0346] This is a computer system that receives data transmitted from terminals via a network, performs analysis, and generates solutions. The server uses the latest image recognition technology and generative AI models to analyze emotional data from the emotion engine.

[0347] 2. Software Configuration

[0348] Emotional engine:

[0349] The system analyzes the user's voice and facial expressions to recognize their emotional state. This analysis uses an open-source emotion analysis library (e.g., "OpenFace").

[0350] Generative AI models:

[0351] Generative AI models such as "GPT-4(registered trademark)" are used for data analysis and solution generation. These models can analyze error logs and screenshots to find appropriate solutions.

[0352] 3. Data Processing

[0353] Data collection:

[0354] When a user launches the support app and reports a problem, the device collects system data (error logs, configuration information) and display information.

[0355] The emotion engine analyzes the user's voice and facial expressions to generate emotion data.

[0356] Data transmission and analysis:

[0357] The device encrypts the data it collects and sends it to the server using a secure channel (e.g., HTTPS). The server decompresses and decodes the received data and analyzes it using a generative AI model.

[0358] Generating and providing solutions:

[0359] The server generates specific solution steps based on the analysis results. These solution steps include detailed explanations with images, videos, text, and audio.

[0360] Furthermore, based on emotional data, it generates adaptive support information tailored to the user's emotional state. For example, if a user is feeling frustrated, it will provide a more detailed explanation of solutions and add encouraging messages.

[0361] 4. Specific Examples

[0362] Examples of resolving app crashes

[0363] A user discovers a problem where a specific app repeatedly crashes and launches a support app to report the issue. The emotion engine analyzes the user's voice and facial expressions to recognize that they are feeling frustrated.

[0364] The device receives a problem report, collects error logs and screenshots from the crash, and sends them to the server along with sentiment data.

[0365] The server analyzes the received data and identifies that a specific library needs updating. Furthermore, because the user is experiencing frustration, the solution will be explained more clearly.

[0366] The server generates videos and text containing library update instructions, providing them to users in an easy-to-understand format. Encouraging messages are added to the videos.

[0367] The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[0368] 5. Example of a prompt statement

[0369] Identify and generate solutions for smartphone app crashes reported by users. Since users are frustrated, particularly clear and easy-to-understand explanations are essential.

[0370] Thus, the system of the present invention provides adaptive support that takes user emotions into consideration, thereby improving customer satisfaction.

[0371] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0372] Step 1:

[0373] Data collection

[0374] The user launches the support app on their smartphone, fills out the input form, and reports the problem.

[0375] Input: User problem report (e.g., "The app crashes").

[0376] The device automatically collects system data (error logs and configuration information) and takes screenshots and screen recordings.

[0377] The emotion engine analyzes the user's voice and facial expressions to generate emotion data (e.g., "frustration: high").

[0378] Output: System data, display information, sentiment data.

[0379] Step 2:

[0380] Sending data

[0381] The system data, display information, and emotional data collected by the device will be encrypted. AES-256 encryption will be used.

[0382] Inputs: System data, display information, sentiment data.

[0383] The device encrypts the data and converts it into secure data packets.

[0384] The device sends encrypted data to the server using a secure channel ("HTTPS").

[0385] Output: Encrypted data packets.

[0386] Step 3:

[0387] Data reception and analysis

[0388] The server receives encrypted data packets.

[0389] The server decompresses and decrypts the received data.

[0390] Input: Encrypted data packet.

[0391] The server uses a decompression tool ("gzip") to decompress the data and then decrypts it using the "AES-256" key.

[0392] Output: Decoded system data, display information, sentiment data.

[0393] The server uses a generated AI model (e.g., "GPT-4") to analyze the data and identify the cause of the problem.

[0394] Input: Decrypted system data, display information.

[0395] The server analyzes error logs and screenshots, and a generated AI model identifies that "a specific library needs to be updated."

[0396] The server analyzes emotional data to understand the user's emotional state (e.g., "frustration: high").

[0397] Output: Cause of the problem, emotional state.

[0398] Step 4:

[0399] Solution generation

[0400] The server generates specific troubleshooting steps based on the identified cause of the problem.

[0401] Input: The cause of the problem (e.g., "Library update required"), emotional state.

[0402] The server generates detailed descriptions using images, videos, text, and audio.

[0403] Example of a solution:

[0404] Video: "Tutorial video showing the procedure for updating the library"

[0405] Text: "Steps to select Settings > System > Update > Library Update"

[0406] Based on emotional data, the explanation will be made more detailed, and encouraging messages will be added.

[0407] Output: Solution steps information, encouraging message.

[0408] Step 5:

[0409] Sending and providing support information

[0410] The server packages the resolution procedure information it generates, encrypts it again, and sends it to the terminal.

[0411] Input: Solution steps information, encouraging messages.

[0412] The server compiles the resolution steps in JSON format, encrypts them, and packages them.

[0413] The server sends encrypted support information to the terminal.

[0414] Output: Encrypted support information package.

[0415] The terminal receives the support information package and performs decompression and decryption.

[0416] Input: Encrypted support information package.

[0417] The device uses a decryption tool to unpack and decrypt the data.

[0418] Output: Decrypted resolution information, encouraging message.

[0419] The device displays and plays troubleshooting information to the user through its user interface.

[0420] Input: Decrypted resolution step information, encouraging message.

[0421] The device displays text guides in the "Support" section within the app, while video guides play in the video player.

[0422] The device will apply an emotionally sensitive delivery method and display encouraging messages.

[0423] Output: Solution steps and encouraging messages provided to the user.

[0424] The user follows the provided instructions to resolve the issue.

[0425] Input: Solution steps information, encouraging messages.

[0426] The user performs the app update procedure and updates the necessary libraries.

[0427] The user confirms that the problem has been resolved and provides feedback.

[0428] Output: Resolved issues, user feedback.

[0429] (Application Example 2)

[0430] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0431] In conventional systems, when customers reported problems with their smartphones or robots, the system often provided a uniform solution that disregarded their emotional state. This could cause stress for both customers and technicians, making it difficult to provide appropriate and effective support. This invention aims to improve customer satisfaction by considering the emotional state of customers and technicians and providing quick and adaptive solutions.

[0432] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting data from the customer's terminal, means for transmitting the collected data to the server, means including an emotion engine for analyzing the customer's emotions, means for generating solutions to problems using images, videos, text, and audio, means for adaptively providing solutions based on the emotion engine, and means for providing the generated solutions to the customer's terminal. This makes it possible to provide solutions that correspond to the emotional state of the customer and the technician, enabling more effective and satisfying support.

[0433] A "terminal" refers to a portable information processing device, such as a smartphone or tablet, used by a user.

[0434] "Means of collecting data" refers to a function that automatically collects relevant system data (error logs, configuration information) from the device and also takes screenshots and screen recordings.

[0435] "Means of transmitting data" refers to the means of encrypting the collected data and sending it to the server through a secure communication channel.

[0436] "Means of analyzing data and identifying problems" refers to the function of analyzing received data to identify the cause of errors or troubles in networks and robotic systems.

[0437] An "emotion engine" is an engine that includes technology for recognizing a user's emotional state by analyzing their voice and facial expressions.

[0438] "Means for generating solutions" refers to a function that generates specific operating procedures and solutions based on analysis results, using images, videos, text, and audio.

[0439] "Means of adaptively providing solutions" refers to means of adjusting the method of providing solutions based on the user's emotional state recognized by the emotion engine.

[0440] "Means of providing solutions" refers to a function that displays the generated solutions on the user's device and provides them through audio or video.

[0441] To implement this invention, the following system is constructed. The system consists of a customer's terminal, a server, and software to link them together.

[0442] Hardware and software to be used

[0443] hardware

[0444] Customer devices: Portable information processing devices such as smartphones and tablets.

[0445] Robots used in factories

[0446] software

[0447] Emotion engine: Microsoft® Azure® Cognitive Services and Google® Cloud Emotion API, etc.

[0448] Image recognition technology: OpenCV, TensorFlow

[0449] Generative AI model: GPT-4 from OpenAI (registered trademark)

[0450] Encryption and communication: TLS / SSL

[0451] Development environment: Android Studio, Xcode

[0452] Program processing details

[0453] 1. Data Collection

[0454] The customer's device automatically collects relevant system data (error logs, configuration information) when a problem occurs. The device also simultaneously captures screenshots and screen recordings. Furthermore, it uses an emotion engine to analyze the user's voice and facial expressions to recognize their emotional state. For example, if a malfunction is reported in a robot used in a factory, the device collects the robot's operation log and the technician's voice and facial expressions at that moment.

[0455] 2. Sending data

[0456] The collected data is encrypted and sent to the server via a secure communication channel (TLS / SSL). For example, robot anomaly logs and technician sentiment data are sent to the server simultaneously.

[0457] 3. Data analysis and problem identification

[0458] The server analyzes the received data. Using image recognition technologies (OpenCV, TensorFlow) and generative AI models (GPT-4), it identifies the cause of problems based on display information and system data. It also analyzes emotion data from the emotion engine to understand the user's emotional state. For example, the server can identify the cause of a robot malfunction and recognize that the technician is experiencing stress.

[0459] 4. Generating support information

[0460] Based on the cause of the problem, the server generates specific troubleshooting steps. These steps are detailed and include images, videos, text, and audio. Based on emotional data, the server adjusts how the troubleshooting steps are explained. For example, if a technician is feeling stressed, the server generates a more polite and easy-to-understand explanatory video and encouraging messages.

[0461] 5. Providing support information

[0462] Support information is encrypted and transmitted to the customer's device via a secure channel. The device verifies the received support information and provides it to the user through a user interface. For example, a technician's device might display a video and text showing the troubleshooting steps, and play an audio guide.

[0463] Specific example

[0464] For example, if a robot arm operating in a factory malfunctions, a technician can launch a smartphone app to report the problem. An emotion engine analyzes the technician's voice and facial expressions to determine if they are stressed, and a server analyzes the robot's operation logs to identify that a specific part needs replacing. The server generates a video and text of the replacement procedure and sends it to the technician's smartphone along with an encouraging message.

[0465] Examples of input prompts for a generative AI model

[0466] Identify the cause of the robot arm malfunction and provide a video and text-based explanation of the replacement procedure. Please include a message of encouragement, as the technicians are under stress.

[0467] Thus, the system of this invention can provide adaptive support based on the emotional state of customers and engineers, enabling more effective and satisfying solutions.

[0468] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0469] Step 1:

[0470] The user launches a smartphone app and reports a problem with the robot. The input includes a description of the problem (text) and images or videos of the problem occurring. The device collects this data and obtains robot operation logs and error logs. Simultaneously, the device's built-in emotion engine captures the user's voice and facial expressions to generate emotion data. The output includes system data, image / video data, and emotion data.

[0471] Step 2:

[0472] The device encrypts the collected data and sends it to the server using a secure communication channel (TLS / SSL). Inputs include system data, image / video data, and sentiment data, each encrypted before transmission. The output is the encrypted data received by the server. Specifically, the device encrypts the data, and the encrypted data is sent to the server over the network.

[0473] Step 3:

[0474] The server decompresses and decodes the received data. Input includes encrypted system data, image / video data, and sentiment data. The server decrypts these and outputs them as data ready for analysis. Specifically, the server decrypts the data and converts it into an analyzable format.

[0475] Step 4:

[0476] The server analyzes data using image recognition technology (OpenCV, TensorFlow) and a generative AI model (GPT-4). Inputs include decompressed system data, image / video data, and emotion data. The server identifies the cause of a problem and understands the user's emotional state. Outputs include specific cause information (text) and emotion analysis results (emotional state). The specific operation involves analysis using an AI model and recognition of the emotional state by an emotion engine.

[0477] Step 5:

[0478] The server generates solutions based on the analysis results. Inputs include specific cause information of the problem and sentiment analysis results. The server generates detailed solutions using images, videos, text, and audio, adapting the explanation method based on sentiment data. The output is a solution package (images, videos, text, audio) for the user. Specifically, the generating AI model generates explanatory videos and text, including adaptive explanations that respond to sentiment.

[0479] Step 6:

[0480] The generated solution package is encrypted and sent to the user's terminal via a secure communication channel. The input is the solution package (images, videos, text, audio). The data is encrypted upon reaching the terminal, and the output is the encrypted data sent from the server. Specifically, the server encrypts the data, and the encrypted data is sent to the user's terminal over the network.

[0481] Step 7:

[0482] The user's device verifies, decompresses, and decodes the received solution package. The input is an encrypted solution package. The device decrypts it and converts it into a format that can be displayed in the user interface. The output is a displayable solution (image, video, text, audio). Specifically, the device decrypts the data and provides the solution through the user interface.

[0483] Step 8:

[0484] The user resolves the problem by following the provided solutions. Input includes support information (images, videos, text, audio). The user then repairs or modifies the robot's settings based on this information. For example, instructions for replacing parts to ensure the robot's arm functions correctly are provided. Output includes feedback to the server via the terminal indicating that the problem has been resolved. Specific actions involve the user following instructions and reporting progress through the support application.

[0485] The above describes the specific processing flow and operation of the system program for realizing this invention.

[0486] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0487] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0488] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0489] [Second Embodiment]

[0490] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0491] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0492] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0493] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0494] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0495] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0496] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0497] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0498] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0499] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0500] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0501] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0502] overview

[0503] This invention is a system that assists in troubleshooting customers' smartphones. This system collects data from the customer's device, sends the collected data to a server, the server analyzes the received data to identify the problem, generates a solution to that problem, and provides it to the customer's device.

[0504] System operation

[0505] The present invention's system mainly consists of the following elements:

[0506] 1. Terminal:

[0507] A terminal is a portable information processing device such as a smartphone or tablet used by the user. When a user reports a problem, the terminal collects system data and display information (screenshots, screen recordings).

[0508] 2. Server:

[0509] A server is a computer system that receives data transmitted from terminals over a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[0510] System processing flow

[0511] 1. Data Collection

[0512] A user discovers a problem with their smartphone, launches a support app, and reports the issue.

[0513] The device receives user reports and automatically collects relevant system data (error logs, configuration information). Furthermore, it requests permission from the user to take screenshots or screen recordings to collect display information at the time the problem occurred.

[0514] 2. Sending data

[0515] The collected data is encrypted on the device and sent to the server using a secure channel. The system confirms that data transmission is complete and notifies the user of the successful transmission.

[0516] 3. Data Analysis

[0517] The server decompresses and decodes the received data. It verifies the correct format of the received data and analyzes it using a generative AI model. Based on display information and system data, it identifies the cause of the problem.

[0518] 4. Generating support information

[0519] After the cause of the problem is identified, the server generates a solution using images, videos, text, and audio. This solution includes specific steps and precautions.

[0520] 5. Providing support information

[0521] The generated support information is sent from the server to the terminal. The terminal analyzes the received data and provides it to the user through the user interface. The user can then resolve the problem according to the solution.

[0522] Specific example

[0523] Examples of resolving app crashes

[0524] 1. A user discovers a problem where a specific app repeatedly crashes and launches the support app to report the problem.

[0525] 2. The device receives the problem report and collects the error log and screenshots from the crash. The device encrypts the collected data and sends it to the server.

[0526] 3. The server analyzes the received data and identifies that a specific library needs to be updated.

[0527] 4. The server generates videos and text containing library update procedures, providing users with easy-to-understand instructions on how to perform the specific operations.

[0528] 5. The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[0529] In the form described above, the present invention is a system that enables efficient and intuitive troubleshooting of smartphones. This significantly reduces the time and effort required to resolve customer problems, thereby improving customer satisfaction.

[0530] The following describes the processing flow.

[0531] Step 1: Data Collection

[0532] User:

[0533] Users launch the support app when they encounter problems with their smartphone.

[0534] Tap the "Report a Problem" button in the support app and enter a summary of the problem or describe it verbally.

[0535] Terminal:

[0536] Upon receiving user input, the terminal automatically collects relevant system data (error logs and configuration information).

[0537] The device will display a message asking the user for permission to take screenshots or screen recordings.

[0538] If the user grants permission, the device will take screenshots and screen recordings when the problem occurs.

[0539] The collected data is saved on the device as a temporary file.

[0540] Step 2: Send

[0541] Terminal:

[0542] The collected data is encrypted and sent to the server using a secure channel.

[0543] Confirm that data transmission is complete and notify the user of its success.

[0544] Step 3: Data Analysis

[0545] server:

[0546] The received data is decompressed, and its format is verified.

[0547] The generative AI model is launched, and data analysis begins.

[0548] Image recognition technology is used to analyze display information (screenshots and screen recordings) and identify the area where the problem is displayed.

[0549] Analyze system data (error logs and configuration information) to identify the root cause of the problem.

[0550] Step 4: Generating support information

[0551] server:

[0552] Based on the cause of the problem, generate specific solution steps.

[0553] The troubleshooting steps are created in detail using images, videos, text, and audio. For example, a video of the app update procedure, screenshots of the settings change procedure, text instructions, and an audio guide.

[0554] The generated support information is packaged and prepared for transmission to the device.

[0555] Step 5: Providing support information

[0556] Terminal:

[0557] Receive support information sent from the server.

[0558] Verify the integrity of the received data and confirm that there are no problems.

[0559] Support information is displayed or played to the user through the user interface. For example, a video of the troubleshooting procedure or text explanation may be displayed.

[0560] User:

[0561] Follow the displayed solution steps.

[0562] If necessary, watch the provided explanatory videos and listen to the audio guide.

[0563] Confirm that the problem has been resolved and provide feedback through the support app.

[0564] In this way, the system of the present invention supports users in efficiently troubleshooting their smartphones.

[0565] (Example 1)

[0566] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0567] Modern information processing devices offer numerous functions, but the number of problems and issues users face is also increasing. These problems are often complex and difficult to resolve on their own. Furthermore, troubleshooting requires significant time and effort, leading to decreased user satisfaction. This invention aims to provide a system that efficiently and intuitively resolves problems users encounter with information processing devices, thereby reducing the burden on users.

[0568] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0569] In this invention, the server includes means for collecting data from the user's information processing device, means for transmitting the collected data to the processing device, and means for the processing device to analyze the received data and identify a problem. This enables the user to solve complex problems quickly and efficiently.

[0570] A "user" is an individual or group that uses an information processing device.

[0571] An "information processing device" is a device that includes hardware and software for collecting, transmitting, receiving, analyzing, and displaying data.

[0572] "Data" refers to various types of information and records within an information processing device, such as error logs, configuration information, screenshots, and screen recordings.

[0573] A "processing device" is a computer system that receives data, analyzes it, and generates results.

[0574] "Analysis" is the process of understanding the content of received data and identifying the cause of a problem.

[0575] A "solution" is information that includes steps and methods for a user to resolve a problem, based on the cause of the problem.

[0576] An "image" is a still image used to convey information visually.

[0577] A "video" is a dynamic medium that conveys visual information along a time axis.

[0578] "Text" refers to information expressed using characters.

[0579] "Sound" refers to sound signals that transmit information through hearing.

[0580] "To transmit" means to move information or data from one device or system to another.

[0581] "To collect" means to gather and incorporate necessary data and information.

[0582] Modes for carrying out the invention

[0583] overview

[0584] This invention is a system that assists in troubleshooting a user's information processing device (e.g., a smartphone or tablet). The system collects data from the user's device and sends the collected data to a server. The server analyzes the received data, identifies the problem the user is facing, generates a solution to that problem, and provides it to the user's device.

[0585] System Configuration

[0586] This system mainly consists of the following elements:

[0587] 1. User's device:

[0588] This refers to information processing devices such as smartphones and tablets used by users. When a user reports a problem, the device collects system data (error logs, configuration information) and display information (screenshots, screen recordings).

[0589] 2. Server:

[0590] This computer system receives data transmitted from terminals via a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[0591] Hardware and software to be used

[0592] 1. On the device side:

[0593] OS: Android or iOS

[0594] Log collection tools: "Logcat" for Android, "Console" for iOS

[0595] Cryptographic library: OpenSSL

[0596] Communication protocol: HTTPS

[0597] 2. Server side:

[0598] Data analysis tools: TensorFlow (image recognition), OpenCV (image analysis)

[0599] Generative AI models: BERT, GPT-3

[0600] Video processing tool: FFMPEG

[0601] System processing flow

[0602] Data collection

[0603] When a user discovers a problem with their smartphone, they launch the support app and report the issue. Upon receiving the report, the device automatically collects relevant system data (error logs, configuration information). It also collects screenshots and screen recordings. This allows for a detailed understanding of the user's specific problem.

[0604] Sending data

[0605] The collected data is encrypted on the device and sent to the server using HTTPS. The user is notified upon successful transmission. This ensures secure data transmission and timely communication with the user.

[0606] Data analysis and solution generation

[0607] The server decompresses the received data and parses the JSON formatted data. TensorFlow and OpenCV are used to analyze screenshots and screen recordings, and a generative AI model (such as GPT-3) is used to analyze error logs. This allows for rapid identification of user problems and generation of appropriate solutions.

[0608] Providing solutions

[0609] The generated solutions are sent from the server to the terminal in text, image, video, and audio formats. The server creates content containing detailed steps and notes for the solution, which the terminal displays in its user interface. The user refers to this content and follows the instructions to resolve the problem.

[0610] Specific examples and prompt statements

[0611] Specific example

[0612] For example, suppose a user reports that a particular app is repeatedly crashing. In this case, the device uses Logcat to collect error logs from the crash and also takes screenshots. This data is encrypted with OpenSSL and sent to the server via HTTPS. The server analyzes the received data and uses a generative AI model to identify that "a specific library needs to be updated." Then, it generates a video using FFMPEG explaining the library update procedure and provides it to the user.

[0613] Example of a prompt

[0614] "Please identify the cause of the problem from this error log."

[0615] In this way, this system can efficiently solve user problems by linking terminals and servers. By automating everything from data collection and analysis to solution generation and delivery, it is expected to significantly reduce the burden on users and improve their satisfaction.

[0616] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0617] Step 1:

[0618] Report the problem

[0619] Input: User reports of problems occurring on their smartphone (e.g., app crashes)

[0620] Action: The user launches the support app and clicks the "Report a problem" button.

[0621] Output: A problem report trigger is sent to the system.

[0622] Step 2:

[0623] Data collection

[0624] Input: User problem report

[0625] Action 1: The terminal automatically collects system data (e.g., error logs, configuration information).

[0626] For Android: Use the "Logcat" tool to collect error logs.

[0627] For iOS: Use the "Console" app to collect error logs.

[0628] Action 2: The device requests permission from the user to take screenshots or screen recordings, and collects them if the user grants permission.

[0629] Output: Collected system data and display information (screenshots, screen recordings) are generated.

[0630] Step 3:

[0631] Sending data

[0632] Input: Collected system data and display information

[0633] Action 1: The device encrypts the collected data.

[0634] We will use OpenSSL as the encryption library.

[0635] Action 2: Send encrypted data to the server via the HTTPS protocol.

[0636] Action 3: Confirm that data transmission is complete and notify the user if successful.

[0637] Output: The encrypted data is sent to the server, and the user receives a notification that "Data transmission complete."

[0638] Step 4:

[0639] Data reception and analysis

[0640] Input: Encrypted data

[0641] Action 1: The server decompresses and decodes the received data.

[0642] Use a decompression tool to extract the ZIP file and read the data file in JSON format.

[0643] Action 2: Verify that the received data is in the correct format.

[0644] Step 3: Analyze the error log using a generative AI model (e.g., GPT-3).

[0645] Enter the prompt message "Identify the cause of the problem from this error log" to obtain the analysis results.

[0646] Step 4: Analyze screenshots and screen recordings using image recognition technology (e.g., TensorFlow, OpenCV).

[0647] Output: The cause of the problem and, if necessary, additional information (image analysis results) are identified.

[0648] Step 5:

[0649] Solution generation

[0650] Input: Analysis results

[0651] Action 1: The server generates a solution using the generated AI model.

[0652] Enter the prompt message "Generate steps and precautions to resolve this issue" to obtain the solution.

[0653] Action 2: Create a video explaining the solution using FFMPEG as needed.

[0654] Output: Solutions in image, video, text, and audio formats are generated.

[0655] Step 6:

[0656] Providing solutions

[0657] Input: Generated solution (image, video, text, audio)

[0658] Action 1: The server encrypts the generated solution and sends it to the terminal via the HTTPS protocol.

[0659] Operation 2: The terminal decompresses the received data and provides it to the user through the user interface.

[0660] Action 3: The user solves the problem by following the solution.

[0661] Output: The user is provided with a concrete solution, and the problem is resolved.

[0662] (Application Example 1)

[0663] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0664] Because autonomous vehicles are controlled by numerous electronic devices and software, rapid and accurate troubleshooting is essential to ensure their proper operation. However, current troubleshooting of autonomous vehicles often requires on-site response by specialized technicians, which can be time-consuming and laborious. Especially in emergencies, a rapid response is crucial, and there is a need for a way to resolve the problem as quickly as possible.

[0665] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0666] In this invention, the server includes means for collecting data from the customer's terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server to identify the customer's problem, means for generating a solution to the problem using images, videos, text, and audio, means for providing the generated solution to the customer's terminal, means for the central processing unit in the autonomous vehicle to collect and manage the data necessary for the execution of these means, and means for providing procedures for resolving problems in the autonomous vehicle. This enables a rapid and accurate response to problems in the autonomous vehicle, and allows the user to resolve the problem themselves.

[0667] A "terminal" refers to a portable information processing device used by a customer, such as a smartphone, tablet, in-car console display, or smart glasses.

[0668] A "server" is a computer system that receives data transmitted from terminals via a network, analyzes it, generates appropriate solutions, and provides them to the terminals.

[0669] A "data collection method" is a function that automatically collects relevant system data, sensor data, and software version information when a terminal receives a problem report.

[0670] "Data transmission means" refers to the function of encrypting collected data and sending it to the server using a secure channel.

[0671] "Data analysis means" refers to the function of a server that decompresses and decodes received data and uses a generated AI model to identify the cause of a problem.

[0672] A "solution generation method" is a function in which, after the cause of a problem has been identified, the server generates a solution that includes specific operating procedures and precautions using images, videos, text, and audio.

[0673] A "solution provisioning mechanism" is a function that sends a generated solution from a server to a terminal, which then receives it and provides it to the user through a user interface.

[0674] The "Central Processing Unit" is the main computing unit within an autonomous vehicle that collects and manages data and effectively executes various functions.

[0675] "Troubleshooting" refers to a series of processes for quickly identifying and addressing various problems that may occur in autonomous vehicles.

[0676] This invention is a system that assists in troubleshooting in autonomous vehicles. This system collects system data from the customer's terminal (e.g., an in-car console display or smart glasses), sends that data to a server for analysis, identifies problems, generates solutions, and provides them to the customer's terminal.

[0677] Specifically, this system works as follows:

[0678] When a user discovers a specific problem in an autonomous vehicle, they report it using the in-car console display or smart glasses. The device receives the report and automatically collects relevant system data (error logs, sensor data, software version information, etc.). Furthermore, it collects screenshots and screen recordings of display information and important messages.

[0679] The collected data is encrypted within the device and sent to the server via a secure communication channel. Once the data transmission is complete, the device notifies the user of the successful transmission.

[0680] The server decompresses and decodes the received data and analyzes it using a generative AI model. Based on the received system data and display information, the server identifies the cause of the problem.

[0681] After the cause of the problem is identified, the server generates a solution using images, videos, text, and audio. The solution includes specific operating procedures and precautions. For example, a solution for a sensor error might provide instructions for reinstalling the sensor driver, using both video and text.

[0682] The generated solution is sent from the server to the terminal, which then provides the received solution to the user through a user interface. The user can then follow the displayed instructions to resolve the problem with the autonomous vehicle.

[0683] Specific examples of hardware and software to be used

[0684] Hardware: Central computer in the autonomous vehicle, in-car console display, smart glasses

[0685] Software: Python program, encryption library (Fernet), HTTP library for data transmission (requests)

[0686] Examples of specific cases and prompt statements

[0687] As a concrete example, consider a scenario where a user discovers a sensor error in an autonomous vehicle and reports the problem via the console display. In this case, the system collects an error log and sends it to a server, which analyzes the error log and generates a procedure for reinstalling the sensor driver. This procedure is then displayed on the console display.

[0688] Example of a prompt:

[0689] text

[0690] A user detects a sensor error in an autonomous vehicle and reports the problem using the in-vehicle console display. The system collects the error log and sends it to the server. As a solution, instructions for reinstalling the sensor driver are generated and displayed on the in-vehicle display.

[0691] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0692] Step 1:

[0693] The user discovers a specific problem in the autonomous vehicle (e.g., a sensor error) and reports the problem using the in-vehicle console display or smart glasses. The input is the problem report information provided by the user. The output is a system confirmation notification indicating that a problem has been reported.

[0694] Step 2:

[0695] The device receives user problem reports and automatically collects relevant system data (error logs, sensor data, software version information, etc.). The input is the user problem report, and the output is the collected system data. The device also collects screenshots and screen recordings.

[0696] Step 3:

[0697] The terminal encrypts the data it collects and sends it to the server using a secure communication channel. The input is the collected system data, and the output is the encrypted data and a notification that the transmission is complete. The terminal uses an encryption library (Fernet) to protect the data and an HTTP library (requests) to send the data.

[0698] Step 4:

[0699] The server decompresses and decodes the received data. The input is encrypted system data, and the output is the decompressed and decoded data. The server verifies the integrity of the data and proceeds with appropriate analysis.

[0700] Step 5:

[0701] The server uses a generated AI model to analyze the decompressed data and identify the root cause of the problem. The input is the decompressed and decoded system data, and the output is the identified cause of the problem. The server uses the AI ​​model to analyze patterns in the data and find the root cause of the error.

[0702] Step 6:

[0703] Based on the cause of the problem identified by the server, a solution is generated using images, videos, text, and audio. The input is the identified cause of the problem, and the output is the content of the generated solution. The solution includes specific operating procedures and precautions.

[0704] Step 7:

[0705] The server sends the generated solution to the terminal. The input is the content of the generated solution, and the output is the data of the solution sent to the terminal. The server uses the HTTP protocol to securely transmit the data.

[0706] Step 8:

[0707] The terminal analyzes the received solution data and provides it to the user through a user interface. The input is the solution data sent from the server, and the output is the content of the solution displayed to the user. The terminal provides the solution to the user in an easy-to-understand format, such as video or text.

[0708] The coordination of the above processing steps enables efficient troubleshooting within autonomous vehicles. This also allows users to quickly obtain appropriate solutions and resolve problems.

[0709] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0710] overview

[0711] This invention is a system that assists in troubleshooting customers' smartphones and incorporates an emotion engine that recognizes user emotions. The system collects data from the customer's device, sends the collected data to a server, the server analyzes the received data to identify the problem, generates a solution to that problem, and provides it to the customer's device. It also features the ability to recognize user emotions and provide emotion-based adaptive support.

[0712] System operation

[0713] The system of the present invention consists of the following elements:

[0714] 1. Terminal:

[0715] A terminal is a portable information processing device such as a smartphone or tablet used by the user. When a user reports a problem, the terminal collects system data and display information (screenshots, screen recordings).

[0716] It is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their emotions.

[0717] 2. Server:

[0718] A server is a computer system that receives data transmitted from terminals over a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[0719] Based on the emotional data transmitted from the emotion engine, the method of providing solutions is adjusted.

[0720] System processing flow

[0721] 1. Data Collection

[0722] The user discovers a problem with their smartphone and launches the support app to report the issue. If the user's voice and camera are enabled, the emotion engine analyzes their voice and facial expressions to recognize their emotional state.

[0723] The device automatically collects relevant system data (error logs, configuration information) and simultaneously takes screenshots and screen recordings. The emotion engine analyzes the user's voice and facial expressions to collect emotion data.

[0724] 2. Sending data

[0725] The device encrypts the collected data and sends it to the server using a secure channel. The transmitted data includes sentiment data. The device confirms that data transmission is complete and notifies the user of its success.

[0726] 3. Data Analysis

[0727] The server decompresses and decodes the received data. It verifies that the format of the received data is correct. It analyzes the data using a generative AI model and identifies the cause of problems based on display information and system data. It also analyzes emotion data from the emotion engine to understand the user's emotional state.

[0728] 4. Generating support information

[0729] The server generates specific troubleshooting steps based on the cause of the problem. These steps are detailed and include images, videos, text, and audio. For example, a video of the app update process, screenshots of settings changes, text instructions, and audio guides.

[0730] Based on emotional data, it generates adaptive support information tailored to the user's emotional state. For example, if a user is feeling frustrated, it provides a more polite and easy-to-understand explanation.

[0731] The generated support information is packaged and prepared for transmission to the device.

[0732] 5. Providing support information

[0733] The terminal receives support information sent from the server. It verifies the integrity of the received data and confirms that there are no problems. The support information is displayed or played back to the user through the user interface.

[0734] We adjust the delivery method based on emotional data. For example, if the emotional state is negative, we add a message of encouragement.

[0735] The user follows the displayed troubleshooting steps. If necessary, they watch the provided explanatory videos and listen to the audio guide. They then confirm that the problem has been resolved and provide feedback through the support app.

[0736] Specific example

[0737] Examples of resolving app crashes

[0738] 1. The user discovers a problem where a specific app repeatedly crashes and launches the support app to report the issue. The emotion engine analyzes the user's voice and facial expressions and recognizes that they are feeling frustrated.

[0739] 2. The device receives the problem report and collects the error log and screenshots from the crash. The device encrypts the collected data and sentiment data and sends it to the server.

[0740] 3. Analyze the data received by the server and identify that a specific library needs updating. Also, since the user is feeling frustrated, provide a more detailed explanation of the solution.

[0741] 4. The server generates videos and text containing library update instructions, providing them to the user in an easy-to-understand format. Encouraging messages are added to the videos.

[0742] 5. The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[0743] As described above, the system of the present invention, which combines an emotion engine, supports efficient troubleshooting of users' smartphones and aims to improve customer satisfaction.

[0744] The following describes the processing flow.

[0745] Step 1: Report the problem

[0746] User:

[0747] The user discovers a problem with their smartphone and launches a support app.

[0748] Tap the "Report a Problem" button in the support app and enter a summary of the problem or describe it verbally.

[0749] Step 2: Data Collection

[0750] Terminal:

[0751] Upon receiving user input, the terminal automatically collects relevant system data (error logs, configuration information).

[0752] At the same time, the user is asked for permission to take screenshots and screen recordings.

[0753] If the user grants permission, the device will take screenshots and screen recordings when the problem occurs.

[0754] The emotion engine analyzes the user's voice and facial expressions to collect emotional data.

[0755] The collected data and sentiment data are saved to a temporary file.

[0756] Step 3: Data transmission

[0757] Terminal:

[0758] The collected data and sentiment data are encrypted and sent to the server using a secure channel.

[0759] Confirm that data transmission is complete and notify the user of its success.

[0760] Step 4: Data Analysis

[0761] server:

[0762] Decompresses and decodes the received data.

[0763] Verify that the format of the received data is correct.

[0764] Start analyzing the data using a generative AI model.

[0765] Image recognition technology is used to analyze display information (screenshots and screen recordings) and identify the area where the problem is displayed.

[0766] Analyze system data (error logs and configuration information) to identify the root cause of the problem.

[0767] We analyze emotional data from the emotion engine to understand the user's emotional state.

[0768] Step 5: Generating support information

[0769] server:

[0770] Based on the cause of the problem, generate specific solution steps.

[0771] The solution steps are created in detail using images, videos, text, and audio.

[0772] Based on emotional data, it generates adaptive support information tailored to the user's emotional state.

[0773] For example, if a user is feeling frustrated, provide a more thorough and easy-to-understand explanation.

[0774] The generated support information is packaged and prepared for transmission to the device.

[0775] Step 6: Providing support information

[0776] Terminal:

[0777] Receive support information sent from the server.

[0778] Verify the integrity of the received data and confirm that there are no problems.

[0779] Support information is displayed and played for the user through the user interface.

[0780] The delivery method is adjusted based on emotional data; for example, if the emotional state is negative, an encouraging message is added.

[0781] Step 7: Problem Solving

[0782] User:

[0783] Follow the displayed solution steps.

[0784] If necessary, watch the provided explanatory videos and listen to the audio guide.

[0785] Confirm that the problem has been resolved and provide feedback through the support app.

[0786] (Example 2)

[0787] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0788] Conventional smartphone troubleshooting systems often provided only general explanations without considering the user's emotional state when resolving their problems. This could lead to user frustration and dissatisfaction, potentially lowering customer satisfaction. Furthermore, identifying the user's problem and generating solutions could be inefficient, resulting in prolonged problem-solving times. This invention aims to enable efficient and highly satisfying troubleshooting by recognizing the user's emotional state in real time and providing adaptive support based on that emotion.

[0789] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0790] In this invention, the server includes means for collecting system data, display information, and emotional data from the customer's terminal; means for transmitting the collected system data, display information, and emotional data to the server; means for analyzing the system data, display information, and emotional data received by the server to identify the customer's problem and emotional state; means for generating a solution to the problem using adaptive information based on images, videos, text, audio, and emotions; and means for providing the generated solution to the customer's terminal. This enables adaptive support that takes the user's emotions into consideration, resulting in efficient and highly satisfactory troubleshooting.

[0791] "System data" refers to technical information related to the operation of a terminal, including error logs and configuration information.

[0792] "Display information" refers to information captured from the content displayed on the device's screen, including screenshots and screen recordings.

[0793] "Emotional data" refers to digital data that indicates a user's emotional state, obtained by analyzing the user's voice and facial expressions.

[0794] A "server" is a computer system that can receive, analyze, and transmit data over a network.

[0795] "Analysis" is the process of thoroughly investigating and examining received data to identify problems and understand emotional states.

[0796] A "solution" is a proposed plan that includes specific action steps to address an identified problem, and is generated using adaptive information based on images, videos, text, audio, and emotions.

[0797] "Emotion-based adaptive information" refers to information that includes explanations and support messages tailored to the user's emotional state (e.g., frustration or anxiety).

[0798] "To collect" means the act of selecting and gathering necessary data.

[0799] "To transmit" refers to the act of transferring data to another device or system.

[0800] "Generating" is the act of creating new data or information based on specific input.

[0801] "Providing" means the act of handing over information, such as generated solutions, in a form that users can use.

[0802] This invention is a system that assists customers in troubleshooting their smartphones, and by combining it with an emotion engine that recognizes the user's emotions, it provides more efficient and satisfying support. This system consists of the following elements:

[0803] 1. Hardware Configuration

[0804] Terminal:

[0805] These are portable information processing devices such as smartphones and tablets. Users use these devices to report problems and receive support information.

[0806] The device is equipped with sensors and software to collect system data, display information (screenshots, screen recordings), and an emotion engine.

[0807] server:

[0808] This is a computer system that receives data transmitted from terminals via a network, performs analysis, and generates solutions. The server uses the latest image recognition technology and generative AI models to analyze emotional data from the emotion engine.

[0809] 2. Software Configuration

[0810] Emotional engine:

[0811] The system analyzes the user's voice and facial expressions to recognize their emotional state. This analysis uses an open-source emotion analysis library (e.g., "OpenFace").

[0812] Generative AI models:

[0813] Generative AI models such as "GPT-4" are used for data analysis and solution generation. These models can analyze error logs and screenshots to find appropriate solutions.

[0814] 3. Data Processing

[0815] Data collection:

[0816] When a user launches the support app and reports a problem, the device collects system data (error logs, configuration information) and display information.

[0817] The emotion engine analyzes the user's voice and facial expressions to generate emotion data.

[0818] Data transmission and analysis:

[0819] The device encrypts the data it collects and sends it to the server using a secure channel (e.g., HTTPS). The server decompresses and decodes the received data and analyzes it using a generative AI model.

[0820] Generating and providing solutions:

[0821] The server generates specific solution steps based on the analysis results. These solution steps include detailed explanations with images, videos, text, and audio.

[0822] Furthermore, based on emotional data, it generates adaptive support information tailored to the user's emotional state. For example, if a user is feeling frustrated, it will provide a more detailed explanation of solutions and add encouraging messages.

[0823] 4. Specific Examples

[0824] Examples of resolving app crashes

[0825] A user discovers a problem where a specific app repeatedly crashes and launches a support app to report the issue. The emotion engine analyzes the user's voice and facial expressions to recognize that they are feeling frustrated.

[0826] The device receives a problem report, collects error logs and screenshots from the crash, and sends them to the server along with sentiment data.

[0827] The server analyzes the received data and identifies that a specific library needs updating. Furthermore, because the user is experiencing frustration, the solution will be explained more clearly.

[0828] The server generates videos and text containing library update instructions, providing them to users in an easy-to-understand format. Encouraging messages are added to the videos.

[0829] The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[0830] 5. Example of a prompt statement

[0831] Identify and generate solutions for smartphone app crashes reported by users. Since users are frustrated, particularly clear and easy-to-understand explanations are essential.

[0832] Thus, the system of the present invention provides adaptive support that takes user emotions into consideration, thereby improving customer satisfaction.

[0833] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0834] Step 1:

[0835] Data collection

[0836] The user launches the support app on their smartphone, fills out the input form, and reports the problem.

[0837] Input: User problem report (e.g., "The app crashes").

[0838] The device automatically collects system data (error logs and configuration information) and takes screenshots and screen recordings.

[0839] The emotion engine analyzes the user's voice and facial expressions to generate emotion data (e.g., "frustration: high").

[0840] Output: System data, display information, sentiment data.

[0841] Step 2:

[0842] Sending data

[0843] The system data, display information, and emotional data collected by the device will be encrypted. AES-256 encryption will be used.

[0844] Inputs: System data, display information, sentiment data.

[0845] The device encrypts the data and converts it into secure data packets.

[0846] The device sends encrypted data to the server using a secure channel ("HTTPS").

[0847] Output: Encrypted data packets.

[0848] Step 3:

[0849] Data reception and analysis

[0850] The server receives encrypted data packets.

[0851] The server decompresses and decrypts the received data.

[0852] Input: Encrypted data packet.

[0853] The server uses a decompression tool ("gzip") to decompress the data and then decrypts it using the "AES-256" key.

[0854] Output: Decoded system data, display information, sentiment data.

[0855] The server uses a generated AI model (e.g., "GPT-4") to analyze the data and identify the cause of the problem.

[0856] Input: Decrypted system data, display information.

[0857] The server analyzes error logs and screenshots, and a generated AI model identifies that "a specific library needs to be updated."

[0858] The server analyzes emotional data to understand the user's emotional state (e.g., "frustration: high").

[0859] Output: Cause of the problem, emotional state.

[0860] Step 4:

[0861] Solution generation

[0862] The server generates specific troubleshooting steps based on the identified cause of the problem.

[0863] Input: The cause of the problem (e.g., "Library update required"), emotional state.

[0864] The server generates detailed descriptions using images, videos, text, and audio.

[0865] Example of a solution:

[0866] Video: "Tutorial video showing the procedure for updating the library"

[0867] Text: "Steps to select Settings > System > Update > Library Update"

[0868] Based on emotional data, the explanation will be made more detailed, and encouraging messages will be added.

[0869] Output: Solution steps information, encouraging message.

[0870] Step 5:

[0871] Sending and providing support information

[0872] The server packages the resolution procedure information it generates, encrypts it again, and sends it to the terminal.

[0873] Input: Solution steps information, encouraging messages.

[0874] The server compiles the resolution steps in JSON format, encrypts them, and packages them.

[0875] The server sends encrypted support information to the terminal.

[0876] Output: Encrypted support information package.

[0877] The terminal receives the support information package and performs decompression and decryption.

[0878] Input: Encrypted support information package.

[0879] The device uses a decryption tool to unpack and decrypt the data.

[0880] Output: Decrypted resolution information, encouraging message.

[0881] The device displays and plays troubleshooting information to the user through its user interface.

[0882] Input: Decrypted resolution step information, encouraging message.

[0883] The device displays text guides in the "Support" section within the app, while video guides play in the video player.

[0884] The device will apply an emotionally sensitive delivery method and display encouraging messages.

[0885] Output: Solution steps and encouraging messages provided to the user.

[0886] The user follows the provided instructions to resolve the issue.

[0887] Input: Solution steps information, encouraging messages.

[0888] The user performs the app update procedure and updates the necessary libraries.

[0889] The user confirms that the problem has been resolved and provides feedback.

[0890] Output: Resolved issues, user feedback.

[0891] (Application Example 2)

[0892] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0893] In conventional systems, when customers reported problems with their smartphones or robots, the system often provided a uniform solution that disregarded their emotional state. This could cause stress for both customers and technicians, making it difficult to provide appropriate and effective support. This invention aims to improve customer satisfaction by considering the emotional state of customers and technicians and providing quick and adaptive solutions.

[0894] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting data from the customer's terminal, means for transmitting the collected data to the server, means including an emotion engine for analyzing the customer's emotions, means for generating solutions to problems using images, videos, text, and audio, means for adaptively providing solutions based on the emotion engine, and means for providing the generated solutions to the customer's terminal. This makes it possible to provide solutions that correspond to the emotional state of the customer and the technician, enabling more effective and satisfying support.

[0895] A "terminal" refers to a portable information processing device, such as a smartphone or tablet, used by a user.

[0896] "Means of collecting data" refers to a function that automatically collects relevant system data (error logs, configuration information) from the device and also takes screenshots and screen recordings.

[0897] "Means of transmitting data" refers to the means of encrypting the collected data and sending it to the server through a secure communication channel.

[0898] "Means of analyzing data and identifying problems" refers to the function of analyzing received data to identify the cause of errors or troubles in networks and robotic systems.

[0899] An "emotion engine" is an engine that includes technology for recognizing a user's emotional state by analyzing their voice and facial expressions.

[0900] "Means for generating solutions" refers to a function that generates specific operating procedures and solutions based on analysis results, using images, videos, text, and audio.

[0901] "Means of adaptively providing solutions" refers to means of adjusting the method of providing solutions based on the user's emotional state recognized by the emotion engine.

[0902] "Means of providing solutions" refers to a function that displays the generated solutions on the user's device and provides them through audio or video.

[0903] To implement this invention, the following system is constructed. The system consists of a customer's terminal, a server, and software to link them together.

[0904] Hardware and software to be used

[0905] hardware

[0906] Customer devices: Portable information processing devices such as smartphones and tablets.

[0907] Robots used in factories

[0908] software

[0909] Emotion engines: Microsoft Azure Cognitive Services, Google Cloud Emotion API, etc.

[0910] Image recognition technology: OpenCV, TensorFlow

[0911] Generative AI model: OpenAI's GPT-4

[0912] Encryption and communication: TLS / SSL

[0913] Development environment: Android Studio, Xcode

[0914] Program processing details

[0915] 1. Data Collection

[0916] The customer's device automatically collects relevant system data (error logs, configuration information) when a problem occurs. The device also simultaneously captures screenshots and screen recordings. Furthermore, it uses an emotion engine to analyze the user's voice and facial expressions to recognize their emotional state. For example, if a malfunction is reported in a robot used in a factory, the device collects the robot's operation log and the technician's voice and facial expressions at that moment.

[0917] 2. Sending data

[0918] The collected data is encrypted and sent to the server via a secure communication channel (TLS / SSL). For example, robot anomaly logs and technician sentiment data are sent to the server simultaneously.

[0919] 3. Data analysis and problem identification

[0920] The server analyzes the received data. Using image recognition technologies (OpenCV, TensorFlow) and generative AI models (GPT-4), it identifies the cause of problems based on display information and system data. It also analyzes emotion data from the emotion engine to understand the user's emotional state. For example, the server can identify the cause of a robot malfunction and recognize that the technician is experiencing stress.

[0921] 4. Generating support information

[0922] Based on the cause of the problem, the server generates specific troubleshooting steps. These steps are detailed and include images, videos, text, and audio. Based on emotional data, the server adjusts how the troubleshooting steps are explained. For example, if a technician is feeling stressed, the server generates a more polite and easy-to-understand explanatory video and encouraging messages.

[0923] 5. Providing support information

[0924] Support information is encrypted and transmitted to the customer's device via a secure channel. The device verifies the received support information and provides it to the user through a user interface. For example, a technician's device might display a video and text showing the troubleshooting steps, and play an audio guide.

[0925] Specific example

[0926] For example, if a robot arm operating in a factory malfunctions, a technician can launch a smartphone app to report the problem. An emotion engine analyzes the technician's voice and facial expressions to determine if they are stressed, and a server analyzes the robot's operation logs to identify that a specific part needs replacing. The server generates a video and text of the replacement procedure and sends it to the technician's smartphone along with an encouraging message.

[0927] Examples of input prompts for a generative AI model

[0928] Identify the cause of the robot arm malfunction and provide a video and text-based explanation of the replacement procedure. Please include a message of encouragement, as the technicians are under stress.

[0929] Thus, the system of this invention can provide adaptive support based on the emotional state of customers and engineers, enabling more effective and satisfying solutions.

[0930] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0931] Step 1:

[0932] The user launches a smartphone app and reports a problem with the robot. The input includes a description of the problem (text) and images or videos of the problem occurring. The device collects this data and obtains robot operation logs and error logs. Simultaneously, the device's built-in emotion engine captures the user's voice and facial expressions to generate emotion data. The output includes system data, image / video data, and emotion data.

[0933] Step 2:

[0934] The device encrypts the collected data and sends it to the server using a secure communication channel (TLS / SSL). Inputs include system data, image / video data, and sentiment data, each encrypted before transmission. The output is the encrypted data received by the server. Specifically, the device encrypts the data, and the encrypted data is sent to the server over the network.

[0935] Step 3:

[0936] The server decompresses and decodes the received data. Input includes encrypted system data, image / video data, and sentiment data. The server decrypts these and outputs them as data ready for analysis. Specifically, the server decrypts the data and converts it into an analyzable format.

[0937] Step 4:

[0938] The server analyzes data using image recognition technology (OpenCV, TensorFlow) and a generative AI model (GPT-4). Inputs include decompressed system data, image / video data, and emotion data. The server identifies the cause of a problem and understands the user's emotional state. Outputs include specific cause information (text) and emotion analysis results (emotional state). The specific operation involves analysis using an AI model and recognition of the emotional state by an emotion engine.

[0939] Step 5:

[0940] The server generates solutions based on the analysis results. Inputs include specific cause information of the problem and sentiment analysis results. The server generates detailed solutions using images, videos, text, and audio, adapting the explanation method based on sentiment data. The output is a solution package (images, videos, text, audio) for the user. Specifically, the generating AI model generates explanatory videos and text, including adaptive explanations that respond to sentiment.

[0941] Step 6:

[0942] The generated solution package is encrypted and sent to the user's terminal via a secure communication channel. The input is the solution package (images, videos, text, audio). The data is encrypted upon reaching the terminal, and the output is the encrypted data sent from the server. Specifically, the server encrypts the data, and the encrypted data is sent to the user's terminal over the network.

[0943] Step 7:

[0944] The user's device verifies, decompresses, and decodes the received solution package. The input is an encrypted solution package. The device decrypts it and converts it into a format that can be displayed in the user interface. The output is a displayable solution (image, video, text, audio). Specifically, the device decrypts the data and provides the solution through the user interface.

[0945] Step 8:

[0946] The user resolves the problem by following the provided solutions. Input includes support information (images, videos, text, audio). The user then repairs or modifies the robot's settings based on this information. For example, instructions for replacing parts to ensure the robot's arm functions correctly are provided. Output includes feedback to the server via the terminal indicating that the problem has been resolved. Specific actions involve the user following instructions and reporting progress through the support application.

[0947] The above describes the specific processing flow and operation of the system program for realizing this invention.

[0948] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0949] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0950] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0951] [Third Embodiment]

[0952] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0953] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0954] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0955] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0956] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0957] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0958] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0959] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0960] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0961] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0962] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0963] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0964] overview

[0965] This invention is a system that assists in troubleshooting customers' smartphones. This system collects data from the customer's device, sends the collected data to a server, the server analyzes the received data to identify the problem, generates a solution to that problem, and provides it to the customer's device.

[0966] System operation

[0967] The present invention's system mainly consists of the following elements:

[0968] 1. Terminal:

[0969] A terminal is a portable information processing device such as a smartphone or tablet used by the user. When a user reports a problem, the terminal collects system data and display information (screenshots, screen recordings).

[0970] 2. Server:

[0971] A server is a computer system that receives data transmitted from terminals over a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[0972] System processing flow

[0973] 1. Data Collection

[0974] A user discovers a problem with their smartphone, launches a support app, and reports the issue.

[0975] The device receives user reports and automatically collects relevant system data (error logs, configuration information). Furthermore, it requests permission from the user to take screenshots or screen recordings to collect display information at the time the problem occurred.

[0976] 2. Sending data

[0977] The collected data is encrypted on the device and sent to the server using a secure channel. The system confirms that data transmission is complete and notifies the user of the successful transmission.

[0978] 3. Data Analysis

[0979] The server decompresses and decodes the received data. It verifies the correct format of the received data and analyzes it using a generative AI model. Based on display information and system data, it identifies the cause of the problem.

[0980] 4. Generating support information

[0981] After the cause of the problem is identified, the server generates a solution using images, videos, text, and audio. This solution includes specific steps and precautions.

[0982] 5. Providing support information

[0983] The generated support information is sent from the server to the terminal. The terminal analyzes the received data and provides it to the user through the user interface. The user can then resolve the problem according to the solution.

[0984] Specific example

[0985] Examples of resolving app crashes

[0986] 1. A user discovers a problem where a specific app repeatedly crashes and launches the support app to report the problem.

[0987] 2. The device receives the problem report and collects the error log and screenshots from the crash. The device encrypts the collected data and sends it to the server.

[0988] 3. The server analyzes the received data and identifies that a specific library needs to be updated.

[0989] 4. The server generates videos and text containing library update procedures, providing users with easy-to-understand instructions on how to perform the specific operations.

[0990] 5. The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[0991] In the form described above, the present invention is a system that enables efficient and intuitive troubleshooting of smartphones. This significantly reduces the time and effort required to resolve customer problems, thereby improving customer satisfaction.

[0992] The following describes the processing flow.

[0993] Step 1: Data Collection

[0994] User:

[0995] Users launch the support app when they encounter problems with their smartphone.

[0996] Tap the "Report a Problem" button in the support app and enter a summary of the problem or describe it verbally.

[0997] Terminal:

[0998] Upon receiving user input, the terminal automatically collects relevant system data (error logs and configuration information).

[0999] The device will display a message asking the user for permission to take screenshots or screen recordings.

[1000] If the user grants permission, the device will take screenshots and screen recordings when the problem occurs.

[1001] The collected data is saved on the device as a temporary file.

[1002] Step 2: Send

[1003] Terminal:

[1004] The collected data is encrypted and sent to the server using a secure channel.

[1005] Confirm that data transmission is complete and notify the user of its success.

[1006] Step 3: Data Analysis

[1007] server:

[1008] The received data is decompressed, and its format is verified.

[1009] The generative AI model is launched, and data analysis begins.

[1010] Image recognition technology is used to analyze display information (screenshots and screen recordings) and identify the area where the problem is displayed.

[1011] Analyze system data (error logs and configuration information) to identify the root cause of the problem.

[1012] Step 4: Generating support information

[1013] server:

[1014] Based on the cause of the problem, generate specific solution steps.

[1015] The troubleshooting steps are created in detail using images, videos, text, and audio. For example, a video of the app update procedure, screenshots of the settings change procedure, text instructions, and an audio guide.

[1016] The generated support information is packaged and prepared for transmission to the device.

[1017] Step 5: Providing support information

[1018] Terminal:

[1019] Receive support information sent from the server.

[1020] Verify the integrity of the received data and confirm that there are no problems.

[1021] Support information is displayed or played to the user through the user interface. For example, a video of the troubleshooting procedure or text explanation may be displayed.

[1022] User:

[1023] Follow the displayed solution steps.

[1024] If necessary, watch the provided explanatory videos and listen to the audio guide.

[1025] Confirm that the problem has been resolved and provide feedback through the support app.

[1026] In this way, the system of the present invention supports users in efficiently troubleshooting their smartphones.

[1027] (Example 1)

[1028] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1029] Modern information processing devices offer numerous functions, but the number of problems and issues users face is also increasing. These problems are often complex and difficult to resolve on their own. Furthermore, troubleshooting requires significant time and effort, leading to decreased user satisfaction. This invention aims to provide a system that efficiently and intuitively resolves problems users encounter with information processing devices, thereby reducing the burden on users.

[1030] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1031] In this invention, the server includes means for collecting data from the user's information processing device, means for transmitting the collected data to the processing device, and means for the processing device to analyze the received data and identify a problem. This enables the user to solve complex problems quickly and efficiently.

[1032] A "user" is an individual or group that uses an information processing device.

[1033] An "information processing device" is a device that includes hardware and software for collecting, transmitting, receiving, analyzing, and displaying data.

[1034] "Data" refers to various types of information and records within an information processing device, such as error logs, configuration information, screenshots, and screen recordings.

[1035] A "processing device" is a computer system that receives data, analyzes it, and generates results.

[1036] "Analysis" is the process of understanding the content of received data and identifying the cause of a problem.

[1037] A "solution" is information that includes steps and methods for a user to resolve a problem, based on the cause of the problem.

[1038] An "image" is a still image used to convey information visually.

[1039] A "video" is a dynamic medium that conveys visual information along a time axis.

[1040] "Text" refers to information expressed using characters.

[1041] "Sound" refers to sound signals that transmit information through hearing.

[1042] "To transmit" means to move information or data from one device or system to another.

[1043] "To collect" means to gather and incorporate necessary data and information.

[1044] Modes for carrying out the invention

[1045] overview

[1046] This invention is a system that assists in troubleshooting a user's information processing device (e.g., a smartphone or tablet). The system collects data from the user's device and sends the collected data to a server. The server analyzes the received data, identifies the problem the user is facing, generates a solution to that problem, and provides it to the user's device.

[1047] System Configuration

[1048] This system mainly consists of the following elements:

[1049] 1. User's device:

[1050] This refers to information processing devices such as smartphones and tablets used by users. When a user reports a problem, the device collects system data (error logs, configuration information) and display information (screenshots, screen recordings).

[1051] 2. Server:

[1052] This computer system receives data transmitted from terminals via a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[1053] Hardware and software to be used

[1054] 1. On the device side:

[1055] OS: Android or iOS

[1056] Log collection tools: "Logcat" for Android, "Console" for iOS

[1057] Cryptographic library: OpenSSL

[1058] Communication protocol: HTTPS

[1059] 2. Server side:

[1060] Data analysis tools: TensorFlow (image recognition), OpenCV (image analysis)

[1061] Generative AI models: BERT, GPT-3

[1062] Video processing tool: FFMPEG

[1063] System processing flow

[1064] Data collection

[1065] When a user discovers a problem with their smartphone, they launch the support app and report the issue. Upon receiving the report, the device automatically collects relevant system data (error logs, configuration information). It also collects screenshots and screen recordings. This allows for a detailed understanding of the user's specific problem.

[1066] Sending data

[1067] The collected data is encrypted on the device and sent to the server using HTTPS. The user is notified upon successful transmission. This ensures secure data transmission and timely communication with the user.

[1068] Data analysis and solution generation

[1069] The server decompresses the received data and parses the JSON formatted data. TensorFlow and OpenCV are used to analyze screenshots and screen recordings, and a generative AI model (such as GPT-3) is used to analyze error logs. This allows for rapid identification of user problems and generation of appropriate solutions.

[1070] Providing solutions

[1071] The generated solutions are sent from the server to the terminal in text, image, video, and audio formats. The server creates content containing detailed steps and notes for the solution, which the terminal displays in its user interface. The user refers to this content and follows the instructions to resolve the problem.

[1072] Specific examples and prompt statements

[1073] Specific example

[1074] For example, suppose a user reports that a particular app is repeatedly crashing. In this case, the device uses Logcat to collect error logs from the crash and also takes screenshots. This data is encrypted with OpenSSL and sent to the server via HTTPS. The server analyzes the received data and uses a generative AI model to identify that "a specific library needs to be updated." Then, it generates a video using FFMPEG explaining the library update procedure and provides it to the user.

[1075] Example of a prompt

[1076] "Please identify the cause of the problem from this error log."

[1077] In this way, this system can efficiently solve user problems by linking terminals and servers. By automating everything from data collection and analysis to solution generation and delivery, it is expected to significantly reduce the burden on users and improve their satisfaction.

[1078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1079] Step 1:

[1080] Report the problem

[1081] Input: User reports of problems occurring on their smartphone (e.g., app crashes)

[1082] Action: The user launches the support app and clicks the "Report a problem" button.

[1083] Output: A problem report trigger is sent to the system.

[1084] Step 2:

[1085] Data collection

[1086] Input: User problem report

[1087] Action 1: The terminal automatically collects system data (e.g., error logs, configuration information).

[1088] For Android: Use the "Logcat" tool to collect error logs.

[1089] For iOS: Use the "Console" app to collect error logs.

[1090] Action 2: The device requests permission from the user to take screenshots or screen recordings, and collects them if the user grants permission.

[1091] Output: Collected system data and display information (screenshots, screen recordings) are generated.

[1092] Step 3:

[1093] Sending data

[1094] Input: Collected system data and display information

[1095] Action 1: The device encrypts the collected data.

[1096] We will use OpenSSL as the encryption library.

[1097] Action 2: Send encrypted data to the server via the HTTPS protocol.

[1098] Action 3: Confirm that data transmission is complete and notify the user if successful.

[1099] Output: The encrypted data is sent to the server, and the user receives a notification that "Data transmission complete."

[1100] Step 4:

[1101] Data reception and analysis

[1102] Input: Encrypted data

[1103] Action 1: The server decompresses and decodes the received data.

[1104] Use a decompression tool to extract the ZIP file and read the data file in JSON format.

[1105] Action 2: Verify that the received data is in the correct format.

[1106] Step 3: Analyze the error log using a generative AI model (e.g., GPT-3).

[1107] Enter the prompt message "Identify the cause of the problem from this error log" to obtain the analysis results.

[1108] Step 4: Analyze screenshots and screen recordings using image recognition technology (e.g., TensorFlow, OpenCV).

[1109] Output: The cause of the problem and, if necessary, additional information (image analysis results) are identified.

[1110] Step 5:

[1111] Solution generation

[1112] Input: Analysis results

[1113] Action 1: The server generates a solution using the generated AI model.

[1114] Enter the prompt message "Generate steps and precautions to resolve this issue" to obtain the solution.

[1115] Action 2: Create a video explaining the solution using FFMPEG as needed.

[1116] Output: Solutions in image, video, text, and audio formats are generated.

[1117] Step 6:

[1118] Providing solutions

[1119] Input: Generated solution (image, video, text, audio)

[1120] Action 1: The server encrypts the generated solution and sends it to the terminal via the HTTPS protocol.

[1121] Operation 2: The terminal decompresses the received data and provides it to the user through the user interface.

[1122] Action 3: The user solves the problem by following the solution.

[1123] Output: The user is provided with a concrete solution, and the problem is resolved.

[1124] (Application Example 1)

[1125] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1126] Because autonomous vehicles are controlled by numerous electronic devices and software, rapid and accurate troubleshooting is essential to ensure their proper operation. However, current troubleshooting of autonomous vehicles often requires on-site response by specialized technicians, which can be time-consuming and laborious. Especially in emergencies, a rapid response is crucial, and there is a need for a way to resolve the problem as quickly as possible.

[1127] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1128] In this invention, the server includes means for collecting data from the customer's terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server to identify the customer's problem, means for generating a solution to the problem using images, videos, text, and audio, means for providing the generated solution to the customer's terminal, means for the central processing unit in the autonomous vehicle to collect and manage the data necessary for the execution of these means, and means for providing procedures for resolving problems in the autonomous vehicle. This enables a rapid and accurate response to problems in the autonomous vehicle, and allows the user to resolve the problem themselves.

[1129] A "terminal" refers to a portable information processing device used by a customer, such as a smartphone, tablet, in-car console display, or smart glasses.

[1130] A "server" is a computer system that receives data transmitted from terminals via a network, analyzes it, generates appropriate solutions, and provides them to the terminals.

[1131] A "data collection method" is a function that automatically collects relevant system data, sensor data, and software version information when a terminal receives a problem report.

[1132] "Data transmission means" refers to the function of encrypting collected data and sending it to the server using a secure channel.

[1133] "Data analysis means" refers to the function of a server that decompresses and decodes received data and uses a generated AI model to identify the cause of a problem.

[1134] A "solution generation method" is a function in which, after the cause of a problem has been identified, the server generates a solution that includes specific operating procedures and precautions using images, videos, text, and audio.

[1135] A "solution provisioning mechanism" is a function that sends a generated solution from a server to a terminal, which then receives it and provides it to the user through a user interface.

[1136] The "Central Processing Unit" is the main computing unit within an autonomous vehicle that collects and manages data and effectively executes various functions.

[1137] "Troubleshooting" refers to a series of processes for quickly identifying and addressing various problems that may occur in autonomous vehicles.

[1138] This invention is a system that assists in troubleshooting in autonomous vehicles. This system collects system data from the customer's terminal (e.g., an in-car console display or smart glasses), sends that data to a server for analysis, identifies problems, generates solutions, and provides them to the customer's terminal.

[1139] Specifically, this system works as follows:

[1140] When a user discovers a specific problem in an autonomous vehicle, they report it using the in-car console display or smart glasses. The device receives the report and automatically collects relevant system data (error logs, sensor data, software version information, etc.). Furthermore, it collects screenshots and screen recordings of display information and important messages.

[1141] The collected data is encrypted within the device and sent to the server via a secure communication channel. Once the data transmission is complete, the device notifies the user of the successful transmission.

[1142] The server decompresses and decodes the received data and analyzes it using a generative AI model. Based on the received system data and display information, the server identifies the cause of the problem.

[1143] After the cause of the problem is identified, the server generates a solution using images, videos, text, and audio. The solution includes specific operating procedures and precautions. For example, a solution for a sensor error might provide instructions for reinstalling the sensor driver, using both video and text.

[1144] The generated solution is sent from the server to the terminal, which then provides the received solution to the user through a user interface. The user can then follow the displayed instructions to resolve the problem with the autonomous vehicle.

[1145] Specific examples of hardware and software to be used

[1146] Hardware: Central computer in the autonomous vehicle, in-car console display, smart glasses

[1147] Software: Python program, encryption library (Fernet), HTTP library for data transmission (requests)

[1148] Examples of specific cases and prompt statements

[1149] As a concrete example, consider a scenario where a user discovers a sensor error in an autonomous vehicle and reports the problem via the console display. In this case, the system collects an error log and sends it to a server, which analyzes the error log and generates a procedure for reinstalling the sensor driver. This procedure is then displayed on the console display.

[1150] Example of a prompt:

[1151] text

[1152] A user detects a sensor error in an autonomous vehicle and reports the problem using the in-vehicle console display. The system collects the error log and sends it to the server. As a solution, instructions for reinstalling the sensor driver are generated and displayed on the in-vehicle display.

[1153] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1154] Step 1:

[1155] The user discovers a specific problem in the autonomous vehicle (e.g., a sensor error) and reports the problem using the in-vehicle console display or smart glasses. The input is the problem report information provided by the user. The output is a system confirmation notification indicating that a problem has been reported.

[1156] Step 2:

[1157] The device receives user problem reports and automatically collects relevant system data (error logs, sensor data, software version information, etc.). The input is the user problem report, and the output is the collected system data. The device also collects screenshots and screen recordings.

[1158] Step 3:

[1159] The terminal encrypts the data it collects and sends it to the server using a secure communication channel. The input is the collected system data, and the output is the encrypted data and a notification that the transmission is complete. The terminal uses an encryption library (Fernet) to protect the data and an HTTP library (requests) to send the data.

[1160] Step 4:

[1161] The server decompresses and decodes the received data. The input is encrypted system data, and the output is the decompressed and decoded data. The server verifies the integrity of the data and proceeds with appropriate analysis.

[1162] Step 5:

[1163] The server uses a generated AI model to analyze the decompressed data and identify the root cause of the problem. The input is the decompressed and decoded system data, and the output is the identified cause of the problem. The server uses the AI ​​model to analyze patterns in the data and find the root cause of the error.

[1164] Step 6:

[1165] Based on the cause of the problem identified by the server, a solution is generated using images, videos, text, and audio. The input is the identified cause of the problem, and the output is the content of the generated solution. The solution includes specific operating procedures and precautions.

[1166] Step 7:

[1167] The server sends the generated solution to the terminal. The input is the content of the generated solution, and the output is the data of the solution sent to the terminal. The server uses the HTTP protocol to securely transmit the data.

[1168] Step 8:

[1169] The terminal analyzes the received solution data and provides it to the user through a user interface. The input is the solution data sent from the server, and the output is the content of the solution displayed to the user. The terminal provides the solution to the user in an easy-to-understand format, such as video or text.

[1170] The coordination of the above processing steps enables efficient troubleshooting within autonomous vehicles. This also allows users to quickly obtain appropriate solutions and resolve problems.

[1171] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1172] overview

[1173] This invention is a system that assists in troubleshooting customers' smartphones and incorporates an emotion engine that recognizes user emotions. The system collects data from the customer's device, sends the collected data to a server, the server analyzes the received data to identify the problem, generates a solution to that problem, and provides it to the customer's device. It also features the ability to recognize user emotions and provide emotion-based adaptive support.

[1174] System operation

[1175] The system of the present invention consists of the following elements:

[1176] 1. Terminal:

[1177] A terminal is a portable information processing device such as a smartphone or tablet used by the user. When a user reports a problem, the terminal collects system data and display information (screenshots, screen recordings).

[1178] It is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their emotions.

[1179] 2. Server:

[1180] A server is a computer system that receives data transmitted from terminals over a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[1181] Based on the emotional data transmitted from the emotion engine, the method of providing solutions is adjusted.

[1182] System processing flow

[1183] 1. Data Collection

[1184] The user discovers a problem with their smartphone and launches the support app to report the issue. If the user's voice and camera are enabled, the emotion engine analyzes their voice and facial expressions to recognize their emotional state.

[1185] The device automatically collects relevant system data (error logs, configuration information) and simultaneously takes screenshots and screen recordings. The emotion engine analyzes the user's voice and facial expressions to collect emotion data.

[1186] 2. Sending data

[1187] The device encrypts the collected data and sends it to the server using a secure channel. The transmitted data includes sentiment data. The device confirms that data transmission is complete and notifies the user of its success.

[1188] 3. Data Analysis

[1189] The server decompresses and decodes the received data. It verifies that the format of the received data is correct. It analyzes the data using a generative AI model and identifies the cause of problems based on display information and system data. It also analyzes emotion data from the emotion engine to understand the user's emotional state.

[1190] 4. Generating support information

[1191] The server generates specific troubleshooting steps based on the cause of the problem. These steps are detailed and include images, videos, text, and audio. For example, a video of the app update process, screenshots of settings changes, text instructions, and audio guides.

[1192] Based on emotional data, it generates adaptive support information tailored to the user's emotional state. For example, if a user is feeling frustrated, it provides a more polite and easy-to-understand explanation.

[1193] The generated support information is packaged and prepared for transmission to the device.

[1194] 5. Providing support information

[1195] The terminal receives support information sent from the server. It verifies the integrity of the received data and confirms that there are no problems. The support information is displayed or played back to the user through the user interface.

[1196] We adjust the delivery method based on emotional data. For example, if the emotional state is negative, we add a message of encouragement.

[1197] The user follows the displayed troubleshooting steps. If necessary, they watch the provided explanatory videos and listen to the audio guide. They then confirm that the problem has been resolved and provide feedback through the support app.

[1198] Specific example

[1199] Examples of resolving app crashes

[1200] 1. The user discovers a problem where a specific app repeatedly crashes and launches the support app to report the issue. The emotion engine analyzes the user's voice and facial expressions and recognizes that they are feeling frustrated.

[1201] 2. The device receives the problem report and collects the error log and screenshots from the crash. The device encrypts the collected data and sentiment data and sends it to the server.

[1202] 3. Analyze the data received by the server and identify that a specific library needs updating. Also, since the user is feeling frustrated, provide a more detailed explanation of the solution.

[1203] 4. The server generates videos and text containing library update instructions, providing them to the user in an easy-to-understand format. Encouraging messages are added to the videos.

[1204] 5. The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[1205] As described above, the system of the present invention, which combines an emotion engine, supports efficient troubleshooting of users' smartphones and aims to improve customer satisfaction.

[1206] The following describes the processing flow.

[1207] Step 1: Report the problem

[1208] User:

[1209] The user discovers a problem with their smartphone and launches a support app.

[1210] Tap the "Report a Problem" button in the support app and enter a summary of the problem or describe it verbally.

[1211] Step 2: Data Collection

[1212] Terminal:

[1213] Upon receiving user input, the terminal automatically collects relevant system data (error logs, configuration information).

[1214] At the same time, the user is asked for permission to take screenshots and screen recordings.

[1215] If the user grants permission, the device will take screenshots and screen recordings when the problem occurs.

[1216] The emotion engine analyzes the user's voice and facial expressions to collect emotional data.

[1217] The collected data and sentiment data are saved to a temporary file.

[1218] Step 3: Data transmission

[1219] Terminal:

[1220] The collected data and sentiment data are encrypted and sent to the server using a secure channel.

[1221] Confirm that data transmission is complete and notify the user of its success.

[1222] Step 4: Data Analysis

[1223] server:

[1224] Decompresses and decodes the received data.

[1225] Verify that the format of the received data is correct.

[1226] Start analyzing the data using a generative AI model.

[1227] Image recognition technology is used to analyze display information (screenshots and screen recordings) and identify the area where the problem is displayed.

[1228] Analyze system data (error logs and configuration information) to identify the root cause of the problem.

[1229] We analyze emotional data from the emotion engine to understand the user's emotional state.

[1230] Step 5: Generating support information

[1231] server:

[1232] Based on the cause of the problem, generate specific solution steps.

[1233] The solution steps are created in detail using images, videos, text, and audio.

[1234] Based on emotional data, it generates adaptive support information tailored to the user's emotional state.

[1235] For example, if a user is feeling frustrated, provide a more thorough and easy-to-understand explanation.

[1236] The generated support information is packaged and prepared for transmission to the device.

[1237] Step 6: Providing support information

[1238] Terminal:

[1239] Receive support information sent from the server.

[1240] Verify the integrity of the received data and confirm that there are no problems.

[1241] Support information is displayed and played for the user through the user interface.

[1242] The delivery method is adjusted based on emotional data; for example, if the emotional state is negative, an encouraging message is added.

[1243] Step 7: Problem Solving

[1244] User:

[1245] Follow the displayed solution steps.

[1246] If necessary, watch the provided explanatory videos and listen to the audio guide.

[1247] Confirm that the problem has been resolved and provide feedback through the support app.

[1248] (Example 2)

[1249] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1250] Conventional smartphone troubleshooting systems often provided only general explanations without considering the user's emotional state when resolving their problems. This could lead to user frustration and dissatisfaction, potentially lowering customer satisfaction. Furthermore, identifying the user's problem and generating solutions could be inefficient, resulting in prolonged problem-solving times. This invention aims to enable efficient and highly satisfying troubleshooting by recognizing the user's emotional state in real time and providing adaptive support based on that emotion.

[1251] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1252] In this invention, the server includes means for collecting system data, display information, and emotional data from the customer's terminal; means for transmitting the collected system data, display information, and emotional data to the server; means for analyzing the system data, display information, and emotional data received by the server to identify the customer's problem and emotional state; means for generating a solution to the problem using adaptive information based on images, videos, text, audio, and emotions; and means for providing the generated solution to the customer's terminal. This enables adaptive support that takes the user's emotions into consideration, resulting in efficient and highly satisfactory troubleshooting.

[1253] "System data" refers to technical information related to the operation of a terminal, including error logs and configuration information.

[1254] "Display information" refers to information captured from the content displayed on the device's screen, including screenshots and screen recordings.

[1255] "Emotional data" refers to digital data that indicates a user's emotional state, obtained by analyzing the user's voice and facial expressions.

[1256] A "server" is a computer system that can receive, analyze, and transmit data over a network.

[1257] "Analysis" is the process of thoroughly investigating and examining received data to identify problems and understand emotional states.

[1258] A "solution" is a proposed plan that includes specific action steps to address an identified problem, and is generated using adaptive information based on images, videos, text, audio, and emotions.

[1259] "Emotion-based adaptive information" refers to information that includes explanations and support messages tailored to the user's emotional state (e.g., frustration or anxiety).

[1260] "To collect" means the act of selecting and gathering necessary data.

[1261] "To transmit" refers to the act of transferring data to another device or system.

[1262] "Generating" is the act of creating new data or information based on specific input.

[1263] "Providing" means the act of handing over information, such as generated solutions, in a form that users can use.

[1264] This invention is a system that assists customers in troubleshooting their smartphones, and by combining it with an emotion engine that recognizes the user's emotions, it provides more efficient and satisfying support. This system consists of the following elements:

[1265] 1. Hardware Configuration

[1266] Terminal:

[1267] These are portable information processing devices such as smartphones and tablets. Users use these devices to report problems and receive support information.

[1268] The device is equipped with sensors and software to collect system data, display information (screenshots, screen recordings), and an emotion engine.

[1269] server:

[1270] This is a computer system that receives data transmitted from terminals via a network, performs analysis, and generates solutions. The server uses the latest image recognition technology and generative AI models to analyze emotional data from the emotion engine.

[1271] 2. Software Configuration

[1272] Emotional engine:

[1273] The system analyzes the user's voice and facial expressions to recognize their emotional state. This analysis uses an open-source emotion analysis library (e.g., "OpenFace").

[1274] Generative AI models:

[1275] Generative AI models such as "GPT-4" are used for data analysis and solution generation. These models can analyze error logs and screenshots to find appropriate solutions.

[1276] 3. Data Processing

[1277] Data collection:

[1278] When a user launches the support app and reports a problem, the device collects system data (error logs, configuration information) and display information.

[1279] The emotion engine analyzes the user's voice and facial expressions to generate emotion data.

[1280] Data transmission and analysis:

[1281] The device encrypts the data it collects and sends it to the server using a secure channel (e.g., HTTPS). The server decompresses and decodes the received data and analyzes it using a generative AI model.

[1282] Generating and providing solutions:

[1283] The server generates specific solution steps based on the analysis results. These solution steps include detailed explanations with images, videos, text, and audio.

[1284] Furthermore, based on emotional data, it generates adaptive support information tailored to the user's emotional state. For example, if a user is feeling frustrated, it will provide a more detailed explanation of solutions and add encouraging messages.

[1285] 4. Specific Examples

[1286] Examples of resolving app crashes

[1287] A user discovers a problem where a specific app repeatedly crashes and launches a support app to report the issue. The emotion engine analyzes the user's voice and facial expressions to recognize that they are feeling frustrated.

[1288] The device receives a problem report, collects error logs and screenshots from the crash, and sends them to the server along with sentiment data.

[1289] The server analyzes the received data and identifies that a specific library needs updating. Furthermore, because the user is experiencing frustration, the solution will be explained more clearly.

[1290] The server generates videos and text containing library update instructions, providing them to users in an easy-to-understand format. Encouraging messages are added to the videos.

[1291] The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[1292] 5. Example of a prompt statement

[1293] Identify and generate solutions for smartphone app crashes reported by users. Since users are frustrated, particularly clear and easy-to-understand explanations are essential.

[1294] Thus, the system of the present invention provides adaptive support that takes user emotions into consideration, thereby improving customer satisfaction.

[1295] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1296] Step 1:

[1297] Data collection

[1298] The user launches the support app on their smartphone, fills out the input form, and reports the problem.

[1299] Input: User problem report (e.g., "The app crashes").

[1300] The device automatically collects system data (error logs and configuration information) and takes screenshots and screen recordings.

[1301] The emotion engine analyzes the user's voice and facial expressions to generate emotion data (e.g., "frustration: high").

[1302] Output: System data, display information, sentiment data.

[1303] Step 2:

[1304] Sending data

[1305] The system data, display information, and emotional data collected by the device will be encrypted. AES-256 encryption will be used.

[1306] Inputs: System data, display information, sentiment data.

[1307] The device encrypts the data and converts it into secure data packets.

[1308] The device sends encrypted data to the server using a secure channel ("HTTPS").

[1309] Output: Encrypted data packets.

[1310] Step 3:

[1311] Data reception and analysis

[1312] The server receives encrypted data packets.

[1313] The server decompresses and decrypts the received data.

[1314] Input: Encrypted data packet.

[1315] The server uses a decompression tool ("gzip") to decompress the data and then decrypts it using the "AES-256" key.

[1316] Output: Decoded system data, display information, sentiment data.

[1317] The server uses a generated AI model (e.g., "GPT-4") to analyze the data and identify the cause of the problem.

[1318] Input: Decrypted system data, display information.

[1319] The server analyzes error logs and screenshots, and a generated AI model identifies that "a specific library needs to be updated."

[1320] The server analyzes emotional data to understand the user's emotional state (e.g., "frustration: high").

[1321] Output: Cause of the problem, emotional state.

[1322] Step 4:

[1323] Solution generation

[1324] The server generates specific troubleshooting steps based on the identified cause of the problem.

[1325] Input: The cause of the problem (e.g., "Library update required"), emotional state.

[1326] The server generates detailed descriptions using images, videos, text, and audio.

[1327] Example of a solution:

[1328] Video: "Tutorial video showing the procedure for updating the library"

[1329] Text: "Steps to select Settings > System > Update > Library Update"

[1330] Based on emotional data, the explanation will be made more detailed, and encouraging messages will be added.

[1331] Output: Solution steps information, encouraging message.

[1332] Step 5:

[1333] Sending and providing support information

[1334] The server packages the resolution procedure information it generates, encrypts it again, and sends it to the terminal.

[1335] Input: Solution steps information, encouraging messages.

[1336] The server compiles the resolution steps in JSON format, encrypts them, and packages them.

[1337] The server sends encrypted support information to the terminal.

[1338] Output: Encrypted support information package.

[1339] The terminal receives the support information package and performs decompression and decryption.

[1340] Input: Encrypted support information package.

[1341] The device uses a decryption tool to unpack and decrypt the data.

[1342] Output: Decrypted resolution information, encouraging message.

[1343] The device displays and plays troubleshooting information to the user through its user interface.

[1344] Input: Decrypted resolution step information, encouraging message.

[1345] The device displays text guides in the "Support" section within the app, while video guides play in the video player.

[1346] The device will apply an emotionally sensitive delivery method and display encouraging messages.

[1347] Output: Solution steps and encouraging messages provided to the user.

[1348] The user follows the provided instructions to resolve the issue.

[1349] Input: Solution steps information, encouraging messages.

[1350] The user performs the app update procedure and updates the necessary libraries.

[1351] The user confirms that the problem has been resolved and provides feedback.

[1352] Output: Resolved issues, user feedback.

[1353] (Application Example 2)

[1354] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1355] In conventional systems, when customers reported problems with their smartphones or robots, the system often provided a uniform solution that disregarded their emotional state. This could cause stress for both customers and technicians, making it difficult to provide appropriate and effective support. This invention aims to improve customer satisfaction by considering the emotional state of customers and technicians and providing quick and adaptive solutions.

[1356] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting data from the customer's terminal, means for transmitting the collected data to the server, means including an emotion engine for analyzing the customer's emotions, means for generating solutions to problems using images, videos, text, and audio, means for adaptively providing solutions based on the emotion engine, and means for providing the generated solutions to the customer's terminal. This makes it possible to provide solutions that correspond to the emotional state of the customer and the technician, enabling more effective and satisfying support.

[1357] A "terminal" refers to a portable information processing device, such as a smartphone or tablet, used by a user.

[1358] "Means of collecting data" refers to a function that automatically collects relevant system data (error logs, configuration information) from the device and also takes screenshots and screen recordings.

[1359] "Means of transmitting data" refers to the means of encrypting the collected data and sending it to the server through a secure communication channel.

[1360] "Means of analyzing data and identifying problems" refers to the function of analyzing received data to identify the cause of errors or troubles in networks and robotic systems.

[1361] An "emotion engine" is an engine that includes technology for recognizing a user's emotional state by analyzing their voice and facial expressions.

[1362] "Means for generating solutions" refers to a function that generates specific operating procedures and solutions based on analysis results, using images, videos, text, and audio.

[1363] "Means of adaptively providing solutions" refers to means of adjusting the method of providing solutions based on the user's emotional state recognized by the emotion engine.

[1364] "Means of providing solutions" refers to a function that displays the generated solutions on the user's device and provides them through audio or video.

[1365] To implement this invention, the following system is constructed. The system consists of a customer's terminal, a server, and software to link them together.

[1366] Hardware and software to be used

[1367] hardware

[1368] Customer devices: Portable information processing devices such as smartphones and tablets.

[1369] Robots used in factories

[1370] software

[1371] Emotion engines: Microsoft Azure Cognitive Services, Google Cloud Emotion API, etc.

[1372] Image recognition technology: OpenCV, TensorFlow

[1373] Generative AI model: OpenAI's GPT-4

[1374] Encryption and communication: TLS / SSL

[1375] Development environment: Android Studio, Xcode

[1376] Program processing details

[1377] 1. Data Collection

[1378] The customer's device automatically collects relevant system data (error logs, configuration information) when a problem occurs. The device also simultaneously captures screenshots and screen recordings. Furthermore, it uses an emotion engine to analyze the user's voice and facial expressions to recognize their emotional state. For example, if a malfunction is reported in a robot used in a factory, the device collects the robot's operation log and the technician's voice and facial expressions at that moment.

[1379] 2. Sending data

[1380] The collected data is encrypted and sent to the server via a secure communication channel (TLS / SSL). For example, robot anomaly logs and technician sentiment data are sent to the server simultaneously.

[1381] 3. Data analysis and problem identification

[1382] The server analyzes the received data. Using image recognition technologies (OpenCV, TensorFlow) and generative AI models (GPT-4), it identifies the cause of problems based on display information and system data. It also analyzes emotion data from the emotion engine to understand the user's emotional state. For example, the server can identify the cause of a robot malfunction and recognize that the technician is experiencing stress.

[1383] 4. Generating support information

[1384] Based on the cause of the problem, the server generates specific troubleshooting steps. These steps are detailed and include images, videos, text, and audio. Based on emotional data, the server adjusts how the troubleshooting steps are explained. For example, if a technician is feeling stressed, the server generates a more polite and easy-to-understand explanatory video and encouraging messages.

[1385] 5. Providing support information

[1386] Support information is encrypted and transmitted to the customer's device via a secure channel. The device verifies the received support information and provides it to the user through a user interface. For example, a technician's device might display a video and text showing the troubleshooting steps, and play an audio guide.

[1387] Specific example

[1388] For example, if a robot arm operating in a factory malfunctions, a technician can launch a smartphone app to report the problem. An emotion engine analyzes the technician's voice and facial expressions to determine if they are stressed, and a server analyzes the robot's operation logs to identify that a specific part needs replacing. The server generates a video and text of the replacement procedure and sends it to the technician's smartphone along with an encouraging message.

[1389] Examples of input prompts for a generative AI model

[1390] Identify the cause of the robot arm malfunction and provide a video and text-based explanation of the replacement procedure. Please include a message of encouragement, as the technicians are under stress.

[1391] Thus, the system of this invention can provide adaptive support based on the emotional state of customers and engineers, enabling more effective and satisfying solutions.

[1392] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1393] Step 1:

[1394] The user launches a smartphone app and reports a problem with the robot. The input includes a description of the problem (text) and images or videos of the problem occurring. The device collects this data and obtains robot operation logs and error logs. Simultaneously, the device's built-in emotion engine captures the user's voice and facial expressions to generate emotion data. The output includes system data, image / video data, and emotion data.

[1395] Step 2:

[1396] The device encrypts the collected data and sends it to the server using a secure communication channel (TLS / SSL). Inputs include system data, image / video data, and sentiment data, each encrypted before transmission. The output is the encrypted data received by the server. Specifically, the device encrypts the data, and the encrypted data is sent to the server over the network.

[1397] Step 3:

[1398] The server decompresses and decodes the received data. Input includes encrypted system data, image / video data, and sentiment data. The server decrypts these and outputs them as data ready for analysis. Specifically, the server decrypts the data and converts it into an analyzable format.

[1399] Step 4:

[1400] The server analyzes data using image recognition technology (OpenCV, TensorFlow) and a generative AI model (GPT-4). Inputs include decompressed system data, image / video data, and emotion data. The server identifies the cause of a problem and understands the user's emotional state. Outputs include specific cause information (text) and emotion analysis results (emotional state). The specific operation involves analysis using an AI model and recognition of the emotional state by an emotion engine.

[1401] Step 5:

[1402] The server generates solutions based on the analysis results. Inputs include specific cause information of the problem and sentiment analysis results. The server generates detailed solutions using images, videos, text, and audio, adapting the explanation method based on sentiment data. The output is a solution package (images, videos, text, audio) for the user. Specifically, the generating AI model generates explanatory videos and text, including adaptive explanations that respond to sentiment.

[1403] Step 6:

[1404] The generated solution package is encrypted and sent to the user's terminal via a secure communication channel. The input is the solution package (images, videos, text, audio). The data is encrypted upon reaching the terminal, and the output is the encrypted data sent from the server. Specifically, the server encrypts the data, and the encrypted data is sent to the user's terminal over the network.

[1405] Step 7:

[1406] The user's device verifies, decompresses, and decodes the received solution package. The input is an encrypted solution package. The device decrypts it and converts it into a format that can be displayed in the user interface. The output is a displayable solution (image, video, text, audio). Specifically, the device decrypts the data and provides the solution through the user interface.

[1407] Step 8:

[1408] The user resolves the problem by following the provided solutions. Input includes support information (images, videos, text, audio). The user then repairs or modifies the robot's settings based on this information. For example, instructions for replacing parts to ensure the robot's arm functions correctly are provided. Output includes feedback to the server via the terminal indicating that the problem has been resolved. Specific actions involve the user following instructions and reporting progress through the support application.

[1409] The above describes the specific processing flow and operation of the system program for realizing this invention.

[1410] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1411] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1412] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1413] [Fourth Embodiment]

[1414] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1415] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1416] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1417] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1418] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1419] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1420] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1421] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1422] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1423] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1424] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1425] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1426] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1427] overview

[1428] This invention is a system that assists in troubleshooting customers' smartphones. This system collects data from the customer's device, sends the collected data to a server, the server analyzes the received data to identify the problem, generates a solution to that problem, and provides it to the customer's device.

[1429] System operation

[1430] The present invention's system mainly consists of the following elements:

[1431] 1. Terminal:

[1432] A terminal is a portable information processing device such as a smartphone or tablet used by the user. When a user reports a problem, the terminal collects system data and display information (screenshots, screen recordings).

[1433] 2. Server:

[1434] A server is a computer system that receives data transmitted from terminals over a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[1435] System processing flow

[1436] 1. Data Collection

[1437] A user discovers a problem with their smartphone, launches a support app, and reports the issue.

[1438] The device receives user reports and automatically collects relevant system data (error logs, configuration information). Furthermore, it requests permission from the user to take screenshots or screen recordings to collect display information at the time the problem occurred.

[1439] 2. Sending data

[1440] The collected data is encrypted on the device and sent to the server using a secure channel. The system confirms that data transmission is complete and notifies the user of the successful transmission.

[1441] 3. Data Analysis

[1442] The server decompresses and decodes the received data. It verifies the correct format of the received data and analyzes it using a generative AI model. Based on display information and system data, it identifies the cause of the problem.

[1443] 4. Generating support information

[1444] After the cause of the problem is identified, the server generates a solution using images, videos, text, and audio. This solution includes specific steps and precautions.

[1445] 5. Providing support information

[1446] The generated support information is sent from the server to the terminal. The terminal analyzes the received data and provides it to the user through the user interface. The user can then resolve the problem according to the solution.

[1447] Specific example

[1448] Examples of resolving app crashes

[1449] 1. A user discovers a problem where a specific app repeatedly crashes and launches the support app to report the problem.

[1450] 2. The device receives the problem report and collects the error log and screenshots from the crash. The device encrypts the collected data and sends it to the server.

[1451] 3. The server analyzes the received data and identifies that a specific library needs to be updated.

[1452] 4. The server generates videos and text containing library update procedures, providing users with easy-to-understand instructions on how to perform the specific operations.

[1453] 5. The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[1454] In the form described above, the present invention is a system that enables efficient and intuitive troubleshooting of smartphones. This significantly reduces the time and effort required to resolve customer problems, thereby improving customer satisfaction.

[1455] The following describes the processing flow.

[1456] Step 1: Data Collection

[1457] User:

[1458] Users launch the support app when they encounter problems with their smartphone.

[1459] Tap the "Report a Problem" button in the support app and enter a summary of the problem or describe it verbally.

[1460] Terminal:

[1461] Upon receiving user input, the terminal automatically collects relevant system data (error logs and configuration information).

[1462] The device will display a message asking the user for permission to take screenshots or screen recordings.

[1463] If the user grants permission, the device will take screenshots and screen recordings when the problem occurs.

[1464] The collected data is saved on the device as a temporary file.

[1465] Step 2: Send

[1466] Terminal:

[1467] The collected data is encrypted and sent to the server using a secure channel.

[1468] Confirm that data transmission is complete and notify the user of its success.

[1469] Step 3: Data Analysis

[1470] server:

[1471] The received data is decompressed, and its format is verified.

[1472] The generative AI model is launched, and data analysis begins.

[1473] Image recognition technology is used to analyze display information (screenshots and screen recordings) and identify the area where the problem is displayed.

[1474] Analyze system data (error logs and configuration information) to identify the root cause of the problem.

[1475] Step 4: Generating support information

[1476] server:

[1477] Based on the cause of the problem, generate specific solution steps.

[1478] The troubleshooting steps are created in detail using images, videos, text, and audio. For example, a video of the app update procedure, screenshots of the settings change procedure, text instructions, and an audio guide.

[1479] The generated support information is packaged and prepared for transmission to the device.

[1480] Step 5: Providing support information

[1481] Terminal:

[1482] Receive support information sent from the server.

[1483] Verify the integrity of the received data and confirm that there are no problems.

[1484] Support information is displayed or played to the user through the user interface. For example, a video of the troubleshooting procedure or text explanation may be displayed.

[1485] User:

[1486] Follow the displayed solution steps.

[1487] If necessary, watch the provided explanatory videos and listen to the audio guide.

[1488] Confirm that the problem has been resolved and provide feedback through the support app.

[1489] In this way, the system of the present invention supports users in efficiently troubleshooting their smartphones.

[1490] (Example 1)

[1491] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1492] Modern information processing devices offer numerous functions, but the number of problems and issues users face is also increasing. These problems are often complex and difficult to resolve on their own. Furthermore, troubleshooting requires significant time and effort, leading to decreased user satisfaction. This invention aims to provide a system that efficiently and intuitively resolves problems users encounter with information processing devices, thereby reducing the burden on users.

[1493] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1494] In this invention, the server includes means for collecting data from the user's information processing device, means for transmitting the collected data to the processing device, and means for the processing device to analyze the received data and identify a problem. This enables the user to solve complex problems quickly and efficiently.

[1495] A "user" is an individual or group that uses an information processing device.

[1496] An "information processing device" is a device that includes hardware and software for collecting, transmitting, receiving, analyzing, and displaying data.

[1497] "Data" refers to various types of information and records within an information processing device, such as error logs, configuration information, screenshots, and screen recordings.

[1498] A "processing device" is a computer system that receives data, analyzes it, and generates results.

[1499] "Analysis" is the process of understanding the content of received data and identifying the cause of a problem.

[1500] A "solution" is information that includes steps and methods for a user to resolve a problem, based on the cause of the problem.

[1501] An "image" is a still image used to convey information visually.

[1502] A "video" is a dynamic medium that conveys visual information along a time axis.

[1503] "Text" refers to information expressed using characters.

[1504] "Sound" refers to sound signals that transmit information through hearing.

[1505] "To transmit" means to move information or data from one device or system to another.

[1506] "To collect" means to gather and incorporate necessary data and information.

[1507] Modes for carrying out the invention

[1508] overview

[1509] This invention is a system that assists in troubleshooting a user's information processing device (e.g., a smartphone or tablet). The system collects data from the user's device and sends the collected data to a server. The server analyzes the received data, identifies the problem the user is facing, generates a solution to that problem, and provides it to the user's device.

[1510] System Configuration

[1511] This system mainly consists of the following elements:

[1512] 1. User's device:

[1513] This refers to information processing devices such as smartphones and tablets used by users. When a user reports a problem, the device collects system data (error logs, configuration information) and display information (screenshots, screen recordings).

[1514] 2. Server:

[1515] This computer system receives data transmitted from terminals via a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[1516] Hardware and software to be used

[1517] 1. On the device side:

[1518] OS: Android or iOS

[1519] Log collection tools: "Logcat" for Android, "Console" for iOS

[1520] Cryptographic library: OpenSSL

[1521] Communication protocol: HTTPS

[1522] 2. Server side:

[1523] Data analysis tools: TensorFlow (image recognition), OpenCV (image analysis)

[1524] Generative AI models: BERT, GPT-3

[1525] Video processing tool: FFMPEG

[1526] System processing flow

[1527] Data collection

[1528] When a user discovers a problem with their smartphone, they launch the support app and report the issue. Upon receiving the report, the device automatically collects relevant system data (error logs, configuration information). It also collects screenshots and screen recordings. This allows for a detailed understanding of the user's specific problem.

[1529] Sending data

[1530] The collected data is encrypted on the device and sent to the server using HTTPS. The user is notified upon successful transmission. This ensures secure data transmission and timely communication with the user.

[1531] Data analysis and solution generation

[1532] The server decompresses the received data and parses the JSON formatted data. TensorFlow and OpenCV are used to analyze screenshots and screen recordings, and a generative AI model (such as GPT-3) is used to analyze error logs. This allows for rapid identification of user problems and generation of appropriate solutions.

[1533] Providing solutions

[1534] The generated solutions are sent from the server to the terminal in text, image, video, and audio formats. The server creates content containing detailed steps and notes for the solution, which the terminal displays in its user interface. The user refers to this content and follows the instructions to resolve the problem.

[1535] Specific examples and prompt statements

[1536] Specific example

[1537] For example, suppose a user reports that a particular app is repeatedly crashing. In this case, the device uses Logcat to collect error logs from the crash and also takes screenshots. This data is encrypted with OpenSSL and sent to the server via HTTPS. The server analyzes the received data and uses a generative AI model to identify that "a specific library needs to be updated." Then, it generates a video using FFMPEG explaining the library update procedure and provides it to the user.

[1538] Example of a prompt

[1539] "Please identify the cause of the problem from this error log."

[1540] In this way, this system can efficiently solve user problems by linking terminals and servers. By automating everything from data collection and analysis to solution generation and delivery, it is expected to significantly reduce the burden on users and improve their satisfaction.

[1541] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1542] Step 1:

[1543] Report the problem

[1544] Input: User reports of problems occurring on their smartphone (e.g., app crashes)

[1545] Action: The user launches the support app and clicks the "Report a problem" button.

[1546] Output: A problem report trigger is sent to the system.

[1547] Step 2:

[1548] Data collection

[1549] Input: User problem report

[1550] Action 1: The terminal automatically collects system data (e.g., error logs, configuration information).

[1551] For Android: Use the "Logcat" tool to collect error logs.

[1552] For iOS: Use the "Console" app to collect error logs.

[1553] Action 2: The device requests permission from the user to take screenshots or screen recordings, and collects them if the user grants permission.

[1554] Output: Collected system data and display information (screenshots, screen recordings) are generated.

[1555] Step 3:

[1556] Sending data

[1557] Input: Collected system data and display information

[1558] Action 1: The device encrypts the collected data.

[1559] We will use OpenSSL as the encryption library.

[1560] Action 2: Send encrypted data to the server via the HTTPS protocol.

[1561] Action 3: Confirm that data transmission is complete and notify the user if successful.

[1562] Output: The encrypted data is sent to the server, and the user receives a notification that "Data transmission complete."

[1563] Step 4:

[1564] Data reception and analysis

[1565] Input: Encrypted data

[1566] Action 1: The server decompresses and decodes the received data.

[1567] Use a decompression tool to extract the ZIP file and read the data file in JSON format.

[1568] Action 2: Verify that the received data is in the correct format.

[1569] Step 3: Analyze the error log using a generative AI model (e.g., GPT-3).

[1570] Enter the prompt message "Identify the cause of the problem from this error log" to obtain the analysis results.

[1571] Step 4: Analyze screenshots and screen recordings using image recognition technology (e.g., TensorFlow, OpenCV).

[1572] Output: The cause of the problem and, if necessary, additional information (image analysis results) are identified.

[1573] Step 5:

[1574] Solution generation

[1575] Input: Analysis results

[1576] Action 1: The server generates a solution using the generated AI model.

[1577] Enter the prompt message "Generate steps and precautions to resolve this issue" to obtain the solution.

[1578] Action 2: Create a video explaining the solution using FFMPEG as needed.

[1579] Output: Solutions in image, video, text, and audio formats are generated.

[1580] Step 6:

[1581] Providing solutions

[1582] Input: Generated solution (image, video, text, audio)

[1583] Action 1: The server encrypts the generated solution and sends it to the terminal via the HTTPS protocol.

[1584] Operation 2: The terminal decompresses the received data and provides it to the user through the user interface.

[1585] Action 3: The user solves the problem by following the solution.

[1586] Output: The user is provided with a concrete solution, and the problem is resolved.

[1587] (Application Example 1)

[1588] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1589] Because autonomous vehicles are controlled by numerous electronic devices and software, rapid and accurate troubleshooting is essential to ensure their proper operation. However, current troubleshooting of autonomous vehicles often requires on-site response by specialized technicians, which can be time-consuming and laborious. Especially in emergencies, a rapid response is crucial, and there is a need for a way to resolve the problem as quickly as possible.

[1590] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1591] In this invention, the server includes means for collecting data from the customer's terminal, means for transmitting the collected data to the server, means for analyzing the data received by the server to identify the customer's problem, means for generating a solution to the problem using images, videos, text, and audio, means for providing the generated solution to the customer's terminal, means for the central processing unit in the autonomous vehicle to collect and manage the data necessary for the execution of these means, and means for providing procedures for resolving problems in the autonomous vehicle. This enables a rapid and accurate response to problems in the autonomous vehicle, and allows the user to resolve the problem themselves.

[1592] A "terminal" refers to a portable information processing device used by a customer, such as a smartphone, tablet, in-car console display, or smart glasses.

[1593] A "server" is a computer system that receives data transmitted from terminals via a network, analyzes it, generates appropriate solutions, and provides them to the terminals.

[1594] A "data collection method" is a function that automatically collects relevant system data, sensor data, and software version information when a terminal receives a problem report.

[1595] "Data transmission means" refers to the function of encrypting collected data and sending it to the server using a secure channel.

[1596] "Data analysis means" refers to the function of a server that decompresses and decodes received data and uses a generated AI model to identify the cause of a problem.

[1597] A "solution generation method" is a function in which, after the cause of a problem has been identified, the server generates a solution that includes specific operating procedures and precautions using images, videos, text, and audio.

[1598] A "solution provisioning mechanism" is a function that sends a generated solution from a server to a terminal, which then receives it and provides it to the user through a user interface.

[1599] The "Central Processing Unit" is the main computing unit within an autonomous vehicle that collects and manages data and effectively executes various functions.

[1600] "Troubleshooting" refers to a series of processes for quickly identifying and addressing various problems that may occur in autonomous vehicles.

[1601] This invention is a system that assists in troubleshooting in autonomous vehicles. This system collects system data from the customer's terminal (e.g., an in-car console display or smart glasses), sends that data to a server for analysis, identifies problems, generates solutions, and provides them to the customer's terminal.

[1602] Specifically, this system works as follows:

[1603] When a user discovers a specific problem in an autonomous vehicle, they report it using the in-car console display or smart glasses. The device receives the report and automatically collects relevant system data (error logs, sensor data, software version information, etc.). Furthermore, it collects screenshots and screen recordings of display information and important messages.

[1604] The collected data is encrypted within the device and sent to the server via a secure communication channel. Once the data transmission is complete, the device notifies the user of the successful transmission.

[1605] The server decompresses and decodes the received data and analyzes it using a generative AI model. Based on the received system data and display information, the server identifies the cause of the problem.

[1606] After the cause of the problem is identified, the server generates a solution using images, videos, text, and audio. The solution includes specific operating procedures and precautions. For example, a solution for a sensor error might provide instructions for reinstalling the sensor driver, using both video and text.

[1607] The generated solution is sent from the server to the terminal, which then provides the received solution to the user through a user interface. The user can then follow the displayed instructions to resolve the problem with the autonomous vehicle.

[1608] Specific examples of hardware and software to be used

[1609] Hardware: Central computer in the autonomous vehicle, in-car console display, smart glasses

[1610] Software: Python program, encryption library (Fernet), HTTP library for data transmission (requests)

[1611] Examples of specific cases and prompt statements

[1612] As a concrete example, consider a scenario where a user discovers a sensor error in an autonomous vehicle and reports the problem via the console display. In this case, the system collects an error log and sends it to a server, which analyzes the error log and generates a procedure for reinstalling the sensor driver. This procedure is then displayed on the console display.

[1613] Example of a prompt:

[1614] text

[1615] A user detects a sensor error in an autonomous vehicle and reports the problem using the in-vehicle console display. The system collects the error log and sends it to the server. As a solution, instructions for reinstalling the sensor driver are generated and displayed on the in-vehicle display.

[1616] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1617] Step 1:

[1618] The user discovers a specific problem in the autonomous vehicle (e.g., a sensor error) and reports the problem using the in-vehicle console display or smart glasses. The input is the problem report information provided by the user. The output is a system confirmation notification indicating that a problem has been reported.

[1619] Step 2:

[1620] The device receives user problem reports and automatically collects relevant system data (error logs, sensor data, software version information, etc.). The input is the user problem report, and the output is the collected system data. The device also collects screenshots and screen recordings.

[1621] Step 3:

[1622] The terminal encrypts the data it collects and sends it to the server using a secure communication channel. The input is the collected system data, and the output is the encrypted data and a notification that the transmission is complete. The terminal uses an encryption library (Fernet) to protect the data and an HTTP library (requests) to send the data.

[1623] Step 4:

[1624] The server decompresses and decodes the received data. The input is encrypted system data, and the output is the decompressed and decoded data. The server verifies the integrity of the data and proceeds with appropriate analysis.

[1625] Step 5:

[1626] The server uses a generated AI model to analyze the decompressed data and identify the root cause of the problem. The input is the decompressed and decoded system data, and the output is the identified cause of the problem. The server uses the AI ​​model to analyze patterns in the data and find the root cause of the error.

[1627] Step 6:

[1628] Based on the cause of the problem identified by the server, a solution is generated using images, videos, text, and audio. The input is the identified cause of the problem, and the output is the content of the generated solution. The solution includes specific operating procedures and precautions.

[1629] Step 7:

[1630] The server sends the generated solution to the terminal. The input is the content of the generated solution, and the output is the data of the solution sent to the terminal. The server uses the HTTP protocol to securely transmit the data.

[1631] Step 8:

[1632] The terminal analyzes the received solution data and provides it to the user through a user interface. The input is the solution data sent from the server, and the output is the content of the solution displayed to the user. The terminal provides the solution to the user in an easy-to-understand format, such as video or text.

[1633] The coordination of the above processing steps enables efficient troubleshooting within autonomous vehicles. This also allows users to quickly obtain appropriate solutions and resolve problems.

[1634] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1635] overview

[1636] This invention is a system that assists in troubleshooting customers' smartphones and incorporates an emotion engine that recognizes user emotions. The system collects data from the customer's device, sends the collected data to a server, the server analyzes the received data to identify the problem, generates a solution to that problem, and provides it to the customer's device. It also features the ability to recognize user emotions and provide emotion-based adaptive support.

[1637] System operation

[1638] The system of the present invention consists of the following elements:

[1639] 1. Terminal:

[1640] A terminal is a portable information processing device such as a smartphone or tablet used by the user. When a user reports a problem, the terminal collects system data and display information (screenshots, screen recordings).

[1641] It is equipped with an emotion engine that analyzes the user's voice and facial expressions to recognize their emotions.

[1642] 2. Server:

[1643] A server is a computer system that receives data transmitted from terminals over a network and performs problem analysis and generates solutions. It uses the latest image recognition technology and generative AI models to identify problems and generate solutions.

[1644] Based on the emotional data transmitted from the emotion engine, the method of providing solutions is adjusted.

[1645] System processing flow

[1646] 1. Data Collection

[1647] The user discovers a problem with their smartphone and launches the support app to report the issue. If the user's voice and camera are enabled, the emotion engine analyzes their voice and facial expressions to recognize their emotional state.

[1648] The device automatically collects relevant system data (error logs, configuration information) and simultaneously takes screenshots and screen recordings. The emotion engine analyzes the user's voice and facial expressions to collect emotion data.

[1649] 2. Sending data

[1650] The device encrypts the collected data and sends it to the server using a secure channel. The transmitted data includes sentiment data. The device confirms that data transmission is complete and notifies the user of its success.

[1651] 3. Data Analysis

[1652] The server decompresses and decodes the received data. It verifies that the format of the received data is correct. It analyzes the data using a generative AI model and identifies the cause of problems based on display information and system data. It also analyzes emotion data from the emotion engine to understand the user's emotional state.

[1653] 4. Generating support information

[1654] The server generates specific troubleshooting steps based on the cause of the problem. These steps are detailed and include images, videos, text, and audio. For example, a video of the app update process, screenshots of settings changes, text instructions, and audio guides.

[1655] Based on emotional data, it generates adaptive support information tailored to the user's emotional state. For example, if a user is feeling frustrated, it provides a more polite and easy-to-understand explanation.

[1656] The generated support information is packaged and prepared for transmission to the device.

[1657] 5. Providing support information

[1658] The terminal receives support information sent from the server. It verifies the integrity of the received data and confirms that there are no problems. The support information is displayed or played back to the user through the user interface.

[1659] We adjust the delivery method based on emotional data. For example, if the emotional state is negative, we add a message of encouragement.

[1660] The user follows the displayed troubleshooting steps. If necessary, they watch the provided explanatory videos and listen to the audio guide. They then confirm that the problem has been resolved and provide feedback through the support app.

[1661] Specific example

[1662] Examples of resolving app crashes

[1663] 1. The user discovers a problem where a specific app repeatedly crashes and launches the support app to report the issue. The emotion engine analyzes the user's voice and facial expressions and recognizes that they are feeling frustrated.

[1664] 2. The device receives the problem report and collects the error log and screenshots from the crash. The device encrypts the collected data and sentiment data and sends it to the server.

[1665] 3. Analyze the data received by the server and identify that a specific library needs updating. Also, since the user is feeling frustrated, provide a more detailed explanation of the solution.

[1666] 4. The server generates videos and text containing library update instructions, providing them to the user in an easy-to-understand format. Encouraging messages are added to the videos.

[1667] 5. The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[1668] As described above, the system of the present invention, which combines an emotion engine, supports efficient troubleshooting of users' smartphones and aims to improve customer satisfaction.

[1669] The following describes the processing flow.

[1670] Step 1: Report the problem

[1671] User:

[1672] The user discovers a problem with their smartphone and launches a support app.

[1673] Tap the "Report a Problem" button in the support app and enter a summary of the problem or describe it verbally.

[1674] Step 2: Data Collection

[1675] Terminal:

[1676] Upon receiving user input, the terminal automatically collects relevant system data (error logs, configuration information).

[1677] At the same time, the user is asked for permission to take screenshots and screen recordings.

[1678] If the user grants permission, the device will take screenshots and screen recordings when the problem occurs.

[1679] The emotion engine analyzes the user's voice and facial expressions to collect emotional data.

[1680] The collected data and sentiment data are saved to a temporary file.

[1681] Step 3: Data transmission

[1682] Terminal:

[1683] The collected data and sentiment data are encrypted and sent to the server using a secure channel.

[1684] Confirm that data transmission is complete and notify the user of its success.

[1685] Step 4: Data Analysis

[1686] server:

[1687] Decompresses and decodes the received data.

[1688] Verify that the format of the received data is correct.

[1689] Start analyzing the data using a generative AI model.

[1690] Image recognition technology is used to analyze display information (screenshots and screen recordings) and identify the area where the problem is displayed.

[1691] Analyze system data (error logs and configuration information) to identify the root cause of the problem.

[1692] We analyze emotional data from the emotion engine to understand the user's emotional state.

[1693] Step 5: Generating support information

[1694] server:

[1695] Based on the cause of the problem, generate specific solution steps.

[1696] The solution steps are created in detail using images, videos, text, and audio.

[1697] Based on emotional data, it generates adaptive support information tailored to the user's emotional state.

[1698] For example, if a user is feeling frustrated, provide a more thorough and easy-to-understand explanation.

[1699] The generated support information is packaged and prepared for transmission to the device.

[1700] Step 6: Providing support information

[1701] Terminal:

[1702] Receive support information sent from the server.

[1703] Verify the integrity of the received data and confirm that there are no problems.

[1704] Support information is displayed and played for the user through the user interface.

[1705] The delivery method is adjusted based on emotional data; for example, if the emotional state is negative, an encouraging message is added.

[1706] Step 7: Problem Solving

[1707] User:

[1708] Follow the displayed solution steps.

[1709] If necessary, watch the provided explanatory videos and listen to the audio guide.

[1710] Confirm that the problem has been resolved and provide feedback through the support app.

[1711] (Example 2)

[1712] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1713] Conventional smartphone troubleshooting systems often provided only general explanations without considering the user's emotional state when resolving their problems. This could lead to user frustration and dissatisfaction, potentially lowering customer satisfaction. Furthermore, identifying the user's problem and generating solutions could be inefficient, resulting in prolonged problem-solving times. This invention aims to enable efficient and highly satisfying troubleshooting by recognizing the user's emotional state in real time and providing adaptive support based on that emotion.

[1714] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1715] In this invention, the server includes means for collecting system data, display information, and emotional data from the customer's terminal; means for transmitting the collected system data, display information, and emotional data to the server; means for analyzing the system data, display information, and emotional data received by the server to identify the customer's problem and emotional state; means for generating a solution to the problem using adaptive information based on images, videos, text, audio, and emotions; and means for providing the generated solution to the customer's terminal. This enables adaptive support that takes the user's emotions into consideration, resulting in efficient and highly satisfactory troubleshooting.

[1716] "System data" refers to technical information related to the operation of a terminal, including error logs and configuration information.

[1717] "Display information" refers to information captured from the content displayed on the device's screen, including screenshots and screen recordings.

[1718] "Emotional data" refers to digital data that indicates a user's emotional state, obtained by analyzing the user's voice and facial expressions.

[1719] A "server" is a computer system that can receive, analyze, and transmit data over a network.

[1720] "Analysis" is the process of thoroughly investigating and examining received data to identify problems and understand emotional states.

[1721] A "solution" is a proposed plan that includes specific action steps to address an identified problem, and is generated using adaptive information based on images, videos, text, audio, and emotions.

[1722] "Emotion-based adaptive information" refers to information that includes explanations and support messages tailored to the user's emotional state (e.g., frustration or anxiety).

[1723] "To collect" means the act of selecting and gathering necessary data.

[1724] "To transmit" refers to the act of transferring data to another device or system.

[1725] "Generating" is the act of creating new data or information based on specific input.

[1726] "Providing" means the act of handing over information, such as generated solutions, in a form that users can use.

[1727] This invention is a system that assists customers in troubleshooting their smartphones, and by combining it with an emotion engine that recognizes the user's emotions, it provides more efficient and satisfying support. This system consists of the following elements:

[1728] 1. Hardware Configuration

[1729] Terminal:

[1730] These are portable information processing devices such as smartphones and tablets. Users use these devices to report problems and receive support information.

[1731] The device is equipped with sensors and software to collect system data, display information (screenshots, screen recordings), and an emotion engine.

[1732] server:

[1733] This is a computer system that receives data transmitted from terminals via a network, performs analysis, and generates solutions. The server uses the latest image recognition technology and generative AI models to analyze emotional data from the emotion engine.

[1734] 2. Software Configuration

[1735] Emotional engine:

[1736] The system analyzes the user's voice and facial expressions to recognize their emotional state. This analysis uses an open-source emotion analysis library (e.g., "OpenFace").

[1737] Generative AI models:

[1738] Generative AI models such as "GPT-4" are used for data analysis and solution generation. These models can analyze error logs and screenshots to find appropriate solutions.

[1739] 3. Data Processing

[1740] Data collection:

[1741] When a user launches the support app and reports a problem, the device collects system data (error logs, configuration information) and display information.

[1742] The emotion engine analyzes the user's voice and facial expressions to generate emotion data.

[1743] Data transmission and analysis:

[1744] The device encrypts the data it collects and sends it to the server using a secure channel (e.g., HTTPS). The server decompresses and decodes the received data and analyzes it using a generative AI model.

[1745] Generating and providing solutions:

[1746] The server generates specific solution steps based on the analysis results. These solution steps include detailed explanations with images, videos, text, and audio.

[1747] Furthermore, based on emotional data, it generates adaptive support information tailored to the user's emotional state. For example, if a user is feeling frustrated, it will provide a more detailed explanation of solutions and add encouraging messages.

[1748] 4. Specific Examples

[1749] Examples of resolving app crashes

[1750] A user discovers a problem where a specific app repeatedly crashes and launches a support app to report the issue. The emotion engine analyzes the user's voice and facial expressions to recognize that they are feeling frustrated.

[1751] The device receives a problem report, collects error logs and screenshots from the crash, and sends them to the server along with sentiment data.

[1752] The server analyzes the received data and identifies that a specific library needs updating. Furthermore, because the user is experiencing frustration, the solution will be explained more clearly.

[1753] The server generates videos and text containing library update instructions, providing them to users in an easy-to-understand format. Encouraging messages are added to the videos.

[1754] The device receives the generated support information and displays it to the user through the user interface. The user updates the app's library by following the provided instructions and resolves the issue.

[1755] 5. Example of a prompt statement

[1756] Identify and generate solutions for smartphone app crashes reported by users. Since users are frustrated, particularly clear and easy-to-understand explanations are essential.

[1757] Thus, the system of the present invention provides adaptive support that takes user emotions into consideration, thereby improving customer satisfaction.

[1758] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1759] Step 1:

[1760] Data collection

[1761] The user launches the support app on their smartphone, fills out the input form, and reports the problem.

[1762] Input: User problem report (e.g., "The app crashes").

[1763] The device automatically collects system data (error logs and configuration information) and takes screenshots and screen recordings.

[1764] The emotion engine analyzes the user's voice and facial expressions to generate emotion data (e.g., "frustration: high").

[1765] Output: System data, display information, sentiment data.

[1766] Step 2:

[1767] Sending data

[1768] The system data, display information, and emotional data collected by the device will be encrypted. AES-256 encryption will be used.

[1769] Inputs: System data, display information, sentiment data.

[1770] The device encrypts the data and converts it into secure data packets.

[1771] The device sends encrypted data to the server using a secure channel ("HTTPS").

[1772] Output: Encrypted data packets.

[1773] Step 3:

[1774] Data reception and analysis

[1775] The server receives encrypted data packets.

[1776] The server decompresses and decrypts the received data.

[1777] Input: Encrypted data packet.

[1778] The server uses a decompression tool ("gzip") to decompress the data and then decrypts it using the "AES-256" key.

[1779] Output: Decoded system data, display information, sentiment data.

[1780] The server uses a generated AI model (e.g., "GPT-4") to analyze the data and identify the cause of the problem.

[1781] Input: Decrypted system data, display information.

[1782] The server analyzes error logs and screenshots, and a generated AI model identifies that "a specific library needs to be updated."

[1783] The server analyzes emotional data to understand the user's emotional state (e.g., "frustration: high").

[1784] Output: Cause of the problem, emotional state.

[1785] Step 4:

[1786] Solution generation

[1787] The server generates specific troubleshooting steps based on the identified cause of the problem.

[1788] Input: The cause of the problem (e.g., "Library update required"), emotional state.

[1789] The server generates detailed descriptions using images, videos, text, and audio.

[1790] Example of a solution:

[1791] Video: "Tutorial video showing the procedure for updating the library"

[1792] Text: "Steps to select Settings > System > Update > Library Update"

[1793] Based on emotional data, the explanation will be made more detailed, and encouraging messages will be added.

[1794] Output: Solution steps information, encouraging message.

[1795] Step 5:

[1796] Sending and providing support information

[1797] The server packages the resolution procedure information it generates, encrypts it again, and sends it to the terminal.

[1798] Input: Solution steps information, encouraging messages.

[1799] The server compiles the resolution steps in JSON format, encrypts them, and packages them.

[1800] The server sends encrypted support information to the terminal.

[1801] Output: Encrypted support information package.

[1802] The terminal receives the support information package and performs decompression and decryption.

[1803] Input: Encrypted support information package.

[1804] The device uses a decryption tool to unpack and decrypt the data.

[1805] Output: Decrypted resolution information, encouraging message.

[1806] The device displays and plays troubleshooting information to the user through its user interface.

[1807] Input: Decrypted resolution step information, encouraging message.

[1808] The device displays text guides in the "Support" section within the app, while video guides play in the video player.

[1809] The device will apply an emotionally sensitive delivery method and display encouraging messages.

[1810] Output: Solution steps and encouraging messages provided to the user.

[1811] The user follows the provided instructions to resolve the issue.

[1812] Input: Solution steps information, encouraging messages.

[1813] The user performs the app update procedure and updates the necessary libraries.

[1814] The user confirms that the problem has been resolved and provides feedback.

[1815] Output: Resolved issues, user feedback.

[1816] (Application Example 2)

[1817] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1818] In conventional systems, when customers reported problems with their smartphones or robots, the system often provided a uniform solution that disregarded their emotional state. This could cause stress for both customers and technicians, making it difficult to provide appropriate and effective support. This invention aims to improve customer satisfaction by considering the emotional state of customers and technicians and providing quick and adaptive solutions.

[1819] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting data from the customer's terminal, means for transmitting the collected data to the server, means including an emotion engine for analyzing the customer's emotions, means for generating solutions to problems using images, videos, text, and audio, means for adaptively providing solutions based on the emotion engine, and means for providing the generated solutions to the customer's terminal. This makes it possible to provide solutions that correspond to the emotional state of the customer and the technician, enabling more effective and satisfying support.

[1820] A "terminal" refers to a portable information processing device, such as a smartphone or tablet, used by a user.

[1821] "Means of collecting data" refers to a function that automatically collects relevant system data (error logs, configuration information) from the device and also takes screenshots and screen recordings.

[1822] "Means of transmitting data" refers to the means of encrypting the collected data and sending it to the server through a secure communication channel.

[1823] "Means of analyzing data and identifying problems" refers to the function of analyzing received data to identify the cause of errors or troubles in networks and robotic systems.

[1824] An "emotion engine" is an engine that includes technology for recognizing a user's emotional state by analyzing their voice and facial expressions.

[1825] "Means for generating solutions" refers to a function that generates specific operating procedures and solutions based on analysis results, using images, videos, text, and audio.

[1826] "Means of adaptively providing solutions" refers to means of adjusting the method of providing solutions based on the user's emotional state recognized by the emotion engine.

[1827] "Means of providing solutions" refers to a function that displays the generated solutions on the user's device and provides them through audio or video.

[1828] To implement this invention, the following system is constructed. The system consists of a customer's terminal, a server, and software to link them together.

[1829] Hardware and software to be used

[1830] hardware

[1831] Customer devices: Portable information processing devices such as smartphones and tablets.

[1832] Robots used in factories

[1833] software

[1834] Emotion engines: Microsoft Azure Cognitive Services, Google Cloud Emotion API, etc.

[1835] Image recognition technology: OpenCV, TensorFlow

[1836] Generative AI model: OpenAI's GPT-4

[1837] Encryption and communication: TLS / SSL

[1838] Development environment: Android Studio, Xcode

[1839] Program processing details

[1840] 1. Data Collection

[1841] The customer's device automatically collects relevant system data (error logs, configuration information) when a problem occurs. The device also simultaneously captures screenshots and screen recordings. Furthermore, it uses an emotion engine to analyze the user's voice and facial expressions to recognize their emotional state. For example, if a malfunction is reported in a robot used in a factory, the device collects the robot's operation log and the technician's voice and facial expressions at that moment.

[1842] 2. Sending data

[1843] The collected data is encrypted and sent to the server via a secure communication channel (TLS / SSL). For example, robot anomaly logs and technician sentiment data are sent to the server simultaneously.

[1844] 3. Data analysis and problem identification

[1845] The server analyzes the received data. Using image recognition technologies (OpenCV, TensorFlow) and generative AI models (GPT-4), it identifies the cause of problems based on display information and system data. It also analyzes emotion data from the emotion engine to understand the user's emotional state. For example, the server can identify the cause of a robot malfunction and recognize that the technician is experiencing stress.

[1846] 4. Generating support information

[1847] Based on the cause of the problem, the server generates specific troubleshooting steps. These steps are detailed and include images, videos, text, and audio. Based on emotional data, the server adjusts how the troubleshooting steps are explained. For example, if a technician is feeling stressed, the server generates a more polite and easy-to-understand explanatory video and encouraging messages.

[1848] 5. Providing support information

[1849] Support information is encrypted and transmitted to the customer's device via a secure channel. The device verifies the received support information and provides it to the user through a user interface. For example, a technician's device might display a video and text showing the troubleshooting steps, and play an audio guide.

[1850] Specific example

[1851] For example, if a robot arm operating in a factory malfunctions, a technician can launch a smartphone app to report the problem. An emotion engine analyzes the technician's voice and facial expressions to determine if they are stressed, and a server analyzes the robot's operation logs to identify that a specific part needs replacing. The server generates a video and text of the replacement procedure and sends it to the technician's smartphone along with an encouraging message.

[1852] Examples of input prompts for a generative AI model

[1853] Identify the cause of the robot arm malfunction and provide a video and text-based explanation of the replacement procedure. Please include a message of encouragement, as the technicians are under stress.

[1854] Thus, the system of this invention can provide adaptive support based on the emotional state of customers and engineers, enabling more effective and satisfying solutions.

[1855] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1856] Step 1:

[1857] The user launches a smartphone app and reports a problem with the robot. The input includes a description of the problem (text) and images or videos of the problem occurring. The device collects this data and obtains robot operation logs and error logs. Simultaneously, the device's built-in emotion engine captures the user's voice and facial expressions to generate emotion data. The output includes system data, image / video data, and emotion data.

[1858] Step 2:

[1859] The device encrypts the collected data and sends it to the server using a secure communication channel (TLS / SSL). Inputs include system data, image / video data, and sentiment data, each encrypted before transmission. The output is the encrypted data received by the server. Specifically, the device encrypts the data, and the encrypted data is sent to the server over the network.

[1860] Step 3:

[1861] The server decompresses and decodes the received data. Input includes encrypted system data, image / video data, and sentiment data. The server decrypts these and outputs them as data ready for analysis. Specifically, the server decrypts the data and converts it into an analyzable format.

[1862] Step 4:

[1863] The server analyzes data using image recognition technology (OpenCV, TensorFlow) and a generative AI model (GPT-4). Inputs include decompressed system data, image / video data, and emotion data. The server identifies the cause of a problem and understands the user's emotional state. Outputs include specific cause information (text) and emotion analysis results (emotional state). The specific operation involves analysis using an AI model and recognition of the emotional state by an emotion engine.

[1864] Step 5:

[1865] The server generates solutions based on the analysis results. Inputs include specific cause information of the problem and sentiment analysis results. The server generates detailed solutions using images, videos, text, and audio, adapting the explanation method based on sentiment data. The output is a solution package (images, videos, text, audio) for the user. Specifically, the generating AI model generates explanatory videos and text, including adaptive explanations that respond to sentiment.

[1866] Step 6:

[1867] The generated solution package is encrypted and sent to the user's terminal via a secure communication channel. The input is the solution package (images, videos, text, audio). The data is encrypted upon reaching the terminal, and the output is the encrypted data sent from the server. Specifically, the server encrypts the data, and the encrypted data is sent to the user's terminal over the network.

[1868] Step 7:

[1869] The user's device verifies, decompresses, and decodes the received solution package. The input is an encrypted solution package. The device decrypts it and converts it into a format that can be displayed in the user interface. The output is a displayable solution (image, video, text, audio). Specifically, the device decrypts the data and provides the solution through the user interface.

[1870] Step 8:

[1871] The user resolves the problem by following the provided solutions. Input includes support information (images, videos, text, audio). The user then repairs or modifies the robot's settings based on this information. For example, instructions for replacing parts to ensure the robot's arm functions correctly are provided. Output includes feedback to the server via the terminal indicating that the problem has been resolved. Specific actions involve the user following instructions and reporting progress through the support application.

[1872] The above describes the specific processing flow and operation of the system program for realizing this invention.

[1873] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1874] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1875] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1876] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1877] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1878] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1879] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1880] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1881] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1882] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1883] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1884] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1885] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1886] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1887] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1888] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1889] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1890] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1891] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1892] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1893] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1894] The following is further disclosed regarding the embodiments described above.

[1895] (Claim 1)

[1896] Means of collecting data from customer devices,

[1897] A means of sending the collected data to the server,

[1898] A means of analyzing the data received by the server to identify the customer's problem,

[1899] A means of generating solutions to problems using images, videos, text, and audio,

[1900] A means of providing the generated solution to the customer's terminal,

[1901] A system that includes this.

[1902] (Claim 2)

[1903] The system according to claim 1, wherein the collected data includes error logs and configuration information.

[1904] (Claim 3)

[1905] The system according to claim 1, wherein the solution generated based on the analysis results includes specific operating procedures.

[1906] "Example 1"

[1907] (Claim 1)

[1908] A means of collecting data from a user's information processing device,

[1909] Means for transmitting collected data to a processing device,

[1910] A means for the processing unit to analyze the received data and identify the problem,

[1911] A means of generating solutions using images, videos, text, and audio based on the analyzed data,

[1912] Means for providing the generated solution to the user's information processing device,

[1913] A system that includes this.

[1914] (Claim 2)

[1915] The system according to claim 1, wherein the collected data includes error logs and configuration information.

[1916] (Claim 3)

[1917] The system according to claim 1, wherein the solution generated based on the analysis results includes specific operating procedures.

[1918] "Application Example 1"

[1919] (Claim 1)

[1920] Means of collecting data from customer devices,

[1921] A means of sending the collected data to the server,

[1922] A means of analyzing the data received by the server to identify the customer's problem,

[1923] A means of generating solutions to problems using images, videos, text, and audio,

[1924] A means of providing the generated solution to the customer's terminal,

[1925] The central processing unit within the autonomous vehicle collects and manages the data necessary to perform these actions,

[1926] This provides a means to offer procedures for resolving problems with autonomous vehicles,

[1927] A system that includes this.

[1928] (Claim 2)

[1929] The system according to claim 1, wherein the collected data includes error logs, configuration information, sensor data, and software version information.

[1930] (Claim 3)

[1931] The system according to claim 1, wherein the solution generated based on the analysis results includes specific operating procedures and is presented through a console display or smart glasses.

[1932] "Example 2 of combining an emotion engine"

[1933] (Claim 1)

[1934] A means for collecting system data, display information, and sentiment data from a customer's terminal,

[1935] A means for transmitting collected system data, display information, and emotion data to a server,

[1936] A means for a server to analyze received system data, display information, and emotional data to identify customer problems and emotional states,

[1937] A means for generating solutions to problems using adaptive information based on images, videos, text, audio, and emotions,

[1938] A means of providing the generated solution to the customer's terminal,

[1939] A system that includes this.

[1940] (Claim 2)

[1941] The system according to claim 1, wherein the collected system data includes error logs and configuration information.

[1942] (Claim 3)

[1943] The system according to claim 1, wherein the solution generated based on the analysis results includes specific operating procedures and adaptive explanations tailored to the user's emotional state.

[1944] "Application example 2 when combining with an emotional engine"

[1945] (Claim 1)

[1946] Means of collecting data from customer devices,

[1947] A means of sending the collected data to the server,

[1948] A means of analyzing the data received by the server to identify the customer's problem,

[1949] A means including an emotion engine for analyzing customer emotions,

[1950] A means of generating solutions to problems using images, videos, text, and audio,

[1951] A means of adaptively providing solutions based on an emotional engine,

[1952] A means of providing the generated solution to the customer's terminal,

[1953] A system that includes this.

[1954] (Claim 2)

[1955] The system according to claim 1, wherein the collected data includes error logs and configuration information.

[1956] (Claim 3)

[1957] The system according to claim 1, wherein the solution generated based on the analysis results includes specific operating procedures. [Explanation of Symbols]

[1958] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means of collecting data from customer devices, A means of sending the collected data to the server, A means of analyzing the data received by the server to identify the customer's problem, A means of generating solutions to problems using images, videos, text, and audio, A means of providing the generated solution to the customer's terminal, A system that includes this.

2. The system according to claim 1, wherein the collected data includes error logs and configuration information.

3. The system according to claim 1, wherein the solution generated based on the analysis results includes specific operating procedures.

Images