System

Abstract

A system is provided.SOLUTION: A system including a means for detecting a plurality of applications executed by a user, a means for collecting interaction data and situation information of the user from each of the detected applications, a means for operating an AI model generated based on the collected data to analyze and infer an interaction between the applications, a means for generating automatic completion input data for each of the applications based on an analysis result, and a means for transmitting the generated automatic completion input data to each of the applications.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current xR applications are often designed to be launched in a single task, and in environments where multiple applications are used simultaneously, there is a lack of collaboration between the applications. This means that while a user is using one application, they are unable to share information or link with other applications. In such situations, the user experience is limited, and the potential of each application is not fully realized. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for detecting multiple applications run by a user, a means for collecting user interaction data and situation information from each detected application, a means for running a generative AI model based on the collected data to analyze and predict interactions between applications, a means for generating auto-complete input data for each application based on the analysis results, and a means for sending the generated auto-complete input data to each application. This system analyzes interactions when multiple xR applications are used simultaneously and auto-completes user input for each application, thereby realizing collaboration between applications. As a result, the user experience can be improved and the potential of each application can be enhanced.

[0006] "User" means the end user who runs and operates a computer system or application.

[0007] "Application" means a software program designed to provide a particular function or service.

[0008] A "generative AI model" refers to an algorithm or framework that uses artificial intelligence technology to analyze data and make predictions or inferences.

[0009] "Interaction data" refers to data related to operations and inputs made by a user when using an application.

[0010] "Status information" refers to data about the execution state or current status of an application.

[0011] "Interaction" refers to the relationship between information and actions between multiple applications.

[0012] "Analysis" refers to the process of finding meaning from collected data and making sense of that information.

[0013] "Inference" refers to the process of predicting unknown information from given data.

[0014] "Auto-complete input data" refers to data generated to complete a user's input based on the results of analysis by a generative AI model.

[0015] "Federation" refers to different applications working together to provide an integrated user experience. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model, and generates auto-complete input data for each application, realizing collaboration between applications.

[0038] Server Processing

[0039] 1. Application Discovery

[0040] The server communicates with the user's device to detect each application the user is using. This allows the server to obtain information about all applications running on the device. For example, if a user is using a virtual movie theater app and a chat app at the same time, that information is notified to the server.

[0041] 2. Data Collection

[0042] The server then collects user interaction data and context information from each detected application, including movie scene information from a virtual cinema app and user comment history from a chat app.

[0043] 3. Generative AI model in action

[0044] A generative AI model is run based on the collected data. For example, movie scene information and utterances from a chat app are used as input. The model analyzes and infers the data to generate topics and questions related to movie scenes.

[0045] 4. Input completion data generation

[0046] The server generates auto-complete input data for each application based on the analysis results of the generative AI model. For example, questions and topics related to movie scenes are generated for a chat app.

[0047] 5. Data Transmission

[0048] Finally, the server sends the generated auto-complete input data to each application, allowing the chat app to suggest new topics and questions related to movie scenes to the user.

[0049] Terminal handling

[0050] 1. Receiving notifications

[0051] The user's device receives the auto-complete input data sent from the server, which allows the new data to be reflected in real time.

[0052] 2. Data Application

[0053] The device applies the received data to each application. For example, it might display topics related to movie scenes in a chat app to encourage user interaction. It might also update the application's UI and notify the user of new input.

[0054] User Action

[0055] 1. App Usage

[0056] Users can use multiple xR applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[0057] 2. Complementary Experiences

[0058] Users experience auto-complete interactions with each application, for example, sharing topics with friends based on a question related to a movie scene displayed in a chat app, resulting in a more natural and rich user experience.

[0059] Specific examples

[0060] Example 1: Linking a virtual movie theater app with a chat app

[0061] Consider a case where a user is watching a movie on a virtual cinema app while chatting with a friend on a chat app. The server collects information about movie scenes and the content of comments made in the chat app. The generative AI model uses this data to generate questions and topics related to the movie scenes. The generated data is sent to the chat app, which receives it and presents it to the user. This allows the user to share topics based on movie scenes with their friends.

[0062] Example 2: Integration of an educational app and a note-taking app

[0063] Consider a case where a user is taking a class using an educational app while recording what they learned using a note-taking app. The server collects the class content data and the contents of the notes. The generative AI model uses this data to generate supplementary information to be added to the notes and hints for related problems. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, the user can easily record supplementary information related to the class.

[0064] Through these processes, the system of the present invention realizes information sharing and collaboration between multiple applications, improving the user experience.

[0065] The processing flow will be explained below.

[0066] Server Processing

[0067] Step 1:

[0068] Application Discovery

[0069] The server communicates with the user's device to discover all currently running applications, specifically by calling the device's API to retrieve the list of active applications.

[0070] Step 2:

[0071] Data collection

[0072] The server collects user interaction data and situation information from each detected application. Specifically, it obtains movie playback time and scene information from the movie theater app, and speech history from the chat app. This is also done through each application's API.

[0073] Step 3:

[0074] Generative AI model in action

[0075] The server inputs the collected data and runs a generative AI model. Specifically, it inputs movie scene information and speech history into the model, analyzes them, and predicts interactions.

[0076] Step 4:

[0077] Input completion data generation

[0078] Based on the analysis results of the generative AI model, auto-complete input data is generated for each application. For example, it generates chat topics related to movie scene information.

[0079] Step 5:

[0080] Data transmission

[0081] The server sends the generated auto-complete input data to each application, again via the API of each application.

[0082] Terminal handling

[0083] Step 1:

[0084] Receive notifications

[0085] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, it periodically checks communications from the server and acquires new data.

[0086] Step 2:

[0087] Data Application

[0088] The device applies the received data to the corresponding application, for example updating the UI to display new topics or questions in a chat app.

[0089] User Action

[0090] Step 1:

[0091] App use

[0092] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[0093] Step 2:

[0094] Complementary Experiences

[0095] Users experience auto-completed interactions, specifically continuing conversations with friends based on topics and questions related to movie scenes displayed in the chat app.

[0096] In this way, through the specific operations at each step, cooperation between the server, terminal, and user is realized, providing a richer user experience.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] Modern users often use multiple applications simultaneously and require enhanced user experiences through information sharing and collaboration between these applications. However, data sharing and auto-completion input between different applications is often performed manually, making it difficult to provide an efficient and natural user experience. For this reason, there is a demand for a system that automatically shares information and collaborates between multiple applications to improve user interaction.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes means for detecting multiple applications run by a user, means for collecting user interaction data and situation information from each detected application, means for running a generative AI model based on the collected data to analyze and predict interactions between applications, means for generating auto-complete input data for each application based on the analysis results, and means for transmitting the generated auto-complete input data to each application, thereby enabling automatic information sharing and collaboration between different applications to provide an efficient and natural user experience.

[0102] "User" refers to a person or entity that uses a system or application.

[0103] "Application" is a general term for a software program that a user runs on a terminal.

[0104] "Interaction Data" refers to data regarding inputs and interactions when a user operates an application.

[0105] "Status information" refers to information related to the user's usage environment and status, including, for example, the application being used and the status of its content.

[0106] A "generative AI model" refers to an artificial intelligence model that analyzes collected data and automatically generates input data.

[0107] "Auto-complete input data" refers to the input data generated by the generative AI model to be applied to each application.

[0108] "Server" refers to a computer system that communicates with a user's terminal, collects data, analyzes the data, and transmits the generated data.

[0109] "Terminal" refers to a device that a user uses to run an application, and examples include a computer and a smartphone.

[0110] "API" stands for Application Programming Interface and refers to a means by which different pieces of software communicate with each other.

[0111] An "HTTP POST request" is a part of the HTTP protocol used to send data to a server.

[0112] This invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model, and generates auto-complete input data for each application, realizing collaboration between applications.

[0113] This system mainly consists of three main components: a server, a terminal, and a user.

[0114] Server Roles

[0115] The server first communicates with the user's terminal and detects each application the user is running. For example, the netstat or ps command is used to obtain information about applications running on the terminal. This information is sent to the server using the UDP or TCP protocol. For example, the netstat -an | grep 'ESTABLISHED' command can be used to identify connected applications.

[0116] Next, the server collects user interaction data and context information from each detected application. The collected data is obtained through API calls and log data analysis. For example, movie scene information can be collected from a virtual movie theater app, and user comment history can be collected from a chat app.

[0117] Based on the collected data, the server runs a generative AI model. For example, GPT-4 (registered trademark) can be used as the generative AI model. By preprocessing the data using Python and inputting it into the generative AI model, topics and questions related to the movie scenes can be generated using movie scene information and comments from the chat app as input.

[0118] Next, the server generates auto-complete input data based on the analysis results of the generative AI model. For example, if the generative AI model generates a "question about a movie scene," it converts it into JSON format so that the chat app can understand it. This completion data is sent to each application using an HTTP POST request. For example, you can POST the data to http: / / chatapp.example.com / api / v1 / messages to reflect the new question in the chat app.

[0119] Device Role

[0120] The user's device receives the auto-complete input data sent from the server. You can receive the data in real time using a listener in the device (for example, a WebSocket server). Create an onMessage event handler to process the data as soon as it arrives.

[0121] The device then applies the received data to each application, for example updating the UI of a chat app to show the new question, or if you're using React, updating the state and re-rendering the component to show the new question.

[0122] User Roles

[0123] Users use multiple applications simultaneously. For example, they may watch a movie on a virtual cinema app while chatting with friends on a chat app. As users interact with the applications, their activities are sent to a server and data is collected.

[0124] The collected data is then automatically filled in and applied to the device, allowing users to enjoy new experiences. For example, a user can share a topic with a friend based on a question related to a movie scene displayed in a chat app. This allows the user's conversation to progress more naturally, resulting in richer communication.

[0125] Specific examples

[0126] Example 1: Linking a virtual movie theater app with a chat app

[0127] When a user watches a movie in a virtual cinema app and chats with a friend in a chat app, the server collects information about the movie's scenes and the content of the chat app's comments. The generative AI model uses this data to generate questions and topics related to the movie's scenes. The generated data is sent to the chat app, which receives it and presents it to the user. In this way, the user can share topics based on the movie's scenes with their friends.

[0128] Examples of specific prompts include:

[0129] "What do you think about the characters in the scene you're watching?"

[0130] "Let's talk about the events behind this scene."

[0131] Example 2: Integration of an educational app and a note-taking app

[0132] When a user takes a class using an educational app and records their learning using a note-taking app, the server collects the class content data and the contents of the notes. The generative AI model uses this data to generate supplementary information to be added to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, users can easily record supplementary information related to the class.

[0133] Examples of specific prompts include:

[0134] "Please note down any important points related to this topic."

[0135] "Let's dig deeper into the ideas explained in class."

[0136] Through these processes, the system of the present invention can realize information sharing and collaboration between multiple applications, thereby improving the user experience.

[0137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0138] Step 1:

[0139] The server detects multiple applications that a user runs.

[0140] Specifically, the server communicates with the user's terminal via TCP or UDP protocol and uses the netstat or ps command to collect information about running applications. The information obtained includes the application name, version, and status information indicating that it is running. Based on this information, it identifies which applications are currently running.

[0141] Input: Application execution status on user device

[0142] Output: Information about the running application (name, version, status, etc.)

[0143] Step 2:

[0144] The server collects user interaction data and context information from each detected application.

[0145] Specifically, the server uses the API of each application to obtain information such as what operations the user performed and what content is being played. For example, it collects information about the currently playing scene from a virtual movie theater app, and the user's comment history from a chat app. This information is collected by calling APIs and analyzing log files.

[0146] Input: Running application information

[0147] Output: Interaction data and context information for each application

[0148] Step 3:

[0149] The server runs a generative AI model based on the collected data.

[0150] Specifically, the server preprocesses the data using Python and inputs it into a generative AI model (e.g., GPT-4). The input data includes movie scene information and comments from a chat app. The generative AI model analyzes this data and generates appropriate complementary information (e.g., new questions or topics).

[0151] Input: Interaction data and context information

[0152] Output: Complementary information (e.g., questions and topics) generated by the generative AI model

[0153] Step 4:

[0154] The server generates input completion data based on the analysis results of the generative AI model.

[0155] Specifically, the server converts the data obtained from the generative AI model into an appropriate format, such as JSON, and formats it in a way that each application can understand. For example, it converts the generated question into a format like {"type": "question", "content": "Who are the characters in this scene?"}.

[0156] Input: Analysis results of the generative AI model

[0157] Output: Input completion data applicable to each application

[0158] Step 5:

[0159] The server transmits the generated auto-completion input data to each application.

[0160] Specifically, the server sends data using an HTTP POST request, for example to http: / / chatapp.example.com / api / v1 / messages, to reflect a new question in the chat app.

[0161] Input: Input completion data

[0162] Output: Data sent to each application

[0163] Step 6:

[0164] The user's terminal receives the auto-complete input data sent from the server.

[0165] Specifically, the device receives data in real time using a locally running listener (e.g., a WebSocket server). When the onMessage event handler receives the data, it executes the appropriate process.

[0166] Input: Data sent to each application

[0167] Output: Type-ahead data received on the device

[0168] Step 7:

[0169] The terminal applies the received data to each application and notifies the user.

[0170] Specifically, the device reflects the received data in the UI of the corresponding application and notifies the user of new input. For example, if you are using React, it will update the state and re-render the component displaying the new question.

[0171] Input: Received type-ahead data

[0172] Output: Updated application UI, notification to the user

[0173] Step 8:

[0174] The user uses the application and experiences the autocompleted interaction.

[0175] Specifically, users can engage in conversations with friends based on questions related to movie scenes presented in a chat app, resulting in a more natural and enriching user experience.

[0176] Input: Updated application UI

[0177] Output: Improved user interaction and experience

[0178] (Application example 1)

[0179] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0180] When users use multiple content distribution services on the Internet, each service operates independently, making it difficult to provide a unified information experience. It has also been pointed out that there is a lack of means to provide real-time updates or additional content related to the media content being viewed or listened to. This results in a fragmented experience, as users are unable to link information between content or receive seamless information.

[0181] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0182] In this invention, the server includes means for detecting multiple programs run by a user, means for collecting user interaction data and situation information from each detected program, means for running a generative AI model based on the collected data to analyze and predict interactions between programs, means for generating auto-complete input data for each program based on the analysis results, means for sending the generated auto-complete input data to each program, and means for automatically generating news and additional information related to the media content being viewed or listened to in cooperation with the content distribution service to which each program belongs. This enables information related to the media content being viewed or listened to to be provided in real time when a user is using multiple content distribution services, thereby realizing a seamless information experience.

[0183] A "program" is a set of instructions or code that runs on a computer to accomplish a particular task or function.

[0184] "Interaction data" is data generated when a user operates a program, and includes operation logs such as clicks, taps, inputs, and viewing time.

[0185] "Contextual information" is data that describes the context or conditions identified during program execution, including the user's geographic location, device state, current activity, etc.

[0186] A "generative AI model" is an artificial intelligence algorithm that analyzes patterns and relationships based on collected data and makes predictions and inferences. In this case, it is used specifically for natural language generation and data completion.

[0187] "Auto-complete input data" refers to suggested data or additional information generated by a generative AI model and used as input data for a specific program.

[0188] A "content distribution service" is a service that provides users with multimedia content such as videos, music, and news via the Internet.

[0189] "Related news and additional information" refers to information or content that is related to the media content that the user is watching or listening to and is provided in a manner that complements it.

[0190] MODE FOR CARRYING OUT THE INVENTION

[0191] This invention relates to a system that analyzes the interactions between multiple programs when a user is using them simultaneously, improving the user experience. The system uses a generative AI model to automatically generate related information based on interaction data and context information, and establishes collaboration between the programs.

[0192] Server Processing

[0193] The server has the following means:

[0194] 1. Program detection methods

[0195] It communicates with the user's device and detects the multiple programs the user is running. This method identifies all programs running on the device. For example, if a user is using a video streaming service and a news app at the same time, that information is notified to the server.

[0196] 2. Data Collection Methods

[0197] For each program detected, it collects user interaction data and contextual information, such as information about the video being watched, browsing history in news apps, etc.

[0198] 3. Means of running the generative AI model

[0199] Based on the collected data, a generative AI model is run to analyze the interactions between programs. The generative AI model analyzes the user's interaction data and generates related information based on it. For example, it generates additional information or topics related to the movie or news being watched.

[0200] 4. Auto-complete input data generation method

[0201] Based on the analysis, it generates auto-complete entries for each program, including news and additional information related to the content being viewed.

[0202] 5. Data Transmission Method

[0203] The generated auto-complete input data is sent to each program, which allows the news app to present new related information and news articles to the user.

[0204] Terminal handling

[0205] The terminal has the following means:

[0206] 1. Notification Reception Method

[0207] Receives auto-complete input data sent from the server, allowing the device to reflect new data in real time.

[0208] 2. Data application method

[0209] The received data is applied to each program and notified to the user, allowing the news app to display relevant news and additional information to the user in real time.

[0210] User Action

[0211] The user has the following means:

[0212] 1. How to use the program

[0213] Use multiple programs at the same time. For example, watch a movie on a streaming service while checking the latest news on a news app.

[0214] 2. Complementary Experiences

[0215] Experience auto-complete information in each program. For example, you can browse more interesting topics based on movie-related news displayed in a news app.

[0216] Specific examples

[0217] As a specific example, consider a case where while a user is watching a "documentary about climate change" on a video streaming service, the system of this invention automatically generates and displays news articles about "the latest research results on climate change" and "the latest trends in the environmental protection movement" in a news app. In this way, additional information that is closely related to the content the user is watching can be provided in real time.

[0218] Example prompt sentence:

[0219] Input: Viewing data for the video "Documentary on Climate Change"

[0220] Output: Related news articles (title, content)

[0221] This allows users to seamlessly experience related information and news while using content distribution services.

[0222] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0223] Step 1:

[0224] Program Detection

[0225] The server communicates with the user's terminal and detects all programs the user is running.

[0226] Input: A list of running programs from the user's terminal

[0227] Output: Detected program information

[0228] Specific operation: A monitoring agent installed on the user's terminal collects information about running programs and sends it to the server.

[0229] Step 2:

[0230] Data collection

[0231] The server collects user interaction data and context information from each detected program.

[0232] Input: Detected program information

[0233] Output: Interaction data and context information from each program

[0234] Specific operations: Collects data such as video playback time, content being viewed, and user operation logs.

[0235] Step 3:

[0236] Generative AI model in operation

[0237] The server runs a generative AI model based on the collected data and analyzes interactions between programs.

[0238] Input: Interaction data and context information

[0239] Output: Analysis results (related news and additional information)

[0240] Specific operation: The collected data is input into a generative AI model for analysis. The generative AI model used here is GPT-3 (registered trademark), etc.

[0241] Step 4:

[0242] Auto-complete input data generation

[0243] The server generates auto-complete input data for each program based on the analysis results of the generative AI model.

[0244] Input: Analysis results

[0245] Output: Auto-complete input data

[0246] Specific operation: Based on the generated data, templates for news articles and additional information are created and their content is automatically generated.

[0247] Step 5:

[0248] Data transmission

[0249] The server transmits the generated auto-completion input data to each program.

[0250] Input:Auto-complete input data

[0251] Output: Data to send to each program

[0252] Specific operation: Encode and send the generated data in an appropriate format to the corresponding program in real time.

[0253] Step 6:

[0254] Receive notifications

[0255] The terminal receives the auto-completion input data sent from the server.

[0256] Input: Data to be sent from the server

[0257] Output: Received complementary data

[0258] Specific operation: The program on the terminal receives data from the server and stores the contents in temporary storage.

[0259] Step 7:

[0260] Data Application

[0261] The terminal applies the received data to each program and notifies the user.

[0262] Input: Received complementary data

[0263] Output: Update information for each program

[0264] Specific behavior: Update the UI of each program to apply the received data and notify the user that new information is available.

[0265] Step 8:

[0266] Program Use

[0267] A user may use multiple programs simultaneously.

[0268] Input: Related news and additional information

[0269] Output: User Interaction

[0270] Specific operations: A user watches a movie on a video streaming service while checking related information on a news app.

[0271] Step 9:

[0272] Complementary Experiences

[0273] Users experience auto-completed information in each program.

[0274] Input: User interaction

[0275] Output: Improving the user experience

[0276] Specific behavior: By reading news articles related to a movie displayed in a news app, you can deepen your understanding of the movie's content.

[0277] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0278] This invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model and an emotion engine, and generates auto-complete input data for each application, realizing collaboration between applications.

[0279] Server Processing

[0280] 1. Application Discovery

[0281] The server communicates with the user's device to detect all currently running applications. Specifically, it calls the device's API to obtain a list of active applications. For example, if a user is using a virtual movie theater app and a chat app at the same time, that information is notified to the server.

[0282] 2. Data Collection

[0283] The server then collects user interaction data and context information from each detected application, such as movie scene information from a virtual cinema app or user comment history from a chat app, and provides the collected data to a generative AI model and emotion engine.

[0284] 3. Generative AI model and emotion engine in action

[0285] The server uses the collected data as input to run a generative AI model and an emotion engine. The generative AI model analyzes movie scene information and chat app comment history to infer interactions. Meanwhile, the emotion engine analyzes the user's emotions from the user's interaction data and situational information. For example, the emotion engine infers the user's emotional state from the user's tone of voice and text content.

[0286] 4. Input completion data generation

[0287] Based on the analysis results of the generative AI model and emotion engine, auto-complete input data for each application is generated. For example, data containing appropriate tone and topics according to the user's emotional state is generated, in addition to casual conversation topics related to movie scene information.

[0288] 5. Data Transmission

[0289] Finally, the server sends the generated auto-complete input data to each application, allowing the chat app to present new topics and questions to the user that are adapted to the movie scenes and the user's emotions.

[0290] Terminal handling

[0291] 1. Receiving notifications

[0292] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, it periodically checks communications from the server and acquires new data.

[0293] 2. Data Application

[0294] The device applies the received data to the corresponding application, for example, displaying a new topic or question in a chat app, or updating the application's UI to notify the user of the new input.

[0295] User Action

[0296] 1. App Usage

[0297] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[0298] 2. Complementary Experiences

[0299] Users experience auto-complete interactions, such as continuing a conversation with a friend based on a movie scene displayed in a chat app or topics and questions related to the user's emotions, resulting in a more natural and rich user experience.

[0300] Specific examples

[0301] Example 1: Linking a virtual movie theater app with a chat app

[0302] Consider a case where a user is watching a movie on a virtual cinema app while chatting with a friend on a chat app. The server collects information about movie scenes, comments made on the chat app, and the user's emotional data analyzed by the emotion engine. Based on this data, the generative AI model and emotion engine generate questions and topics related to the movie scenes and adapted to the user's emotional state. The generated data is sent to the chat app, which receives it on the device and presents it to the user. In this way, the user can share topics based on the movie scenes and their own emotions with their friends.

[0303] Example 2: Integration of an educational app and a note-taking app

[0304] Consider a case where a user takes a class using an educational app while recording their learning using a note-taking app. The server collects the class content data, the note content, and the user's emotional data analyzed by the emotion engine. Based on this data, the generative AI model and emotion engine generate supplementary information to add to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, users can not only easily record supplementary information related to the class, but also receive optimal learning experiences tailored to their emotional state.

[0305] Through these processes, the system of the present invention enables information sharing and collaboration between multiple applications, not only improving the user experience but also providing optimal interactions according to the user's emotional state.

[0306] The processing flow will be explained below.

[0307] Server Processing

[0308] Step 1:

[0309] Application Discovery

[0310] The server communicates with the user's device to discover all currently running applications, specifically by calling the device's API to retrieve the list of active applications.

[0311] Step 2:

[0312] Data collection

[0313] The server collects user interaction data and context information from each detected application. Specifically, it obtains movie playback time and scene information from the virtual cinema app, and speech history from the chat app. It also collects the user's voice tone and text content as input data for the emotion engine.

[0314] Step 3:

[0315] Generative AI model and emotion engine in action

[0316] The server uses the collected data as input to run a generative AI model and an emotion engine. The generative AI model analyzes movie scene information and chat app comment history to predict interactions. The emotion engine analyzes the user's voice tone and text content to evaluate the user's emotional state.

[0317] Step 4:

[0318] Applying Emotion Data

[0319] Based on the analysis results of the emotion engine, the generative AI model adjusts the auto-complete input data for each application. For example, if the user is excited, it generates input data that is appropriate for that excited state.

[0320] Step 5:

[0321] Input completion data generation

[0322] Based on the analysis results of the generative AI model and emotion engine, auto-complete input data for each application is generated. For example, data is generated that includes not only casual conversation topics related to movie scene information, but also tone and topics that correspond to the user's emotional state.

[0323] Step 6:

[0324] Data transmission

[0325] The server sends the generated auto-completion input data to each application via the application's API, where it is applied by the corresponding application.

[0326] Terminal handling

[0327] Step 1:

[0328] Receive notifications

[0329] The device monitors and receives auto-complete input data sent from the server in real time, and periodically checks for communication from the server to see if there is new data.

[0330] Step 2:

[0331] Data Application

[0332] The device applies the received data to the corresponding application, for example, updating the UI to display new topics or questions in a chat app. Based on the emotion data, the device presents content tailored to the user.

[0333] User Action

[0334] Step 1:

[0335] App use

[0336] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[0337] Step 2:

[0338] Complementary Experiences

[0339] Users experience auto-complete interactions, such as continuing a conversation with a friend based on a movie scene displayed in a chat app or topics and questions related to the user's emotions, providing a more natural and enriching experience.

[0340] Specific examples

[0341] Example 1: Linking a virtual movie theater app with a chat app

[0342] Step 1:

[0343] The server detects the use of the virtual cinema app and the chat app.

[0344] Step 2:

[0345] The server collects movie playback scene information, chat app comment history, and the user's voice tone.

[0346] Step 3:

[0347] The server uses a generative AI model to analyze movie scene information and speech history, and uses an emotion engine to evaluate the user's emotional state.

[0348] Step 4:

[0349] Based on the results of the emotion engine, the generative AI model adjusts chat topics to suit the user's emotional state.

[0350] Step 5:

[0351] The server generates input data adjusted based on the emotional state and sends it to the chat app.

[0352] Step 6:

[0353] The device receives the transmitted data, reflects it in the chat app, and presents new topics to the user.

[0354] Example 2: Integration of an educational app and a note-taking app

[0355] Step 1:

[0356] The server detects the use of educational apps and note-taking apps.

[0357] Step 2:

[0358] The server collects lesson content data, note content, and user text content.

[0359] Step 3:

[0360] The server uses a generative AI model to analyze the lesson content and an emotion engine to evaluate the user's emotional state.

[0361] Step 4:

[0362] Based on the results of the emotion engine, a generative AI model tailors supplemental information and hints that correspond to the user's emotional state.

[0363] Step 5:

[0364] The server generates complementary data adjusted based on the emotional state and sends it to the note-taking app.

[0365] Step 6:

[0366] The device receives the transmitted data, reflects it in the note application, and adds supplementary information to the user's notes.

[0367] Through these processes, the system of the present invention can realize information sharing and collaboration among multiple applications, and can also provide optimal interactions according to the user's emotional state.

[0368] Example 2

[0369] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0370] In conventional systems, when a user operates multiple programs simultaneously, there is a problem of a fragmented user experience due to a lack of information sharing or collaboration between the programs. Therefore, there is a need for a system that allows users to smoothly exchange information between different programs. There is also a need to provide a more natural and rich experience by realizing interactions that take the user's emotional state into consideration.

[0371] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0372] In this invention, the server includes means for detecting multiple programs run by a user, means for collecting user operation data and situation information from each detected program, means for running a generative AI model and an emotion analysis engine based on the collected data to analyze and predict interactions between programs, means for generating auto-complete input data for each program based on the analysis results, and means for transmitting the generated auto-complete input data to each program. This enables information sharing and collaboration between different programs, and can provide optimal interactions to the user according to their emotional state.

[0373] "Program" refers to multiple applications and software that can be operated by a user.

[0374] "Operation data" refers to information about a series of operations performed by a user within a program.

[0375] "Status Information" refers to information relating to the state of a program and its contents.

[0376] "Generative AI model" refers to an artificial intelligence model that analyzes and predicts interactions between programs based on collected interaction data and situational information.

[0377] "Emotion analysis engine" refers to an engine for analyzing a user's emotional state from the user's interaction data and context information.

[0378] "Auto-completion input data" refers to data generated to complement a user's operation.

[0379] "Information sharing" refers to the exchange and sharing of information between different programs.

[0380] "Integration" refers to multiple programs working together to provide a unified experience.

[0381] An "interaction" refers to a series of exchanges or operations between a user and a program.

[0382] "Analysis Results" refers to the analysis results of data obtained by the generative AI model and sentiment analysis engine.

[0383] "User experience" refers to the overall impression and satisfaction a user feels when operating each program.

[0384] This invention relates to a system that analyzes the interactions between multiple programs when a user is using them simultaneously, improving the user experience. This system collects user operation data and situational information, analyzes them using a generative AI model and an emotion analysis engine, and generates auto-complete input data for each program, realizing collaboration between programs.

[0385] Specifically, the server operates as follows.

[0386] The server first communicates with the user's device to detect all currently running programs. A specific example of this operation is to call the device's API (e.g., ActivityManager in ANDROID (registered trademark) or UIApplication in iOS) to obtain a list of active programs. Information about the programs detected in this way is then notified to the server.

[0387] Next, the server collects user operation data and context information from each detected program. This information is collected using the API of each program. For example, a virtual movie theater app might collect information about the scene currently playing in a movie, while a chat app might collect the user's comment history. This information is then provided to the generative AI model and sentiment analysis engine.

[0388] The server uses the collected data as input to run a generative AI model and an emotion analysis engine. The generative AI model, for example, analyzes movie scene information and chat app comment history to infer the interactions between them. Meanwhile, the emotion analysis engine infers the user's emotional state from the tone of their voice and the content of their chat messages.

[0389] Based on the analysis results of the generative AI model and the emotion analysis engine, the server generates auto-complete input data for each program. As a specific example, data is generated that includes not only chat topics related to movie scene information, but also appropriate tones and topics according to the user's emotional state. For example, appropriate questions and comments related to a moving movie scene are generated and sent to a chat app.

[0390] The auto-complete input data sent from the server is received by the device. The device periodically checks for communication from the server and obtains new data. This received data is applied to the corresponding program, for example, a new topic or question is displayed in a chat app. The device uses this data to update the program's UI and notify the user that new input has been received. Specifically, this is done by updating the RecyclerView on Android, or by using the refresh function of UITableView on iOS.

[0391] Users can enjoy a more natural and richer experience with this updated program. For example, when watching a movie in a virtual cinema app and chatting with friends in a chat app, they can enjoy auto-complete interactions. Topics and questions related to the movie scene and their own emotions are displayed, making the conversation with friends more natural and smooth.

[0392] Below are some examples of specific prompt sentences.

[0393] Example 1: Linking a virtual movie theater app with a chat app

[0394] When a user watches a movie in a virtual cinema app and chats with a friend in a chat app, the server collects information about the movie's scenes and the chat app's comments and emotional data. The generative AI model and emotion analysis engine use this data to generate questions and topics related to the movie's scenes and adapted to the user's emotional state. The generated data is sent to the chat app, which receives it and presents it to the user. This process allows the user to share topics based on the movie's scenes and their emotions with their friends.

[0395] Example 2: Integration of an educational app and a note-taking app

[0396] When a user takes a class using an educational app and records their learning using a note-taking app, the server collects the class content data, the note content, and emotional data. Based on this data, the generative AI model and emotional analysis engine generate supplementary information to add to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. This process allows users to easily record supplementary information related to the class and provides optimal learning based on their emotional state.

[0397] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0398] Step 1:

[0399] The server communicates with the user's device and detects all currently running programs. Specifically, the server calls the device's API (e.g., ActivityManager on Android or UIApplication on iOS) to obtain a list of active programs. The input data is information about the programs running on the device, and the output is a list of running programs.

[0400] Step 2:

[0401] The server collects user operation data and status information from each detected program. This is done through each program's API. For example, a virtual movie theater app collects information about the scene currently playing in a movie, and a chat app collects the user's comment history. The input data is the API call results for each program, and the output is the operation data and status information collected from each program.

[0402] Step 3:

[0403] The server provides the collected operation data and situation information as input to the generative AI model and the emotion analysis engine. In this step, the generative AI model analyzes the movie scene information and the chat app's comment history to infer the interactions between them. Meanwhile, the emotion analysis engine analyzes the user's emotional state from the user's tone of voice and the content of the chat messages. The input is the collected data, and the output is the analysis results and the emotional state.

[0404] Step 4:

[0405] The server generates auto-completion input data for each program based on the analysis results of the generative AI model and the emotion analysis engine. This includes, for example, automatically generating topics and questions related to moving movie scenes for a chat app. The input is the analysis results and emotional state, and the output is auto-completion input data.

[0406] Step 5:

[0407] The server sends the generated auto-completion input data to each program. Specifically, it sends the data using HTTP requests, WebSockets, etc. The input is the auto-completion input data, and the output is the data received by the corresponding program.

[0408] Step 6:

[0409] The terminal receives the auto-completion input data sent from the server. It monitors the communication from the server in real time and acquires new data. The input is the communication data from the server, and the output is the received auto-completion input data.

[0410] Step 7:

[0411] The device applies the received auto-completion input data to the corresponding program, for example, displaying a new topic or question in a chat app and updating the application UI to notify the user of the new input. The input is the received auto-completion input data, and the output is the updated UI.

[0412] Step 8:

[0413] Users use multiple programs simultaneously and experience auto-completed interactions. For example, they can watch a movie in a virtual cinema app while chatting with friends in a chat app, continuing the conversation based on newly generated topics and questions. The input is an updated UI, and the output is the user's new interaction experience.

[0414] (Application example 2)

[0415] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0416] Conventional systems provided simple navigation and entertainment information without considering the user's emotions or surrounding environmental information while driving, which did not sufficiently improve the user experience.In addition, there were few ways to accurately analyze and coordinate the interactions between multiple applications used simultaneously by the user, making it difficult to reduce stress and inconvenience while driving.

[0417] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0418] In this invention, the server includes means for detecting multiple applications run by a user, means for collecting user interaction data and situation information from each detected application, means for running a generative AI model based on the collected data to analyze and predict interactions between applications, means for generating auto-complete input data for each application based on the analysis results, means for sending the generated auto-complete input data to each application, and means for generating navigation information and entertainment information that takes into account the emotional state and surrounding environmental information of the means of transportation driven by the user based on the analyzed interaction data and situation information. This makes it possible to provide optimal navigation information and entertainment information in real time according to the user's emotions and surrounding environment, reducing stress and inconvenience while driving and improving the user experience.

[0419] The "means for detecting multiple applications run by a user" refers to a method and apparatus for identifying and listing all software and programs currently running on a user's terminal.

[0420] "Means for collecting user interaction data and situation information" refers to methods and devices for acquiring and accumulating data about the operations performed by users in each application and data about the surrounding environment.

[0421] "Means for running a generative AI model based on collected data to analyze and infer interactions between applications" refers to a method and device that uses acquired user operation history and environmental data as input, analyzes the relationships and interactions between applications using AI technology, and infers the results.

[0422] "Means for generating auto-complete input data for each application based on the analysis results" refers to a method and device for automatically creating appropriate data and information for each application based on the analysis results of the AI ​​model.

[0423] The "means for sending the generated auto-completion input data to each application" refers to a method and apparatus for transferring the generated data to the corresponding application.

[0424] "Means for generating navigation information and entertainment information that takes into account the emotional state and surrounding environmental information of the means of transportation being driven by the user based on analyzed interaction data and situational information" refers to a method and device for analyzing emotions and surrounding conditions from the user's operation history and environmental data, and providing useful route guidance and entertainment content while driving based on that.

[0425] As an embodiment of the present invention, a user assistance system for an autonomous driving vehicle will be taken as an example.

[0426] In this system, the server connects to the user's smartphone to detect multiple applications and collects interaction data and context information from each application. Based on the collected data, it runs a generative AI model to analyze the interactions between applications and generate the necessary auto-complete input data. This auto-complete input data is then sent to the corresponding application. Specifically, the system includes the following steps:

[0427] The server uses the device's API to discover all applications the user is running, which results in a list of currently active applications.

[0428] The server then collects data from each detected application about the user's operation history and surrounding environment, including GPS data while driving, the vehicle's current location, speed, surrounding traffic and weather information, and voice inputs. This data is then fed into a generative AI model to analyze the user's current situation and emotional state.

[0429] The generative AI model uses this data to analyze interactions between applications and generate optimal navigation and entertainment information to improve the user's driving experience. For example, if the user is feeling stressed, it can provide relaxing music or smooth route guidance based on that emotion.

[0430] The server then sends the generated navigation and entertainment information to each application. The device applies the received data to the corresponding application and notifies the user in real time. For example, a prompt such as "Please provide optimal driving advice and entertainment based on the current traffic situation and emotional state" could be used.

[0431] This system uses EmotionEngine to analyze the user's emotions and NavigationSystem (e.g., Google® Maps API) to provide the optimal navigation route.It also improves the user's driving experience by generating entertainment information based on the collected data according to the user's driving situation and emotional state.

[0432] For example, if a user is stuck in traffic while driving an autonomous vehicle, the system uses the Emotion Engine to detect that the user is frustrated. The navigation system takes into account the current traffic situation and suggests an alternative route. The system also notifies the user, saying, "You are feeling stressed by the current traffic jam, so we will play relaxing music and suggest the optimal route to avoid it." In this way, the system can reduce stress and inconvenience while the user is driving, providing a more comfortable driving experience.

[0433] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0434] Step 1:

[0435] The server communicates with the user's device and uses an API to discover all currently running applications. Specifically, it calls the device's API to retrieve the list of active applications. The device's status information is given as input, and the list of active applications is obtained as output.

[0436] Step 2:

[0437] The server collects user interaction data and context information from each detected application, including GPS data, voice input, the vehicle's current location, speed, and environmental data such as traffic and weather. The input is the list of active applications, and the output is the collected interaction data and context information.

[0438] Step 3:

[0439] The server inputs the collected data into a generative AI model to analyze and predict interactions between applications. The generative AI model analyzes the user's operation history and environmental data to analyze the relationships and interactions between applications. The collected data is given as input, and the analysis and prediction of interactions between applications is obtained as output.

[0440] Step 4:

[0441] The server generates auto-completion input data for each application based on the analysis results of the generative AI model. For example, it creates navigation information or entertainment information taking into account the user's emotional state and surrounding environment. The analysis results are given as input, and the auto-completion input data is generated as output.

[0442] Step 5:

[0443] The server sends the generated auto-completion input data to each application. Each application receives and applies this data. The auto-completion input data is given as input, and the corresponding application is updated as output.

[0444] Step 6:

[0445] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, the device periodically checks for communication from the server and acquires new data. Data from the server is given as input, and the received data is saved on the device as output.

[0446] Step 7:

[0447] The device applies the received data to the corresponding application and notifies the user, for example, displaying new route directions in a navigation application or instructing a music application to play relaxing music. The received data is given as input, and the application's UI is updated as output.

[0448] Step 8:

[0449] The user experiences complementary interactions, for example, receiving new route directions from a navigation app while driving and playing relaxing music from a music app, with application notifications as input and an improved driving experience as output.

[0450] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0451] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0452] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0453] [Second embodiment]

[0454] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0455] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0456] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0457] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0458] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0459] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0460] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0461] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0462] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0463] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0464] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0465] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0466] The present invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model, and generates auto-complete input data for each application, realizing collaboration between applications.

[0467] Server Processing

[0468] 1. Application Discovery

[0469] The server communicates with the user's device to detect each application the user is using. This allows the server to obtain information about all applications running on the device. For example, if a user is using a virtual movie theater app and a chat app at the same time, that information is notified to the server.

[0470] 2. Data Collection

[0471] The server then collects user interaction data and context information from each detected application, including movie scene information from a virtual cinema app and user comment history from a chat app.

[0472] 3. Generative AI model in action

[0473] A generative AI model is run based on the collected data. For example, movie scene information and utterances from a chat app are used as input. The model analyzes and infers the data to generate topics and questions related to movie scenes.

[0474] 4. Input completion data generation

[0475] The server generates auto-complete input data for each application based on the analysis results of the generative AI model. For example, questions and topics related to movie scenes are generated for a chat app.

[0476] 5. Data Transmission

[0477] Finally, the server sends the generated auto-complete input data to each application, allowing the chat app to suggest new topics and questions related to movie scenes to the user.

[0478] Terminal handling

[0479] 1. Receiving notifications

[0480] The user's device receives the auto-complete input data sent from the server, which allows the new data to be reflected in real time.

[0481] 2. Data Application

[0482] The device applies the received data to each application. For example, it might display topics related to movie scenes in a chat app to encourage user interaction. It might also update the application's UI and notify the user of new input.

[0483] User Action

[0484] 1. App Usage

[0485] Users can use multiple xR applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[0486] 2. Complementary Experiences

[0487] Users experience auto-complete interactions with each application, for example, sharing topics with friends based on a question related to a movie scene displayed in a chat app, resulting in a more natural and rich user experience.

[0488] Specific examples

[0489] Example 1: Linking a virtual movie theater app with a chat app

[0490] Consider a case where a user is watching a movie on a virtual cinema app while chatting with a friend on a chat app. The server collects information about movie scenes and the content of comments made in the chat app. The generative AI model uses this data to generate questions and topics related to the movie scenes. The generated data is sent to the chat app, which receives it and presents it to the user. This allows the user to share topics based on movie scenes with their friends.

[0491] Example 2: Integration of an educational app and a note-taking app

[0492] Consider a case where a user is taking a class using an educational app while recording what they learned using a note-taking app. The server collects the class content data and the contents of the notes. The generative AI model uses this data to generate supplementary information to be added to the notes and hints for related problems. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, the user can easily record supplementary information related to the class.

[0493] Through these processes, the system of the present invention realizes information sharing and collaboration between multiple applications, improving the user experience.

[0494] The processing flow will be explained below.

[0495] Server Processing

[0496] Step 1:

[0497] Application Discovery

[0498] The server communicates with the user's device to discover all currently running applications, specifically by calling the device's API to retrieve the list of active applications.

[0499] Step 2:

[0500] Data collection

[0501] The server collects user interaction data and situation information from each detected application. Specifically, it obtains movie playback time and scene information from the movie theater app, and speech history from the chat app. This is also done through each application's API.

[0502] Step 3:

[0503] Generative AI model in action

[0504] The server inputs the collected data and runs a generative AI model. Specifically, it inputs movie scene information and speech history into the model, analyzes them, and predicts interactions.

[0505] Step 4:

[0506] Input completion data generation

[0507] Based on the analysis results of the generative AI model, auto-complete input data is generated for each application. For example, it generates chat topics related to movie scene information.

[0508] Step 5:

[0509] Data transmission

[0510] The server sends the generated auto-complete input data to each application, again via the API of each application.

[0511] Terminal handling

[0512] Step 1:

[0513] Receive notifications

[0514] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, it periodically checks communications from the server and acquires new data.

[0515] Step 2:

[0516] Data Application

[0517] The device applies the received data to the corresponding application, for example updating the UI to display new topics or questions in a chat app.

[0518] User Action

[0519] Step 1:

[0520] App use

[0521] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[0522] Step 2:

[0523] Complementary Experiences

[0524] Users experience auto-completed interactions, specifically continuing conversations with friends based on topics and questions related to movie scenes displayed in the chat app.

[0525] In this way, through the specific operations at each step, cooperation between the server, terminal, and user is realized, providing a richer user experience.

[0526] Example 1

[0527] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0528] Modern users often use multiple applications simultaneously and require enhanced user experiences through information sharing and collaboration between these applications. However, data sharing and auto-completion input between different applications is often performed manually, making it difficult to provide an efficient and natural user experience. For this reason, there is a demand for a system that automatically shares information and collaborates between multiple applications to improve user interaction.

[0529] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0530] In this invention, the server includes means for detecting multiple applications run by a user, means for collecting user interaction data and situation information from each detected application, means for running a generative AI model based on the collected data to analyze and predict interactions between applications, means for generating auto-complete input data for each application based on the analysis results, and means for transmitting the generated auto-complete input data to each application, thereby enabling automatic information sharing and collaboration between different applications to provide an efficient and natural user experience.

[0531] "User" refers to a person or entity that uses a system or application.

[0532] "Application" is a general term for a software program that a user runs on a terminal.

[0533] "Interaction Data" refers to data regarding inputs and interactions when a user operates an application.

[0534] "Status information" refers to information related to the user's usage environment and status, including, for example, the application being used and the status of its content.

[0535] A "generative AI model" refers to an artificial intelligence model that analyzes collected data and automatically generates input data.

[0536] "Auto-complete input data" refers to the input data generated by the generative AI model to be applied to each application.

[0537] "Server" refers to a computer system that communicates with a user's terminal, collects data, analyzes the data, and transmits the generated data.

[0538] "Terminal" refers to a device that a user uses to run an application, and examples include a computer and a smartphone.

[0539] "API" stands for Application Programming Interface and refers to a means by which different pieces of software communicate with each other.

[0540] An "HTTP POST request" is a part of the HTTP protocol used to send data to a server.

[0541] This invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model, and generates auto-complete input data for each application, realizing collaboration between applications.

[0542] This system mainly consists of three main components: a server, a terminal, and a user.

[0543] Server Roles

[0544] The server first communicates with the user's terminal and detects each application the user is running. For example, the netstat or ps command is used to obtain information about applications running on the terminal. This information is sent to the server using the UDP or TCP protocol. For example, the netstat -an | grep 'ESTABLISHED' command can be used to identify connected applications.

[0545] Next, the server collects user interaction data and context information from each detected application. The collected data is obtained through API calls and log data analysis. For example, movie scene information can be collected from a virtual movie theater app, and user comment history can be collected from a chat app.

[0546] Based on the collected data, the server runs a generative AI model. For example, GPT-4 can be used as the generative AI model. By preprocessing the data using Python and inputting it into the generative AI model, topics and questions related to the movie scenes can be generated using movie scene information and the content of comments made in the chat app as input.

[0547] Next, the server generates auto-complete input data based on the analysis results of the generative AI model. For example, if the generative AI model generates a "question about a movie scene," it converts it into JSON format so that the chat app can understand it. This completion data is sent to each application using an HTTP POST request. For example, you can POST the data to http: / / chatapp.example.com / api / v1 / messages to reflect the new question in the chat app.

[0548] Device Role

[0549] The user's device receives the auto-complete input data sent from the server. You can receive the data in real time using a listener in the device (for example, a WebSocket server). Create an onMessage event handler to process the data as soon as it arrives.

[0550] The device then applies the received data to each application, for example updating the UI of a chat app to show the new question, or if you're using React, updating the state and re-rendering the component to show the new question.

[0551] User Roles

[0552] Users use multiple applications simultaneously. For example, they may watch a movie on a virtual cinema app while chatting with friends on a chat app. As users interact with the applications, their activities are sent to a server and data is collected.

[0553] The collected data is then automatically filled in and applied to the device, allowing users to enjoy new experiences. For example, a user can share a topic with a friend based on a question related to a movie scene displayed in a chat app. This allows the user's conversation to progress more naturally, resulting in richer communication.

[0554] Specific examples

[0555] Example 1: Linking a virtual movie theater app with a chat app

[0556] When a user watches a movie in a virtual cinema app and chats with a friend in a chat app, the server collects information about the movie's scenes and the content of the chat app's comments. The generative AI model uses this data to generate questions and topics related to the movie's scenes. The generated data is sent to the chat app, which receives it and presents it to the user. In this way, the user can share topics based on the movie's scenes with their friends.

[0557] Examples of specific prompts include:

[0558] "What do you think about the characters in the scene you're watching?"

[0559] "Let's talk about the events behind this scene."

[0560] Example 2: Integration of an educational app and a note-taking app

[0561] When a user takes a class using an educational app and records their learning using a note-taking app, the server collects the class content data and the contents of the notes. The generative AI model uses this data to generate supplementary information to be added to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, users can easily record supplementary information related to the class.

[0562] Examples of specific prompts include:

[0563] "Please note down any important points related to this topic."

[0564] "Let's dig deeper into the ideas explained in class."

[0565] Through these processes, the system of the present invention can realize information sharing and collaboration between multiple applications, thereby improving the user experience.

[0566] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0567] Step 1:

[0568] The server detects multiple applications that a user runs.

[0569] Specifically, the server communicates with the user's terminal via TCP or UDP protocol and uses the netstat or ps command to collect information about running applications. The information obtained includes the application name, version, and status information indicating that it is running. Based on this information, it identifies which applications are currently running.

[0570] Input: Application execution status on user device

[0571] Output: Information about the running application (name, version, status, etc.)

[0572] Step 2:

[0573] The server collects user interaction data and context information from each detected application.

[0574] Specifically, the server uses the API of each application to obtain information such as what operations the user performed and what content is being played. For example, it collects information about the currently playing scene from a virtual movie theater app, and the user's comment history from a chat app. This information is collected by calling APIs and analyzing log files.

[0575] Input: Running application information

[0576] Output: Interaction data and context information for each application

[0577] Step 3:

[0578] The server runs a generative AI model based on the collected data.

[0579] Specifically, the server preprocesses the data using Python and inputs it into a generative AI model (e.g., GPT-4). The input data includes movie scene information and comments from a chat app. The generative AI model analyzes this data and generates appropriate complementary information (e.g., new questions or topics).

[0580] Input: Interaction data and context information

[0581] Output: Complementary information (e.g., questions and topics) generated by the generative AI model

[0582] Step 4:

[0583] The server generates input completion data based on the analysis results of the generative AI model.

[0584] Specifically, the server converts the data obtained from the generative AI model into an appropriate format, such as JSON, and formats it in a way that each application can understand. For example, it converts the generated question into a format like {"type": "question", "content": "Who are the characters in this scene?"}.

[0585] Input: Analysis results of the generative AI model

[0586] Output: Input completion data applicable to each application

[0587] Step 5:

[0588] The server transmits the generated auto-completion input data to each application.

[0589] Specifically, the server sends data using an HTTP POST request, for example to http: / / chatapp.example.com / api / v1 / messages, to reflect a new question in the chat app.

[0590] Input: Input completion data

[0591] Output: Data sent to each application

[0592] Step 6:

[0593] The user's terminal receives the auto-complete input data sent from the server.

[0594] Specifically, the device receives data in real time using a locally running listener (e.g., a WebSocket server). When the onMessage event handler receives the data, it executes the appropriate process.

[0595] Input: Data sent to each application

[0596] Output: Type-ahead data received on the device

[0597] Step 7:

[0598] The terminal applies the received data to each application and notifies the user.

[0599] Specifically, the device reflects the received data in the UI of the corresponding application and notifies the user of new input. For example, if you are using React, it will update the state and re-render the component displaying the new question.

[0600] Input: Received type-ahead data

[0601] Output: Updated application UI, notification to the user

[0602] Step 8:

[0603] The user uses the application and experiences the autocompleted interaction.

[0604] Specifically, users can engage in conversations with friends based on questions related to movie scenes presented in a chat app, resulting in a more natural and enriching user experience.

[0605] Input: Updated application UI

[0606] Output: Improved user interaction and experience

[0607] (Application example 1)

[0608] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0609] When users use multiple content distribution services on the Internet, each service operates independently, making it difficult to provide a unified information experience. It has also been pointed out that there is a lack of means to provide real-time updates or additional content related to the media content being viewed or listened to. This results in a fragmented experience, as users are unable to link information between content or receive seamless information.

[0610] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0611] In this invention, the server includes means for detecting multiple programs run by a user, means for collecting user interaction data and situation information from each detected program, means for running a generative AI model based on the collected data to analyze and predict interactions between programs, means for generating auto-complete input data for each program based on the analysis results, means for sending the generated auto-complete input data to each program, and means for automatically generating news and additional information related to the media content being viewed or listened to in cooperation with the content distribution service to which each program belongs. This enables information related to the media content being viewed or listened to to be provided in real time when a user is using multiple content distribution services, thereby realizing a seamless information experience.

[0612] A "program" is a set of instructions or code that runs on a computer to accomplish a particular task or function.

[0613] "Interaction data" is data generated when a user operates a program, and includes operation logs such as clicks, taps, inputs, and viewing time.

[0614] "Contextual information" is data that describes the context or conditions identified during program execution, including the user's geographic location, device state, current activity, etc.

[0615] A "generative AI model" is an artificial intelligence algorithm that analyzes patterns and relationships based on collected data and makes predictions and inferences. In this case, it is used specifically for natural language generation and data completion.

[0616] "Auto-complete input data" refers to suggested data or additional information generated by a generative AI model and used as input data for a specific program.

[0617] A "content distribution service" is a service that provides users with multimedia content such as videos, music, and news via the Internet.

[0618] "Related news and additional information" refers to information or content that is related to the media content that the user is watching or listening to and is provided in a manner that complements it.

[0619] MODE FOR CARRYING OUT THE INVENTION

[0620] This invention relates to a system that analyzes the interactions between multiple programs when a user is using them simultaneously, improving the user experience. The system uses a generative AI model to automatically generate related information based on interaction data and context information, and establishes collaboration between the programs.

[0621] Server Processing

[0622] The server has the following means:

[0623] 1. Program detection methods

[0624] It communicates with the user's device and detects the multiple programs the user is running. This method identifies all programs running on the device. For example, if a user is using a video streaming service and a news app at the same time, that information is notified to the server.

[0625] 2. Data Collection Methods

[0626] For each program detected, it collects user interaction data and contextual information, such as information about the video being watched, browsing history in news apps, etc.

[0627] 3. Means of running the generative AI model

[0628] Based on the collected data, a generative AI model is run to analyze the interactions between programs. The generative AI model analyzes the user's interaction data and generates related information based on it. For example, it generates additional information or topics related to the movie or news being watched.

[0629] 4. Auto-complete input data generation method

[0630] Based on the analysis, it generates auto-complete entries for each program, including news and additional information related to the content being viewed.

[0631] 5. Data Transmission Method

[0632] The generated auto-complete input data is sent to each program, which allows the news app to present new related information and news articles to the user.

[0633] Terminal handling

[0634] The terminal has the following means:

[0635] 1. Notification Reception Method

[0636] Receives auto-complete input data sent from the server, allowing the device to reflect new data in real time.

[0637] 2. Data application method

[0638] The received data is applied to each program and notified to the user, allowing the news app to display relevant news and additional information to the user in real time.

[0639] User Action

[0640] The user has the following means:

[0641] 1. How to use the program

[0642] Use multiple programs at the same time. For example, watch a movie on a streaming service while checking the latest news on a news app.

[0643] 2. Complementary Experiences

[0644] Experience auto-complete information in each program. For example, you can browse more interesting topics based on movie-related news displayed in a news app.

[0645] Specific examples

[0646] As a specific example, consider a case where while a user is watching a "documentary about climate change" on a video streaming service, the system of this invention automatically generates and displays news articles about "the latest research results on climate change" and "the latest trends in the environmental protection movement" in a news app. In this way, additional information that is closely related to the content the user is watching can be provided in real time.

[0647] Example prompt sentence:

[0648] Input: Viewing data for the video "Documentary on Climate Change"

[0649] Output: Related news articles (title, content)

[0650] This allows users to seamlessly experience related information and news while using content distribution services.

[0651] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0652] Step 1:

[0653] Program Detection

[0654] The server communicates with the user's terminal and detects all programs the user is running.

[0655] Input: A list of running programs from the user's terminal

[0656] Output: Detected program information

[0657] Specific operation: A monitoring agent installed on the user's terminal collects information about running programs and sends it to the server.

[0658] Step 2:

[0659] Data collection

[0660] The server collects user interaction data and context information from each detected program.

[0661] Input: Detected program information

[0662] Output: Interaction data and context information from each program

[0663] Specific operations: Collects data such as video playback time, content being viewed, and user operation logs.

[0664] Step 3:

[0665] Generative AI model in operation

[0666] The server runs a generative AI model based on the collected data and analyzes interactions between programs.

[0667] Input: Interaction data and context information

[0668] Output: Analysis results (related news and additional information)

[0669] Specific operation: The collected data is input into a generative AI model for analysis. The generative AI model used here is GPT-3.

[0670] Step 4:

[0671] Auto-complete input data generation

[0672] The server generates auto-complete input data for each program based on the analysis results of the generative AI model.

[0673] Input: Analysis results

[0674] Output: Auto-complete input data

[0675] Specific operation: Based on the generated data, templates for news articles and additional information are created and their content is automatically generated.

[0676] Step 5:

[0677] Data transmission

[0678] The server transmits the generated auto-completion input data to each program.

[0679] Input:Auto-complete input data

[0680] Output: Data to send to each program

[0681] Specific operation: Encode and send the generated data in an appropriate format to the corresponding program in real time.

[0682] Step 6:

[0683] Receive notifications

[0684] The terminal receives the auto-completion input data sent from the server.

[0685] Input: Data to be sent from the server

[0686] Output: Received complementary data

[0687] Specific operation: The program on the terminal receives data from the server and stores the contents in temporary storage.

[0688] Step 7:

[0689] Data Application

[0690] The terminal applies the received data to each program and notifies the user.

[0691] Input: Received complementary data

[0692] Output: Update information for each program

[0693] Specific behavior: Update the UI of each program to apply the received data and notify the user that new information is available.

[0694] Step 8:

[0695] Program Use

[0696] A user may use multiple programs simultaneously.

[0697] Input: Related news and additional information

[0698] Output: User Interaction

[0699] Specific operations: A user watches a movie on a video streaming service while checking related information on a news app.

[0700] Step 9:

[0701] Complementary Experiences

[0702] Users experience auto-completed information in each program.

[0703] Input: User interaction

[0704] Output: Improving the user experience

[0705] Specific behavior: By reading news articles related to a movie displayed in a news app, you can deepen your understanding of the movie's content.

[0706] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0707] This invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model and an emotion engine, and generates auto-complete input data for each application, realizing collaboration between applications.

[0708] Server Processing

[0709] 1. Application Discovery

[0710] The server communicates with the user's device to detect all currently running applications. Specifically, it calls the device's API to obtain a list of active applications. For example, if a user is using a virtual movie theater app and a chat app at the same time, that information is notified to the server.

[0711] 2. Data Collection

[0712] The server then collects user interaction data and context information from each detected application, such as movie scene information from a virtual cinema app or user comment history from a chat app, and provides the collected data to a generative AI model and emotion engine.

[0713] 3. Generative AI model and emotion engine in action

[0714] The server uses the collected data as input to run a generative AI model and an emotion engine. The generative AI model analyzes movie scene information and chat app comment history to infer interactions. Meanwhile, the emotion engine analyzes the user's emotions from the user's interaction data and situational information. For example, the emotion engine infers the user's emotional state from the user's tone of voice and text content.

[0715] 4. Input completion data generation

[0716] Based on the analysis results of the generative AI model and emotion engine, auto-complete input data for each application is generated. For example, data containing appropriate tone and topics according to the user's emotional state is generated, in addition to casual conversation topics related to movie scene information.

[0717] 5. Data Transmission

[0718] Finally, the server sends the generated auto-complete input data to each application, allowing the chat app to present new topics and questions to the user that are adapted to the movie scenes and the user's emotions.

[0719] Terminal handling

[0720] 1. Receiving notifications

[0721] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, it periodically checks communications from the server and acquires new data.

[0722] 2. Data Application

[0723] The device applies the received data to the corresponding application, for example, displaying a new topic or question in a chat app, or updating the application's UI to notify the user of the new input.

[0724] User Action

[0725] 1. App Usage

[0726] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[0727] 2. Complementary Experiences

[0728] Users experience auto-complete interactions, such as continuing a conversation with a friend based on a movie scene displayed in a chat app or topics and questions related to the user's emotions, resulting in a more natural and rich user experience.

[0729] Specific examples

[0730] Example 1: Linking a virtual movie theater app with a chat app

[0731] Consider a case where a user is watching a movie on a virtual cinema app while chatting with a friend on a chat app. The server collects information about movie scenes, comments made on the chat app, and the user's emotional data analyzed by the emotion engine. Based on this data, the generative AI model and emotion engine generate questions and topics related to the movie scenes and adapted to the user's emotional state. The generated data is sent to the chat app, which receives it on the device and presents it to the user. In this way, the user can share topics based on the movie scenes and their own emotions with their friends.

[0732] Example 2: Integration of an educational app and a note-taking app

[0733] Consider a case where a user takes a class using an educational app while recording their learning using a note-taking app. The server collects the class content data, the note content, and the user's emotional data analyzed by the emotion engine. Based on this data, the generative AI model and emotion engine generate supplementary information to add to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, users can not only easily record supplementary information related to the class, but also receive optimal learning experiences tailored to their emotional state.

[0734] Through these processes, the system of the present invention enables information sharing and collaboration between multiple applications, not only improving the user experience but also providing optimal interactions according to the user's emotional state.

[0735] The processing flow will be explained below.

[0736] Server Processing

[0737] Step 1:

[0738] Application Discovery

[0739] The server communicates with the user's device to discover all currently running applications, specifically by calling the device's API to retrieve the list of active applications.

[0740] Step 2:

[0741] Data collection

[0742] The server collects user interaction data and context information from each detected application. Specifically, it obtains movie playback time and scene information from the virtual cinema app, and speech history from the chat app. It also collects the user's voice tone and text content as input data for the emotion engine.

[0743] Step 3:

[0744] Generative AI model and emotion engine in action

[0745] The server uses the collected data as input to run a generative AI model and an emotion engine. The generative AI model analyzes movie scene information and chat app comment history to predict interactions. The emotion engine analyzes the user's voice tone and text content to evaluate the user's emotional state.

[0746] Step 4:

[0747] Applying Emotion Data

[0748] Based on the analysis results of the emotion engine, the generative AI model adjusts the auto-complete input data for each application. For example, if the user is excited, it generates input data that is appropriate for that excited state.

[0749] Step 5:

[0750] Input completion data generation

[0751] Based on the analysis results of the generative AI model and emotion engine, auto-complete input data for each application is generated. For example, data is generated that includes not only casual conversation topics related to movie scene information, but also tone and topics that correspond to the user's emotional state.

[0752] Step 6:

[0753] Data transmission

[0754] The server sends the generated auto-completion input data to each application via the application's API, where it is applied by the corresponding application.

[0755] Terminal handling

[0756] Step 1:

[0757] Receive notifications

[0758] The device monitors and receives auto-complete input data sent from the server in real time, and periodically checks for communication from the server to see if there is new data.

[0759] Step 2:

[0760] Data Application

[0761] The device applies the received data to the corresponding application, for example, updating the UI to display new topics or questions in a chat app. Based on the emotion data, the device presents content tailored to the user.

[0762] User Action

[0763] Step 1:

[0764] App use

[0765] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[0766] Step 2:

[0767] Complementary Experiences

[0768] Users experience auto-complete interactions, such as continuing a conversation with a friend based on a movie scene displayed in a chat app or topics and questions related to the user's emotions, providing a more natural and enriching experience.

[0769] Specific examples

[0770] Example 1: Linking a virtual movie theater app with a chat app

[0771] Step 1:

[0772] The server detects the use of the virtual cinema app and the chat app.

[0773] Step 2:

[0774] The server collects movie playback scene information, chat app comment history, and the user's voice tone.

[0775] Step 3:

[0776] The server uses a generative AI model to analyze movie scene information and speech history, and uses an emotion engine to evaluate the user's emotional state.

[0777] Step 4:

[0778] Based on the results of the emotion engine, the generative AI model adjusts chat topics to suit the user's emotional state.

[0779] Step 5:

[0780] The server generates input data adjusted based on the emotional state and sends it to the chat app.

[0781] Step 6:

[0782] The device receives the transmitted data, reflects it in the chat app, and presents new topics to the user.

[0783] Example 2: Integration of an educational app and a note-taking app

[0784] Step 1:

[0785] The server detects the use of educational apps and note-taking apps.

[0786] Step 2:

[0787] The server collects lesson content data, note content, and user text content.

[0788] Step 3:

[0789] The server uses a generative AI model to analyze the lesson content and an emotion engine to evaluate the user's emotional state.

[0790] Step 4:

[0791] Based on the results of the emotion engine, a generative AI model tailors supplemental information and hints that correspond to the user's emotional state.

[0792] Step 5:

[0793] The server generates complementary data adjusted based on the emotional state and sends it to the note-taking app.

[0794] Step 6:

[0795] The device receives the transmitted data, reflects it in the note application, and adds supplementary information to the user's notes.

[0796] Through these processes, the system of the present invention can realize information sharing and collaboration among multiple applications, and can also provide optimal interactions according to the user's emotional state.

[0797] Example 2

[0798] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0799] In conventional systems, when a user operates multiple programs simultaneously, there is a problem of a fragmented user experience due to a lack of information sharing or collaboration between the programs. Therefore, there is a need for a system that allows users to smoothly exchange information between different programs. There is also a need to provide a more natural and rich experience by realizing interactions that take the user's emotional state into consideration.

[0800] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0801] In this invention, the server includes means for detecting multiple programs run by a user, means for collecting user operation data and situation information from each detected program, means for running a generative AI model and an emotion analysis engine based on the collected data to analyze and predict interactions between programs, means for generating auto-complete input data for each program based on the analysis results, and means for transmitting the generated auto-complete input data to each program. This enables information sharing and collaboration between different programs, and can provide optimal interactions to the user according to their emotional state.

[0802] "Program" refers to multiple applications and software that can be operated by a user.

[0803] "Operation data" refers to information about a series of operations performed by a user within a program.

[0804] "Status Information" refers to information relating to the state of a program and its contents.

[0805] "Generative AI model" refers to an artificial intelligence model that analyzes and predicts interactions between programs based on collected interaction data and situational information.

[0806] "Emotion analysis engine" refers to an engine for analyzing a user's emotional state from the user's interaction data and context information.

[0807] "Auto-completion input data" refers to data generated to complement a user's operation.

[0808] "Information sharing" refers to the exchange and sharing of information between different programs.

[0809] "Integration" refers to multiple programs working together to provide a unified experience.

[0810] An "interaction" refers to a series of exchanges or operations between a user and a program.

[0811] "Analysis Results" refers to the analysis results of data obtained by the generative AI model and sentiment analysis engine.

[0812] "User experience" refers to the overall impression and satisfaction a user feels when operating each program.

[0813] This invention relates to a system that analyzes the interactions between multiple programs when a user is using them simultaneously, improving the user experience. This system collects user operation data and situational information, analyzes them using a generative AI model and an emotion analysis engine, and generates auto-complete input data for each program, realizing collaboration between programs.

[0814] Specifically, the server operates as follows.

[0815] The server first communicates with the user's device to detect all programs currently running. A specific example of this operation is to call the device's API (e.g., ActivityManager in Android or UIApplication in iOS) to obtain a list of active programs. The information about the programs detected in this way is then notified to the server.

[0816] Next, the server collects user operation data and context information from each detected program. This information is collected using the API of each program. For example, a virtual movie theater app might collect information about the scene currently playing in a movie, while a chat app might collect the user's comment history. This information is then provided to the generative AI model and sentiment analysis engine.

[0817] The server uses the collected data as input to run a generative AI model and an emotion analysis engine. The generative AI model, for example, analyzes movie scene information and chat app comment history to infer the interactions between them. Meanwhile, the emotion analysis engine infers the user's emotional state from the tone of their voice and the content of their chat messages.

[0818] Based on the analysis results of the generative AI model and the emotion analysis engine, the server generates auto-complete input data for each program. As a specific example, data is generated that includes not only chat topics related to movie scene information, but also appropriate tones and topics according to the user's emotional state. For example, appropriate questions and comments related to a moving movie scene are generated and sent to a chat app.

[0819] The auto-complete input data sent from the server is received by the device. The device periodically checks for communication from the server and obtains new data. This received data is applied to the corresponding program, for example, a new topic or question is displayed in a chat app. The device uses this data to update the program's UI and notify the user that new input has been received. Specifically, this is done by updating the RecyclerView on Android, or by using the refresh function of UITableView on iOS.

[0820] Users can enjoy a more natural and richer experience with this updated program. For example, when watching a movie in a virtual cinema app and chatting with friends in a chat app, they can enjoy auto-complete interactions. Topics and questions related to the movie scene and their own emotions are displayed, making the conversation with friends more natural and smooth.

[0821] Below are some examples of specific prompt sentences.

[0822] Example 1: Linking a virtual movie theater app with a chat app

[0823] When a user watches a movie in a virtual cinema app and chats with a friend in a chat app, the server collects information about the movie's scenes and the chat app's comments and emotional data. The generative AI model and emotion analysis engine use this data to generate questions and topics related to the movie's scenes and adapted to the user's emotional state. The generated data is sent to the chat app, which receives it and presents it to the user. This process allows the user to share topics based on the movie's scenes and their emotions with their friends.

[0824] Example 2: Integration of an educational app and a note-taking app

[0825] When a user takes a class using an educational app and records their learning using a note-taking app, the server collects the class content data, the note content, and emotional data. Based on this data, the generative AI model and emotional analysis engine generate supplementary information to add to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. This process allows users to easily record supplementary information related to the class and provides optimal learning based on their emotional state.

[0826] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0827] Step 1:

[0828] The server communicates with the user's device and detects all currently running programs. Specifically, the server calls the device's API (e.g., ActivityManager on Android or UIApplication on iOS) to obtain a list of active programs. The input data is information about the programs running on the device, and the output is a list of running programs.

[0829] Step 2:

[0830] The server collects user operation data and status information from each detected program. This is done through each program's API. For example, a virtual movie theater app collects information about the scene currently playing in a movie, and a chat app collects the user's comment history. The input data is the API call results for each program, and the output is the operation data and status information collected from each program.

[0831] Step 3:

[0832] The server provides the collected operation data and situation information as input to the generative AI model and the emotion analysis engine. In this step, the generative AI model analyzes the movie scene information and the chat app's comment history to infer the interactions between them. Meanwhile, the emotion analysis engine analyzes the user's emotional state from the user's tone of voice and the content of the chat messages. The input is the collected data, and the output is the analysis results and the emotional state.

[0833] Step 4:

[0834] The server generates auto-completion input data for each program based on the analysis results of the generative AI model and the emotion analysis engine. This includes, for example, automatically generating topics and questions related to moving movie scenes for a chat app. The input is the analysis results and emotional state, and the output is auto-completion input data.

[0835] Step 5:

[0836] The server sends the generated auto-completion input data to each program. Specifically, it sends the data using HTTP requests, WebSockets, etc. The input is the auto-completion input data, and the output is the data received by the corresponding program.

[0837] Step 6:

[0838] The terminal receives the auto-completion input data sent from the server. It monitors the communication from the server in real time and acquires new data. The input is the communication data from the server, and the output is the received auto-completion input data.

[0839] Step 7:

[0840] The device applies the received auto-completion input data to the corresponding program, for example, displaying a new topic or question in a chat app and updating the application UI to notify the user of the new input. The input is the received auto-completion input data, and the output is the updated UI.

[0841] Step 8:

[0842] Users use multiple programs simultaneously and experience auto-completed interactions. For example, they can watch a movie in a virtual cinema app while chatting with friends in a chat app, continuing the conversation based on newly generated topics and questions. The input is an updated UI, and the output is the user's new interaction experience.

[0843] (Application example 2)

[0844] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0845] Conventional systems provided simple navigation and entertainment information without considering the user's emotions or surrounding environmental information while driving, which did not sufficiently improve the user experience.In addition, there were few ways to accurately analyze and coordinate the interactions between multiple applications used simultaneously by the user, making it difficult to reduce stress and inconvenience while driving.

[0846] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0847] In this invention, the server includes means for detecting multiple applications run by a user, means for collecting user interaction data and situation information from each detected application, means for running a generative AI model based on the collected data to analyze and predict interactions between applications, means for generating auto-complete input data for each application based on the analysis results, means for sending the generated auto-complete input data to each application, and means for generating navigation information and entertainment information that takes into account the emotional state and surrounding environmental information of the means of transportation driven by the user based on the analyzed interaction data and situation information. This makes it possible to provide optimal navigation information and entertainment information in real time according to the user's emotions and surrounding environment, reducing stress and inconvenience while driving and improving the user experience.

[0848] The "means for detecting multiple applications run by a user" refers to a method and apparatus for identifying and listing all software and programs currently running on a user's terminal.

[0849] "Means for collecting user interaction data and situation information" refers to methods and devices for acquiring and accumulating data about the operations performed by users in each application and data about the surrounding environment.

[0850] "Means for running a generative AI model based on collected data to analyze and infer interactions between applications" refers to a method and device that uses acquired user operation history and environmental data as input, analyzes the relationships and interactions between applications using AI technology, and infers the results.

[0851] "Means for generating auto-complete input data for each application based on the analysis results" refers to a method and device for automatically creating appropriate data and information for each application based on the analysis results of the AI ​​model.

[0852] The "means for sending the generated auto-completion input data to each application" refers to a method and apparatus for transferring the generated data to the corresponding application.

[0853] "Means for generating navigation information and entertainment information that takes into account the emotional state and surrounding environmental information of the means of transportation being driven by the user based on analyzed interaction data and situational information" refers to a method and device for analyzing emotions and surrounding conditions from the user's operation history and environmental data, and providing useful route guidance and entertainment content while driving based on that.

[0854] As an embodiment of the present invention, a user assistance system for an autonomous driving vehicle will be taken as an example.

[0855] In this system, the server connects to the user's smartphone to detect multiple applications and collects interaction data and context information from each application. Based on the collected data, it runs a generative AI model to analyze the interactions between applications and generate the necessary auto-complete input data. This auto-complete input data is then sent to the corresponding application. Specifically, the system includes the following steps:

[0856] The server uses the device's API to discover all applications the user is running, which results in a list of currently active applications.

[0857] The server then collects data from each detected application about the user's operation history and surrounding environment, including GPS data while driving, the vehicle's current location, speed, surrounding traffic and weather information, and voice inputs. This data is then fed into a generative AI model to analyze the user's current situation and emotional state.

[0858] The generative AI model uses this data to analyze interactions between applications and generate optimal navigation and entertainment information to improve the user's driving experience. For example, if the user is feeling stressed, it can provide relaxing music or smooth route guidance based on that emotion.

[0859] The server then sends the generated navigation and entertainment information to each application. The device applies the received data to the corresponding application and notifies the user in real time. For example, a prompt such as "Please provide optimal driving advice and entertainment based on the current traffic situation and emotional state" could be used.

[0860] This system uses EmotionEngine to analyze the user's emotions and NavigationSystem (e.g., Google Maps API) to provide the optimal navigation route.It also improves the user's driving experience by generating entertainment information based on the collected data according to the user's driving situation and emotional state.

[0861] For example, if a user is stuck in traffic while driving an autonomous vehicle, the system uses the Emotion Engine to detect that the user is frustrated. The navigation system takes into account the current traffic situation and suggests an alternative route. The system also notifies the user, saying, "You are feeling stressed by the current traffic jam, so we will play relaxing music and suggest the optimal route to avoid it." In this way, the system can reduce stress and inconvenience while the user is driving, providing a more comfortable driving experience.

[0862] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0863] Step 1:

[0864] The server communicates with the user's device and uses an API to discover all currently running applications. Specifically, it calls the device's API to retrieve the list of active applications. The device's status information is given as input, and the list of active applications is obtained as output.

[0865] Step 2:

[0866] The server collects user interaction data and context information from each detected application, including GPS data, voice input, the vehicle's current location, speed, and environmental data such as traffic and weather. The input is the list of active applications, and the output is the collected interaction data and context information.

[0867] Step 3:

[0868] The server inputs the collected data into a generative AI model to analyze and predict interactions between applications. The generative AI model analyzes the user's operation history and environmental data to analyze the relationships and interactions between applications. The collected data is given as input, and the analysis and prediction of interactions between applications is obtained as output.

[0869] Step 4:

[0870] The server generates auto-completion input data for each application based on the analysis results of the generative AI model. For example, it creates navigation information or entertainment information taking into account the user's emotional state and surrounding environment. The analysis results are given as input, and the auto-completion input data is generated as output.

[0871] Step 5:

[0872] The server sends the generated auto-completion input data to each application. Each application receives and applies this data. The auto-completion input data is given as input, and the corresponding application is updated as output.

[0873] Step 6:

[0874] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, the device periodically checks for communication from the server and acquires new data. Data from the server is given as input, and the received data is saved on the device as output.

[0875] Step 7:

[0876] The device applies the received data to the corresponding application and notifies the user, for example, displaying new route directions in a navigation application or instructing a music application to play relaxing music. The received data is given as input, and the application's UI is updated as output.

[0877] Step 8:

[0878] The user experiences complementary interactions, for example, receiving new route directions from a navigation app while driving and playing relaxing music from a music app, with application notifications as input and an improved driving experience as output.

[0879] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0880] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0881] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0882] [Third embodiment]

[0883] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0884] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0885] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0886] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0887] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0888] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0889] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0890] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0891] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0892] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0893] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0894] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0895] The present invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model, and generates auto-complete input data for each application, realizing collaboration between applications.

[0896] Server Processing

[0897] 1. Application Discovery

[0898] The server communicates with the user's device to detect each application the user is using. This allows the server to obtain information about all applications running on the device. For example, if a user is using a virtual movie theater app and a chat app at the same time, that information is notified to the server.

[0899] 2. Data Collection

[0900] The server then collects user interaction data and context information from each detected application, including movie scene information from a virtual cinema app and user comment history from a chat app.

[0901] 3. Generative AI model in action

[0902] A generative AI model is run based on the collected data. For example, movie scene information and utterances from a chat app are used as input. The model analyzes and infers the data to generate topics and questions related to movie scenes.

[0903] 4. Input completion data generation

[0904] The server generates auto-complete input data for each application based on the analysis results of the generative AI model. For example, questions and topics related to movie scenes are generated for a chat app.

[0905] 5. Data Transmission

[0906] Finally, the server sends the generated auto-complete input data to each application, allowing the chat app to suggest new topics and questions related to movie scenes to the user.

[0907] Terminal handling

[0908] 1. Receiving notifications

[0909] The user's device receives the auto-complete input data sent from the server, which allows the new data to be reflected in real time.

[0910] 2. Data Application

[0911] The device applies the received data to each application. For example, it might display topics related to movie scenes in a chat app to encourage user interaction. It might also update the application's UI and notify the user of new input.

[0912] User Action

[0913] 1. App Usage

[0914] Users can use multiple xR applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[0915] 2. Complementary Experiences

[0916] Users experience auto-complete interactions with each application, for example, sharing topics with friends based on a question related to a movie scene displayed in a chat app, resulting in a more natural and rich user experience.

[0917] Specific examples

[0918] Example 1: Linking a virtual movie theater app with a chat app

[0919] Consider a case where a user is watching a movie on a virtual cinema app while chatting with a friend on a chat app. The server collects information about movie scenes and the content of comments made in the chat app. The generative AI model uses this data to generate questions and topics related to the movie scenes. The generated data is sent to the chat app, which receives it and presents it to the user. This allows the user to share topics based on movie scenes with their friends.

[0920] Example 2: Integration of an educational app and a note-taking app

[0921] Consider a case where a user is taking a class using an educational app while recording what they learned using a note-taking app. The server collects the class content data and the contents of the notes. The generative AI model uses this data to generate supplementary information to be added to the notes and hints for related problems. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, the user can easily record supplementary information related to the class.

[0922] Through these processes, the system of the present invention realizes information sharing and collaboration between multiple applications, improving the user experience.

[0923] The processing flow will be explained below.

[0924] Server Processing

[0925] Step 1:

[0926] Application Discovery

[0927] The server communicates with the user's device to discover all currently running applications, specifically by calling the device's API to retrieve the list of active applications.

[0928] Step 2:

[0929] Data collection

[0930] The server collects user interaction data and situation information from each detected application. Specifically, it obtains movie playback time and scene information from the movie theater app, and speech history from the chat app. This is also done through each application's API.

[0931] Step 3:

[0932] Generative AI model in action

[0933] The server inputs the collected data and runs a generative AI model. Specifically, it inputs movie scene information and speech history into the model, analyzes them, and predicts interactions.

[0934] Step 4:

[0935] Input completion data generation

[0936] Based on the analysis results of the generative AI model, auto-complete input data is generated for each application. For example, it generates chat topics related to movie scene information.

[0937] Step 5:

[0938] Data transmission

[0939] The server sends the generated auto-complete input data to each application, again via the API of each application.

[0940] Terminal handling

[0941] Step 1:

[0942] Receive notifications

[0943] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, it periodically checks communications from the server and acquires new data.

[0944] Step 2:

[0945] Data Application

[0946] The device applies the received data to the corresponding application, for example updating the UI to display new topics or questions in a chat app.

[0947] User Action

[0948] Step 1:

[0949] App use

[0950] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[0951] Step 2:

[0952] Complementary Experiences

[0953] Users experience auto-completed interactions, specifically continuing conversations with friends based on topics and questions related to movie scenes displayed in the chat app.

[0954] In this way, through the specific operations at each step, cooperation between the server, terminal, and user is realized, providing a richer user experience.

[0955] Example 1

[0956] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0957] Modern users often use multiple applications simultaneously and require enhanced user experiences through information sharing and collaboration between these applications. However, data sharing and auto-completion input between different applications is often performed manually, making it difficult to provide an efficient and natural user experience. For this reason, there is a demand for a system that automatically shares information and collaborates between multiple applications to improve user interaction.

[0958] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0959] In this invention, the server includes means for detecting multiple applications run by a user, means for collecting user interaction data and situation information from each detected application, means for running a generative AI model based on the collected data to analyze and predict interactions between applications, means for generating auto-complete input data for each application based on the analysis results, and means for transmitting the generated auto-complete input data to each application, thereby enabling automatic information sharing and collaboration between different applications to provide an efficient and natural user experience.

[0960] "User" refers to a person or entity that uses a system or application.

[0961] "Application" is a general term for a software program that a user runs on a terminal.

[0962] "Interaction Data" refers to data regarding inputs and interactions when a user operates an application.

[0963] "Status information" refers to information related to the user's usage environment and status, including, for example, the application being used and the status of its content.

[0964] A "generative AI model" refers to an artificial intelligence model that analyzes collected data and automatically generates input data.

[0965] "Auto-complete input data" refers to the input data generated by the generative AI model to be applied to each application.

[0966] "Server" refers to a computer system that communicates with a user's terminal, collects data, analyzes the data, and transmits the generated data.

[0967] "Terminal" refers to a device that a user uses to run an application, and examples include a computer and a smartphone.

[0968] "API" stands for Application Programming Interface and refers to a means by which different pieces of software communicate with each other.

[0969] An "HTTP POST request" is a part of the HTTP protocol used to send data to a server.

[0970] This invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model, and generates auto-complete input data for each application, realizing collaboration between applications.

[0971] This system mainly consists of three main components: a server, a terminal, and a user.

[0972] Server Roles

[0973] The server first communicates with the user's terminal and detects each application the user is running. For example, the netstat or ps command is used to obtain information about applications running on the terminal. This information is sent to the server using the UDP or TCP protocol. For example, the netstat -an | grep 'ESTABLISHED' command can be used to identify connected applications.

[0974] Next, the server collects user interaction data and context information from each detected application. The collected data is obtained through API calls and log data analysis. For example, movie scene information can be collected from a virtual movie theater app, and user comment history can be collected from a chat app.

[0975] Based on the collected data, the server runs a generative AI model. For example, GPT-4 can be used as the generative AI model. By preprocessing the data using Python and inputting it into the generative AI model, topics and questions related to the movie scenes can be generated using movie scene information and the content of comments made in the chat app as input.

[0976] Next, the server generates auto-complete input data based on the analysis results of the generative AI model. For example, if the generative AI model generates a "question about a movie scene," it converts it into JSON format so that the chat app can understand it. This completion data is sent to each application using an HTTP POST request. For example, you can POST the data to http: / / chatapp.example.com / api / v1 / messages to reflect the new question in the chat app.

[0977] Device Role

[0978] The user's device receives the auto-complete input data sent from the server. You can receive the data in real time using a listener in the device (for example, a WebSocket server). Create an onMessage event handler to process the data as soon as it arrives.

[0979] The device then applies the received data to each application, for example updating the UI of a chat app to show the new question, or if you're using React, updating the state and re-rendering the component to show the new question.

[0980] User Roles

[0981] Users use multiple applications simultaneously. For example, they may watch a movie on a virtual cinema app while chatting with friends on a chat app. As users interact with the applications, their activities are sent to a server and data is collected.

[0982] The collected data is then automatically filled in and applied to the device, allowing users to enjoy new experiences. For example, a user can share a topic with a friend based on a question related to a movie scene displayed in a chat app. This allows the user's conversation to progress more naturally, resulting in richer communication.

[0983] Specific examples

[0984] Example 1: Linking a virtual movie theater app with a chat app

[0985] When a user watches a movie in a virtual cinema app and chats with a friend in a chat app, the server collects information about the movie's scenes and the content of the chat app's comments. The generative AI model uses this data to generate questions and topics related to the movie's scenes. The generated data is sent to the chat app, which receives it and presents it to the user. In this way, the user can share topics based on the movie's scenes with their friends.

[0986] Examples of specific prompts include:

[0987] "What do you think about the characters in the scene you're watching?"

[0988] "Let's talk about the events behind this scene."

[0989] Example 2: Integration of an educational app and a note-taking app

[0990] When a user takes a class using an educational app and records their learning using a note-taking app, the server collects the class content data and the contents of the notes. The generative AI model uses this data to generate supplementary information to be added to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, users can easily record supplementary information related to the class.

[0991] Examples of specific prompts include:

[0992] "Please note down any important points related to this topic."

[0993] "Let's dig deeper into the ideas explained in class."

[0994] Through these processes, the system of the present invention can realize information sharing and collaboration between multiple applications, thereby improving the user experience.

[0995] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0996] Step 1:

[0997] The server detects multiple applications that a user runs.

[0998] Specifically, the server communicates with the user's terminal via TCP or UDP protocol and uses the netstat or ps command to collect information about running applications. The information obtained includes the application name, version, and status information indicating that it is running. Based on this information, it identifies which applications are currently running.

[0999] Input: Application execution status on user device

[1000] Output: Information about the running application (name, version, status, etc.)

[1001] Step 2:

[1002] The server collects user interaction data and context information from each detected application.

[1003] Specifically, the server uses the API of each application to obtain information such as what operations the user performed and what content is being played. For example, it collects information about the currently playing scene from a virtual movie theater app, and the user's comment history from a chat app. This information is collected by calling APIs and analyzing log files.

[1004] Input: Running application information

[1005] Output: Interaction data and context information for each application

[1006] Step 3:

[1007] The server runs a generative AI model based on the collected data.

[1008] Specifically, the server preprocesses the data using Python and inputs it into a generative AI model (e.g., GPT-4). The input data includes movie scene information and comments from a chat app. The generative AI model analyzes this data and generates appropriate complementary information (e.g., new questions or topics).

[1009] Input: Interaction data and context information

[1010] Output: Complementary information (e.g., questions and topics) generated by the generative AI model

[1011] Step 4:

[1012] The server generates input completion data based on the analysis results of the generative AI model.

[1013] Specifically, the server converts the data obtained from the generative AI model into an appropriate format, such as JSON, and formats it in a way that each application can understand. For example, it converts the generated question into a format like {"type": "question", "content": "Who are the characters in this scene?"}.

[1014] Input: Analysis results of the generative AI model

[1015] Output: Input completion data applicable to each application

[1016] Step 5:

[1017] The server transmits the generated auto-completion input data to each application.

[1018] Specifically, the server sends data using an HTTP POST request, for example to http: / / chatapp.example.com / api / v1 / messages, to reflect a new question in the chat app.

[1019] Input: Input completion data

[1020] Output: Data sent to each application

[1021] Step 6:

[1022] The user's terminal receives the auto-complete input data sent from the server.

[1023] Specifically, the device receives data in real time using a locally running listener (e.g., a WebSocket server). When the onMessage event handler receives the data, it executes the appropriate process.

[1024] Input: Data sent to each application

[1025] Output: Type-ahead data received on the device

[1026] Step 7:

[1027] The terminal applies the received data to each application and notifies the user.

[1028] Specifically, the device reflects the received data in the UI of the corresponding application and notifies the user of new input. For example, if you are using React, it will update the state and re-render the component displaying the new question.

[1029] Input: Received type-ahead data

[1030] Output: Updated application UI, notification to the user

[1031] Step 8:

[1032] The user uses the application and experiences the autocompleted interaction.

[1033] Specifically, users can engage in conversations with friends based on questions related to movie scenes presented in a chat app, resulting in a more natural and enriching user experience.

[1034] Input: Updated application UI

[1035] Output: Improved user interaction and experience

[1036] (Application example 1)

[1037] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1038] When users use multiple content distribution services on the Internet, each service operates independently, making it difficult to provide a unified information experience. It has also been pointed out that there is a lack of means to provide real-time updates or additional content related to the media content being viewed or listened to. This results in a fragmented experience, as users are unable to link information between content or receive seamless information.

[1039] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1040] In this invention, the server includes means for detecting multiple programs run by a user, means for collecting user interaction data and situation information from each detected program, means for running a generative AI model based on the collected data to analyze and predict interactions between programs, means for generating auto-complete input data for each program based on the analysis results, means for sending the generated auto-complete input data to each program, and means for automatically generating news and additional information related to the media content being viewed or listened to in cooperation with the content distribution service to which each program belongs. This enables information related to the media content being viewed or listened to to be provided in real time when a user is using multiple content distribution services, thereby realizing a seamless information experience.

[1041] A "program" is a set of instructions or code that runs on a computer to accomplish a particular task or function.

[1042] "Interaction data" is data generated when a user operates a program, and includes operation logs such as clicks, taps, inputs, and viewing time.

[1043] "Contextual information" is data that describes the context or conditions identified during program execution, including the user's geographic location, device state, current activity, etc.

[1044] A "generative AI model" is an artificial intelligence algorithm that analyzes patterns and relationships based on collected data and makes predictions and inferences. In this case, it is used specifically for natural language generation and data completion.

[1045] "Auto-complete input data" refers to suggested data or additional information generated by a generative AI model and used as input data for a specific program.

[1046] A "content distribution service" is a service that provides users with multimedia content such as videos, music, and news via the Internet.

[1047] "Related news and additional information" refers to information or content that is related to the media content that the user is watching or listening to and is provided in a manner that complements it.

[1048] MODE FOR CARRYING OUT THE INVENTION

[1049] This invention relates to a system that analyzes the interactions between multiple programs when a user is using them simultaneously, improving the user experience. The system uses a generative AI model to automatically generate related information based on interaction data and context information, and establishes collaboration between the programs.

[1050] Server Processing

[1051] The server has the following means:

[1052] 1. Program detection methods

[1053] It communicates with the user's device and detects the multiple programs the user is running. This method identifies all programs running on the device. For example, if a user is using a video streaming service and a news app at the same time, that information is notified to the server.

[1054] 2. Data Collection Methods

[1055] For each program detected, it collects user interaction data and contextual information, such as information about the video being watched, browsing history in news apps, etc.

[1056] 3. Means of running the generative AI model

[1057] Based on the collected data, a generative AI model is run to analyze the interactions between programs. The generative AI model analyzes the user's interaction data and generates related information based on it. For example, it generates additional information or topics related to the movie or news being watched.

[1058] 4. Auto-complete input data generation method

[1059] Based on the analysis, it generates auto-complete entries for each program, including news and additional information related to the content being viewed.

[1060] 5. Data Transmission Method

[1061] The generated auto-complete input data is sent to each program, which allows the news app to present new related information and news articles to the user.

[1062] Terminal handling

[1063] The terminal has the following means:

[1064] 1. Notification Reception Method

[1065] Receives auto-complete input data sent from the server, allowing the device to reflect new data in real time.

[1066] 2. Data application method

[1067] The received data is applied to each program and notified to the user, allowing the news app to display relevant news and additional information to the user in real time.

[1068] User Action

[1069] The user has the following means:

[1070] 1. How to use the program

[1071] Use multiple programs at the same time. For example, watch a movie on a streaming service while checking the latest news on a news app.

[1072] 2. Complementary Experiences

[1073] Experience auto-complete information in each program. For example, you can browse more interesting topics based on movie-related news displayed in a news app.

[1074] Specific examples

[1075] As a specific example, consider a case where while a user is watching a "documentary about climate change" on a video streaming service, the system of this invention automatically generates and displays news articles about "the latest research results on climate change" and "the latest trends in the environmental protection movement" in a news app. In this way, additional information that is closely related to the content the user is watching can be provided in real time.

[1076] Example prompt sentence:

[1077] Input: Viewing data for the video "Documentary on Climate Change"

[1078] Output: Related news articles (title, content)

[1079] This allows users to seamlessly experience related information and news while using content distribution services.

[1080] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1081] Step 1:

[1082] Program Detection

[1083] The server communicates with the user's terminal and detects all programs the user is running.

[1084] Input: A list of running programs from the user's terminal

[1085] Output: Detected program information

[1086] Specific operation: A monitoring agent installed on the user's terminal collects information about running programs and sends it to the server.

[1087] Step 2:

[1088] Data collection

[1089] The server collects user interaction data and context information from each detected program.

[1090] Input: Detected program information

[1091] Output: Interaction data and context information from each program

[1092] Specific operations: Collects data such as video playback time, content being viewed, and user operation logs.

[1093] Step 3:

[1094] Generative AI model in operation

[1095] The server runs a generative AI model based on the collected data and analyzes interactions between programs.

[1096] Input: Interaction data and context information

[1097] Output: Analysis results (related news and additional information)

[1098] Specific operation: The collected data is input into a generative AI model for analysis. The generative AI model used here is GPT-3.

[1099] Step 4:

[1100] Auto-complete input data generation

[1101] The server generates auto-complete input data for each program based on the analysis results of the generative AI model.

[1102] Input: Analysis results

[1103] Output: Auto-complete input data

[1104] Specific operation: Based on the generated data, templates for news articles and additional information are created and their content is automatically generated.

[1105] Step 5:

[1106] Data transmission

[1107] The server transmits the generated auto-completion input data to each program.

[1108] Input:Auto-complete input data

[1109] Output: Data to send to each program

[1110] Specific operation: Encode and send the generated data in an appropriate format to the corresponding program in real time.

[1111] Step 6:

[1112] Receive notifications

[1113] The terminal receives the auto-completion input data sent from the server.

[1114] Input: Data to be sent from the server

[1115] Output: Received complementary data

[1116] Specific operation: The program on the terminal receives data from the server and stores the contents in temporary storage.

[1117] Step 7:

[1118] Data Application

[1119] The terminal applies the received data to each program and notifies the user.

[1120] Input: Received complementary data

[1121] Output: Update information for each program

[1122] Specific behavior: Update the UI of each program to apply the received data and notify the user that new information is available.

[1123] Step 8:

[1124] Program Use

[1125] A user may use multiple programs simultaneously.

[1126] Input: Related news and additional information

[1127] Output: User Interaction

[1128] Specific operations: A user watches a movie on a video streaming service while checking related information on a news app.

[1129] Step 9:

[1130] Complementary Experiences

[1131] Users experience auto-completed information in each program.

[1132] Input: User interaction

[1133] Output: Improving the user experience

[1134] Specific behavior: By reading news articles related to a movie displayed in a news app, you can deepen your understanding of the movie's content.

[1135] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1136] This invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model and an emotion engine, and generates auto-complete input data for each application, realizing collaboration between applications.

[1137] Server Processing

[1138] 1. Application Discovery

[1139] The server communicates with the user's device to detect all currently running applications. Specifically, it calls the device's API to obtain a list of active applications. For example, if a user is using a virtual movie theater app and a chat app at the same time, that information is notified to the server.

[1140] 2. Data Collection

[1141] The server then collects user interaction data and context information from each detected application, such as movie scene information from a virtual cinema app or user comment history from a chat app, and provides the collected data to a generative AI model and emotion engine.

[1142] 3. Generative AI model and emotion engine in action

[1143] The server uses the collected data as input to run a generative AI model and an emotion engine. The generative AI model analyzes movie scene information and chat app comment history to infer interactions. Meanwhile, the emotion engine analyzes the user's emotions from the user's interaction data and situational information. For example, the emotion engine infers the user's emotional state from the user's tone of voice and text content.

[1144] 4. Input completion data generation

[1145] Based on the analysis results of the generative AI model and emotion engine, auto-complete input data for each application is generated. For example, data containing appropriate tone and topics according to the user's emotional state is generated, in addition to casual conversation topics related to movie scene information.

[1146] 5. Data Transmission

[1147] Finally, the server sends the generated auto-complete input data to each application, allowing the chat app to present new topics and questions to the user that are adapted to the movie scenes and the user's emotions.

[1148] Terminal handling

[1149] 1. Receiving notifications

[1150] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, it periodically checks communications from the server and acquires new data.

[1151] 2. Data Application

[1152] The device applies the received data to the corresponding application, for example, displaying a new topic or question in a chat app, or updating the application's UI to notify the user of the new input.

[1153] User Action

[1154] 1. App Usage

[1155] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[1156] 2. Complementary Experiences

[1157] Users experience auto-complete interactions, such as continuing a conversation with a friend based on a movie scene displayed in a chat app or topics and questions related to the user's emotions, resulting in a more natural and rich user experience.

[1158] Specific examples

[1159] Example 1: Linking a virtual movie theater app with a chat app

[1160] Consider a case where a user is watching a movie on a virtual cinema app while chatting with a friend on a chat app. The server collects information about movie scenes, comments made on the chat app, and the user's emotional data analyzed by the emotion engine. Based on this data, the generative AI model and emotion engine generate questions and topics related to the movie scenes and adapted to the user's emotional state. The generated data is sent to the chat app, which receives it on the device and presents it to the user. In this way, the user can share topics based on the movie scenes and their own emotions with their friends.

[1161] Example 2: Integration of an educational app and a note-taking app

[1162] Consider a case where a user takes a class using an educational app while recording their learning using a note-taking app. The server collects the class content data, the note content, and the user's emotional data analyzed by the emotion engine. Based on this data, the generative AI model and emotion engine generate supplementary information to add to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, users can not only easily record supplementary information related to the class, but also receive optimal learning experiences tailored to their emotional state.

[1163] Through these processes, the system of the present invention enables information sharing and collaboration between multiple applications, not only improving the user experience but also providing optimal interactions according to the user's emotional state.

[1164] The processing flow will be explained below.

[1165] Server Processing

[1166] Step 1:

[1167] Application Discovery

[1168] The server communicates with the user's device to discover all currently running applications, specifically by calling the device's API to retrieve the list of active applications.

[1169] Step 2:

[1170] Data collection

[1171] The server collects user interaction data and context information from each detected application. Specifically, it obtains movie playback time and scene information from the virtual cinema app, and speech history from the chat app. It also collects the user's voice tone and text content as input data for the emotion engine.

[1172] Step 3:

[1173] Generative AI model and emotion engine in action

[1174] The server uses the collected data as input to run a generative AI model and an emotion engine. The generative AI model analyzes movie scene information and chat app comment history to predict interactions. The emotion engine analyzes the user's voice tone and text content to evaluate the user's emotional state.

[1175] Step 4:

[1176] Applying Emotion Data

[1177] Based on the analysis results of the emotion engine, the generative AI model adjusts the auto-complete input data for each application. For example, if the user is excited, it generates input data that is appropriate for that excited state.

[1178] Step 5:

[1179] Input completion data generation

[1180] Based on the analysis results of the generative AI model and emotion engine, auto-complete input data for each application is generated. For example, data is generated that includes not only casual conversation topics related to movie scene information, but also tone and topics that correspond to the user's emotional state.

[1181] Step 6:

[1182] Data transmission

[1183] The server sends the generated auto-completion input data to each application via the application's API, where it is applied by the corresponding application.

[1184] Terminal handling

[1185] Step 1:

[1186] Receive notifications

[1187] The device monitors and receives auto-complete input data sent from the server in real time, and periodically checks for communication from the server to see if there is new data.

[1188] Step 2:

[1189] Data Application

[1190] The device applies the received data to the corresponding application, for example, updating the UI to display new topics or questions in a chat app. Based on the emotion data, the device presents content tailored to the user.

[1191] User Action

[1192] Step 1:

[1193] App use

[1194] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[1195] Step 2:

[1196] Complementary Experiences

[1197] Users experience auto-complete interactions, such as continuing a conversation with a friend based on a movie scene displayed in a chat app or topics and questions related to the user's emotions, providing a more natural and enriching experience.

[1198] Specific examples

[1199] Example 1: Linking a virtual movie theater app with a chat app

[1200] Step 1:

[1201] The server detects the use of the virtual cinema app and the chat app.

[1202] Step 2:

[1203] The server collects movie playback scene information, chat app comment history, and the user's voice tone.

[1204] Step 3:

[1205] The server uses a generative AI model to analyze movie scene information and speech history, and uses an emotion engine to evaluate the user's emotional state.

[1206] Step 4:

[1207] Based on the results of the emotion engine, the generative AI model adjusts chat topics to suit the user's emotional state.

[1208] Step 5:

[1209] The server generates input data adjusted based on the emotional state and sends it to the chat app.

[1210] Step 6:

[1211] The device receives the transmitted data, reflects it in the chat app, and presents new topics to the user.

[1212] Example 2: Integration of an educational app and a note-taking app

[1213] Step 1:

[1214] The server detects the use of educational apps and note-taking apps.

[1215] Step 2:

[1216] The server collects lesson content data, note content, and user text content.

[1217] Step 3:

[1218] The server uses a generative AI model to analyze the lesson content and an emotion engine to evaluate the user's emotional state.

[1219] Step 4:

[1220] Based on the results of the emotion engine, a generative AI model tailors supplemental information and hints that correspond to the user's emotional state.

[1221] Step 5:

[1222] The server generates complementary data adjusted based on the emotional state and sends it to the note-taking app.

[1223] Step 6:

[1224] The device receives the transmitted data, reflects it in the note application, and adds supplementary information to the user's notes.

[1225] Through these processes, the system of the present invention can realize information sharing and collaboration among multiple applications, and can also provide optimal interactions according to the user's emotional state.

[1226] Example 2

[1227] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1228] In conventional systems, when a user operates multiple programs simultaneously, there is a problem of a fragmented user experience due to a lack of information sharing or collaboration between the programs. Therefore, there is a need for a system that allows users to smoothly exchange information between different programs. There is also a need to provide a more natural and rich experience by realizing interactions that take the user's emotional state into consideration.

[1229] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1230] In this invention, the server includes means for detecting multiple programs run by a user, means for collecting user operation data and situation information from each detected program, means for running a generative AI model and an emotion analysis engine based on the collected data to analyze and predict interactions between programs, means for generating auto-complete input data for each program based on the analysis results, and means for transmitting the generated auto-complete input data to each program. This enables information sharing and collaboration between different programs, and can provide optimal interactions to the user according to their emotional state.

[1231] "Program" refers to multiple applications and software that can be operated by a user.

[1232] "Operation data" refers to information about a series of operations performed by a user within a program.

[1233] "Status Information" refers to information relating to the state of a program and its contents.

[1234] "Generative AI model" refers to an artificial intelligence model that analyzes and predicts interactions between programs based on collected interaction data and situational information.

[1235] "Emotion analysis engine" refers to an engine for analyzing a user's emotional state from the user's interaction data and context information.

[1236] "Auto-completion input data" refers to data generated to complement a user's operation.

[1237] "Information sharing" refers to the exchange and sharing of information between different programs.

[1238] "Integration" refers to multiple programs working together to provide a unified experience.

[1239] An "interaction" refers to a series of exchanges or operations between a user and a program.

[1240] "Analysis Results" refers to the analysis results of data obtained by the generative AI model and sentiment analysis engine.

[1241] "User experience" refers to the overall impression and satisfaction a user feels when operating each program.

[1242] This invention relates to a system that analyzes the interactions between multiple programs when a user is using them simultaneously, improving the user experience. This system collects user operation data and situational information, analyzes them using a generative AI model and an emotion analysis engine, and generates auto-complete input data for each program, realizing collaboration between programs.

[1243] Specifically, the server operates as follows.

[1244] The server first communicates with the user's device to detect all programs currently running. A specific example of this operation is to call the device's API (e.g., ActivityManager in Android or UIApplication in iOS) to obtain a list of active programs. The information about the programs detected in this way is then notified to the server.

[1245] Next, the server collects user operation data and context information from each detected program. This information is collected using the API of each program. For example, a virtual movie theater app might collect information about the scene currently playing in a movie, while a chat app might collect the user's comment history. This information is then provided to the generative AI model and sentiment analysis engine.

[1246] The server uses the collected data as input to run a generative AI model and an emotion analysis engine. The generative AI model, for example, analyzes movie scene information and chat app comment history to infer the interactions between them. Meanwhile, the emotion analysis engine infers the user's emotional state from the tone of their voice and the content of their chat messages.

[1247] Based on the analysis results of the generative AI model and the emotion analysis engine, the server generates auto-complete input data for each program. As a specific example, data is generated that includes not only chat topics related to movie scene information, but also appropriate tones and topics according to the user's emotional state. For example, appropriate questions and comments related to a moving movie scene are generated and sent to a chat app.

[1248] The auto-complete input data sent from the server is received by the device. The device periodically checks for communication from the server and obtains new data. This received data is applied to the corresponding program, for example, a new topic or question is displayed in a chat app. The device uses this data to update the program's UI and notify the user that new input has been received. Specifically, this is done by updating the RecyclerView on Android, or by using the refresh function of UITableView on iOS.

[1249] Users can enjoy a more natural and richer experience with this updated program. For example, when watching a movie in a virtual cinema app and chatting with friends in a chat app, they can enjoy auto-complete interactions. Topics and questions related to the movie scene and their own emotions are displayed, making the conversation with friends more natural and smooth.

[1250] Below are some examples of specific prompt sentences.

[1251] Example 1: Linking a virtual movie theater app with a chat app

[1252] When a user watches a movie in a virtual cinema app and chats with a friend in a chat app, the server collects information about the movie's scenes and the chat app's comments and emotional data. The generative AI model and emotion analysis engine use this data to generate questions and topics related to the movie's scenes and adapted to the user's emotional state. The generated data is sent to the chat app, which receives it and presents it to the user. This process allows the user to share topics based on the movie's scenes and their emotions with their friends.

[1253] Example 2: Integration of an educational app and a note-taking app

[1254] When a user takes a class using an educational app and records their learning using a note-taking app, the server collects the class content data, the note content, and emotional data. Based on this data, the generative AI model and emotional analysis engine generate supplementary information to add to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. This process allows users to easily record supplementary information related to the class and provides optimal learning based on their emotional state.

[1255] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1256] Step 1:

[1257] The server communicates with the user's device and detects all currently running programs. Specifically, the server calls the device's API (e.g., ActivityManager on Android or UIApplication on iOS) to obtain a list of active programs. The input data is information about the programs running on the device, and the output is a list of running programs.

[1258] Step 2:

[1259] The server collects user operation data and status information from each detected program. This is done through each program's API. For example, a virtual movie theater app collects information about the scene currently playing in a movie, and a chat app collects the user's comment history. The input data is the API call results for each program, and the output is the operation data and status information collected from each program.

[1260] Step 3:

[1261] The server provides the collected operation data and situation information as input to the generative AI model and the emotion analysis engine. In this step, the generative AI model analyzes the movie scene information and the chat app's comment history to infer the interactions between them. Meanwhile, the emotion analysis engine analyzes the user's emotional state from the user's tone of voice and the content of the chat messages. The input is the collected data, and the output is the analysis results and the emotional state.

[1262] Step 4:

[1263] The server generates auto-completion input data for each program based on the analysis results of the generative AI model and the emotion analysis engine. This includes, for example, automatically generating topics and questions related to moving movie scenes for a chat app. The input is the analysis results and emotional state, and the output is auto-completion input data.

[1264] Step 5:

[1265] The server sends the generated auto-completion input data to each program. Specifically, it sends the data using HTTP requests, WebSockets, etc. The input is the auto-completion input data, and the output is the data received by the corresponding program.

[1266] Step 6:

[1267] The terminal receives the auto-completion input data sent from the server. It monitors the communication from the server in real time and acquires new data. The input is the communication data from the server, and the output is the received auto-completion input data.

[1268] Step 7:

[1269] The device applies the received auto-completion input data to the corresponding program, for example, displaying a new topic or question in a chat app and updating the application UI to notify the user of the new input. The input is the received auto-completion input data, and the output is the updated UI.

[1270] Step 8:

[1271] Users use multiple programs simultaneously and experience auto-completed interactions. For example, they can watch a movie in a virtual cinema app while chatting with friends in a chat app, continuing the conversation based on newly generated topics and questions. The input is an updated UI, and the output is the user's new interaction experience.

[1272] (Application example 2)

[1273] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1274] Conventional systems provided simple navigation and entertainment information without considering the user's emotions or surrounding environmental information while driving, which did not sufficiently improve the user experience.In addition, there were few ways to accurately analyze and coordinate the interactions between multiple applications used simultaneously by the user, making it difficult to reduce stress and inconvenience while driving.

[1275] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1276] In this invention, the server includes means for detecting multiple applications run by a user, means for collecting user interaction data and situation information from each detected application, means for running a generative AI model based on the collected data to analyze and predict interactions between applications, means for generating auto-complete input data for each application based on the analysis results, means for sending the generated auto-complete input data to each application, and means for generating navigation information and entertainment information that takes into account the emotional state and surrounding environmental information of the means of transportation driven by the user based on the analyzed interaction data and situation information. This makes it possible to provide optimal navigation information and entertainment information in real time according to the user's emotions and surrounding environment, reducing stress and inconvenience while driving and improving the user experience.

[1277] The "means for detecting multiple applications run by a user" refers to a method and apparatus for identifying and listing all software and programs currently running on a user's terminal.

[1278] "Means for collecting user interaction data and situation information" refers to methods and devices for acquiring and accumulating data about the operations performed by users in each application and data about the surrounding environment.

[1279] "Means for running a generative AI model based on collected data to analyze and infer interactions between applications" refers to a method and device that uses acquired user operation history and environmental data as input, analyzes the relationships and interactions between applications using AI technology, and infers the results.

[1280] "Means for generating auto-complete input data for each application based on the analysis results" refers to a method and device for automatically creating appropriate data and information for each application based on the analysis results of the AI ​​model.

[1281] The "means for sending the generated auto-completion input data to each application" refers to a method and apparatus for transferring the generated data to the corresponding application.

[1282] "Means for generating navigation information and entertainment information that takes into account the emotional state and surrounding environmental information of the means of transportation being driven by the user based on analyzed interaction data and situational information" refers to a method and device for analyzing emotions and surrounding conditions from the user's operation history and environmental data, and providing useful route guidance and entertainment content while driving based on that.

[1283] As an embodiment of the present invention, a user assistance system for an autonomous driving vehicle will be taken as an example.

[1284] In this system, the server connects to the user's smartphone to detect multiple applications and collects interaction data and context information from each application. Based on the collected data, it runs a generative AI model to analyze the interactions between applications and generate the necessary auto-complete input data. This auto-complete input data is then sent to the corresponding application. Specifically, the system includes the following steps:

[1285] The server uses the device's API to discover all applications the user is running, which results in a list of currently active applications.

[1286] The server then collects data from each detected application about the user's operation history and surrounding environment, including GPS data while driving, the vehicle's current location, speed, surrounding traffic and weather information, and voice inputs. This data is then fed into a generative AI model to analyze the user's current situation and emotional state.

[1287] The generative AI model uses this data to analyze interactions between applications and generate optimal navigation and entertainment information to improve the user's driving experience. For example, if the user is feeling stressed, it can provide relaxing music or smooth route guidance based on that emotion.

[1288] The server then sends the generated navigation and entertainment information to each application. The device applies the received data to the corresponding application and notifies the user in real time. For example, a prompt such as "Please provide optimal driving advice and entertainment based on the current traffic situation and emotional state" could be used.

[1289] This system uses EmotionEngine to analyze the user's emotions and NavigationSystem (e.g., Google Maps API) to provide the optimal navigation route.It also improves the user's driving experience by generating entertainment information based on the collected data according to the user's driving situation and emotional state.

[1290] For example, if a user is stuck in traffic while driving an autonomous vehicle, the system uses the Emotion Engine to detect that the user is frustrated. The navigation system takes into account the current traffic situation and suggests an alternative route. The system also notifies the user, saying, "You are feeling stressed by the current traffic jam, so we will play relaxing music and suggest the optimal route to avoid it." In this way, the system can reduce stress and inconvenience while the user is driving, providing a more comfortable driving experience.

[1291] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1292] Step 1:

[1293] The server communicates with the user's device and uses an API to discover all currently running applications. Specifically, it calls the device's API to retrieve the list of active applications. The device's status information is given as input, and the list of active applications is obtained as output.

[1294] Step 2:

[1295] The server collects user interaction data and context information from each detected application, including GPS data, voice input, the vehicle's current location, speed, and environmental data such as traffic and weather. The input is the list of active applications, and the output is the collected interaction data and context information.

[1296] Step 3:

[1297] The server inputs the collected data into a generative AI model to analyze and predict interactions between applications. The generative AI model analyzes the user's operation history and environmental data to analyze the relationships and interactions between applications. The collected data is given as input, and the analysis and prediction of interactions between applications is obtained as output.

[1298] Step 4:

[1299] The server generates auto-completion input data for each application based on the analysis results of the generative AI model. For example, it creates navigation information or entertainment information taking into account the user's emotional state and surrounding environment. The analysis results are given as input, and the auto-completion input data is generated as output.

[1300] Step 5:

[1301] The server sends the generated auto-completion input data to each application. Each application receives and applies this data. The auto-completion input data is given as input, and the corresponding application is updated as output.

[1302] Step 6:

[1303] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, the device periodically checks for communication from the server and acquires new data. Data from the server is given as input, and the received data is saved on the device as output.

[1304] Step 7:

[1305] The device applies the received data to the corresponding application and notifies the user, for example, displaying new route directions in a navigation application or instructing a music application to play relaxing music. The received data is given as input, and the application's UI is updated as output.

[1306] Step 8:

[1307] The user experiences complementary interactions, for example, receiving new route directions from a navigation app while driving and playing relaxing music from a music app, with application notifications as input and an improved driving experience as output.

[1308] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1309] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1310] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1311] [Fourth embodiment]

[1312] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1313] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1314] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1315] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1316] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1317] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1318] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1319] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1320] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1321] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1322] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1323] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1324] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1325] The present invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model, and generates auto-complete input data for each application, realizing collaboration between applications.

[1326] Server Processing

[1327] 1. Application Discovery

[1328] The server communicates with the user's device to detect each application the user is using. This allows the server to obtain information about all applications running on the device. For example, if a user is using a virtual movie theater app and a chat app at the same time, that information is notified to the server.

[1329] 2. Data Collection

[1330] The server then collects user interaction data and context information from each detected application, including movie scene information from a virtual cinema app and user comment history from a chat app.

[1331] 3. Generative AI model in action

[1332] A generative AI model is run based on the collected data. For example, movie scene information and utterances from a chat app are used as input. The model analyzes and infers the data to generate topics and questions related to movie scenes.

[1333] 4. Input completion data generation

[1334] The server generates auto-complete input data for each application based on the analysis results of the generative AI model. For example, questions and topics related to movie scenes are generated for a chat app.

[1335] 5. Data Transmission

[1336] Finally, the server sends the generated auto-complete input data to each application, allowing the chat app to suggest new topics and questions related to movie scenes to the user.

[1337] Terminal handling

[1338] 1. Receiving notifications

[1339] The user's device receives the auto-complete input data sent from the server, which allows the new data to be reflected in real time.

[1340] 2. Data Application

[1341] The device applies the received data to each application. For example, it might display topics related to movie scenes in a chat app to encourage user interaction. It might also update the application's UI and notify the user of new input.

[1342] User Action

[1343] 1. App Usage

[1344] Users can use multiple xR applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[1345] 2. Complementary Experiences

[1346] Users experience auto-complete interactions with each application, for example, sharing topics with friends based on a question related to a movie scene displayed in a chat app, resulting in a more natural and rich user experience.

[1347] Specific examples

[1348] Example 1: Linking a virtual movie theater app with a chat app

[1349] Consider a case where a user is watching a movie on a virtual cinema app while chatting with a friend on a chat app. The server collects information about movie scenes and the content of comments made in the chat app. The generative AI model uses this data to generate questions and topics related to the movie scenes. The generated data is sent to the chat app, which receives it and presents it to the user. This allows the user to share topics based on movie scenes with their friends.

[1350] Example 2: Integration of an educational app and a note-taking app

[1351] Consider a case where a user is taking a class using an educational app while recording what they learned using a note-taking app. The server collects the class content data and the contents of the notes. The generative AI model uses this data to generate supplementary information to be added to the notes and hints for related problems. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, the user can easily record supplementary information related to the class.

[1352] Through these processes, the system of the present invention realizes information sharing and collaboration between multiple applications, improving the user experience.

[1353] The processing flow will be explained below.

[1354] Server Processing

[1355] Step 1:

[1356] Application Discovery

[1357] The server communicates with the user's device to discover all currently running applications, specifically by calling the device's API to retrieve the list of active applications.

[1358] Step 2:

[1359] Data collection

[1360] The server collects user interaction data and situation information from each detected application. Specifically, it obtains movie playback time and scene information from the movie theater app, and speech history from the chat app. This is also done through each application's API.

[1361] Step 3:

[1362] Generative AI model in action

[1363] The server inputs the collected data and runs a generative AI model. Specifically, it inputs movie scene information and speech history into the model, analyzes them, and predicts interactions.

[1364] Step 4:

[1365] Input completion data generation

[1366] Based on the analysis results of the generative AI model, auto-complete input data is generated for each application. For example, it generates chat topics related to movie scene information.

[1367] Step 5:

[1368] Data transmission

[1369] The server sends the generated auto-complete input data to each application, again via the API of each application.

[1370] Terminal handling

[1371] Step 1:

[1372] Receive notifications

[1373] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, it periodically checks communications from the server and acquires new data.

[1374] Step 2:

[1375] Data Application

[1376] The device applies the received data to the corresponding application, for example updating the UI to display new topics or questions in a chat app.

[1377] User Action

[1378] Step 1:

[1379] App use

[1380] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[1381] Step 2:

[1382] Complementary Experiences

[1383] Users experience auto-completed interactions, specifically continuing conversations with friends based on topics and questions related to movie scenes displayed in the chat app.

[1384] In this way, through the specific operations at each step, cooperation between the server, terminal, and user is realized, providing a richer user experience.

[1385] Example 1

[1386] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1387] Modern users often use multiple applications simultaneously and require enhanced user experiences through information sharing and collaboration between these applications. However, data sharing and auto-completion input between different applications is often performed manually, making it difficult to provide an efficient and natural user experience. For this reason, there is a demand for a system that automatically shares information and collaborates between multiple applications to improve user interaction.

[1388] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1389] In this invention, the server includes means for detecting multiple applications run by a user, means for collecting user interaction data and situation information from each detected application, means for running a generative AI model based on the collected data to analyze and predict interactions between applications, means for generating auto-complete input data for each application based on the analysis results, and means for transmitting the generated auto-complete input data to each application, thereby enabling automatic information sharing and collaboration between different applications to provide an efficient and natural user experience.

[1390] "User" refers to a person or entity that uses a system or application.

[1391] "Application" is a general term for a software program that a user runs on a terminal.

[1392] "Interaction Data" refers to data regarding inputs and interactions when a user operates an application.

[1393] "Status information" refers to information related to the user's usage environment and status, including, for example, the application being used and the status of its content.

[1394] A "generative AI model" refers to an artificial intelligence model that analyzes collected data and automatically generates input data.

[1395] "Auto-complete input data" refers to the input data generated by the generative AI model to be applied to each application.

[1396] "Server" refers to a computer system that communicates with a user's terminal, collects data, analyzes the data, and transmits the generated data.

[1397] "Terminal" refers to a device that a user uses to run an application, and examples include a computer and a smartphone.

[1398] "API" stands for Application Programming Interface and refers to a means by which different pieces of software communicate with each other.

[1399] An "HTTP POST request" is a part of the HTTP protocol used to send data to a server.

[1400] This invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model, and generates auto-complete input data for each application, realizing collaboration between applications.

[1401] This system mainly consists of three main components: a server, a terminal, and a user.

[1402] Server Roles

[1403] The server first communicates with the user's terminal and detects each application the user is running. For example, the netstat or ps command is used to obtain information about applications running on the terminal. This information is sent to the server using the UDP or TCP protocol. For example, the netstat -an | grep 'ESTABLISHED' command can be used to identify connected applications.

[1404] Next, the server collects user interaction data and context information from each detected application. The collected data is obtained through API calls and log data analysis. For example, movie scene information can be collected from a virtual movie theater app, and user comment history can be collected from a chat app.

[1405] Based on the collected data, the server runs a generative AI model. For example, GPT-4 can be used as the generative AI model. By preprocessing the data using Python and inputting it into the generative AI model, topics and questions related to the movie scenes can be generated using movie scene information and the content of comments made in the chat app as input.

[1406] Next, the server generates auto-complete input data based on the analysis results of the generative AI model. For example, if the generative AI model generates a "question about a movie scene," it converts it into JSON format so that the chat app can understand it. This completion data is sent to each application using an HTTP POST request. For example, you can POST the data to http: / / chatapp.example.com / api / v1 / messages to reflect the new question in the chat app.

[1407] Device Role

[1408] The user's device receives the auto-complete input data sent from the server. You can receive the data in real time using a listener in the device (for example, a WebSocket server). Create an onMessage event handler to process the data as soon as it arrives.

[1409] The device then applies the received data to each application, for example updating the UI of a chat app to show the new question, or if you're using React, updating the state and re-rendering the component to show the new question.

[1410] User Roles

[1411] Users use multiple applications simultaneously. For example, they may watch a movie on a virtual cinema app while chatting with friends on a chat app. As users interact with the applications, their activities are sent to a server and data is collected.

[1412] The collected data is then automatically filled in and applied to the device, allowing users to enjoy new experiences. For example, a user can share a topic with a friend based on a question related to a movie scene displayed in a chat app. This allows the user's conversation to progress more naturally, resulting in richer communication.

[1413] Specific examples

[1414] Example 1: Linking a virtual movie theater app with a chat app

[1415] When a user watches a movie in a virtual cinema app and chats with a friend in a chat app, the server collects information about the movie's scenes and the content of the chat app's comments. The generative AI model uses this data to generate questions and topics related to the movie's scenes. The generated data is sent to the chat app, which receives it and presents it to the user. In this way, the user can share topics based on the movie's scenes with their friends.

[1416] Examples of specific prompts include:

[1417] "What do you think about the characters in the scene you're watching?"

[1418] "Let's talk about the events behind this scene."

[1419] Example 2: Integration of an educational app and a note-taking app

[1420] When a user takes a class using an educational app and records their learning using a note-taking app, the server collects the class content data and the contents of the notes. The generative AI model uses this data to generate supplementary information to be added to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, users can easily record supplementary information related to the class.

[1421] Examples of specific prompts include:

[1422] "Please note down any important points related to this topic."

[1423] "Let's dig deeper into the ideas explained in class."

[1424] Through these processes, the system of the present invention can realize information sharing and collaboration between multiple applications, thereby improving the user experience.

[1425] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1426] Step 1:

[1427] The server detects multiple applications that a user runs.

[1428] Specifically, the server communicates with the user's terminal via TCP or UDP protocol and uses the netstat or ps command to collect information about running applications. The information obtained includes the application name, version, and status information indicating that it is running. Based on this information, it identifies which applications are currently running.

[1429] Input: Application execution status on user device

[1430] Output: Information about the running application (name, version, status, etc.)

[1431] Step 2:

[1432] The server collects user interaction data and context information from each detected application.

[1433] Specifically, the server uses the API of each application to obtain information such as what operations the user performed and what content is being played. For example, it collects information about the currently playing scene from a virtual movie theater app, and the user's comment history from a chat app. This information is collected by calling APIs and analyzing log files.

[1434] Input: Running application information

[1435] Output: Interaction data and context information for each application

[1436] Step 3:

[1437] The server runs a generative AI model based on the collected data.

[1438] Specifically, the server preprocesses the data using Python and inputs it into a generative AI model (e.g., GPT-4). The input data includes movie scene information and comments from a chat app. The generative AI model analyzes this data and generates appropriate complementary information (e.g., new questions or topics).

[1439] Input: Interaction data and context information

[1440] Output: Complementary information (e.g., questions and topics) generated by the generative AI model

[1441] Step 4:

[1442] The server generates input completion data based on the analysis results of the generative AI model.

[1443] Specifically, the server converts the data obtained from the generative AI model into an appropriate format, such as JSON, and formats it in a way that each application can understand. For example, it converts the generated question into a format like {"type": "question", "content": "Who are the characters in this scene?"}.

[1444] Input: Analysis results of the generative AI model

[1445] Output: Input completion data applicable to each application

[1446] Step 5:

[1447] The server transmits the generated auto-completion input data to each application.

[1448] Specifically, the server sends data using an HTTP POST request, for example to http: / / chatapp.example.com / api / v1 / messages, to reflect a new question in the chat app.

[1449] Input: Input completion data

[1450] Output: Data sent to each application

[1451] Step 6:

[1452] The user's terminal receives the auto-complete input data sent from the server.

[1453] Specifically, the device receives data in real time using a locally running listener (e.g., a WebSocket server). When the onMessage event handler receives the data, it executes the appropriate process.

[1454] Input: Data sent to each application

[1455] Output: Type-ahead data received on the device

[1456] Step 7:

[1457] The terminal applies the received data to each application and notifies the user.

[1458] Specifically, the device reflects the received data in the UI of the corresponding application and notifies the user of new input. For example, if you are using React, it will update the state and re-render the component displaying the new question.

[1459] Input: Received type-ahead data

[1460] Output: Updated application UI, notification to the user

[1461] Step 8:

[1462] The user uses the application and experiences the autocompleted interaction.

[1463] Specifically, users can engage in conversations with friends based on questions related to movie scenes presented in a chat app, resulting in a more natural and enriching user experience.

[1464] Input: Updated application UI

[1465] Output: Improved user interaction and experience

[1466] (Application example 1)

[1467] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1468] When users use multiple content distribution services on the Internet, each service operates independently, making it difficult to provide a unified information experience. It has also been pointed out that there is a lack of means to provide real-time updates or additional content related to the media content being viewed or listened to. This results in a fragmented experience, as users are unable to link information between content or receive seamless information.

[1469] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1470] In this invention, the server includes means for detecting multiple programs run by a user, means for collecting user interaction data and situation information from each detected program, means for running a generative AI model based on the collected data to analyze and predict interactions between programs, means for generating auto-complete input data for each program based on the analysis results, means for sending the generated auto-complete input data to each program, and means for automatically generating news and additional information related to the media content being viewed or listened to in cooperation with the content distribution service to which each program belongs. This enables information related to the media content being viewed or listened to to be provided in real time when a user is using multiple content distribution services, thereby realizing a seamless information experience.

[1471] A "program" is a set of instructions or code that runs on a computer to accomplish a particular task or function.

[1472] "Interaction data" is data generated when a user operates a program, and includes operation logs such as clicks, taps, inputs, and viewing time.

[1473] "Contextual information" is data that describes the context or conditions identified during program execution, including the user's geographic location, device state, current activity, etc.

[1474] A "generative AI model" is an artificial intelligence algorithm that analyzes patterns and relationships based on collected data and makes predictions and inferences. In this case, it is used specifically for natural language generation and data completion.

[1475] "Auto-complete input data" refers to suggested data or additional information generated by a generative AI model and used as input data for a specific program.

[1476] A "content distribution service" is a service that provides users with multimedia content such as videos, music, and news via the Internet.

[1477] "Related news and additional information" refers to information or content that is related to the media content that the user is watching or listening to and is provided in a manner that complements it.

[1478] MODE FOR CARRYING OUT THE INVENTION

[1479] This invention relates to a system that analyzes the interactions between multiple programs when a user is using them simultaneously, improving the user experience. The system uses a generative AI model to automatically generate related information based on interaction data and context information, and establishes collaboration between the programs.

[1480] Server Processing

[1481] The server has the following means:

[1482] 1. Program detection methods

[1483] It communicates with the user's device and detects the multiple programs the user is running. This method identifies all programs running on the device. For example, if a user is using a video streaming service and a news app at the same time, that information is notified to the server.

[1484] 2. Data Collection Methods

[1485] For each program detected, it collects user interaction data and contextual information, such as information about the video being watched, browsing history in news apps, etc.

[1486] 3. Means of running the generative AI model

[1487] Based on the collected data, a generative AI model is run to analyze the interactions between programs. The generative AI model analyzes the user's interaction data and generates related information based on it. For example, it generates additional information or topics related to the movie or news being watched.

[1488] 4. Auto-complete input data generation method

[1489] Based on the analysis, it generates auto-complete entries for each program, including news and additional information related to the content being viewed.

[1490] 5. Data Transmission Method

[1491] The generated auto-complete input data is sent to each program, which allows the news app to present new related information and news articles to the user.

[1492] Terminal handling

[1493] The terminal has the following means:

[1494] 1. Notification Reception Method

[1495] Receives auto-complete input data sent from the server, allowing the device to reflect new data in real time.

[1496] 2. Data application method

[1497] The received data is applied to each program and notified to the user, allowing the news app to display relevant news and additional information to the user in real time.

[1498] User Action

[1499] The user has the following means:

[1500] 1. How to use the program

[1501] Use multiple programs at the same time. For example, watch a movie on a streaming service while checking the latest news on a news app.

[1502] 2. Complementary Experiences

[1503] Experience auto-complete information in each program. For example, you can browse more interesting topics based on movie-related news displayed in a news app.

[1504] Specific examples

[1505] As a specific example, consider a case where while a user is watching a "documentary about climate change" on a video streaming service, the system of this invention automatically generates and displays news articles about "the latest research results on climate change" and "the latest trends in the environmental protection movement" in a news app. In this way, additional information that is closely related to the content the user is watching can be provided in real time.

[1506] Example prompt sentence:

[1507] Input: Viewing data for the video "Documentary on Climate Change"

[1508] Output: Related news articles (title, content)

[1509] This allows users to seamlessly experience related information and news while using content distribution services.

[1510] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1511] Step 1:

[1512] Program Detection

[1513] The server communicates with the user's terminal and detects all programs the user is running.

[1514] Input: A list of running programs from the user's terminal

[1515] Output: Detected program information

[1516] Specific operation: A monitoring agent installed on the user's terminal collects information about running programs and sends it to the server.

[1517] Step 2:

[1518] Data collection

[1519] The server collects user interaction data and context information from each detected program.

[1520] Input: Detected program information

[1521] Output: Interaction data and context information from each program

[1522] Specific operations: Collects data such as video playback time, content being viewed, and user operation logs.

[1523] Step 3:

[1524] Generative AI model in operation

[1525] The server runs a generative AI model based on the collected data and analyzes interactions between programs.

[1526] Input: Interaction data and context information

[1527] Output: Analysis results (related news and additional information)

[1528] Specific operation: The collected data is input into a generative AI model for analysis. The generative AI model used here is GPT-3.

[1529] Step 4:

[1530] Auto-complete input data generation

[1531] The server generates auto-complete input data for each program based on the analysis results of the generative AI model.

[1532] Input: Analysis results

[1533] Output: Auto-complete input data

[1534] Specific operation: Based on the generated data, templates for news articles and additional information are created and their content is automatically generated.

[1535] Step 5:

[1536] Data transmission

[1537] The server transmits the generated auto-completion input data to each program.

[1538] Input:Auto-complete input data

[1539] Output: Data to send to each program

[1540] Specific operation: Encode and send the generated data in an appropriate format to the corresponding program in real time.

[1541] Step 6:

[1542] Receive notifications

[1543] The terminal receives the auto-completion input data sent from the server.

[1544] Input: Data to be sent from the server

[1545] Output: Received complementary data

[1546] Specific operation: The program on the terminal receives data from the server and stores the contents in temporary storage.

[1547] Step 7:

[1548] Data Application

[1549] The terminal applies the received data to each program and notifies the user.

[1550] Input: Received complementary data

[1551] Output: Update information for each program

[1552] Specific behavior: Update the UI of each program to apply the received data and notify the user that new information is available.

[1553] Step 8:

[1554] Program Use

[1555] A user may use multiple programs simultaneously.

[1556] Input: Related news and additional information

[1557] Output: User Interaction

[1558] Specific operations: A user watches a movie on a video streaming service while checking related information on a news app.

[1559] Step 9:

[1560] Complementary Experiences

[1561] Users experience auto-completed information in each program.

[1562] Input: User interaction

[1563] Output: Improving the user experience

[1564] Specific behavior: By reading news articles related to a movie displayed in a news app, you can deepen your understanding of the movie's content.

[1565] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1566] This invention relates to a system that analyzes interactions between multiple applications when a user is using them simultaneously, improving the user experience. This system collects user interaction data and situational information, analyzes them using a generative AI model and an emotion engine, and generates auto-complete input data for each application, realizing collaboration between applications.

[1567] Server Processing

[1568] 1. Application Discovery

[1569] The server communicates with the user's device to detect all currently running applications. Specifically, it calls the device's API to obtain a list of active applications. For example, if a user is using a virtual movie theater app and a chat app at the same time, that information is notified to the server.

[1570] 2. Data Collection

[1571] The server then collects user interaction data and context information from each detected application, such as movie scene information from a virtual cinema app or user comment history from a chat app, and provides the collected data to a generative AI model and emotion engine.

[1572] 3. Generative AI model and emotion engine in action

[1573] The server uses the collected data as input to run a generative AI model and an emotion engine. The generative AI model analyzes movie scene information and chat app comment history to infer interactions. Meanwhile, the emotion engine analyzes the user's emotions from the user's interaction data and situational information. For example, the emotion engine infers the user's emotional state from the user's tone of voice and text content.

[1574] 4. Input completion data generation

[1575] Based on the analysis results of the generative AI model and emotion engine, auto-complete input data for each application is generated. For example, data containing appropriate tone and topics according to the user's emotional state is generated, in addition to casual conversation topics related to movie scene information.

[1576] 5. Data Transmission

[1577] Finally, the server sends the generated auto-complete input data to each application, allowing the chat app to present new topics and questions to the user that are adapted to the movie scenes and the user's emotions.

[1578] Terminal handling

[1579] 1. Receiving notifications

[1580] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, it periodically checks communications from the server and acquires new data.

[1581] 2. Data Application

[1582] The device applies the received data to the corresponding application, for example, displaying a new topic or question in a chat app, or updating the application's UI to notify the user of the new input.

[1583] User Action

[1584] 1. App Usage

[1585] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[1586] 2. Complementary Experiences

[1587] Users experience auto-complete interactions, such as continuing a conversation with a friend based on a movie scene displayed in a chat app or topics and questions related to the user's emotions, resulting in a more natural and rich user experience.

[1588] Specific examples

[1589] Example 1: Linking a virtual movie theater app with a chat app

[1590] Consider a case where a user is watching a movie on a virtual cinema app while chatting with a friend on a chat app. The server collects information about movie scenes, comments made on the chat app, and the user's emotional data analyzed by the emotion engine. Based on this data, the generative AI model and emotion engine generate questions and topics related to the movie scenes and adapted to the user's emotional state. The generated data is sent to the chat app, which receives it on the device and presents it to the user. In this way, the user can share topics based on the movie scenes and their own emotions with their friends.

[1591] Example 2: Integration of an educational app and a note-taking app

[1592] Consider a case where a user takes a class using an educational app while recording their learning using a note-taking app. The server collects the class content data, the note content, and the user's emotional data analyzed by the emotion engine. Based on this data, the generative AI model and emotion engine generate supplementary information to add to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. In this way, users can not only easily record supplementary information related to the class, but also receive optimal learning experiences tailored to their emotional state.

[1593] Through these processes, the system of the present invention enables information sharing and collaboration between multiple applications, not only improving the user experience but also providing optimal interactions according to the user's emotional state.

[1594] The processing flow will be explained below.

[1595] Server Processing

[1596] Step 1:

[1597] Application Discovery

[1598] The server communicates with the user's device to discover all currently running applications, specifically by calling the device's API to retrieve the list of active applications.

[1599] Step 2:

[1600] Data collection

[1601] The server collects user interaction data and context information from each detected application. Specifically, it obtains movie playback time and scene information from the virtual cinema app, and speech history from the chat app. It also collects the user's voice tone and text content as input data for the emotion engine.

[1602] Step 3:

[1603] Generative AI model and emotion engine in action

[1604] The server uses the collected data as input to run a generative AI model and an emotion engine. The generative AI model analyzes movie scene information and chat app comment history to predict interactions. The emotion engine analyzes the user's voice tone and text content to evaluate the user's emotional state.

[1605] Step 4:

[1606] Applying Emotion Data

[1607] Based on the analysis results of the emotion engine, the generative AI model adjusts the auto-complete input data for each application. For example, if the user is excited, it generates input data that is appropriate for that excited state.

[1608] Step 5:

[1609] Input completion data generation

[1610] Based on the analysis results of the generative AI model and emotion engine, auto-complete input data for each application is generated. For example, data is generated that includes not only casual conversation topics related to movie scene information, but also tone and topics that correspond to the user's emotional state.

[1611] Step 6:

[1612] Data transmission

[1613] The server sends the generated auto-completion input data to each application via the application's API, where it is applied by the corresponding application.

[1614] Terminal handling

[1615] Step 1:

[1616] Receive notifications

[1617] The device monitors and receives auto-complete input data sent from the server in real time, and periodically checks for communication from the server to see if there is new data.

[1618] Step 2:

[1619] Data Application

[1620] The device applies the received data to the corresponding application, for example, updating the UI to display new topics or questions in a chat app. Based on the emotion data, the device presents content tailored to the user.

[1621] User Action

[1622] Step 1:

[1623] App use

[1624] Users can use multiple applications simultaneously, for example, watching a movie in a virtual cinema app while chatting with friends in a chat app.

[1625] Step 2:

[1626] Complementary Experiences

[1627] Users experience auto-complete interactions, such as continuing a conversation with a friend based on a movie scene displayed in a chat app or topics and questions related to the user's emotions, providing a more natural and enriching experience.

[1628] Specific examples

[1629] Example 1: Linking a virtual movie theater app with a chat app

[1630] Step 1:

[1631] The server detects the use of the virtual cinema app and the chat app.

[1632] Step 2:

[1633] The server collects movie playback scene information, chat app comment history, and the user's voice tone.

[1634] Step 3:

[1635] The server uses a generative AI model to analyze movie scene information and speech history, and uses an emotion engine to evaluate the user's emotional state.

[1636] Step 4:

[1637] Based on the results of the emotion engine, the generative AI model adjusts chat topics to suit the user's emotional state.

[1638] Step 5:

[1639] The server generates input data adjusted based on the emotional state and sends it to the chat app.

[1640] Step 6:

[1641] The device receives the transmitted data, reflects it in the chat app, and presents new topics to the user.

[1642] Example 2: Integration of an educational app and a note-taking app

[1643] Step 1:

[1644] The server detects the use of educational apps and note-taking apps.

[1645] Step 2:

[1646] The server collects lesson content data, note content, and user text content.

[1647] Step 3:

[1648] The server uses a generative AI model to analyze the lesson content and an emotion engine to evaluate the user's emotional state.

[1649] Step 4:

[1650] Based on the results of the emotion engine, a generative AI model tailors supplemental information and hints that correspond to the user's emotional state.

[1651] Step 5:

[1652] The server generates complementary data adjusted based on the emotional state and sends it to the note-taking app.

[1653] Step 6:

[1654] The device receives the transmitted data, reflects it in the note application, and adds supplementary information to the user's notes.

[1655] Through these processes, the system of the present invention can realize information sharing and collaboration among multiple applications, and can also provide optimal interactions according to the user's emotional state.

[1656] Example 2

[1657] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1658] In conventional systems, when a user operates multiple programs simultaneously, there is a problem of a fragmented user experience due to a lack of information sharing or collaboration between the programs. Therefore, there is a need for a system that allows users to smoothly exchange information between different programs. There is also a need to provide a more natural and rich experience by realizing interactions that take the user's emotional state into consideration.

[1659] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1660] In this invention, the server includes means for detecting multiple programs run by a user, means for collecting user operation data and situation information from each detected program, means for running a generative AI model and an emotion analysis engine based on the collected data to analyze and predict interactions between programs, means for generating auto-complete input data for each program based on the analysis results, and means for transmitting the generated auto-complete input data to each program. This enables information sharing and collaboration between different programs, and can provide optimal interactions to the user according to their emotional state.

[1661] "Program" refers to multiple applications and software that can be operated by a user.

[1662] "Operation data" refers to information about a series of operations performed by a user within a program.

[1663] "Status Information" refers to information relating to the state of a program and its contents.

[1664] "Generative AI model" refers to an artificial intelligence model that analyzes and predicts interactions between programs based on collected interaction data and situational information.

[1665] "Emotion analysis engine" refers to an engine for analyzing a user's emotional state from the user's interaction data and context information.

[1666] "Auto-completion input data" refers to data generated to complement a user's operation.

[1667] "Information sharing" refers to the exchange and sharing of information between different programs.

[1668] "Integration" refers to multiple programs working together to provide a unified experience.

[1669] An "interaction" refers to a series of exchanges or operations between a user and a program.

[1670] "Analysis Results" refers to the analysis results of data obtained by the generative AI model and sentiment analysis engine.

[1671] "User experience" refers to the overall impression and satisfaction a user feels when operating each program.

[1672] This invention relates to a system that analyzes the interactions between multiple programs when a user is using them simultaneously, improving the user experience. This system collects user operation data and situational information, analyzes them using a generative AI model and an emotion analysis engine, and generates auto-complete input data for each program, realizing collaboration between programs.

[1673] Specifically, the server operates as follows.

[1674] The server first communicates with the user's device to detect all programs currently running. A specific example of this operation is to call the device's API (e.g., ActivityManager in Android or UIApplication in iOS) to obtain a list of active programs. The information about the programs detected in this way is then notified to the server.

[1675] Next, the server collects user operation data and context information from each detected program. This information is collected using the API of each program. For example, a virtual movie theater app might collect information about the scene currently playing in a movie, while a chat app might collect the user's comment history. This information is then provided to the generative AI model and sentiment analysis engine.

[1676] The server uses the collected data as input to run a generative AI model and an emotion analysis engine. The generative AI model, for example, analyzes movie scene information and chat app comment history to infer the interactions between them. Meanwhile, the emotion analysis engine infers the user's emotional state from the tone of their voice and the content of their chat messages.

[1677] Based on the analysis results of the generative AI model and the emotion analysis engine, the server generates auto-complete input data for each program. As a specific example, data is generated that includes not only chat topics related to movie scene information, but also appropriate tones and topics according to the user's emotional state. For example, appropriate questions and comments related to a moving movie scene are generated and sent to a chat app.

[1678] The auto-complete input data sent from the server is received by the device. The device periodically checks for communication from the server and obtains new data. This received data is applied to the corresponding program, for example, a new topic or question is displayed in a chat app. The device uses this data to update the program's UI and notify the user that new input has been received. Specifically, this is done by updating the RecyclerView on Android, or by using the refresh function of UITableView on iOS.

[1679] Users can enjoy a more natural and richer experience with this updated program. For example, when watching a movie in a virtual cinema app and chatting with friends in a chat app, they can enjoy auto-complete interactions. Topics and questions related to the movie scene and their own emotions are displayed, making the conversation with friends more natural and smooth.

[1680] Below are some examples of specific prompt sentences.

[1681] Example 1: Linking a virtual movie theater app with a chat app

[1682] When a user watches a movie in a virtual cinema app and chats with a friend in a chat app, the server collects information about the movie's scenes and the chat app's comments and emotional data. The generative AI model and emotion analysis engine use this data to generate questions and topics related to the movie's scenes and adapted to the user's emotional state. The generated data is sent to the chat app, which receives it and presents it to the user. This process allows the user to share topics based on the movie's scenes and their emotions with their friends.

[1683] Example 2: Integration of an educational app and a note-taking app

[1684] When a user takes a class using an educational app and records their learning using a note-taking app, the server collects the class content data, the note content, and emotional data. Based on this data, the generative AI model and emotional analysis engine generate supplementary information to add to the notes and hints for related questions. The generated data is sent to the note-taking app, and the device receives it and reflects it in the user's notes. This process allows users to easily record supplementary information related to the class and provides optimal learning based on their emotional state.

[1685] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1686] Step 1:

[1687] The server communicates with the user's device and detects all currently running programs. Specifically, the server calls the device's API (e.g., ActivityManager on Android or UIApplication on iOS) to obtain a list of active programs. The input data is information about the programs running on the device, and the output is a list of running programs.

[1688] Step 2:

[1689] The server collects user operation data and status information from each detected program. This is done through each program's API. For example, a virtual movie theater app collects information about the scene currently playing in a movie, and a chat app collects the user's comment history. The input data is the API call results for each program, and the output is the operation data and status information collected from each program.

[1690] Step 3:

[1691] The server provides the collected operation data and situation information as input to the generative AI model and the emotion analysis engine. In this step, the generative AI model analyzes the movie scene information and the chat app's comment history to infer the interactions between them. Meanwhile, the emotion analysis engine analyzes the user's emotional state from the user's tone of voice and the content of the chat messages. The input is the collected data, and the output is the analysis results and the emotional state.

[1692] Step 4:

[1693] The server generates auto-completion input data for each program based on the analysis results of the generative AI model and the emotion analysis engine. This includes, for example, automatically generating topics and questions related to moving movie scenes for a chat app. The input is the analysis results and emotional state, and the output is auto-completion input data.

[1694] Step 5:

[1695] The server sends the generated auto-completion input data to each program. Specifically, it sends the data using HTTP requests, WebSockets, etc. The input is the auto-completion input data, and the output is the data received by the corresponding program.

[1696] Step 6:

[1697] The terminal receives the auto-completion input data sent from the server. It monitors the communication from the server in real time and acquires new data. The input is the communication data from the server, and the output is the received auto-completion input data.

[1698] Step 7:

[1699] The device applies the received auto-completion input data to the corresponding program, for example, displaying a new topic or question in a chat app and updating the application UI to notify the user of the new input. The input is the received auto-completion input data, and the output is the updated UI.

[1700] Step 8:

[1701] Users use multiple programs simultaneously and experience auto-completed interactions. For example, they can watch a movie in a virtual cinema app while chatting with friends in a chat app, continuing the conversation based on newly generated topics and questions. The input is an updated UI, and the output is the user's new interaction experience.

[1702] (Application example 2)

[1703] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1704] Conventional systems provided simple navigation and entertainment information without considering the user's emotions or surrounding environmental information while driving, which did not sufficiently improve the user experience.In addition, there were few ways to accurately analyze and coordinate the interactions between multiple applications used simultaneously by the user, making it difficult to reduce stress and inconvenience while driving.

[1705] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1706] In this invention, the server includes means for detecting multiple applications run by a user, means for collecting user interaction data and situation information from each detected application, means for running a generative AI model based on the collected data to analyze and predict interactions between applications, means for generating auto-complete input data for each application based on the analysis results, means for sending the generated auto-complete input data to each application, and means for generating navigation information and entertainment information that takes into account the emotional state and surrounding environmental information of the means of transportation driven by the user based on the analyzed interaction data and situation information. This makes it possible to provide optimal navigation information and entertainment information in real time according to the user's emotions and surrounding environment, reducing stress and inconvenience while driving and improving the user experience.

[1707] The "means for detecting multiple applications run by a user" refers to a method and apparatus for identifying and listing all software and programs currently running on a user's terminal.

[1708] "Means for collecting user interaction data and situation information" refers to methods and devices for acquiring and accumulating data about the operations performed by users in each application and data about the surrounding environment.

[1709] "Means for running a generative AI model based on collected data to analyze and infer interactions between applications" refers to a method and device that uses acquired user operation history and environmental data as input, analyzes the relationships and interactions between applications using AI technology, and infers the results.

[1710] "Means for generating auto-complete input data for each application based on the analysis results" refers to a method and device for automatically creating appropriate data and information for each application based on the analysis results of the AI ​​model.

[1711] The "means for sending the generated auto-completion input data to each application" refers to a method and apparatus for transferring the generated data to the corresponding application.

[1712] "Means for generating navigation information and entertainment information that takes into account the emotional state and surrounding environmental information of the means of transportation being driven by the user based on analyzed interaction data and situational information" refers to a method and device for analyzing emotions and surrounding conditions from the user's operation history and environmental data, and providing useful route guidance and entertainment content while driving based on that.

[1713] As an embodiment of the present invention, a user assistance system for an autonomous driving vehicle will be taken as an example.

[1714] In this system, the server connects to the user's smartphone to detect multiple applications and collects interaction data and context information from each application. Based on the collected data, it runs a generative AI model to analyze the interactions between applications and generate the necessary auto-complete input data. This auto-complete input data is then sent to the corresponding application. Specifically, the system includes the following steps:

[1715] The server uses the device's API to discover all applications the user is running, which results in a list of currently active applications.

[1716] The server then collects data from each detected application about the user's operation history and surrounding environment, including GPS data while driving, the vehicle's current location, speed, surrounding traffic and weather information, and voice inputs. This data is then fed into a generative AI model to analyze the user's current situation and emotional state.

[1717] The generative AI model uses this data to analyze interactions between applications and generate optimal navigation and entertainment information to improve the user's driving experience. For example, if the user is feeling stressed, it can provide relaxing music or smooth route guidance based on that emotion.

[1718] The server then sends the generated navigation and entertainment information to each application. The device applies the received data to the corresponding application and notifies the user in real time. For example, a prompt such as "Please provide optimal driving advice and entertainment based on the current traffic situation and emotional state" could be used.

[1719] This system uses EmotionEngine to analyze the user's emotions and NavigationSystem (e.g., Google Maps API) to provide the optimal navigation route.It also improves the user's driving experience by generating entertainment information based on the collected data according to the user's driving situation and emotional state.

[1720] For example, if a user is stuck in traffic while driving an autonomous vehicle, the system uses the Emotion Engine to detect that the user is frustrated. The navigation system takes into account the current traffic situation and suggests an alternative route. The system also notifies the user, saying, "You are feeling stressed by the current traffic jam, so we will play relaxing music and suggest the optimal route to avoid it." In this way, the system can reduce stress and inconvenience while the user is driving, providing a more comfortable driving experience.

[1721] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1722] Step 1:

[1723] The server communicates with the user's device and uses an API to discover all currently running applications. Specifically, it calls the device's API to retrieve the list of active applications. The device's status information is given as input, and the list of active applications is obtained as output.

[1724] Step 2:

[1725] The server collects user interaction data and context information from each detected application, including GPS data, voice input, the vehicle's current location, speed, and environmental data such as traffic and weather. The input is the list of active applications, and the output is the collected interaction data and context information.

[1726] Step 3:

[1727] The server inputs the collected data into a generative AI model to analyze and predict interactions between applications. The generative AI model analyzes the user's operation history and environmental data to analyze the relationships and interactions between applications. The collected data is given as input, and the analysis and prediction of interactions between applications is obtained as output.

[1728] Step 4:

[1729] The server generates auto-completion input data for each application based on the analysis results of the generative AI model. For example, it creates navigation information or entertainment information taking into account the user's emotional state and surrounding environment. The analysis results are given as input, and the auto-completion input data is generated as output.

[1730] Step 5:

[1731] The server sends the generated auto-completion input data to each application. Each application receives and applies this data. The auto-completion input data is given as input, and the corresponding application is updated as output.

[1732] Step 6:

[1733] The device monitors and receives auto-complete input data sent from the server in real time. Specifically, the device periodically checks for communication from the server and acquires new data. Data from the server is given as input, and the received data is saved on the device as output.

[1734] Step 7:

[1735] The device applies the received data to the corresponding application and notifies the user, for example, displaying new route directions in a navigation application or instructing a music application to play relaxing music. The received data is given as input, and the application's UI is updated as output.

[1736] Step 8:

[1737] The user experiences complementary interactions, for example, receiving new route directions from a navigation app while driving and playing relaxing music from a music app, with application notifications as input and an improved driving experience as output.

[1738] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1739] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1740] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1741] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1742] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1743] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1744] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1745] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1746] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1747] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1748] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1749] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1750] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1751] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1752] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1753] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1754] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1755] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1756] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1757] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1758] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1759] The following is further disclosed regarding the above embodiment.

[1760] (Claim 1)

[1761] means for detecting multiple applications executed by a user;

[1762] means for collecting user interaction data and contextual information from each detected application;

[1763] A means to run generative AI models based on collected data to analyze and infer interactions between applications;

[1764] A means for generating auto-completion input data for each application based on the analysis results;

[1765] A means for transmitting the generated auto-complete input data to each application;

[1766] A system including:

[1767] (Claim 2)

[1768] means for applying the received auto-complete input data to a corresponding application and notifying a user;

[1769] 10. The system of claim 1, further comprising:

[1770] (Claim 3)

[1771] The generative AI model has the means to perform analysis to enable information sharing and collaboration between different applications used by the user.

[1772] 10. The system of claim 1.

[1773] "Example 1"

[1774] (Claim 1)

[1775] means for detecting multiple applications executed by a user;

[1776] means for collecting user interaction data and contextual information from each detected application;

[1777] A means to run generative AI models based on collected data to analyze and infer interactions between applications;

[1778] A means for generating auto-completion input data for each application based on the analysis results;

[1779] A means for transmitting the generated auto-complete input data to each application;

[1780] A system including:

[1781] (Claim 2)

[1782] means for applying the received auto-complete input data to a corresponding application and notifying a user;

[1783] A way to update the application's UI to present the new input to the user;

[1784] 10. The system of claim 1, further comprising:

[1785] (Claim 3)

[1786] The generative AI model has the means to perform analysis to enable information sharing and collaboration between different applications used by the user.

[1787] A means for acquiring user operation information using the API of each application;

[1788] further comprising means for transmitting the data using an HTTP POST request;

[1789] 10. The system of claim 1.

[1790] "Application Example 1"

[1791] (Claim 1)

[1792] means for detecting multiple programs executed by a user;

[1793] means for collecting user interaction data and contextual information from each detected program;

[1794] A means to run a generative AI model based on the collected data to analyze and infer interactions between programs;

[1795] A means for generating auto-completion input data for each program based on the analysis results;

[1796] A means for transmitting the generated auto-complete input data to each program;

[1797] It works in conjunction with the content distribution service to which each program belongs, and automatically generates news and additional information related to the media content being viewed or listened to.

[1798] A system including:

[1799] (Claim 2)

[1800] means for applying the received auto-complete input data to a corresponding program and notifying the user;

[1801] 10. The system of claim 1.

[1802] (Claim 3)

[1803] The generative AI model has the means to perform analysis to enable information sharing and collaboration between different programs used by the user.

[1804] 10. The system of claim 1.

[1805] "Example 2: Combining Emotion Engines"

[1806] (Claim 1)

[1807] means for detecting multiple programs executed by a user;

[1808] means for collecting user operation data and situation information from each detected program;

[1809] A means to analyze and infer interactions between programs by running a generative AI model and an emotion analysis engine based on the collected data;

[1810] A means for generating auto-completion input data for each program based on the analysis results;

[1811] A means for transmitting the generated auto-complete input data to each program;

[1812] A system including:

[1813] (Claim 2)

[1814] means for applying the received auto-complete input data to a corresponding program and notifying the user;

[1815] 10. The system of claim 1, further comprising:

[1816] (Claim 3)

[1817] The generative AI model and sentiment analysis engine have the means to perform analysis to enable information sharing and collaboration between different programs used by the user.

[1818] 10. The system of claim 1.

[1819] "Application example 2 when combining emotion engines"

[1820] (Claim 1)

[1821] means for detecting multiple applications executed by a user;

[1822] means for collecting user interaction data and contextual information from each detected application;

[1823] A means to run generative AI models based on collected data to analyze and infer interactions between applications;

[1824] A means for generating auto-completion input data for each application based on the analysis results;

[1825] A means for transmitting the generated auto-complete input data to each application;

[1826] A means for generating navigation information and entertainment information based on the analyzed interaction data and situation information, taking into consideration the emotional state and surrounding environment information of the means of transportation driven by the user;

[1827] A system including:

[1828] (Claim 2)

[1829] means for applying the received auto-complete input data to a corresponding application and notifying a user;

[1830] 10. The system of claim 1, further comprising:

[1831] (Claim 3)

[1832] The generative AI model performs analysis to enable information sharing and collaboration between different applications used by the user, and has a means to generate prompts including navigation and entertainment information that take into account the user's emotional state and surrounding environmental information.

[1833] 10. The system of claim 1. [Explanation of symbols]

[1834] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for detecting multiple applications executed by a user; means for collecting user interaction data and contextual information from each detected application; A means to run generative AI models based on collected data to analyze and infer interactions between applications; A means for generating auto-completion input data for each application based on the analysis results; A means for transmitting the generated auto-complete input data to each application; A system including:

2. means for applying the received auto-complete input data to a corresponding application and notifying a user; The system of claim 1 further comprising:

3. The generative AI model has the means to perform analysis to enable information sharing and collaboration between different applications used by the user. The system of claim 1 .

Images