System

Abstract

A system is provided.SOLUTION: A system including means for registering various services in a plug-in form, means for installing the registered plug-in in a terminal, means for collecting operation data of a user, means for storing the collected data, means for fine-tuning a generative artificial intelligence model using the stored data, means for delivering the fine-tuned artificial intelligence model to the terminal, means for providing a service based on the fine-tuned artificial intelligence model to the user, and means for collecting feedback from the user and improving the service.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, users utilize a variety of digital services, but each service is independent, making it difficult to effectively utilize users' thoughts and behavioral patterns. This results in a large amount of information being lost over time, and individual originality and thinking are not adequately preserved. Furthermore, it is difficult to achieve a consistent user experience across multiple services, resulting in a decline in usability. To solve these issues, a new system is needed that seamlessly connects various services and utilizes generative artificial intelligence (AI) to preserve and utilize individual users' thoughts and behaviors. [Means for solving the problem]

[0005] The present invention provides a system including means for registering various services in the form of plug-ins, means for installing the registered plug-ins on a terminal, means for collecting user operation data, means for saving the collected data, means for fine-tuning a generative AI model using the saved data, means for delivering the fine-tuned AI model to the terminal, means for providing a service based on the fine-tuned AI model to the user, and means for collecting feedback from the user and improving the service.The system of the present invention comprehensively manages operation data such as calls, messages, and notes made by users, and uses the collected data to fine-tune the AI ​​model, making it possible to provide a service that reflects the thoughts and actions of individual users.Furthermore, users can seamlessly use multiple services on the same platform, achieving an efficient operation experience.

[0006] A "plugin" is a type of extension module that is added to a major system or application, and is an independent program that implements specific functions.

[0007] A "terminal" is a device that can be directly operated by a user, including, but not limited to, a smartphone, tablet, or PC.

[0008] "User" refers to an individual or corporation that uses a system or service, and is also called a user.

[0009] "Operation data" refers to data that records the history and content of various operations performed by a user, and includes call content, message content, memo content, and the like.

[0010] A "generative artificial intelligence (AI) model" is a form of artificial intelligence that has the ability to learn patterns from data, generate new data, and make inferences.

[0011] "Fine-tuning" is the process of making fine adjustments to an existing artificial intelligence model based on specific objectives and data, thereby improving the model's performance.

[0012] "Distribution to terminal" refers to the operation of transmitting data or programs from a server to a terminal, making the data or programs available on the terminal side.

[0013] A "service provision means" is a system or process for providing specific functions or content to users.

[0014] "Feedback" refers to information such as evaluations, opinions, and requests for improvement obtained from users, and is used to improve and adjust services. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system for registering various services in a plug-in format and managing and providing them in an integrated manner. Specific embodiments for carrying out the present invention and each processing step are described below in natural language.

[0037] Registering a service plugin

[0038] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, and notes, and provides a mechanism for incorporating these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[0039] Plugin installation and user authentication

[0040] The device installs the plug-in downloaded from the server, enabling the use of various services. The user accesses the plug-in service through the device and enters authentication information for authentication. If authentication is successful, the user can access all registered services.

[0041] Data collection and storage

[0042] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. This database stores the operation history for each user.

[0043] Data preprocessing and cleansing

[0044] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and shaped for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[0045] Fine-tuning generative artificial intelligence

[0046] The server uses the preprocessed data to fine-tune the generative artificial intelligence (AI) model. The saved operation data is used as training data for the AI ​​model, improving its performance. Once fine-tuned, the AI ​​model is sent to the device, where it is updated.

[0047] To provide services and improve user experience

[0048] The device uses a fine-tuned AI model to provide individually optimized services to users. When users compose messages, the AI ​​model provides assistance and suggestions based on the user's past usage history. This streamlines user operations and provides a seamless user experience.

[0049] Feedback and Service Improvement

[0050] The server receives and collects feedback from users. Users can post to the server any problems or requests for improvements they have experienced while using the service. The server analyzes the collected feedback and identifies areas for improvement in the service. The service is updated and improved, and a more advanced service is provided to users.

[0051] Specific examples

[0052] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to the server. The server uses this data to fine-tune the generative AI and learns the user's unique writing style and expressions. The updated model is then used by the AI ​​to make predictions and completions as the user composes messages. This improves the user's operational efficiency.

[0053] The above is a specific embodiment for carrying out the present invention. The present invention provides users with a more personalized and seamless service experience, thereby improving usability.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] The server exposes an API for registering various services in the form of plugins. Once services such as calls, messages, and notes are registered as plugins, the server integrates them into the system.

[0057] Step 2:

[0058] The device will be able to download plugins from the server, and if a user wants to use a particular service, they will be required to install the corresponding plugin on their device.

[0059] Step 3:

[0060] Users access various services through plug-ins installed on their devices. When accessing the services, the user enters their authentication information and is authenticated by the server.

[0061] Step 4:

[0062] The device collects real-time data on user operations such as calls, messages, and notes, and stores this data as logs in the background.

[0063] Step 5:

[0064] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and pre-processed as necessary.

[0065] Step 6:

[0066] The server uses the stored data to fine-tune a generative artificial intelligence (AI) model, allowing the model to learn user interaction patterns and behavioral characteristics.

[0067] Step 7:

[0068] The server then distributes the fine-tuned artificial intelligence model to the device, which updates the AI ​​model on the device.

[0069] Step 8:

[0070] The device will utilize fine-tuned AI models to provide individually optimized services for users, such as in a note-taking app where AI predicts what you're typing and auto-completes the text, improving operational efficiency.

[0071] Step 9:

[0072] Users can provide feedback to the server about any problems or requests for improvements they may have while using the system. This feedback can be easily sent via the terminal.

[0073] Step 10:

[0074] The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements are determined, the server updates the plugins and AI models and distributes the new versions to devices.

[0075] This is the specific program processing flow, which allows users to enjoy a consistent and seamless service experience.

[0076] Example 1

[0077] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0078] In conventional systems, plugins for providing various services are managed separately, making it difficult to provide integrated services. Furthermore, to effectively utilize user operation data and provide individually optimized services, data collection, storage, preprocessing, analysis, and fine-tuning of generative AI models are required, but these processes are often not consistently integrated. This results in the challenge of making it difficult to improve the quality of the user experience.

[0079] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0080] In this invention, the server includes a means for allowing developers to register various services in the form of plug-ins, a means for registering plug-ins using an API provided by the server, and a means for installing the registered plug-ins on a terminal. This enables integrated management and provision of various services. The terminal also includes a means for entering user authentication information to access services, a means for collecting user operation data in real time, and a means for storing the collected operation data. The server further includes a means for preprocessing the stored data and removing noise and unnecessary data, a means for fine-tuning a generative AI model using the preprocessed data, a means for delivering the fine-tuned AI model to the terminal, a means for optimizing and providing services to users based on the fine-tuned AI model, and a means for collecting and analyzing user feedback to improve the service. This enables a personalized and seamless service experience based on user operation data, improving usability.

[0081] A "plug-in format" is a small program module that adds functionality to software.

[0082] "API" stands for Application Programming Interface, a means by which different software applications can communicate with each other.

[0083] A "terminal" is a hardware device (e.g., a smartphone or computer) that a user operates to access various services.

[0084] "Authentication information" refers to information such as ID and password used by users when accessing a system.

[0085] "Operation data" refers to data generated when a user operates the system (for example, call logs, message history, memo records, etc.).

[0086] "Real-time" means that data is processed immediately at the moment an action is taken.

[0087] "Preprocessing" refers to cleansing and shaping data before analyzing it.

[0088] "Noise" refers to unnecessary or meaningless data in data analysis.

[0089] A "generative artificial intelligence model" is an artificial intelligence algorithm that generates new data based on past data.

[0090] "Fine-tuning" is the process of further optimizing an existing artificial intelligence model using a specific dataset.

[0091] "Feedback" refers to opinions and information provided by users when they report improvements or problems with the system.

[0092] "Usability" is a concept that refers to how easy a system or service is to use.

[0093] This invention is a system that registers various services in a plug-in format and manages and provides them in an integrated manner. In this system, the server, terminal, and user each play different roles, realizing seamless and personalized services.

[0094] Registering a service plugin

[0095] The server provides an interface that allows developers to register various services in the form of plugins. This interface is implemented using the API provided by the server. Developers can register services such as calls, messages, and memos as plugins on the server. For example, if a developer wants to provide a new memo function as a plugin, they send this plugin to the server via the API. The server receives it and incorporates it into the system by storing it in a database.

[0096] Plugin installation and user authentication

[0097] The device installs the plug-in downloaded from the server. The user accesses the plug-in service through the device and authenticates by entering their authentication information (e.g., ID and password). If authentication is successful, the user will be able to access all registered services. For example, if a user installs a message service plug-in and authenticates, the message service will become available on that device.

[0098] Data collection and storage

[0099] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background and sent to the server periodically or in real time. The server receives the operational data sent from the device and stores it in a secure database. The database stores the operation history for each user.

[0100] Data preprocessing and cleansing

[0101] The server preprocesses the collected data to remove noise and unnecessary data. For example, meaningless spaces and duplicate data are removed from message content. The preprocessed data is then classified and formatted for analysis, and different types of data, such as call logs and message logs, are organized chronologically.

[0102] Fine-tuning generative AI models

[0103] The server uses the preprocessed data to fine-tune the generative AI model. The user's past operational data is used as training data for the AI ​​model. Through this process, the model learns the user's unique behavioral patterns and expression methods, improving its performance. Once fine-tuned, the artificial intelligence model is delivered to the device, updating the device's AI engine.

[0104] To provide services and improve user experience

[0105] The device uses a fine-tuned AI model to provide individually optimized services to users. For example, when a user composes a message, the AI ​​model suggests the most appropriate words and grammar based on past usage history. When a user begins to type "Hello," the AI ​​will provide predictive completion, such as "Hello, what are your plans for today?", supporting the user's operations.

[0106] Feedback and Service Improvement

[0107] The server collects and analyzes feedback from users. Users can post to the server via their devices any problems or requests for improvements they may have while using the service. For example, feedback such as "predictive text doesn't work well" is sent to the server. The server analyzes this information and collects information to improve the service. The service is then updated and improved, and the improved plug-in is distributed to the device again.

[0108] Examples of concrete examples and prompts

[0109] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to a server. The server uses this data to fine-tune a generative AI model, learning the user's unique writing style and expressions. The updated model is then delivered to the device, allowing the AI ​​to make predictions and completions as the user composes messages.

[0110] An example of a prompt sentence is "Generate the next message based on the user's past message history."

[0111] The above description clarifies specific embodiments for carrying out the present invention, which provides a user with a personalized and seamless service experience and improves usability.

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1: Registering the service plugin

[0114] The server provides an interface that allows developers to register various services in the form of plugins. As input, developers provide data for service plugins such as calls, messages, and notes. The server receives this data and stores it in the system's database. As output, the registered plugins are recorded as available in the system. For example, a developer can send a new note plugin to the server via API, and the server will receive it and store it in the database.

[0115] Step 2: Install the plugin

[0116] The terminal connects to the server and gets a list of required plugins. As input, it receives the plugin list information from the server. The terminal downloads and installs the required plugins based on the user's selection. As output, the installed plugins become available on the terminal. For example, the terminal connects to the server to download and install a new message plugin.

[0117] Step 3: User authentication

[0118] A user accesses a plugged-in service through a terminal and enters authentication information. As input, the user provides authentication information such as ID and password. The server receives the authentication information and compares it with the information in the database. If authentication is successful, the output is that the user can access all registered services. For example, when a user logs in to a messaging service, the terminal sends the ID and password to the server for authentication.

[0119] Step 4: Data collection

[0120] The device collects operation data for each service used by the user in real time. As input, it acquires user operation data (call history, message sending history, memo records, etc.). The device saves this data as a log in the background and sends it to the server. As output, the collection of operation data and its transfer to the server is completed. For example, when a user makes a call, the call history is recorded on the device and sent to the server.

[0121] Step 5: Save Data

[0122] The server receives operation data sent from the terminal and stores it in a secure database. As input, it receives operation data from the terminal. The server converts this data into an appropriate format and stores it in the database. As output, the operation history for each user is stored in the database. For example, the server records the message history received from the terminal in the database.

[0123] Step 6: Data preprocessing and cleansing

[0124] The server preprocesses the stored data to remove noise and unnecessary information. It uses the stored data in the database as input. The server cleanses the data, extracting and formatting only the necessary parts. As output, a clean dataset suitable for analysis is generated. For example, the server removes unnecessary whitespace and duplicate data from message data.

[0125] Step 7: Fine-tuning the generative AI model

[0126] The server uses the preprocessed data to fine-tune a generative AI model. It uses a clean, preprocessed dataset as input. The server trains the AI ​​model on this data to optimize performance. The output is a fine-tuned AI model. For example, the server trains an AI model that learns writing style and expressions based on a user's past messaging history.

[0127] Step 8: Deploy and update the AI ​​model

[0128] The fine-tuned AI model is then sent from the server to the device, updating the device's AI engine. The fine-tuned AI model on the server is used as input. The server then sends the AI ​​model to the device via the internet. As output, the AI ​​engine on the device is updated based on the latest AI model. For example, the server sends the latest generative AI model to the device, enhancing the device's AI capabilities.

[0129] Step 9: Improving service delivery and user experience

[0130] The device uses a fine-tuned AI model to provide optimized services to users. As input, it uses the updated AI model and real-time user operation data. The device makes suggestions and completions to assist the user's operations. As output, a seamless service individually optimized for the user is provided. For example, when a user composes a new message, the AI ​​suggests the most appropriate predictive text.

[0131] Step 10: Gather feedback

[0132] The server collects and analyzes feedback from users. As input, it receives feedback information provided by users while using the service. The server analyzes this and identifies areas for improvement. As output, improvements to the service are compiled and reflected in the next update. For example, a user may send feedback to the server that "predictive text conversion did not work properly," and the server will analyze the problem.

[0133] Step 11: Service Improvement

[0134] The server improves the service based on the feedback. It uses the collected feedback information and analysis results as input. The server improves the plugins and AI models based on this and distributes them as the next update. As output, the improved plugins and AI models are provided to users in the next release. For example, the server makes improvements to improve the accuracy of predictive text conversion and releases it as a new version.

[0135] Through the above steps, this system provides various services that are managed in an integrated manner, improving usability.

[0136] (Application example 1)

[0137] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0138] In conventional brick-and-mortar stores, product recommendations based on user behavior and past purchase history were not adequately provided, and improving the user experience was an issue. In addition, there was a lack of means to quickly and appropriately provide users with the information they needed, which often made shopping in brick-and-mortar stores cumbersome.

[0139] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0140] In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for fine-tuning a generative AI model using the saved data, means for delivering the fine-tuned AI model to the terminal, means for providing a service based on the fine-tuned AI model to the user, means for collecting feedback from the user and improving the service, means for the terminal to make product suggestions to assist the user in a physical store, and means for displaying the suggestions on a display device of the terminal. This makes it possible to make appropriate product suggestions based on the user's behavior and past purchase history, significantly improving the shopping experience in a physical store.

[0141] A "plug-in" is an additional module that extends the functionality of a particular piece of software.

[0142] "Terminal" refers to an electronic device used by a user, and examples include smartphones and smart glasses.

[0143] "User operation data" refers to information relating to operations performed by a user, including messages, calls, notes, actions taken within the store, and the like.

[0144] A "generative artificial intelligence model" is a machine learning model that generates new information and suggestions based on data.

[0145] "Fine-tuning" is the optimization process of adapting an existing generative artificial intelligence model to a specific dataset to improve its accuracy.

[0146] "Feedback" refers to information such as ratings and opinions provided by users, and is used to improve services.

[0147] "Product suggestions" refers to proposing appropriate products and services based on a user's past purchasing history and behavior.

[0148] "Display device" refers to a device or screen for visually displaying information, including, for example, the display of smart glasses.

[0149] 1. System Programming

[0150] To realize this application example, a series of programs are required to register various services in plug-in format, install them on the device, collect, save, and analyze user operation data, and fine-tune the generative AI model. The specific implementation method is explained below.

[0151] 2. Explanation of the generated program processing

[0152] The server provides an interface for developers to register various services in the form of plug-ins, allowing them to add new services. The registered plug-ins are then downloaded and installed on the device. The first application installed is the shopping assistant application, which includes a user authentication function.

[0153] The terminal uses hardware such as smart glasses or smartphones to collect user operation data in real time. This operation data includes call data, message data, memo data, and in-store behavior data. The collected data is sent to a backend server and stored in a secure database.

[0154] The server preprocesses the stored data and fine-tunes the generative AI model. For example, it uses Python data analysis libraries (such as NumPy and Pandas) to cleanse, classify, and shape the data. It then fine-tunes the data using a generative AI model (such as TensorFlow or PyTorch). This generates a model customized for each user.

[0155] The fine-tuned AI model is then sent back to the device, where personalized product suggestions are made to the user. Specifically, products based on the user's past purchase history and preferred styles are displayed on the smart glasses' display. Suggestions such as "This jacket is similar to a style you liked in the past. Would you like to try it on?" are displayed, improving the user's shopping experience.

[0156] 3. Examples and prompts

[0157] For example, if a user is in a clothing store in a shopping mall, the smart glasses can suggest new products based on the user's past purchases and preferred styles. The user can look at the display and receive suggestions such as, "This jacket is similar to a style you liked in the past. Would you like to try it on?"

[0158] An example prompt for a generative AI model might look something like this:

[0159] Generate new product suggestions based on the user's past purchases, and take into account the user's current location to make the most of your in-store product data.

[0160] In this way, the present invention significantly improves the shopping experience in physical stores by suggesting products based on user behavior data and past purchase history.

[0161] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0162] Step 1: Register the plugin

[0163] The server provides an interface for registering various services in the form of plugins. The input includes information about the various services registered by the developer. The server saves the entered service information in a database and makes it manageable as a plugin. This allows the various services to be registered on the server and ready to be distributed to devices.

[0164] Step 2: Install the plugin

[0165] The device downloads the specified plugin from the server and installs it on the system. The input includes plugin data sent from the server. The device receives this data, stores it in its file system, and the registered service becomes available.

[0166] Step 3: User authentication

[0167] The user uses the terminal to input authentication information to access various services. The input includes the user's authentication information (e.g., user ID, password). The terminal sends this authentication information to the server, which then verifies it against a database. If authentication is successful, the terminal becomes able to access all registered services.

[0168] Step 4: Collecting operational data

[0169] The terminal collects user operation data (calls, messages, notes, and in-store actions) in real time. The input includes the user's operation behavior. The terminal converts and prepares this data in the required format to send to the backend server.

[0170] Step 5: Save Data

[0171] The server stores the operation data sent from the terminal in a secure database. The input includes the operation data sent from the terminal. The server stores this data in a database and manages the operation history for each user.

[0172] Step 6: Preprocessing the data

[0173] The server preprocesses the stored operational data to remove noise and unnecessary data. The input includes raw data stored in a database. The server performs data cleansing using data analysis libraries (e.g., NumPy, Pandas) to generate a clean dataset.

[0174] Step 7: Fine-tuning the generative AI model

[0175] The server uses the preprocessed data to fine-tune a generative artificial intelligence model. The input includes a clean dataset. The server optimizes the AI ​​model using machine learning libraries (e.g., TensorFlow, PyTorch) and generates a customized model for each user.

[0176] Step 8: Serving the Model

[0177] The server delivers the fine-tuned generative artificial intelligence model to the device, which includes the optimized AI model as input. The server sends the model to the device, which then integrates the model into its system.

[0178] Step 9: Service Delivery

[0179] The device uses a fine-tuned generative artificial intelligence model to provide personalized product recommendations to users. Inputs include the optimized AI model and user operation data. Specifically, the device displays product recommendations based on past purchase history and behavior on the smart glasses display.

[0180] Step 10: Gather feedback

[0181] Users can input feedback about problems they encounter while using the service or requests for improvements. This feedback includes user ratings and opinions. The device sends this feedback to the server, which then analyzes it as data for improving the service.

[0182] Through these steps, the present invention improves the overall shopping experience in brick-and-mortar stores.

[0183] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0184] The present invention is a system that provides individually optimized services by combining a system for registering various services in a plug-in format and managing and providing them in an integrated manner with an emotion engine that recognizes the user's emotions. Specific embodiments for implementing the present invention and each processing step are described below in natural language.

[0185] Registering a service plugin

[0186] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, and notes, and provides a mechanism for integrating these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[0187] Plugin installation and user authentication

[0188] The device will be able to download plugins from the server, and if a user wants to use a particular service, they will be required to install the corresponding plugin on their device.

[0189] Users access various services through the plug-in installed on their devices and enter their authentication information for authentication. Once authentication is successful, users can access all registered services.

[0190] Data collection and storage

[0191] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. The saved data is classified by user and pre-processed as needed.

[0192] Data preprocessing and cleansing

[0193] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and formatted for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[0194] Introducing the Emotion Engine

[0195] The server recognizes the user's emotions based on the stored operation data using an emotion engine, which extracts emotion data from the content of the user's messages and calls using text and voice analysis.

[0196] Fine-tuning generative artificial intelligence

[0197] The server uses the preprocessed operation data and emotion data to fine-tune the generative artificial intelligence (AI) model. By incorporating the emotion data, the AI ​​model learns individual needs and tendencies based on the user's emotions. Once the fine-tuned AI model is complete, it is distributed to the device, where the AI ​​model is updated.

[0198] To provide services and improve user experience

[0199] The device will use fine-tuned AI models to provide individually optimized services to users. For example, in a memo app, AI will provide assistance and suggestions based on the user's emotions, improving operational efficiency.

[0200] Feedback and Service Improvement

[0201] The server receives and collects feedback from users. Users can post any problems or requests for improvements they have noticed while using the service to the server. The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements have been decided, the server updates the plugins and AI models and distributes the new versions to devices.

[0202] Specific examples

[0203] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to the server. The server uses this data and an emotion engine to recognize the user's emotional state and fine-tune the generative AI. The updated model delivered to the device is then used by the AI ​​to make predictions and completions when the user composes a message. Taking the emotion data into account, the AI ​​can make optimal suggestions based on the user's emotional state. This increases user efficiency and satisfaction.

[0204] The above is a specific embodiment for carrying out the present invention. The present invention provides users with a more personalized and seamless service experience, thereby improving usability.

[0205] The processing flow will be explained below.

[0206] Step 1:

[0207] The server exposes an API for registering various services in the form of plugins. Once services such as calls, messages, and notes are registered as plugins, the server integrates them into the system.

[0208] Step 2:

[0209] The device downloads and installs the necessary plugins from the server. When a user wants to use a particular service, they are required to install the corresponding plugin for that service on their device.

[0210] Step 3:

[0211] Users access various services through a plug-in installed on their device and enter their authentication information for authentication. Once authentication is successful, the user can access all registered services.

[0212] Step 4:

[0213] The device collects real-time data on user operations such as calls, messages, and notes, and stores this data as logs in the background.

[0214] Step 5:

[0215] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and pre-processed as necessary.

[0216] Step 6:

[0217] The server preprocesses the stored operational data to remove noise and unnecessary data, resulting in a clean dataset.

[0218] Step 7:

[0219] The server recognizes the user's emotions based on the pre-processed data using an emotion engine, which uses text analysis of messages and audio analysis of calls to identify the user's emotional state.

[0220] Step 8:

[0221] The server uses the interaction data and recognized emotion data to fine-tune a generative artificial intelligence (AI) model, enabling advanced pattern recognition that takes the user's emotional state into account.

[0222] Step 9:

[0223] The server then distributes the fine-tuned AI model to the device, which updates the AI ​​model on the device to reflect the latest user information.

[0224] Step 10:

[0225] The device will utilize fine-tuned AI models to provide optimized services to users, such as messaging services, where the AI ​​will provide assistance and suggestions based on the user's emotions.

[0226] Step 11:

[0227] Users can provide feedback to the server about any problems or requests for improvements they may have while using the system. This feedback can be easily sent via the terminal.

[0228] Step 12:

[0229] The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements are determined, the server updates the plugins and AI models and distributes the new versions to devices.

[0230] Specific examples

[0231] For example, when a user uses a messaging service, the device collects the content of the message sent by the user and their emotional state at the time of sending it. The server uses this data and an emotion engine to recognize the user's emotional state and fine-tune the AI ​​model. Next, using the updated model delivered to the device, the AI ​​makes emotion-based predictions and completions when composing messages. For example, if the AI ​​recognizes that the user is feeling angry, it can assist in smoother communication by suggesting a calmer message. This results in improved user efficiency and satisfaction.

[0232] This concludes the detailed explanation of each processing step. This system enables us to provide users with a seamless and personalized service experience.

[0233] Example 2

[0234] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0235] In today's diverse service delivery systems, there is a demand for services that are individually optimized for each user, but conventional systems require individual settings for each service, making it difficult to efficiently manage data and optimize individual services based on emotion recognition. Therefore, a system is needed that can manage a large number of different services in an integrated manner and provide optimized services that also respond to the user's emotional state.

[0236] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0237] In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for preprocessing the saved data and removing noise and unnecessary data, means for recognizing a user's emotion with an emotion engine using the preprocessed data, means for fine-tuning a generative AI model using the recognized emotion data, means for delivering the fine-tuned AI model to the terminal, service providing means for providing a service to the user based on the fine-tuned AI model, and means for collecting feedback from the user and improving the service. This makes it possible to manage a variety of services in an integrated manner and provide optimized services that also correspond to the emotional state of the user.

[0238] "Various services" refers to different functions and offerings that users can use, such as calls, messages, and notes.

[0239] The "plug-in format" refers to a format in which various services are modularized and can be easily added or removed from the system.

[0240] "Terminal" refers to a computing device, such as a smartphone, tablet, or PC, that a user uses to access various services.

[0241] "Operational Data" refers to information generated or collected when a user uses various services, such as call records, message sending history, and memo input data.

[0242] "Preprocessing" refers to the process of removing noise and unnecessary data from stored data and preparing it in a form that is easy to analyze.

[0243] "Noise" refers to unnecessary information that is not needed in the data processing process and that may adversely affect the results of the analysis.

[0244] "Emotion Engine" refers to an algorithm or software that uses text and speech analysis techniques to recognize and extract a user's emotional state.

[0245] A "generative artificial intelligence model" refers to a machine learning model that can learn large amounts of data and generate new content, such as GPT-3 or BERT.

[0246] "Fine-tuning" refers to the process of adjusting an existing machine learning model to optimize its performance for a specific dataset or application.

[0247] "Feedback" refers to opinions and reports that users provide to the server regarding problems or requests for improvement they have experienced while using the service.

[0248] A "secure database" refers to a data management system in which stored data is protected from unauthorized access and confidentiality, integrity, and availability are ensured.

[0249] MODE FOR CARRYING OUT THE INVENTION

[0250] The present invention is a system that provides individually optimized services by combining a system that registers various services in a plug-in format and manages and provides them in an integrated manner with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described in detail below.

[0251] Registering a service plugin

[0252] The server provides an API interface that allows developers to register various services in the form of plugins. If a developer wants to provide new functions, such as a call recording service, messaging service, or memo service, they can register them as plugins. Specifically, the plugins are uploaded using the API provided by the server and integrated into the entire system.

[0253] Plugin installation and user authentication

[0254] The device has the ability to download and install plug-ins from the server. When a user wants to use a particular service, such as a call recording service, the plug-in is installed on the device. The user then accesses the service through the installed plug-in on the device and is authenticated by entering authentication information, such as a username and password. If the authentication is successful, the user can access all registered services.

[0255] Data collection and storage

[0256] The device collects real-time operational data, such as calls, messages, and notes, used by the user. This operational data is saved as a log file on the device and periodically sent to the server. The server stores this data in a secure database, where it is categorized by user.

[0257] Data preprocessing and cleansing

[0258] The server preprocesses the stored operational data to remove noise and unnecessary data. Noise refers to unnecessary information that is not needed in the data processing process and may adversely affect the analysis results. This preprocessing generates a clean dataset. The preprocessed data is organized into time-series data, making it easier to analyze.

[0259] Introducing the Emotion Engine

[0260] The server uses the cleansed operation data to recognize the user's emotions with an emotion engine. This emotion engine uses text analysis (e.g., natural language processing technology) and voice analysis (e.g., voice recognition technology) to extract emotion data from the user's messages and call content. For example, emotions such as "tired" or "happy" can be detected from messages sent by the user.

[0261] Fine-tuning generative artificial intelligence

[0262] The server uses the preprocessed operation data and emotion data to fine-tune a generative artificial intelligence (AI) model. For example, it uses deep learning models such as GPT-3 and BERT to improve the model's ability to respond and make suggestions based on the user's emotional changes. The fine-tuned AI model is then distributed to the device, where it can be used.

[0263] To provide services and improve user experience

[0264] The device uses fine-tuned generative AI models to provide individually optimized services to users. For example, in a notes app, when a user types "I'm tired," it automatically suggests "Make time to take a break." This increases user efficiency and satisfaction.

[0265] Prompt Sentence Examples

[0266] Based on the following conversation history, we present examples of prompt sentences that recognize the user's emotions and provide appropriate suggestions.

[0267] "Based on the following conversation history, please recognize the user's emotions and provide appropriate suggestions.

[0268] Conversation History:

[0269] User: 'I'm extremely busy and stressed today'

[0270] Generative AI model: 'Sounds like a lot of work. Good luck.'

[0271] Also, please complete the following message:

[0272] By inputting this prompt into a generative AI model, it is possible to understand the user's emotions and provide the most appropriate message for the situation.

[0273] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0274] Step 1: Registering the service plugin

[0275] The server provides an API interface that allows developers to register various services in the form of plugins. Specifically, developers create new services as plugins and send the plugin's metadata and files to the server. The inputs are the plugin specifications and executable files, and the output is that the plugin is registered in the system. This makes services such as call logs, messages, and notes available in the system.

[0276] Step 2: Installing the plugin and authenticating the user

[0277] The device has the function of downloading and installing registered plugins from the server. When a user uses a specific service, the device allows access to the service via the installed plugin. During this process, the plugin file from the server and the user's authentication information (username, password) are used as input, and the service becomes available as output. If authentication is successful, the user will be able to access the corresponding service.

[0278] Step 3: Data collection and storage

[0279] The device collects operational data such as calls, messages, and notes used by the user in real time. This data is stored in the device as a log file and periodically sent to the server. The input is user operation data (call records, message contents, note contents), and the output is the saved log file and sent data. The server saves the received operation data in a secure database. The saved data is categorized by user.

[0280] Step 4: Preprocessing and cleansing the data

[0281] The server preprocesses the stored operational data, specifically removing noise, filling in missing data, and detecting outliers. The input is the stored operational data, and the output is a clean dataset, improving the quality of the data and making it easier to analyze.

[0282] Step 5: Implementing the Emotion Engine

[0283] The server uses the preprocessed operation data to run an emotion engine. This emotion engine uses text analysis and speech analysis techniques to recognize the user's emotional state. The preprocessed operation data (message content, call content) is used as input, and emotion data is generated as output. For example, emotions such as "I'm stressed" can be detected through text analysis.

[0284] Step 6: Fine-tuning the generative AI

[0285] The server uses the preprocessed interaction data and emotion data to fine-tune a generative artificial intelligence (AI) model. Specifically, it applies it to a generative AI model (e.g., GPT-3) to improve its ability to generate responses based on user emotions. The clean interaction data and emotion data are input, and a fine-tuned AI model is generated as output. This model learns optimal responses based on individual needs.

[0286] Step 7: Providing services and improving user experience

[0287] The device uses a fine-tuned generative AI model to provide optimal services to users. For example, if a user uses a memo app and enters "I'm tired," the AI ​​generates a suggestion such as "I recommend you take a rest." The inputs include user operations and emotional data, and the output is the AI's optimal suggestion. This improves user efficiency and satisfaction.

[0288] Step 8: Feedback and Service Improvement

[0289] The server collects feedback from users and uses it to improve the service. Users can post any problems or requests for improvements they have noticed while using the service to the server. The input is user feedback, and the output is analyzed improvements. The server analyzes the feedback, updates the plugins and AI models, and delivers the improved versions to the devices. This allows for continuous improvement of the service.

[0290] (Application example 2)

[0291] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0292] Current content delivery systems recommend content uniformly without considering the user's emotional state, resulting in a decrease in user satisfaction and experience value. Specifically, they may be unable to recommend relaxing content to a stressed user, while recommending inappropriate content to a user in an energetic state. A system that can solve these problems and realize optimal content recommendations based on the user's emotions is needed.

[0293] The specification processing by specification processing unit 290 of data processing device 12 in application example 2 is realized by the following means. In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for fine-tuning the generative AI model using the saved data, means for delivering the fine-tuned AI model to the terminal, means for generating emotion data using an emotion engine that recognizes the user's emotions, means for recommending optimal content to the user based on the generated emotion data, service providing means for providing the user with a service based on the fine-tuned generative AI model, and means for collecting user feedback and improving the service. This makes it possible to recommend optimal content according to the user's emotional state.

[0294] "Various services" are abstract means that provide functions and actions such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis that users can use.

[0295] The "plug-in format" is a method of modularizing various services and providing them in a form that can be easily installed and updated.

[0296] "Terminal" refers to a device used by a user, including smartphones, tablets, head-mounted displays, etc.

[0297] "User operation data" refers to call records, message history, memo contents, heart rate, facial expressions, and voice data generated when a user operates a terminal.

[0298] "Stored data" refers to the state in which collected user operation data is stored in a secure database.

[0299] A "generative artificial intelligence model" is a machine learning model that learns from collected data and generates useful information and services for users.

[0300] "Fine-tuning" is the act of fine-tuning an existing generative artificial intelligence model to optimize its performance for a specific dataset or purpose.

[0301] An "emotion engine" is a software component that analyzes user operation data and recognizes and extracts their emotional state.

[0302] "Emotion data" is a data set that is analyzed using an emotion engine and indicates the user's emotional state.

[0303] "Content" refers to media files that can be viewed or listened to, and is an information resource that includes movies, music videos, documentaries, etc.

[0304] "Recommendation" is the act of the system selecting and providing content or services that it deems most suitable based on the user's data and emotional state.

[0305] "Feedback" refers to information that indicates evaluations, impressions, and requests for improvement provided by users regarding the system.

[0306] This invention is a system that provides individually optimized content by combining a system that registers various services in a plug-in format and manages and provides them in an integrated manner with an emotion engine that recognizes the user's emotions.

[0307] Registering a service plugin

[0308] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis, and integrates these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[0309] Plugin installation and user authentication

[0310] The device downloads and installs the plug-in from the server. When a user wants to use a specific service, the device is required to install the plug-in corresponding to that service. The user accesses various services through the plug-in installed on the device and is authenticated by entering authentication information. If authentication is successful, the user can access all registered services.

[0311] Data collection and storage

[0312] The device collects real-time operational data, such as user calls, messages, notes, heart rate measurements, facial expressions, and voice analysis. The collected operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. The saved data is classified by user and preprocessed as necessary.

[0313] Data preprocessing and cleansing

[0314] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and shaped for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[0315] Introducing the Emotion Engine

[0316] The server recognizes the user's emotions based on the stored operation data using an emotion engine, which uses text analysis and voice analysis to extract emotion data from the user's messages, call content, heart rate, and facial expressions.

[0317] Fine-tuning generative artificial intelligence

[0318] The server uses the preprocessed operation data and emotion data to fine-tune the generative artificial intelligence (AI) model. By incorporating the emotion data, the AI ​​model learns individual needs and tendencies based on the user's emotions. Once the fine-tuned AI model is complete, it is distributed to the device, where the AI ​​model is updated.

[0319] To provide services and improve user experience

[0320] The device uses a fine-tuned AI model to recommend content that is individually optimized for each user. For example, if a user using a head-mounted display is recognized as feeling stressed, movies and music with a relaxing effect will be recommended. A user in an energetic state will be recommended action movies and dance videos. In this way, personalized content is provided according to the user's current emotional state.

[0321] Feedback and Service Improvement

[0322] The server receives and collects feedback from users. Users can post any problems or requests for improvements they have noticed while using the service to the server. The server analyzes the collected feedback and identifies areas for improvement in the service. Once the improvements have been decided, the server updates the plugins and AI models and distributes the new versions to devices.

[0323] For example, a user may want to relax late at night and put on a head-mounted display. The system can recognize the user's heart rate and facial expression as indicating fatigue or a desire for relaxation, and recommend relaxing music or landscape videos. If the user feels energetic and energetic, action movies or dance videos may be recommended.

[0324] An example of a prompt sentence to be input to a generative artificial intelligence model is:

[0325] "If a user is in a state where they want to relax, what are the best movies to recommend?"

[0326] "Provide effective music when the user's heart rate is high and they are feeling fatigued."

[0327] There is.

[0328] The above is a specific embodiment for carrying out the present invention. This system makes it possible to provide users with a more personalized and seamless service experience, thereby improving usability.

[0329] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0330] Step 1:

[0331] The server provides an interface for registering various services in the form of plugins. Developers use the API to register services such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis. In this step, service data is received as input and output is obtained for each plugin to be integrated into the server.

[0332] Step 2:

[0333] The device downloads and installs the plug-in from the server. The user installs the plug-in to use a specific service through the device. In this step, an installation request is input and the device outputs the state in which the plug-in has been installed.

[0334] Step 3:

[0335] The user accesses various services through the plug-in installed on the terminal and enters authentication information to be authenticated. If authentication is successful, the user can access all registered services. In this step, authentication information is entered and the result of authentication success / failure is output.

[0336] Step 4:

[0337] The device collects user operation data in real time, including call records, message history, memo contents, heart rate data, facial expression data, and voice data. In this step, operation data is input and the collected data log is output.

[0338] Step 5:

[0339] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and preprocessed as needed. In this step, operation data is input and a preprocessed clean dataset is output.

[0340] Step 6:

[0341] The server preprocesses the stored data to remove noise and unnecessary data. This process generates a clean dataset and organizes data in different formats in chronological order. In this step, the data to be preprocessed is input and a cleaned dataset is output.

[0342] Step 7:

[0343] The server uses an emotion engine to recognize the user's emotions based on the stored operation data. It extracts emotion data from the content of the user's messages and calls using text analysis and voice analysis. In this step, operation data is input and emotion data is output.

[0344] Step 8:

[0345] The server uses the collected and preprocessed operation data and emotion data to fine-tune the generative AI model, thereby learning the user's individual needs and tendencies. In this step, the preprocessed data and emotion data are input, and a fine-tuned AI model is output.

[0346] Step 9:

[0347] The server delivers the fine-tuned generative AI model to the device, which then uses the updated AI model to provide personalized content to the user. In this step, an AI model is input and a fine-tuned AI model is output that is delivered to the device.

[0348] Step 10:

[0349] The device uses a fine-tuned AI model to recommend content that is individually optimized for the user. For example, if the system recognizes that the user is feeling stressed, it will recommend content that has a relaxing effect. In this step, emotional data is input and recommended content is output.

[0350] Step 11:

[0351] The server collects and analyzes user feedback. Based on the feedback, it identifies improvements to the system and updates the plugins and AI models. In this step, feedback data is input and an improved system is output.

[0352] Step 12:

[0353] For example, if the emotion recognition result indicates "fatigue" when a user wants to relax, relaxing music and landscape videos will be recommended. If the user is feeling energetic and lively, action movies and dance videos will be recommended. In this step, the emotional state is input and optimized content is output.

[0354] A concrete example of a prompt is:

[0355] "If a user is in a state where they want to relax, what are the best movies to recommend?"

[0356] "Provide effective music when the user's heart rate is high and they are feeling fatigued."

[0357] There is.

[0358] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0359] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0360] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0361] [Second embodiment]

[0362] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0363] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0364] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0365] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0366] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0367] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0368] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0369] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0370] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0371] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0372] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0373] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0374] The present invention is a system for registering various services in a plug-in format and managing and providing them in an integrated manner. Specific embodiments for carrying out the present invention and each processing step are described below in natural language.

[0375] Registering a service plugin

[0376] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, and notes, and provides a mechanism for incorporating these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[0377] Plugin installation and user authentication

[0378] The device installs the plug-in downloaded from the server, enabling the use of various services. The user accesses the plug-in service through the device and enters authentication information for authentication. If authentication is successful, the user can access all registered services.

[0379] Data collection and storage

[0380] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. This database stores the operation history for each user.

[0381] Data preprocessing and cleansing

[0382] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and shaped for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[0383] Fine-tuning generative artificial intelligence

[0384] The server uses the preprocessed data to fine-tune the generative artificial intelligence (AI) model. The saved operation data is used as training data for the AI ​​model, improving its performance. Once fine-tuned, the AI ​​model is sent to the device, where it is updated.

[0385] To provide services and improve user experience

[0386] The device uses a fine-tuned AI model to provide individually optimized services to users. When users compose messages, the AI ​​model provides assistance and suggestions based on the user's past usage history. This streamlines user operations and provides a seamless user experience.

[0387] Feedback and Service Improvement

[0388] The server receives and collects feedback from users. Users can post to the server any problems or requests for improvements they have experienced while using the service. The server analyzes the collected feedback and identifies areas for improvement in the service. The service is updated and improved, and a more advanced service is provided to users.

[0389] Specific examples

[0390] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to the server. The server uses this data to fine-tune the generative AI and learns the user's unique writing style and expressions. The updated model is then used by the AI ​​to make predictions and completions as the user composes messages. This improves the user's operational efficiency.

[0391] The above is a specific embodiment for carrying out the present invention. The present invention provides users with a more personalized and seamless service experience, thereby improving usability.

[0392] The processing flow will be explained below.

[0393] Step 1:

[0394] The server exposes an API for registering various services in the form of plugins. Once services such as calls, messages, and notes are registered as plugins, the server integrates them into the system.

[0395] Step 2:

[0396] The device will be able to download plugins from the server, and if a user wants to use a particular service, they will be required to install the corresponding plugin on their device.

[0397] Step 3:

[0398] Users access various services through plug-ins installed on their devices. When accessing the services, the user enters their authentication information and is authenticated by the server.

[0399] Step 4:

[0400] The device collects real-time data on user operations such as calls, messages, and notes, and stores this data as logs in the background.

[0401] Step 5:

[0402] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and pre-processed as necessary.

[0403] Step 6:

[0404] The server uses the stored data to fine-tune a generative artificial intelligence (AI) model, allowing the model to learn user interaction patterns and behavioral characteristics.

[0405] Step 7:

[0406] The server then distributes the fine-tuned artificial intelligence model to the device, which updates the AI ​​model on the device.

[0407] Step 8:

[0408] The device will utilize fine-tuned AI models to provide individually optimized services for users, such as in a note-taking app where AI predicts what you're typing and auto-completes the text, improving operational efficiency.

[0409] Step 9:

[0410] Users can provide feedback to the server about any problems or requests for improvements they may have while using the system. This feedback can be easily sent via the terminal.

[0411] Step 10:

[0412] The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements are determined, the server updates the plugins and AI models and distributes the new versions to devices.

[0413] This is the specific program processing flow, which allows users to enjoy a consistent and seamless service experience.

[0414] Example 1

[0415] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0416] In conventional systems, plugins for providing various services are managed separately, making it difficult to provide integrated services. Furthermore, to effectively utilize user operation data and provide individually optimized services, data collection, storage, preprocessing, analysis, and fine-tuning of generative AI models are required, but these processes are often not consistently integrated. This results in the challenge of making it difficult to improve the quality of the user experience.

[0417] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0418] In this invention, the server includes a means for allowing developers to register various services in the form of plug-ins, a means for registering plug-ins using an API provided by the server, and a means for installing the registered plug-ins on a terminal. This enables integrated management and provision of various services. The terminal also includes a means for entering user authentication information to access services, a means for collecting user operation data in real time, and a means for storing the collected operation data. The server further includes a means for preprocessing the stored data and removing noise and unnecessary data, a means for fine-tuning a generative AI model using the preprocessed data, a means for delivering the fine-tuned AI model to the terminal, a means for optimizing and providing services to users based on the fine-tuned AI model, and a means for collecting and analyzing user feedback to improve the service. This enables a personalized and seamless service experience based on user operation data, improving usability.

[0419] A "plug-in format" is a small program module that adds functionality to software.

[0420] "API" stands for Application Programming Interface, a means by which different software applications can communicate with each other.

[0421] A "terminal" is a hardware device (e.g., a smartphone or computer) that a user operates to access various services.

[0422] "Authentication information" refers to information such as ID and password used by users when accessing a system.

[0423] "Operation data" refers to data generated when a user operates the system (for example, call logs, message history, memo records, etc.).

[0424] "Real-time" means that data is processed immediately at the moment an action is taken.

[0425] "Preprocessing" refers to cleansing and shaping data before analyzing it.

[0426] "Noise" refers to unnecessary or meaningless data in data analysis.

[0427] A "generative artificial intelligence model" is an artificial intelligence algorithm that generates new data based on past data.

[0428] "Fine-tuning" is the process of further optimizing an existing artificial intelligence model using a specific dataset.

[0429] "Feedback" refers to opinions and information provided by users when they report improvements or problems with the system.

[0430] "Usability" is a concept that refers to how easy a system or service is to use.

[0431] This invention is a system that registers various services in a plug-in format and manages and provides them in an integrated manner. In this system, the server, terminal, and user each play different roles, realizing seamless and personalized services.

[0432] Registering a service plugin

[0433] The server provides an interface that allows developers to register various services in the form of plugins. This interface is implemented using the API provided by the server. Developers can register services such as calls, messages, and memos as plugins on the server. For example, if a developer wants to provide a new memo function as a plugin, they send this plugin to the server via the API. The server receives it and incorporates it into the system by storing it in a database.

[0434] Plugin installation and user authentication

[0435] The device installs the plug-in downloaded from the server. The user accesses the plug-in service through the device and authenticates by entering their authentication information (e.g., ID and password). If authentication is successful, the user will be able to access all registered services. For example, if a user installs a message service plug-in and authenticates, the message service will become available on that device.

[0436] Data collection and storage

[0437] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background and sent to the server periodically or in real time. The server receives the operational data sent from the device and stores it in a secure database. The database stores the operation history for each user.

[0438] Data preprocessing and cleansing

[0439] The server preprocesses the collected data to remove noise and unnecessary data. For example, meaningless spaces and duplicate data are removed from message content. The preprocessed data is then classified and formatted for analysis, and different types of data, such as call logs and message logs, are organized chronologically.

[0440] Fine-tuning generative AI models

[0441] The server uses the preprocessed data to fine-tune the generative AI model. The user's past operational data is used as training data for the AI ​​model. Through this process, the model learns the user's unique behavioral patterns and expression methods, improving its performance. Once fine-tuned, the artificial intelligence model is delivered to the device, updating the device's AI engine.

[0442] To provide services and improve user experience

[0443] The device uses a fine-tuned AI model to provide individually optimized services to users. For example, when a user composes a message, the AI ​​model suggests the most appropriate words and grammar based on past usage history. When a user begins to type "Hello," the AI ​​will provide predictive completion, such as "Hello, what are your plans for today?", supporting the user's operations.

[0444] Feedback and Service Improvement

[0445] The server collects and analyzes feedback from users. Users can post to the server via their devices any problems or requests for improvements they may have while using the service. For example, feedback such as "predictive text doesn't work well" is sent to the server. The server analyzes this information and collects information to improve the service. The service is then updated and improved, and the improved plug-in is distributed to the device again.

[0446] Examples of concrete examples and prompts

[0447] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to a server. The server uses this data to fine-tune a generative AI model, learning the user's unique writing style and expressions. The updated model is then delivered to the device, allowing the AI ​​to make predictions and completions as the user composes messages.

[0448] An example of a prompt sentence is "Generate the next message based on the user's past message history."

[0449] The above description clarifies specific embodiments for carrying out the present invention, which provides a user with a personalized and seamless service experience and improves usability.

[0450] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0451] Step 1: Registering the service plugin

[0452] The server provides an interface that allows developers to register various services in the form of plugins. As input, developers provide data for service plugins such as calls, messages, and notes. The server receives this data and stores it in the system's database. As output, the registered plugins are recorded as available in the system. For example, a developer can send a new note plugin to the server via API, and the server will receive it and store it in the database.

[0453] Step 2: Install the plugin

[0454] The terminal connects to the server and gets a list of required plugins. As input, it receives the plugin list information from the server. The terminal downloads and installs the required plugins based on the user's selection. As output, the installed plugins become available on the terminal. For example, the terminal connects to the server to download and install a new message plugin.

[0455] Step 3: User authentication

[0456] A user accesses a plugged-in service through a terminal and enters authentication information. As input, the user provides authentication information such as ID and password. The server receives the authentication information and compares it with the information in the database. If authentication is successful, the output is that the user can access all registered services. For example, when a user logs in to a messaging service, the terminal sends the ID and password to the server for authentication.

[0457] Step 4: Data collection

[0458] The device collects operation data for each service used by the user in real time. As input, it acquires user operation data (call history, message sending history, memo records, etc.). The device saves this data as a log in the background and sends it to the server. As output, the collection of operation data and its transfer to the server is completed. For example, when a user makes a call, the call history is recorded on the device and sent to the server.

[0459] Step 5: Save Data

[0460] The server receives operation data sent from the terminal and stores it in a secure database. As input, it receives operation data from the terminal. The server converts this data into an appropriate format and stores it in the database. As output, the operation history for each user is stored in the database. For example, the server records the message history received from the terminal in the database.

[0461] Step 6: Data preprocessing and cleansing

[0462] The server preprocesses the stored data to remove noise and unnecessary information. It uses the stored data in the database as input. The server cleanses the data, extracting and formatting only the necessary parts. As output, a clean dataset suitable for analysis is generated. For example, the server removes unnecessary whitespace and duplicate data from message data.

[0463] Step 7: Fine-tuning the generative AI model

[0464] The server uses the preprocessed data to fine-tune a generative AI model. It uses a clean, preprocessed dataset as input. The server trains the AI ​​model on this data to optimize performance. The output is a fine-tuned AI model. For example, the server trains an AI model that learns writing style and expressions based on a user's past messaging history.

[0465] Step 8: Deploy and update the AI ​​model

[0466] The fine-tuned AI model is then sent from the server to the device, updating the device's AI engine. The fine-tuned AI model on the server is used as input. The server then sends the AI ​​model to the device via the internet. As output, the AI ​​engine on the device is updated based on the latest AI model. For example, the server sends the latest generative AI model to the device, enhancing the device's AI capabilities.

[0467] Step 9: Improving service delivery and user experience

[0468] The device uses a fine-tuned AI model to provide optimized services to users. As input, it uses the updated AI model and real-time user operation data. The device makes suggestions and completions to assist the user's operations. As output, a seamless service individually optimized for the user is provided. For example, when a user composes a new message, the AI ​​suggests the most appropriate predictive text.

[0469] Step 10: Gather feedback

[0470] The server collects and analyzes feedback from users. As input, it receives feedback information provided by users while using the service. The server analyzes this and identifies areas for improvement. As output, improvements to the service are compiled and reflected in the next update. For example, a user may send feedback to the server that "predictive text conversion did not work properly," and the server will analyze the problem.

[0471] Step 11: Service Improvement

[0472] The server improves the service based on the feedback. It uses the collected feedback information and analysis results as input. The server improves the plugins and AI models based on this and distributes them as the next update. As output, the improved plugins and AI models are provided to users in the next release. For example, the server makes improvements to improve the accuracy of predictive text conversion and releases it as a new version.

[0473] Through the above steps, this system provides various services that are managed in an integrated manner, improving usability.

[0474] (Application example 1)

[0475] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0476] In conventional brick-and-mortar stores, product recommendations based on user behavior and past purchase history were not adequately provided, and improving the user experience was an issue. In addition, there was a lack of means to quickly and appropriately provide users with the information they needed, which often made shopping in brick-and-mortar stores cumbersome.

[0477] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0478] In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for fine-tuning a generative AI model using the saved data, means for delivering the fine-tuned AI model to the terminal, means for providing a service based on the fine-tuned AI model to the user, means for collecting feedback from the user and improving the service, means for the terminal to make product suggestions to assist the user in a physical store, and means for displaying the suggestions on a display device of the terminal. This makes it possible to make appropriate product suggestions based on the user's behavior and past purchase history, significantly improving the shopping experience in a physical store.

[0479] A "plug-in" is an additional module that extends the functionality of a particular piece of software.

[0480] "Terminal" refers to an electronic device used by a user, and examples include smartphones and smart glasses.

[0481] "User operation data" refers to information relating to operations performed by a user, including messages, calls, notes, actions taken within the store, and the like.

[0482] A "generative artificial intelligence model" is a machine learning model that generates new information and suggestions based on data.

[0483] "Fine-tuning" is the optimization process of adapting an existing generative artificial intelligence model to a specific dataset to improve its accuracy.

[0484] "Feedback" refers to information such as ratings and opinions provided by users, and is used to improve services.

[0485] "Product suggestions" refers to proposing appropriate products and services based on a user's past purchasing history and behavior.

[0486] "Display device" refers to a device or screen for visually displaying information, including, for example, the display of smart glasses.

[0487] 1. System Programming

[0488] To realize this application example, a series of programs are required to register various services in plug-in format, install them on the device, collect, save, and analyze user operation data, and fine-tune the generative AI model. The specific implementation method is explained below.

[0489] 2. Explanation of the generated program processing

[0490] The server provides an interface for developers to register various services in the form of plug-ins, allowing them to add new services. The registered plug-ins are then downloaded and installed on the device. The first application installed is the shopping assistant application, which includes a user authentication function.

[0491] The terminal uses hardware such as smart glasses or smartphones to collect user operation data in real time. This operation data includes call data, message data, memo data, and in-store behavior data. The collected data is sent to a backend server and stored in a secure database.

[0492] The server preprocesses the stored data and fine-tunes the generative AI model. For example, it uses Python data analysis libraries (such as NumPy and Pandas) to cleanse, classify, and shape the data. It then fine-tunes the data using a generative AI model (such as TensorFlow or PyTorch). This generates a model customized for each user.

[0493] The fine-tuned AI model is then sent back to the device, where personalized product suggestions are made to the user. Specifically, products based on the user's past purchase history and preferred styles are displayed on the smart glasses' display. Suggestions such as "This jacket is similar to a style you liked in the past. Would you like to try it on?" are displayed, improving the user's shopping experience.

[0494] 3. Examples and prompts

[0495] For example, if a user is in a clothing store in a shopping mall, the smart glasses can suggest new products based on the user's past purchases and preferred styles. The user can look at the display and receive suggestions such as, "This jacket is similar to a style you liked in the past. Would you like to try it on?"

[0496] An example prompt for a generative AI model might look something like this:

[0497] Generate new product suggestions based on the user's past purchases, and take into account the user's current location to make the most of your in-store product data.

[0498] In this way, the present invention significantly improves the shopping experience in physical stores by suggesting products based on user behavior data and past purchase history.

[0499] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0500] Step 1: Register the plugin

[0501] The server provides an interface for registering various services in the form of plugins. The input includes information about the various services registered by the developer. The server saves the entered service information in a database and makes it manageable as a plugin. This allows the various services to be registered on the server and ready to be distributed to devices.

[0502] Step 2: Install the plugin

[0503] The device downloads the specified plugin from the server and installs it on the system. The input includes plugin data sent from the server. The device receives this data, stores it in its file system, and the registered service becomes available.

[0504] Step 3: User authentication

[0505] The user uses the terminal to input authentication information to access various services. The input includes the user's authentication information (e.g., user ID, password). The terminal sends this authentication information to the server, which then verifies it against a database. If authentication is successful, the terminal becomes able to access all registered services.

[0506] Step 4: Collecting operational data

[0507] The terminal collects user operation data (calls, messages, notes, and in-store actions) in real time. The input includes the user's operation behavior. The terminal converts and prepares this data in the required format to send to the backend server.

[0508] Step 5: Save Data

[0509] The server stores the operation data sent from the terminal in a secure database. The input includes the operation data sent from the terminal. The server stores this data in a database and manages the operation history for each user.

[0510] Step 6: Preprocessing the data

[0511] The server preprocesses the stored operational data to remove noise and unnecessary data. The input includes raw data stored in a database. The server performs data cleansing using data analysis libraries (e.g., NumPy, Pandas) to generate a clean dataset.

[0512] Step 7: Fine-tuning the generative AI model

[0513] The server uses the preprocessed data to fine-tune a generative artificial intelligence model. The input includes a clean dataset. The server optimizes the AI ​​model using machine learning libraries (e.g., TensorFlow, PyTorch) and generates a customized model for each user.

[0514] Step 8: Serving the Model

[0515] The server delivers the fine-tuned generative artificial intelligence model to the device, which includes the optimized AI model as input. The server sends the model to the device, which then integrates the model into its system.

[0516] Step 9: Service Delivery

[0517] The device uses a fine-tuned generative artificial intelligence model to provide personalized product recommendations to users. Inputs include the optimized AI model and user operation data. Specifically, the device displays product recommendations based on past purchase history and behavior on the smart glasses display.

[0518] Step 10: Gather feedback

[0519] Users can input feedback about problems they encounter while using the service or requests for improvements. This feedback includes user ratings and opinions. The device sends this feedback to the server, which then analyzes it as data for improving the service.

[0520] Through these steps, the present invention improves the overall shopping experience in brick-and-mortar stores.

[0521] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0522] The present invention is a system that provides individually optimized services by combining a system for registering various services in a plug-in format and managing and providing them in an integrated manner with an emotion engine that recognizes the user's emotions. Specific embodiments for implementing the present invention and each processing step are described below in natural language.

[0523] Registering a service plugin

[0524] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, and notes, and provides a mechanism for integrating these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[0525] Plugin installation and user authentication

[0526] The device will be able to download plugins from the server, and if a user wants to use a particular service, they will be required to install the corresponding plugin on their device.

[0527] Users access various services through the plug-in installed on their devices and enter their authentication information for authentication. Once authentication is successful, users can access all registered services.

[0528] Data collection and storage

[0529] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. The saved data is classified by user and pre-processed as needed.

[0530] Data preprocessing and cleansing

[0531] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and formatted for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[0532] Introducing the Emotion Engine

[0533] The server recognizes the user's emotions based on the stored operation data using an emotion engine, which extracts emotion data from the content of the user's messages and calls using text and voice analysis.

[0534] Fine-tuning generative artificial intelligence

[0535] The server uses the preprocessed operation data and emotion data to fine-tune the generative artificial intelligence (AI) model. By incorporating the emotion data, the AI ​​model learns individual needs and tendencies based on the user's emotions. Once the fine-tuned AI model is complete, it is distributed to the device, where the AI ​​model is updated.

[0536] To provide services and improve user experience

[0537] The device will use fine-tuned AI models to provide individually optimized services to users. For example, in a memo app, AI will provide assistance and suggestions based on the user's emotions, improving operational efficiency.

[0538] Feedback and Service Improvement

[0539] The server receives and collects feedback from users. Users can post any problems or requests for improvements they have noticed while using the service to the server. The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements have been decided, the server updates the plugins and AI models and distributes the new versions to devices.

[0540] Specific examples

[0541] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to the server. The server uses this data and an emotion engine to recognize the user's emotional state and fine-tune the generative AI. The updated model delivered to the device is then used by the AI ​​to make predictions and completions when the user composes a message. Taking the emotion data into account, the AI ​​can make optimal suggestions based on the user's emotional state. This increases user efficiency and satisfaction.

[0542] The above is a specific embodiment for carrying out the present invention. The present invention provides users with a more personalized and seamless service experience, thereby improving usability.

[0543] The processing flow will be explained below.

[0544] Step 1:

[0545] The server exposes an API for registering various services in the form of plugins. Once services such as calls, messages, and notes are registered as plugins, the server integrates them into the system.

[0546] Step 2:

[0547] The device downloads and installs the necessary plugins from the server. When a user wants to use a particular service, they are required to install the corresponding plugin for that service on their device.

[0548] Step 3:

[0549] Users access various services through a plug-in installed on their device and enter their authentication information for authentication. Once authentication is successful, the user can access all registered services.

[0550] Step 4:

[0551] The device collects real-time data on user operations such as calls, messages, and notes, and stores this data as logs in the background.

[0552] Step 5:

[0553] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and pre-processed as necessary.

[0554] Step 6:

[0555] The server preprocesses the stored operational data to remove noise and unnecessary data, resulting in a clean dataset.

[0556] Step 7:

[0557] The server recognizes the user's emotions based on the pre-processed data using an emotion engine, which uses text analysis of messages and audio analysis of calls to identify the user's emotional state.

[0558] Step 8:

[0559] The server uses the interaction data and recognized emotion data to fine-tune a generative artificial intelligence (AI) model, enabling advanced pattern recognition that takes the user's emotional state into account.

[0560] Step 9:

[0561] The server then distributes the fine-tuned AI model to the device, which updates the AI ​​model on the device to reflect the latest user information.

[0562] Step 10:

[0563] The device will utilize fine-tuned AI models to provide optimized services to users, such as messaging services, where the AI ​​will provide assistance and suggestions based on the user's emotions.

[0564] Step 11:

[0565] Users can provide feedback to the server about any problems or requests for improvements they may have while using the system. This feedback can be easily sent via the terminal.

[0566] Step 12:

[0567] The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements are determined, the server updates the plugins and AI models and distributes the new versions to devices.

[0568] Specific examples

[0569] For example, when a user uses a messaging service, the device collects the content of the message sent by the user and their emotional state at the time of sending it. The server uses this data and an emotion engine to recognize the user's emotional state and fine-tune the AI ​​model. Next, using the updated model delivered to the device, the AI ​​makes emotion-based predictions and completions when composing messages. For example, if the AI ​​recognizes that the user is feeling angry, it can assist in smoother communication by suggesting a calmer message. This results in improved user efficiency and satisfaction.

[0570] This concludes the detailed explanation of each processing step. This system enables us to provide users with a seamless and personalized service experience.

[0571] Example 2

[0572] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0573] In today's diverse service delivery systems, there is a demand for services that are individually optimized for each user, but conventional systems require individual settings for each service, making it difficult to efficiently manage data and optimize individual services based on emotion recognition. Therefore, a system is needed that can manage a large number of different services in an integrated manner and provide optimized services that also respond to the user's emotional state.

[0574] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0575] In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for preprocessing the saved data and removing noise and unnecessary data, means for recognizing a user's emotion with an emotion engine using the preprocessed data, means for fine-tuning a generative AI model using the recognized emotion data, means for delivering the fine-tuned AI model to the terminal, service providing means for providing a service to the user based on the fine-tuned AI model, and means for collecting feedback from the user and improving the service. This makes it possible to manage a variety of services in an integrated manner and provide optimized services that also correspond to the emotional state of the user.

[0576] "Various services" refers to different functions and offerings that users can use, such as calls, messages, and notes.

[0577] The "plug-in format" refers to a format in which various services are modularized and can be easily added or removed from the system.

[0578] "Terminal" refers to a computing device, such as a smartphone, tablet, or PC, that a user uses to access various services.

[0579] "Operational Data" refers to information generated or collected when a user uses various services, such as call records, message sending history, and memo input data.

[0580] "Preprocessing" refers to the process of removing noise and unnecessary data from stored data and preparing it in a form that is easy to analyze.

[0581] "Noise" refers to unnecessary information that is not needed in the data processing process and that may adversely affect the results of the analysis.

[0582] "Emotion Engine" refers to an algorithm or software that uses text and speech analysis techniques to recognize and extract a user's emotional state.

[0583] A "generative artificial intelligence model" refers to a machine learning model that can learn large amounts of data and generate new content, such as GPT-3 or BERT.

[0584] "Fine-tuning" refers to the process of adjusting an existing machine learning model to optimize its performance for a specific dataset or application.

[0585] "Feedback" refers to opinions and reports that users provide to the server regarding problems or requests for improvement they have experienced while using the service.

[0586] A "secure database" refers to a data management system in which stored data is protected from unauthorized access and confidentiality, integrity, and availability are ensured.

[0587] MODE FOR CARRYING OUT THE INVENTION

[0588] The present invention is a system that provides individually optimized services by combining a system that registers various services in a plug-in format and manages and provides them in an integrated manner with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described in detail below.

[0589] Registering a service plugin

[0590] The server provides an API interface that allows developers to register various services in the form of plugins. If a developer wants to provide new functions, such as a call recording service, messaging service, or memo service, they can register them as plugins. Specifically, the plugins are uploaded using the API provided by the server and integrated into the entire system.

[0591] Plugin installation and user authentication

[0592] The device has the ability to download and install plug-ins from the server. When a user wants to use a particular service, such as a call recording service, the plug-in is installed on the device. The user then accesses the service through the installed plug-in on the device and is authenticated by entering authentication information, such as a username and password. If the authentication is successful, the user can access all registered services.

[0593] Data collection and storage

[0594] The device collects real-time operational data, such as calls, messages, and notes, used by the user. This operational data is saved as a log file on the device and periodically sent to the server. The server stores this data in a secure database, where it is categorized by user.

[0595] Data preprocessing and cleansing

[0596] The server preprocesses the stored operational data to remove noise and unnecessary data. Noise refers to unnecessary information that is not needed in the data processing process and may adversely affect the analysis results. This preprocessing generates a clean dataset. The preprocessed data is organized into time-series data, making it easier to analyze.

[0597] Introducing the Emotion Engine

[0598] The server uses the cleansed operation data to recognize the user's emotions with an emotion engine. This emotion engine uses text analysis (e.g., natural language processing technology) and voice analysis (e.g., voice recognition technology) to extract emotion data from the user's messages and call content. For example, emotions such as "tired" or "happy" can be detected from messages sent by the user.

[0599] Fine-tuning generative artificial intelligence

[0600] The server uses the preprocessed operation data and emotion data to fine-tune a generative artificial intelligence (AI) model. For example, it uses deep learning models such as GPT-3 and BERT to improve the model's ability to respond and make suggestions based on the user's emotional changes. The fine-tuned AI model is then distributed to the device, where it can be used.

[0601] To provide services and improve user experience

[0602] The device uses fine-tuned generative AI models to provide individually optimized services to users. For example, in a notes app, when a user types "I'm tired," it automatically suggests "Make time to take a break." This increases user efficiency and satisfaction.

[0603] Prompt Sentence Examples

[0604] Based on the following conversation history, we present examples of prompt sentences that recognize the user's emotions and provide appropriate suggestions.

[0605] "Based on the following conversation history, please recognize the user's emotions and provide appropriate suggestions.

[0606] Conversation History:

[0607] User: 'I'm extremely busy and stressed today'

[0608] Generative AI model: 'Sounds like a lot of work. Good luck.'

[0609] Also, please complete the following message:

[0610] By inputting this prompt into a generative AI model, it is possible to understand the user's emotions and provide the most appropriate message for the situation.

[0611] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0612] Step 1: Registering the service plugin

[0613] The server provides an API interface that allows developers to register various services in the form of plugins. Specifically, developers create new services as plugins and send the plugin's metadata and files to the server. The inputs are the plugin specifications and executable files, and the output is that the plugin is registered in the system. This makes services such as call logs, messages, and notes available in the system.

[0614] Step 2: Installing the plugin and authenticating the user

[0615] The device has the function of downloading and installing registered plugins from the server. When a user uses a specific service, the device allows access to the service via the installed plugin. During this process, the plugin file from the server and the user's authentication information (username, password) are used as input, and the service becomes available as output. If authentication is successful, the user will be able to access the corresponding service.

[0616] Step 3: Data collection and storage

[0617] The device collects operational data such as calls, messages, and notes used by the user in real time. This data is stored in the device as a log file and periodically sent to the server. The input is user operation data (call records, message contents, note contents), and the output is the saved log file and sent data. The server saves the received operation data in a secure database. The saved data is categorized by user.

[0618] Step 4: Preprocessing and cleansing the data

[0619] The server preprocesses the stored operational data, specifically removing noise, filling in missing data, and detecting outliers. The input is the stored operational data, and the output is a clean dataset, improving the quality of the data and making it easier to analyze.

[0620] Step 5: Implementing the Emotion Engine

[0621] The server uses the preprocessed operation data to run an emotion engine. This emotion engine uses text analysis and speech analysis techniques to recognize the user's emotional state. The preprocessed operation data (message content, call content) is used as input, and emotion data is generated as output. For example, emotions such as "I'm stressed" can be detected through text analysis.

[0622] Step 6: Fine-tuning the generative AI

[0623] The server uses the preprocessed interaction data and emotion data to fine-tune a generative artificial intelligence (AI) model. Specifically, it applies it to a generative AI model (e.g., GPT-3) to improve its ability to generate responses based on user emotions. The clean interaction data and emotion data are input, and a fine-tuned AI model is generated as output. This model learns optimal responses based on individual needs.

[0624] Step 7: Providing services and improving user experience

[0625] The device uses a fine-tuned generative AI model to provide optimal services to users. For example, if a user uses a memo app and enters "I'm tired," the AI ​​generates a suggestion such as "I recommend you take a rest." The inputs include user operations and emotional data, and the output is the AI's optimal suggestion. This improves user efficiency and satisfaction.

[0626] Step 8: Feedback and Service Improvement

[0627] The server collects feedback from users and uses it to improve the service. Users can post any problems or requests for improvements they have noticed while using the service to the server. The input is user feedback, and the output is analyzed improvements. The server analyzes the feedback, updates the plugins and AI models, and delivers the improved versions to the devices. This allows for continuous improvement of the service.

[0628] (Application example 2)

[0629] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0630] Current content delivery systems recommend content uniformly without considering the user's emotional state, resulting in a decrease in user satisfaction and experience value. Specifically, they may be unable to recommend relaxing content to a stressed user, while recommending inappropriate content to a user in an energetic state. A system that can solve these problems and realize optimal content recommendations based on the user's emotions is needed.

[0631] The specification processing by specification processing unit 290 of data processing device 12 in application example 2 is realized by the following means. In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for fine-tuning the generative AI model using the saved data, means for delivering the fine-tuned AI model to the terminal, means for generating emotion data using an emotion engine that recognizes the user's emotions, means for recommending optimal content to the user based on the generated emotion data, service providing means for providing the user with a service based on the fine-tuned generative AI model, and means for collecting user feedback and improving the service. This makes it possible to recommend optimal content according to the user's emotional state.

[0632] "Various services" are abstract means that provide functions and actions such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis that users can use.

[0633] The "plug-in format" is a method of modularizing various services and providing them in a form that can be easily installed and updated.

[0634] "Terminal" refers to a device used by a user, including smartphones, tablets, head-mounted displays, etc.

[0635] "User operation data" refers to call records, message history, memo contents, heart rate, facial expressions, and voice data generated when a user operates a terminal.

[0636] "Stored data" refers to the state in which collected user operation data is stored in a secure database.

[0637] A "generative artificial intelligence model" is a machine learning model that learns from collected data and generates useful information and services for users.

[0638] "Fine-tuning" is the act of fine-tuning an existing generative artificial intelligence model to optimize its performance for a specific dataset or purpose.

[0639] An "emotion engine" is a software component that analyzes user operation data and recognizes and extracts their emotional state.

[0640] "Emotion data" is a data set that is analyzed using an emotion engine and indicates the user's emotional state.

[0641] "Content" refers to media files that can be viewed or listened to, and is an information resource that includes movies, music videos, documentaries, etc.

[0642] "Recommendation" is the act of the system selecting and providing content or services that it deems most suitable based on the user's data and emotional state.

[0643] "Feedback" refers to information that indicates evaluations, impressions, and requests for improvement provided by users regarding the system.

[0644] This invention is a system that provides individually optimized content by combining a system that registers various services in a plug-in format and manages and provides them in an integrated manner with an emotion engine that recognizes the user's emotions.

[0645] Registering a service plugin

[0646] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis, and integrates these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[0647] Plugin installation and user authentication

[0648] The device downloads and installs the plug-in from the server. When a user wants to use a specific service, the device is required to install the plug-in corresponding to that service. The user accesses various services through the plug-in installed on the device and is authenticated by entering authentication information. If authentication is successful, the user can access all registered services.

[0649] Data collection and storage

[0650] The device collects real-time operational data, such as user calls, messages, notes, heart rate measurements, facial expressions, and voice analysis. The collected operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. The saved data is classified by user and preprocessed as necessary.

[0651] Data preprocessing and cleansing

[0652] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and shaped for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[0653] Introducing the Emotion Engine

[0654] The server recognizes the user's emotions based on the stored operation data using an emotion engine, which uses text analysis and voice analysis to extract emotion data from the user's messages, call content, heart rate, and facial expressions.

[0655] Fine-tuning generative artificial intelligence

[0656] The server uses the preprocessed operation data and emotion data to fine-tune the generative artificial intelligence (AI) model. By incorporating the emotion data, the AI ​​model learns individual needs and tendencies based on the user's emotions. Once the fine-tuned AI model is complete, it is distributed to the device, where the AI ​​model is updated.

[0657] To provide services and improve user experience

[0658] The device uses a fine-tuned AI model to recommend content that is individually optimized for each user. For example, if a user using a head-mounted display is recognized as feeling stressed, movies and music with a relaxing effect will be recommended. A user in an energetic state will be recommended action movies and dance videos. In this way, personalized content is provided according to the user's current emotional state.

[0659] Feedback and Service Improvement

[0660] The server receives and collects feedback from users. Users can post any problems or requests for improvements they have noticed while using the service to the server. The server analyzes the collected feedback and identifies areas for improvement in the service. Once the improvements have been decided, the server updates the plugins and AI models and distributes the new versions to devices.

[0661] For example, a user may want to relax late at night and put on a head-mounted display. The system can recognize the user's heart rate and facial expression as indicating fatigue or a desire for relaxation, and recommend relaxing music or landscape videos. If the user feels energetic and energetic, action movies or dance videos may be recommended.

[0662] An example of a prompt sentence to be input to a generative artificial intelligence model is:

[0663] "If a user is in a state where they want to relax, what are the best movies to recommend?"

[0664] "Provide effective music when the user's heart rate is high and they are feeling fatigued."

[0665] There is.

[0666] The above is a specific embodiment for carrying out the present invention. This system makes it possible to provide users with a more personalized and seamless service experience, thereby improving usability.

[0667] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0668] Step 1:

[0669] The server provides an interface for registering various services in the form of plugins. Developers use the API to register services such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis. In this step, service data is received as input and output is obtained for each plugin to be integrated into the server.

[0670] Step 2:

[0671] The device downloads and installs the plug-in from the server. The user installs the plug-in to use a specific service through the device. In this step, an installation request is input and the device outputs the state in which the plug-in has been installed.

[0672] Step 3:

[0673] The user accesses various services through the plug-in installed on the terminal and enters authentication information to be authenticated. If authentication is successful, the user can access all registered services. In this step, authentication information is entered and the result of authentication success / failure is output.

[0674] Step 4:

[0675] The device collects user operation data in real time, including call records, message history, memo contents, heart rate data, facial expression data, and voice data. In this step, operation data is input and the collected data log is output.

[0676] Step 5:

[0677] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and preprocessed as needed. In this step, operation data is input and a preprocessed clean dataset is output.

[0678] Step 6:

[0679] The server preprocesses the stored data to remove noise and unnecessary data. This process generates a clean dataset and organizes data in different formats in chronological order. In this step, the data to be preprocessed is input and a cleaned dataset is output.

[0680] Step 7:

[0681] The server uses an emotion engine to recognize the user's emotions based on the stored operation data. It extracts emotion data from the content of the user's messages and calls using text analysis and voice analysis. In this step, operation data is input and emotion data is output.

[0682] Step 8:

[0683] The server uses the collected and preprocessed operation data and emotion data to fine-tune the generative AI model, thereby learning the user's individual needs and tendencies. In this step, the preprocessed data and emotion data are input, and a fine-tuned AI model is output.

[0684] Step 9:

[0685] The server delivers the fine-tuned generative AI model to the device, which then uses the updated AI model to provide personalized content to the user. In this step, an AI model is input and a fine-tuned AI model is output that is delivered to the device.

[0686] Step 10:

[0687] The device uses a fine-tuned AI model to recommend content that is individually optimized for the user. For example, if the system recognizes that the user is feeling stressed, it will recommend content that has a relaxing effect. In this step, emotional data is input and recommended content is output.

[0688] Step 11:

[0689] The server collects and analyzes user feedback. Based on the feedback, it identifies improvements to the system and updates the plugins and AI models. In this step, feedback data is input and an improved system is output.

[0690] Step 12:

[0691] For example, if the emotion recognition result indicates "fatigue" when a user wants to relax, relaxing music and landscape videos will be recommended. If the user is feeling energetic and lively, action movies and dance videos will be recommended. In this step, the emotional state is input and optimized content is output.

[0692] A concrete example of a prompt is:

[0693] "If a user is in a state where they want to relax, what are the best movies to recommend?"

[0694] "Provide effective music when the user's heart rate is high and they are feeling fatigued."

[0695] There is.

[0696] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0697] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0698] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0699] [Third embodiment]

[0700] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0701] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0702] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0703] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0704] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0705] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0706] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0707] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0708] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0709] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0710] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0711] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0712] The present invention is a system for registering various services in a plug-in format and managing and providing them in an integrated manner. Specific embodiments for carrying out the present invention and each processing step are described below in natural language.

[0713] Registering a service plugin

[0714] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, and notes, and provides a mechanism for incorporating these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[0715] Plugin installation and user authentication

[0716] The device installs the plug-in downloaded from the server, enabling the use of various services. The user accesses the plug-in service through the device and enters authentication information for authentication. If authentication is successful, the user can access all registered services.

[0717] Data collection and storage

[0718] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. This database stores the operation history for each user.

[0719] Data preprocessing and cleansing

[0720] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and shaped for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[0721] Fine-tuning generative artificial intelligence

[0722] The server uses the preprocessed data to fine-tune the generative artificial intelligence (AI) model. The saved operation data is used as training data for the AI ​​model, improving its performance. Once fine-tuned, the AI ​​model is sent to the device, where it is updated.

[0723] To provide services and improve user experience

[0724] The device uses a fine-tuned AI model to provide individually optimized services to users. When users compose messages, the AI ​​model provides assistance and suggestions based on the user's past usage history. This streamlines user operations and provides a seamless user experience.

[0725] Feedback and Service Improvement

[0726] The server receives and collects feedback from users. Users can post to the server any problems or requests for improvements they have experienced while using the service. The server analyzes the collected feedback and identifies areas for improvement in the service. The service is updated and improved, and a more advanced service is provided to users.

[0727] Specific examples

[0728] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to the server. The server uses this data to fine-tune the generative AI and learns the user's unique writing style and expressions. The updated model is then used by the AI ​​to make predictions and completions as the user composes messages. This improves the user's operational efficiency.

[0729] The above is a specific embodiment for carrying out the present invention. The present invention provides users with a more personalized and seamless service experience, thereby improving usability.

[0730] The processing flow will be explained below.

[0731] Step 1:

[0732] The server exposes an API for registering various services in the form of plugins. Once services such as calls, messages, and notes are registered as plugins, the server integrates them into the system.

[0733] Step 2:

[0734] The device will be able to download plugins from the server, and if a user wants to use a particular service, they will be required to install the corresponding plugin on their device.

[0735] Step 3:

[0736] Users access various services through plug-ins installed on their devices. When accessing the services, the user enters their authentication information and is authenticated by the server.

[0737] Step 4:

[0738] The device collects real-time data on user operations such as calls, messages, and notes, and stores this data as logs in the background.

[0739] Step 5:

[0740] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and pre-processed as necessary.

[0741] Step 6:

[0742] The server uses the stored data to fine-tune a generative artificial intelligence (AI) model, allowing the model to learn user interaction patterns and behavioral characteristics.

[0743] Step 7:

[0744] The server then distributes the fine-tuned artificial intelligence model to the device, which updates the AI ​​model on the device.

[0745] Step 8:

[0746] The device will utilize fine-tuned AI models to provide individually optimized services for users, such as in a note-taking app where AI predicts what you're typing and auto-completes the text, improving operational efficiency.

[0747] Step 9:

[0748] Users can provide feedback to the server about any problems or requests for improvements they may have while using the system. This feedback can be easily sent via the terminal.

[0749] Step 10:

[0750] The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements are determined, the server updates the plugins and AI models and distributes the new versions to devices.

[0751] This is the specific program processing flow, which allows users to enjoy a consistent and seamless service experience.

[0752] Example 1

[0753] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0754] In conventional systems, plugins for providing various services are managed separately, making it difficult to provide integrated services. Furthermore, to effectively utilize user operation data and provide individually optimized services, data collection, storage, preprocessing, analysis, and fine-tuning of generative AI models are required, but these processes are often not consistently integrated. This results in the challenge of making it difficult to improve the quality of the user experience.

[0755] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0756] In this invention, the server includes a means for allowing developers to register various services in the form of plug-ins, a means for registering plug-ins using an API provided by the server, and a means for installing the registered plug-ins on a terminal. This enables integrated management and provision of various services. The terminal also includes a means for entering user authentication information to access services, a means for collecting user operation data in real time, and a means for storing the collected operation data. The server further includes a means for preprocessing the stored data and removing noise and unnecessary data, a means for fine-tuning a generative AI model using the preprocessed data, a means for delivering the fine-tuned AI model to the terminal, a means for optimizing and providing services to users based on the fine-tuned AI model, and a means for collecting and analyzing user feedback to improve the service. This enables a personalized and seamless service experience based on user operation data, improving usability.

[0757] A "plug-in format" is a small program module that adds functionality to software.

[0758] "API" stands for Application Programming Interface, a means by which different software applications can communicate with each other.

[0759] A "terminal" is a hardware device (e.g., a smartphone or computer) that a user operates to access various services.

[0760] "Authentication information" refers to information such as ID and password used by users when accessing a system.

[0761] "Operation data" refers to data generated when a user operates the system (for example, call logs, message history, memo records, etc.).

[0762] "Real-time" means that data is processed immediately at the moment an action is taken.

[0763] "Preprocessing" refers to cleansing and shaping data before analyzing it.

[0764] "Noise" refers to unnecessary or meaningless data in data analysis.

[0765] A "generative artificial intelligence model" is an artificial intelligence algorithm that generates new data based on past data.

[0766] "Fine-tuning" is the process of further optimizing an existing artificial intelligence model using a specific dataset.

[0767] "Feedback" refers to opinions and information provided by users when they report improvements or problems with the system.

[0768] "Usability" is a concept that refers to how easy a system or service is to use.

[0769] This invention is a system that registers various services in a plug-in format and manages and provides them in an integrated manner. In this system, the server, terminal, and user each play different roles, realizing seamless and personalized services.

[0770] Registering a service plugin

[0771] The server provides an interface that allows developers to register various services in the form of plugins. This interface is implemented using the API provided by the server. Developers can register services such as calls, messages, and memos as plugins on the server. For example, if a developer wants to provide a new memo function as a plugin, they send this plugin to the server via the API. The server receives it and incorporates it into the system by storing it in a database.

[0772] Plugin installation and user authentication

[0773] The device installs the plug-in downloaded from the server. The user accesses the plug-in service through the device and authenticates by entering their authentication information (e.g., ID and password). If authentication is successful, the user will be able to access all registered services. For example, if a user installs a message service plug-in and authenticates, the message service will become available on that device.

[0774] Data collection and storage

[0775] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background and sent to the server periodically or in real time. The server receives the operational data sent from the device and stores it in a secure database. The database stores the operation history for each user.

[0776] Data preprocessing and cleansing

[0777] The server preprocesses the collected data to remove noise and unnecessary data. For example, meaningless spaces and duplicate data are removed from message content. The preprocessed data is then classified and formatted for analysis, and different types of data, such as call logs and message logs, are organized chronologically.

[0778] Fine-tuning generative AI models

[0779] The server uses the preprocessed data to fine-tune the generative AI model. The user's past operational data is used as training data for the AI ​​model. Through this process, the model learns the user's unique behavioral patterns and expression methods, improving its performance. Once fine-tuned, the artificial intelligence model is delivered to the device, updating the device's AI engine.

[0780] To provide services and improve user experience

[0781] The device uses a fine-tuned AI model to provide individually optimized services to users. For example, when a user composes a message, the AI ​​model suggests the most appropriate words and grammar based on past usage history. When a user begins to type "Hello," the AI ​​will provide predictive completion, such as "Hello, what are your plans for today?", supporting the user's operations.

[0782] Feedback and Service Improvement

[0783] The server collects and analyzes feedback from users. Users can post to the server via their devices any problems or requests for improvements they may have while using the service. For example, feedback such as "predictive text doesn't work well" is sent to the server. The server analyzes this information and collects information to improve the service. The service is then updated and improved, and the improved plug-in is distributed to the device again.

[0784] Examples of concrete examples and prompts

[0785] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to a server. The server uses this data to fine-tune a generative AI model, learning the user's unique writing style and expressions. The updated model is then delivered to the device, allowing the AI ​​to make predictions and completions as the user composes messages.

[0786] An example of a prompt sentence is "Generate the next message based on the user's past message history."

[0787] The above description clarifies specific embodiments for carrying out the present invention, which provides a user with a personalized and seamless service experience and improves usability.

[0788] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0789] Step 1: Registering the service plugin

[0790] The server provides an interface that allows developers to register various services in the form of plugins. As input, developers provide data for service plugins such as calls, messages, and notes. The server receives this data and stores it in the system's database. As output, the registered plugins are recorded as available in the system. For example, a developer can send a new note plugin to the server via API, and the server will receive it and store it in the database.

[0791] Step 2: Install the plugin

[0792] The terminal connects to the server and gets a list of required plugins. As input, it receives the plugin list information from the server. The terminal downloads and installs the required plugins based on the user's selection. As output, the installed plugins become available on the terminal. For example, the terminal connects to the server to download and install a new message plugin.

[0793] Step 3: User authentication

[0794] A user accesses a plugged-in service through a terminal and enters authentication information. As input, the user provides authentication information such as ID and password. The server receives the authentication information and compares it with the information in the database. If authentication is successful, the output is that the user can access all registered services. For example, when a user logs in to a messaging service, the terminal sends the ID and password to the server for authentication.

[0795] Step 4: Data collection

[0796] The device collects operation data for each service used by the user in real time. As input, it acquires user operation data (call history, message sending history, memo records, etc.). The device saves this data as a log in the background and sends it to the server. As output, the collection of operation data and its transfer to the server is completed. For example, when a user makes a call, the call history is recorded on the device and sent to the server.

[0797] Step 5: Save Data

[0798] The server receives operation data sent from the terminal and stores it in a secure database. As input, it receives operation data from the terminal. The server converts this data into an appropriate format and stores it in the database. As output, the operation history for each user is stored in the database. For example, the server records the message history received from the terminal in the database.

[0799] Step 6: Data preprocessing and cleansing

[0800] The server preprocesses the stored data to remove noise and unnecessary information. It uses the stored data in the database as input. The server cleanses the data, extracting and formatting only the necessary parts. As output, a clean dataset suitable for analysis is generated. For example, the server removes unnecessary whitespace and duplicate data from message data.

[0801] Step 7: Fine-tuning the generative AI model

[0802] The server uses the preprocessed data to fine-tune a generative AI model. It uses a clean, preprocessed dataset as input. The server trains the AI ​​model on this data to optimize performance. The output is a fine-tuned AI model. For example, the server trains an AI model that learns writing style and expressions based on a user's past messaging history.

[0803] Step 8: Deploy and update the AI ​​model

[0804] The fine-tuned AI model is then sent from the server to the device, updating the device's AI engine. The fine-tuned AI model on the server is used as input. The server then sends the AI ​​model to the device via the internet. As output, the AI ​​engine on the device is updated based on the latest AI model. For example, the server sends the latest generative AI model to the device, enhancing the device's AI capabilities.

[0805] Step 9: Improving service delivery and user experience

[0806] The device uses a fine-tuned AI model to provide optimized services to users. As input, it uses the updated AI model and real-time user operation data. The device makes suggestions and completions to assist the user's operations. As output, a seamless service individually optimized for the user is provided. For example, when a user composes a new message, the AI ​​suggests the most appropriate predictive text.

[0807] Step 10: Gather feedback

[0808] The server collects and analyzes feedback from users. As input, it receives feedback information provided by users while using the service. The server analyzes this and identifies areas for improvement. As output, improvements to the service are compiled and reflected in the next update. For example, a user may send feedback to the server that "predictive text conversion did not work properly," and the server will analyze the problem.

[0809] Step 11: Service Improvement

[0810] The server improves the service based on the feedback. It uses the collected feedback information and analysis results as input. The server improves the plugins and AI models based on this and distributes them as the next update. As output, the improved plugins and AI models are provided to users in the next release. For example, the server makes improvements to improve the accuracy of predictive text conversion and releases it as a new version.

[0811] Through the above steps, this system provides various services that are managed in an integrated manner, improving usability.

[0812] (Application example 1)

[0813] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0814] In conventional brick-and-mortar stores, product recommendations based on user behavior and past purchase history were not adequately provided, and improving the user experience was an issue. In addition, there was a lack of means to quickly and appropriately provide users with the information they needed, which often made shopping in brick-and-mortar stores cumbersome.

[0815] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0816] In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for fine-tuning a generative AI model using the saved data, means for delivering the fine-tuned AI model to the terminal, means for providing a service based on the fine-tuned AI model to the user, means for collecting feedback from the user and improving the service, means for the terminal to make product suggestions to assist the user in a physical store, and means for displaying the suggestions on a display device of the terminal. This makes it possible to make appropriate product suggestions based on the user's behavior and past purchase history, significantly improving the shopping experience in a physical store.

[0817] A "plug-in" is an additional module that extends the functionality of a particular piece of software.

[0818] "Terminal" refers to an electronic device used by a user, and examples include smartphones and smart glasses.

[0819] "User operation data" refers to information relating to operations performed by a user, including messages, calls, notes, actions taken within the store, and the like.

[0820] A "generative artificial intelligence model" is a machine learning model that generates new information and suggestions based on data.

[0821] "Fine-tuning" is the optimization process of adapting an existing generative artificial intelligence model to a specific dataset to improve its accuracy.

[0822] "Feedback" refers to information such as ratings and opinions provided by users, and is used to improve services.

[0823] "Product suggestions" refers to proposing appropriate products and services based on a user's past purchasing history and behavior.

[0824] "Display device" refers to a device or screen for visually displaying information, including, for example, the display of smart glasses.

[0825] 1. System Programming

[0826] To realize this application example, a series of programs are required to register various services in plug-in format, install them on the device, collect, save, and analyze user operation data, and fine-tune the generative AI model. The specific implementation method is explained below.

[0827] 2. Explanation of the generated program processing

[0828] The server provides an interface for developers to register various services in the form of plug-ins, allowing them to add new services. The registered plug-ins are then downloaded and installed on the device. The first application installed is the shopping assistant application, which includes a user authentication function.

[0829] The terminal uses hardware such as smart glasses or smartphones to collect user operation data in real time. This operation data includes call data, message data, memo data, and in-store behavior data. The collected data is sent to a backend server and stored in a secure database.

[0830] The server preprocesses the stored data and fine-tunes the generative AI model. For example, it uses Python data analysis libraries (such as NumPy and Pandas) to cleanse, classify, and shape the data. It then fine-tunes the data using a generative AI model (such as TensorFlow or PyTorch). This generates a model customized for each user.

[0831] The fine-tuned AI model is then sent back to the device, where personalized product suggestions are made to the user. Specifically, products based on the user's past purchase history and preferred styles are displayed on the smart glasses' display. Suggestions such as "This jacket is similar to a style you liked in the past. Would you like to try it on?" are displayed, improving the user's shopping experience.

[0832] 3. Examples and prompts

[0833] For example, if a user is in a clothing store in a shopping mall, the smart glasses can suggest new products based on the user's past purchases and preferred styles. The user can look at the display and receive suggestions such as, "This jacket is similar to a style you liked in the past. Would you like to try it on?"

[0834] An example prompt for a generative AI model might look something like this:

[0835] Generate new product suggestions based on the user's past purchases, and take into account the user's current location to make the most of your in-store product data.

[0836] In this way, the present invention significantly improves the shopping experience in physical stores by suggesting products based on user behavior data and past purchase history.

[0837] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0838] Step 1: Register the plugin

[0839] The server provides an interface for registering various services in the form of plugins. The input includes information about the various services registered by the developer. The server saves the entered service information in a database and makes it manageable as a plugin. This allows the various services to be registered on the server and ready to be distributed to devices.

[0840] Step 2: Install the plugin

[0841] The device downloads the specified plugin from the server and installs it on the system. The input includes plugin data sent from the server. The device receives this data, stores it in its file system, and the registered service becomes available.

[0842] Step 3: User authentication

[0843] The user uses the terminal to input authentication information to access various services. The input includes the user's authentication information (e.g., user ID, password). The terminal sends this authentication information to the server, which then verifies it against a database. If authentication is successful, the terminal becomes able to access all registered services.

[0844] Step 4: Collecting operational data

[0845] The terminal collects user operation data (calls, messages, notes, and in-store actions) in real time. The input includes the user's operation behavior. The terminal converts and prepares this data in the required format to send to the backend server.

[0846] Step 5: Save Data

[0847] The server stores the operation data sent from the terminal in a secure database. The input includes the operation data sent from the terminal. The server stores this data in a database and manages the operation history for each user.

[0848] Step 6: Preprocessing the data

[0849] The server preprocesses the stored operational data to remove noise and unnecessary data. The input includes raw data stored in a database. The server performs data cleansing using data analysis libraries (e.g., NumPy, Pandas) to generate a clean dataset.

[0850] Step 7: Fine-tuning the generative AI model

[0851] The server uses the preprocessed data to fine-tune a generative artificial intelligence model. The input includes a clean dataset. The server optimizes the AI ​​model using machine learning libraries (e.g., TensorFlow, PyTorch) and generates a customized model for each user.

[0852] Step 8: Serving the Model

[0853] The server delivers the fine-tuned generative artificial intelligence model to the device, which includes the optimized AI model as input. The server sends the model to the device, which then integrates the model into its system.

[0854] Step 9: Service Delivery

[0855] The device uses a fine-tuned generative artificial intelligence model to provide personalized product recommendations to users. Inputs include the optimized AI model and user operation data. Specifically, the device displays product recommendations based on past purchase history and behavior on the smart glasses display.

[0856] Step 10: Gather feedback

[0857] Users can input feedback about problems they encounter while using the service or requests for improvements. This feedback includes user ratings and opinions. The device sends this feedback to the server, which then analyzes it as data for improving the service.

[0858] Through these steps, the present invention improves the overall shopping experience in brick-and-mortar stores.

[0859] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0860] The present invention is a system that provides individually optimized services by combining a system for registering various services in a plug-in format and managing and providing them in an integrated manner with an emotion engine that recognizes the user's emotions. Specific embodiments for implementing the present invention and each processing step are described below in natural language.

[0861] Registering a service plugin

[0862] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, and notes, and provides a mechanism for integrating these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[0863] Plugin installation and user authentication

[0864] The device will be able to download plugins from the server, and if a user wants to use a particular service, they will be required to install the corresponding plugin on their device.

[0865] Users access various services through the plug-in installed on their devices and enter their authentication information for authentication. Once authentication is successful, users can access all registered services.

[0866] Data collection and storage

[0867] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. The saved data is classified by user and pre-processed as needed.

[0868] Data preprocessing and cleansing

[0869] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and formatted for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[0870] Introducing the Emotion Engine

[0871] The server recognizes the user's emotions based on the stored operation data using an emotion engine, which extracts emotion data from the content of the user's messages and calls using text and voice analysis.

[0872] Fine-tuning generative artificial intelligence

[0873] The server uses the preprocessed operation data and emotion data to fine-tune the generative artificial intelligence (AI) model. By incorporating the emotion data, the AI ​​model learns individual needs and tendencies based on the user's emotions. Once the fine-tuned AI model is complete, it is distributed to the device, where the AI ​​model is updated.

[0874] To provide services and improve user experience

[0875] The device will use fine-tuned AI models to provide individually optimized services to users. For example, in a memo app, AI will provide assistance and suggestions based on the user's emotions, improving operational efficiency.

[0876] Feedback and Service Improvement

[0877] The server receives and collects feedback from users. Users can post any problems or requests for improvements they have noticed while using the service to the server. The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements have been decided, the server updates the plugins and AI models and distributes the new versions to devices.

[0878] Specific examples

[0879] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to the server. The server uses this data and an emotion engine to recognize the user's emotional state and fine-tune the generative AI. The updated model delivered to the device is then used by the AI ​​to make predictions and completions when the user composes a message. Taking the emotion data into account, the AI ​​can make optimal suggestions based on the user's emotional state. This increases user efficiency and satisfaction.

[0880] The above is a specific embodiment for carrying out the present invention. The present invention provides users with a more personalized and seamless service experience, thereby improving usability.

[0881] The processing flow will be explained below.

[0882] Step 1:

[0883] The server exposes an API for registering various services in the form of plugins. Once services such as calls, messages, and notes are registered as plugins, the server integrates them into the system.

[0884] Step 2:

[0885] The device downloads and installs the necessary plugins from the server. When a user wants to use a particular service, they are required to install the corresponding plugin for that service on their device.

[0886] Step 3:

[0887] Users access various services through a plug-in installed on their device and enter their authentication information for authentication. Once authentication is successful, the user can access all registered services.

[0888] Step 4:

[0889] The device collects real-time data on user operations such as calls, messages, and notes, and stores this data as logs in the background.

[0890] Step 5:

[0891] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and pre-processed as necessary.

[0892] Step 6:

[0893] The server preprocesses the stored operational data to remove noise and unnecessary data, resulting in a clean dataset.

[0894] Step 7:

[0895] The server recognizes the user's emotions based on the pre-processed data using an emotion engine, which uses text analysis of messages and audio analysis of calls to identify the user's emotional state.

[0896] Step 8:

[0897] The server uses the interaction data and recognized emotion data to fine-tune a generative artificial intelligence (AI) model, enabling advanced pattern recognition that takes the user's emotional state into account.

[0898] Step 9:

[0899] The server then distributes the fine-tuned AI model to the device, which updates the AI ​​model on the device to reflect the latest user information.

[0900] Step 10:

[0901] The device will utilize fine-tuned AI models to provide optimized services to users, such as messaging services, where the AI ​​will provide assistance and suggestions based on the user's emotions.

[0902] Step 11:

[0903] Users can provide feedback to the server about any problems or requests for improvements they may have while using the system. This feedback can be easily sent via the terminal.

[0904] Step 12:

[0905] The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements are determined, the server updates the plugins and AI models and distributes the new versions to devices.

[0906] Specific examples

[0907] For example, when a user uses a messaging service, the device collects the content of the message sent by the user and their emotional state at the time of sending it. The server uses this data and an emotion engine to recognize the user's emotional state and fine-tune the AI ​​model. Next, using the updated model delivered to the device, the AI ​​makes emotion-based predictions and completions when composing messages. For example, if the AI ​​recognizes that the user is feeling angry, it can assist in smoother communication by suggesting a calmer message. This results in improved user efficiency and satisfaction.

[0908] This concludes the detailed explanation of each processing step. This system enables us to provide users with a seamless and personalized service experience.

[0909] Example 2

[0910] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0911] In today's diverse service delivery systems, there is a demand for services that are individually optimized for each user, but conventional systems require individual settings for each service, making it difficult to efficiently manage data and optimize individual services based on emotion recognition. Therefore, a system is needed that can manage a large number of different services in an integrated manner and provide optimized services that also respond to the user's emotional state.

[0912] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0913] In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for preprocessing the saved data and removing noise and unnecessary data, means for recognizing a user's emotion with an emotion engine using the preprocessed data, means for fine-tuning a generative AI model using the recognized emotion data, means for delivering the fine-tuned AI model to the terminal, service providing means for providing a service to the user based on the fine-tuned AI model, and means for collecting feedback from the user and improving the service. This makes it possible to manage a variety of services in an integrated manner and provide optimized services that also correspond to the emotional state of the user.

[0914] "Various services" refers to different functions and offerings that users can use, such as calls, messages, and notes.

[0915] The "plug-in format" refers to a format in which various services are modularized and can be easily added or removed from the system.

[0916] "Terminal" refers to a computing device, such as a smartphone, tablet, or PC, that a user uses to access various services.

[0917] "Operational Data" refers to information generated or collected when a user uses various services, such as call records, message sending history, and memo input data.

[0918] "Preprocessing" refers to the process of removing noise and unnecessary data from stored data and preparing it in a form that is easy to analyze.

[0919] "Noise" refers to unnecessary information that is not needed in the data processing process and that may adversely affect the results of the analysis.

[0920] "Emotion Engine" refers to an algorithm or software that uses text and speech analysis techniques to recognize and extract a user's emotional state.

[0921] A "generative artificial intelligence model" refers to a machine learning model that can learn large amounts of data and generate new content, such as GPT-3 or BERT.

[0922] "Fine-tuning" refers to the process of adjusting an existing machine learning model to optimize its performance for a specific dataset or application.

[0923] "Feedback" refers to opinions and reports that users provide to the server regarding problems or requests for improvement they have experienced while using the service.

[0924] A "secure database" refers to a data management system in which stored data is protected from unauthorized access and confidentiality, integrity, and availability are ensured.

[0925] MODE FOR CARRYING OUT THE INVENTION

[0926] The present invention is a system that provides individually optimized services by combining a system that registers various services in a plug-in format and manages and provides them in an integrated manner with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described in detail below.

[0927] Registering a service plugin

[0928] The server provides an API interface that allows developers to register various services in the form of plugins. If a developer wants to provide new functions, such as a call recording service, messaging service, or memo service, they can register them as plugins. Specifically, the plugins are uploaded using the API provided by the server and integrated into the entire system.

[0929] Plugin installation and user authentication

[0930] The device has the ability to download and install plug-ins from the server. When a user wants to use a particular service, such as a call recording service, the plug-in is installed on the device. The user then accesses the service through the installed plug-in on the device and is authenticated by entering authentication information, such as a username and password. If the authentication is successful, the user can access all registered services.

[0931] Data collection and storage

[0932] The device collects real-time operational data, such as calls, messages, and notes, used by the user. This operational data is saved as a log file on the device and periodically sent to the server. The server stores this data in a secure database, where it is categorized by user.

[0933] Data preprocessing and cleansing

[0934] The server preprocesses the stored operational data to remove noise and unnecessary data. Noise refers to unnecessary information that is not needed in the data processing process and may adversely affect the analysis results. This preprocessing generates a clean dataset. The preprocessed data is organized into time-series data, making it easier to analyze.

[0935] Introducing the Emotion Engine

[0936] The server uses the cleansed operation data to recognize the user's emotions with an emotion engine. This emotion engine uses text analysis (e.g., natural language processing technology) and voice analysis (e.g., voice recognition technology) to extract emotion data from the user's messages and call content. For example, emotions such as "tired" or "happy" can be detected from messages sent by the user.

[0937] Fine-tuning generative artificial intelligence

[0938] The server uses the preprocessed operation data and emotion data to fine-tune a generative artificial intelligence (AI) model. For example, it uses deep learning models such as GPT-3 and BERT to improve the model's ability to respond and make suggestions based on the user's emotional changes. The fine-tuned AI model is then distributed to the device, where it can be used.

[0939] To provide services and improve user experience

[0940] The device uses fine-tuned generative AI models to provide individually optimized services to users. For example, in a notes app, when a user types "I'm tired," it automatically suggests "Make time to take a break." This increases user efficiency and satisfaction.

[0941] Prompt Sentence Examples

[0942] Based on the following conversation history, we present examples of prompt sentences that recognize the user's emotions and provide appropriate suggestions.

[0943] "Based on the following conversation history, please recognize the user's emotions and provide appropriate suggestions.

[0944] Conversation History:

[0945] User: 'I'm extremely busy and stressed today'

[0946] Generative AI model: 'Sounds like a lot of work. Good luck.'

[0947] Also, please complete the following message:

[0948] By inputting this prompt into a generative AI model, it is possible to understand the user's emotions and provide the most appropriate message for the situation.

[0949] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0950] Step 1: Registering the service plugin

[0951] The server provides an API interface that allows developers to register various services in the form of plugins. Specifically, developers create new services as plugins and send the plugin's metadata and files to the server. The inputs are the plugin specifications and executable files, and the output is that the plugin is registered in the system. This makes services such as call logs, messages, and notes available in the system.

[0952] Step 2: Installing the plugin and authenticating the user

[0953] The device has the function of downloading and installing registered plugins from the server. When a user uses a specific service, the device allows access to the service via the installed plugin. During this process, the plugin file from the server and the user's authentication information (username, password) are used as input, and the service becomes available as output. If authentication is successful, the user will be able to access the corresponding service.

[0954] Step 3: Data collection and storage

[0955] The device collects operational data such as calls, messages, and notes used by the user in real time. This data is stored in the device as a log file and periodically sent to the server. The input is user operation data (call records, message contents, note contents), and the output is the saved log file and sent data. The server saves the received operation data in a secure database. The saved data is categorized by user.

[0956] Step 4: Preprocessing and cleansing the data

[0957] The server preprocesses the stored operational data, specifically removing noise, filling in missing data, and detecting outliers. The input is the stored operational data, and the output is a clean dataset, improving the quality of the data and making it easier to analyze.

[0958] Step 5: Implementing the Emotion Engine

[0959] The server uses the preprocessed operation data to run an emotion engine. This emotion engine uses text analysis and speech analysis techniques to recognize the user's emotional state. The preprocessed operation data (message content, call content) is used as input, and emotion data is generated as output. For example, emotions such as "I'm stressed" can be detected through text analysis.

[0960] Step 6: Fine-tuning the generative AI

[0961] The server uses the preprocessed interaction data and emotion data to fine-tune a generative artificial intelligence (AI) model. Specifically, it applies it to a generative AI model (e.g., GPT-3) to improve its ability to generate responses based on user emotions. The clean interaction data and emotion data are input, and a fine-tuned AI model is generated as output. This model learns optimal responses based on individual needs.

[0962] Step 7: Providing services and improving user experience

[0963] The device uses a fine-tuned generative AI model to provide optimal services to users. For example, if a user uses a memo app and enters "I'm tired," the AI ​​generates a suggestion such as "I recommend you take a rest." The inputs include user operations and emotional data, and the output is the AI's optimal suggestion. This improves user efficiency and satisfaction.

[0964] Step 8: Feedback and Service Improvement

[0965] The server collects feedback from users and uses it to improve the service. Users can post any problems or requests for improvements they have noticed while using the service to the server. The input is user feedback, and the output is analyzed improvements. The server analyzes the feedback, updates the plugins and AI models, and delivers the improved versions to the devices. This allows for continuous improvement of the service.

[0966] (Application example 2)

[0967] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0968] Current content delivery systems recommend content uniformly without considering the user's emotional state, resulting in a decrease in user satisfaction and experience value. Specifically, they may be unable to recommend relaxing content to a stressed user, while recommending inappropriate content to a user in an energetic state. A system that can solve these problems and realize optimal content recommendations based on the user's emotions is needed.

[0969] The specification processing by specification processing unit 290 of data processing device 12 in application example 2 is realized by the following means. In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for fine-tuning the generative AI model using the saved data, means for delivering the fine-tuned AI model to the terminal, means for generating emotion data using an emotion engine that recognizes the user's emotions, means for recommending optimal content to the user based on the generated emotion data, service providing means for providing the user with a service based on the fine-tuned generative AI model, and means for collecting user feedback and improving the service. This makes it possible to recommend optimal content according to the user's emotional state.

[0970] "Various services" are abstract means that provide functions and actions such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis that users can use.

[0971] The "plug-in format" is a method of modularizing various services and providing them in a form that can be easily installed and updated.

[0972] "Terminal" refers to a device used by a user, including smartphones, tablets, head-mounted displays, etc.

[0973] "User operation data" refers to call records, message history, memo contents, heart rate, facial expressions, and voice data generated when a user operates a terminal.

[0974] "Stored data" refers to the state in which collected user operation data is stored in a secure database.

[0975] A "generative artificial intelligence model" is a machine learning model that learns from collected data and generates useful information and services for users.

[0976] "Fine-tuning" is the act of fine-tuning an existing generative artificial intelligence model to optimize its performance for a specific dataset or purpose.

[0977] An "emotion engine" is a software component that analyzes user operation data and recognizes and extracts their emotional state.

[0978] "Emotion data" is a data set that is analyzed using an emotion engine and indicates the user's emotional state.

[0979] "Content" refers to media files that can be viewed or listened to, and is an information resource that includes movies, music videos, documentaries, etc.

[0980] "Recommendation" is the act of the system selecting and providing content or services that it deems most suitable based on the user's data and emotional state.

[0981] "Feedback" refers to information that indicates evaluations, impressions, and requests for improvement provided by users regarding the system.

[0982] This invention is a system that provides individually optimized content by combining a system that registers various services in a plug-in format and manages and provides them in an integrated manner with an emotion engine that recognizes the user's emotions.

[0983] Registering a service plugin

[0984] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis, and integrates these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[0985] Plugin installation and user authentication

[0986] The device downloads and installs the plug-in from the server. When a user wants to use a specific service, the device is required to install the plug-in corresponding to that service. The user accesses various services through the plug-in installed on the device and is authenticated by entering authentication information. If authentication is successful, the user can access all registered services.

[0987] Data collection and storage

[0988] The device collects real-time operational data, such as user calls, messages, notes, heart rate measurements, facial expressions, and voice analysis. The collected operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. The saved data is classified by user and preprocessed as necessary.

[0989] Data preprocessing and cleansing

[0990] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and shaped for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[0991] Introducing the Emotion Engine

[0992] The server recognizes the user's emotions based on the stored operation data using an emotion engine, which uses text analysis and voice analysis to extract emotion data from the user's messages, call content, heart rate, and facial expressions.

[0993] Fine-tuning generative artificial intelligence

[0994] The server uses the preprocessed operation data and emotion data to fine-tune the generative artificial intelligence (AI) model. By incorporating the emotion data, the AI ​​model learns individual needs and tendencies based on the user's emotions. Once the fine-tuned AI model is complete, it is distributed to the device, where the AI ​​model is updated.

[0995] To provide services and improve user experience

[0996] The device uses a fine-tuned AI model to recommend content that is individually optimized for each user. For example, if a user using a head-mounted display is recognized as feeling stressed, movies and music with a relaxing effect will be recommended. A user in an energetic state will be recommended action movies and dance videos. In this way, personalized content is provided according to the user's current emotional state.

[0997] Feedback and Service Improvement

[0998] The server receives and collects feedback from users. Users can post any problems or requests for improvements they have noticed while using the service to the server. The server analyzes the collected feedback and identifies areas for improvement in the service. Once the improvements have been decided, the server updates the plugins and AI models and distributes the new versions to devices.

[0999] For example, a user may want to relax late at night and put on a head-mounted display. The system can recognize the user's heart rate and facial expression as indicating fatigue or a desire for relaxation, and recommend relaxing music or landscape videos. If the user feels energetic and energetic, action movies or dance videos may be recommended.

[1000] An example of a prompt sentence to be input to a generative artificial intelligence model is:

[1001] "If a user is in a state where they want to relax, what are the best movies to recommend?"

[1002] "Provide effective music when the user's heart rate is high and they are feeling fatigued."

[1003] There is.

[1004] The above is a specific embodiment for carrying out the present invention. This system makes it possible to provide users with a more personalized and seamless service experience, thereby improving usability.

[1005] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1006] Step 1:

[1007] The server provides an interface for registering various services in the form of plugins. Developers use the API to register services such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis. In this step, service data is received as input and output is obtained for each plugin to be integrated into the server.

[1008] Step 2:

[1009] The device downloads and installs the plug-in from the server. The user installs the plug-in to use a specific service through the device. In this step, an installation request is input and the device outputs the state in which the plug-in has been installed.

[1010] Step 3:

[1011] The user accesses various services through the plug-in installed on the terminal and enters authentication information to be authenticated. If authentication is successful, the user can access all registered services. In this step, authentication information is entered and the result of authentication success / failure is output.

[1012] Step 4:

[1013] The device collects user operation data in real time, including call records, message history, memo contents, heart rate data, facial expression data, and voice data. In this step, operation data is input and the collected data log is output.

[1014] Step 5:

[1015] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and preprocessed as needed. In this step, operation data is input and a preprocessed clean dataset is output.

[1016] Step 6:

[1017] The server preprocesses the stored data to remove noise and unnecessary data. This process generates a clean dataset and organizes data in different formats in chronological order. In this step, the data to be preprocessed is input and a cleaned dataset is output.

[1018] Step 7:

[1019] The server uses an emotion engine to recognize the user's emotions based on the stored operation data. It extracts emotion data from the content of the user's messages and calls using text analysis and voice analysis. In this step, operation data is input and emotion data is output.

[1020] Step 8:

[1021] The server uses the collected and preprocessed operation data and emotion data to fine-tune the generative AI model, thereby learning the user's individual needs and tendencies. In this step, the preprocessed data and emotion data are input, and a fine-tuned AI model is output.

[1022] Step 9:

[1023] The server delivers the fine-tuned generative AI model to the device, which then uses the updated AI model to provide personalized content to the user. In this step, an AI model is input and a fine-tuned AI model is output that is delivered to the device.

[1024] Step 10:

[1025] The device uses a fine-tuned AI model to recommend content that is individually optimized for the user. For example, if the system recognizes that the user is feeling stressed, it will recommend content that has a relaxing effect. In this step, emotional data is input and recommended content is output.

[1026] Step 11:

[1027] The server collects and analyzes user feedback. Based on the feedback, it identifies improvements to the system and updates the plugins and AI models. In this step, feedback data is input and an improved system is output.

[1028] Step 12:

[1029] For example, if the emotion recognition result indicates "fatigue" when a user wants to relax, relaxing music and landscape videos will be recommended. If the user is feeling energetic and lively, action movies and dance videos will be recommended. In this step, the emotional state is input and optimized content is output.

[1030] A concrete example of a prompt is:

[1031] "If a user is in a state where they want to relax, what are the best movies to recommend?"

[1032] "Provide effective music when the user's heart rate is high and they are feeling fatigued."

[1033] There is.

[1034] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1035] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1036] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1037] [Fourth embodiment]

[1038] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1039] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1040] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1041] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1042] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1043] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1044] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1045] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1046] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1047] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1048] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1049] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1050] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1051] The present invention is a system for registering various services in a plug-in format and managing and providing them in an integrated manner. Specific embodiments for carrying out the present invention and each processing step are described below in natural language.

[1052] Registering a service plugin

[1053] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, and notes, and provides a mechanism for incorporating these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[1054] Plugin installation and user authentication

[1055] The device installs the plug-in downloaded from the server, enabling the use of various services. The user accesses the plug-in service through the device and enters authentication information for authentication. If authentication is successful, the user can access all registered services.

[1056] Data collection and storage

[1057] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. This database stores the operation history for each user.

[1058] Data preprocessing and cleansing

[1059] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and shaped for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[1060] Fine-tuning generative artificial intelligence

[1061] The server uses the preprocessed data to fine-tune the generative artificial intelligence (AI) model. The saved operation data is used as training data for the AI ​​model, improving its performance. Once fine-tuned, the AI ​​model is sent to the device, where it is updated.

[1062] To provide services and improve user experience

[1063] The device uses a fine-tuned AI model to provide individually optimized services to users. When users compose messages, the AI ​​model provides assistance and suggestions based on the user's past usage history. This streamlines user operations and provides a seamless user experience.

[1064] Feedback and Service Improvement

[1065] The server receives and collects feedback from users. Users can post to the server any problems or requests for improvements they have experienced while using the service. The server analyzes the collected feedback and identifies areas for improvement in the service. The service is updated and improved, and a more advanced service is provided to users.

[1066] Specific examples

[1067] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to the server. The server uses this data to fine-tune the generative AI and learns the user's unique writing style and expressions. The updated model is then used by the AI ​​to make predictions and completions as the user composes messages. This improves the user's operational efficiency.

[1068] The above is a specific embodiment for carrying out the present invention. The present invention provides users with a more personalized and seamless service experience, thereby improving usability.

[1069] The processing flow will be explained below.

[1070] Step 1:

[1071] The server exposes an API for registering various services in the form of plugins. Once services such as calls, messages, and notes are registered as plugins, the server integrates them into the system.

[1072] Step 2:

[1073] The device will be able to download plugins from the server, and if a user wants to use a particular service, they will be required to install the corresponding plugin on their device.

[1074] Step 3:

[1075] Users access various services through plug-ins installed on their devices. When accessing the services, the user enters their authentication information and is authenticated by the server.

[1076] Step 4:

[1077] The device collects real-time data on user operations such as calls, messages, and notes, and stores this data as logs in the background.

[1078] Step 5:

[1079] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and pre-processed as necessary.

[1080] Step 6:

[1081] The server uses the stored data to fine-tune a generative artificial intelligence (AI) model, allowing the model to learn user interaction patterns and behavioral characteristics.

[1082] Step 7:

[1083] The server then distributes the fine-tuned artificial intelligence model to the device, which updates the AI ​​model on the device.

[1084] Step 8:

[1085] The device will utilize fine-tuned AI models to provide individually optimized services for users, such as in a note-taking app where AI predicts what you're typing and auto-completes the text, improving operational efficiency.

[1086] Step 9:

[1087] Users can provide feedback to the server about any problems or requests for improvements they may have while using the system. This feedback can be easily sent via the terminal.

[1088] Step 10:

[1089] The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements are determined, the server updates the plugins and AI models and distributes the new versions to devices.

[1090] This is the specific program processing flow, which allows users to enjoy a consistent and seamless service experience.

[1091] Example 1

[1092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1093] In conventional systems, plugins for providing various services are managed separately, making it difficult to provide integrated services. Furthermore, to effectively utilize user operation data and provide individually optimized services, data collection, storage, preprocessing, analysis, and fine-tuning of generative AI models are required, but these processes are often not consistently integrated. This results in the challenge of making it difficult to improve the quality of the user experience.

[1094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1095] In this invention, the server includes a means for allowing developers to register various services in the form of plug-ins, a means for registering plug-ins using an API provided by the server, and a means for installing the registered plug-ins on a terminal. This enables integrated management and provision of various services. The terminal also includes a means for entering user authentication information to access services, a means for collecting user operation data in real time, and a means for storing the collected operation data. The server further includes a means for preprocessing the stored data and removing noise and unnecessary data, a means for fine-tuning a generative AI model using the preprocessed data, a means for delivering the fine-tuned AI model to the terminal, a means for optimizing and providing services to users based on the fine-tuned AI model, and a means for collecting and analyzing user feedback to improve the service. This enables a personalized and seamless service experience based on user operation data, improving usability.

[1096] A "plug-in format" is a small program module that adds functionality to software.

[1097] "API" stands for Application Programming Interface, a means by which different software applications can communicate with each other.

[1098] A "terminal" is a hardware device (e.g., a smartphone or computer) that a user operates to access various services.

[1099] "Authentication information" refers to information such as ID and password used by users when accessing a system.

[1100] "Operation data" refers to data generated when a user operates the system (for example, call logs, message history, memo records, etc.).

[1101] "Real-time" means that data is processed immediately at the moment an action is taken.

[1102] "Preprocessing" refers to cleansing and shaping data before analyzing it.

[1103] "Noise" refers to unnecessary or meaningless data in data analysis.

[1104] A "generative artificial intelligence model" is an artificial intelligence algorithm that generates new data based on past data.

[1105] "Fine-tuning" is the process of further optimizing an existing artificial intelligence model using a specific dataset.

[1106] "Feedback" refers to opinions and information provided by users when they report improvements or problems with the system.

[1107] "Usability" is a concept that refers to how easy a system or service is to use.

[1108] This invention is a system that registers various services in a plug-in format and manages and provides them in an integrated manner. In this system, the server, terminal, and user each play different roles, realizing seamless and personalized services.

[1109] Registering a service plugin

[1110] The server provides an interface that allows developers to register various services in the form of plugins. This interface is implemented using the API provided by the server. Developers can register services such as calls, messages, and memos as plugins on the server. For example, if a developer wants to provide a new memo function as a plugin, they send this plugin to the server via the API. The server receives it and incorporates it into the system by storing it in a database.

[1111] Plugin installation and user authentication

[1112] The device installs the plug-in downloaded from the server. The user accesses the plug-in service through the device and authenticates by entering their authentication information (e.g., ID and password). If authentication is successful, the user will be able to access all registered services. For example, if a user installs a message service plug-in and authenticates, the message service will become available on that device.

[1113] Data collection and storage

[1114] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background and sent to the server periodically or in real time. The server receives the operational data sent from the device and stores it in a secure database. The database stores the operation history for each user.

[1115] Data preprocessing and cleansing

[1116] The server preprocesses the collected data to remove noise and unnecessary data. For example, meaningless spaces and duplicate data are removed from message content. The preprocessed data is then classified and formatted for analysis, and different types of data, such as call logs and message logs, are organized chronologically.

[1117] Fine-tuning generative AI models

[1118] The server uses the preprocessed data to fine-tune the generative AI model. The user's past operational data is used as training data for the AI ​​model. Through this process, the model learns the user's unique behavioral patterns and expression methods, improving its performance. Once fine-tuned, the artificial intelligence model is delivered to the device, updating the device's AI engine.

[1119] To provide services and improve user experience

[1120] The device uses a fine-tuned AI model to provide individually optimized services to users. For example, when a user composes a message, the AI ​​model suggests the most appropriate words and grammar based on past usage history. When a user begins to type "Hello," the AI ​​will provide predictive completion, such as "Hello, what are your plans for today?", supporting the user's operations.

[1121] Feedback and Service Improvement

[1122] The server collects and analyzes feedback from users. Users can post to the server via their devices any problems or requests for improvements they may have while using the service. For example, feedback such as "predictive text doesn't work well" is sent to the server. The server analyzes this information and collects information to improve the service. The service is then updated and improved, and the improved plug-in is distributed to the device again.

[1123] Examples of concrete examples and prompts

[1124] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to a server. The server uses this data to fine-tune a generative AI model, learning the user's unique writing style and expressions. The updated model is then delivered to the device, allowing the AI ​​to make predictions and completions as the user composes messages.

[1125] An example of a prompt sentence is "Generate the next message based on the user's past message history."

[1126] The above description clarifies specific embodiments for carrying out the present invention, which provides a user with a personalized and seamless service experience and improves usability.

[1127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1128] Step 1: Registering the service plugin

[1129] The server provides an interface that allows developers to register various services in the form of plugins. As input, developers provide data for service plugins such as calls, messages, and notes. The server receives this data and stores it in the system's database. As output, the registered plugins are recorded as available in the system. For example, a developer can send a new note plugin to the server via API, and the server will receive it and store it in the database.

[1130] Step 2: Install the plugin

[1131] The terminal connects to the server and gets a list of required plugins. As input, it receives the plugin list information from the server. The terminal downloads and installs the required plugins based on the user's selection. As output, the installed plugins become available on the terminal. For example, the terminal connects to the server to download and install a new message plugin.

[1132] Step 3: User authentication

[1133] A user accesses a plugged-in service through a terminal and enters authentication information. As input, the user provides authentication information such as ID and password. The server receives the authentication information and compares it with the information in the database. If authentication is successful, the output is that the user can access all registered services. For example, when a user logs in to a messaging service, the terminal sends the ID and password to the server for authentication.

[1134] Step 4: Data collection

[1135] The device collects operation data for each service used by the user in real time. As input, it acquires user operation data (call history, message sending history, memo records, etc.). The device saves this data as a log in the background and sends it to the server. As output, the collection of operation data and its transfer to the server is completed. For example, when a user makes a call, the call history is recorded on the device and sent to the server.

[1136] Step 5: Save Data

[1137] The server receives operation data sent from the terminal and stores it in a secure database. As input, it receives operation data from the terminal. The server converts this data into an appropriate format and stores it in the database. As output, the operation history for each user is stored in the database. For example, the server records the message history received from the terminal in the database.

[1138] Step 6: Data preprocessing and cleansing

[1139] The server preprocesses the stored data to remove noise and unnecessary information. It uses the stored data in the database as input. The server cleanses the data, extracting and formatting only the necessary parts. As output, a clean dataset suitable for analysis is generated. For example, the server removes unnecessary whitespace and duplicate data from message data.

[1140] Step 7: Fine-tuning the generative AI model

[1141] The server uses the preprocessed data to fine-tune a generative AI model. It uses a clean, preprocessed dataset as input. The server trains the AI ​​model on this data to optimize performance. The output is a fine-tuned AI model. For example, the server trains an AI model that learns writing style and expressions based on a user's past messaging history.

[1142] Step 8: Deploy and update the AI ​​model

[1143] The fine-tuned AI model is then sent from the server to the device, updating the device's AI engine. The fine-tuned AI model on the server is used as input. The server then sends the AI ​​model to the device via the internet. As output, the AI ​​engine on the device is updated based on the latest AI model. For example, the server sends the latest generative AI model to the device, enhancing the device's AI capabilities.

[1144] Step 9: Improving service delivery and user experience

[1145] The device uses a fine-tuned AI model to provide optimized services to users. As input, it uses the updated AI model and real-time user operation data. The device makes suggestions and completions to assist the user's operations. As output, a seamless service individually optimized for the user is provided. For example, when a user composes a new message, the AI ​​suggests the most appropriate predictive text.

[1146] Step 10: Gather feedback

[1147] The server collects and analyzes feedback from users. As input, it receives feedback information provided by users while using the service. The server analyzes this and identifies areas for improvement. As output, improvements to the service are compiled and reflected in the next update. For example, a user may send feedback to the server that "predictive text conversion did not work properly," and the server will analyze the problem.

[1148] Step 11: Service Improvement

[1149] The server improves the service based on the feedback. It uses the collected feedback information and analysis results as input. The server improves the plugins and AI models based on this and distributes them as the next update. As output, the improved plugins and AI models are provided to users in the next release. For example, the server makes improvements to improve the accuracy of predictive text conversion and releases it as a new version.

[1150] Through the above steps, this system provides various services that are managed in an integrated manner, improving usability.

[1151] (Application example 1)

[1152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1153] In conventional brick-and-mortar stores, product recommendations based on user behavior and past purchase history were not adequately provided, and improving the user experience was an issue. In addition, there was a lack of means to quickly and appropriately provide users with the information they needed, which often made shopping in brick-and-mortar stores cumbersome.

[1154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1155] In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for fine-tuning a generative AI model using the saved data, means for delivering the fine-tuned AI model to the terminal, means for providing a service based on the fine-tuned AI model to the user, means for collecting feedback from the user and improving the service, means for the terminal to make product suggestions to assist the user in a physical store, and means for displaying the suggestions on a display device of the terminal. This makes it possible to make appropriate product suggestions based on the user's behavior and past purchase history, significantly improving the shopping experience in a physical store.

[1156] A "plug-in" is an additional module that extends the functionality of a particular piece of software.

[1157] "Terminal" refers to an electronic device used by a user, and examples include smartphones and smart glasses.

[1158] "User operation data" refers to information relating to operations performed by a user, including messages, calls, notes, actions taken within the store, and the like.

[1159] A "generative artificial intelligence model" is a machine learning model that generates new information and suggestions based on data.

[1160] "Fine-tuning" is the optimization process of adapting an existing generative artificial intelligence model to a specific dataset to improve its accuracy.

[1161] "Feedback" refers to information such as ratings and opinions provided by users, and is used to improve services.

[1162] "Product suggestions" refers to proposing appropriate products and services based on a user's past purchasing history and behavior.

[1163] "Display device" refers to a device or screen for visually displaying information, including, for example, the display of smart glasses.

[1164] 1. System Programming

[1165] To realize this application example, a series of programs are required to register various services in plug-in format, install them on the device, collect, save, and analyze user operation data, and fine-tune the generative AI model. The specific implementation method is explained below.

[1166] 2. Explanation of the generated program processing

[1167] The server provides an interface for developers to register various services in the form of plug-ins, allowing them to add new services. The registered plug-ins are then downloaded and installed on the device. The first application installed is the shopping assistant application, which includes a user authentication function.

[1168] The terminal uses hardware such as smart glasses or smartphones to collect user operation data in real time. This operation data includes call data, message data, memo data, and in-store behavior data. The collected data is sent to a backend server and stored in a secure database.

[1169] The server preprocesses the stored data and fine-tunes the generative AI model. For example, it uses Python data analysis libraries (such as NumPy and Pandas) to cleanse, classify, and shape the data. It then fine-tunes the data using a generative AI model (such as TensorFlow or PyTorch). This generates a model customized for each user.

[1170] The fine-tuned AI model is then sent back to the device, where personalized product suggestions are made to the user. Specifically, products based on the user's past purchase history and preferred styles are displayed on the smart glasses' display. Suggestions such as "This jacket is similar to a style you liked in the past. Would you like to try it on?" are displayed, improving the user's shopping experience.

[1171] 3. Examples and prompts

[1172] For example, if a user is in a clothing store in a shopping mall, the smart glasses can suggest new products based on the user's past purchases and preferred styles. The user can look at the display and receive suggestions such as, "This jacket is similar to a style you liked in the past. Would you like to try it on?"

[1173] An example prompt for a generative AI model might look something like this:

[1174] Generate new product suggestions based on the user's past purchases, and take into account the user's current location to make the most of your in-store product data.

[1175] In this way, the present invention significantly improves the shopping experience in physical stores by suggesting products based on user behavior data and past purchase history.

[1176] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1177] Step 1: Register the plugin

[1178] The server provides an interface for registering various services in the form of plugins. The input includes information about the various services registered by the developer. The server saves the entered service information in a database and makes it manageable as a plugin. This allows the various services to be registered on the server and ready to be distributed to devices.

[1179] Step 2: Install the plugin

[1180] The device downloads the specified plugin from the server and installs it on the system. The input includes plugin data sent from the server. The device receives this data, stores it in its file system, and the registered service becomes available.

[1181] Step 3: User authentication

[1182] The user uses the terminal to input authentication information to access various services. The input includes the user's authentication information (e.g., user ID, password). The terminal sends this authentication information to the server, which then verifies it against a database. If authentication is successful, the terminal becomes able to access all registered services.

[1183] Step 4: Collecting operational data

[1184] The terminal collects user operation data (calls, messages, notes, and in-store actions) in real time. The input includes the user's operation behavior. The terminal converts and prepares this data in the required format to send to the backend server.

[1185] Step 5: Save Data

[1186] The server stores the operation data sent from the terminal in a secure database. The input includes the operation data sent from the terminal. The server stores this data in a database and manages the operation history for each user.

[1187] Step 6: Preprocessing the data

[1188] The server preprocesses the stored operational data to remove noise and unnecessary data. The input includes raw data stored in a database. The server performs data cleansing using data analysis libraries (e.g., NumPy, Pandas) to generate a clean dataset.

[1189] Step 7: Fine-tuning the generative AI model

[1190] The server uses the preprocessed data to fine-tune a generative artificial intelligence model. The input includes a clean dataset. The server optimizes the AI ​​model using machine learning libraries (e.g., TensorFlow, PyTorch) and generates a customized model for each user.

[1191] Step 8: Serving the Model

[1192] The server delivers the fine-tuned generative artificial intelligence model to the device, which includes the optimized AI model as input. The server sends the model to the device, which then integrates the model into its system.

[1193] Step 9: Service Delivery

[1194] The device uses a fine-tuned generative artificial intelligence model to provide personalized product recommendations to users. Inputs include the optimized AI model and user operation data. Specifically, the device displays product recommendations based on past purchase history and behavior on the smart glasses display.

[1195] Step 10: Gather feedback

[1196] Users can input feedback about problems they encounter while using the service or requests for improvements. This feedback includes user ratings and opinions. The device sends this feedback to the server, which then analyzes it as data for improving the service.

[1197] Through these steps, the present invention improves the overall shopping experience in brick-and-mortar stores.

[1198] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1199] The present invention is a system that provides individually optimized services by combining a system for registering various services in a plug-in format and managing and providing them in an integrated manner with an emotion engine that recognizes the user's emotions. Specific embodiments for implementing the present invention and each processing step are described below in natural language.

[1200] Registering a service plugin

[1201] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, and notes, and provides a mechanism for integrating these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[1202] Plugin installation and user authentication

[1203] The device will be able to download plugins from the server, and if a user wants to use a particular service, they will be required to install the corresponding plugin on their device.

[1204] Users access various services through the plug-in installed on their devices and enter their authentication information for authentication. Once authentication is successful, users can access all registered services.

[1205] Data collection and storage

[1206] The device collects operational data such as calls, messages, and notes used by the user in real time. This operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. The saved data is classified by user and pre-processed as needed.

[1207] Data preprocessing and cleansing

[1208] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and formatted for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[1209] Introducing the Emotion Engine

[1210] The server recognizes the user's emotions based on the stored operation data using an emotion engine, which extracts emotion data from the content of the user's messages and calls using text and voice analysis.

[1211] Fine-tuning generative artificial intelligence

[1212] The server uses the preprocessed operation data and emotion data to fine-tune the generative artificial intelligence (AI) model. By incorporating the emotion data, the AI ​​model learns individual needs and tendencies based on the user's emotions. Once the fine-tuned AI model is complete, it is distributed to the device, where the AI ​​model is updated.

[1213] To provide services and improve user experience

[1214] The device will use fine-tuned AI models to provide individually optimized services to users. For example, in a memo app, AI will provide assistance and suggestions based on the user's emotions, improving operational efficiency.

[1215] Feedback and Service Improvement

[1216] The server receives and collects feedback from users. Users can post any problems or requests for improvements they have noticed while using the service to the server. The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements have been decided, the server updates the plugins and AI models and distributes the new versions to devices.

[1217] Specific examples

[1218] For example, when a user uses a messaging service, the device collects the user's message sending history and sends it to the server. The server uses this data and an emotion engine to recognize the user's emotional state and fine-tune the generative AI. The updated model delivered to the device is then used by the AI ​​to make predictions and completions when the user composes a message. Taking the emotion data into account, the AI ​​can make optimal suggestions based on the user's emotional state. This increases user efficiency and satisfaction.

[1219] The above is a specific embodiment for carrying out the present invention. The present invention provides users with a more personalized and seamless service experience, thereby improving usability.

[1220] The processing flow will be explained below.

[1221] Step 1:

[1222] The server exposes an API for registering various services in the form of plugins. Once services such as calls, messages, and notes are registered as plugins, the server integrates them into the system.

[1223] Step 2:

[1224] The device downloads and installs the necessary plugins from the server. When a user wants to use a particular service, they are required to install the corresponding plugin for that service on their device.

[1225] Step 3:

[1226] Users access various services through a plug-in installed on their device and enter their authentication information for authentication. Once authentication is successful, the user can access all registered services.

[1227] Step 4:

[1228] The device collects real-time data on user operations such as calls, messages, and notes, and stores this data as logs in the background.

[1229] Step 5:

[1230] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and pre-processed as necessary.

[1231] Step 6:

[1232] The server preprocesses the stored operational data to remove noise and unnecessary data, resulting in a clean dataset.

[1233] Step 7:

[1234] The server recognizes the user's emotions based on the pre-processed data using an emotion engine, which uses text analysis of messages and audio analysis of calls to identify the user's emotional state.

[1235] Step 8:

[1236] The server uses the interaction data and recognized emotion data to fine-tune a generative artificial intelligence (AI) model, enabling advanced pattern recognition that takes the user's emotional state into account.

[1237] Step 9:

[1238] The server then distributes the fine-tuned AI model to the device, which updates the AI ​​model on the device to reflect the latest user information.

[1239] Step 10:

[1240] The device will utilize fine-tuned AI models to provide optimized services to users, such as messaging services, where the AI ​​will provide assistance and suggestions based on the user's emotions.

[1241] Step 11:

[1242] Users can provide feedback to the server about any problems or requests for improvements they may have while using the system. This feedback can be easily sent via the terminal.

[1243] Step 12:

[1244] The server analyzes the collected feedback and identifies areas for improvement in the service. Once improvements are determined, the server updates the plugins and AI models and distributes the new versions to devices.

[1245] Specific examples

[1246] For example, when a user uses a messaging service, the device collects the content of the message sent by the user and their emotional state at the time of sending it. The server uses this data and an emotion engine to recognize the user's emotional state and fine-tune the AI ​​model. Next, using the updated model delivered to the device, the AI ​​makes emotion-based predictions and completions when composing messages. For example, if the AI ​​recognizes that the user is feeling angry, it can assist in smoother communication by suggesting a calmer message. This results in improved user efficiency and satisfaction.

[1247] This concludes the detailed explanation of each processing step. This system enables us to provide users with a seamless and personalized service experience.

[1248] Example 2

[1249] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1250] In today's diverse service delivery systems, there is a demand for services that are individually optimized for each user, but conventional systems require individual settings for each service, making it difficult to efficiently manage data and optimize individual services based on emotion recognition. Therefore, a system is needed that can manage a large number of different services in an integrated manner and provide optimized services that also respond to the user's emotional state.

[1251] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1252] In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for preprocessing the saved data and removing noise and unnecessary data, means for recognizing a user's emotion with an emotion engine using the preprocessed data, means for fine-tuning a generative AI model using the recognized emotion data, means for delivering the fine-tuned AI model to the terminal, service providing means for providing a service to the user based on the fine-tuned AI model, and means for collecting feedback from the user and improving the service. This makes it possible to manage a variety of services in an integrated manner and provide optimized services that also correspond to the emotional state of the user.

[1253] "Various services" refers to different functions and offerings that users can use, such as calls, messages, and notes.

[1254] The "plug-in format" refers to a format in which various services are modularized and can be easily added or removed from the system.

[1255] "Terminal" refers to a computing device, such as a smartphone, tablet, or PC, that a user uses to access various services.

[1256] "Operational Data" refers to information generated or collected when a user uses various services, such as call records, message sending history, and memo input data.

[1257] "Preprocessing" refers to the process of removing noise and unnecessary data from stored data and preparing it in a form that is easy to analyze.

[1258] "Noise" refers to unnecessary information that is not needed in the data processing process and that may adversely affect the results of the analysis.

[1259] "Emotion Engine" refers to an algorithm or software that uses text and speech analysis techniques to recognize and extract a user's emotional state.

[1260] A "generative artificial intelligence model" refers to a machine learning model that can learn large amounts of data and generate new content, such as GPT-3 or BERT.

[1261] "Fine-tuning" refers to the process of adjusting an existing machine learning model to optimize its performance for a specific dataset or application.

[1262] "Feedback" refers to opinions and reports that users provide to the server regarding problems or requests for improvement they have experienced while using the service.

[1263] A "secure database" refers to a data management system in which stored data is protected from unauthorized access and confidentiality, integrity, and availability are ensured.

[1264] MODE FOR CARRYING OUT THE INVENTION

[1265] The present invention is a system that provides individually optimized services by combining a system that registers various services in a plug-in format and manages and provides them in an integrated manner with an emotion engine that recognizes the user's emotions. Specific embodiments for carrying out the present invention will be described in detail below.

[1266] Registering a service plugin

[1267] The server provides an API interface that allows developers to register various services in the form of plugins. If a developer wants to provide new functions, such as a call recording service, messaging service, or memo service, they can register them as plugins. Specifically, the plugins are uploaded using the API provided by the server and integrated into the entire system.

[1268] Plugin installation and user authentication

[1269] The device has the ability to download and install plug-ins from the server. When a user wants to use a particular service, such as a call recording service, the plug-in is installed on the device. The user then accesses the service through the installed plug-in on the device and is authenticated by entering authentication information, such as a username and password. If the authentication is successful, the user can access all registered services.

[1270] Data collection and storage

[1271] The device collects real-time operational data, such as calls, messages, and notes, used by the user. This operational data is saved as a log file on the device and periodically sent to the server. The server stores this data in a secure database, where it is categorized by user.

[1272] Data preprocessing and cleansing

[1273] The server preprocesses the stored operational data to remove noise and unnecessary data. Noise refers to unnecessary information that is not needed in the data processing process and may adversely affect the analysis results. This preprocessing generates a clean dataset. The preprocessed data is organized into time-series data, making it easier to analyze.

[1274] Introducing the Emotion Engine

[1275] The server uses the cleansed operation data to recognize the user's emotions with an emotion engine. This emotion engine uses text analysis (e.g., natural language processing technology) and voice analysis (e.g., voice recognition technology) to extract emotion data from the user's messages and call content. For example, emotions such as "tired" or "happy" can be detected from messages sent by the user.

[1276] Fine-tuning generative artificial intelligence

[1277] The server uses the preprocessed operation data and emotion data to fine-tune a generative artificial intelligence (AI) model. For example, it uses deep learning models such as GPT-3 and BERT to improve the model's ability to respond and make suggestions based on the user's emotional changes. The fine-tuned AI model is then distributed to the device, where it can be used.

[1278] To provide services and improve user experience

[1279] The device uses fine-tuned generative AI models to provide individually optimized services to users. For example, in a notes app, when a user types "I'm tired," it automatically suggests "Make time to take a break." This increases user efficiency and satisfaction.

[1280] Prompt Sentence Examples

[1281] Based on the following conversation history, we present examples of prompt sentences that recognize the user's emotions and provide appropriate suggestions.

[1282] "Based on the following conversation history, please recognize the user's emotions and provide appropriate suggestions.

[1283] Conversation History:

[1284] User: 'I'm extremely busy and stressed today'

[1285] Generative AI model: 'Sounds like a lot of work. Good luck.'

[1286] Also, please complete the following message:

[1287] By inputting this prompt into a generative AI model, it is possible to understand the user's emotions and provide the most appropriate message for the situation.

[1288] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1289] Step 1: Registering the service plugin

[1290] The server provides an API interface that allows developers to register various services in the form of plugins. Specifically, developers create new services as plugins and send the plugin's metadata and files to the server. The inputs are the plugin specifications and executable files, and the output is that the plugin is registered in the system. This makes services such as call logs, messages, and notes available in the system.

[1291] Step 2: Installing the plugin and authenticating the user

[1292] The device has the function of downloading and installing registered plugins from the server. When a user uses a specific service, the device allows access to the service via the installed plugin. During this process, the plugin file from the server and the user's authentication information (username, password) are used as input, and the service becomes available as output. If authentication is successful, the user will be able to access the corresponding service.

[1293] Step 3: Data collection and storage

[1294] The device collects operational data such as calls, messages, and notes used by the user in real time. This data is stored in the device as a log file and periodically sent to the server. The input is user operation data (call records, message contents, note contents), and the output is the saved log file and sent data. The server saves the received operation data in a secure database. The saved data is categorized by user.

[1295] Step 4: Preprocessing and cleansing the data

[1296] The server preprocesses the stored operational data, specifically removing noise, filling in missing data, and detecting outliers. The input is the stored operational data, and the output is a clean dataset, improving the quality of the data and making it easier to analyze.

[1297] Step 5: Implementing the Emotion Engine

[1298] The server uses the preprocessed operation data to run an emotion engine. This emotion engine uses text analysis and speech analysis techniques to recognize the user's emotional state. The preprocessed operation data (message content, call content) is used as input, and emotion data is generated as output. For example, emotions such as "I'm stressed" can be detected through text analysis.

[1299] Step 6: Fine-tuning the generative AI

[1300] The server uses the preprocessed interaction data and emotion data to fine-tune a generative artificial intelligence (AI) model. Specifically, it applies it to a generative AI model (e.g., GPT-3) to improve its ability to generate responses based on user emotions. The clean interaction data and emotion data are input, and a fine-tuned AI model is generated as output. This model learns optimal responses based on individual needs.

[1301] Step 7: Providing services and improving user experience

[1302] The device uses a fine-tuned generative AI model to provide optimal services to users. For example, if a user uses a memo app and enters "I'm tired," the AI ​​generates a suggestion such as "I recommend you take a rest." The inputs include user operations and emotional data, and the output is the AI's optimal suggestion. This improves user efficiency and satisfaction.

[1303] Step 8: Feedback and Service Improvement

[1304] The server collects feedback from users and uses it to improve the service. Users can post any problems or requests for improvements they have noticed while using the service to the server. The input is user feedback, and the output is analyzed improvements. The server analyzes the feedback, updates the plugins and AI models, and delivers the improved versions to the devices. This allows for continuous improvement of the service.

[1305] (Application example 2)

[1306] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1307] Current content delivery systems recommend content uniformly without considering the user's emotional state, resulting in a decrease in user satisfaction and experience value. Specifically, they may be unable to recommend relaxing content to a stressed user, while recommending inappropriate content to a user in an energetic state. A system that can solve these problems and realize optimal content recommendations based on the user's emotions is needed.

[1308] The specification processing by specification processing unit 290 of data processing device 12 in application example 2 is realized by the following means. In this invention, the server includes means for registering various services in the form of plug-ins, means for installing the registered plug-ins on the terminal, means for collecting user operation data, means for saving the collected data, means for fine-tuning the generative AI model using the saved data, means for delivering the fine-tuned AI model to the terminal, means for generating emotion data using an emotion engine that recognizes the user's emotions, means for recommending optimal content to the user based on the generated emotion data, service providing means for providing the user with a service based on the fine-tuned generative AI model, and means for collecting user feedback and improving the service. This makes it possible to recommend optimal content according to the user's emotional state.

[1309] "Various services" are abstract means that provide functions and actions such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis that users can use.

[1310] The "plug-in format" is a method of modularizing various services and providing them in a form that can be easily installed and updated.

[1311] "Terminal" refers to a device used by a user, including smartphones, tablets, head-mounted displays, etc.

[1312] "User operation data" refers to call records, message history, memo contents, heart rate, facial expressions, and voice data generated when a user operates a terminal.

[1313] "Stored data" refers to the state in which collected user operation data is stored in a secure database.

[1314] A "generative artificial intelligence model" is a machine learning model that learns from collected data and generates useful information and services for users.

[1315] "Fine-tuning" is the act of fine-tuning an existing generative artificial intelligence model to optimize its performance for a specific dataset or purpose.

[1316] An "emotion engine" is a software component that analyzes user operation data and recognizes and extracts their emotional state.

[1317] "Emotion data" is a data set that is analyzed using an emotion engine and indicates the user's emotional state.

[1318] "Content" refers to media files that can be viewed or listened to, and is an information resource that includes movies, music videos, documentaries, etc.

[1319] "Recommendation" is the act of the system selecting and providing content or services that it deems most suitable based on the user's data and emotional state.

[1320] "Feedback" refers to information that indicates evaluations, impressions, and requests for improvement provided by users regarding the system.

[1321] This invention is a system that provides individually optimized content by combining a system that registers various services in a plug-in format and manages and provides them in an integrated manner with an emotion engine that recognizes the user's emotions.

[1322] Registering a service plugin

[1323] The server provides an interface that allows developers to register various services in the form of plugins. The server accepts services such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis, and integrates these into the system as plugins. Developers use the API provided by the server to register service plugins on the server.

[1324] Plugin installation and user authentication

[1325] The device downloads and installs the plug-in from the server. When a user wants to use a specific service, the device is required to install the plug-in corresponding to that service. The user accesses various services through the plug-in installed on the device and is authenticated by entering authentication information. If authentication is successful, the user can access all registered services.

[1326] Data collection and storage

[1327] The device collects real-time operational data, such as user calls, messages, notes, heart rate measurements, facial expressions, and voice analysis. The collected operational data is saved as a log in the background. The server receives the operational data sent from the device and saves it in a secure database. The saved data is classified by user and preprocessed as necessary.

[1328] Data preprocessing and cleansing

[1329] The server preprocesses the stored data to remove noise and unnecessary data, resulting in a clean dataset. The preprocessed data is then classified and shaped for analysis, and different types of data, such as messages and call logs, are organized chronologically.

[1330] Introducing the Emotion Engine

[1331] The server recognizes the user's emotions based on the stored operation data using an emotion engine, which uses text analysis and voice analysis to extract emotion data from the user's messages, call content, heart rate, and facial expressions.

[1332] Fine-tuning generative artificial intelligence

[1333] The server uses the preprocessed operation data and emotion data to fine-tune the generative artificial intelligence (AI) model. By incorporating the emotion data, the AI ​​model learns individual needs and tendencies based on the user's emotions. Once the fine-tuned AI model is complete, it is distributed to the device, where the AI ​​model is updated.

[1334] To provide services and improve user experience

[1335] The device uses a fine-tuned AI model to recommend content that is individually optimized for each user. For example, if a user using a head-mounted display is recognized as feeling stressed, movies and music with a relaxing effect will be recommended. A user in an energetic state will be recommended action movies and dance videos. In this way, personalized content is provided according to the user's current emotional state.

[1336] Feedback and Service Improvement

[1337] The server receives and collects feedback from users. Users can post any problems or requests for improvements they have noticed while using the service to the server. The server analyzes the collected feedback and identifies areas for improvement in the service. Once the improvements have been decided, the server updates the plugins and AI models and distributes the new versions to devices.

[1338] For example, a user may want to relax late at night and put on a head-mounted display. The system can recognize the user's heart rate and facial expression as indicating fatigue or a desire for relaxation, and recommend relaxing music or landscape videos. If the user feels energetic and energetic, action movies or dance videos may be recommended.

[1339] An example of a prompt sentence to be input to a generative artificial intelligence model is:

[1340] "If a user is in a state where they want to relax, what are the best movies to recommend?"

[1341] "Provide effective music when the user's heart rate is high and they are feeling fatigued."

[1342] There is.

[1343] The above is a specific embodiment for carrying out the present invention. This system makes it possible to provide users with a more personalized and seamless service experience, thereby improving usability.

[1344] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1345] Step 1:

[1346] The server provides an interface for registering various services in the form of plugins. Developers use the API to register services such as calls, messages, notes, heart rate measurement, facial expression analysis, and voice analysis. In this step, service data is received as input and output is obtained for each plugin to be integrated into the server.

[1347] Step 2:

[1348] The device downloads and installs the plug-in from the server. The user installs the plug-in to use a specific service through the device. In this step, an installation request is input and the device outputs the state in which the plug-in has been installed.

[1349] Step 3:

[1350] The user accesses various services through the plug-in installed on the terminal and enters authentication information to be authenticated. If authentication is successful, the user can access all registered services. In this step, authentication information is entered and the result of authentication success / failure is output.

[1351] Step 4:

[1352] The device collects user operation data in real time, including call records, message history, memo contents, heart rate data, facial expression data, and voice data. In this step, operation data is input and the collected data log is output.

[1353] Step 5:

[1354] The server receives the operation data sent from the device and stores it in a secure database. The stored data is classified by user and preprocessed as needed. In this step, operation data is input and a preprocessed clean dataset is output.

[1355] Step 6:

[1356] The server preprocesses the stored data to remove noise and unnecessary data. This process generates a clean dataset and organizes data in different formats in chronological order. In this step, the data to be preprocessed is input and a cleaned dataset is output.

[1357] Step 7:

[1358] The server uses an emotion engine to recognize the user's emotions based on the stored operation data. It extracts emotion data from the content of the user's messages and calls using text analysis and voice analysis. In this step, operation data is input and emotion data is output.

[1359] Step 8:

[1360] The server uses the collected and preprocessed operation data and emotion data to fine-tune the generative AI model, thereby learning the user's individual needs and tendencies. In this step, the preprocessed data and emotion data are input, and a fine-tuned AI model is output.

[1361] Step 9:

[1362] The server delivers the fine-tuned generative AI model to the device, which then uses the updated AI model to provide personalized content to the user. In this step, an AI model is input and a fine-tuned AI model is output that is delivered to the device.

[1363] Step 10:

[1364] The device uses a fine-tuned AI model to recommend content that is individually optimized for the user. For example, if the system recognizes that the user is feeling stressed, it will recommend content that has a relaxing effect. In this step, emotional data is input and recommended content is output.

[1365] Step 11:

[1366] The server collects and analyzes user feedback. Based on the feedback, it identifies improvements to the system and updates the plugins and AI models. In this step, feedback data is input and an improved system is output.

[1367] Step 12:

[1368] For example, if the emotion recognition result indicates "fatigue" when a user wants to relax, relaxing music and landscape videos will be recommended. If the user is feeling energetic and lively, action movies and dance videos will be recommended. In this step, the emotional state is input and optimized content is output.

[1369] A concrete example of a prompt is:

[1370] "If a user is in a state where they want to relax, what are the best movies to recommend?"

[1371] "Provide effective music when the user's heart rate is high and they are feeling fatigued."

[1372] There is.

[1373] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1374] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1375] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1376] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1377] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1378] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1379] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1380] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1381] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1382] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1383] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1384] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1385] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1386] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1387] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1388] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1389] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1390] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1391] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1392] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1393] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1394] The following is further disclosed regarding the above embodiment.

[1395] (Claim 1)

[1396] A means to register various services in plugin format,

[1397] A means for installing the registered plugin on the device;

[1398] A means for collecting user operation data;

[1399] a means of storing the collected data; and

[1400] means for fine-tuning a generative artificial intelligence model using the stored data; and

[1401] A means for delivering the fine-tuned artificial intelligence model to the device; and

[1402] a service providing means for providing a service based on a fine-tuned artificial intelligence model to a user;

[1403] A system that includes a means of collecting feedback from users and improving the service.

[1404] (Claim 2)

[1405] 2. The system according to claim 1, wherein the collected user operation data includes call data, message data, and memo data.

[1406] (Claim 3)

[1407] The system according to claim 1, characterized in that fine-tuning of the generative artificial intelligence model is performed individually for each user.

[1408] "Example 1"

[1409] (Claim 1)

[1410] A means for developers to register various services in the form of plugins,

[1411] A means of registering plugins using the API provided by the server,

[1412] A means for installing the registered plugin on the device;

[1413] A means for entering user credentials to access the service;

[1414] A means for collecting user operation data in real time;

[1415] a means for storing the collected operational data;

[1416] A means for preprocessing the stored data to remove noise and unnecessary data;

[1417] means for fine-tuning a generative artificial intelligence model using the preprocessed data; and

[1418] A means for delivering the fine-tuned artificial intelligence model to the device; and

[1419] A means for optimizing and providing services to users based on fine-tuned artificial intelligence models;

[1420] A system that includes a means to collect and analyze feedback from users to improve services.

[1421] (Claim 2)

[1422] 2. The system according to claim 1, wherein the collected user operation data includes call data, message data, and memo data.

[1423] (Claim 3)

[1424] The system according to claim 1, characterized in that fine-tuning of the generative artificial intelligence model is performed individually for each user.

[1425] "Application Example 1"

[1426] (Claim 1)

[1427] A means to register various services in plugin format,

[1428] A means for installing the registered plugin on the device;

[1429] A means for collecting user operation data;

[1430] a means of storing the collected data; and

[1431] means for fine-tuning a generative artificial intelligence model using the stored data; and

[1432] A means for delivering the fine-tuned artificial intelligence model to the device; and

[1433] a service providing means for providing a service based on a fine-tuned artificial intelligence model to a user;

[1434] A means of collecting user feedback and improving the service;

[1435] A means for the terminal to make product suggestions to assist the user in the physical store;

[1436] The system includes means for displaying the proposal on a display device of the terminal.

[1437] (Claim 2)

[1438] 2. The system according to claim 1, wherein the collected user operation data includes call data, message data, memo data, and in-store behavior data.

[1439] (Claim 3)

[1440] The system described in claim 1, characterized in that the generative artificial intelligence model is fine-tuned individually for each user, and product suggestions are made based on the user's past purchase history.

[1441] "Example 2: Combining Emotion Engines"

[1442] (Claim 1)

[1443] A means to register various services in plugin format,

[1444] A means for installing the registered plugin on the device;

[1445] A means for collecting user operation data;

[1446] a means of storing the collected data; and

[1447] A means for preprocessing the stored data to remove noise and unnecessary data;

[1448] means for recognizing a user's emotion with an emotion engine using the preprocessed data;

[1449] a means for fine-tuning a generative artificial intelligence model using the recognized emotion data; and

[1450] A means for delivering the fine-tuned artificial intelligence model to the device; and

[1451] a service providing means for providing a service based on a fine-tuned artificial intelligence model to a user;

[1452] A system that includes a means of collecting feedback from users and improving the service.

[1453] (Claim 2)

[1454] 2. The system according to claim 1, wherein the collected user operation data includes call data, message data, and memo data.

[1455] (Claim 3)

[1456] The system according to claim 1, characterized in that fine-tuning of the generative artificial intelligence model is performed individually for each user.

[1457] "Application example 2 when combining emotion engines"

[1458] (Claim 1)

[1459] A means to register various services in plugin format,

[1460] A means for installing the registered plugin on the device;

[1461] A means for collecting user operation data;

[1462] a means of storing the collected data; and

[1463] means for fine-tuning a generative artificial intelligence model using the stored data; and

[1464] A means for delivering the fine-tuned artificial intelligence model to the device; and

[1465] A means for generating emotion data using an emotion engine that recognizes the emotion of a user;

[1466] A means for recommending optimal content to a user based on the generated emotion data;

[1467] a service providing means for providing a service based on a fine-tuned generative artificial intelligence model to a user;

[1468] A system that includes a means of collecting feedback from users and improving the service.

[1469] (Claim 2)

[1470] 2. The system according to claim 1, wherein the collected user operation data includes call data, message data, memo data, heart rate data, facial expression data, and voice data.

[1471] (Claim 3)

[1472] The system described in claim 1, characterized in that the generative artificial intelligence model is fine-tuned individually for each user and is optimized according to the user's emotional state. [Explanation of symbols]

[1473] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means to register various services in plugin format, A means for installing the registered plugin on the device; A means for collecting user operation data; a means of storing the collected data; and means for fine-tuning a generative artificial intelligence model using the stored data; and A means for delivering the fine-tuned artificial intelligence model to the device; and a service providing means for providing a service based on a fine-tuned artificial intelligence model to a user; A system that includes a means of collecting feedback from users and improving the service.

2. 2. The system according to claim 1, wherein the collected user operation data includes call data, message data, and memo data.

3. 2. The system according to claim 1, wherein fine-tuning of the generative artificial intelligence model is performed individually for each user.

Images