System

Abstract

A system is provided.SOLUTION: A system comprising: means for storing personal information using a local database and restricting transmission to the outside; means for performing clustering based on feature data of a user; means for constructing a federated learning model using data collected from a plurality of users; and means for transmitting a learning parameter of the federated learning model to an external server.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] ---

[0005] While generative AI models have high performance and are widely used in the market, their generalization makes them difficult to differentiate. Furthermore, the risk of personal information leaks prevents major companies from using generative AI models. To address these issues, there is a need to provide an environment in which companies can incorporate generative AI models into their operations without worrying about the risk of information leaks. [Means for solving the problem]

[0006] The present invention provides a system that includes a means for storing personal information using a local database and restricting external transmission, a means for clustering based on user feature data, a means for building a federated learning model using data collected from multiple users, and a means for transmitting the learning parameters of the federated learning model to an external server, thereby significantly reducing the risk of personal information leakage and creating an environment in which companies can safely utilize generative AI models. Furthermore, the present invention realizes a system that meets the needs of both users and companies by providing a means for data sovereignty management of personal information and management functions for providing optimal services while protecting user privacy.

[0007] ---

[0008] A "local database" is a database that stores data on the user's device or server and restricts data transmission to the outside.

[0009] "Personal information" is information that can identify a specific individual, and includes data such as name, address, telephone number, and email address.

[0010] "Feature data" is data that indicates the nature or characteristics of a specific object (in this case, a user) and is used for clustering and analysis.

[0011] "Clustering" is a machine learning technique that divides data into groups with similar characteristics, and is used in classification problems and data analysis.

[0012] "Federated learning" is a machine learning technique in which multiple appliances collaborate to train a model, allowing each device to participate in the learning process without sending its own data.

[0013] "Learning parameters" are parameters that a machine learning model learns from data, including weights and biases to perform tasks such as prediction and classification.

[0014] An "external server" is a remote server accessible via the Internet or other communication means that stores and processes data.

[0015] "Data sovereignty" is a mechanism that allows users to manage and control how their data is used, with the aim of enhancing privacy and security.

[0016] "Management functions" are functions that enable the system to provide optimal services while ensuring user privacy and data security. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] ---

[0039] This invention relates to a generative AI model system that uses a local database to store personal information and restricts its transmission to external servers. Specifically, it performs clustering based on user feature data, builds a federated learning model using data collected from multiple users, and transmits the learning parameters to an external server. The invention also provides a means for data sovereignty management of personal information and a management function for providing optimal services while protecting user privacy.

[0040] First, the server uses a local database to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. The server checks the existence of a local database file during initialization, and creates a new one if it does not exist. This file stores each user's personal information and usage data.

[0041] The server then performs clustering based on the user feature data. The feature data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm to classify users with similar features into the same group. This makes it easier to analyze the feature data and provide customized services.

[0042] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only learning parameters are sent to the external server. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leakage.

[0043] As a specific example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and clusters users with similar purchasing patterns. It then inputs all users' purchasing data into a federated learning algorithm to create an integrated model. This model can then provide optimized product recommendations for each individual user.

[0044] From the perspective of privacy management, the server also has a means of data sovereignty management. This is a function to strictly manage personal information so that it is not leaked to the outside. Data sovereignty management includes data encryption, access control, and obtaining usage permission. This allows users to safely manage how their data is used.

[0045] Furthermore, the server has management functions to provide optimal services while protecting users' privacy. These management functions allow the server to provide services tailored to users' requests and needs. For example, the server can provide personalized suggestions and advice by taking into account users' usage history and feedback.

[0046] As described above, the present invention provides a system that provides high-performance generative AI models while preventing the leakage of personal information. This allows companies to incorporate generative AI models into their business with peace of mind, and users can use them with peace of mind, knowing that their privacy is protected.

[0047] The processing flow will be explained below.

[0048] ---

[0049] Step 1:

[0050] When the server initializes the program, it checks for the existence of a local database file. If the file exists, the server reads it and loads the user data into local memory. If it does not exist, the server creates an empty database and creates a new file.

[0051] Step 2:

[0052] If the user provides new data, the server updates the local database by calling the update_local_db method. During this time, the data received from the user is merged with the existing data and updated to the latest state.

[0053] Step 3:

[0054] After the server updates the local database, it calls the save_data method to save the database to a file, which makes the user data persistent.

[0055] Step 4:

[0056] The server performs clustering based on the user feature data. To do this, the server takes all users' feature data and performs clustering using the KMeans algorithm. Each user is assigned to an appropriate cluster based on their feature data.

[0057] Step 5:

[0058] After clustering is complete, the server assigns each user a cluster ID, which allows users with similar characteristics to be grouped together for efficient data analysis and processing.

[0059] Step 6:

[0060] The server uses all user data to perform federated learning. Using data collected from each user, it locally trains a model and aggregates the results to build an overall AI model. Individual user data is not transmitted externally, protecting privacy.

[0061] Step 7:

[0062] The server sends the learning parameters obtained as a result of federated learning to an external server, which makes it possible to remotely utilize more accurate models while protecting personal information.

[0063] Step 8:

[0064] The server will manage data sovereignty and protect users' personal information from leaks to the outside world, by encrypting data, controlling access, and obtaining permission to use the data.

[0065] Step 9:

[0066] The server has management functions to provide optimized services while protecting the user's privacy, allowing the server to provide customized services according to the user's requests and needs.

[0067] ---

[0068] The above is the specific processing flow of the "MyData-GPT" system. The roles of the server, terminal, and user are clearly distinguished, and the specific operations performed at each step are explained.

[0069] Example 1

[0070] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0071] In modern information systems, users' personal information is collected by many service providers, increasing the possibility of it being leaked to external parties. This increases the risk of user privacy being violated. It is also difficult to effectively utilize individual data to create advanced AI models. Furthermore, there is insufficient data authority management for personal information, making it difficult to use services with peace of mind.

[0072] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0073] In this invention, the server includes means for storing personal information using local storage and restricting external transmission, means for clustering based on user attribute data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external computer, and means for aggregating the learning parameters and updating the final model. This makes it possible to construct a highly accurate AI model without leaking personal information to the outside, and to provide optimal services while protecting user privacy.

[0074] "Local storage" refers to data storage that is stored directly on the user's device and is a storage device for holding data that is not transferred to an external server.

[0075] "Personal Information" means identifiable information about a user, including name, address, telephone number, email address, purchase history, etc.

[0076] "Clustering" is a machine learning algorithm for analyzing a collection of data and grouping data with similar attributes.

[0077] "Federated learning" is a technology that performs partial learning on each terminal without collecting data from multiple distributed terminals, and then collects only the resulting learning parameters to update the overall model.

[0078] "Learning parameters" are internal numerical information that is adjusted by training a machine learning model, and refer to coefficients and biases that improve the model's predictive accuracy.

[0079] An "external computer" is a computing resource for a server or cloud service that is located in a location separate from the user's device or local storage.

[0080] "Data Rights Management" is a security measure that manages access rights and usage permissions for personal information, allowing users to control how their data is used.

[0081] "Management functions for providing optimal services while protecting privacy" refers to system functions that analyze collected data and provide customized services and recommendations tailored to each user while protecting the user's privacy.

[0082] This invention relates to a generative AI model system that uses local storage to store personal information and restricts its external transmission. Specifically, it performs clustering based on user attribute data, builds a federated learning model using data collected from multiple users, and transmits the learning parameters to an external computer. The invention also includes a means for managing data authority for personal information and a management function for providing optimal services while protecting user privacy.

[0083] First, the server uses local storage to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. When the server is initialized, it checks for the existence of a local storage file, and if the file does not exist, it creates a new one. This file stores each user's personal information and usage data. For example, a database file called "user_data.db" is created using SQLite, and a "users" table and a "user_logs" table are placed in it.

[0084] Next, the server performs clustering based on the user's attribute data. The attribute data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm, specifically K-means clustering, to classify users with similar characteristics into the same group. This makes it easier to analyze based on the attribute data and enables the provision of customized services.

[0085] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only the learning parameters are sent to the external computer. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leaks. Specifically, learning is carried out using machine learning frameworks such as Pytorch and TensorFlow.

[0086] As a specific example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and clusters users with similar purchasing patterns. It then inputs the purchasing data of all users into a federated learning algorithm to create an integrated model. This model can then make product recommendations optimized for each individual user.

[0087] From the perspective of privacy management, the server also has a means for data permission management. This is a function for strictly managing personal information to prevent it from leaking to the outside. Data permission management includes data encryption, access control, and obtaining permission to use. This allows users to safely manage how their data is used.

[0088] Furthermore, the server has a management function to provide optimal services while protecting the user's privacy. This management function allows the server to provide services tailored to the user's requests and needs. For example, it can provide personalized suggestions and advice based on the user's usage history and feedback.

[0089] An example prompt is:

[0090] "This system will recommend the best products based on your recent purchase history. Would you be interested in any of these products?"

[0091] As described above, the present invention provides a system that provides high-performance generative AI models while preventing the leakage of personal information. This allows companies to incorporate generative AI models into their business operations with peace of mind, and users can use them with peace of mind, knowing that their privacy is protected.

[0092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0093] Step 1: Initialize and configure the local database

[0094] During system initialization, the server checks for the existence of a local database file. As input, it references the path to the database file on the file system. If the file does not exist, the server creates a new database file and initializes the appropriate tables. The database contains personal information for each user. As output, it creates an SQLite database called "user_data.db" and the "users" and "user_logs" tables within it.

[0095] Specifically, the server performs the following operations:

[0096] Checking for the existence of "user_data.db"

[0097] Creating a database if it does not exist

[0098] Initializing the "users" and "user_logs" tables

[0099] Step 2: Collect user data

[0100] The device collects data according to the user's operations. As input, the user's operation log and input data are acquired in real time. This data is stored in a local database. As output, the input data is added to the "user_logs" table.

[0101] Specifically, the device performs the following operations:

[0102] Monitoring user clicks and input information

[0103] Insert the obtained data into a local database

[0104] Step 3: Clustering feature data

[0105] The server periodically clusters user data. User attribute data obtained from a local database is used as input. The server uses this data to run a machine learning algorithm (e.g., K-means clustering) to group users. The output is the clustering results.

[0106] Specifically, the server performs the following operations:

[0107] Execute "SELECT FROM user_logs WHERE timestamp > {last_scan_timestamp}"

[0108] Extracting attribute data and calculating features

[0109] Clustering with machine learning algorithms

[0110] Step 4: Running federated learning

[0111] The server then starts federated learning based on the data collected from all users. The local data of each device is used as input. The device uses that data to train a model locally and sends the training parameters to the server. The server integrates the collected parameters to update the final AI model. The output is an updated AI model.

[0112] Specifically, the server and the device perform the following operations:

[0113] Learning task distribution from the server

[0114] On-device model training (e.g., Pytorch or TensorFlow)

[0115] Sending learning parameters and aggregating them on the server

[0116] Step 5: Optimized service delivery

[0117] Finally, the server uses the updated AI model to provide the optimal service to the user. The inputs are the updated AI model and the user's usage history and feedback. The server generates personalized recommendations and proposals based on this and sends them to the device. The output is optimized recommendation data.

[0118] Specifically, the server and the device perform the following operations:

[0119] Data analysis using updated AI models

[0120] Generating recommendations

[0121] Sending recommendation data to devices

[0122] Viewing recommendations on your device

[0123] (Application example 1)

[0124] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] Conventional electronic payment services often do not adequately protect privacy when transmitting users' purchase history and personal information to external parties, resulting in stress for users and the risk of information leaks. Furthermore, they lack functionality for providing personalized promotions and offers, leaving the user experience unsatisfactory. The purpose of this invention is to solve these problems and provide a safe and effective personalized service.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0127] In this invention, the server includes a means for storing personal information using a local database and restricting transmission to the outside, a means for clustering based on user feature data, a means for building a federated learning model using data collected from multiple users, and a smartphone application including a management function for providing personalized promotions and offers, thereby enabling the provision of optimized promotions and offers while protecting user privacy.

[0128] A "local database" is a database that protects privacy by storing a user's personal information on the device and restricting its transmission to the outside.

[0129] "Personal information" refers to a user's unique identification information, purchasing history, behavioral patterns, etc.

[0130] "Feature data" is data that includes user behavior patterns and attribute information, and is used for clustering.

[0131] "Clustering" is the process of grouping users with similar characteristics based on their feature data.

[0132] A "federated learning model" is a machine learning model constructed using data collected from multiple users, in which only the learning parameters are sent to an external server.

[0133] "Learning parameters" are parameters sent to an external server to improve the accuracy of the model in federated learning.

[0134] The "external server" is a server that receives the learning parameters of the federated learning model and constructs an integrated model.

[0135] "Personalized promotions and offers" are promotions and offers that provide personalized suggestions and benefits to individual users based on their user characteristics data.

[0136] A "smartphone application" is an application that runs on a smartphone, handles users' personal information, and provides users with personalized services through clustering and federated learning.

[0137] "Data sovereignty management" refers to the means of managing users' personal information to prevent it from being leaked to the outside, and includes encryption and access control.

[0138] "Privacy protection" refers to measures to prevent users' personal information from being misused or leaked.

[0139] This invention specifically describes a system for protecting users' personal information and providing personalized promotions in electronic payment services. This system has the functions of storing personal information using a local database and restricting external transmission, clustering users based on feature data, and building a federated learning model.

[0140] First, the device (e.g., a smartphone) stores the user's purchasing history and behavioral patterns in a local database. This database is built using SQLite, and the user's personal information is securely encrypted and stored using the Fernet encryption method.

[0141] The device then performs clustering based on the user feature data, using the KMeans algorithm to group similar users based on their behavioral patterns and attribute information.

[0142] In the federated learning step, each user's data is trained locally and the training parameters are sent to an external server. TensorFlow Federated (TFF) is used to build the federated learning model. This ensures privacy by keeping individual user data local and only sending the training parameters to the external server.

[0143] Furthermore, the terminal will utilize the constructed model to provide users with personalized promotions and offers, which will suggest optimal offers and deals for them, improving their user experience.

[0144] For example, when a user records their past purchase history using a smartphone application, that data is encrypted and stored in a local database. Using a model built through federated learning, the user is then presented with "recommended coupons." These coupons are personalized based on the user's purchase history and behavioral patterns.

[0145] Example prompt sentence:

[0146] I want to create a feature that uses a user's past purchase history data to display coupon recommendations while maintaining privacy. How can I use federated learning and clustering to provide personalized promotions?

[0147] The system is realized using a smartphone, a local database (SQLite), an encryption library (Fernet), a clustering algorithm (KMeans), and a federated learning framework (TensorFlow Federated).

[0148] As described above, this invention is a system that provides safe and effective personal information protection and personalized services through a smartphone application. This system allows users to use electronic payment services with peace of mind, and allows companies to provide highly reliable services.

[0149] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0150] Step 1:

[0151] Initializing the local database

[0152] The device initializes a local database to store the user's personal information. Specifically, it uses SQLite and creates a new database file if one does not exist. It checks the state of the local file system as input and generates a database file to store the user information as output.

[0153] Step 2:

[0154] Data encryption and storage

[0155] The device encrypts the user's purchase history and behavioral patterns and stores them in a local database. Specifically, it uses the Fernet encryption library to encrypt user data. It receives unencrypted purchase history and behavioral patterns as input and stores the encrypted data in the local database as output.

[0156] Step 3:

[0157] Feature data collection and clustering

[0158] The device collects user feature data and performs clustering using the KMeans algorithm. Collected user behavior patterns and attribute information are used as input, and users classified into clusters are output.

[0159] Step 4:

[0160] Implementing federated learning

[0161] The device learns each user's data locally and sends the learning parameters to an external server. Specifically, federated learning is performed using TensorFlow Federated (TFF). Each user's purchase history data is provided as input, and the learning parameters of the federated learning model are generated as output and sent to the external server.

[0162] Step 5:

[0163] Model integration with external servers

[0164] The server receives the learning parameters sent from the devices and constructs an integrated federated learning model. Learning parameters from multiple devices are received as input, and an integrated, high-precision federated learning model is generated as output.

[0165] Step 6:

[0166] Providing personalized service

[0167] The device uses the constructed federated learning model to provide personalized promotions and offers to the user. Using the federated model and user feature data as input, the device displays promotions and offers optimized for the user as output.

[0168] In this way, the system protects users' personal information while providing personalized promotions and offers that are most relevant to them.

[0169] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0170] ---

[0171] This invention combines a generative AI model system that uses a local database to store personal information and restricts external transmission with an emotion engine that recognizes user emotions. Specifically, it performs clustering based on user feature data and builds a federated learning model using data collected from multiple users. It also has a management function that transmits the learning parameters of the federated learning model to an external server, performs data sovereignty management of personal information, and provides optimal services while protecting user privacy. The system also has the function of recognizing the user's emotional state using the emotion engine and adjusting services.

[0172] First, the server uses a local database to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. The server checks the existence of a local database file during initialization, and creates a new one if it does not exist. This file stores each user's personal information and usage data.

[0173] The server then performs clustering based on the user feature data. The feature data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm to classify users with similar features into the same group. This makes it easier to analyze the feature data and provide customized services.

[0174] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only learning parameters are sent to the external server. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leakage.

[0175] Furthermore, by combining it with an emotion engine, it becomes possible to recognize the user's emotional state. The server utilizes the emotion engine to identify emotions from user input data, voice data, and image data. For example, when a user types a text message, the server reads the user's emotions from the context and expression and stores them as emotion data. This emotion data is added to the user's feature data to improve the accuracy of clustering and federated learning models.

[0176] As a concrete example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and emotional state and clusters users with similar patterns. It then inputs all users' purchasing data and emotional data into a federated learning algorithm to create an integrated model. This model can then provide optimized product recommendations for each individual user.

[0177] The emotion data recognized by the emotion engine complements the user's feature data and improves the accuracy of the clustering and federated learning models. The server utilizes this to provide detailed services tailored to the user's needs and status.

[0178] Finally, the server has the ability to adjust service content in real time. Based on the user's emotional state as recognized by the emotion engine, the server can change the content and delivery method of the service accordingly. For example, if the user is feeling stressed, the server can respond in line with the user's emotions by recommending content that has a relaxing effect.

[0179] The above is a specific implementation of the "MyData-GPT" system, which combines an emotion engine. This system makes it possible to provide highly accurate and optimized services to users while protecting personal information.

[0180] The processing flow will be explained below.

[0181] ---

[0182] Step 1:

[0183] When the server initializes the program, it checks for the existence of a local database file. If the file exists, the server reads it and loads the user data into local memory. If it does not exist, the server creates an empty database and creates a new file.

[0184] Step 2:

[0185] When the user provides new data, the server updates the local database by calling the update_local_db method. The data received from the user is merged with the existing data and updated to the latest state.

[0186] Step 3:

[0187] After updating the local database, the server calls the save_data method to save the database to a file. This process makes the user data persistent.

[0188] Step 4:

[0189] The emotion engine processes user input data, voice data, and image data to identify the user's emotional state. For example, it analyzes emotions from text input and generates emotion data.

[0190] Step 5:

[0191] The server performs clustering based on the user's feature data and emotion data. The server extracts the feature data and emotion data of all users and performs clustering using the KMeans algorithm. Each user is assigned to an appropriate cluster based on their feature data.

[0192] Step 6:

[0193] After clustering is complete, the server assigns each user a cluster ID, which allows users with similar characteristics to be grouped together for efficient data analysis and processing.

[0194] Step 7:

[0195] The server performs federated learning using all user data and emotional data. The server learns each user's data locally and aggregates the results to build an overall AI model. Individual user data is not transmitted externally, protecting privacy.

[0196] Step 8:

[0197] The server sends the learning parameters obtained as a result of federated learning to an external server, making it possible to use highly accurate models remotely while protecting personal information.

[0198] Step 9:

[0199] The server will manage data sovereignty and protect users' personal information from leaks to the outside world, by encrypting data, controlling access, and obtaining permission to use the data.

[0200] Step 10:

[0201] The server has management functions to provide optimized services while protecting the user's privacy, allowing the server to provide customized services according to the user's requests and needs.

[0202] Step 11:

[0203] The server adjusts the service content in real time based on the user's emotional state recognized by the emotion engine. For example, if the user is feeling stressed, it can respond in line with the user's emotions by recommending content that has a relaxing effect.

[0204] ---

[0205] The above is the specific processing flow of the "MyData-GPT" system, which combines an emotion engine. The roles of the server, device, and user are clarified, and the specific operations performed at each step are explained.

[0206] Example 2

[0207] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0208] In modern society, protecting users' personal information and providing an optimized user experience are urgent issues. However, existing systems face privacy risks due to the external transmission of personal information, and it is difficult to adjust services in real time based on the user's emotional state. A system that solves these issues and provides optimal services using emotion recognition while protecting user privacy is needed.

[0209] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0210] In this invention, the server includes means for storing personal information using a local database and restricting external transmission, means for clustering based on user feature data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external server, means for utilizing an emotion engine that recognizes the user's emotional state, and means for adjusting service content in real time based on the emotional state. This makes it possible to provide optimal services based on the user's emotional state while protecting the user's personal information.

[0211] A "local database" is a database used to store a user's personal information on the device and restrict its transmission to external devices.

[0212] "Clustering" is a method of grouping users with similar characteristics based on user characteristic data.

[0213] A "federated learning model" is a machine learning model that is constructed by learning data collected from multiple users locally on each device and aggregating only the learning results (parameters).

[0214] "Learning parameters" are data that indicate the results of local learning by each device in a federated learning model, and by aggregating these data, the accuracy of the model is improved.

[0215] An "emotion engine" is a technology that analyzes user input data (text, voice, images, etc.) and recognizes their emotions.

[0216] "Emotional State" means the current mental and emotional state identified from data entered by the User.

[0217] "Real-time adjustment of service content" is a function that instantly changes the content and method of the service provided based on the user's emotional state recognized by the emotion engine.

[0218] "Data sovereignty management" is a management method that ensures that users' personal information is used under their own control and prevents it from leaking to the outside.

[0219] "Privacy protection" refers to measures to prevent users' personal information and emotional data from being misused by third parties.

[0220] MODE FOR CARRYING OUT THE INVENTION

[0221] This invention is a system that provides advanced services while protecting privacy by storing users' personal information in a local database and restricting its transmission to external parties. This system combines clustering based on user feature data, the construction of a federated learning model, and an emotion recognition engine to adjust service content in real time.

[0222] Data storage method

[0223] First, when the system starts up, the server checks for the existence of a local database, and if the file does not exist, it creates a new one. This database stores each user's personal information and usage data. Specifically, it uses a database management system such as SQLite. For example, user information is stored in a file named "users.db."

[0224] Clustering Method

[0225] Next, the server collects feature data for clustering based on user behavior patterns and attribute information. This process uses machine learning libraries such as scikit-learn. Specifically, it uses the K-means algorithm and classifies users into five groups by setting "n_clusters=5," for example.

[0226] Federated Learning Methods

[0227] In the federated learning step, the server collects data from multiple devices and performs local learning on each device. Each device performs learning using, for example, TensorFlow Federated, and sends the resulting learning parameters to the server. The server aggregates these parameters and builds an overall AI model.

[0228] Emotion recognition method

[0229] When a user inputs text, voice, and image data, the server uses an emotion engine to recognize the user's emotional state from these input data. Specifically, the emotion recognition engine uses a general emotion recognition API. For example, if a user inputs a message such as "I had so much fun today!", the server identifies the emotion "joy" from the context and stores it in a database.

[0230] Real-time adjustment method

[0231] The server adjusts the services it provides in real time based on the user's emotional state as recognized by the emotion engine. For example, if a user is recognized as feeling "stressed," it will recommend music and videos with a relaxing effect. As a specific example, a general music streaming API can be used for music recommendations. The server reflects these recommendations on the user's device in real time.

[0232] Examples and prompts

[0233] As a specific example, the server performs federated learning using purchasing data and emotional data collected from each user. For example, it analyzes each user's purchasing patterns and emotional state and clusters users with similar patterns. Then, it inputs all users' purchasing data and emotional data into a federated learning algorithm to create an integrated model. This model then makes product recommendations optimized for each individual user.

[0234] Example prompt sentence:

[0235] "Use federated learning to build the optimal product recommendation model using user purchasing data and sentiment data."

[0236] "Recognize emotions from user text input and tailor your service based on those emotions."

[0237] As described above, the system according to the present invention provides optimal services based on the user's emotions while protecting personal information.

[0238] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0239] How the system handles programs

[0240] Step 1:

[0241] The server initializes the local database.

[0242] Input: System startup state

[0243] BEHAVIOR: The server checks for the existence of an SQLite database file called "users.db" and creates it if it doesn't exist. It then connects to the database using the sqlite3 library and sets up the users table.

[0244] Output: Initialized local database

[0245] Step 2:

[0246] The server collects user feature data and performs clustering.

[0247] Input: User behavior patterns and attribute information (e.g., age, gender, purchase history)

[0248] How it works: The server uses the scikit-learn library to perform K-means clustering on the feature data. For example, run kmeans = KMeans(n_clusters=5) to classify users into 5 groups. After clustering, it saves a list of users classified into groups.

[0249] Output: Clustered user data

[0250] Step 3:

[0251] The server collects data for implementing federated learning and performs the learning.

[0252] Input: User data collected from each device

[0253] How it works: The server prepares the data to be handed over to each device, and uses the TensorFlow Federated library to perform local training. The trained parameters on each device are sent to the server, which aggregates these parameters and builds the overall AI model.

[0254] Output: AI model with aggregated learning parameters

[0255] Step 4:

[0256] The server uses an emotion engine to recognize the user's emotional state.

[0257] Input: User input data (text, audio, images, etc.)

[0258] How it works: The server uses an emotion recognition engine to identify the user's emotions. For example, using an emotion recognition API, it analyzes the text data "I had so much fun today!", identifies the emotional state "joy," and stores it in the database.

[0259] Output: Recognized emotion data

[0260] Step 5:

[0261] The server adjusts the service content in real time based on the user's emotional state.

[0262] Input: Recognized emotion data

[0263] How it works: The server adjusts the content of the service provided to the user based on the emotion data. For example, if the server detects that the user is feeling stressed, it will use a music streaming API to retrieve an appropriate relaxation playlist and recommend it to the user's device to provide relaxing music.

[0264] Output: Tailored service delivery

[0265] Step 6:

[0266] The server monitors the overall system function, gathering feedback and making improvements as needed.

[0267] Input: Usage of the provided service and user feedback

[0268] How it works: The server monitors user feedback and usage data to analyze the system's performance and accuracy, which in turn allows it to make improvements to improve the accuracy of the federated learning model and emotion recognition engine.

[0269] Output: New version of improved system functionality

[0270] Through this series of steps, the system can provide advanced and optimized services based on user sentiment while protecting user privacy.

[0271] (Application example 2)

[0272] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0273] Conventional service provision systems not only pose privacy risks by transmitting users' personal information to external parties, but also lack the ability to adapt to users' emotional states. This makes it difficult to provide appropriate services that reflect users' real-time emotions. Furthermore, in today's world where both personal information protection and service quality are required, providing optimal services while ensuring user privacy is a challenge.

[0274] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0275] In this invention, the server includes means for storing personal information using a local database and restricting external transmission, means for clustering based on user feature data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external server, means for analyzing the emotional state of the user in real time using an emotion recognition engine, and means for dynamically adjusting the content and method of service provision based on the analysis results, thereby enabling the provision of real-time services according to emotions while protecting user privacy.

[0276] A "local database" is a database that stores users' personal information and restricts its transmission to external parties.

[0277] "Clustering" is a technique for grouping multiple users based on their characteristic data.

[0278] A "federated learning model" is a model that is constructed using data collected from multiple users and transmits only the learning parameters to an external server.

[0279] A "learning parameter" is a value that indicates the characteristics of the data used when constructing an associative learning model.

[0280] The "external server" is a server that receives the learning parameters and performs analysis and processing.

[0281] An "emotion recognition engine" is a system for analyzing a user's emotional state in real time.

[0282] "Dynamic adjustment" is the process of changing service content and delivery methods in real time based on analysis results.

[0283] "Protecting privacy" means managing users' personal information to prevent it from being leaked to the outside without permission.

[0284] "User characteristic data" refers to data such as user behavior patterns and attribute information.

[0285] "Real-time analysis" means instantly determining the user's emotional state on the spot.

[0286] This invention is a system that analyzes user emotions in real time in brick-and-mortar stores and dynamically adjusts service content based on that data. It uses a server and smart glasses in combination to provide optimal services while protecting user privacy.

[0287] The server uses a local database to store personal information and restricts external transmission. This local database stores each user's personal information and usage data. Clustering methods are used to classify users based on their feature data, and a federated learning model is constructed using data collected from multiple users. Learning parameters are sent to an external server, but the personal information itself is not sent externally, protecting privacy.

[0288] The smart glasses have a built-in camera that acts as an input device for analyzing the user's facial expressions and voice. The emotion recognition engine uses this camera to analyze the user's emotions in real time. The analyzed emotion data is sent to the server, which dynamically adjusts the service based on the user's interests and state.

[0289] Specifically, it works as follows:

[0290] 1. The server analyzes the user's emotional state using image data acquired from the smart glasses.

[0291] 2. The analyzed emotion data is stored in a local database and classified into groups of users with similar emotion patterns using clustering methods.

[0292] 3. The data analyzed by the emotion recognition engine is reflected in the overall AI model using a federated learning model.

[0293] 4. Based on this model, the server recommends the most suitable products and services for each user and adjusts the delivery method in real time.

[0294] The main hardware used is smart glasses and a server. The software uses OpenCV and Keras (with TensorFlow backend). OpenCV functions as a face detector, and Keras is used for the sentiment analysis model. Data is processed in real time, and appropriate services are automatically provided based on the analysis results.

[0295] For example, if a store staff member is wearing smart glasses and a camera captures a customer's facial expression, the image will be used by an emotion recognition engine to analyze the customer's emotional state. If the customer is feeling stressed, the system will recommend products that have a relaxing effect. This will improve the customer experience and enhance the quality of service at the store.

[0296] Examples of prompts include:

[0297] "What emotional state is this customer in right now?"

[0298] "Provide product recommendations that match customer emotions."

[0299] This will realize a system that can provide optimal services based on emotions in real time while protecting user privacy.

[0300] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0301] Step 1:

[0302] The smart glasses capture the user's facial image using the built-in camera. The image data obtained from the camera is the input. This image data is passed to the next processing step.

[0303] Step 2:

[0304] The server receives the image data and performs face detection using OpenCV. Specifically, it performs grayscale conversion and histogram equalization on the image data, and inputs it into a face detector. The face detector identifies the face in the image and outputs its coordinate information and the face image.

[0305] Step 3:

[0306] The server inputs the facial image obtained from the face detector into the emotion recognition engine. This facial image is passed to a pre-trained Keras emotion recognition model for emotion analysis. The emotion recognition model classifies the emotional state (e.g., anger, happiness, surprise, etc.) from the input facial image and outputs the result as emotion data.

[0307] Step 4:

[0308] The server receives emotion data from the emotion recognition engine and stores it in a local database. The local database classifies this emotion data using a clustering method and reconstructs it as user feature data. This allows emotion patterns for each individual user to be stored in the database.

[0309] Step 5:

[0310] The server updates the federated learning model using feature data collected from multiple users based on the clustered emotion data. This federated learning model trains locally using each user's data and sends the model parameters obtained as a result of the training to an external server. The external server receives this and updates the overall AI model.

[0311] Step 6:

[0312] The server receives the updated AI model returned from the external server and combines it with the emotional data to recommend the most suitable products and services for the user. This recommendation information is notified to the staff in real time via the smart glasses. For example, a message such as "Please recommend a relaxing aroma" may appear on the display of the staff member's smart glasses.

[0313] Step 7:

[0314] Store staff can provide dynamically tailored services to users based on the recommendations displayed on the smart glasses. This process enables real-time emotion recognition and service adjustment, improving user satisfaction.

[0315] Through these steps, a system will be realized in which the server, smart glasses, emotion recognition engine, federated learning model, etc. work together to provide optimal services to users.

[0316] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0317] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0318] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0319] [Second embodiment]

[0320] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0321] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0322] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0323] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0324] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0325] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0326] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0327] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0328] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0329] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0330] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0331] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0332] ---

[0333] This invention relates to a generative AI model system that uses a local database to store personal information and restricts its transmission to external servers. Specifically, it performs clustering based on user feature data, builds a federated learning model using data collected from multiple users, and transmits the learning parameters to an external server. The invention also provides a means for data sovereignty management of personal information and a management function for providing optimal services while protecting user privacy.

[0334] First, the server uses a local database to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. The server checks the existence of a local database file during initialization, and creates a new one if it does not exist. This file stores each user's personal information and usage data.

[0335] The server then performs clustering based on the user feature data. The feature data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm to classify users with similar features into the same group. This makes it easier to analyze the feature data and provide customized services.

[0336] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only learning parameters are sent to the external server. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leakage.

[0337] As a specific example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and clusters users with similar purchasing patterns. It then inputs all users' purchasing data into a federated learning algorithm to create an integrated model. This model can then provide optimized product recommendations for each individual user.

[0338] From the perspective of privacy management, the server also has a means of data sovereignty management. This is a function to strictly manage personal information so that it is not leaked to the outside. Data sovereignty management includes data encryption, access control, and obtaining usage permission. This allows users to safely manage how their data is used.

[0339] Furthermore, the server has management functions to provide optimal services while protecting users' privacy. These management functions allow the server to provide services tailored to users' requests and needs. For example, the server can provide personalized suggestions and advice by taking into account users' usage history and feedback.

[0340] As described above, the present invention provides a system that provides high-performance generative AI models while preventing the leakage of personal information. This allows companies to incorporate generative AI models into their business with peace of mind, and users can use them with peace of mind, knowing that their privacy is protected.

[0341] The processing flow will be explained below.

[0342] ---

[0343] Step 1:

[0344] When the server initializes the program, it checks for the existence of a local database file. If the file exists, the server reads it and loads the user data into local memory. If it does not exist, the server creates an empty database and creates a new file.

[0345] Step 2:

[0346] If the user provides new data, the server updates the local database by calling the update_local_db method. During this time, the data received from the user is merged with the existing data and updated to the latest state.

[0347] Step 3:

[0348] After the server updates the local database, it calls the save_data method to save the database to a file, which makes the user data persistent.

[0349] Step 4:

[0350] The server performs clustering based on the user feature data. To do this, the server takes all users' feature data and performs clustering using the KMeans algorithm. Each user is assigned to an appropriate cluster based on their feature data.

[0351] Step 5:

[0352] After clustering is complete, the server assigns each user a cluster ID, which allows users with similar characteristics to be grouped together for efficient data analysis and processing.

[0353] Step 6:

[0354] The server uses all user data to perform federated learning. Using data collected from each user, it locally trains a model and aggregates the results to build an overall AI model. Individual user data is not transmitted externally, protecting privacy.

[0355] Step 7:

[0356] The server sends the learning parameters obtained as a result of federated learning to an external server, which makes it possible to remotely utilize more accurate models while protecting personal information.

[0357] Step 8:

[0358] The server will manage data sovereignty and protect users' personal information from leaks to the outside world, by encrypting data, controlling access, and obtaining permission to use the data.

[0359] Step 9:

[0360] The server has management functions to provide optimized services while protecting the user's privacy, allowing the server to provide customized services according to the user's requests and needs.

[0361] ---

[0362] The above is the specific processing flow of the "MyData-GPT" system. The roles of the server, terminal, and user are clearly distinguished, and the specific operations performed at each step are explained.

[0363] Example 1

[0364] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0365] In modern information systems, users' personal information is collected by many service providers, increasing the possibility of it being leaked to external parties. This increases the risk of user privacy being violated. It is also difficult to effectively utilize individual data to create advanced AI models. Furthermore, there is insufficient data authority management for personal information, making it difficult to use services with peace of mind.

[0366] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0367] In this invention, the server includes means for storing personal information using local storage and restricting external transmission, means for clustering based on user attribute data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external computer, and means for aggregating the learning parameters and updating the final model. This makes it possible to construct a highly accurate AI model without leaking personal information to the outside, and to provide optimal services while protecting user privacy.

[0368] "Local storage" refers to data storage that is stored directly on the user's device and is a storage device for holding data that is not transferred to an external server.

[0369] "Personal Information" means identifiable information about a user, including name, address, telephone number, email address, purchase history, etc.

[0370] "Clustering" is a machine learning algorithm for analyzing a collection of data and grouping data with similar attributes.

[0371] "Federated learning" is a technology that performs partial learning on each terminal without collecting data from multiple distributed terminals, and then collects only the resulting learning parameters to update the overall model.

[0372] "Learning parameters" are internal numerical information that is adjusted by training a machine learning model, and refer to coefficients and biases that improve the model's predictive accuracy.

[0373] An "external computer" is a computing resource for a server or cloud service that is located in a location separate from the user's device or local storage.

[0374] "Data Rights Management" is a security measure that manages access rights and usage permissions for personal information, allowing users to control how their data is used.

[0375] "Management functions for providing optimal services while protecting privacy" refers to system functions that analyze collected data and provide customized services and recommendations tailored to each user while protecting the user's privacy.

[0376] This invention relates to a generative AI model system that uses local storage to store personal information and restricts its external transmission. Specifically, it performs clustering based on user attribute data, builds a federated learning model using data collected from multiple users, and transmits the learning parameters to an external computer. The invention also includes a means for managing data authority for personal information and a management function for providing optimal services while protecting user privacy.

[0377] First, the server uses local storage to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. When the server is initialized, it checks for the existence of a local storage file, and if the file does not exist, it creates a new one. This file stores each user's personal information and usage data. For example, a database file called "user_data.db" is created using SQLite, and a "users" table and a "user_logs" table are placed in it.

[0378] Next, the server performs clustering based on the user's attribute data. The attribute data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm, specifically K-means clustering, to classify users with similar characteristics into the same group. This makes it easier to analyze based on the attribute data and enables the provision of customized services.

[0379] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only the learning parameters are sent to the external computer. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leaks. Specifically, learning is carried out using machine learning frameworks such as Pytorch and TensorFlow.

[0380] As a specific example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and clusters users with similar purchasing patterns. It then inputs the purchasing data of all users into a federated learning algorithm to create an integrated model. This model can then make product recommendations optimized for each individual user.

[0381] From the perspective of privacy management, the server also has a means for data permission management. This is a function for strictly managing personal information to prevent it from leaking to the outside. Data permission management includes data encryption, access control, and obtaining permission to use. This allows users to safely manage how their data is used.

[0382] Furthermore, the server has a management function to provide optimal services while protecting the user's privacy. This management function allows the server to provide services tailored to the user's requests and needs. For example, it can provide personalized suggestions and advice based on the user's usage history and feedback.

[0383] An example prompt is:

[0384] "This system will recommend the best products based on your recent purchase history. Would you be interested in any of these products?"

[0385] As described above, the present invention provides a system that provides high-performance generative AI models while preventing the leakage of personal information. This allows companies to incorporate generative AI models into their business operations with peace of mind, and users can use them with peace of mind, knowing that their privacy is protected.

[0386] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0387] Step 1: Initialize and configure the local database

[0388] During system initialization, the server checks for the existence of a local database file. As input, it references the path to the database file on the file system. If the file does not exist, the server creates a new database file and initializes the appropriate tables. The database contains personal information for each user. As output, it creates an SQLite database called "user_data.db" and the "users" and "user_logs" tables within it.

[0389] Specifically, the server performs the following operations:

[0390] Checking for the existence of "user_data.db"

[0391] Creating a database if it does not exist

[0392] Initializing the "users" and "user_logs" tables

[0393] Step 2: Collect user data

[0394] The device collects data according to the user's operations. As input, the user's operation log and input data are acquired in real time. This data is stored in a local database. As output, the input data is added to the "user_logs" table.

[0395] Specifically, the device performs the following operations:

[0396] Monitoring user clicks and input information

[0397] Insert the obtained data into a local database

[0398] Step 3: Clustering feature data

[0399] The server periodically clusters user data. User attribute data obtained from a local database is used as input. The server uses this data to run a machine learning algorithm (e.g., K-means clustering) to group users. The output is the clustering results.

[0400] Specifically, the server performs the following operations:

[0401] Execute "SELECT FROM user_logs WHERE timestamp > {last_scan_timestamp}"

[0402] Extracting attribute data and calculating features

[0403] Clustering with machine learning algorithms

[0404] Step 4: Running federated learning

[0405] The server then starts federated learning based on the data collected from all users. The local data of each device is used as input. The device uses that data to train a model locally and sends the training parameters to the server. The server integrates the collected parameters to update the final AI model. The output is an updated AI model.

[0406] Specifically, the server and the device perform the following operations:

[0407] Learning task distribution from the server

[0408] On-device model training (e.g., Pytorch or TensorFlow)

[0409] Sending learning parameters and aggregating them on the server

[0410] Step 5: Optimized service delivery

[0411] Finally, the server uses the updated AI model to provide the optimal service to the user. The inputs are the updated AI model and the user's usage history and feedback. The server generates personalized recommendations and proposals based on this and sends them to the device. The output is optimized recommendation data.

[0412] Specifically, the server and the device perform the following operations:

[0413] Data analysis using updated AI models

[0414] Generating recommendations

[0415] Sending recommendation data to devices

[0416] Viewing recommendations on your device

[0417] (Application example 1)

[0418] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0419] Conventional electronic payment services often do not adequately protect privacy when transmitting users' purchase history and personal information to external parties, resulting in stress for users and the risk of information leaks. Furthermore, they lack functionality for providing personalized promotions and offers, leaving the user experience unsatisfactory. The purpose of this invention is to solve these problems and provide a safe and effective personalized service.

[0420] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0421] In this invention, the server includes a means for storing personal information using a local database and restricting transmission to the outside, a means for clustering based on user feature data, a means for building a federated learning model using data collected from multiple users, and a smartphone application including a management function for providing personalized promotions and offers, thereby enabling the provision of optimized promotions and offers while protecting user privacy.

[0422] A "local database" is a database that protects privacy by storing a user's personal information on the device and restricting its transmission to the outside.

[0423] "Personal information" refers to a user's unique identification information, purchasing history, behavioral patterns, etc.

[0424] "Feature data" is data that includes user behavior patterns and attribute information, and is used for clustering.

[0425] "Clustering" is the process of grouping users with similar characteristics based on their feature data.

[0426] A "federated learning model" is a machine learning model constructed using data collected from multiple users, in which only the learning parameters are sent to an external server.

[0427] "Learning parameters" are parameters sent to an external server to improve the accuracy of the model in federated learning.

[0428] The "external server" is a server that receives the learning parameters of the federated learning model and constructs an integrated model.

[0429] "Personalized promotions and offers" are promotions and offers that provide personalized suggestions and benefits to individual users based on their user characteristics data.

[0430] A "smartphone application" is an application that runs on a smartphone, handles users' personal information, and provides users with personalized services through clustering and federated learning.

[0431] "Data sovereignty management" refers to the means of managing users' personal information to prevent it from being leaked to the outside, and includes encryption and access control.

[0432] "Privacy protection" refers to measures to prevent users' personal information from being misused or leaked.

[0433] This invention specifically describes a system for protecting users' personal information and providing personalized promotions in electronic payment services. This system has the functions of storing personal information using a local database and restricting external transmission, clustering users based on feature data, and building a federated learning model.

[0434] First, the device (e.g., a smartphone) stores the user's purchasing history and behavioral patterns in a local database. This database is built using SQLite, and the user's personal information is securely encrypted and stored using the Fernet encryption method.

[0435] The device then performs clustering based on the user feature data, using the KMeans algorithm to group similar users based on their behavioral patterns and attribute information.

[0436] In the federated learning step, each user's data is trained locally and the training parameters are sent to an external server. TensorFlow Federated (TFF) is used to build the federated learning model. This ensures privacy by keeping individual user data local and only sending the training parameters to the external server.

[0437] Furthermore, the terminal will utilize the constructed model to provide users with personalized promotions and offers, which will suggest optimal offers and deals for them, improving their user experience.

[0438] For example, when a user records their past purchase history using a smartphone application, that data is encrypted and stored in a local database. Using a model built through federated learning, the user is then presented with "recommended coupons." These coupons are personalized based on the user's purchase history and behavioral patterns.

[0439] Example prompt sentence:

[0440] I want to create a feature that uses a user's past purchase history data to display coupon recommendations while maintaining privacy. How can I use federated learning and clustering to provide personalized promotions?

[0441] The system is realized using a smartphone, a local database (SQLite), an encryption library (Fernet), a clustering algorithm (KMeans), and a federated learning framework (TensorFlow Federated).

[0442] As described above, this invention is a system that provides safe and effective personal information protection and personalized services through a smartphone application. This system allows users to use electronic payment services with peace of mind, and allows companies to provide highly reliable services.

[0443] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0444] Step 1:

[0445] Initializing the local database

[0446] The device initializes a local database to store the user's personal information. Specifically, it uses SQLite and creates a new database file if one does not exist. It checks the state of the local file system as input and generates a database file to store the user information as output.

[0447] Step 2:

[0448] Data encryption and storage

[0449] The device encrypts the user's purchase history and behavioral patterns and stores them in a local database. Specifically, it uses the Fernet encryption library to encrypt user data. It receives unencrypted purchase history and behavioral patterns as input and stores the encrypted data in the local database as output.

[0450] Step 3:

[0451] Feature data collection and clustering

[0452] The device collects user feature data and performs clustering using the KMeans algorithm. Collected user behavior patterns and attribute information are used as input, and users classified into clusters are output.

[0453] Step 4:

[0454] Implementing federated learning

[0455] The device learns each user's data locally and sends the learning parameters to an external server. Specifically, federated learning is performed using TensorFlow Federated (TFF). Each user's purchase history data is provided as input, and the learning parameters of the federated learning model are generated as output and sent to the external server.

[0456] Step 5:

[0457] Model integration with external servers

[0458] The server receives the learning parameters sent from the devices and constructs an integrated federated learning model. Learning parameters from multiple devices are received as input, and an integrated, high-precision federated learning model is generated as output.

[0459] Step 6:

[0460] Providing personalized service

[0461] The device uses the constructed federated learning model to provide personalized promotions and offers to the user. Using the federated model and user feature data as input, the device displays promotions and offers optimized for the user as output.

[0462] In this way, the system protects users' personal information while providing personalized promotions and offers that are most relevant to them.

[0463] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0464] ---

[0465] This invention combines a generative AI model system that uses a local database to store personal information and restricts external transmission with an emotion engine that recognizes user emotions. Specifically, it performs clustering based on user feature data and builds a federated learning model using data collected from multiple users. It also has a management function that transmits the learning parameters of the federated learning model to an external server, performs data sovereignty management of personal information, and provides optimal services while protecting user privacy. The system also has the function of recognizing the user's emotional state using the emotion engine and adjusting services.

[0466] First, the server uses a local database to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. The server checks the existence of a local database file during initialization, and creates a new one if it does not exist. This file stores each user's personal information and usage data.

[0467] The server then performs clustering based on the user feature data. The feature data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm to classify users with similar features into the same group. This makes it easier to analyze the feature data and provide customized services.

[0468] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only learning parameters are sent to the external server. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leakage.

[0469] Furthermore, by combining it with an emotion engine, it becomes possible to recognize the user's emotional state. The server utilizes the emotion engine to identify emotions from user input data, voice data, and image data. For example, when a user types a text message, the server reads the user's emotions from the context and expression and stores them as emotion data. This emotion data is added to the user's feature data to improve the accuracy of clustering and federated learning models.

[0470] As a concrete example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and emotional state and clusters users with similar patterns. It then inputs all users' purchasing data and emotional data into a federated learning algorithm to create an integrated model. This model can then provide optimized product recommendations for each individual user.

[0471] The emotion data recognized by the emotion engine complements the user's feature data and improves the accuracy of the clustering and federated learning models. The server utilizes this to provide detailed services tailored to the user's needs and status.

[0472] Finally, the server has the ability to adjust service content in real time. Based on the user's emotional state as recognized by the emotion engine, the server can change the content and delivery method of the service accordingly. For example, if the user is feeling stressed, the server can respond in line with the user's emotions by recommending content that has a relaxing effect.

[0473] The above is a specific implementation of the "MyData-GPT" system, which combines an emotion engine. This system makes it possible to provide highly accurate and optimized services to users while protecting personal information.

[0474] The processing flow will be explained below.

[0475] ---

[0476] Step 1:

[0477] When the server initializes the program, it checks for the existence of a local database file. If the file exists, the server reads it and loads the user data into local memory. If it does not exist, the server creates an empty database and creates a new file.

[0478] Step 2:

[0479] When the user provides new data, the server updates the local database by calling the update_local_db method. The data received from the user is merged with the existing data and updated to the latest state.

[0480] Step 3:

[0481] After updating the local database, the server calls the save_data method to save the database to a file. This process makes the user data persistent.

[0482] Step 4:

[0483] The emotion engine processes user input data, voice data, and image data to identify the user's emotional state. For example, it analyzes emotions from text input and generates emotion data.

[0484] Step 5:

[0485] The server performs clustering based on the user's feature data and emotion data. The server extracts the feature data and emotion data of all users and performs clustering using the KMeans algorithm. Each user is assigned to an appropriate cluster based on their feature data.

[0486] Step 6:

[0487] After clustering is complete, the server assigns each user a cluster ID, which allows users with similar characteristics to be grouped together for efficient data analysis and processing.

[0488] Step 7:

[0489] The server performs federated learning using all user data and emotional data. The server learns each user's data locally and aggregates the results to build an overall AI model. Individual user data is not transmitted externally, protecting privacy.

[0490] Step 8:

[0491] The server sends the learning parameters obtained as a result of federated learning to an external server, making it possible to use highly accurate models remotely while protecting personal information.

[0492] Step 9:

[0493] The server will manage data sovereignty and protect users' personal information from leaks to the outside world, by encrypting data, controlling access, and obtaining permission to use the data.

[0494] Step 10:

[0495] The server has management functions to provide optimized services while protecting the user's privacy, allowing the server to provide customized services according to the user's requests and needs.

[0496] Step 11:

[0497] The server adjusts the service content in real time based on the user's emotional state recognized by the emotion engine. For example, if the user is feeling stressed, it can respond in line with the user's emotions by recommending content that has a relaxing effect.

[0498] ---

[0499] The above is the specific processing flow of the "MyData-GPT" system, which combines an emotion engine. The roles of the server, device, and user are clarified, and the specific operations performed at each step are explained.

[0500] Example 2

[0501] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0502] In modern society, protecting users' personal information and providing an optimized user experience are urgent issues. However, existing systems face privacy risks due to the external transmission of personal information, and it is difficult to adjust services in real time based on the user's emotional state. A system that solves these issues and provides optimal services using emotion recognition while protecting user privacy is needed.

[0503] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0504] In this invention, the server includes means for storing personal information using a local database and restricting external transmission, means for clustering based on user feature data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external server, means for utilizing an emotion engine that recognizes the user's emotional state, and means for adjusting service content in real time based on the emotional state. This makes it possible to provide optimal services based on the user's emotional state while protecting the user's personal information.

[0505] A "local database" is a database used to store a user's personal information on the device and restrict its transmission to external devices.

[0506] "Clustering" is a method of grouping users with similar characteristics based on user characteristic data.

[0507] A "federated learning model" is a machine learning model that is constructed by learning data collected from multiple users locally on each device and aggregating only the learning results (parameters).

[0508] "Learning parameters" are data that indicate the results of local learning by each device in a federated learning model, and by aggregating these data, the accuracy of the model is improved.

[0509] An "emotion engine" is a technology that analyzes user input data (text, voice, images, etc.) and recognizes their emotions.

[0510] "Emotional State" means the current mental and emotional state identified from data entered by the User.

[0511] "Real-time adjustment of service content" is a function that instantly changes the content and method of the service provided based on the user's emotional state recognized by the emotion engine.

[0512] "Data sovereignty management" is a management method that ensures that users' personal information is used under their own control and prevents it from leaking to the outside.

[0513] "Privacy protection" refers to measures to prevent users' personal information and emotional data from being misused by third parties.

[0514] MODE FOR CARRYING OUT THE INVENTION

[0515] This invention is a system that provides advanced services while protecting privacy by storing users' personal information in a local database and restricting its transmission to external parties. This system combines clustering based on user feature data, the construction of a federated learning model, and an emotion recognition engine to adjust service content in real time.

[0516] Data storage method

[0517] First, when the system starts up, the server checks for the existence of a local database, and if the file does not exist, it creates a new one. This database stores each user's personal information and usage data. Specifically, it uses a database management system such as SQLite. For example, user information is stored in a file named "users.db."

[0518] Clustering Method

[0519] Next, the server collects feature data for clustering based on user behavior patterns and attribute information. This process uses machine learning libraries such as scikit-learn. Specifically, it uses the K-means algorithm and classifies users into five groups by setting "n_clusters=5," for example.

[0520] Federated Learning Methods

[0521] In the federated learning step, the server collects data from multiple devices and performs local learning on each device. Each device performs learning using, for example, TensorFlow Federated, and sends the resulting learning parameters to the server. The server aggregates these parameters and builds an overall AI model.

[0522] Emotion recognition method

[0523] When a user inputs text, voice, and image data, the server uses an emotion engine to recognize the user's emotional state from these input data. Specifically, the emotion recognition engine uses a general emotion recognition API. For example, if a user inputs a message such as "I had so much fun today!", the server identifies the emotion "joy" from the context and stores it in a database.

[0524] Real-time adjustment method

[0525] The server adjusts the services it provides in real time based on the user's emotional state as recognized by the emotion engine. For example, if a user is recognized as feeling "stressed," it will recommend music and videos with a relaxing effect. As a specific example, a general music streaming API can be used for music recommendations. The server reflects these recommendations on the user's device in real time.

[0526] Examples and prompts

[0527] As a specific example, the server performs federated learning using purchasing data and emotional data collected from each user. For example, it analyzes each user's purchasing patterns and emotional state and clusters users with similar patterns. Then, it inputs all users' purchasing data and emotional data into a federated learning algorithm to create an integrated model. This model then makes product recommendations optimized for each individual user.

[0528] Example prompt sentence:

[0529] "Use federated learning to build the optimal product recommendation model using user purchasing data and sentiment data."

[0530] "Recognize emotions from user text input and tailor your service based on those emotions."

[0531] As described above, the system according to the present invention provides optimal services based on the user's emotions while protecting personal information.

[0532] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0533] How the system handles programs

[0534] Step 1:

[0535] The server initializes the local database.

[0536] Input: System startup state

[0537] BEHAVIOR: The server checks for the existence of an SQLite database file called "users.db" and creates it if it doesn't exist. It then connects to the database using the sqlite3 library and sets up the users table.

[0538] Output: Initialized local database

[0539] Step 2:

[0540] The server collects user feature data and performs clustering.

[0541] Input: User behavior patterns and attribute information (e.g., age, gender, purchase history)

[0542] How it works: The server uses the scikit-learn library to perform K-means clustering on the feature data. For example, run kmeans = KMeans(n_clusters=5) to classify users into 5 groups. After clustering, it saves a list of users classified into groups.

[0543] Output: Clustered user data

[0544] Step 3:

[0545] The server collects data for implementing federated learning and performs the learning.

[0546] Input: User data collected from each device

[0547] How it works: The server prepares the data to be handed over to each device, and uses the TensorFlow Federated library to perform local training. The trained parameters on each device are sent to the server, which aggregates these parameters and builds the overall AI model.

[0548] Output: AI model with aggregated learning parameters

[0549] Step 4:

[0550] The server uses an emotion engine to recognize the user's emotional state.

[0551] Input: User input data (text, audio, images, etc.)

[0552] How it works: The server uses an emotion recognition engine to identify the user's emotions. For example, using an emotion recognition API, it analyzes the text data "I had so much fun today!", identifies the emotional state "joy," and stores it in the database.

[0553] Output: Recognized emotion data

[0554] Step 5:

[0555] The server adjusts the service content in real time based on the user's emotional state.

[0556] Input: Recognized emotion data

[0557] How it works: The server adjusts the content of the service provided to the user based on the emotion data. For example, if the server detects that the user is feeling stressed, it will use a music streaming API to retrieve an appropriate relaxation playlist and recommend it to the user's device to provide relaxing music.

[0558] Output: Tailored service delivery

[0559] Step 6:

[0560] The server monitors the overall system function, gathering feedback and making improvements as needed.

[0561] Input: Usage of the provided service and user feedback

[0562] How it works: The server monitors user feedback and usage data to analyze the system's performance and accuracy, which in turn allows it to make improvements to improve the accuracy of the federated learning model and emotion recognition engine.

[0563] Output: New version of improved system functionality

[0564] Through this series of steps, the system can provide advanced and optimized services based on user sentiment while protecting user privacy.

[0565] (Application example 2)

[0566] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0567] Conventional service provision systems not only pose privacy risks by transmitting users' personal information to external parties, but also lack the ability to adapt to users' emotional states. This makes it difficult to provide appropriate services that reflect users' real-time emotions. Furthermore, in today's world where both personal information protection and service quality are required, providing optimal services while ensuring user privacy is a challenge.

[0568] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0569] In this invention, the server includes means for storing personal information using a local database and restricting external transmission, means for clustering based on user feature data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external server, means for analyzing the emotional state of the user in real time using an emotion recognition engine, and means for dynamically adjusting the content and method of service provision based on the analysis results, thereby enabling the provision of real-time services according to emotions while protecting user privacy.

[0570] A "local database" is a database that stores users' personal information and restricts its transmission to external parties.

[0571] "Clustering" is a technique for grouping multiple users based on their characteristic data.

[0572] A "federated learning model" is a model that is constructed using data collected from multiple users and transmits only the learning parameters to an external server.

[0573] A "learning parameter" is a value that indicates the characteristics of the data used when constructing an associative learning model.

[0574] The "external server" is a server that receives the learning parameters and performs analysis and processing.

[0575] An "emotion recognition engine" is a system for analyzing a user's emotional state in real time.

[0576] "Dynamic adjustment" is the process of changing service content and delivery methods in real time based on analysis results.

[0577] "Protecting privacy" means managing users' personal information to prevent it from being leaked to the outside without permission.

[0578] "User characteristic data" refers to data such as user behavior patterns and attribute information.

[0579] "Real-time analysis" means instantly determining the user's emotional state on the spot.

[0580] This invention is a system that analyzes user emotions in real time in brick-and-mortar stores and dynamically adjusts service content based on that data. It uses a server and smart glasses in combination to provide optimal services while protecting user privacy.

[0581] The server uses a local database to store personal information and restricts external transmission. This local database stores each user's personal information and usage data. Clustering methods are used to classify users based on their feature data, and a federated learning model is constructed using data collected from multiple users. Learning parameters are sent to an external server, but the personal information itself is not sent externally, protecting privacy.

[0582] The smart glasses have a built-in camera that acts as an input device for analyzing the user's facial expressions and voice. The emotion recognition engine uses this camera to analyze the user's emotions in real time. The analyzed emotion data is sent to the server, which dynamically adjusts the service based on the user's interests and state.

[0583] Specifically, it works as follows:

[0584] 1. The server analyzes the user's emotional state using image data acquired from the smart glasses.

[0585] 2. The analyzed emotion data is stored in a local database and classified into groups of users with similar emotion patterns using clustering methods.

[0586] 3. The data analyzed by the emotion recognition engine is reflected in the overall AI model using a federated learning model.

[0587] 4. Based on this model, the server recommends the most suitable products and services for each user and adjusts the delivery method in real time.

[0588] The main hardware used is smart glasses and a server. The software uses OpenCV and Keras (with TensorFlow backend). OpenCV functions as a face detector, and Keras is used for the sentiment analysis model. Data is processed in real time, and appropriate services are automatically provided based on the analysis results.

[0589] For example, if a store staff member is wearing smart glasses and a camera captures a customer's facial expression, the image will be used by an emotion recognition engine to analyze the customer's emotional state. If the customer is feeling stressed, the system will recommend products that have a relaxing effect. This will improve the customer experience and enhance the quality of service at the store.

[0590] Examples of prompts include:

[0591] "What emotional state is this customer in right now?"

[0592] "Provide product recommendations that match customer emotions."

[0593] This will realize a system that can provide optimal services based on emotions in real time while protecting user privacy.

[0594] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0595] Step 1:

[0596] The smart glasses capture the user's facial image using the built-in camera. The image data obtained from the camera is the input. This image data is passed to the next processing step.

[0597] Step 2:

[0598] The server receives the image data and performs face detection using OpenCV. Specifically, it performs grayscale conversion and histogram equalization on the image data, and inputs it into a face detector. The face detector identifies the face in the image and outputs its coordinate information and the face image.

[0599] Step 3:

[0600] The server inputs the facial image obtained from the face detector into the emotion recognition engine. This facial image is passed to a pre-trained Keras emotion recognition model for emotion analysis. The emotion recognition model classifies the emotional state (e.g., anger, happiness, surprise, etc.) from the input facial image and outputs the result as emotion data.

[0601] Step 4:

[0602] The server receives emotion data from the emotion recognition engine and stores it in a local database. The local database classifies this emotion data using a clustering method and reconstructs it as user feature data. This allows emotion patterns for each individual user to be stored in the database.

[0603] Step 5:

[0604] The server updates the federated learning model using feature data collected from multiple users based on the clustered emotion data. This federated learning model trains locally using each user's data and sends the model parameters obtained as a result of the training to an external server. The external server receives this and updates the overall AI model.

[0605] Step 6:

[0606] The server receives the updated AI model returned from the external server and combines it with the emotional data to recommend the most suitable products and services for the user. This recommendation information is notified to the staff in real time via the smart glasses. For example, a message such as "Please recommend a relaxing aroma" may appear on the display of the staff member's smart glasses.

[0607] Step 7:

[0608] Store staff can provide dynamically tailored services to users based on the recommendations displayed on the smart glasses. This process enables real-time emotion recognition and service adjustment, improving user satisfaction.

[0609] Through these steps, a system will be realized in which the server, smart glasses, emotion recognition engine, federated learning model, etc. work together to provide optimal services to users.

[0610] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0611] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0612] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0613] [Third embodiment]

[0614] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0615] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0616] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0617] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0618] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0619] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0620] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0621] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0622] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0623] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0624] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0625] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0626] ---

[0627] This invention relates to a generative AI model system that uses a local database to store personal information and restricts its transmission to external servers. Specifically, it performs clustering based on user feature data, builds a federated learning model using data collected from multiple users, and transmits the learning parameters to an external server. The invention also provides a means for data sovereignty management of personal information and a management function for providing optimal services while protecting user privacy.

[0628] First, the server uses a local database to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. The server checks the existence of a local database file during initialization, and creates a new one if it does not exist. This file stores each user's personal information and usage data.

[0629] The server then performs clustering based on the user feature data. The feature data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm to classify users with similar features into the same group. This makes it easier to analyze the feature data and provide customized services.

[0630] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only learning parameters are sent to the external server. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leakage.

[0631] As a specific example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and clusters users with similar purchasing patterns. It then inputs all users' purchasing data into a federated learning algorithm to create an integrated model. This model can then provide optimized product recommendations for each individual user.

[0632] From the perspective of privacy management, the server also has a means of data sovereignty management. This is a function to strictly manage personal information so that it is not leaked to the outside. Data sovereignty management includes data encryption, access control, and obtaining usage permission. This allows users to safely manage how their data is used.

[0633] Furthermore, the server has management functions to provide optimal services while protecting users' privacy. These management functions allow the server to provide services tailored to users' requests and needs. For example, the server can provide personalized suggestions and advice by taking into account users' usage history and feedback.

[0634] As described above, the present invention provides a system that provides high-performance generative AI models while preventing the leakage of personal information. This allows companies to incorporate generative AI models into their business with peace of mind, and users can use them with peace of mind, knowing that their privacy is protected.

[0635] The processing flow will be explained below.

[0636] ---

[0637] Step 1:

[0638] When the server initializes the program, it checks for the existence of a local database file. If the file exists, the server reads it and loads the user data into local memory. If it does not exist, the server creates an empty database and creates a new file.

[0639] Step 2:

[0640] If the user provides new data, the server updates the local database by calling the update_local_db method. During this time, the data received from the user is merged with the existing data and updated to the latest state.

[0641] Step 3:

[0642] After the server updates the local database, it calls the save_data method to save the database to a file, which makes the user data persistent.

[0643] Step 4:

[0644] The server performs clustering based on the user feature data. To do this, the server takes all users' feature data and performs clustering using the KMeans algorithm. Each user is assigned to an appropriate cluster based on their feature data.

[0645] Step 5:

[0646] After clustering is complete, the server assigns each user a cluster ID, which allows users with similar characteristics to be grouped together for efficient data analysis and processing.

[0647] Step 6:

[0648] The server uses all user data to perform federated learning. Using data collected from each user, it locally trains a model and aggregates the results to build an overall AI model. Individual user data is not transmitted externally, protecting privacy.

[0649] Step 7:

[0650] The server sends the learning parameters obtained as a result of federated learning to an external server, which makes it possible to remotely utilize more accurate models while protecting personal information.

[0651] Step 8:

[0652] The server will manage data sovereignty and protect users' personal information from leaks to the outside world, by encrypting data, controlling access, and obtaining permission to use the data.

[0653] Step 9:

[0654] The server has management functions to provide optimized services while protecting the user's privacy, allowing the server to provide customized services according to the user's requests and needs.

[0655] ---

[0656] The above is the specific processing flow of the "MyData-GPT" system. The roles of the server, terminal, and user are clearly distinguished, and the specific operations performed at each step are explained.

[0657] Example 1

[0658] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0659] In modern information systems, users' personal information is collected by many service providers, increasing the possibility of it being leaked to external parties. This increases the risk of user privacy being violated. It is also difficult to effectively utilize individual data to create advanced AI models. Furthermore, there is insufficient data authority management for personal information, making it difficult to use services with peace of mind.

[0660] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0661] In this invention, the server includes means for storing personal information using local storage and restricting external transmission, means for clustering based on user attribute data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external computer, and means for aggregating the learning parameters and updating the final model. This makes it possible to construct a highly accurate AI model without leaking personal information to the outside, and to provide optimal services while protecting user privacy.

[0662] "Local storage" refers to data storage that is stored directly on the user's device and is a storage device for holding data that is not transferred to an external server.

[0663] "Personal Information" means identifiable information about a user, including name, address, telephone number, email address, purchase history, etc.

[0664] "Clustering" is a machine learning algorithm for analyzing a collection of data and grouping data with similar attributes.

[0665] "Federated learning" is a technology that performs partial learning on each terminal without collecting data from multiple distributed terminals, and then collects only the resulting learning parameters to update the overall model.

[0666] "Learning parameters" are internal numerical information that is adjusted by training a machine learning model, and refer to coefficients and biases that improve the model's predictive accuracy.

[0667] An "external computer" is a computing resource for a server or cloud service that is located in a location separate from the user's device or local storage.

[0668] "Data Rights Management" is a security measure that manages access rights and usage permissions for personal information, allowing users to control how their data is used.

[0669] "Management functions for providing optimal services while protecting privacy" refers to system functions that analyze collected data and provide customized services and recommendations tailored to each user while protecting the user's privacy.

[0670] This invention relates to a generative AI model system that uses local storage to store personal information and restricts its external transmission. Specifically, it performs clustering based on user attribute data, builds a federated learning model using data collected from multiple users, and transmits the learning parameters to an external computer. The invention also includes a means for managing data authority for personal information and a management function for providing optimal services while protecting user privacy.

[0671] First, the server uses local storage to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. When the server is initialized, it checks for the existence of a local storage file, and if the file does not exist, it creates a new one. This file stores each user's personal information and usage data. For example, a database file called "user_data.db" is created using SQLite, and a "users" table and a "user_logs" table are placed in it.

[0672] Next, the server performs clustering based on the user's attribute data. The attribute data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm, specifically K-means clustering, to classify users with similar characteristics into the same group. This makes it easier to analyze based on the attribute data and enables the provision of customized services.

[0673] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only the learning parameters are sent to the external computer. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leaks. Specifically, learning is carried out using machine learning frameworks such as Pytorch and TensorFlow.

[0674] As a specific example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and clusters users with similar purchasing patterns. It then inputs the purchasing data of all users into a federated learning algorithm to create an integrated model. This model can then make product recommendations optimized for each individual user.

[0675] From the perspective of privacy management, the server also has a means for data permission management. This is a function for strictly managing personal information to prevent it from leaking to the outside. Data permission management includes data encryption, access control, and obtaining permission to use. This allows users to safely manage how their data is used.

[0676] Furthermore, the server has a management function to provide optimal services while protecting the user's privacy. This management function allows the server to provide services tailored to the user's requests and needs. For example, it can provide personalized suggestions and advice based on the user's usage history and feedback.

[0677] An example prompt is:

[0678] "This system will recommend the best products based on your recent purchase history. Would you be interested in any of these products?"

[0679] As described above, the present invention provides a system that provides high-performance generative AI models while preventing the leakage of personal information. This allows companies to incorporate generative AI models into their business operations with peace of mind, and users can use them with peace of mind, knowing that their privacy is protected.

[0680] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0681] Step 1: Initialize and configure the local database

[0682] During system initialization, the server checks for the existence of a local database file. As input, it references the path to the database file on the file system. If the file does not exist, the server creates a new database file and initializes the appropriate tables. The database contains personal information for each user. As output, it creates an SQLite database called "user_data.db" and the "users" and "user_logs" tables within it.

[0683] Specifically, the server performs the following operations:

[0684] Checking for the existence of "user_data.db"

[0685] Creating a database if it does not exist

[0686] Initializing the "users" and "user_logs" tables

[0687] Step 2: Collect user data

[0688] The device collects data according to the user's operations. As input, the user's operation log and input data are acquired in real time. This data is stored in a local database. As output, the input data is added to the "user_logs" table.

[0689] Specifically, the device performs the following operations:

[0690] Monitoring user clicks and input information

[0691] Insert the obtained data into a local database

[0692] Step 3: Clustering feature data

[0693] The server periodically clusters user data. User attribute data obtained from a local database is used as input. The server uses this data to run a machine learning algorithm (e.g., K-means clustering) to group users. The output is the clustering results.

[0694] Specifically, the server performs the following operations:

[0695] Execute "SELECT FROM user_logs WHERE timestamp > {last_scan_timestamp}"

[0696] Extracting attribute data and calculating features

[0697] Clustering with machine learning algorithms

[0698] Step 4: Running federated learning

[0699] The server then starts federated learning based on the data collected from all users. The local data of each device is used as input. The device uses that data to train a model locally and sends the training parameters to the server. The server integrates the collected parameters to update the final AI model. The output is an updated AI model.

[0700] Specifically, the server and the device perform the following operations:

[0701] Learning task distribution from the server

[0702] On-device model training (e.g., Pytorch or TensorFlow)

[0703] Sending learning parameters and aggregating them on the server

[0704] Step 5: Optimized service delivery

[0705] Finally, the server uses the updated AI model to provide the optimal service to the user. The inputs are the updated AI model and the user's usage history and feedback. The server generates personalized recommendations and proposals based on this and sends them to the device. The output is optimized recommendation data.

[0706] Specifically, the server and the device perform the following operations:

[0707] Data analysis using updated AI models

[0708] Generating recommendations

[0709] Sending recommendation data to devices

[0710] Viewing recommendations on your device

[0711] (Application example 1)

[0712] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0713] Conventional electronic payment services often do not adequately protect privacy when transmitting users' purchase history and personal information to external parties, resulting in stress for users and the risk of information leaks. Furthermore, they lack functionality for providing personalized promotions and offers, leaving the user experience unsatisfactory. The purpose of this invention is to solve these problems and provide a safe and effective personalized service.

[0714] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0715] In this invention, the server includes a means for storing personal information using a local database and restricting transmission to the outside, a means for clustering based on user feature data, a means for building a federated learning model using data collected from multiple users, and a smartphone application including a management function for providing personalized promotions and offers, thereby enabling the provision of optimized promotions and offers while protecting user privacy.

[0716] A "local database" is a database that protects privacy by storing a user's personal information on the device and restricting its transmission to the outside.

[0717] "Personal information" refers to a user's unique identification information, purchasing history, behavioral patterns, etc.

[0718] "Feature data" is data that includes user behavior patterns and attribute information, and is used for clustering.

[0719] "Clustering" is the process of grouping users with similar characteristics based on their feature data.

[0720] A "federated learning model" is a machine learning model constructed using data collected from multiple users, in which only the learning parameters are sent to an external server.

[0721] "Learning parameters" are parameters sent to an external server to improve the accuracy of the model in federated learning.

[0722] The "external server" is a server that receives the learning parameters of the federated learning model and constructs an integrated model.

[0723] "Personalized promotions and offers" are promotions and offers that provide personalized suggestions and benefits to individual users based on their user characteristics data.

[0724] A "smartphone application" is an application that runs on a smartphone, handles users' personal information, and provides users with personalized services through clustering and federated learning.

[0725] "Data sovereignty management" refers to the means of managing users' personal information to prevent it from being leaked to the outside, and includes encryption and access control.

[0726] "Privacy protection" refers to measures to prevent users' personal information from being misused or leaked.

[0727] This invention specifically describes a system for protecting users' personal information and providing personalized promotions in electronic payment services. This system has the functions of storing personal information using a local database and restricting external transmission, clustering users based on feature data, and building a federated learning model.

[0728] First, the device (e.g., a smartphone) stores the user's purchasing history and behavioral patterns in a local database. This database is built using SQLite, and the user's personal information is securely encrypted and stored using the Fernet encryption method.

[0729] The device then performs clustering based on the user feature data, using the KMeans algorithm to group similar users based on their behavioral patterns and attribute information.

[0730] In the federated learning step, each user's data is trained locally and the training parameters are sent to an external server. TensorFlow Federated (TFF) is used to build the federated learning model. This ensures privacy by keeping individual user data local and only sending the training parameters to the external server.

[0731] Furthermore, the terminal will utilize the constructed model to provide users with personalized promotions and offers, which will suggest optimal offers and deals for them, improving their user experience.

[0732] For example, when a user records their past purchase history using a smartphone application, that data is encrypted and stored in a local database. Using a model built through federated learning, the user is then presented with "recommended coupons." These coupons are personalized based on the user's purchase history and behavioral patterns.

[0733] Example prompt sentence:

[0734] I want to create a feature that uses a user's past purchase history data to display coupon recommendations while maintaining privacy. How can I use federated learning and clustering to provide personalized promotions?

[0735] The system is realized using a smartphone, a local database (SQLite), an encryption library (Fernet), a clustering algorithm (KMeans), and a federated learning framework (TensorFlow Federated).

[0736] As described above, this invention is a system that provides safe and effective personal information protection and personalized services through a smartphone application. This system allows users to use electronic payment services with peace of mind, and allows companies to provide highly reliable services.

[0737] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0738] Step 1:

[0739] Initializing the local database

[0740] The device initializes a local database to store the user's personal information. Specifically, it uses SQLite and creates a new database file if one does not exist. It checks the state of the local file system as input and generates a database file to store the user information as output.

[0741] Step 2:

[0742] Data encryption and storage

[0743] The device encrypts the user's purchase history and behavioral patterns and stores them in a local database. Specifically, it uses the Fernet encryption library to encrypt user data. It receives unencrypted purchase history and behavioral patterns as input and stores the encrypted data in the local database as output.

[0744] Step 3:

[0745] Feature data collection and clustering

[0746] The device collects user feature data and performs clustering using the KMeans algorithm. Collected user behavior patterns and attribute information are used as input, and users classified into clusters are output.

[0747] Step 4:

[0748] Implementing federated learning

[0749] The device learns each user's data locally and sends the learning parameters to an external server. Specifically, federated learning is performed using TensorFlow Federated (TFF). Each user's purchase history data is provided as input, and the learning parameters of the federated learning model are generated as output and sent to the external server.

[0750] Step 5:

[0751] Model integration with external servers

[0752] The server receives the learning parameters sent from the devices and constructs an integrated federated learning model. Learning parameters from multiple devices are received as input, and an integrated, high-precision federated learning model is generated as output.

[0753] Step 6:

[0754] Providing personalized service

[0755] The device uses the constructed federated learning model to provide personalized promotions and offers to the user. Using the federated model and user feature data as input, the device displays promotions and offers optimized for the user as output.

[0756] In this way, the system protects users' personal information while providing personalized promotions and offers that are most relevant to them.

[0757] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0758] ---

[0759] This invention combines a generative AI model system that uses a local database to store personal information and restricts external transmission with an emotion engine that recognizes user emotions. Specifically, it performs clustering based on user feature data and builds a federated learning model using data collected from multiple users. It also has a management function that transmits the learning parameters of the federated learning model to an external server, performs data sovereignty management of personal information, and provides optimal services while protecting user privacy. The system also has the function of recognizing the user's emotional state using the emotion engine and adjusting services.

[0760] First, the server uses a local database to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. The server checks the existence of a local database file during initialization, and creates a new one if it does not exist. This file stores each user's personal information and usage data.

[0761] The server then performs clustering based on the user feature data. The feature data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm to classify users with similar features into the same group. This makes it easier to analyze the feature data and provide customized services.

[0762] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only learning parameters are sent to the external server. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leakage.

[0763] Furthermore, by combining it with an emotion engine, it becomes possible to recognize the user's emotional state. The server utilizes the emotion engine to identify emotions from user input data, voice data, and image data. For example, when a user types a text message, the server reads the user's emotions from the context and expression and stores them as emotion data. This emotion data is added to the user's feature data to improve the accuracy of clustering and federated learning models.

[0764] As a concrete example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and emotional state and clusters users with similar patterns. It then inputs all users' purchasing data and emotional data into a federated learning algorithm to create an integrated model. This model can then provide optimized product recommendations for each individual user.

[0765] The emotion data recognized by the emotion engine complements the user's feature data and improves the accuracy of the clustering and federated learning models. The server utilizes this to provide detailed services tailored to the user's needs and status.

[0766] Finally, the server has the ability to adjust service content in real time. Based on the user's emotional state as recognized by the emotion engine, the server can change the content and delivery method of the service accordingly. For example, if the user is feeling stressed, the server can respond in line with the user's emotions by recommending content that has a relaxing effect.

[0767] The above is a specific implementation of the "MyData-GPT" system, which combines an emotion engine. This system makes it possible to provide highly accurate and optimized services to users while protecting personal information.

[0768] The processing flow will be explained below.

[0769] ---

[0770] Step 1:

[0771] When the server initializes the program, it checks for the existence of a local database file. If the file exists, the server reads it and loads the user data into local memory. If it does not exist, the server creates an empty database and creates a new file.

[0772] Step 2:

[0773] When the user provides new data, the server updates the local database by calling the update_local_db method. The data received from the user is merged with the existing data and updated to the latest state.

[0774] Step 3:

[0775] After updating the local database, the server calls the save_data method to save the database to a file. This process makes the user data persistent.

[0776] Step 4:

[0777] The emotion engine processes user input data, voice data, and image data to identify the user's emotional state. For example, it analyzes emotions from text input and generates emotion data.

[0778] Step 5:

[0779] The server performs clustering based on the user's feature data and emotion data. The server extracts the feature data and emotion data of all users and performs clustering using the KMeans algorithm. Each user is assigned to an appropriate cluster based on their feature data.

[0780] Step 6:

[0781] After clustering is complete, the server assigns each user a cluster ID, which allows users with similar characteristics to be grouped together for efficient data analysis and processing.

[0782] Step 7:

[0783] The server performs federated learning using all user data and emotional data. The server learns each user's data locally and aggregates the results to build an overall AI model. Individual user data is not transmitted externally, protecting privacy.

[0784] Step 8:

[0785] The server sends the learning parameters obtained as a result of federated learning to an external server, making it possible to use highly accurate models remotely while protecting personal information.

[0786] Step 9:

[0787] The server will manage data sovereignty and protect users' personal information from leaks to the outside world, by encrypting data, controlling access, and obtaining permission to use the data.

[0788] Step 10:

[0789] The server has management functions to provide optimized services while protecting the user's privacy, allowing the server to provide customized services according to the user's requests and needs.

[0790] Step 11:

[0791] The server adjusts the service content in real time based on the user's emotional state recognized by the emotion engine. For example, if the user is feeling stressed, it can respond in line with the user's emotions by recommending content that has a relaxing effect.

[0792] ---

[0793] The above is the specific processing flow of the "MyData-GPT" system, which combines an emotion engine. The roles of the server, device, and user are clarified, and the specific operations performed at each step are explained.

[0794] Example 2

[0795] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0796] In modern society, protecting users' personal information and providing an optimized user experience are urgent issues. However, existing systems face privacy risks due to the external transmission of personal information, and it is difficult to adjust services in real time based on the user's emotional state. A system that solves these issues and provides optimal services using emotion recognition while protecting user privacy is needed.

[0797] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0798] In this invention, the server includes means for storing personal information using a local database and restricting external transmission, means for clustering based on user feature data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external server, means for utilizing an emotion engine that recognizes the user's emotional state, and means for adjusting service content in real time based on the emotional state. This makes it possible to provide optimal services based on the user's emotional state while protecting the user's personal information.

[0799] A "local database" is a database used to store a user's personal information on the device and restrict its transmission to external devices.

[0800] "Clustering" is a method of grouping users with similar characteristics based on user characteristic data.

[0801] A "federated learning model" is a machine learning model that is constructed by learning data collected from multiple users locally on each device and aggregating only the learning results (parameters).

[0802] "Learning parameters" are data that indicate the results of local learning by each device in a federated learning model, and by aggregating these data, the accuracy of the model is improved.

[0803] An "emotion engine" is a technology that analyzes user input data (text, voice, images, etc.) and recognizes their emotions.

[0804] "Emotional State" means the current mental and emotional state identified from data entered by the User.

[0805] "Real-time adjustment of service content" is a function that instantly changes the content and method of the service provided based on the user's emotional state recognized by the emotion engine.

[0806] "Data sovereignty management" is a management method that ensures that users' personal information is used under their own control and prevents it from leaking to the outside.

[0807] "Privacy protection" refers to measures to prevent users' personal information and emotional data from being misused by third parties.

[0808] MODE FOR CARRYING OUT THE INVENTION

[0809] This invention is a system that provides advanced services while protecting privacy by storing users' personal information in a local database and restricting its transmission to external parties. This system combines clustering based on user feature data, the construction of a federated learning model, and an emotion recognition engine to adjust service content in real time.

[0810] Data storage method

[0811] First, when the system starts up, the server checks for the existence of a local database, and if the file does not exist, it creates a new one. This database stores each user's personal information and usage data. Specifically, it uses a database management system such as SQLite. For example, user information is stored in a file named "users.db."

[0812] Clustering Method

[0813] Next, the server collects feature data for clustering based on user behavior patterns and attribute information. This process uses machine learning libraries such as scikit-learn. Specifically, it uses the K-means algorithm and classifies users into five groups by setting "n_clusters=5," for example.

[0814] Federated Learning Methods

[0815] In the federated learning step, the server collects data from multiple devices and performs local learning on each device. Each device performs learning using, for example, TensorFlow Federated, and sends the resulting learning parameters to the server. The server aggregates these parameters and builds an overall AI model.

[0816] Emotion recognition method

[0817] When a user inputs text, voice, and image data, the server uses an emotion engine to recognize the user's emotional state from these input data. Specifically, the emotion recognition engine uses a general emotion recognition API. For example, if a user inputs a message such as "I had so much fun today!", the server identifies the emotion "joy" from the context and stores it in a database.

[0818] Real-time adjustment method

[0819] The server adjusts the services it provides in real time based on the user's emotional state as recognized by the emotion engine. For example, if a user is recognized as feeling "stressed," it will recommend music and videos with a relaxing effect. As a specific example, a general music streaming API can be used for music recommendations. The server reflects these recommendations on the user's device in real time.

[0820] Examples and prompts

[0821] As a specific example, the server performs federated learning using purchasing data and emotional data collected from each user. For example, it analyzes each user's purchasing patterns and emotional state and clusters users with similar patterns. Then, it inputs all users' purchasing data and emotional data into a federated learning algorithm to create an integrated model. This model then makes product recommendations optimized for each individual user.

[0822] Example prompt sentence:

[0823] "Use federated learning to build the optimal product recommendation model using user purchasing data and sentiment data."

[0824] "Recognize emotions from user text input and tailor your service based on those emotions."

[0825] As described above, the system according to the present invention provides optimal services based on the user's emotions while protecting personal information.

[0826] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0827] How the system handles programs

[0828] Step 1:

[0829] The server initializes the local database.

[0830] Input: System startup state

[0831] BEHAVIOR: The server checks for the existence of an SQLite database file called "users.db" and creates it if it doesn't exist. It then connects to the database using the sqlite3 library and sets up the users table.

[0832] Output: Initialized local database

[0833] Step 2:

[0834] The server collects user feature data and performs clustering.

[0835] Input: User behavior patterns and attribute information (e.g., age, gender, purchase history)

[0836] How it works: The server uses the scikit-learn library to perform K-means clustering on the feature data. For example, run kmeans = KMeans(n_clusters=5) to classify users into 5 groups. After clustering, it saves a list of users classified into groups.

[0837] Output: Clustered user data

[0838] Step 3:

[0839] The server collects data for implementing federated learning and performs the learning.

[0840] Input: User data collected from each device

[0841] How it works: The server prepares the data to be handed over to each device, and uses the TensorFlow Federated library to perform local training. The trained parameters on each device are sent to the server, which aggregates these parameters and builds the overall AI model.

[0842] Output: AI model with aggregated learning parameters

[0843] Step 4:

[0844] The server uses an emotion engine to recognize the user's emotional state.

[0845] Input: User input data (text, audio, images, etc.)

[0846] How it works: The server uses an emotion recognition engine to identify the user's emotions. For example, using an emotion recognition API, it analyzes the text data "I had so much fun today!", identifies the emotional state "joy," and stores it in the database.

[0847] Output: Recognized emotion data

[0848] Step 5:

[0849] The server adjusts the service content in real time based on the user's emotional state.

[0850] Input: Recognized emotion data

[0851] How it works: The server adjusts the content of the service provided to the user based on the emotion data. For example, if the server detects that the user is feeling stressed, it will use a music streaming API to retrieve an appropriate relaxation playlist and recommend it to the user's device to provide relaxing music.

[0852] Output: Tailored service delivery

[0853] Step 6:

[0854] The server monitors the overall system function, gathering feedback and making improvements as needed.

[0855] Input: Usage of the provided service and user feedback

[0856] How it works: The server monitors user feedback and usage data to analyze the system's performance and accuracy, which in turn allows it to make improvements to improve the accuracy of the federated learning model and emotion recognition engine.

[0857] Output: New version of improved system functionality

[0858] Through this series of steps, the system can provide advanced and optimized services based on user sentiment while protecting user privacy.

[0859] (Application example 2)

[0860] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0861] Conventional service provision systems not only pose privacy risks by transmitting users' personal information to external parties, but also lack the ability to adapt to users' emotional states. This makes it difficult to provide appropriate services that reflect users' real-time emotions. Furthermore, in today's world where both personal information protection and service quality are required, providing optimal services while ensuring user privacy is a challenge.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0863] In this invention, the server includes means for storing personal information using a local database and restricting external transmission, means for clustering based on user feature data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external server, means for analyzing the emotional state of the user in real time using an emotion recognition engine, and means for dynamically adjusting the content and method of service provision based on the analysis results, thereby enabling the provision of real-time services according to emotions while protecting user privacy.

[0864] A "local database" is a database that stores users' personal information and restricts its transmission to external parties.

[0865] "Clustering" is a technique for grouping multiple users based on their characteristic data.

[0866] A "federated learning model" is a model that is constructed using data collected from multiple users and transmits only the learning parameters to an external server.

[0867] A "learning parameter" is a value that indicates the characteristics of the data used when constructing an associative learning model.

[0868] The "external server" is a server that receives the learning parameters and performs analysis and processing.

[0869] An "emotion recognition engine" is a system for analyzing a user's emotional state in real time.

[0870] "Dynamic adjustment" is the process of changing service content and delivery methods in real time based on analysis results.

[0871] "Protecting privacy" means managing users' personal information to prevent it from being leaked to the outside without permission.

[0872] "User characteristic data" refers to data such as user behavior patterns and attribute information.

[0873] "Real-time analysis" means instantly determining the user's emotional state on the spot.

[0874] This invention is a system that analyzes user emotions in real time in brick-and-mortar stores and dynamically adjusts service content based on that data. It uses a server and smart glasses in combination to provide optimal services while protecting user privacy.

[0875] The server uses a local database to store personal information and restricts external transmission. This local database stores each user's personal information and usage data. Clustering methods are used to classify users based on their feature data, and a federated learning model is constructed using data collected from multiple users. Learning parameters are sent to an external server, but the personal information itself is not sent externally, protecting privacy.

[0876] The smart glasses have a built-in camera that acts as an input device for analyzing the user's facial expressions and voice. The emotion recognition engine uses this camera to analyze the user's emotions in real time. The analyzed emotion data is sent to the server, which dynamically adjusts the service based on the user's interests and state.

[0877] Specifically, it works as follows:

[0878] 1. The server analyzes the user's emotional state using image data acquired from the smart glasses.

[0879] 2. The analyzed emotion data is stored in a local database and classified into groups of users with similar emotion patterns using clustering methods.

[0880] 3. The data analyzed by the emotion recognition engine is reflected in the overall AI model using a federated learning model.

[0881] 4. Based on this model, the server recommends the most suitable products and services for each user and adjusts the delivery method in real time.

[0882] The main hardware used is smart glasses and a server. The software uses OpenCV and Keras (with TensorFlow backend). OpenCV functions as a face detector, and Keras is used for the sentiment analysis model. Data is processed in real time, and appropriate services are automatically provided based on the analysis results.

[0883] For example, if a store staff member is wearing smart glasses and a camera captures a customer's facial expression, the image will be used by an emotion recognition engine to analyze the customer's emotional state. If the customer is feeling stressed, the system will recommend products that have a relaxing effect. This will improve the customer experience and enhance the quality of service at the store.

[0884] Examples of prompts include:

[0885] "What emotional state is this customer in right now?"

[0886] "Provide product recommendations that match customer emotions."

[0887] This will realize a system that can provide optimal services based on emotions in real time while protecting user privacy.

[0888] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0889] Step 1:

[0890] The smart glasses capture the user's facial image using the built-in camera. The image data obtained from the camera is the input. This image data is passed to the next processing step.

[0891] Step 2:

[0892] The server receives the image data and performs face detection using OpenCV. Specifically, it performs grayscale conversion and histogram equalization on the image data, and inputs it into a face detector. The face detector identifies the face in the image and outputs its coordinate information and the face image.

[0893] Step 3:

[0894] The server inputs the facial image obtained from the face detector into the emotion recognition engine. This facial image is passed to a pre-trained Keras emotion recognition model for emotion analysis. The emotion recognition model classifies the emotional state (e.g., anger, happiness, surprise, etc.) from the input facial image and outputs the result as emotion data.

[0895] Step 4:

[0896] The server receives emotion data from the emotion recognition engine and stores it in a local database. The local database classifies this emotion data using a clustering method and reconstructs it as user feature data. This allows emotion patterns for each individual user to be stored in the database.

[0897] Step 5:

[0898] The server updates the federated learning model using feature data collected from multiple users based on the clustered emotion data. This federated learning model trains locally using each user's data and sends the model parameters obtained as a result of the training to an external server. The external server receives this and updates the overall AI model.

[0899] Step 6:

[0900] The server receives the updated AI model returned from the external server and combines it with the emotional data to recommend the most suitable products and services for the user. This recommendation information is notified to the staff in real time via the smart glasses. For example, a message such as "Please recommend a relaxing aroma" may appear on the display of the staff member's smart glasses.

[0901] Step 7:

[0902] Store staff can provide dynamically tailored services to users based on the recommendations displayed on the smart glasses. This process enables real-time emotion recognition and service adjustment, improving user satisfaction.

[0903] Through these steps, a system will be realized in which the server, smart glasses, emotion recognition engine, federated learning model, etc. work together to provide optimal services to users.

[0904] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0905] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0906] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0907] [Fourth embodiment]

[0908] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0909] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0910] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0911] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0912] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0913] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0914] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0915] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0916] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0917] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0918] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0919] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0920] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0921] ---

[0922] This invention relates to a generative AI model system that uses a local database to store personal information and restricts its transmission to external servers. Specifically, it performs clustering based on user feature data, builds a federated learning model using data collected from multiple users, and transmits the learning parameters to an external server. The invention also provides a means for data sovereignty management of personal information and a management function for providing optimal services while protecting user privacy.

[0923] First, the server uses a local database to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. The server checks the existence of a local database file during initialization, and creates a new one if it does not exist. This file stores each user's personal information and usage data.

[0924] The server then performs clustering based on the user feature data. The feature data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm to classify users with similar features into the same group. This makes it easier to analyze the feature data and provide customized services.

[0925] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only learning parameters are sent to the external server. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leakage.

[0926] As a specific example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and clusters users with similar purchasing patterns. It then inputs all users' purchasing data into a federated learning algorithm to create an integrated model. This model can then provide optimized product recommendations for each individual user.

[0927] From the perspective of privacy management, the server also has a means of data sovereignty management. This is a function to strictly manage personal information so that it is not leaked to the outside. Data sovereignty management includes data encryption, access control, and obtaining usage permission. This allows users to safely manage how their data is used.

[0928] Furthermore, the server has management functions to provide optimal services while protecting users' privacy. These management functions allow the server to provide services tailored to users' requests and needs. For example, the server can provide personalized suggestions and advice by taking into account users' usage history and feedback.

[0929] As described above, the present invention provides a system that provides high-performance generative AI models while preventing the leakage of personal information. This allows companies to incorporate generative AI models into their business with peace of mind, and users can use them with peace of mind, knowing that their privacy is protected.

[0930] The processing flow will be explained below.

[0931] ---

[0932] Step 1:

[0933] When the server initializes the program, it checks for the existence of a local database file. If the file exists, the server reads it and loads the user data into local memory. If it does not exist, the server creates an empty database and creates a new file.

[0934] Step 2:

[0935] If the user provides new data, the server updates the local database by calling the update_local_db method. During this time, the data received from the user is merged with the existing data and updated to the latest state.

[0936] Step 3:

[0937] After the server updates the local database, it calls the save_data method to save the database to a file, which makes the user data persistent.

[0938] Step 4:

[0939] The server performs clustering based on the user feature data. To do this, the server takes all users' feature data and performs clustering using the KMeans algorithm. Each user is assigned to an appropriate cluster based on their feature data.

[0940] Step 5:

[0941] After clustering is complete, the server assigns each user a cluster ID, which allows users with similar characteristics to be grouped together for efficient data analysis and processing.

[0942] Step 6:

[0943] The server uses all user data to perform federated learning. Using data collected from each user, it locally trains a model and aggregates the results to build an overall AI model. Individual user data is not transmitted externally, protecting privacy.

[0944] Step 7:

[0945] The server sends the learning parameters obtained as a result of federated learning to an external server, which makes it possible to remotely utilize more accurate models while protecting personal information.

[0946] Step 8:

[0947] The server will manage data sovereignty and protect users' personal information from leaks to the outside world, by encrypting data, controlling access, and obtaining permission to use the data.

[0948] Step 9:

[0949] The server has management functions to provide optimized services while protecting the user's privacy, allowing the server to provide customized services according to the user's requests and needs.

[0950] ---

[0951] The above is the specific processing flow of the "MyData-GPT" system. The roles of the server, terminal, and user are clearly distinguished, and the specific operations performed at each step are explained.

[0952] Example 1

[0953] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0954] In modern information systems, users' personal information is collected by many service providers, increasing the possibility of it being leaked to external parties. This increases the risk of user privacy being violated. It is also difficult to effectively utilize individual data to create advanced AI models. Furthermore, there is insufficient data authority management for personal information, making it difficult to use services with peace of mind.

[0955] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0956] In this invention, the server includes means for storing personal information using local storage and restricting external transmission, means for clustering based on user attribute data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external computer, and means for aggregating the learning parameters and updating the final model. This makes it possible to construct a highly accurate AI model without leaking personal information to the outside, and to provide optimal services while protecting user privacy.

[0957] "Local storage" refers to data storage that is stored directly on the user's device and is a storage device for holding data that is not transferred to an external server.

[0958] "Personal Information" means identifiable information about a user, including name, address, telephone number, email address, purchase history, etc.

[0959] "Clustering" is a machine learning algorithm for analyzing a collection of data and grouping data with similar attributes.

[0960] "Federated learning" is a technology that performs partial learning on each terminal without collecting data from multiple distributed terminals, and then collects only the resulting learning parameters to update the overall model.

[0961] "Learning parameters" are internal numerical information that is adjusted by training a machine learning model, and refer to coefficients and biases that improve the model's predictive accuracy.

[0962] An "external computer" is a computing resource for a server or cloud service that is located in a location separate from the user's device or local storage.

[0963] "Data Rights Management" is a security measure that manages access rights and usage permissions for personal information, allowing users to control how their data is used.

[0964] "Management functions for providing optimal services while protecting privacy" refers to system functions that analyze collected data and provide customized services and recommendations tailored to each user while protecting the user's privacy.

[0965] This invention relates to a generative AI model system that uses local storage to store personal information and restricts its external transmission. Specifically, it performs clustering based on user attribute data, builds a federated learning model using data collected from multiple users, and transmits the learning parameters to an external computer. The invention also includes a means for managing data authority for personal information and a management function for providing optimal services while protecting user privacy.

[0966] First, the server uses local storage to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. When the server is initialized, it checks for the existence of a local storage file, and if the file does not exist, it creates a new one. This file stores each user's personal information and usage data. For example, a database file called "user_data.db" is created using SQLite, and a "users" table and a "user_logs" table are placed in it.

[0967] Next, the server performs clustering based on the user's attribute data. The attribute data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm, specifically K-means clustering, to classify users with similar characteristics into the same group. This makes it easier to analyze based on the attribute data and enables the provision of customized services.

[0968] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only the learning parameters are sent to the external computer. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leaks. Specifically, learning is carried out using machine learning frameworks such as Pytorch and TensorFlow.

[0969] As a specific example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and clusters users with similar purchasing patterns. It then inputs the purchasing data of all users into a federated learning algorithm to create an integrated model. This model can then make product recommendations optimized for each individual user.

[0970] From the perspective of privacy management, the server also has a means for data permission management. This is a function for strictly managing personal information to prevent it from leaking to the outside. Data permission management includes data encryption, access control, and obtaining permission to use. This allows users to safely manage how their data is used.

[0971] Furthermore, the server has a management function to provide optimal services while protecting the user's privacy. This management function allows the server to provide services tailored to the user's requests and needs. For example, it can provide personalized suggestions and advice based on the user's usage history and feedback.

[0972] An example prompt is:

[0973] "This system will recommend the best products based on your recent purchase history. Would you be interested in any of these products?"

[0974] As described above, the present invention provides a system that provides high-performance generative AI models while preventing the leakage of personal information. This allows companies to incorporate generative AI models into their business operations with peace of mind, and users can use them with peace of mind, knowing that their privacy is protected.

[0975] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0976] Step 1: Initialize and configure the local database

[0977] During system initialization, the server checks for the existence of a local database file. As input, it references the path to the database file on the file system. If the file does not exist, the server creates a new database file and initializes the appropriate tables. The database contains personal information for each user. As output, it creates an SQLite database called "user_data.db" and the "users" and "user_logs" tables within it.

[0978] Specifically, the server performs the following operations:

[0979] Checking for the existence of "user_data.db"

[0980] Creating a database if it does not exist

[0981] Initializing the "users" and "user_logs" tables

[0982] Step 2: Collect user data

[0983] The device collects data according to the user's operations. As input, the user's operation log and input data are acquired in real time. This data is stored in a local database. As output, the input data is added to the "user_logs" table.

[0984] Specifically, the device performs the following operations:

[0985] Monitoring user clicks and input information

[0986] Insert the obtained data into a local database

[0987] Step 3: Clustering feature data

[0988] The server periodically clusters user data. User attribute data obtained from a local database is used as input. The server uses this data to run a machine learning algorithm (e.g., K-means clustering) to group users. The output is the clustering results.

[0989] Specifically, the server performs the following operations:

[0990] Execute "SELECT FROM user_logs WHERE timestamp > {last_scan_timestamp}"

[0991] Extracting attribute data and calculating features

[0992] Clustering with machine learning algorithms

[0993] Step 4: Running federated learning

[0994] The server then starts federated learning based on the data collected from all users. The local data of each device is used as input. The device uses that data to train a model locally and sends the training parameters to the server. The server integrates the collected parameters to update the final AI model. The output is an updated AI model.

[0995] Specifically, the server and the device perform the following operations:

[0996] Learning task distribution from the server

[0997] On-device model training (e.g., Pytorch or TensorFlow)

[0998] Sending learning parameters and aggregating them on the server

[0999] Step 5: Optimized service delivery

[1000] Finally, the server uses the updated AI model to provide the optimal service to the user. The inputs are the updated AI model and the user's usage history and feedback. The server generates personalized recommendations and proposals based on this and sends them to the device. The output is optimized recommendation data.

[1001] Specifically, the server and the device perform the following operations:

[1002] Data analysis using updated AI models

[1003] Generating recommendations

[1004] Sending recommendation data to devices

[1005] Viewing recommendations on your device

[1006] (Application example 1)

[1007] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1008] Conventional electronic payment services often do not adequately protect privacy when transmitting users' purchase history and personal information to external parties, resulting in stress for users and the risk of information leaks. Furthermore, they lack functionality for providing personalized promotions and offers, leaving the user experience unsatisfactory. The purpose of this invention is to solve these problems and provide a safe and effective personalized service.

[1009] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1010] In this invention, the server includes a means for storing personal information using a local database and restricting transmission to the outside, a means for clustering based on user feature data, a means for building a federated learning model using data collected from multiple users, and a smartphone application including a management function for providing personalized promotions and offers, thereby enabling the provision of optimized promotions and offers while protecting user privacy.

[1011] A "local database" is a database that protects privacy by storing a user's personal information on the device and restricting its transmission to the outside.

[1012] "Personal information" refers to a user's unique identification information, purchasing history, behavioral patterns, etc.

[1013] "Feature data" is data that includes user behavior patterns and attribute information, and is used for clustering.

[1014] "Clustering" is the process of grouping users with similar characteristics based on their feature data.

[1015] A "federated learning model" is a machine learning model constructed using data collected from multiple users, in which only the learning parameters are sent to an external server.

[1016] "Learning parameters" are parameters sent to an external server to improve the accuracy of the model in federated learning.

[1017] The "external server" is a server that receives the learning parameters of the federated learning model and constructs an integrated model.

[1018] "Personalized promotions and offers" are promotions and offers that provide personalized suggestions and benefits to individual users based on their user characteristics data.

[1019] A "smartphone application" is an application that runs on a smartphone, handles users' personal information, and provides users with personalized services through clustering and federated learning.

[1020] "Data sovereignty management" refers to the means of managing users' personal information to prevent it from being leaked to the outside, and includes encryption and access control.

[1021] "Privacy protection" refers to measures to prevent users' personal information from being misused or leaked.

[1022] This invention specifically describes a system for protecting users' personal information and providing personalized promotions in electronic payment services. This system has the functions of storing personal information using a local database and restricting external transmission, clustering users based on feature data, and building a federated learning model.

[1023] First, the device (e.g., a smartphone) stores the user's purchasing history and behavioral patterns in a local database. This database is built using SQLite, and the user's personal information is securely encrypted and stored using the Fernet encryption method.

[1024] The device then performs clustering based on the user feature data, using the KMeans algorithm to group similar users based on their behavioral patterns and attribute information.

[1025] In the federated learning step, each user's data is trained locally and the training parameters are sent to an external server. TensorFlow Federated (TFF) is used to build the federated learning model. This ensures privacy by keeping individual user data local and only sending the training parameters to the external server.

[1026] Furthermore, the terminal will utilize the constructed model to provide users with personalized promotions and offers, which will suggest optimal offers and deals for them, improving their user experience.

[1027] For example, when a user records their past purchase history using a smartphone application, that data is encrypted and stored in a local database. Using a model built through federated learning, the user is then presented with "recommended coupons." These coupons are personalized based on the user's purchase history and behavioral patterns.

[1028] Example prompt sentence:

[1029] I want to create a feature that uses a user's past purchase history data to display coupon recommendations while maintaining privacy. How can I use federated learning and clustering to provide personalized promotions?

[1030] The system is realized using a smartphone, a local database (SQLite), an encryption library (Fernet), a clustering algorithm (KMeans), and a federated learning framework (TensorFlow Federated).

[1031] As described above, this invention is a system that provides safe and effective personal information protection and personalized services through a smartphone application. This system allows users to use electronic payment services with peace of mind, and allows companies to provide highly reliable services.

[1032] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1033] Step 1:

[1034] Initializing the local database

[1035] The device initializes a local database to store the user's personal information. Specifically, it uses SQLite and creates a new database file if one does not exist. It checks the state of the local file system as input and generates a database file to store the user information as output.

[1036] Step 2:

[1037] Data encryption and storage

[1038] The device encrypts the user's purchase history and behavioral patterns and stores them in a local database. Specifically, it uses the Fernet encryption library to encrypt user data. It receives unencrypted purchase history and behavioral patterns as input and stores the encrypted data in the local database as output.

[1039] Step 3:

[1040] Feature data collection and clustering

[1041] The device collects user feature data and performs clustering using the KMeans algorithm. Collected user behavior patterns and attribute information are used as input, and users classified into clusters are output.

[1042] Step 4:

[1043] Implementing federated learning

[1044] The device learns each user's data locally and sends the learning parameters to an external server. Specifically, federated learning is performed using TensorFlow Federated (TFF). Each user's purchase history data is provided as input, and the learning parameters of the federated learning model are generated as output and sent to the external server.

[1045] Step 5:

[1046] Model integration with external servers

[1047] The server receives the learning parameters sent from the devices and constructs an integrated federated learning model. Learning parameters from multiple devices are received as input, and an integrated, high-precision federated learning model is generated as output.

[1048] Step 6:

[1049] Providing personalized service

[1050] The device uses the constructed federated learning model to provide personalized promotions and offers to the user. Using the federated model and user feature data as input, the device displays promotions and offers optimized for the user as output.

[1051] In this way, the system protects users' personal information while providing personalized promotions and offers that are most relevant to them.

[1052] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1053] ---

[1054] This invention combines a generative AI model system that uses a local database to store personal information and restricts external transmission with an emotion engine that recognizes user emotions. Specifically, it performs clustering based on user feature data and builds a federated learning model using data collected from multiple users. It also has a management function that transmits the learning parameters of the federated learning model to an external server, performs data sovereignty management of personal information, and provides optimal services while protecting user privacy. The system also has the function of recognizing the user's emotional state using the emotion engine and adjusting services.

[1055] First, the server uses a local database to store personal information. This is to ensure privacy by keeping user data within the device and restricting external transmission. The server checks the existence of a local database file during initialization, and creates a new one if it does not exist. This file stores each user's personal information and usage data.

[1056] The server then performs clustering based on the user feature data. The feature data includes user behavior patterns and attribute information, and is used to group multiple users. For example, the server uses a machine learning algorithm to classify users with similar features into the same group. This makes it easier to analyze the feature data and provide customized services.

[1057] In the federated learning step, the server utilizes data collected from all users. The server learns each user's data locally and aggregates the results to build an overall AI model. During this process, no personal information is sent externally; only learning parameters are sent to the external server. This method makes it possible to create a highly accurate AI model while significantly reducing the risk of information leakage.

[1058] Furthermore, by combining it with an emotion engine, it becomes possible to recognize the user's emotional state. The server utilizes the emotion engine to identify emotions from user input data, voice data, and image data. For example, when a user types a text message, the server reads the user's emotions from the context and expression and stores them as emotion data. This emotion data is added to the user's feature data to improve the accuracy of clustering and federated learning models.

[1059] As a concrete example, the server performs federated learning using purchasing data collected from each user. It analyzes each user's purchasing patterns and emotional state and clusters users with similar patterns. It then inputs all users' purchasing data and emotional data into a federated learning algorithm to create an integrated model. This model can then provide optimized product recommendations for each individual user.

[1060] The emotion data recognized by the emotion engine complements the user's feature data and improves the accuracy of the clustering and federated learning models. The server utilizes this to provide detailed services tailored to the user's needs and status.

[1061] Finally, the server has the ability to adjust service content in real time. Based on the user's emotional state as recognized by the emotion engine, the server can change the content and delivery method of the service accordingly. For example, if the user is feeling stressed, the server can respond in line with the user's emotions by recommending content that has a relaxing effect.

[1062] The above is a specific implementation of the "MyData-GPT" system, which combines an emotion engine. This system makes it possible to provide highly accurate and optimized services to users while protecting personal information.

[1063] The processing flow will be explained below.

[1064] ---

[1065] Step 1:

[1066] When the server initializes the program, it checks for the existence of a local database file. If the file exists, the server reads it and loads the user data into local memory. If it does not exist, the server creates an empty database and creates a new file.

[1067] Step 2:

[1068] When the user provides new data, the server updates the local database by calling the update_local_db method. The data received from the user is merged with the existing data and updated to the latest state.

[1069] Step 3:

[1070] After updating the local database, the server calls the save_data method to save the database to a file. This process makes the user data persistent.

[1071] Step 4:

[1072] The emotion engine processes user input data, voice data, and image data to identify the user's emotional state. For example, it analyzes emotions from text input and generates emotion data.

[1073] Step 5:

[1074] The server performs clustering based on the user's feature data and emotion data. The server extracts the feature data and emotion data of all users and performs clustering using the KMeans algorithm. Each user is assigned to an appropriate cluster based on their feature data.

[1075] Step 6:

[1076] After clustering is complete, the server assigns each user a cluster ID, which allows users with similar characteristics to be grouped together for efficient data analysis and processing.

[1077] Step 7:

[1078] The server performs federated learning using all user data and emotional data. The server learns each user's data locally and aggregates the results to build an overall AI model. Individual user data is not transmitted externally, protecting privacy.

[1079] Step 8:

[1080] The server sends the learning parameters obtained as a result of federated learning to an external server, making it possible to use highly accurate models remotely while protecting personal information.

[1081] Step 9:

[1082] The server will manage data sovereignty and protect users' personal information from leaks to the outside world, by encrypting data, controlling access, and obtaining permission to use the data.

[1083] Step 10:

[1084] The server has management functions to provide optimized services while protecting the user's privacy, allowing the server to provide customized services according to the user's requests and needs.

[1085] Step 11:

[1086] The server adjusts the service content in real time based on the user's emotional state recognized by the emotion engine. For example, if the user is feeling stressed, it can respond in line with the user's emotions by recommending content that has a relaxing effect.

[1087] ---

[1088] The above is the specific processing flow of the "MyData-GPT" system, which combines an emotion engine. The roles of the server, device, and user are clarified, and the specific operations performed at each step are explained.

[1089] Example 2

[1090] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1091] In modern society, protecting users' personal information and providing an optimized user experience are urgent issues. However, existing systems face privacy risks due to the external transmission of personal information, and it is difficult to adjust services in real time based on the user's emotional state. A system that solves these issues and provides optimal services using emotion recognition while protecting user privacy is needed.

[1092] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1093] In this invention, the server includes means for storing personal information using a local database and restricting external transmission, means for clustering based on user feature data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external server, means for utilizing an emotion engine that recognizes the user's emotional state, and means for adjusting service content in real time based on the emotional state. This makes it possible to provide optimal services based on the user's emotional state while protecting the user's personal information.

[1094] A "local database" is a database used to store a user's personal information on the device and restrict its transmission to external devices.

[1095] "Clustering" is a method of grouping users with similar characteristics based on user characteristic data.

[1096] A "federated learning model" is a machine learning model that is constructed by learning data collected from multiple users locally on each device and aggregating only the learning results (parameters).

[1097] "Learning parameters" are data that indicate the results of local learning by each device in a federated learning model, and by aggregating these data, the accuracy of the model is improved.

[1098] An "emotion engine" is a technology that analyzes user input data (text, voice, images, etc.) and recognizes their emotions.

[1099] "Emotional State" means the current mental and emotional state identified from data entered by the User.

[1100] "Real-time adjustment of service content" is a function that instantly changes the content and method of the service provided based on the user's emotional state recognized by the emotion engine.

[1101] "Data sovereignty management" is a management method that ensures that users' personal information is used under their own control and prevents it from leaking to the outside.

[1102] "Privacy protection" refers to measures to prevent users' personal information and emotional data from being misused by third parties.

[1103] MODE FOR CARRYING OUT THE INVENTION

[1104] This invention is a system that provides advanced services while protecting privacy by storing users' personal information in a local database and restricting its transmission to external parties. This system combines clustering based on user feature data, the construction of a federated learning model, and an emotion recognition engine to adjust service content in real time.

[1105] Data storage method

[1106] First, when the system starts up, the server checks for the existence of a local database, and if the file does not exist, it creates a new one. This database stores each user's personal information and usage data. Specifically, it uses a database management system such as SQLite. For example, user information is stored in a file named "users.db."

[1107] Clustering Method

[1108] Next, the server collects feature data for clustering based on user behavior patterns and attribute information. This process uses machine learning libraries such as scikit-learn. Specifically, it uses the K-means algorithm and classifies users into five groups by setting "n_clusters=5," for example.

[1109] Federated Learning Methods

[1110] In the federated learning step, the server collects data from multiple devices and performs local learning on each device. Each device performs learning using, for example, TensorFlow Federated, and sends the resulting learning parameters to the server. The server aggregates these parameters and builds an overall AI model.

[1111] Emotion recognition method

[1112] When a user inputs text, voice, and image data, the server uses an emotion engine to recognize the user's emotional state from these input data. Specifically, the emotion recognition engine uses a general emotion recognition API. For example, if a user inputs a message such as "I had so much fun today!", the server identifies the emotion "joy" from the context and stores it in a database.

[1113] Real-time adjustment method

[1114] The server adjusts the services it provides in real time based on the user's emotional state as recognized by the emotion engine. For example, if a user is recognized as feeling "stressed," it will recommend music and videos with a relaxing effect. As a specific example, a general music streaming API can be used for music recommendations. The server reflects these recommendations on the user's device in real time.

[1115] Examples and prompts

[1116] As a specific example, the server performs federated learning using purchasing data and emotional data collected from each user. For example, it analyzes each user's purchasing patterns and emotional state and clusters users with similar patterns. Then, it inputs all users' purchasing data and emotional data into a federated learning algorithm to create an integrated model. This model then makes product recommendations optimized for each individual user.

[1117] Example prompt sentence:

[1118] "Use federated learning to build the optimal product recommendation model using user purchasing data and sentiment data."

[1119] "Recognize emotions from user text input and tailor your service based on those emotions."

[1120] As described above, the system according to the present invention provides optimal services based on the user's emotions while protecting personal information.

[1121] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1122] How the system handles programs

[1123] Step 1:

[1124] The server initializes the local database.

[1125] Input: System startup state

[1126] BEHAVIOR: The server checks for the existence of an SQLite database file called "users.db" and creates it if it doesn't exist. It then connects to the database using the sqlite3 library and sets up the users table.

[1127] Output: Initialized local database

[1128] Step 2:

[1129] The server collects user feature data and performs clustering.

[1130] Input: User behavior patterns and attribute information (e.g., age, gender, purchase history)

[1131] How it works: The server uses the scikit-learn library to perform K-means clustering on the feature data. For example, run kmeans = KMeans(n_clusters=5) to classify users into 5 groups. After clustering, it saves a list of users classified into groups.

[1132] Output: Clustered user data

[1133] Step 3:

[1134] The server collects data for implementing federated learning and performs the learning.

[1135] Input: User data collected from each device

[1136] How it works: The server prepares the data to be handed over to each device, and uses the TensorFlow Federated library to perform local training. The trained parameters on each device are sent to the server, which aggregates these parameters and builds the overall AI model.

[1137] Output: AI model with aggregated learning parameters

[1138] Step 4:

[1139] The server uses an emotion engine to recognize the user's emotional state.

[1140] Input: User input data (text, audio, images, etc.)

[1141] How it works: The server uses an emotion recognition engine to identify the user's emotions. For example, using an emotion recognition API, it analyzes the text data "I had so much fun today!", identifies the emotional state "joy," and stores it in the database.

[1142] Output: Recognized emotion data

[1143] Step 5:

[1144] The server adjusts the service content in real time based on the user's emotional state.

[1145] Input: Recognized emotion data

[1146] How it works: The server adjusts the content of the service provided to the user based on the emotion data. For example, if the server detects that the user is feeling stressed, it will use a music streaming API to retrieve an appropriate relaxation playlist and recommend it to the user's device to provide relaxing music.

[1147] Output: Tailored service delivery

[1148] Step 6:

[1149] The server monitors the overall system function, gathering feedback and making improvements as needed.

[1150] Input: Usage of the provided service and user feedback

[1151] How it works: The server monitors user feedback and usage data to analyze the system's performance and accuracy, which in turn allows it to make improvements to improve the accuracy of the federated learning model and emotion recognition engine.

[1152] Output: New version of improved system functionality

[1153] Through this series of steps, the system can provide advanced and optimized services based on user sentiment while protecting user privacy.

[1154] (Application example 2)

[1155] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1156] Conventional service provision systems not only pose privacy risks by transmitting users' personal information to external parties, but also lack the ability to adapt to users' emotional states. This makes it difficult to provide appropriate services that reflect users' real-time emotions. Furthermore, in today's world where both personal information protection and service quality are required, providing optimal services while ensuring user privacy is a challenge.

[1157] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1158] In this invention, the server includes means for storing personal information using a local database and restricting external transmission, means for clustering based on user feature data, means for constructing a federated learning model using data collected from multiple users, means for transmitting learning parameters of the federated learning model to an external server, means for analyzing the emotional state of the user in real time using an emotion recognition engine, and means for dynamically adjusting the content and method of service provision based on the analysis results, thereby enabling the provision of real-time services according to emotions while protecting user privacy.

[1159] A "local database" is a database that stores users' personal information and restricts its transmission to external parties.

[1160] "Clustering" is a technique for grouping multiple users based on their characteristic data.

[1161] A "federated learning model" is a model that is constructed using data collected from multiple users and transmits only the learning parameters to an external server.

[1162] A "learning parameter" is a value that indicates the characteristics of the data used when constructing an associative learning model.

[1163] The "external server" is a server that receives the learning parameters and performs analysis and processing.

[1164] An "emotion recognition engine" is a system for analyzing a user's emotional state in real time.

[1165] "Dynamic adjustment" is the process of changing service content and delivery methods in real time based on analysis results.

[1166] "Protecting privacy" means managing users' personal information to prevent it from being leaked to the outside without permission.

[1167] "User characteristic data" refers to data such as user behavior patterns and attribute information.

[1168] "Real-time analysis" means instantly determining the user's emotional state on the spot.

[1169] This invention is a system that analyzes user emotions in real time in brick-and-mortar stores and dynamically adjusts service content based on that data. It uses a server and smart glasses in combination to provide optimal services while protecting user privacy.

[1170] The server uses a local database to store personal information and restricts external transmission. This local database stores each user's personal information and usage data. Clustering methods are used to classify users based on their feature data, and a federated learning model is constructed using data collected from multiple users. Learning parameters are sent to an external server, but the personal information itself is not sent externally, protecting privacy.

[1171] The smart glasses have a built-in camera that acts as an input device for analyzing the user's facial expressions and voice. The emotion recognition engine uses this camera to analyze the user's emotions in real time. The analyzed emotion data is sent to the server, which dynamically adjusts the service based on the user's interests and state.

[1172] Specifically, it works as follows:

[1173] 1. The server analyzes the user's emotional state using image data acquired from the smart glasses.

[1174] 2. The analyzed emotion data is stored in a local database and classified into groups of users with similar emotion patterns using clustering methods.

[1175] 3. The data analyzed by the emotion recognition engine is reflected in the overall AI model using a federated learning model.

[1176] 4. Based on this model, the server recommends the most suitable products and services for each user and adjusts the delivery method in real time.

[1177] The main hardware used is smart glasses and a server. The software uses OpenCV and Keras (with TensorFlow backend). OpenCV functions as a face detector, and Keras is used for the sentiment analysis model. Data is processed in real time, and appropriate services are automatically provided based on the analysis results.

[1178] For example, if a store staff member is wearing smart glasses and a camera captures a customer's facial expression, the image will be used by an emotion recognition engine to analyze the customer's emotional state. If the customer is feeling stressed, the system will recommend products that have a relaxing effect. This will improve the customer experience and enhance the quality of service at the store.

[1179] Examples of prompts include:

[1180] "What emotional state is this customer in right now?"

[1181] "Provide product recommendations that match customer emotions."

[1182] This will realize a system that can provide optimal services based on emotions in real time while protecting user privacy.

[1183] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1184] Step 1:

[1185] The smart glasses capture the user's facial image using the built-in camera. The image data obtained from the camera is the input. This image data is passed to the next processing step.

[1186] Step 2:

[1187] The server receives the image data and performs face detection using OpenCV. Specifically, it performs grayscale conversion and histogram equalization on the image data, and inputs it into a face detector. The face detector identifies the face in the image and outputs its coordinate information and the face image.

[1188] Step 3:

[1189] The server inputs the facial image obtained from the face detector into the emotion recognition engine. This facial image is passed to a pre-trained Keras emotion recognition model for emotion analysis. The emotion recognition model classifies the emotional state (e.g., anger, happiness, surprise, etc.) from the input facial image and outputs the result as emotion data.

[1190] Step 4:

[1191] The server receives emotion data from the emotion recognition engine and stores it in a local database. The local database classifies this emotion data using a clustering method and reconstructs it as user feature data. This allows emotion patterns for each individual user to be stored in the database.

[1192] Step 5:

[1193] The server updates the federated learning model using feature data collected from multiple users based on the clustered emotion data. This federated learning model trains locally using each user's data and sends the model parameters obtained as a result of the training to an external server. The external server receives this and updates the overall AI model.

[1194] Step 6:

[1195] The server receives the updated AI model returned from the external server and combines it with the emotional data to recommend the most suitable products and services for the user. This recommendation information is notified to the staff in real time via the smart glasses. For example, a message such as "Please recommend a relaxing aroma" may appear on the display of the staff member's smart glasses.

[1196] Step 7:

[1197] Store staff can provide dynamically tailored services to users based on the recommendations displayed on the smart glasses. This process enables real-time emotion recognition and service adjustment, improving user satisfaction.

[1198] Through these steps, a system will be realized in which the server, smart glasses, emotion recognition engine, federated learning model, etc. work together to provide optimal services to users.

[1199] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1200] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1201] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1202] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1203] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1204] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1205] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1206] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1207] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1208] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1209] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1210] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1211] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1212] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1213] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1214] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1215] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1216] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1217] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1218] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1219] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1220] The following is further disclosed regarding the above embodiment.

[1221] ---

[1222] (Claim 1)

[1223] Use of a local database to store personal information and limit its transmission;

[1224] A method for clustering based on user feature data;

[1225] A means for building a federated learning model using data collected from multiple users;

[1226] means for transmitting learning parameters of the federated learning model to an external server;

[1227] A system including:

[1228] (Claim 2)

[1229] 2. The system of claim 1,

[1230] The system further comprising means for data sovereignty management of the personal information.

[1231] (Claim 3)

[1232] 2. The system of claim 1,

[1233] A system with management functions to provide optimal service while protecting user privacy.

[1234] "Example 1"

[1235] (Claim 1)

[1236] Use local storage to store personal information and limit its transmission;

[1237] A means for performing clustering based on user attribute data;

[1238] A means for constructing a federated learning model using data collected from multiple users;

[1239] means for transmitting the learning parameters of the associative learning model to an external computer;

[1240] means for aggregating the learning parameters and updating a final model;

[1241] A system including:

[1242] (Claim 2)

[1243] 10. The system of claim 1, further comprising means for performing data authority management of personal information.

[1244] (Claim 3)

[1245] 10. The system according to claim 1, further comprising a management function for providing optimal service while protecting user privacy.

[1246] "Application Example 1"

[1247] (Claim 1)

[1248] Use of a local database to store personal information and limit its transmission;

[1249] A method for clustering based on user feature data;

[1250] A means for building a federated learning model using data collected from multiple users;

[1251] means for transmitting learning parameters of the federated learning model to an external server;

[1252] a smartphone application means including administrative functions for providing personalized promotions and offers;

[1253] A system including:

[1254] (Claim 2)

[1255] 10. The system of claim 1, further comprising means for performing data sovereignty management of personal information.

[1256] (Claim 3)

[1257] 10. The system according to claim 1, further comprising a management function for providing optimal service while protecting user privacy.

[1258] "Example 2: Combining Emotion Engines"

[1259] (Claim 1)

[1260] Use of a local database to store personal information and limit its transmission;

[1261] A method for clustering based on user feature data;

[1262] A means for building a federated learning model using data collected from multiple users;

[1263] means for transmitting learning parameters of the federated learning model to an external server;

[1264] a means for utilizing an emotion engine that recognizes the user's emotional state;

[1265] a means for adjusting service content in real time based on emotional state;

[1266] A system including:

[1267] (Claim 2)

[1268] The system of claim 1, which performs data sovereignty management of personal information.

[1269] (Claim 3)

[1270] 10. The system according to claim 1, further comprising a management function for providing optimal service while protecting user privacy.

[1271] "Application example 2 when combining emotion engines"

[1272] (Claim 1)

[1273] Use of a local database to store personal information and limit its transmission;

[1274] A method for clustering based on user feature data;

[1275] A means for building a federated learning model using data collected from multiple users;

[1276] means for transmitting learning parameters of the federated learning model to an external server;

[1277] A means for analyzing the user's emotional state in real time using an emotion recognition engine;

[1278] A means for dynamically adjusting the content and delivery method of services based on the analysis results;

[1279] A system including:

[1280] (Claim 2)

[1281] 10. The system of claim 1, further comprising means for performing data sovereignty management of personal information.

[1282] (Claim 3)

[1283] 10. The system according to claim 1, further comprising a management function for providing optimal service while protecting user privacy. [Explanation of symbols]

[1284] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. Use of a local database to store personal information and limit its transmission; A method for clustering based on user feature data; A means for building a federated learning model using data collected from multiple users; means for transmitting learning parameters of the federated learning model to an external server; A system including:

2. 10. The system of claim 1, The system further comprising means for data sovereignty management of the personal information.

3. 10. The system of claim 1, A system with management functions to provide optimal service while protecting user privacy.

Images