system

Abstract

We provide the system. [Solution] Means for collecting agent usage data and user feedback data, A means for analyzing the collected data and quantifying the performance of each agent, A means for issuing digital identification information to an agent whose performance has been quantified and for recording its ownership, A means of providing a market that makes the aforementioned digital identification information tradable, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional agent market, there are problems that the performance and reliability of agents are opaque and it is difficult for users to select an optimal agent. Also, agent developers cannot obtain legitimate profits according to their performance, and there is a lack of incentives for new technology development. It is necessary to solve such problems.

Means for Solving the Problems

[0005] This invention provides a means for collecting and analyzing agent usage data and user feedback to quantify the performance of each agent. Based on this quantified performance, digital identification information (NFTs) is issued, and ownership is recorded using blockchain technology to ensure transparent management. Furthermore, a marketplace is provided where this digital identification information can be traded, and a fair trading environment is provided by recording transaction history and distributing revenue to developers.

[0006] An "agent" is a piece of software that autonomously performs specific tasks or services.

[0007] "Usage data" refers to data that shows how often and how the agent is being used.

[0008] "User feedback" refers to information including ratings and comments from users who have used the agent.

[0009] "Data analysis" is the process of analyzing collected data to evaluate the performance of an agent.

[0010] "Quantifying performance" refers to expressing the agent's operational results and reliability numerically based on indicators.

[0011] "Digital identification information" refers to information that indicates the ownership and characteristics of an asset managed in digital format.

[0012] "Recording ownership" refers to digitally storing the rights and relationships regarding agents and their related information.

[0013] Blockchain technology is a system that uses distributed ledger technology to ensure the security and transparency of data.

[0014] "To make tradable" refers to making certain digital assets available for buying and selling on the market.

[0015] The "market" refers to the platform where digital assets and services are traded.

[0016] The "transaction history" refers to the information indicating the records of past sales and purchases.

[0017] "Distribute the proceeds" refers to the process of dividing the profits obtained from transactions and service provision among the relevant parties.

Brief Description of the Drawings

[0018] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12]It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the language used in the following description will be explained.

[0021] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] Embodiments of this invention relate to a system for evaluating agent performance and managing digital ownership. This system mainly consists of a server, terminals, and users, and operates in the following manner.

[0040] Data collection and analysis

[0041] The server prepares an interface for collecting agent usage data and user feedback, and retrieves this data periodically. Specifically, it collects data on which agents were used, to what extent, and how users evaluated those agents, through APIs and log file analysis.

[0042] The server uses an AI model based on the collected data to quantify the performance of each agent. This quantification process takes into account various metrics such as usage frequency, user satisfaction, and bug rate.

[0043] Issuance of digital identification information

[0044] The server generates digital identification information for each agent based on quantified performance evaluations. This identification information is issued on the blockchain and represents ownership of the agent.

[0045] The server uses smart contracts to securely and transparently manage ownership and transaction history of this digital identification information on the blockchain.

[0046] Marketplace transactions

[0047] Users can access a marketplace provided via the web or application to view a list of agents, their ratings, and their corresponding digital identification information.

[0048] Users select the digital identification information of agents they are interested in and proceed with the purchase on the marketplace. This process is typically carried out using cryptocurrency, with payment made through a wallet application.

[0049] Profit sharing

[0050] The server updates the record immediately upon completion of a transaction and distributes the revenue based on the transaction to the agent's creator or developer. This ensures that developers receive fair compensation for the revenue generated from the agents they create.

[0051] In this way, agent usage, evaluation, and transactions can be centrally managed, providing a fair and transparent environment for developers and users. This can improve the reliability of agent selection and transactions, while also fostering new incentives for developers.

[0052] The following describes the processing flow.

[0053] Step 1:

[0054] The server activates a logging system to monitor agent usage data. This allows for real-time recording of each agent's usage frequency and execution time.

[0055] Step 2:

[0056] The server automatically displays a feedback form when a user uses the agent. The feedback is collected as evaluation scores and comments and stored in a database.

[0057] Step 3:

[0058] The server runs an AI engine to analyze the collected usage data and feedback. The AI ​​evaluates this data and calculates a numerical performance score for each agent.

[0059] Step 4:

[0060] The server generates a unique digital identification (NFT) for each agent based on its performance score and issues it on the blockchain. This identification includes the agent's ownership and evaluation information.

[0061] Step 5:

[0062] The server publishes the generated digital identification information through the marketplace platform. This allows users to view detailed information about each agent and the NFTs available for trading.

[0063] Step 6:

[0064] Users select an agent they are interested in from those available for trading on the marketplace and begin the purchase process. Secure transactions are conducted via smart contracts using cryptocurrency.

[0065] Step 7:

[0066] The server records the transaction history via the blockchain when a transaction is completed and immediately distributes the transaction revenue to the developer. This allows the developer to receive revenue based on their agent.

[0067] (Example 1)

[0068] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0069] In modern society, the proliferation of information devices and software agents has made performance evaluation and promoting their appropriate use crucial issues. However, there are current challenges in establishing appropriate performance evaluation methods, ensuring fair and transparent ownership management, and guaranteeing the reliability of transactions based on these evaluations. There is a need for a system that can solve these problems, enabling improved performance and increased use of information devices, as well as fair compensation for developers.

[0070] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0071] In this invention, the server includes means for collecting data on the operating status of information devices and user evaluations; means for analyzing the collected data using an AI algorithm and quantifying the performance of each information device; and means for generating digital identification information for the information devices whose performance has been quantified and recording their ownership on a distributed ledger. This provides a system that enables appropriate performance evaluation of information devices, fair and transparent management of their ownership, and reliable transactions based on this.

[0072] "Information equipment" refers to all devices that process and communicate digital data, and includes computers and terminals with communication functions.

[0073] "User" refers to an individual or organization that utilizes the functions provided by information devices or software agents.

[0074] "Means of data collection" refers to the function of providing a mechanism for collecting and storing necessary information from information devices and users.

[0075] An "AI algorithm" refers to a calculation procedure that uses artificial intelligence technology to analyze data and determine the performance of equipment.

[0076] "Digital identification information" refers to unique digital data associated with information devices or agents, which can be used to identify ownership and attributes.

[0077] A "distributed ledger" refers to a system that records transaction information and other data synchronously across multiple locations in a way that makes tampering difficult, and includes technologies such as blockchain.

[0078] An "e-commerce environment" refers to a marketplace where goods and services can be bought and sold via the internet or other means.

[0079] "Transaction records" refer to data that records the details of transactions conducted in an e-commerce environment, and typically include the date and time of the sale, the goods, and the price.

[0080] The embodiments for carrying out the invention are described below.

[0081] This invention is a system that enables performance evaluation of information devices and management of their digital identification information. This system mainly consists of three components: a server, a terminal, and a user.

[0082] The server acquires usage data and user ratings from information devices using APIs and log analysis techniques. Specifically, the hardware consists of a computer server for managing the database, while the software includes APIs for data collection and log analysis tools. The server then analyzes this data using AI algorithms (e.g., deep learning models utilizing TENSORFLOW®) to quantify the performance metrics of each information device. This allows for the output of specific figures, such as, for example, that device A is used 20 times a day and has an 80% satisfaction rate.

[0083] After performance evaluation is complete, the server generates digital identification information for each information device and records this identification information and transaction history on a distributed ledger (e.g., blockchain technology). This operation also utilizes smart contract technology to ensure the transparent and secure management of identification information.

[0084] The terminal (a computer or smartphone operated by the user) connects to an e-commerce environment provided through a web browser or dedicated application and is used to view a list of information devices, their ratings, and digital identification information. For example, when a user accesses the market using their terminal, they can view ratings of information devices of interest and decide whether to purchase them.

[0085] Users complete the purchase process in the e-commerce environment through their devices and make payments using cryptocurrency. Payments are made via a wallet application, and based on the confirmed transaction from the server, legitimate revenue is automatically distributed to the developers.

[0086] As a concrete example, consider the evaluation process for a voice assistant application. When a user uses assistant application A, the server automatically collects usage frequency and evaluation data, which is then analyzed by an AI model to calculate a performance score. Based on this data, digital identification information for application A is generated, and the user confirms this information in the marketplace before making a purchase. Ultimately, the developer receives revenue based on sales.

[0087] Examples of input prompts for the generating AI model include: "Evaluate the agent's performance and generate new incentive suggestions that take user feedback into consideration."

[0088] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0089] Step 1:

[0090] The server collects usage data and user evaluation data through the APIs and log files of information devices. Inputs include the information device's identification ID, usage count, and user satisfaction rating. This data is stored in a database and output as organized information based on a data schema for use in subsequent analysis.

[0091] Step 2:

[0092] The server loads the collected data as input to an AI algorithm. Specifically, it uses a deep learning framework such as TensorFlow to run a data analysis model. The AI ​​algorithm evaluates metrics such as usage frequency, satisfaction level, and bug rate, and quantifies the performance of each information device. As a result, it outputs a specific performance score for each information device.

[0093] Step 3:

[0094] The server generates unique digital identification information for each information device based on its performance score. This identification information is registered in a distributed ledger system using a smart contract. The input includes the generated performance score and the information device's identification ID. The output is a secure and transparent recording of digital identification information that can be verified on the blockchain.

[0095] Step 4:

[0096] The terminal provides an information commerce environment in response to user access requests. The terminal displays an interface for users to browse a list of information devices and verify their evaluations and digital identification information. Inputs include the user's search criteria and the IDs of information devices of interest. Outputs include performance evaluations and purchase options for the information devices displayed on the interface.

[0097] Step 5:

[0098] The user uses a terminal to select the digital identification information of the information device they are interested in and begins the purchase process. The user makes the payment using a wallet application. Inputs include the user's wallet information and the purchase amount. Output is a confirmation message indicating that the payment has been completed. Transaction information is also recorded on a distributed ledger.

[0099] Step 6:

[0100] The server updates the transaction record as soon as the transaction is completed and distributes the revenue to the information device developers. Inputs include transaction history and developer information. Outputs include the allocation of legitimate revenue to the developers' wallets through an automated transfer process via smart contracts.

[0101] (Application Example 1)

[0102] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0103] There is a need to accurately evaluate agent usage and performance, and to manage digital ownership securely and transparently. It is also crucial to provide users with an environment where they can easily verify agent performance and conduct secure transactions. Traditional methods struggle to meet these requirements, resulting in inefficiencies and a lack of reliability.

[0104] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0105] In this invention, the server includes a device for collecting information on agent usage and user feedback, a device for analyzing the collected information and quantifying the performance of each agent, and a device for issuing digital identification information to the agents whose performance has been quantified and recording their ownership. This enables efficient and reliable evaluation of agent performance and management of digital ownership.

[0106] An "agent" is a piece of software or program that automatically performs a specific task or function.

[0107] "User feedback" refers to the impressions and opinions provided by users after using an agent, and is an important indicator in performance evaluation.

[0108] "Digital identification information" refers to electronic identification information issued based on the performance and usage of an agent, and is used to manage ownership and transaction history on the blockchain.

[0109] Blockchain technology is a technology that functions as a distributed ledger, enabling data transparency and preventing tampering.

[0110] The "marketplace" is a platform for trading agents' digital identification information, where users can buy and rate agents.

[0111] A "transaction history" is a record of an agent's buying, selling, and exchange activities in the market, and is recorded to ensure transparency and reliability.

[0112] A "profit-distributing device" is a device equipped with the function of appropriately distributing the profits obtained after a market transaction is completed to the agent providers.

[0113] This system works in conjunction with servers, terminals, and users to securely and efficiently manage agent performance evaluation and digital ownership. The system is structured as follows, with each element playing its own role.

[0114] The server provides an interface for collecting agent usage data and user feedback. Through APIs and log file analysis, it obtains data such as agent usage frequency, user satisfaction, and bug rates. The collected data is analyzed using an AI model, and the performance of each agent is quantified. This quantified data forms the basis for generating digital identification information.

[0115] The terminal displays information from the agent marketplace as a device accessed by the user. Users can access the marketplace via a device such as a smartphone and view the agent list and individual ratings. Blockchain technology is used to securely manage ownership of digital identification information and transaction history. In this process, ownership transfers and revenue distribution are automatically carried out by smart contracts.

[0116] The system is designed using mobile app development frameworks such as Flutter® and React Native, and can utilize Ethereum's Web3.js as its blockchain API. This ensures smooth operation and security for the entire system.

[0117] As a concrete example, a user uses a terminal to search for an agent they are interested in and check its performance evaluation. When purchasing an agent they like, the transaction is conducted via cryptocurrency, and after the purchase, ownership is managed on the blockchain. An example of a prompt using a generated AI model is, "Retrieve data to list the agent's transaction history and optimize the process of verifying ownership on the blockchain." This prompt allows the system to process the request efficiently and accurately.

[0118] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0119] Step 1:

[0120] The server collects data on agent usage and user feedback. Inputs are agent usage logs and user reviews, and output is a database of this information. Data processing involves retrieving information via API and organizing it into the database. Specifically, it periodically collects usage frequency and user ratings for each agent.

[0121] Step 2:

[0122] The server analyzes the collected data and quantifies the performance of each agent. The input is the data collected in step 1, and the output is the quantified performance evaluation. For data calculation, an AI model is used to generate a score that takes into account usage frequency, user satisfaction, and bug occurrence rate. Specifically, a machine learning algorithm is applied to automatically calculate the performance evaluation.

[0123] Step 3:

[0124] The server issues digital identification information to agents whose performance has been quantified, and records ownership of that information on the blockchain. The input is the performance evaluation data obtained in step 2, and the output is the registration of the digital identification information on the blockchain. Specifically, a smart contract is used to generate the identification information and publish it on the blockchain.

[0125] Step 4:

[0126] The terminal allows the user to access the marketplace and view a list of agent performance ratings. The input is the digital identification information obtained in step 3, and the output is the agent list displayed to the user. Specifically, the evaluation information is visually organized and provided in a format that is easy for the user to compare.

[0127] Step 5:

[0128] The user selects the digital identification information of an agent they are interested in and proceeds with the purchase. The input is the displayed agent information, and the output is a confirmation of the purchased agent. Specifically, the user completes the purchase by processing payment in cryptocurrency through their wallet.

[0129] Step 6:

[0130] The server updates the record after a transaction is completed and distributes the revenue to the agent provider. The input is transaction data, and the output is updated revenue-related data. Specifically, revenue is automatically sent via a smart contract based on defined distribution rules.

[0131] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0132] Embodiments of this invention are systems aimed at reflecting user emotions in the performance evaluation of agents, and primarily operate with a configuration including a server, terminal, user, and emotion engine.

[0133] Data collection and analysis

[0134] The server provides an interface for collecting user emotion data, in addition to agent usage data and user feedback. Emotion data is acquired via the camera and microphone installed on the device and obtained by analyzing the user's voice tone and facial expressions.

[0135] The server uses an emotion engine to analyze the user's emotions and measure the emotional changes in real time. For example, if the voice changes from a calm tone to an excited tone while the user is using the agent, this could be attributed to the agent's effectiveness or difficulty level. This information plays a crucial role in evaluating the agent's performance.

[0136] Quantification of performance evaluation

[0137] The server uses AI to analyze all collected data, including emotional data, and quantifies the agent's performance. This performance evaluation includes changes in emotional responses and the tone of user feedback.

[0138] Issuance of digital identification information

[0139] The server generates digital identification information (NFTs) for each agent based on quantified evaluations and issues them on the blockchain. This identification information records detailed agent performance data, including sentiment ratings.

[0140] Marketplace transactions

[0141] Users can access the marketplace to check the performance and emotional responses of agents. This allows them to select the most suitable agent, taking emotional data into consideration.

[0142] Users purchase and trade the digital identification information of specific agents. Transactions are conducted securely using cryptocurrency and via smart contracts.

[0143] Profit sharing

[0144] The server records transaction history and distributes revenue to developers based on the value of agents that utilize emotional data. This allows developers to more accurately understand the quality of products by considering emotional data, thereby promoting the development of agents that contribute to improving the user experience.

[0145] The introduction of this system will enable agent evaluation and transaction processes that reflect user sentiment, improving convenience and reliability for both users and developers.

[0146] The following describes the processing flow.

[0147] Step 1:

[0148] As soon as the agent starts operating, the device uses its camera and microphone to collect the user's facial expressions and voice in real time. This data is sent to the emotion engine as a feed to evaluate the user's emotions.

[0149] Step 2:

[0150] The server analyzes user emotion data sent from the terminal using an emotion engine. This analysis allows the server to capture the user's emotional state, such as excitement, stress, and satisfaction, as numerical values.

[0151] Step 3:

[0152] The server integrates emotional data analyzed by the emotion engine, agent usage frequency data, and user feedback data, and calculates an overall performance evaluation score using an AI model. This score accurately reflects the agent's effectiveness and the user experience.

[0153] Step 4:

[0154] The server generates digital identification information for each agent based on the calculated performance evaluation score. This digital identification information is recorded on the blockchain and issued as an NFT representing the agent's digital ownership.

[0155] Step 5:

[0156] The server lists digital identification information, including performance evaluation scores and sentiment data analysis results, on the marketplace platform, making it accessible to users.

[0157] Step 6:

[0158] Users browse a list of agents from the marketplace and select the best agent based on sentiment data. When purchasing an agent's NFT, payment is made via cryptocurrency using a wallet app.

[0159] Step 7:

[0160] The server automatically updates the record on the blockchain once a transaction is completed and distributes the revenue based on the transaction to the developers. This revenue includes value based on sentiment data and is fairly returned to the developers.

[0161] (Example 2)

[0162] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0163] Existing agent evaluation systems have a problem in that they do not adequately evaluate agent performance while considering user emotions, resulting in the user experience not being fully utilized. Furthermore, there is a need for suggestions for agent improvements based on evaluation results, as well as ensuring transparency and reliability in the evaluation process.

[0164] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0165] In this invention, the server includes means for acquiring user voice and video via a terminal and analyzing emotions; means for integrating agent usage, user feedback, and emotion analysis data to quantify the performance of each agent; and means for generating digital identification information for the agents whose performance has been quantified and managing that identification information using distributed ledger technology. This enables highly accurate evaluation of agents that reflect user emotions and transparent system operation with suggestions for improvement.

[0166] A "terminal" is a device that acquires the user's voice and video and transmits that data to a server.

[0167] "Audio and video" refers to acoustic and visual data that indicates the user's state, and these are sources of information used to analyze emotions.

[0168] "Methods for analyzing emotions" refer to processes and technologies that estimate a user's emotions in real time based on acquired audio and video data.

[0169] An "agent" is software or hardware that performs a specific function and interacts with the user.

[0170] "Usage" refers to information about how the agent is being used by users.

[0171] "User feedback" refers to information that shows the opinions and impressions that users provide regarding their use of the agent.

[0172] "Emotional analysis data" refers to data obtained by analyzing the emotional state of users, and it forms the basis for evaluation.

[0173] "Methods for quantifying performance" refer to the process of expressing the agent's performance numerically based on collected data.

[0174] "Digital identification information" refers to unique digital data generated based on the performance evaluation of an agent, and is used for ownership and transactions.

[0175] "Distributed ledger technology" is a technology for managing and recording digital identification information in a tamper-proof manner, and blockchain technology is generally used for this purpose.

[0176] One embodiment of this invention provides a system for performing real-time performance evaluation of an agent while taking user emotions into consideration. This system mainly consists of a server, a terminal, a user, and an emotion analysis engine.

[0177] First, the device uses its camera and microphone to capture the user's voice and video. This allows it to capture the user's instantaneous emotional state. The voice data is used to analyze features such as voice tone and volume, and the video data is used for facial expression recognition. This data is transmitted to the server in real time.

[0178] The server inputs audio and video data received from the terminal into an emotion analysis engine to analyze the user's emotions. Specifically, it uses an audio analysis model to classify voice tone and an expression recognition model to analyze the user's facial movements. This allows the user's emotions to be estimated in real time with labels such as "joy," "sadness," and "surprise."

[0179] Subsequently, the server integrates agent usage data, user feedback data, and analyzed sentiment data, and uses AI to quantify the agent's performance. This quantified data is then used in conjunction with a generative AI model. An example of a prompt used in this process is, "What improvements can be made to the agent based on this sentiment pattern?"

[0180] Next, the server generates digital identification information based on quantified performance evaluations and manages this information in a tamper-proof manner using distributed ledger technology, such as blockchain. Blockchain management enhances the transparency and reliability of agent performance evaluations.

[0181] Finally, users can view agent performance data on the marketplace and trade the digital identification information of agents they are interested in. These transactions are conducted using cryptocurrency and are secured by smart contracts.

[0182] In this way, by utilizing user sentiment data to evaluate and improve agent performance, it is possible to provide a better user experience.

[0183] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0184] Step 1:

[0185] The device acquires the user's voice and video using a camera and microphone. The input consists of the user's real-time voice and facial images, which are transmitted to the server as digital data. Specifically, the voice recognition software installed on the device captures the user's voice and digitizes its waveform. In addition, the video capture function captures the user's facial movements frame by frame, converts them into data, and transmits it.

[0186] Step 2:

[0187] The server receives the input and passes it to the emotion analysis engine. The input here consists of digitized audio and video. The server inputs the audio data into the audio analysis model and the video data into the facial recognition model. As part of the data processing, it analyzes the pitch and speed of the audio and performs calculations to extract facial feature points from the video. The output is the user's emotion label (e.g., "joy," "anger," "anxiety," etc.).

[0188] Step 3:

[0189] The server then integrates the sentiment analysis results with agent usage data and user feedback data. Inputs include the aforementioned sentiment labels, agent activity logs, and user text feedback. For data processing, this data is subjected to AI analysis, and a generative AI model is used to quantify the agent's performance. The output is the agent's evaluation score.

[0190] Step 4:

[0191] The server uses this evaluation score to generate digital identification information. This identification information includes a unique agent ID and evaluation score. The output is a digital identification information (NFT), which is recorded in a distributed ledger system. Specifically, blockchain technology is used to execute a process that stores this data in a tamper-proof manner.

[0192] Step 5:

[0193] Users verify the agent's digital identification information on the marketplace and conduct transactions. The input is the digital identification information of the agent selected by the user. Specifically, the user operates the interface to access the marketplace and completes the purchase procedure using cryptocurrency. The output is the transaction history stored in the user's wallet.

[0194] (Application Example 2)

[0195] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0196] The objective of this invention is to provide a system that reflects user emotions in real time, enables more precise evaluation of agent performance, and optimizes the operation of a robotic device that acts in accordance with user emotions.

[0197] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0198] In this invention, the server includes means for collecting agent usage data and user feedback data; means for analyzing the collected data and emotion data estimated from the user's voice and facial expressions to quantify the performance of each agent; means for issuing digital identification information to the agents whose performance has been quantified and recording their ownership; and means for operating a robot device that changes its behavioral patterns based on the user's emotion data. This enables highly accurate agent performance evaluation that reflects the user's emotions and adaptive behavioral control of the robot device in accordance with the user's emotions.

[0199] An "agent" is a program or device that interacts with users and provides information and support.

[0200] "Usage data" refers to data about how the agent is being used.

[0201] "User feedback" refers to the evaluations and opinions that users provide after using the agent.

[0202] "Analysis" is the act of examining collected data to extract useful information.

[0203] "Emotional data" refers to data on the user's psychological state, estimated based on their voice and facial expressions.

[0204] "Quantifying" means expressing data or information as numerical values.

[0205] "Digital identification information" refers to digital data that records the evaluation and characteristics of an agent.

[0206] "Ownership" refers to a property right relating to a specific object or information.

[0207] "To record" is the act of saving information in a medium.

[0208] A "market" is a place and system where buying and selling take place.

[0209] A "robot device" is a mechanical device that can take actions in response to the user's emotions.

[0210] The server collects data on agent usage and user feedback from the user's device. The device is equipped with a camera and microphone, which are used to acquire voice and facial expression data. The server analyzes the user's facial expressions from the camera footage using image processing with the OpenCV library and estimates emotion data using EmotionRecognizer. It also analyzes voice data using the SpeechRecognition library and estimates emotion data based on voice tone.

[0211] User emotion data is used to operate robotic devices that dynamically change their behavior. This information is used with the MusicPlayer module to select appropriate music and provide an environment that is sensitive to the user's psychological state. For example, if a user shows signs of fatigue after a long day, the server detects this and instructs the robotic device to play relaxation music.

[0212] An example of a prompt message is, "If the system detects that the user is tired, it will switch to relaxation mode. Please play calming music to soothe the user." This allows the user to enjoy a system that provides a comfortable experience tailored to their emotions.

[0213] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0214] Step 1:

[0215] The device uses a camera and microphone to collect user facial expressions and audio data. In this process, the camera acquires image frames in real time, and the microphone captures the user's speech as audio data. The input in this process is the user's video and audio, which are then sent to the server.

[0216] Step 2:

[0217] The server analyzes the received video data using the OpenCV library and evaluates the user's emotions through EmotionRecognizer. Specifically, it extracts facial features through image processing and performs data calculations to estimate emotions based on these features. The output is the user's estimated emotion data.

[0218] Step 3:

[0219] The server analyzes the audio data using the SpeechRecognition library to evaluate the speech tone. The audio is converted to text, and speech tone features are extracted from the text and quantified as emotion data. The output is emotion estimation data based on speech tone.

[0220] Step 4:

[0221] The server integrates the emotional data obtained in steps 2 and 3 and uses a generative AI model to perform a comprehensive emotional assessment. This allows for a detailed understanding of the user's current psychological state. The output at this stage is the integrated emotional assessment data.

[0222] Step 5:

[0223] The server determines the specific actions the robot device should take based on emotion evaluation data. If music playback or adjustments to environmental settings are required, the MusicPlayer module is used to select appropriate music and settings. A concrete example of this action might include playing calming music to soothe the user.

[0224] Step 6:

[0225] The robotic device receives instructions from the server and performs actions to adjust the user's environment. This includes playing specified music and adjusting lighting. As a final output, the user experiences a comfortable environment tailored to their emotions.

[0226] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0227] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0228] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0229] [Second Embodiment]

[0230] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0231] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0232] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0233] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0234] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0235] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0236] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0237] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0238] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0239] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0240] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0241] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0242] Embodiments of this invention relate to a system for evaluating agent performance and managing digital ownership. This system mainly consists of a server, terminals, and users, and operates in the following manner.

[0243] Data collection and analysis

[0244] The server prepares an interface for collecting agent usage data and user feedback, and retrieves this data periodically. Specifically, it collects data on which agents were used, to what extent, and how users evaluated those agents, through APIs and log file analysis.

[0245] The server uses an AI model based on the collected data to quantify the performance of each agent. This quantification process takes into account various metrics such as usage frequency, user satisfaction, and bug rate.

[0246] Issuance of digital identification information

[0247] The server generates digital identification information for each agent based on quantified performance evaluations. This identification information is issued on the blockchain and represents ownership of the agent.

[0248] The server uses smart contracts to securely and transparently manage ownership and transaction history of this digital identification information on the blockchain.

[0249] Marketplace transactions

[0250] Users can access a marketplace provided via the web or application to view a list of agents, their ratings, and their corresponding digital identification information.

[0251] Users select the digital identification information of agents they are interested in and proceed with the purchase on the marketplace. This process is typically carried out using cryptocurrency, with payment made through a wallet application.

[0252] Profit sharing

[0253] The server updates the record immediately upon completion of a transaction and distributes the revenue based on the transaction to the agent's creator or developer. This ensures that developers receive fair compensation for the revenue generated from the agents they create.

[0254] In this way, agent usage, evaluation, and transactions can be centrally managed, providing a fair and transparent environment for developers and users. This can improve the reliability of agent selection and transactions, while also fostering new incentives for developers.

[0255] The following describes the processing flow.

[0256] Step 1:

[0257] The server activates a logging system to monitor agent usage data. This allows for real-time recording of each agent's usage frequency and execution time.

[0258] Step 2:

[0259] The server automatically displays a feedback form when a user uses the agent. The feedback is collected as evaluation scores and comments and stored in a database.

[0260] Step 3:

[0261] The server runs an AI engine to analyze the collected usage data and feedback. The AI ​​evaluates this data and calculates a numerical performance score for each agent.

[0262] Step 4:

[0263] The server generates a unique digital identification (NFT) for each agent based on its performance score and issues it on the blockchain. This identification includes the agent's ownership and evaluation information.

[0264] Step 5:

[0265] The server publishes the generated digital identification information through the marketplace platform. This allows users to view detailed information about each agent and the NFTs available for trading.

[0266] Step 6:

[0267] Users select an agent they are interested in from those available for trading on the marketplace and begin the purchase process. Secure transactions are conducted via smart contracts using cryptocurrency.

[0268] Step 7:

[0269] The server records the transaction history via the blockchain when a transaction is completed and immediately distributes the transaction revenue to the developer. This allows the developer to receive revenue based on their agent.

[0270] (Example 1)

[0271] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0272] In modern society, the proliferation of information devices and software agents has made performance evaluation and promoting their appropriate use crucial issues. However, there are current challenges in establishing appropriate performance evaluation methods, ensuring fair and transparent ownership management, and guaranteeing the reliability of transactions based on these evaluations. There is a need for a system that can solve these problems, enabling improved performance and increased use of information devices, as well as fair compensation for developers.

[0273] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0274] In this invention, the server includes means for collecting data on the operating status of information devices and user evaluations; means for analyzing the collected data using an AI algorithm and quantifying the performance of each information device; and means for generating digital identification information for the information devices whose performance has been quantified and recording their ownership on a distributed ledger. This provides a system that enables appropriate performance evaluation of information devices, fair and transparent management of their ownership, and reliable transactions based on this.

[0275] "Information equipment" refers to all devices that process and communicate digital data, and includes computers and terminals with communication functions.

[0276] "User" refers to an individual or organization that utilizes the functions provided by information devices or software agents.

[0277] "Means of data collection" refers to the function of providing a mechanism for collecting and storing necessary information from information devices and users.

[0278] An "AI algorithm" refers to a calculation procedure that uses artificial intelligence technology to analyze data and determine the performance of equipment.

[0279] "Digital identification information" refers to unique digital data associated with information devices or agents, which can be used to identify ownership and attributes.

[0280] A "distributed ledger" refers to a system that records transaction information and other data synchronously across multiple locations in a way that makes tampering difficult, and includes technologies such as blockchain.

[0281] An "e-commerce environment" refers to a marketplace where goods and services can be bought and sold via the internet or other means.

[0282] The "transaction record" refers to data that records the details of transactions conducted in an e-commerce trading environment, usually including the date and time of the sale and purchase, goods, amount, etc.

[0283] The embodiments for implementing the invention will be described below.

[0284] This invention is a system that enables the performance evaluation of information devices and the management of their digital identification information. This system mainly consists of three components: a server, a terminal, and a user.

[0285] The server acquires usage data and user evaluations from information devices using APIs and log analysis techniques. As specific hardware, a computer server for managing databases is used, and the software includes APIs for data collection and log analysis tools. Also, the server analyzes this data using an AI algorithm (e.g., a deep learning model using TensorFlow) to quantify the performance indicators of each information device. As a result, specific numerical values such as device A being used 20 times a day and obtaining a satisfaction level of 80% are output.

[0286] After the performance evaluation is completed, the server generates digital identification information for each information device and records the identification information and transaction history on a distributed ledger (e.g., blockchain technology). In this operation, smart contract technology is also used to ensure the transparent and secure management of the identification information.

[0287] The terminal (a computer or smartphone operated by the user) connects to the e-commerce trading environment provided through a web browser or a dedicated application and is used to view the list of information devices, evaluations, and digital identification information. For example, when the user accesses the market using the terminal, they can view the evaluations of information devices of interest and decide whether to purchase.

[0288] Users complete the purchase process in the e-commerce environment through their devices and make payments using cryptocurrency. Payments are made via a wallet application, and based on the confirmed transaction from the server, legitimate revenue is automatically distributed to the developers.

[0289] As a concrete example, consider the evaluation process for a voice assistant application. When a user uses assistant application A, the server automatically collects usage frequency and evaluation data, which is then analyzed by an AI model to calculate a performance score. Based on this data, digital identification information for application A is generated, and the user confirms this information in the marketplace before making a purchase. Ultimately, the developer receives revenue based on sales.

[0290] Examples of input prompts for the generating AI model include: "Evaluate the agent's performance and generate new incentive suggestions that take user feedback into consideration."

[0291] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0292] Step 1:

[0293] The server collects usage data and user evaluation data through the APIs and log files of information devices. Inputs include the information device's identification ID, usage count, and user satisfaction rating. This data is stored in a database and output as organized information based on a data schema for use in subsequent analysis.

[0294] Step 2:

[0295] The server loads the collected data as input to an AI algorithm. Specifically, it uses a deep learning framework such as TensorFlow to run a data analysis model. The AI ​​algorithm evaluates metrics such as usage frequency, satisfaction level, and bug rate, and quantifies the performance of each information device. As a result, it outputs a specific performance score for each information device.

[0296] Step 3:

[0297] The server generates unique digital identification information for each information device based on its performance score. This identification information is registered in a distributed ledger system using a smart contract. The input includes the generated performance score and the information device's identification ID. The output is a secure and transparent recording of digital identification information that can be verified on the blockchain.

[0298] Step 4:

[0299] The terminal provides an information commerce environment in response to user access requests. The terminal displays an interface for users to browse a list of information devices and verify their evaluations and digital identification information. Inputs include the user's search criteria and the IDs of information devices of interest. Outputs include performance evaluations and purchase options for the information devices displayed on the interface.

[0300] Step 5:

[0301] The user uses a terminal to select the digital identification information of the information device they are interested in and begins the purchase process. The user makes the payment using a wallet application. Inputs include the user's wallet information and the purchase amount. Output is a confirmation message indicating that the payment has been completed. Transaction information is also recorded on a distributed ledger.

[0302] Step 6:

[0303] Once a transaction is completed, the server updates its record and distributes the proceeds to the developers of the information devices. The inputs include the transaction history and developer information. As output, legitimate proceeds are allocated to the developers' wallets through an automated transfer process via smart contracts.

[0304] (Application Example 1)

[0305] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0306] There is a need to accurately evaluate the usage status and performance of agents and to manage digital ownership safely and transparently. It is also important to provide an environment in which users can easily check the performance of agents and conduct secure transactions. Conventional methods have difficulty meeting these requirements, and the issues are a lack of efficiency and reliability.

[0307] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0308] In this invention, the server includes a device that collects information on the usage status of agents and user feedback, a device that analyzes the collected information to quantify the performance of each agent, and a device that issues digital identification information for the agent whose performance has been quantified and records its ownership. This enables efficient and reliable performance evaluation of agents and management of digital ownership.

[0309] An "agent" is software or a program that automatically executes specific tasks or functions.

[0310] "User feedback" refers to the impressions and opinions provided by users after using an agent and is an important indicator in performance evaluation.

[0311] "Digital identification information" refers to electronic identification information issued based on the performance and usage of an agent, and is used to manage ownership and transaction history on the blockchain.

[0312] Blockchain technology is a technology that functions as a distributed ledger, enabling data transparency and preventing tampering.

[0313] The "marketplace" is a platform for trading agents' digital identification information, where users can buy and rate agents.

[0314] A "transaction history" is a record of an agent's buying, selling, and exchange activities in the market, and is recorded to ensure transparency and reliability.

[0315] A "profit-distributing device" is a device equipped with the function of appropriately distributing the profits obtained after a market transaction is completed to the agent providers.

[0316] This system works in conjunction with servers, terminals, and users to securely and efficiently manage agent performance evaluation and digital ownership. The system is structured as follows, with each element playing its own role.

[0317] The server provides an interface for collecting agent usage data and user feedback. Through APIs and log file analysis, it obtains data such as agent usage frequency, user satisfaction, and bug rates. The collected data is analyzed using an AI model, and the performance of each agent is quantified. This quantified data forms the basis for generating digital identification information.

[0318] The terminal displays information from the agent marketplace as a device accessed by the user. Users can access the marketplace via a device such as a smartphone and view the agent list and individual ratings. Blockchain technology is used to securely manage ownership of digital identification information and transaction history. In this process, ownership transfers and revenue distribution are automatically carried out by smart contracts.

[0319] The system is designed using mobile app development frameworks such as Flutter and React Native, and can utilize Ethereum's Web3.js as its blockchain API. This ensures smooth operation and security for the entire system.

[0320] As a concrete example, a user uses a terminal to search for an agent they are interested in and check its performance evaluation. When purchasing an agent they like, the transaction is conducted via cryptocurrency, and after the purchase, ownership is managed on the blockchain. An example of a prompt using a generated AI model is, "Retrieve data to list the agent's transaction history and optimize the process of verifying ownership on the blockchain." This prompt allows the system to process the request efficiently and accurately.

[0321] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0322] Step 1:

[0323] The server collects data on agent usage and user feedback. Inputs are agent usage logs and user reviews, and output is a database of this information. Data processing involves retrieving information via API and organizing it into the database. Specifically, it periodically collects usage frequency and user ratings for each agent.

[0324] Step 2:

[0325] The server analyzes the collected data and quantifies the performance of each agent. The input is the data collected in step 1, and the output is the quantified performance evaluation. For data calculation, an AI model is used to generate a score that takes into account usage frequency, user satisfaction, and bug occurrence rate. Specifically, a machine learning algorithm is applied to automatically calculate the performance evaluation.

[0326] Step 3:

[0327] The server issues digital identification information to agents whose performance has been quantified, and records ownership of that information on the blockchain. The input is the performance evaluation data obtained in step 2, and the output is the registration of the digital identification information on the blockchain. Specifically, a smart contract is used to generate the identification information and publish it on the blockchain.

[0328] Step 4:

[0329] The terminal allows the user to access the marketplace and view a list of agent performance ratings. The input is the digital identification information obtained in step 3, and the output is the agent list displayed to the user. Specifically, the evaluation information is visually organized and provided in a format that is easy for the user to compare.

[0330] Step 5:

[0331] The user selects the digital identification information of an agent they are interested in and proceeds with the purchase. The input is the displayed agent information, and the output is a confirmation of the purchased agent. Specifically, the user completes the purchase by processing payment in cryptocurrency through their wallet.

[0332] Step 6:

[0333] The server updates the record after a transaction is completed and distributes the revenue to the agent provider. The input is transaction data, and the output is updated revenue-related data. Specifically, revenue is automatically sent via a smart contract based on defined distribution rules.

[0334] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0335] Embodiments of this invention are systems aimed at reflecting user emotions in the performance evaluation of agents, and primarily operate with a configuration including a server, terminal, user, and emotion engine.

[0336] Data collection and analysis

[0337] The server provides an interface for collecting user emotion data, in addition to agent usage data and user feedback. Emotion data is acquired via the camera and microphone installed on the device and obtained by analyzing the user's voice tone and facial expressions.

[0338] The server uses an emotion engine to analyze the user's emotions and measure the emotional changes in real time. For example, if the voice changes from a calm tone to an excited tone while the user is using the agent, this could be attributed to the agent's effectiveness or difficulty level. This information plays a crucial role in evaluating the agent's performance.

[0339] Quantification of performance evaluation

[0340] The server uses AI to analyze all collected data, including emotional data, and quantifies the agent's performance. This performance evaluation includes changes in emotional responses and the tone of user feedback.

[0341] Issuance of digital identification information

[0342] The server generates digital identification information (NFTs) for each agent based on quantified evaluations and issues them on the blockchain. This identification information records detailed agent performance data, including sentiment ratings.

[0343] Marketplace transactions

[0344] Users can access the marketplace to check the performance and emotional responses of agents. This allows them to select the most suitable agent, taking emotional data into consideration.

[0345] Users purchase and trade the digital identification information of specific agents. Transactions are conducted securely using cryptocurrency and via smart contracts.

[0346] Profit sharing

[0347] The server records transaction history and distributes revenue to developers based on the value of agents that utilize emotional data. This allows developers to more accurately understand the quality of products by considering emotional data, thereby promoting the development of agents that contribute to improving the user experience.

[0348] The introduction of this system will enable agent evaluation and transaction processes that reflect user sentiment, improving convenience and reliability for both users and developers.

[0349] The following describes the processing flow.

[0350] Step 1:

[0351] As soon as the agent starts operating, the device uses its camera and microphone to collect the user's facial expressions and voice in real time. This data is sent to the emotion engine as a feed to evaluate the user's emotions.

[0352] Step 2:

[0353] The server analyzes user emotion data sent from the terminal using an emotion engine. This analysis allows the server to capture the user's emotional state, such as excitement, stress, and satisfaction, as numerical values.

[0354] Step 3:

[0355] The server integrates emotional data analyzed by the emotion engine, agent usage frequency data, and user feedback data, and calculates an overall performance evaluation score using an AI model. This score accurately reflects the agent's effectiveness and the user experience.

[0356] Step 4:

[0357] The server generates digital identification information for each agent based on the calculated performance evaluation score. This digital identification information is recorded on the blockchain and issued as an NFT representing the agent's digital ownership.

[0358] Step 5:

[0359] The server lists digital identification information, including performance evaluation scores and sentiment data analysis results, on the marketplace platform, making it accessible to users.

[0360] Step 6:

[0361] Users browse a list of agents from the marketplace and select the best agent based on sentiment data. When purchasing an agent's NFT, payment is made via cryptocurrency using a wallet app.

[0362] Step 7:

[0363] The server automatically updates the record on the blockchain once a transaction is completed and distributes the revenue based on the transaction to the developers. This revenue includes value based on sentiment data and is fairly returned to the developers.

[0364] (Example 2)

[0365] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0366] Existing agent evaluation systems have a problem in that they do not adequately evaluate agent performance while considering user emotions, resulting in the user experience not being fully utilized. Furthermore, there is a need for suggestions for agent improvements based on evaluation results, as well as ensuring transparency and reliability in the evaluation process.

[0367] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0368] In this invention, the server includes means for acquiring user voice and video via a terminal and analyzing emotions; means for integrating agent usage, user feedback, and emotion analysis data to quantify the performance of each agent; and means for generating digital identification information for the agents whose performance has been quantified and managing that identification information using distributed ledger technology. This enables highly accurate evaluation of agents that reflect user emotions and transparent system operation with suggestions for improvement.

[0369] A "terminal" is a device that acquires the user's voice and video and transmits that data to a server.

[0370] "Audio and video" refers to acoustic and visual data that indicates the user's state, and these are sources of information used to analyze emotions.

[0371] "Methods for analyzing emotions" refer to processes and technologies that estimate a user's emotions in real time based on acquired audio and video data.

[0372] An "agent" is software or hardware that performs a specific function and interacts with the user.

[0373] "Usage" refers to information about how the agent is being used by users.

[0374] "User feedback" refers to information that shows the opinions and impressions that users provide regarding their use of the agent.

[0375] "Emotional analysis data" refers to data obtained by analyzing the emotional state of users, and it forms the basis for evaluation.

[0376] "Methods for quantifying performance" refer to the process of expressing the agent's performance numerically based on collected data.

[0377] "Digital identification information" refers to unique digital data generated based on the performance evaluation of an agent, and is used for ownership and transactions.

[0378] "Distributed ledger technology" is a technology for managing and recording digital identification information in a tamper-proof manner, and blockchain technology is generally used for this purpose.

[0379] One embodiment of this invention provides a system for performing real-time performance evaluation of an agent while taking user emotions into consideration. This system mainly consists of a server, a terminal, a user, and an emotion analysis engine.

[0380] First, the device uses its camera and microphone to capture the user's voice and video. This allows it to capture the user's instantaneous emotional state. The voice data is used to analyze features such as voice tone and volume, and the video data is used for facial expression recognition. This data is transmitted to the server in real time.

[0381] The server inputs audio and video data received from the terminal into an emotion analysis engine to analyze the user's emotions. Specifically, it uses an audio analysis model to classify voice tone and an expression recognition model to analyze the user's facial movements. This allows the user's emotions to be estimated in real time with labels such as "joy," "sadness," and "surprise."

[0382] Subsequently, the server integrates agent usage data, user feedback data, and analyzed sentiment data, and uses AI to quantify the agent's performance. This quantified data is then used in conjunction with a generative AI model. An example of a prompt used in this process is, "What improvements can be made to the agent based on this sentiment pattern?"

[0383] Next, the server generates digital identification information based on quantified performance evaluations and manages this information in a tamper-proof manner using distributed ledger technology, such as blockchain. Blockchain management enhances the transparency and reliability of agent performance evaluations.

[0384] Finally, users can view agent performance data on the marketplace and trade the digital identification information of agents they are interested in. These transactions are conducted using cryptocurrency and are secured by smart contracts.

[0385] In this way, by utilizing user sentiment data to evaluate and improve agent performance, it is possible to provide a better user experience.

[0386] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0387] Step 1:

[0388] The device acquires the user's voice and video using a camera and microphone. The input consists of the user's real-time voice and facial images, which are transmitted to the server as digital data. Specifically, the voice recognition software installed on the device captures the user's voice and digitizes its waveform. In addition, the video capture function captures the user's facial movements frame by frame, converts them into data, and transmits it.

[0389] Step 2:

[0390] The server receives the input and passes it to the emotion analysis engine. The input here consists of digitized audio and video. The server inputs the audio data into the audio analysis model and the video data into the facial recognition model. As part of the data processing, it analyzes the pitch and speed of the audio and performs calculations to extract facial feature points from the video. The output is the user's emotion label (e.g., "joy," "anger," "anxiety," etc.).

[0391] Step 3:

[0392] The server then integrates the sentiment analysis results with agent usage data and user feedback data. Inputs include the aforementioned sentiment labels, agent activity logs, and user text feedback. For data processing, this data is subjected to AI analysis, and a generative AI model is used to quantify the agent's performance. The output is the agent's evaluation score.

[0393] Step 4:

[0394] The server uses this evaluation score to generate digital identification information. This identification information includes a unique agent ID and evaluation score. The output is a digital identification information (NFT), which is recorded in a distributed ledger system. Specifically, blockchain technology is used to execute a process that stores this data in a tamper-proof manner.

[0395] Step 5:

[0396] Users verify the agent's digital identification information on the marketplace and conduct transactions. The input is the digital identification information of the agent selected by the user. Specifically, the user operates the interface to access the marketplace and completes the purchase procedure using cryptocurrency. The output is the transaction history stored in the user's wallet.

[0397] (Application Example 2)

[0398] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0399] The objective of this invention is to provide a system that reflects user emotions in real time, enables more precise evaluation of agent performance, and optimizes the operation of a robotic device that acts in accordance with user emotions.

[0400] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0401] In this invention, the server includes means for collecting agent usage data and user feedback data; means for analyzing the collected data and emotion data estimated from the user's voice and facial expressions to quantify the performance of each agent; means for issuing digital identification information to the agents whose performance has been quantified and recording their ownership; and means for operating a robot device that changes its behavioral patterns based on the user's emotion data. This enables highly accurate agent performance evaluation that reflects the user's emotions and adaptive behavioral control of the robot device in accordance with the user's emotions.

[0402] An "agent" is a program or device that interacts with users and provides information and support.

[0403] "Usage data" refers to data about how the agent is being used.

[0404] "User feedback" refers to the evaluations and opinions that users provide after using the agent.

[0405] "Analysis" is the act of examining collected data to extract useful information.

[0406] "Emotional data" refers to data on the user's psychological state, estimated based on their voice and facial expressions.

[0407] "Quantifying" means expressing data or information as numerical values.

[0408] "Digital identification information" refers to digital data that records the evaluation and characteristics of an agent.

[0409] "Ownership" refers to a property right relating to a specific object or information.

[0410] "To record" is the act of saving information in a medium.

[0411] A "market" is a place and system where buying and selling take place.

[0412] A "robot device" is a mechanical device that can take actions in response to the user's emotions.

[0413] The server collects data on agent usage and user feedback from the user's device. The device is equipped with a camera and microphone, which are used to acquire voice and facial expression data. The server analyzes the user's facial expressions from the camera footage using image processing with the OpenCV library and estimates emotion data using EmotionRecognizer. It also analyzes voice data using the SpeechRecognition library and estimates emotion data based on voice tone.

[0414] User emotion data is used to operate robotic devices that dynamically change their behavior. This information is used with the MusicPlayer module to select appropriate music and provide an environment that is sensitive to the user's psychological state. For example, if a user shows signs of fatigue after a long day, the server detects this and instructs the robotic device to play relaxation music.

[0415] An example of a prompt message is, "If the system detects that the user is tired, it will switch to relaxation mode. Please play calming music to soothe the user." This allows the user to enjoy a system that provides a comfortable experience tailored to their emotions.

[0416] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0417] Step 1:

[0418] The device uses a camera and microphone to collect user facial expressions and audio data. In this process, the camera acquires image frames in real time, and the microphone captures the user's speech as audio data. The input in this process is the user's video and audio, which are then sent to the server.

[0419] Step 2:

[0420] The server analyzes the received video data using the OpenCV library and evaluates the user's emotions through EmotionRecognizer. Specifically, it extracts facial features through image processing and performs data calculations to estimate emotions based on these features. The output is the user's estimated emotion data.

[0421] Step 3:

[0422] The server analyzes the audio data using the SpeechRecognition library to evaluate the speech tone. The audio is converted to text, and speech tone features are extracted from the text and quantified as emotion data. The output is emotion estimation data based on speech tone.

[0423] Step 4:

[0424] The server integrates the emotional data obtained in steps 2 and 3 and uses a generative AI model to perform a comprehensive emotional assessment. This allows for a detailed understanding of the user's current psychological state. The output at this stage is the integrated emotional assessment data.

[0425] Step 5:

[0426] The server determines the specific actions the robot device should take based on emotion evaluation data. If music playback or adjustments to environmental settings are required, the MusicPlayer module is used to select appropriate music and settings. A concrete example of this action might include playing calming music to soothe the user.

[0427] Step 6:

[0428] The robotic device receives instructions from the server and performs actions to adjust the user's environment. This includes playing specified music and adjusting lighting. As a final output, the user experiences a comfortable environment tailored to their emotions.

[0429] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0430] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0431] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0432] [Third Embodiment]

[0433] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0434] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0435] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0436] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0437] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0438] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0439] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0440] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0441] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0442] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0443] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0444] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0445] Embodiments of this invention relate to a system for evaluating agent performance and managing digital ownership. This system mainly consists of a server, terminals, and users, and operates in the following manner.

[0446] Data collection and analysis

[0447] The server prepares an interface for collecting agent usage data and user feedback, and retrieves this data periodically. Specifically, it collects data on which agents were used, to what extent, and how users evaluated those agents, through APIs and log file analysis.

[0448] The server uses an AI model based on the collected data to quantify the performance of each agent. This quantification process takes into account various metrics such as usage frequency, user satisfaction, and bug rate.

[0449] Issuance of digital identification information

[0450] The server generates digital identification information for each agent based on quantified performance evaluations. This identification information is issued on the blockchain and represents ownership of the agent.

[0451] The server uses smart contracts to securely and transparently manage ownership and transaction history of this digital identification information on the blockchain.

[0452] Marketplace transactions

[0453] Users can access a marketplace provided via the web or application to view a list of agents, their ratings, and their corresponding digital identification information.

[0454] Users select the digital identification information of agents they are interested in and proceed with the purchase on the marketplace. This process is typically carried out using cryptocurrency, with payment made through a wallet application.

[0455] Profit sharing

[0456] The server updates the record immediately upon completion of a transaction and distributes the revenue based on the transaction to the agent's creator or developer. This ensures that developers receive fair compensation for the revenue generated from the agents they create.

[0457] In this way, agent usage, evaluation, and transactions can be centrally managed, providing a fair and transparent environment for developers and users. This can improve the reliability of agent selection and transactions, while also fostering new incentives for developers.

[0458] The following describes the processing flow.

[0459] Step 1:

[0460] The server activates a logging system to monitor agent usage data. This allows for real-time recording of each agent's usage frequency and execution time.

[0461] Step 2:

[0462] The server automatically displays a feedback form when a user uses the agent. The feedback is collected as evaluation scores and comments and stored in a database.

[0463] Step 3:

[0464] The server runs an AI engine to analyze the collected usage data and feedback. The AI ​​evaluates this data and calculates a numerical performance score for each agent.

[0465] Step 4:

[0466] The server generates a unique digital identification (NFT) for each agent based on its performance score and issues it on the blockchain. This identification includes the agent's ownership and evaluation information.

[0467] Step 5:

[0468] The server publishes the generated digital identification information through the marketplace platform. This allows users to view detailed information about each agent and the NFTs available for trading.

[0469] Step 6:

[0470] Users select an agent they are interested in from those available for trading on the marketplace and begin the purchase process. Secure transactions are conducted via smart contracts using cryptocurrency.

[0471] Step 7:

[0472] The server records the transaction history via the blockchain when a transaction is completed and immediately distributes the transaction revenue to the developer. This allows the developer to receive revenue based on their agent.

[0473] (Example 1)

[0474] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0475] In modern society, the proliferation of information devices and software agents has made performance evaluation and promoting their appropriate use crucial issues. However, there are current challenges in establishing appropriate performance evaluation methods, ensuring fair and transparent ownership management, and guaranteeing the reliability of transactions based on these evaluations. There is a need for a system that can solve these problems, enabling improved performance and increased use of information devices, as well as fair compensation for developers.

[0476] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0477] In this invention, the server includes means for collecting data on the operating status of information devices and user evaluations; means for analyzing the collected data using an AI algorithm and quantifying the performance of each information device; and means for generating digital identification information for the information devices whose performance has been quantified and recording their ownership on a distributed ledger. This provides a system that enables appropriate performance evaluation of information devices, fair and transparent management of their ownership, and reliable transactions based on this.

[0478] "Information equipment" refers to all devices that process and communicate digital data, and includes computers and terminals with communication functions.

[0479] "User" refers to an individual or organization that utilizes the functions provided by information devices or software agents.

[0480] "Means of data collection" refers to the function of providing a mechanism for collecting and storing necessary information from information devices and users.

[0481] An "AI algorithm" refers to a calculation procedure that uses artificial intelligence technology to analyze data and determine the performance of equipment.

[0482] "Digital identification information" refers to unique digital data associated with information devices or agents, which can be used to identify ownership and attributes.

[0483] A "distributed ledger" refers to a system that records transaction information and other data synchronously across multiple locations in a way that makes tampering difficult, and includes technologies such as blockchain.

[0484] An "e-commerce environment" refers to a marketplace where goods and services can be bought and sold via the internet or other means.

[0485] "Transaction records" refer to data that records the details of transactions conducted in an e-commerce environment, and typically include the date and time of the sale, the goods, and the price.

[0486] The embodiments for carrying out the invention are described below.

[0487] This invention is a system that enables performance evaluation of information devices and management of their digital identification information. This system mainly consists of three components: a server, a terminal, and a user.

[0488] The server acquires usage data and user ratings from information devices using APIs and log analysis techniques. Specifically, the hardware consists of a computer server for managing the database, while the software includes APIs for data collection and log analysis tools. The server then analyzes this data using AI algorithms (e.g., deep learning models using TensorFlow) to quantify the performance metrics of each information device. This allows for the output of specific figures, such as "Device A was used 20 times a day and achieved an 80% satisfaction rate."

[0489] After performance evaluation is complete, the server generates digital identification information for each information device and records this identification information and transaction history on a distributed ledger (e.g., blockchain technology). This operation also utilizes smart contract technology to ensure the transparent and secure management of identification information.

[0490] The terminal (a computer or smartphone operated by the user) connects to an e-commerce environment provided through a web browser or dedicated application and is used to view a list of information devices, their ratings, and digital identification information. For example, when a user accesses the market using their terminal, they can view ratings of information devices of interest and decide whether to purchase them.

[0491] Users complete the purchase process in the e-commerce environment through their devices and make payments using cryptocurrency. Payments are made via a wallet application, and based on the confirmed transaction from the server, legitimate revenue is automatically distributed to the developers.

[0492] As a concrete example, consider the evaluation process for a voice assistant application. When a user uses assistant application A, the server automatically collects usage frequency and evaluation data, which is then analyzed by an AI model to calculate a performance score. Based on this data, digital identification information for application A is generated, and the user confirms this information in the marketplace before making a purchase. Ultimately, the developer receives revenue based on sales.

[0493] Examples of input prompts for the generating AI model include: "Evaluate the agent's performance and generate new incentive suggestions that take user feedback into consideration."

[0494] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0495] Step 1:

[0496] The server collects usage data and user evaluation data through the APIs and log files of information devices. Inputs include the information device's identification ID, usage count, and user satisfaction rating. This data is stored in a database and output as organized information based on a data schema for use in subsequent analysis.

[0497] Step 2:

[0498] The server loads the collected data as input to an AI algorithm. Specifically, it uses a deep learning framework such as TensorFlow to run a data analysis model. The AI ​​algorithm evaluates metrics such as usage frequency, satisfaction level, and bug rate, and quantifies the performance of each information device. As a result, it outputs a specific performance score for each information device.

[0499] Step 3:

[0500] The server generates unique digital identification information for each information device based on its performance score. This identification information is registered in a distributed ledger system using a smart contract. The input includes the generated performance score and the information device's identification ID. The output is a secure and transparent recording of digital identification information that can be verified on the blockchain.

[0501] Step 4:

[0502] The terminal provides an information commerce environment in response to user access requests. The terminal displays an interface for users to browse a list of information devices and verify their evaluations and digital identification information. Inputs include the user's search criteria and the IDs of information devices of interest. Outputs include performance evaluations and purchase options for the information devices displayed on the interface.

[0503] Step 5:

[0504] The user uses a terminal to select the digital identification information of the information device they are interested in and begins the purchase process. The user makes the payment using a wallet application. Inputs include the user's wallet information and the purchase amount. Output is a confirmation message indicating that the payment has been completed. Transaction information is also recorded on a distributed ledger.

[0505] Step 6:

[0506] The server updates the transaction record as soon as the transaction is completed and distributes the revenue to the information device developers. Inputs include transaction history and developer information. Outputs include the allocation of legitimate revenue to the developers' wallets through an automated transfer process via smart contracts.

[0507] (Application Example 1)

[0508] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0509] There is a need to accurately evaluate agent usage and performance, and to manage digital ownership securely and transparently. It is also crucial to provide users with an environment where they can easily verify agent performance and conduct secure transactions. Traditional methods struggle to meet these requirements, resulting in inefficiencies and a lack of reliability.

[0510] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0511] In this invention, the server includes a device for collecting information on agent usage and user feedback, a device for analyzing the collected information and quantifying the performance of each agent, and a device for issuing digital identification information to the agents whose performance has been quantified and recording their ownership. This enables efficient and reliable evaluation of agent performance and management of digital ownership.

[0512] An "agent" is a piece of software or program that automatically performs a specific task or function.

[0513] "User feedback" refers to the impressions and opinions provided by users after using an agent, and is an important indicator in performance evaluation.

[0514] "Digital identification information" refers to electronic identification information issued based on the performance and usage of an agent, and is used to manage ownership and transaction history on the blockchain.

[0515] Blockchain technology is a technology that functions as a distributed ledger, enabling data transparency and preventing tampering.

[0516] The "marketplace" is a platform for trading agents' digital identification information, where users can buy and rate agents.

[0517] A "transaction history" is a record of an agent's buying, selling, and exchange activities in the market, and is recorded to ensure transparency and reliability.

[0518] A "profit-distributing device" is a device equipped with the function of appropriately distributing the profits obtained after a market transaction is completed to the agent providers.

[0519] This system works in conjunction with servers, terminals, and users to securely and efficiently manage agent performance evaluation and digital ownership. The system is structured as follows, with each element playing its own role.

[0520] The server provides an interface for collecting agent usage data and user feedback. Through APIs and log file analysis, it obtains data such as agent usage frequency, user satisfaction, and bug rates. The collected data is analyzed using an AI model, and the performance of each agent is quantified. This quantified data forms the basis for generating digital identification information.

[0521] The terminal displays information from the agent marketplace as a device accessed by the user. Users can access the marketplace via a device such as a smartphone and view the agent list and individual ratings. Blockchain technology is used to securely manage ownership of digital identification information and transaction history. In this process, ownership transfers and revenue distribution are automatically carried out by smart contracts.

[0522] The system is designed using mobile app development frameworks such as Flutter and React Native, and can utilize Ethereum's Web3.js as its blockchain API. This ensures smooth operation and security for the entire system.

[0523] As a concrete example, a user uses a terminal to search for an agent they are interested in and check its performance evaluation. When purchasing an agent they like, the transaction is conducted via cryptocurrency, and after the purchase, ownership is managed on the blockchain. An example of a prompt using a generated AI model is, "Retrieve data to list the agent's transaction history and optimize the process of verifying ownership on the blockchain." This prompt allows the system to process the request efficiently and accurately.

[0524] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0525] Step 1:

[0526] The server collects data on agent usage and user feedback. Inputs are agent usage logs and user reviews, and output is a database of this information. Data processing involves retrieving information via API and organizing it into the database. Specifically, it periodically collects usage frequency and user ratings for each agent.

[0527] Step 2:

[0528] The server analyzes the collected data and quantifies the performance of each agent. The input is the data collected in step 1, and the output is the quantified performance evaluation. For data calculation, an AI model is used to generate a score that takes into account usage frequency, user satisfaction, and bug occurrence rate. Specifically, a machine learning algorithm is applied to automatically calculate the performance evaluation.

[0529] Step 3:

[0530] The server issues digital identification information to agents whose performance has been quantified, and records ownership of that information on the blockchain. The input is the performance evaluation data obtained in step 2, and the output is the registration of the digital identification information on the blockchain. Specifically, a smart contract is used to generate the identification information and publish it on the blockchain.

[0531] Step 4:

[0532] The terminal allows the user to access the marketplace and view a list of agent performance ratings. The input is the digital identification information obtained in step 3, and the output is the agent list displayed to the user. Specifically, the evaluation information is visually organized and provided in a format that is easy for the user to compare.

[0533] Step 5:

[0534] The user selects the digital identification information of an agent they are interested in and proceeds with the purchase. The input is the displayed agent information, and the output is a confirmation of the purchased agent. Specifically, the user completes the purchase by processing payment in cryptocurrency through their wallet.

[0535] Step 6:

[0536] The server updates the record after a transaction is completed and distributes the revenue to the agent provider. The input is transaction data, and the output is updated revenue-related data. Specifically, revenue is automatically sent via a smart contract based on defined distribution rules.

[0537] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0538] Embodiments of this invention are systems aimed at reflecting user emotions in the performance evaluation of agents, and primarily operate with a configuration including a server, terminal, user, and emotion engine.

[0539] Data collection and analysis

[0540] The server provides an interface for collecting user emotion data, in addition to agent usage data and user feedback. Emotion data is acquired via the camera and microphone installed on the device and obtained by analyzing the user's voice tone and facial expressions.

[0541] The server uses an emotion engine to analyze the user's emotions and measure the emotional changes in real time. For example, if the voice changes from a calm tone to an excited tone while the user is using the agent, this could be attributed to the agent's effectiveness or difficulty level. This information plays a crucial role in evaluating the agent's performance.

[0542] Quantification of performance evaluation

[0543] The server uses AI to analyze all collected data, including emotional data, and quantifies the agent's performance. This performance evaluation includes changes in emotional responses and the tone of user feedback.

[0544] Issuance of digital identification information

[0545] The server generates digital identification information (NFTs) for each agent based on quantified evaluations and issues them on the blockchain. This identification information records detailed agent performance data, including sentiment ratings.

[0546] Marketplace transactions

[0547] Users can access the marketplace to check the performance and emotional responses of agents. This allows them to select the most suitable agent, taking emotional data into consideration.

[0548] Users purchase and trade the digital identification information of specific agents. Transactions are conducted securely using cryptocurrency and via smart contracts.

[0549] Profit sharing

[0550] The server records transaction history and distributes revenue to developers based on the value of agents that utilize emotional data. This allows developers to more accurately understand the quality of products by considering emotional data, thereby promoting the development of agents that contribute to improving the user experience.

[0551] The introduction of this system will enable agent evaluation and transaction processes that reflect user sentiment, improving convenience and reliability for both users and developers.

[0552] The following describes the processing flow.

[0553] Step 1:

[0554] As soon as the agent starts operating, the device uses its camera and microphone to collect the user's facial expressions and voice in real time. This data is sent to the emotion engine as a feed to evaluate the user's emotions.

[0555] Step 2:

[0556] The server analyzes user emotion data sent from the terminal using an emotion engine. This analysis allows the server to capture the user's emotional state, such as excitement, stress, and satisfaction, as numerical values.

[0557] Step 3:

[0558] The server integrates emotional data analyzed by the emotion engine, agent usage frequency data, and user feedback data, and calculates an overall performance evaluation score using an AI model. This score accurately reflects the agent's effectiveness and the user experience.

[0559] Step 4:

[0560] The server generates digital identification information for each agent based on the calculated performance evaluation score. This digital identification information is recorded on the blockchain and issued as an NFT representing the agent's digital ownership.

[0561] Step 5:

[0562] The server lists digital identification information, including performance evaluation scores and sentiment data analysis results, on the marketplace platform, making it accessible to users.

[0563] Step 6:

[0564] Users browse a list of agents from the marketplace and select the best agent based on sentiment data. When purchasing an agent's NFT, payment is made via cryptocurrency using a wallet app.

[0565] Step 7:

[0566] The server automatically updates the record on the blockchain once a transaction is completed and distributes the revenue based on the transaction to the developers. This revenue includes value based on sentiment data and is fairly returned to the developers.

[0567] (Example 2)

[0568] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0569] Existing agent evaluation systems have a problem in that they do not adequately evaluate agent performance while considering user emotions, resulting in the user experience not being fully utilized. Furthermore, there is a need for suggestions for agent improvements based on evaluation results, as well as ensuring transparency and reliability in the evaluation process.

[0570] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0571] In this invention, the server includes means for acquiring user voice and video via a terminal and analyzing emotions; means for integrating agent usage, user feedback, and emotion analysis data to quantify the performance of each agent; and means for generating digital identification information for the agents whose performance has been quantified and managing that identification information using distributed ledger technology. This enables highly accurate evaluation of agents that reflect user emotions and transparent system operation with suggestions for improvement.

[0572] A "terminal" is a device that acquires the user's voice and video and transmits that data to a server.

[0573] "Audio and video" refers to acoustic and visual data that indicates the user's state, and these are sources of information used to analyze emotions.

[0574] "Methods for analyzing emotions" refer to processes and technologies that estimate a user's emotions in real time based on acquired audio and video data.

[0575] An "agent" is software or hardware that performs a specific function and interacts with the user.

[0576] "Usage" refers to information about how the agent is being used by users.

[0577] "User feedback" refers to information that shows the opinions and impressions that users provide regarding their use of the agent.

[0578] "Emotional analysis data" refers to data obtained by analyzing the emotional state of users, and it forms the basis for evaluation.

[0579] "Methods for quantifying performance" refer to the process of expressing the agent's performance numerically based on collected data.

[0580] "Digital identification information" refers to unique digital data generated based on the performance evaluation of an agent, and is used for ownership and transactions.

[0581] "Distributed ledger technology" is a technology for managing and recording digital identification information in a tamper-proof manner, and blockchain technology is generally used for this purpose.

[0582] One embodiment of this invention provides a system for performing real-time performance evaluation of an agent while taking user emotions into consideration. This system mainly consists of a server, a terminal, a user, and an emotion analysis engine.

[0583] First, the device uses its camera and microphone to capture the user's voice and video. This allows it to capture the user's instantaneous emotional state. The voice data is used to analyze features such as voice tone and volume, and the video data is used for facial expression recognition. This data is transmitted to the server in real time.

[0584] The server inputs audio and video data received from the terminal into an emotion analysis engine to analyze the user's emotions. Specifically, it uses an audio analysis model to classify voice tone and an expression recognition model to analyze the user's facial movements. This allows the user's emotions to be estimated in real time with labels such as "joy," "sadness," and "surprise."

[0585] Subsequently, the server integrates agent usage data, user feedback data, and analyzed sentiment data, and uses AI to quantify the agent's performance. This quantified data is then used in conjunction with a generative AI model. An example of a prompt used in this process is, "What improvements can be made to the agent based on this sentiment pattern?"

[0586] Next, the server generates digital identification information based on quantified performance evaluations and manages this information in a tamper-proof manner using distributed ledger technology, such as blockchain. Blockchain management enhances the transparency and reliability of agent performance evaluations.

[0587] Finally, users can view agent performance data on the marketplace and trade the digital identification information of agents they are interested in. These transactions are conducted using cryptocurrency and are secured by smart contracts.

[0588] In this way, by utilizing user sentiment data to evaluate and improve agent performance, it is possible to provide a better user experience.

[0589] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0590] Step 1:

[0591] The device acquires the user's voice and video using a camera and microphone. The input consists of the user's real-time voice and facial images, which are transmitted to the server as digital data. Specifically, the voice recognition software installed on the device captures the user's voice and digitizes its waveform. In addition, the video capture function captures the user's facial movements frame by frame, converts them into data, and transmits it.

[0592] Step 2:

[0593] The server receives the input and passes it to the emotion analysis engine. The input here consists of digitized audio and video. The server inputs the audio data into the audio analysis model and the video data into the facial recognition model. As part of the data processing, it analyzes the pitch and speed of the audio and performs calculations to extract facial feature points from the video. The output is the user's emotion label (e.g., "joy," "anger," "anxiety," etc.).

[0594] Step 3:

[0595] The server then integrates the sentiment analysis results with agent usage data and user feedback data. Inputs include the aforementioned sentiment labels, agent activity logs, and user text feedback. For data processing, this data is subjected to AI analysis, and a generative AI model is used to quantify the agent's performance. The output is the agent's evaluation score.

[0596] Step 4:

[0597] The server uses this evaluation score to generate digital identification information. This identification information includes a unique agent ID and evaluation score. The output is a digital identification information (NFT), which is recorded in a distributed ledger system. Specifically, blockchain technology is used to execute a process that stores this data in a tamper-proof manner.

[0598] Step 5:

[0599] Users verify the agent's digital identification information on the marketplace and conduct transactions. The input is the digital identification information of the agent selected by the user. Specifically, the user operates the interface to access the marketplace and completes the purchase procedure using cryptocurrency. The output is the transaction history stored in the user's wallet.

[0600] (Application Example 2)

[0601] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0602] The objective of this invention is to provide a system that reflects user emotions in real time, enables more precise evaluation of agent performance, and optimizes the operation of a robotic device that acts in accordance with user emotions.

[0603] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0604] In this invention, the server includes means for collecting agent usage data and user feedback data; means for analyzing the collected data and emotion data estimated from the user's voice and facial expressions to quantify the performance of each agent; means for issuing digital identification information to the agents whose performance has been quantified and recording their ownership; and means for operating a robot device that changes its behavioral patterns based on the user's emotion data. This enables highly accurate agent performance evaluation that reflects the user's emotions and adaptive behavioral control of the robot device in accordance with the user's emotions.

[0605] An "agent" is a program or device that interacts with users and provides information and support.

[0606] "Usage data" refers to data about how the agent is being used.

[0607] "User feedback" refers to the evaluations and opinions that users provide after using the agent.

[0608] "Analysis" is the act of examining collected data to extract useful information.

[0609] "Emotional data" refers to data on the user's psychological state, estimated based on their voice and facial expressions.

[0610] "Quantifying" means expressing data or information as numerical values.

[0611] "Digital identification information" refers to digital data that records the evaluation and characteristics of an agent.

[0612] "Ownership" refers to a property right relating to a specific object or information.

[0613] "To record" is the act of saving information in a medium.

[0614] A "market" is a place and system where buying and selling take place.

[0615] A "robot device" is a mechanical device that can take actions in response to the user's emotions.

[0616] The server collects data on agent usage and user feedback from the user's device. The device is equipped with a camera and microphone, which are used to acquire voice and facial expression data. The server analyzes the user's facial expressions from the camera footage using image processing with the OpenCV library and estimates emotion data using EmotionRecognizer. It also analyzes voice data using the SpeechRecognition library and estimates emotion data based on voice tone.

[0617] User emotion data is used to operate robotic devices that dynamically change their behavior. This information is used with the MusicPlayer module to select appropriate music and provide an environment that is sensitive to the user's psychological state. For example, if a user shows signs of fatigue after a long day, the server detects this and instructs the robotic device to play relaxation music.

[0618] An example of a prompt message is, "If the system detects that the user is tired, it will switch to relaxation mode. Please play calming music to soothe the user." This allows the user to enjoy a system that provides a comfortable experience tailored to their emotions.

[0619] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0620] Step 1:

[0621] The device uses a camera and microphone to collect user facial expressions and audio data. In this process, the camera acquires image frames in real time, and the microphone captures the user's speech as audio data. The input in this process is the user's video and audio, which are then sent to the server.

[0622] Step 2:

[0623] The server analyzes the received video data using the OpenCV library and evaluates the user's emotions through EmotionRecognizer. Specifically, it extracts facial features through image processing and performs data calculations to estimate emotions based on these features. The output is the user's estimated emotion data.

[0624] Step 3:

[0625] The server analyzes the audio data using the SpeechRecognition library to evaluate the speech tone. The audio is converted to text, and speech tone features are extracted from the text and quantified as emotion data. The output is emotion estimation data based on speech tone.

[0626] Step 4:

[0627] The server integrates the emotional data obtained in steps 2 and 3 and uses a generative AI model to perform a comprehensive emotional assessment. This allows for a detailed understanding of the user's current psychological state. The output at this stage is the integrated emotional assessment data.

[0628] Step 5:

[0629] The server determines the specific actions the robot device should take based on emotion evaluation data. If music playback or adjustments to environmental settings are required, the MusicPlayer module is used to select appropriate music and settings. A concrete example of this action might include playing calming music to soothe the user.

[0630] Step 6:

[0631] The robotic device receives instructions from the server and performs actions to adjust the user's environment. This includes playing specified music and adjusting lighting. As a final output, the user experiences a comfortable environment tailored to their emotions.

[0632] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0633] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0634] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0635] [Fourth Embodiment]

[0636] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0637] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0638] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0639] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0640] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0641] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0642] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0643] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0644] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0645] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0646] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0647] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0648] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0649] Embodiments of this invention relate to a system for evaluating agent performance and managing digital ownership. This system mainly consists of a server, terminals, and users, and operates in the following manner.

[0650] Data collection and analysis

[0651] The server prepares an interface for collecting agent usage data and user feedback, and retrieves this data periodically. Specifically, it collects data on which agents were used, to what extent, and how users evaluated those agents, through APIs and log file analysis.

[0652] The server uses an AI model based on the collected data to quantify the performance of each agent. This quantification process takes into account various metrics such as usage frequency, user satisfaction, and bug rate.

[0653] Issuance of digital identification information

[0654] The server generates digital identification information for each agent based on quantified performance evaluations. This identification information is issued on the blockchain and represents ownership of the agent.

[0655] The server uses smart contracts to securely and transparently manage ownership and transaction history of this digital identification information on the blockchain.

[0656] Marketplace transactions

[0657] Users can access a marketplace provided via the web or application to view a list of agents, their ratings, and their corresponding digital identification information.

[0658] Users select the digital identification information of agents they are interested in and proceed with the purchase on the marketplace. This process is typically carried out using cryptocurrency, with payment made through a wallet application.

[0659] Profit sharing

[0660] The server updates the record immediately upon completion of a transaction and distributes the revenue based on the transaction to the agent's creator or developer. This ensures that developers receive fair compensation for the revenue generated from the agents they create.

[0661] In this way, agent usage, evaluation, and transactions can be centrally managed, providing a fair and transparent environment for developers and users. This can improve the reliability of agent selection and transactions, while also fostering new incentives for developers.

[0662] The following describes the processing flow.

[0663] Step 1:

[0664] The server activates a logging system to monitor agent usage data. This allows for real-time recording of each agent's usage frequency and execution time.

[0665] Step 2:

[0666] The server automatically displays a feedback form when a user uses the agent. The feedback is collected as evaluation scores and comments and stored in a database.

[0667] Step 3:

[0668] The server runs an AI engine to analyze the collected usage data and feedback. The AI ​​evaluates this data and calculates a numerical performance score for each agent.

[0669] Step 4:

[0670] The server generates a unique digital identification (NFT) for each agent based on its performance score and issues it on the blockchain. This identification includes the agent's ownership and evaluation information.

[0671] Step 5:

[0672] The server publishes the generated digital identification information through the marketplace platform. This allows users to view detailed information about each agent and the NFTs available for trading.

[0673] Step 6:

[0674] Users select an agent they are interested in from those available for trading on the marketplace and begin the purchase process. Secure transactions are conducted via smart contracts using cryptocurrency.

[0675] Step 7:

[0676] The server records the transaction history via the blockchain when a transaction is completed and immediately distributes the transaction revenue to the developer. This allows the developer to receive revenue based on their agent.

[0677] (Example 1)

[0678] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0679] In modern society, the proliferation of information devices and software agents has made performance evaluation and promoting their appropriate use crucial issues. However, there are current challenges in establishing appropriate performance evaluation methods, ensuring fair and transparent ownership management, and guaranteeing the reliability of transactions based on these evaluations. There is a need for a system that can solve these problems, enabling improved performance and increased use of information devices, as well as fair compensation for developers.

[0680] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0681] In this invention, the server includes means for collecting data on the operating status of information devices and user evaluations; means for analyzing the collected data using an AI algorithm and quantifying the performance of each information device; and means for generating digital identification information for the information devices whose performance has been quantified and recording their ownership on a distributed ledger. This provides a system that enables appropriate performance evaluation of information devices, fair and transparent management of their ownership, and reliable transactions based on this.

[0682] "Information equipment" refers to all devices that process and communicate digital data, and includes computers and terminals with communication functions.

[0683] "User" refers to an individual or organization that utilizes the functions provided by information devices or software agents.

[0684] "Means of data collection" refers to the function of providing a mechanism for collecting and storing necessary information from information devices and users.

[0685] An "AI algorithm" refers to a calculation procedure that uses artificial intelligence technology to analyze data and determine the performance of equipment.

[0686] "Digital identification information" refers to unique digital data associated with information devices or agents, which can be used to identify ownership and attributes.

[0687] A "distributed ledger" refers to a system that records transaction information and other data synchronously across multiple locations in a way that makes tampering difficult, and includes technologies such as blockchain.

[0688] An "e-commerce environment" refers to a marketplace where goods and services can be bought and sold via the internet or other means.

[0689] "Transaction records" refer to data that records the details of transactions conducted in an e-commerce environment, and typically include the date and time of the sale, the goods, and the price.

[0690] The embodiments for carrying out the invention are described below.

[0691] This invention is a system that enables performance evaluation of information devices and management of their digital identification information. This system mainly consists of three components: a server, a terminal, and a user.

[0692] The server acquires usage data and user ratings from information devices using APIs and log analysis techniques. Specifically, the hardware consists of a computer server for managing the database, while the software includes APIs for data collection and log analysis tools. The server then analyzes this data using AI algorithms (e.g., deep learning models using TensorFlow) to quantify the performance metrics of each information device. This allows for the output of specific figures, such as "Device A was used 20 times a day and achieved an 80% satisfaction rate."

[0693] After performance evaluation is complete, the server generates digital identification information for each information device and records this identification information and transaction history on a distributed ledger (e.g., blockchain technology). This operation also utilizes smart contract technology to ensure the transparent and secure management of identification information.

[0694] The terminal (a computer or smartphone operated by the user) connects to an e-commerce environment provided through a web browser or dedicated application and is used to view a list of information devices, their ratings, and digital identification information. For example, when a user accesses the market using their terminal, they can view ratings of information devices of interest and decide whether to purchase them.

[0695] Users complete the purchase process in the e-commerce environment through their devices and make payments using cryptocurrency. Payments are made via a wallet application, and based on the confirmed transaction from the server, legitimate revenue is automatically distributed to the developers.

[0696] As a concrete example, consider the evaluation process for a voice assistant application. When a user uses assistant application A, the server automatically collects usage frequency and evaluation data, which is then analyzed by an AI model to calculate a performance score. Based on this data, digital identification information for application A is generated, and the user confirms this information in the marketplace before making a purchase. Ultimately, the developer receives revenue based on sales.

[0697] Examples of input prompts for the generating AI model include: "Evaluate the agent's performance and generate new incentive suggestions that take user feedback into consideration."

[0698] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0699] Step 1:

[0700] The server collects usage data and user evaluation data through the APIs and log files of information devices. Inputs include the information device's identification ID, usage count, and user satisfaction rating. This data is stored in a database and output as organized information based on a data schema for use in subsequent analysis.

[0701] Step 2:

[0702] The server loads the collected data as input to an AI algorithm. Specifically, it uses a deep learning framework such as TensorFlow to run a data analysis model. The AI ​​algorithm evaluates metrics such as usage frequency, satisfaction level, and bug rate, and quantifies the performance of each information device. As a result, it outputs a specific performance score for each information device.

[0703] Step 3:

[0704] The server generates unique digital identification information for each information device based on its performance score. This identification information is registered in a distributed ledger system using a smart contract. The input includes the generated performance score and the information device's identification ID. The output is a secure and transparent recording of digital identification information that can be verified on the blockchain.

[0705] Step 4:

[0706] The terminal provides an information commerce environment in response to user access requests. The terminal displays an interface for users to browse a list of information devices and verify their evaluations and digital identification information. Inputs include the user's search criteria and the IDs of information devices of interest. Outputs include performance evaluations and purchase options for the information devices displayed on the interface.

[0707] Step 5:

[0708] The user uses a terminal to select the digital identification information of the information device they are interested in and begins the purchase process. The user makes the payment using a wallet application. Inputs include the user's wallet information and the purchase amount. Output is a confirmation message indicating that the payment has been completed. Transaction information is also recorded on a distributed ledger.

[0709] Step 6:

[0710] The server updates the transaction record as soon as the transaction is completed and distributes the revenue to the information device developers. Inputs include transaction history and developer information. Outputs include the allocation of legitimate revenue to the developers' wallets through an automated transfer process via smart contracts.

[0711] (Application Example 1)

[0712] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0713] There is a need to accurately evaluate agent usage and performance, and to manage digital ownership securely and transparently. It is also crucial to provide users with an environment where they can easily verify agent performance and conduct secure transactions. Traditional methods struggle to meet these requirements, resulting in inefficiencies and a lack of reliability.

[0714] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0715] In this invention, the server includes a device for collecting information on agent usage and user feedback, a device for analyzing the collected information and quantifying the performance of each agent, and a device for issuing digital identification information to the agents whose performance has been quantified and recording their ownership. This enables efficient and reliable evaluation of agent performance and management of digital ownership.

[0716] An "agent" is a piece of software or program that automatically performs a specific task or function.

[0717] "User feedback" refers to the impressions and opinions provided by users after using an agent, and is an important indicator in performance evaluation.

[0718] "Digital identification information" refers to electronic identification information issued based on the performance and usage of an agent, and is used to manage ownership and transaction history on the blockchain.

[0719] Blockchain technology is a technology that functions as a distributed ledger, enabling data transparency and preventing tampering.

[0720] The "marketplace" is a platform for trading agents' digital identification information, where users can buy and rate agents.

[0721] A "transaction history" is a record of an agent's buying, selling, and exchange activities in the market, and is recorded to ensure transparency and reliability.

[0722] A "profit-distributing device" is a device equipped with the function of appropriately distributing the profits obtained after a market transaction is completed to the agent providers.

[0723] This system works in conjunction with servers, terminals, and users to securely and efficiently manage agent performance evaluation and digital ownership. The system is structured as follows, with each element playing its own role.

[0724] The server provides an interface for collecting agent usage data and user feedback. Through APIs and log file analysis, it obtains data such as agent usage frequency, user satisfaction, and bug rates. The collected data is analyzed using an AI model, and the performance of each agent is quantified. This quantified data forms the basis for generating digital identification information.

[0725] The terminal displays information from the agent marketplace as a device accessed by the user. Users can access the marketplace via a device such as a smartphone and view the agent list and individual ratings. Blockchain technology is used to securely manage ownership of digital identification information and transaction history. In this process, ownership transfers and revenue distribution are automatically carried out by smart contracts.

[0726] The system is designed using mobile app development frameworks such as Flutter and React Native, and can utilize Ethereum's Web3.js as its blockchain API. This ensures smooth operation and security for the entire system.

[0727] As a concrete example, a user uses a terminal to search for an agent they are interested in and check its performance evaluation. When purchasing an agent they like, the transaction is conducted via cryptocurrency, and after the purchase, ownership is managed on the blockchain. An example of a prompt using a generated AI model is, "Retrieve data to list the agent's transaction history and optimize the process of verifying ownership on the blockchain." This prompt allows the system to process the request efficiently and accurately.

[0728] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0729] Step 1:

[0730] The server collects data on agent usage and user feedback. Inputs are agent usage logs and user reviews, and output is a database of this information. Data processing involves retrieving information via API and organizing it into the database. Specifically, it periodically collects usage frequency and user ratings for each agent.

[0731] Step 2:

[0732] The server analyzes the collected data and quantifies the performance of each agent. The input is the data collected in step 1, and the output is the quantified performance evaluation. For data calculation, an AI model is used to generate a score that takes into account usage frequency, user satisfaction, and bug occurrence rate. Specifically, a machine learning algorithm is applied to automatically calculate the performance evaluation.

[0733] Step 3:

[0734] The server issues digital identification information to agents whose performance has been quantified, and records ownership of that information on the blockchain. The input is the performance evaluation data obtained in step 2, and the output is the registration of the digital identification information on the blockchain. Specifically, a smart contract is used to generate the identification information and publish it on the blockchain.

[0735] Step 4:

[0736] The terminal allows the user to access the marketplace and view a list of agent performance ratings. The input is the digital identification information obtained in step 3, and the output is the agent list displayed to the user. Specifically, the evaluation information is visually organized and provided in a format that is easy for the user to compare.

[0737] Step 5:

[0738] The user selects the digital identification information of an agent they are interested in and proceeds with the purchase. The input is the displayed agent information, and the output is a confirmation of the purchased agent. Specifically, the user completes the purchase by processing payment in cryptocurrency through their wallet.

[0739] Step 6:

[0740] The server updates the record after a transaction is completed and distributes the revenue to the agent provider. The input is transaction data, and the output is updated revenue-related data. Specifically, revenue is automatically sent via a smart contract based on defined distribution rules.

[0741] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0742] Embodiments of this invention are systems aimed at reflecting user emotions in the performance evaluation of agents, and primarily operate with a configuration including a server, terminal, user, and emotion engine.

[0743] Data collection and analysis

[0744] The server provides an interface for collecting user emotion data, in addition to agent usage data and user feedback. Emotion data is acquired via the camera and microphone installed on the device and obtained by analyzing the user's voice tone and facial expressions.

[0745] The server uses an emotion engine to analyze the user's emotions and measure the emotional changes in real time. For example, if the voice changes from a calm tone to an excited tone while the user is using the agent, this could be attributed to the agent's effectiveness or difficulty level. This information plays a crucial role in evaluating the agent's performance.

[0746] Quantification of performance evaluation

[0747] The server uses AI to analyze all collected data, including emotional data, and quantifies the agent's performance. This performance evaluation includes changes in emotional responses and the tone of user feedback.

[0748] Issuance of digital identification information

[0749] The server generates digital identification information (NFTs) for each agent based on quantified evaluations and issues them on the blockchain. This identification information records detailed agent performance data, including sentiment ratings.

[0750] Marketplace transactions

[0751] Users can access the marketplace to check the performance and emotional responses of agents. This allows them to select the most suitable agent, taking emotional data into consideration.

[0752] Users purchase and trade the digital identification information of specific agents. Transactions are conducted securely using cryptocurrency and via smart contracts.

[0753] Profit sharing

[0754] The server records transaction history and distributes revenue to developers based on the value of agents that utilize emotional data. This allows developers to more accurately understand the quality of products by considering emotional data, thereby promoting the development of agents that contribute to improving the user experience.

[0755] The introduction of this system will enable agent evaluation and transaction processes that reflect user sentiment, improving convenience and reliability for both users and developers.

[0756] The following describes the processing flow.

[0757] Step 1:

[0758] As soon as the agent starts operating, the device uses its camera and microphone to collect the user's facial expressions and voice in real time. This data is sent to the emotion engine as a feed to evaluate the user's emotions.

[0759] Step 2:

[0760] The server analyzes user emotion data sent from the terminal using an emotion engine. This analysis allows the server to capture the user's emotional state, such as excitement, stress, and satisfaction, as numerical values.

[0761] Step 3:

[0762] The server integrates emotional data analyzed by the emotion engine, agent usage frequency data, and user feedback data, and calculates an overall performance evaluation score using an AI model. This score accurately reflects the agent's effectiveness and the user experience.

[0763] Step 4:

[0764] The server generates digital identification information for each agent based on the calculated performance evaluation score. This digital identification information is recorded on the blockchain and issued as an NFT representing the agent's digital ownership.

[0765] Step 5:

[0766] The server lists digital identification information, including performance evaluation scores and sentiment data analysis results, on the marketplace platform, making it accessible to users.

[0767] Step 6:

[0768] Users browse a list of agents from the marketplace and select the best agent based on sentiment data. When purchasing an agent's NFT, payment is made via cryptocurrency using a wallet app.

[0769] Step 7:

[0770] The server automatically updates the record on the blockchain once a transaction is completed and distributes the revenue based on the transaction to the developers. This revenue includes value based on sentiment data and is fairly returned to the developers.

[0771] (Example 2)

[0772] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0773] Existing agent evaluation systems have a problem in that they do not adequately evaluate agent performance while considering user emotions, resulting in the user experience not being fully utilized. Furthermore, there is a need for suggestions for agent improvements based on evaluation results, as well as ensuring transparency and reliability in the evaluation process.

[0774] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0775] In this invention, the server includes means for acquiring user voice and video via a terminal and analyzing emotions; means for integrating agent usage, user feedback, and emotion analysis data to quantify the performance of each agent; and means for generating digital identification information for the agents whose performance has been quantified and managing that identification information using distributed ledger technology. This enables highly accurate evaluation of agents that reflect user emotions and transparent system operation with suggestions for improvement.

[0776] A "terminal" is a device that acquires the user's voice and video and transmits that data to a server.

[0777] "Audio and video" refers to acoustic and visual data that indicates the user's state, and these are sources of information used to analyze emotions.

[0778] "Methods for analyzing emotions" refer to processes and technologies that estimate a user's emotions in real time based on acquired audio and video data.

[0779] An "agent" is software or hardware that performs a specific function and interacts with the user.

[0780] "Usage" refers to information about how the agent is being used by users.

[0781] "User feedback" refers to information that shows the opinions and impressions that users provide regarding their use of the agent.

[0782] "Emotional analysis data" refers to data obtained by analyzing the emotional state of users, and it forms the basis for evaluation.

[0783] "Methods for quantifying performance" refer to the process of expressing the agent's performance numerically based on collected data.

[0784] "Digital identification information" refers to unique digital data generated based on the performance evaluation of an agent, and is used for ownership and transactions.

[0785] "Distributed ledger technology" is a technology for managing and recording digital identification information in a tamper-proof manner, and blockchain technology is generally used for this purpose.

[0786] One embodiment of this invention provides a system for performing real-time performance evaluation of an agent while taking user emotions into consideration. This system mainly consists of a server, a terminal, a user, and an emotion analysis engine.

[0787] First, the device uses its camera and microphone to capture the user's voice and video. This allows it to capture the user's instantaneous emotional state. The voice data is used to analyze features such as voice tone and volume, and the video data is used for facial expression recognition. This data is transmitted to the server in real time.

[0788] The server inputs audio and video data received from the terminal into an emotion analysis engine to analyze the user's emotions. Specifically, it uses an audio analysis model to classify voice tone and an expression recognition model to analyze the user's facial movements. This allows the user's emotions to be estimated in real time with labels such as "joy," "sadness," and "surprise."

[0789] Subsequently, the server integrates agent usage data, user feedback data, and analyzed sentiment data, and uses AI to quantify the agent's performance. This quantified data is then used in conjunction with a generative AI model. An example of a prompt used in this process is, "What improvements can be made to the agent based on this sentiment pattern?"

[0790] Next, the server generates digital identification information based on quantified performance evaluations and manages this information in a tamper-proof manner using distributed ledger technology, such as blockchain. Blockchain management enhances the transparency and reliability of agent performance evaluations.

[0791] Finally, users can view agent performance data on the marketplace and trade the digital identification information of agents they are interested in. These transactions are conducted using cryptocurrency and are secured by smart contracts.

[0792] In this way, by utilizing user sentiment data to evaluate and improve agent performance, it is possible to provide a better user experience.

[0793] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0794] Step 1:

[0795] The device acquires the user's voice and video using a camera and microphone. The input consists of the user's real-time voice and facial images, which are transmitted to the server as digital data. Specifically, the voice recognition software installed on the device captures the user's voice and digitizes its waveform. In addition, the video capture function captures the user's facial movements frame by frame, converts them into data, and transmits it.

[0796] Step 2:

[0797] The server receives the input and passes it to the emotion analysis engine. The input here consists of digitized audio and video. The server inputs the audio data into the audio analysis model and the video data into the facial recognition model. As part of the data processing, it analyzes the pitch and speed of the audio and performs calculations to extract facial feature points from the video. The output is the user's emotion label (e.g., "joy," "anger," "anxiety," etc.).

[0798] Step 3:

[0799] The server then integrates the sentiment analysis results with agent usage data and user feedback data. Inputs include the aforementioned sentiment labels, agent activity logs, and user text feedback. For data processing, this data is subjected to AI analysis, and a generative AI model is used to quantify the agent's performance. The output is the agent's evaluation score.

[0800] Step 4:

[0801] The server uses this evaluation score to generate digital identification information. This identification information includes a unique agent ID and evaluation score. The output is a digital identification information (NFT), which is recorded in a distributed ledger system. Specifically, blockchain technology is used to execute a process that stores this data in a tamper-proof manner.

[0802] Step 5:

[0803] Users verify the agent's digital identification information on the marketplace and conduct transactions. The input is the digital identification information of the agent selected by the user. Specifically, the user operates the interface to access the marketplace and completes the purchase procedure using cryptocurrency. The output is the transaction history stored in the user's wallet.

[0804] (Application Example 2)

[0805] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0806] The objective of this invention is to provide a system that reflects user emotions in real time, enables more precise evaluation of agent performance, and optimizes the operation of a robotic device that acts in accordance with user emotions.

[0807] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0808] In this invention, the server includes means for collecting agent usage data and user feedback data; means for analyzing the collected data and emotion data estimated from the user's voice and facial expressions to quantify the performance of each agent; means for issuing digital identification information to the agents whose performance has been quantified and recording their ownership; and means for operating a robot device that changes its behavioral patterns based on the user's emotion data. This enables highly accurate agent performance evaluation that reflects the user's emotions and adaptive behavioral control of the robot device in accordance with the user's emotions.

[0809] An "agent" is a program or device that interacts with users and provides information and support.

[0810] "Usage data" refers to data about how the agent is being used.

[0811] "User feedback" refers to the evaluations and opinions that users provide after using the agent.

[0812] "Analysis" is the act of examining collected data to extract useful information.

[0813] "Emotional data" refers to data on the user's psychological state, estimated based on their voice and facial expressions.

[0814] "Quantifying" means expressing data or information as numerical values.

[0815] "Digital identification information" refers to digital data that records the evaluation and characteristics of an agent.

[0816] "Ownership" refers to a property right relating to a specific object or information.

[0817] "To record" is the act of saving information in a medium.

[0818] A "market" is a place and system where buying and selling take place.

[0819] A "robot device" is a mechanical device that can take actions in response to the user's emotions.

[0820] The server collects data on agent usage and user feedback from the user's device. The device is equipped with a camera and microphone, which are used to acquire voice and facial expression data. The server analyzes the user's facial expressions from the camera footage using image processing with the OpenCV library and estimates emotion data using EmotionRecognizer. It also analyzes voice data using the SpeechRecognition library and estimates emotion data based on voice tone.

[0821] User emotion data is used to operate robotic devices that dynamically change their behavior. This information is used with the MusicPlayer module to select appropriate music and provide an environment that is sensitive to the user's psychological state. For example, if a user shows signs of fatigue after a long day, the server detects this and instructs the robotic device to play relaxation music.

[0822] An example of a prompt message is, "If the system detects that the user is tired, it will switch to relaxation mode. Please play calming music to soothe the user." This allows the user to enjoy a system that provides a comfortable experience tailored to their emotions.

[0823] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0824] Step 1:

[0825] The device uses a camera and microphone to collect user facial expressions and audio data. In this process, the camera acquires image frames in real time, and the microphone captures the user's speech as audio data. The input in this process is the user's video and audio, which are then sent to the server.

[0826] Step 2:

[0827] The server analyzes the received video data using the OpenCV library and evaluates the user's emotions through EmotionRecognizer. Specifically, it extracts facial features through image processing and performs data calculations to estimate emotions based on these features. The output is the user's estimated emotion data.

[0828] Step 3:

[0829] The server analyzes the audio data using the SpeechRecognition library to evaluate the speech tone. The audio is converted to text, and speech tone features are extracted from the text and quantified as emotion data. The output is emotion estimation data based on speech tone.

[0830] Step 4:

[0831] The server integrates the emotional data obtained in steps 2 and 3 and uses a generative AI model to perform a comprehensive emotional assessment. This allows for a detailed understanding of the user's current psychological state. The output at this stage is the integrated emotional assessment data.

[0832] Step 5:

[0833] The server determines the specific actions the robot device should take based on emotion evaluation data. If music playback or adjustments to environmental settings are required, the MusicPlayer module is used to select appropriate music and settings. A concrete example of this action might include playing calming music to soothe the user.

[0834] Step 6:

[0835] The robotic device receives instructions from the server and performs actions to adjust the user's environment. This includes playing specified music and adjusting lighting. As a final output, the user experiences a comfortable environment tailored to their emotions.

[0836] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0837] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0838] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0839] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0840] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0841] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0842] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0843] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0844] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0845] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0846] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0847] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0848] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0849] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0850] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0851] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0852] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0853] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0854] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0855] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0856] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0857] The following is further disclosed regarding the embodiments described above.

[0858] (Claim 1)

[0859] Means for collecting agent usage data and user feedback data,

[0860] A means for analyzing the collected data and quantifying the performance of each agent,

[0861] A means for issuing digital identification information to an agent whose performance has been quantified and for recording its ownership,

[0862] A means of providing a market that makes the aforementioned digital identification information tradable,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, wherein the aforementioned digital identification information is managed using blockchain technology.

[0866] (Claim 3)

[0867] The system according to claim 1, further comprising means for recording transaction history in the market and distributing revenue to the agent provider.

[0868] "Example 1"

[0869] (Claim 1)

[0870] A means for collecting data on the operating status of information equipment and user evaluations,

[0871] A means for analyzing the collected data using an AI algorithm and quantifying the performance of each information device,

[0872] A means for generating digital identification information for information devices whose performance has been quantified, and for recording their ownership on a distributed ledger,

[0873] Means for providing an electronic commerce environment that enables the trading of the aforementioned digital identification information,

[0874] After the transaction is completed, the transaction record is updated and the revenue is distributed to the information equipment developers.

[0875] A system that includes this.

[0876] (Claim 2)

[0877] The system according to claim 1, wherein the digital identification information is managed using distributed ledger technology.

[0878] (Claim 3)

[0879] The system according to claim 1 for recording transaction history in the aforementioned e-commerce environment.

[0880] "Application Example 1"

[0881] (Claim 1)

[0882] A device for collecting information on agent usage and user feedback,

[0883] A device that analyzes the collected information and quantifies the performance of each agent,

[0884] A device that issues digital identification information to an agent whose performance has been quantified and records its ownership,

[0885] A device that provides a market for trading the aforementioned digital identification information,

[0886] An information device that displays a list of agent performance evaluations in the aforementioned market and allows for purchase and evaluation,

[0887] A system that includes this.

[0888] (Claim 2)

[0889] The system according to claim 1, wherein the digital identification information is managed using distributed ledger technology.

[0890] (Claim 3)

[0891] The system according to claim 1, comprising a device for recording transaction history in the market and distributing revenue to the agent provider.

[0892] "Example 2 of combining an emotion engine"

[0893] (Claim 1)

[0894] A means of acquiring the user's voice and video via a terminal and analyzing their emotions,

[0895] A means to quantify the performance of each agent by integrating agent usage data, user feedback, and sentiment analysis data,

[0896] A means for generating digital identification information for an agent whose performance has been quantified, and for managing that identification information using distributed ledger technology,

[0897] A means for providing a trading market using the aforementioned digital identification information,

[0898] A system that includes this.

[0899] (Claim 2)

[0900] The system according to claim 1, which includes a process in which revenue is distributed to the agent's developers after a transaction has taken place in the aforementioned market.

[0901] (Claim 3)

[0902] The system according to claim 1, wherein when the aforementioned digital identification information is generated, a process is performed to suggest improvements to the agent using a prompt input model based on the results of sentiment analysis.

[0903] "Application example 2 when combining with an emotional engine"

[0904] (Claim 1)

[0905] Means for collecting agent usage data and user feedback data,

[0906] A means for analyzing the collected data and quantifying the performance of each agent based on emotion data estimated from the user's voice and facial expressions,

[0907] A means for issuing digital identification information to an agent whose performance has been quantified and for recording its ownership,

[0908] A means of providing a market that makes the aforementioned digital identification information tradable,

[0909] Means for operating a robotic device that changes its behavioral patterns based on user emotional data,

[0910] A system that includes this.

[0911] (Claim 2)

[0912] The system according to claim 1, wherein the digital identification information is managed using distributed ledger technology.

[0913] (Claim 3)

[0914] The system according to claim 1, further comprising means for recording transaction history in the market and distributing revenue to the agent provider. [Explanation of symbols]

[0915] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for collecting agent usage data and user feedback data, A means for analyzing the collected data and quantifying the performance of each agent, A means for issuing digital identification information to an agent whose performance has been quantified and for recording its ownership, A means of providing a market that makes the aforementioned digital identification information tradable, A system that includes this.

2. The system according to claim 1, wherein the aforementioned digital identification information is managed using blockchain technology.

3. The system according to claim 1, further comprising means for recording transaction history in the market and distributing revenue to the agent provider.

Images