System

Abstract

A system is provided.SOLUTION: A system comprising: means for recognizing a face of a user; means for storing previous conversation content; means for recognizing a face of a conversation partner; means for acquiring the stored previous conversation content; means for acquiring recent social media post content of the conversation partner; and means for notifying the user of the acquired information by voice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] During business meetings, dinner parties, and other occasions, checking the details of previous conversations and the other person's recent activities requires a lot of time and effort. It is tedious to jot down memories using paper or electronic notes, and updating and managing data is difficult. Furthermore, there are no systems that can appropriately retrieve information from other people's social media accounts and provide topics of conversation based on that information. To solve these problems, there is a need for devices that allow users to efficiently retrieve and utilize information during business meetings and other conversations. [Means for solving the problem]

[0005] The present invention provides a system that recognizes a user's face, stores the content of a previous conversation, recognizes the face of a conversation partner and acquires the stored content of the previous conversation, acquires the conversation partner's recent social media posts, and notifies the user of the acquired information by voice. Specifically, the system includes a means for recognizing the user's face, a means for storing the content of the previous conversation, a means for recognizing the conversation partner's face, a means for acquiring the stored content of the previous conversation, a means for acquiring the conversation partner's recent social media posts, and a means for notifying the user of the acquired information by voice. The system also includes a means for analyzing the content of the previous conversation to identify important keywords, a means for analyzing the social media posts to identify related topics, a means for storing the face photos and names of business partners and friends registered by the user in advance, and a means for linking social media account information. By implementing the system as a glasses-type device with face recognition and voice notification, the user can efficiently acquire information and smoothly progress with business negotiations and conversations.

[0006] "User" refers to a person who wears the glasses-type device and engages in business negotiations or conversations.

[0007] "Facial recognition" refers to a technology that uses a camera to identify a person's face and identify them based on their features.

[0008] "Conversation content" refers to the words and information exchanged between a user and the other party during a business negotiation or conversation.

[0009] "Means for storing" refers to a system or data storage for saving the contents of previous conversations, setting information, etc.

[0010] "Social media" refers to platforms for creating and sharing content online, such as Facebook, Twitter, and Instagram.

[0011] "Voice notification" refers to the function of converting information acquired by the system into voice and conveying it to the user.

[0012] "Business partner" refers to a person with whom you have business conversations or transactions.

[0013] "Analysis" refers to the process of processing acquired information as data and finding meaning and patterns.

[0014] "Keywords" refer to specific important words or phrases found in conversations or posts.

[0015] "Eyeglasses-type device" refers to a device that has the shape of glasses but has built-in electronic functions.

[0016] "Notification means" refers to output devices such as speakers, displays, and vibrations that notify the user of information. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to an eyeglasses-type device that efficiently acquires stored information and new information and notifies the user by voice when the user is engaged in a business negotiation or conversation. Hereinafter, embodiments and specific examples of the present invention will be described.

[0039] System Configuration

[0040] The system mainly includes the following components:

[0041] 1. Terminal

[0042] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[0043] 2. Server

[0044] It is a database and processing system that exists on the cloud and manages and analyzes information on users and business partners, as well as social media data.

[0045] Program processing overview

[0046] 1. User authentication and initial setup

[0047] The user wears the glasses-type device.

[0048] The device uses a built-in camera to perform facial recognition and authenticate the user.

[0049] When using the service for the first time, the user registers information about business partners and friends on the settings screen.

[0050] 2. Recognizing others and acquiring information

[0051] When a user is at a business meeting or dinner, the device uses the camera to recognize the face of the person they are talking to.

[0052] The device sends the recognized facial information to a server and requests data from past conversation history and social media.

[0053] 3. Analysis and provision of information

[0054] The server analyzes past conversation history and social media data to extract important keywords and topics.

[0055] The extracted data is sent to the terminal in text format.

[0056] The device provides this information to the user via a voice assistant.

[0057] Specific examples

[0058] Initial Setup

[0059] 1. The user puts on the glasses-type device and turns it on.

[0060] 2. The device uses a camera to capture the user's face and perform facial authentication.

[0061] 3. When launching the application for the first time, the user opens the application settings screen, registers photos and names of business partners and friends, and links their social media accounts.

[0062] 4. The device sends the registered information to the server and stores it.

[0063] Daily use

[0064] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the other person's face.

[0065] 2. The device sends the recognized facial information to the server and requests past conversation history and the latest social media posts.

[0066] 3. The server searches the database for the corresponding information and performs the analysis.

[0067] 4. The server summarizes the analysis results in text format and sends them to the terminal.

[0068] 5. The device uses the voice assistant to notify the user, "In our last conversation, we talked about ____" or "It seems that ____ has recently started a new project."

[0069] 6. Users can smoothly advance the conversation based on the information provided.

[0070] By implementing the present invention in this manner, users can efficiently prepare for and manage business negotiations and conversations.

[0071] The processing flow will be explained below.

[0072] Program processing steps

[0073] Initial Setup

[0074] Step 1:

[0075] The user wears the glasses-type device.

[0076] The device powers up and starts the system.

[0077] Step 2:

[0078] The device uses the built-in camera to recognize the user's face and complete user authentication.

[0079] Recognized facial data is matched with an internal profile.

[0080] Step 3:

[0081] On first use, the user opens the device's settings screen.

[0082] Users register photos and names of business partners and friends.

[0083] The user configures their social media account access settings.

[0084] Step 4:

[0085] The terminal sends the registration information to the server.

[0086] The server stores the information in a database.

[0087] During a conversation

[0088] Step 1:

[0089] The user initiates a conversation.

[0090] The device's built-in camera captures the face of the person you're talking to and performs facial recognition.

[0091] Step 2:

[0092] The device sends the recognized face data to the server.

[0093] The device requests past conversation history and the latest social media posts.

[0094] Step 3:

[0095] The server searches the database for the corresponding conversation history and social media data.

[0096] The server extracts the relevant information and performs analysis.

[0097] Step 4:

[0098] The server compiles the analysis results in text format and sends them to the terminal.

[0099] The information includes keywords from past conversations and the latest social media posts.

[0100] Step 5:

[0101] The device notifies the user of the received information through the voice assistant.

[0102] The audio will provide information such as, "In our last conversation, we talked about XX," or "X recently started a new project."

[0103] Step 6:

[0104] The user can proceed with the conversation based on the notified information.

[0105] Users can use the provided topics to smoothly conduct business negotiations and dinner meetings.

[0106] By using the above specific processing steps, the present invention can improve the efficiency of preparation and management in business negotiations and conversation scenes, and increase convenience for users.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] In the past, when preparing for and conducting business negotiations or conversations, it was difficult for users to efficiently obtain the content of past conversations or the latest information on the other party. Furthermore, manually managing a large amount of information placed a heavy burden on users. The present invention aims to solve these problems and provide a system that allows users to quickly and efficiently obtain the information they need for business negotiations or conversations.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes means for transmitting face information to an external database system and requesting past conversation history and related information, means for analyzing the past conversation history and online information to extract important keywords and topics, and means for notifying the user of information using a voice assistant, thereby enabling the user to quickly and efficiently obtain information necessary for business negotiations and conversations, and to proceed smoothly.

[0112] "Means for recognizing the user's face" refers to the device's built-in camera and facial recognition software that identifies and authenticates the user's face.

[0113] The "means for storing the contents of the previous conversation" is a function that stores the contents of a conversation that has been held in the past and makes it possible to retrieve the contents again as needed.

[0114] "Means for recognizing the face of the person being spoken to" refers to the device's ability to use a camera and facial recognition software to identify the face of the person with whom the user is speaking.

[0115] The "means for acquiring the stored previous conversation content" is a function for searching and acquiring the saved past conversation content again.

[0116] "Means for obtaining recent posts from the public online information of the interlocutor" refers to a function for obtaining recent posts from the interlocutor's social media or other online platforms.

[0117] The "means for notifying the user of the acquired information by voice" is a function for notifying the user of the acquired information by voice using a voice assistant.

[0118] "Means for sending facial information to an external database system and requesting past conversation history and related information" refers to a function that sends recognized facial information to an external database such as a cloud server and requests past conversation history and related information.

[0119] "Means of extracting important keywords and topics by analyzing past conversation history and online information" refers to a function that uses natural language processing, etc. to identify and extract important keywords and topics based on saved conversation history and acquired online information.

[0120] A "voice assistant" is software that responds to voice instructions from a user and provides information.

[0121] The present invention relates to a glasses-type device that efficiently acquires stored information and new information when a user is engaged in business negotiations or conversations, and notifies the user of the information by voice.

[0122] System Configuration

[0123] The system includes the following components:

[0124] 1. Terminal: The main body of the glasses-type device, which contains a camera, microphone, speaker, processor, memory, and wireless communication module.

[0125] 2. Server: A database and processing system that exists on the cloud and manages and analyzes information on users, business partners, and social media data.

[0126] Program processing overview

[0127] The present invention provides support for users to smoothly conduct business negotiations and conversations through the following program processing.

[0128] 1. User authentication and initialization:

[0129] The user puts on the glasses-type device and turns it on.

[0130] The device takes a picture of the user's face with the built-in camera, performs facial recognition, and authenticates the user. Facial recognition uses technologies such as OpenCV.

[0131] When using the app for the first time, users register photos and names of business partners or friends on the device's application settings screen and link their social media accounts.

[0132] This information is sent from the terminal to the server and stored.

[0133] 2. Recognizing others and obtaining information:

[0134] When a user begins a business negotiation or conversation, the device activates its built-in camera and recognizes the other person's face.

[0135] The device sends the recognized facial information to a server and requests information from past conversation history and social media.

[0136] 3. Analysis and provision of information:

[0137] The server searches and retrieves past conversation history and the latest information from social media from a database, and analyzes it using natural language processing technology (e.g., spaCy).

[0138] As a result of the analysis, important keywords and topics are extracted and sent to the device in text format.

[0139] The device will use a voice assistant such as Google Cloud Text-to-Speech to notify the user via voice.

[0140] Specific examples

[0141] 1. Initial Setup:

[0142] The user puts on the glasses-type device and turns it on. The device then performs facial recognition using a camera.

[0143] Users register information about business partners and friends on the application's settings screen and link their social media accounts.

[0144] The registered information is sent to the server and stored.

[0145] 2. Daily use:

[0146] When a user starts a business negotiation or conversation, the device recognizes the other person's face using the built-in camera and sends that facial information to the server.

[0147] The server searches the database for past conversation history and social media information and analyzes it.

[0148] The analysis results are sent to the device in text format, and the device uses a voice assistant to notify the user, saying things like, "In our last conversation, we talked about XX," or "It seems like XX has recently started a new project."

[0149] Prompt Sentence Examples

[0150] "Know who your next sales call is and show them their latest social media posts."

[0151] "Please extract information from past conversation history that is appropriate for the next topic and tell me."

[0152] In this way, the system of the present invention provides the user with the necessary information quickly and efficiently, and supports smooth progress of business negotiations and conversations.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1:

[0155] User authentication and initialization

[0156] The user puts on the glasses-type device and turns it on.

[0157] Input: Video of the user's face.

[0158] The device captures a picture of the user's face with its built-in camera and uses facial recognition software (e.g., OpenCV) to authenticate the user.

[0159] Output: Authentication complete flag.

[0160] How it works: If authentication is successful, the initial setup process continues: On first use, the user opens the application setup screen on the device.

[0161] Step 2:

[0162] Information registration and linking

[0163] Users operate the application to register photos and names of business partners and friends and link their social media accounts.

[0164] Input: Contact and friend information, social media accounts.

[0165] The device collects this information and sends it to a server in the cloud.

[0166] Output: The registered information is saved on the server.

[0167] How it works: The server stores the received information in a database.

[0168] Step 3:

[0169] Recognizing others and acquiring information

[0170] When a user starts a business meeting or conversation, the device activates the camera and scans the face of the person they are talking to.

[0171] Input: Video of the person you're talking to.

[0172] The device uses facial recognition software to recognize the face of the person you are talking to and sends the facial information to a server.

[0173] Output: ID of the person whose face was recognized.

[0174] What it does: Requests past conversation history and social media data from the server.

[0175] Step 4:

[0176] Searching and retrieving data

[0177] The server searches and retrieves from the database the past conversation history and the latest online postings of the corresponding user or business partner.

[0178] Input: facial recognition data, database query.

[0179] Output: Past conversation history and social media posts.

[0180] Operation: The server retrieves these data and sends them to the device.

[0181] Step 5:

[0182] Analysis of information

[0183] The server analyzes the acquired data and extracts important keywords and topics using natural language processing technology (e.g., spaCy).

[0184] Input: Past conversation history and social media posts.

[0185] Output: Extracted important keywords and topics.

[0186] Operation: The server sends the analysis results to the terminal in text format.

[0187] Step 6:

[0188] Execute voice notification

[0189] The device uses voice assistants such as Google Cloud Text-to-Speech to provide information to the user via voice.

[0190] Input: Analysis results in text format.

[0191] Output: Audio notification.

[0192] What it does: The device notifies the user, "In our last conversation, we talked about ____" or "It seems that ____ has recently started a new project."

[0193] (Application example 1)

[0194] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0195] Systems already exist that use facial recognition technology and social media information to assist in business negotiations and conversations, but applying this to security services would enable patrolling security personnel to efficiently obtain information on the status of facilities and past security incidents in real time.However, current systems do not adequately link real-time facial recognition with the acquisition of past data, making it difficult for security personnel to respond immediately.

[0196] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0197] In this invention, the server includes means for recognizing a user's face, means for storing the content of a previous conversation, means for recognizing the face of the person being talked to, means for acquiring the stored content of the previous conversation, means for acquiring the content of the person being talked to's recent social media posts, means for notifying the user of the acquired information by voice, means for recognizing faces during patrol and acquiring information about past security incidents, and means for notifying the user of the acquired past security information by voice. This enables security personnel to effectively acquire information about the status of a facility and past security incidents in real time during patrol, enabling them to respond quickly.

[0198] "User" means a person who uses the system.

[0199] A "means for recognizing a face" is a device or method that uses a camera or software to detect a person's face and analyze its features.

[0200] The "means for storing the contents of the previous conversation" is a method or device for storing the contents of a previous conversation in a storage device.

[0201] The "means for recognizing the face of the person being spoken to" refers to a device or method for detecting the face of the person being spoken to during the current conversation and analyzing its features.

[0202] The "means for acquiring the stored content of the previous conversation" is a method or device for retrieving the content of the previous conversation from a storage device.

[0203] A "means for obtaining recent social media posts" is a device or method for automatically obtaining the latest social media posts of a target person.

[0204] The "means for notifying the user of the acquired information by voice" refers to a device or method for conveying the acquired information to the user using voice synthesis technology.

[0205] "Means for recognizing faces during patrol and obtaining information on past security incidents" refers to a device or method for recognizing the faces of people seen during patrol and obtaining information on security incidents in which the people have been involved in the past.

[0206] The "means for notifying the user of acquired past security information by voice" refers to a device or method for conveying acquired security-related information to the user using voice synthesis technology.

[0207] The system of the present invention has the function of recognizing faces in real time while the user is on patrol, acquiring information about past security incidents, and notifying the user of the information by voice. Hereinafter, an embodiment of the present invention will be described.

[0208] System Configuration

[0209] The system mainly includes the following components:

[0210] 1. Terminal

[0211] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[0212] 2. Server

[0213] It is a database and processing system that exists on the cloud and manages and analyzes information about users and their conversation partners.

[0214] Program processing overview

[0215] 1. User Authentication

[0216] The device uses a built-in camera to perform facial recognition and authenticate the user.

[0217] 2. Facial Recognition and Information Acquisition

[0218] The device uses a camera to recognize the faces of people it sees while patrolling.

[0219] The device sends the recognized facial information to a server and requests data on past security incidents.

[0220] 3. Analysis and provision of information

[0221] The server searches the database for the corresponding information and performs the analysis.

[0222] The server compiles the analysis results in text format and sends them to the terminal.

[0223] The device provides this information to the user via a voice assistant.

[0224] Technology used

[0225] Hardware:

[0226] Glasses-type device (camera, microphone, speaker, processor)

[0227] software:

[0228] OpenCV: Camera image acquisition and image processing

[0229] face_recognition: Face recognition library

[0230] speech_recognition: Speech recognition library

[0231] pyttsx3: Text-to-speech library

[0232] requests: Communicating with the server

[0233] Specific examples

[0234] For example, if a user spots a suspicious person in a blind spot while patrolling a facility, the system will immediately recognize the person and provide a voice notification with information about their involvement in past security incidents, allowing the user to take immediate action.

[0235] Prompt Sentence Examples

[0236] Design a security assistant application that uses a glasses-type device to obtain real-time information about the facility's status and past security incidents while the user is patrolling, and provides voice notifications.

[0237] 1. User authentication through facial recognition.

[0238] 2. Camera-based facial recognition during patrols.

[0239] 3. Send your facial image to the server and request past security information.

[0240] 4. Voice notification of retrieved information.

[0241] Design your application to include the above functions and generate the appropriate program code.

[0242] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0243] Step 1:

[0244] When a user wears the glasses-type device, the device uses the built-in camera to perform facial recognition and authenticate the user. The input is the camera image and the output is the authentication result. Authentication is performed using OpenCV and the face_recognition library.

[0245] Step 2:

[0246] While the user is patrolling the facility, the device continuously uses the camera to recognize the faces of people around them. The input is the surrounding image, and the output is the face recognition result. The face_recognition library is used for face recognition.

[0247] Step 3:

[0248] The device sends the recognized facial information to the server and requests data about past security incidents. The input is the recognized facial feature data, and the output is a request to the server. The requests library is used to send the data.

[0249] Step 4:

[0250] The server searches a database based on the received facial information to obtain information on related security incidents. The input is facial feature data, and the output is past security information. The search and analysis are performed using a database system within the server.

[0251] Step 5:

[0252] The server compiles the analysis results in text format and sends them to the terminal. The input is the analyzed security information, and the output is text format data. The requests library is used to send the data.

[0253] Step 6:

[0254] The device notifies the user of the received security information using a voice assistant. The input is text-formatted security information, and the output is voice. The voice notification uses the pyttsx3 library.

[0255] Step 7:

[0256] The user takes the necessary action based on the provided information. The input is the security information notified by voice, and the output is the user's response. This allows for a quick response.

[0257] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0258] The present invention relates to a glasses-type device that uses an emotion engine to recognize emotions when a user is engaged in a business negotiation or conversation, and further adjusts the information provision method and topic selection based on the emotion recognition information. The following describes embodiments of the present invention and specific examples thereof.

[0259] System Configuration

[0260] The system mainly includes the following components:

[0261] 1. Terminal

[0262] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[0263] 2. Server

[0264] It is a database and processing system that exists on the cloud and manages and analyzes information on users and business partners, as well as social media data.

[0265] 3. Emotion Engine

[0266] The system uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, recognizing emotions in real time.

[0267] Program processing overview

[0268] 1. User authentication and initial setup

[0269] The user wears the glasses-type device.

[0270] The device uses a built-in camera to perform facial recognition and authenticate the user.

[0271] When using the service for the first time, the user registers information about business partners and friends on the settings screen.

[0272] 2. Recognizing others and acquiring information

[0273] When a user is at a business meeting or dinner, the device uses the camera to recognize the face of the person they are talking to.

[0274] The device sends the recognized facial information to a server and requests data from past conversation history and social media.

[0275] 3. Emotion Recognition Processing

[0276] The device uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, and uses an emotion engine to recognize emotions.

[0277] The recognized emotion information is sent to a server along with other data.

[0278] 4. Analysis and provision of information

[0279] The server analyzes past conversation history and social media data to extract important keywords and topics.

[0280] Based on the emotions recognized by the emotion engine, the way information is presented and the selection of appropriate topics are adjusted.

[0281] The extracted data is sent to the terminal in text format.

[0282] The device provides this information to the user via a voice assistant.

[0283] Specific examples

[0284] Initial Setup

[0285] 1. The user puts on the glasses-type device and turns it on.

[0286] 2. The device uses a camera to capture the user's face and perform facial authentication.

[0287] 3. When launching the application for the first time, the user opens the application settings screen, registers photos and names of business partners and friends, and links their social media accounts.

[0288] 4. The device sends the registered information to the server and stores it.

[0289] Daily use

[0290] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the other person's face.

[0291] 2. The device sends the recognized facial data to the server and requests past conversation history and the latest social media posts.

[0292] 3. The device uses a camera and microphone to recognize the user's emotions in real time and transmits the emotion data to the server.

[0293] 4. The server searches the database for the corresponding information and performs analysis. The analysis results include keywords from past conversations, the latest social media posts, and topics that take into account the user's emotions.

[0294] 5. The server summarizes the analysis results in text format and sends them to the terminal.

[0295] 6. The device uses a voice assistant to notify the user, saying things like, "In our last conversation, we talked about XX," or "X recently started a new project," providing appropriate topics that reflect the user's emotions.

[0296] 7. The user can then proceed with the conversation based on the information provided. The topics and information provided adapt to the user's emotions, allowing for smooth business negotiations and dinner meetings.

[0297] Through the above-described specific processing steps, the present invention enables users to conduct business negotiations and conversations efficiently and smoothly, and in particular, provides adaptive information that takes emotions into consideration.

[0298] The processing flow will be explained below.

[0299] The processing steps of the program (if it includes an emotion engine that recognizes the user's emotions)

[0300] Initial Setup

[0301] Step 1:

[0302] The user wears the glasses-type device.

[0303] The device powers up and starts the system.

[0304] Step 2:

[0305] The device uses the built-in camera to recognize the user's face and complete user authentication.

[0306] Recognized facial data is matched with an internal profile.

[0307] Step 3:

[0308] On first use, the user opens the device's settings screen.

[0309] Users register photos and names of business partners and friends.

[0310] The user configures their social media account access settings.

[0311] Step 4:

[0312] The terminal sends the registration information to the server.

[0313] The server stores the information in a database.

[0314] During a conversation

[0315] Step 1:

[0316] The user initiates a conversation.

[0317] The device's built-in camera captures the face of the person you're talking to and performs facial recognition.

[0318] Step 2:

[0319] The device sends the recognized face data to the server.

[0320] The device requests past conversation history and the latest social media posts.

[0321] Step 3:

[0322] The device uses its built-in camera and microphone to analyze the user's facial expressions and tone of voice, and uses an emotion engine to recognize real-time emotions.

[0323] The recognized emotion data is sent to a server along with other data.

[0324] Step 4:

[0325] The server searches the database for the corresponding conversation history and social media data.

[0326] The server analyzes the relevant information and identifies important keywords and topics.

[0327] Step 5:

[0328] The server adjusts the method and content of the information it provides based on the user's emotional information.

[0329] If the sentiment is negative, adjust it to add supportive information or encouraging messages.

[0330] Step 6:

[0331] The server compiles the analysis results in text format and sends them to the terminal.

[0332] The information includes keywords from past conversations, recent social media posts, and relevant topics based on emotional information.

[0333] Step 7:

[0334] The device notifies the user of the received information through the voice assistant.

[0335] The audio will provide information such as, "In our last conversation, we talked about XX," or "X recently started a new project."

[0336] Depending on the user's emotions, it suggests appropriate topics such as "How about talking about ○○ to relax?"

[0337] Step 8:

[0338] The user can proceed with the conversation based on the notified information.

[0339] The topics and information provided are adjusted to match the user's emotions, making business negotiations and dinner meetings go more smoothly.

[0340] By using the above specific processing steps, the present invention enables the user to efficiently acquire information in business negotiations and conversation scenes, and provides information adaptively according to emotions.

[0341] Example 2

[0342] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0343] Conventional conversation assistance systems provide information without considering the emotions of the user or the person they are talking to, making it difficult to select the right topic at the right time. Furthermore, they lack the functionality to efficiently collect and analyze past conversation content and social media information to provide useful information. This reduces the user's ability to adapt to smoothly progress business negotiations and conversations.

[0344] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0345] In this invention, the server includes means for recognizing the user's face, means for storing the content of the previous conversation, means for recognizing the face of the conversation partner, means for acquiring the stored content of the previous conversation, means for acquiring the content of the conversation partner's recent social media posts, means for recognizing the user's emotions in real time, means for analyzing the acquired information and the recognized emotion information to identify related topics, and means for notifying the user of the acquired information by voice. This allows the user to receive appropriate information based on the emotion of the conversation partner, thereby enabling business negotiations and conversations to proceed more smoothly.

[0346] A "user" is an individual who uses the system to conduct business or conversations.

[0347] "Facial recognition means" refers to technology that captures facial images with a camera and identifies specific people from those images.

[0348] "Means for storing the content of previous conversations" refers to technology that has the function of saving the content of past conversations in a database.

[0349] "Means for recognizing the face of the person being conversed with" refers to technology that uses a camera to capture and identify the face image of the person being conversed with the user.

[0350] "Means for retrieving stored content of previous conversations" refers to technology that has the function of searching and retrieving past conversation data stored in a database.

[0351] "Means of obtaining recent social media posts" refers to technology that has the function of collecting the latest post data from social media such as SNS.

[0352] "Means for recognizing emotions in real time" refers to technology that uses a camera or microphone to analyze a user's facial expressions and tone of voice to identify emotions in real time.

[0353] "Means for identifying relevant topics by analyzing acquired information and recognized emotional information" refers to technology that automatically selects appropriate topics for conversation based on collected data and emotional information.

[0354] "Means of notifying the user of acquired information by voice" refers to technology that uses voice assistants or other means to notify the user of analysis results and related information.

[0355] This invention relates to a system that uses an emotion engine to recognize emotions when a user is engaged in business negotiations or conversations, and then adjusts the information provision method and topic selection based on that information. This system is composed of a glasses-type device (hereinafter referred to as the "terminal") and a cloud server (hereinafter referred to as the "server").

[0356] System Components

[0357] 1. Terminal

[0358] The device is a glasses-type device that incorporates a camera, microphone, speaker, processor, memory, wireless communication module, etc. This device performs facial recognition, emotion recognition, and acts as a voice assistant for the user.

[0359] 2. Server

[0360] The server is located in the cloud and functions as a database and data analysis system, managing and analyzing information about users and their conversation partners, social media data, and past conversation history.

[0361] 3. Emotion Engine

[0362] The emotion engine analyzes the user's facial expressions and tone of voice through the built-in camera and microphone, recognizing emotions in real time.

[0363] Program processing

[0364] This system performs the following main processes:

[0365] 1. Initial Setup and User Authentication

[0366] The user puts on the glasses-type device and turns it on. The device uses the built-in camera to capture the user's face and activates a facial recognition system (e.g., OpenCV or TensorFlow) to authenticate the user. If authentication is successful, the user can register information about business partners and friends on the settings screen. The device then sends this information to the server and stores it in a database.

[0367] 2. Recognizing others and acquiring information

[0368] When a business meeting or conversation begins, the device recognizes the face of the person it is talking to using its built-in camera. The recognized facial data is then sent to the server, which then searches past conversation history and social media data to obtain the necessary information. This communication uses the HTTP or HTTPS protocol.

[0369] 3. Emotion Recognition and Information Transmission

[0370] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine (e.g., Google Cloud Vision API or IBM Watson Tone Analyzer). The recognized emotion data is then sent to a server.

[0371] 4. Analysis and provision of information

[0372] The server analyzes the received emotional data, past conversation history, and social media data. NLP technology is used in the analysis to extract important keywords and topics. The analysis results are compiled in text format and sent to the device. The device then uses Google Assistant, Amazon Alexa, or other voice recognition to notify the user.

[0373] Specific examples

[0374] Example of initial settings

[0375] 1. The user puts on the glasses-type device and presses the power button to start it up.

[0376] 2. The device uses the built-in camera to capture the user's face and processes the facial image in real time.

[0377] 3. The device will use the OpenCV library to perform facial recognition, and if successful, will notify you by voice message saying "Authentication successful."

[0378] 4. The user opens the application's settings screen, enters information about business partners and friends, and connects their social media accounts. This information is sent from the device to the server and stored.

[0379] Examples of everyday use

[0380] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the face of the person they are talking to.

[0381] 2. The device sends the recognized facial data to the server, which then retrieves past conversation history and the latest social media posts from a database.

[0382] 3. The device uses a camera and microphone to recognize the user's emotions in real time and transmits the data to the server.

[0383] 4. The server uses natural language processing technology to analyze and select appropriate topics and information.

[0384] 5. The device uses Google Assistant to notify the user, "In our last conversation, we talked about ____" or "____ recently started a new project."

[0385] 6. Users can advance the conversation based on the information provided.

[0386] Examples of prompt statements

[0387] "Please explain how a user can recognize the emotions of the person they are talking to and offer appropriate topics of conversation when conducting business using a glasses-type device."

[0388] This system allows users to conduct business negotiations and conversations efficiently and smoothly, and provides adaptive information based on emotions.

[0389] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0390] Step 1:

[0391] The user puts on the glasses-type device and turns it on. The device captures the user's face using the built-in camera. It takes the facial image as input and starts a facial recognition system (e.g., OpenCV or TensorFlow) to authenticate the user. If facial recognition is successful, it outputs a message indicating successful authentication. Specifically, the camera takes a picture of the user's face, the image is sent to the processor, and a deep learning model performs facial recognition.

[0392] Step 2:

[0393] The user registers information about business partners and friends on a setup screen. The information entered on the setup screen includes a photo, name, and social media accounts. The device sends this information to the server and stores it in a database. An output confirmation message is displayed to confirm that the input data has been saved in the database. Specifically, the device formats the data to be sent and sends it to the server using an HTTP request.

[0394] Step 3:

[0395] When a user initiates a business meeting or conversation, the device uses its built-in camera to recognize the face of the person it is talking to. The captured facial image is sent as input to a server. The server then analyzes past conversation history and social media information based on this facial data to obtain the necessary data. The analysis results are sent as output to the device. Specifically, the server runs a facial recognition algorithm to search a database and extract relevant information.

[0396] Step 4:

[0397] The device uses a built-in camera and microphone to collect the user's facial expressions and tone of voice in real time. This data is sent to an emotion engine (e.g., Google Cloud Vision API or IBM Watson Tone Analyzer) for analysis. The input data is facial expression and tone of voice information, and the output data is recognized emotion information. Specifically, facial images captured by the camera and audio recorded by the microphone are sent to a processor and analyzed by the emotion recognition engine.

[0398] Step 5:

[0399] The recognized emotion data is sent to a server. The input data includes emotion information, and the server performs further data analysis based on this. The analysis results are sent to the device, and the output includes appropriate topics and keywords. Specifically, the server compares the received emotion data with a database and uses natural language processing technology to extract important keywords and topics.

[0400] Step 6:

[0401] Based on the received data, the device notifies the user using a voice assistant (e.g., Google Assistant or Amazon Alexa). The input includes the analysis results and appropriate topic information, and the output is a voice notification. Specifically, the device converts the analysis results into a voice format and notifies the user, for example, "In our last conversation, we talked about XX," or "X recently started a new project."

[0402] These steps realize a system that supports users in efficiently and smoothly conducting business negotiations and conversations.

[0403] (Application example 2)

[0404] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0405] Conventional customer service systems provide uniform responses without considering customer emotions, making it difficult to improve customer satisfaction or provide appropriate responses. While some systems utilize conversation content and social media information, they are unable to obtain customer emotional information in real time and dynamically provide appropriate information based on this information. Therefore, there was a need for a system that enables individual responses based on customer emotions and supports smoother communication.

[0406] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's face, means for storing the content of the previous conversation, means for recognizing the face of the conversation partner, means for acquiring the stored content of the previous conversation, means for acquiring the content of the conversation partner's recent social media posts, means for notifying the user of the acquired information by voice, means for analyzing the user's facial expression to recognize emotions, and means for providing appropriate information based on the emotions. This makes it possible to provide appropriate information and select topics according to the customer's emotions, thereby improving the quality of customer service.

[0407] "Means for recognizing a user's face" refers to the technology that the system uses to detect and identify the user's face.

[0408] The "means for storing the contents of the previous conversation" is a technique for saving the contents of a previous conversation between the user and the conversation partner.

[0409] The "means for recognizing the face of the interlocutor" is a technique that the system uses to detect and identify the face of the user's interlocutor.

[0410] The "means for acquiring stored previous conversation content" is a technique for calling up and making available the saved content of a previous conversation.

[0411] The "means for acquiring the latest social media posts of the conversation partner" is a technique for collecting the latest information posted on social media by the user's conversation partner.

[0412] The "means for notifying the user of acquired information by voice" is a technique for transmitting collected information to the user by voice.

[0413] "Means for analyzing a user's facial expression for emotion recognition" refers to technology that uses cameras and sensors to analyze a user's facial expression and identify the emotion.

[0414] "Means for providing appropriate information based on emotions" refers to technology that selects and provides optimal topics and information to users in response to recognized emotions.

[0415] MODE FOR CARRYING OUT THE INVENTION

[0416] This invention is a system that uses smart glasses to recognize customer emotions in real time when a user is serving customers in a physical store, and provides appropriate information and topics. The main components of the system are smart glasses (terminals), a cloud server (server), and an emotion engine. Detailed embodiments of the system are described below.

[0417] System Configuration

[0418] 1. Terminal

[0419] The smart glasses contain the following hardware:

[0420] Camera: A device used to capture images of users and customers' faces.

[0421] Microphone: A device for capturing the voice of users and customers.

[0422] Speaker: A device that notifies the user of acquired information by voice.

[0423] Processor: A central processing unit for processing data.

[0424] Memory: Storage for temporarily storing data.

[0425] Wireless communication module: A module for communicating with the server.

[0426] 2. Server

[0427] The servers in the cloud contain the following software and databases:

[0428] Database: Manages user and customer information, past conversation history, and social media data.

[0429] Emotion engine: Algorithms that analyze facial expressions and tone of voice to recognize emotions in users and customers.

[0430] Analysis system: A system that analyzes past conversation history and social media data to extract important keywords and topics.

[0431] Program processing overview

[0432] 1. User authentication and initial setup

[0433] When a user puts on the smart glasses, facial recognition is performed using the device's built-in camera. When using the glasses for the first time, the user registers information about business partners and customers on the settings screen and links their social media accounts. The device then sends the registered information to a server and stores it.

[0434] 2. Customer awareness and information acquisition

[0435] When a user interacts with a customer in a physical store, the device uses a camera to recognize the face of the person they are talking to, and sends the recognized facial information to a server, requesting past conversation history and the latest data from social media.

[0436] 3. Emotion Recognition Processing

[0437] The device uses a built-in camera and microphone to analyze the customer's facial expressions and tone of voice, and then uses an emotion engine to recognize emotions, which are then sent to a server along with other data.

[0438] 4. Analysis and provision of information

[0439] The server analyzes past conversation history and social media data to extract important keywords and topics. Based on the emotions recognized by the emotion engine, it adjusts the way information is presented and the selection of appropriate topics. The extracted data is sent to the device in text format, and the device then provides this information to the user via the voice assistant.

[0440] Specific examples

[0441] Example 1: Smart Customer Service Assistant

[0442] When a customer service staff member wears smart glasses and serves them in a physical store, the camera recognizes the face of the customer as they enter the store. Based on the recognized facial information, the system obtains the customer's past purchase history and preferences, and then suggests appropriate products based on this. If the customer's facial expression shows "satisfaction" or "excitement" regarding a product they are interested in, the system proactively approaches them, and if the customer looks "confused" or "questioning," it provides a detailed explanation.

[0443] Prompt Sentence Examples

[0444] Describe a system that analyzes how a customer reacts in real time using smart glasses that recognize your emotions and provide the most appropriate conversational topic based on that information. For example, if the customer is smiling, the system might provide a positive conversational topic, or if the customer looks anxious, the system might provide words of encouragement. Also describe the facial and emotion recognition algorithms that enable this functionality.

[0445] This system will enable more personalized customer service in physical stores, which is expected to improve customer satisfaction.

[0446] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0447] Step 1:

[0448] The user puts on the smart glasses and turns them on. The device's built-in camera activates and captures a picture of the user's face. Using the captured facial image, a facial recognition algorithm authenticates the user. The input is the user's facial image, and the output is the authentication result. If this authentication process is successful, the smart glasses can be used.

[0449] Step 2:

[0450] When a user starts interacting with a customer in a store, the camera scans the surroundings and recognizes the face of the person they are interacting with. The input is the facial image of the customer in the store, and the output is the customer's identification information. The recognized customer information is sent to the server via the wireless communication module.

[0451] Step 3:

[0452] The server searches a database based on the received customer identification information to retrieve the customer's past conversation history and latest social media posts. The input is customer identification information, and the output is past conversation history and social media data. This data is analyzed within the server to extract important keywords and topics.

[0453] Step 4:

[0454] The device uses a built-in camera and microphone to capture the customer's facial expressions and tone of voice in real time. The input is the customer's facial image and voice data, and the output is the emotion recognition result by the emotion engine. The recognized emotion information is sent to the server.

[0455] Step 5:

[0456] The server combines and analyzes the emotion recognition results with the customer's past data to select topics and information that fit the customer's current emotions. The input is the emotion recognition results, past conversation history, and social media data, and the output is the appropriate topics and information. The results are converted into text format and sent to the device.

[0457] Step 6:

[0458] The device notifies the user of the received information via the voice assistant. The input is text information sent from the server, and the output is a voice message. The user can then proceed with the conversation with the customer based on the information received. This adaptive topic provision improves the quality of customer service and increases customer satisfaction.

[0459] By performing data processing and calculations based on input data at each step and using it as output data for the next step, a system is created in which a series of processes can be carried out seamlessly.

[0460] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0461] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0462] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0463] [Second embodiment]

[0464] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0465] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0466] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0467] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0468] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0469] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0470] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0471] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0472] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0473] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0474] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0475] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0476] The present invention relates to an eyeglasses-type device that efficiently acquires stored information and new information and notifies the user by voice when the user is engaged in a business negotiation or conversation. Hereinafter, embodiments and specific examples of the present invention will be described.

[0477] System Configuration

[0478] The system mainly includes the following components:

[0479] 1. Terminal

[0480] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[0481] 2. Server

[0482] It is a database and processing system that exists on the cloud and manages and analyzes information on users and business partners, as well as social media data.

[0483] Program processing overview

[0484] 1. User authentication and initial setup

[0485] The user wears the glasses-type device.

[0486] The device uses a built-in camera to perform facial recognition and authenticate the user.

[0487] When using the service for the first time, the user registers information about business partners and friends on the settings screen.

[0488] 2. Recognizing others and acquiring information

[0489] When a user is at a business meeting or dinner, the device uses the camera to recognize the face of the person they are talking to.

[0490] The device sends the recognized facial information to a server and requests data from past conversation history and social media.

[0491] 3. Analysis and provision of information

[0492] The server analyzes past conversation history and social media data to extract important keywords and topics.

[0493] The extracted data is sent to the terminal in text format.

[0494] The device provides this information to the user via a voice assistant.

[0495] Specific examples

[0496] Initial Setup

[0497] 1. The user puts on the glasses-type device and turns it on.

[0498] 2. The device uses a camera to capture the user's face and perform facial authentication.

[0499] 3. When launching the application for the first time, the user opens the application settings screen, registers photos and names of business partners and friends, and links their social media accounts.

[0500] 4. The device sends the registered information to the server and stores it.

[0501] Daily use

[0502] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the other person's face.

[0503] 2. The device sends the recognized facial information to the server and requests past conversation history and the latest social media posts.

[0504] 3. The server searches the database for the corresponding information and performs the analysis.

[0505] 4. The server summarizes the analysis results in text format and sends them to the terminal.

[0506] 5. The device uses the voice assistant to notify the user, "In our last conversation, we talked about ____" or "It seems that ____ has recently started a new project."

[0507] 6. Users can smoothly advance the conversation based on the information provided.

[0508] By implementing the present invention in this manner, users can efficiently prepare for and manage business negotiations and conversations.

[0509] The processing flow will be explained below.

[0510] Program processing steps

[0511] Initial Setup

[0512] Step 1:

[0513] The user wears the glasses-type device.

[0514] The device powers up and starts the system.

[0515] Step 2:

[0516] The device uses the built-in camera to recognize the user's face and complete user authentication.

[0517] Recognized facial data is matched with an internal profile.

[0518] Step 3:

[0519] On first use, the user opens the device's settings screen.

[0520] Users register photos and names of business partners and friends.

[0521] The user configures their social media account access settings.

[0522] Step 4:

[0523] The terminal sends the registration information to the server.

[0524] The server stores the information in a database.

[0525] During a conversation

[0526] Step 1:

[0527] The user initiates a conversation.

[0528] The device's built-in camera captures the face of the person you're talking to and performs facial recognition.

[0529] Step 2:

[0530] The device sends the recognized face data to the server.

[0531] The device requests past conversation history and the latest social media posts.

[0532] Step 3:

[0533] The server searches the database for the corresponding conversation history and social media data.

[0534] The server extracts the relevant information and performs analysis.

[0535] Step 4:

[0536] The server compiles the analysis results in text format and sends them to the terminal.

[0537] The information includes keywords from past conversations and the latest social media posts.

[0538] Step 5:

[0539] The device notifies the user of the received information through the voice assistant.

[0540] The audio will provide information such as, "In our last conversation, we talked about XX," or "X recently started a new project."

[0541] Step 6:

[0542] The user can proceed with the conversation based on the notified information.

[0543] Users can use the provided topics to smoothly conduct business negotiations and dinner meetings.

[0544] By using the above specific processing steps, the present invention can improve the efficiency of preparation and management in business negotiations and conversation scenes, and increase convenience for users.

[0545] Example 1

[0546] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0547] In the past, when preparing for and conducting business negotiations or conversations, it was difficult for users to efficiently obtain the content of past conversations or the latest information on the other party. Furthermore, manually managing a large amount of information placed a heavy burden on users. The present invention aims to solve these problems and provide a system that allows users to quickly and efficiently obtain the information they need for business negotiations or conversations.

[0548] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0549] In this invention, the server includes means for transmitting face information to an external database system and requesting past conversation history and related information, means for analyzing the past conversation history and online information to extract important keywords and topics, and means for notifying the user of information using a voice assistant, thereby enabling the user to quickly and efficiently obtain information necessary for business negotiations and conversations, and to proceed smoothly.

[0550] "Means for recognizing the user's face" refers to the device's built-in camera and facial recognition software that identifies and authenticates the user's face.

[0551] The "means for storing the contents of the previous conversation" is a function that stores the contents of a conversation that has been held in the past and makes it possible to retrieve the contents again as needed.

[0552] "Means for recognizing the face of the person being spoken to" refers to the device's ability to use a camera and facial recognition software to identify the face of the person with whom the user is speaking.

[0553] The "means for acquiring the stored previous conversation content" is a function for searching and acquiring the saved past conversation content again.

[0554] "Means for obtaining recent posts from the public online information of the interlocutor" refers to a function for obtaining recent posts from the interlocutor's social media or other online platforms.

[0555] The "means for notifying the user of the acquired information by voice" is a function for notifying the user of the acquired information by voice using a voice assistant.

[0556] "Means for sending facial information to an external database system and requesting past conversation history and related information" refers to a function that sends recognized facial information to an external database such as a cloud server and requests past conversation history and related information.

[0557] "Means of extracting important keywords and topics by analyzing past conversation history and online information" refers to a function that uses natural language processing, etc. to identify and extract important keywords and topics based on saved conversation history and acquired online information.

[0558] A "voice assistant" is software that responds to voice instructions from a user and provides information.

[0559] The present invention relates to a glasses-type device that efficiently acquires stored information and new information when a user is engaged in business negotiations or conversations, and notifies the user of the information by voice.

[0560] System Configuration

[0561] The system includes the following components:

[0562] 1. Terminal: The main body of the glasses-type device, which contains a camera, microphone, speaker, processor, memory, and wireless communication module.

[0563] 2. Server: A database and processing system that exists on the cloud and manages and analyzes information on users, business partners, and social media data.

[0564] Program processing overview

[0565] The present invention provides support for users to smoothly conduct business negotiations and conversations through the following program processing.

[0566] 1. User authentication and initialization:

[0567] The user puts on the glasses-type device and turns it on.

[0568] The device takes a picture of the user's face with the built-in camera, performs facial recognition, and authenticates the user. Facial recognition uses technologies such as OpenCV.

[0569] When using the app for the first time, users register photos and names of business partners or friends on the device's application settings screen and link their social media accounts.

[0570] This information is sent from the terminal to the server and stored.

[0571] 2. Recognizing others and obtaining information:

[0572] When a user begins a business negotiation or conversation, the device activates its built-in camera and recognizes the other person's face.

[0573] The device sends the recognized facial information to a server and requests information from past conversation history and social media.

[0574] 3. Analysis and provision of information:

[0575] The server searches and retrieves past conversation history and the latest information from social media from a database, and analyzes it using natural language processing technology (e.g., spaCy).

[0576] As a result of the analysis, important keywords and topics are extracted and sent to the device in text format.

[0577] The device will use a voice assistant such as Google Cloud Text-to-Speech to notify the user via voice.

[0578] Specific examples

[0579] 1. Initial Setup:

[0580] The user puts on the glasses-type device and turns it on. The device then performs facial recognition using a camera.

[0581] Users register information about business partners and friends on the application's settings screen and link their social media accounts.

[0582] The registered information is sent to the server and stored.

[0583] 2. Daily use:

[0584] When a user starts a business negotiation or conversation, the device recognizes the other person's face using the built-in camera and sends that facial information to the server.

[0585] The server searches the database for past conversation history and social media information and analyzes it.

[0586] The analysis results are sent to the device in text format, and the device uses a voice assistant to notify the user, saying things like, "In our last conversation, we talked about XX," or "It seems like XX has recently started a new project."

[0587] Prompt Sentence Examples

[0588] "Know who your next sales call is and show them their latest social media posts."

[0589] "Please extract information from past conversation history that is appropriate for the next topic and tell me."

[0590] In this way, the system of the present invention provides the user with the necessary information quickly and efficiently, and supports smooth progress of business negotiations and conversations.

[0591] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0592] Step 1:

[0593] User authentication and initialization

[0594] The user puts on the glasses-type device and turns it on.

[0595] Input: Video of the user's face.

[0596] The device captures a picture of the user's face with its built-in camera and uses facial recognition software (e.g., OpenCV) to authenticate the user.

[0597] Output: Authentication complete flag.

[0598] How it works: If authentication is successful, the initial setup process continues: On first use, the user opens the application setup screen on the device.

[0599] Step 2:

[0600] Information registration and linking

[0601] Users operate the application to register photos and names of business partners and friends and link their social media accounts.

[0602] Input: Contact and friend information, social media accounts.

[0603] The device collects this information and sends it to a server in the cloud.

[0604] Output: The registered information is saved on the server.

[0605] How it works: The server stores the received information in a database.

[0606] Step 3:

[0607] Recognizing others and acquiring information

[0608] When a user starts a business meeting or conversation, the device activates the camera and scans the face of the person they are talking to.

[0609] Input: Video of the person you're talking to.

[0610] The device uses facial recognition software to recognize the face of the person you are talking to and sends the facial information to a server.

[0611] Output: ID of the person whose face was recognized.

[0612] What it does: Requests past conversation history and social media data from the server.

[0613] Step 4:

[0614] Searching and retrieving data

[0615] The server searches and retrieves from the database the past conversation history and the latest online postings of the corresponding user or business partner.

[0616] Input: facial recognition data, database query.

[0617] Output: Past conversation history and social media posts.

[0618] Operation: The server retrieves these data and sends them to the device.

[0619] Step 5:

[0620] Analysis of information

[0621] The server analyzes the acquired data and extracts important keywords and topics using natural language processing technology (e.g., spaCy).

[0622] Input: Past conversation history and social media posts.

[0623] Output: Extracted important keywords and topics.

[0624] Operation: The server sends the analysis results to the terminal in text format.

[0625] Step 6:

[0626] Execute voice notification

[0627] The device uses voice assistants such as Google Cloud Text-to-Speech to provide information to the user via voice.

[0628] Input: Analysis results in text format.

[0629] Output: Audio notification.

[0630] What it does: The device notifies the user, "In our last conversation, we talked about ____" or "It seems that ____ has recently started a new project."

[0631] (Application example 1)

[0632] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0633] Systems already exist that use facial recognition technology and social media information to assist in business negotiations and conversations, but applying this to security services would enable patrolling security personnel to efficiently obtain information on the status of facilities and past security incidents in real time.However, current systems do not adequately link real-time facial recognition with the acquisition of past data, making it difficult for security personnel to respond immediately.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0635] In this invention, the server includes means for recognizing a user's face, means for storing the content of a previous conversation, means for recognizing the face of the person being talked to, means for acquiring the stored content of the previous conversation, means for acquiring the content of the person being talked to's recent social media posts, means for notifying the user of the acquired information by voice, means for recognizing faces during patrol and acquiring information about past security incidents, and means for notifying the user of the acquired past security information by voice. This enables security personnel to effectively acquire information about the status of a facility and past security incidents in real time during patrol, enabling them to respond quickly.

[0636] "User" means a person who uses the system.

[0637] A "means for recognizing a face" is a device or method that uses a camera or software to detect a person's face and analyze its features.

[0638] The "means for storing the contents of the previous conversation" is a method or device for storing the contents of a previous conversation in a storage device.

[0639] The "means for recognizing the face of the person being spoken to" refers to a device or method for detecting the face of the person being spoken to during the current conversation and analyzing its features.

[0640] The "means for acquiring the stored content of the previous conversation" is a method or device for retrieving the content of the previous conversation from a storage device.

[0641] A "means for obtaining recent social media posts" is a device or method for automatically obtaining the latest social media posts of a target person.

[0642] The "means for notifying the user of the acquired information by voice" refers to a device or method for conveying the acquired information to the user using voice synthesis technology.

[0643] "Means for recognizing faces during patrol and obtaining information on past security incidents" refers to a device or method for recognizing the faces of people seen during patrol and obtaining information on security incidents in which the people have been involved in the past.

[0644] The "means for notifying the user of acquired past security information by voice" refers to a device or method for conveying acquired security-related information to the user using voice synthesis technology.

[0645] The system of the present invention has the function of recognizing faces in real time while the user is on patrol, acquiring information about past security incidents, and notifying the user of the information by voice. Hereinafter, an embodiment of the present invention will be described.

[0646] System Configuration

[0647] The system mainly includes the following components:

[0648] 1. Terminal

[0649] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[0650] 2. Server

[0651] It is a database and processing system that exists on the cloud and manages and analyzes information about users and their conversation partners.

[0652] Program processing overview

[0653] 1. User Authentication

[0654] The device uses a built-in camera to perform facial recognition and authenticate the user.

[0655] 2. Facial Recognition and Information Acquisition

[0656] The device uses a camera to recognize the faces of people it sees while patrolling.

[0657] The device sends the recognized facial information to a server and requests data on past security incidents.

[0658] 3. Analysis and provision of information

[0659] The server searches the database for the corresponding information and performs the analysis.

[0660] The server compiles the analysis results in text format and sends them to the terminal.

[0661] The device provides this information to the user via a voice assistant.

[0662] Technology used

[0663] Hardware:

[0664] Glasses-type device (camera, microphone, speaker, processor)

[0665] software:

[0666] OpenCV: Camera image acquisition and image processing

[0667] face_recognition: Face recognition library

[0668] speech_recognition: Speech recognition library

[0669] pyttsx3: Text-to-speech library

[0670] requests: Communicating with the server

[0671] Specific examples

[0672] For example, if a user spots a suspicious person in a blind spot while patrolling a facility, the system will immediately recognize the person and provide a voice notification with information about their involvement in past security incidents, allowing the user to take immediate action.

[0673] Prompt Sentence Examples

[0674] Design a security assistant application that uses a glasses-type device to obtain real-time information about the facility's status and past security incidents while the user is patrolling, and provides voice notifications.

[0675] 1. User authentication through facial recognition.

[0676] 2. Camera-based facial recognition during patrols.

[0677] 3. Send your facial image to the server and request past security information.

[0678] 4. Voice notification of retrieved information.

[0679] Design your application to include the above functions and generate the appropriate program code.

[0680] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0681] Step 1:

[0682] When a user wears the glasses-type device, the device uses the built-in camera to perform facial recognition and authenticate the user. The input is the camera image and the output is the authentication result. Authentication is performed using OpenCV and the face_recognition library.

[0683] Step 2:

[0684] While the user is patrolling the facility, the device continuously uses the camera to recognize the faces of people around them. The input is the surrounding image, and the output is the face recognition result. The face_recognition library is used for face recognition.

[0685] Step 3:

[0686] The device sends the recognized facial information to the server and requests data about past security incidents. The input is the recognized facial feature data, and the output is a request to the server. The requests library is used to send the data.

[0687] Step 4:

[0688] The server searches a database based on the received facial information to obtain information on related security incidents. The input is facial feature data, and the output is past security information. The search and analysis are performed using a database system within the server.

[0689] Step 5:

[0690] The server compiles the analysis results in text format and sends them to the terminal. The input is the analyzed security information, and the output is text format data. The requests library is used to send the data.

[0691] Step 6:

[0692] The device notifies the user of the received security information using a voice assistant. The input is text-formatted security information, and the output is voice. The voice notification uses the pyttsx3 library.

[0693] Step 7:

[0694] The user takes the necessary action based on the provided information. The input is the security information notified by voice, and the output is the user's response. This allows for a quick response.

[0695] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0696] The present invention relates to a glasses-type device that uses an emotion engine to recognize emotions when a user is engaged in a business negotiation or conversation, and further adjusts the information provision method and topic selection based on the emotion recognition information. The following describes embodiments of the present invention and specific examples thereof.

[0697] System Configuration

[0698] The system mainly includes the following components:

[0699] 1. Terminal

[0700] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[0701] 2. Server

[0702] It is a database and processing system that exists on the cloud and manages and analyzes information on users and business partners, as well as social media data.

[0703] 3. Emotion Engine

[0704] The system uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, recognizing emotions in real time.

[0705] Program processing overview

[0706] 1. User authentication and initial setup

[0707] The user wears the glasses-type device.

[0708] The device uses a built-in camera to perform facial recognition and authenticate the user.

[0709] When using the service for the first time, the user registers information about business partners and friends on the settings screen.

[0710] 2. Recognizing others and acquiring information

[0711] When a user is at a business meeting or dinner, the device uses the camera to recognize the face of the person they are talking to.

[0712] The device sends the recognized facial information to a server and requests data from past conversation history and social media.

[0713] 3. Emotion Recognition Processing

[0714] The device uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, and uses an emotion engine to recognize emotions.

[0715] The recognized emotion information is sent to a server along with other data.

[0716] 4. Analysis and provision of information

[0717] The server analyzes past conversation history and social media data to extract important keywords and topics.

[0718] Based on the emotions recognized by the emotion engine, the way information is presented and the selection of appropriate topics are adjusted.

[0719] The extracted data is sent to the terminal in text format.

[0720] The device provides this information to the user via a voice assistant.

[0721] Specific examples

[0722] Initial Setup

[0723] 1. The user puts on the glasses-type device and turns it on.

[0724] 2. The device uses a camera to capture the user's face and perform facial authentication.

[0725] 3. When launching the application for the first time, the user opens the application settings screen, registers photos and names of business partners and friends, and links their social media accounts.

[0726] 4. The device sends the registered information to the server and stores it.

[0727] Daily use

[0728] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the other person's face.

[0729] 2. The device sends the recognized facial data to the server and requests past conversation history and the latest social media posts.

[0730] 3. The device uses a camera and microphone to recognize the user's emotions in real time and transmits the emotion data to the server.

[0731] 4. The server searches the database for the corresponding information and performs analysis. The analysis results include keywords from past conversations, the latest social media posts, and topics that take into account the user's emotions.

[0732] 5. The server summarizes the analysis results in text format and sends them to the terminal.

[0733] 6. The device uses a voice assistant to notify the user, saying things like, "In our last conversation, we talked about XX," or "X recently started a new project," providing appropriate topics that reflect the user's emotions.

[0734] 7. The user can then proceed with the conversation based on the information provided. The topics and information provided adapt to the user's emotions, allowing for smooth business negotiations and dinner meetings.

[0735] Through the above-described specific processing steps, the present invention enables users to conduct business negotiations and conversations efficiently and smoothly, and in particular, provides adaptive information that takes emotions into consideration.

[0736] The processing flow will be explained below.

[0737] The processing steps of the program (if it includes an emotion engine that recognizes the user's emotions)

[0738] Initial Setup

[0739] Step 1:

[0740] The user wears the glasses-type device.

[0741] The device powers up and starts the system.

[0742] Step 2:

[0743] The device uses the built-in camera to recognize the user's face and complete user authentication.

[0744] Recognized facial data is matched with an internal profile.

[0745] Step 3:

[0746] On first use, the user opens the device's settings screen.

[0747] Users register photos and names of business partners and friends.

[0748] The user configures their social media account access settings.

[0749] Step 4:

[0750] The terminal sends the registration information to the server.

[0751] The server stores the information in a database.

[0752] During a conversation

[0753] Step 1:

[0754] The user initiates a conversation.

[0755] The device's built-in camera captures the face of the person you're talking to and performs facial recognition.

[0756] Step 2:

[0757] The device sends the recognized face data to the server.

[0758] The device requests past conversation history and the latest social media posts.

[0759] Step 3:

[0760] The device uses its built-in camera and microphone to analyze the user's facial expressions and tone of voice, and uses an emotion engine to recognize real-time emotions.

[0761] The recognized emotion data is sent to a server along with other data.

[0762] Step 4:

[0763] The server searches the database for the corresponding conversation history and social media data.

[0764] The server analyzes the relevant information and identifies important keywords and topics.

[0765] Step 5:

[0766] The server adjusts the method and content of the information it provides based on the user's emotional information.

[0767] If the sentiment is negative, adjust it to add supportive information or encouraging messages.

[0768] Step 6:

[0769] The server compiles the analysis results in text format and sends them to the terminal.

[0770] The information includes keywords from past conversations, recent social media posts, and relevant topics based on emotional information.

[0771] Step 7:

[0772] The device notifies the user of the received information through the voice assistant.

[0773] The audio will provide information such as, "In our last conversation, we talked about XX," or "X recently started a new project."

[0774] Depending on the user's emotions, it suggests appropriate topics such as "How about talking about ○○ to relax?"

[0775] Step 8:

[0776] The user can proceed with the conversation based on the notified information.

[0777] The topics and information provided are adjusted to match the user's emotions, making business negotiations and dinner meetings go more smoothly.

[0778] By using the above specific processing steps, the present invention enables the user to efficiently acquire information in business negotiations and conversation scenes, and provides information adaptively according to emotions.

[0779] Example 2

[0780] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0781] Conventional conversation assistance systems provide information without considering the emotions of the user or the person they are talking to, making it difficult to select the right topic at the right time. Furthermore, they lack the functionality to efficiently collect and analyze past conversation content and social media information to provide useful information. This reduces the user's ability to adapt to smoothly progress business negotiations and conversations.

[0782] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0783] In this invention, the server includes means for recognizing the user's face, means for storing the content of the previous conversation, means for recognizing the face of the conversation partner, means for acquiring the stored content of the previous conversation, means for acquiring the content of the conversation partner's recent social media posts, means for recognizing the user's emotions in real time, means for analyzing the acquired information and the recognized emotion information to identify related topics, and means for notifying the user of the acquired information by voice. This allows the user to receive appropriate information based on the emotion of the conversation partner, thereby enabling business negotiations and conversations to proceed more smoothly.

[0784] A "user" is an individual who uses the system to conduct business or conversations.

[0785] "Facial recognition means" refers to technology that captures facial images with a camera and identifies specific people from those images.

[0786] "Means for storing the content of previous conversations" refers to technology that has the function of saving the content of past conversations in a database.

[0787] "Means for recognizing the face of the person being conversed with" refers to technology that uses a camera to capture and identify the face image of the person being conversed with the user.

[0788] "Means for retrieving stored content of previous conversations" refers to technology that has the function of searching and retrieving past conversation data stored in a database.

[0789] "Means of obtaining recent social media posts" refers to technology that has the function of collecting the latest post data from social media such as SNS.

[0790] "Means for recognizing emotions in real time" refers to technology that uses a camera or microphone to analyze a user's facial expressions and tone of voice to identify emotions in real time.

[0791] "Means for identifying relevant topics by analyzing acquired information and recognized emotional information" refers to technology that automatically selects appropriate topics for conversation based on collected data and emotional information.

[0792] "Means of notifying the user of acquired information by voice" refers to technology that uses voice assistants or other means to notify the user of analysis results and related information.

[0793] This invention relates to a system that uses an emotion engine to recognize emotions when a user is engaged in business negotiations or conversations, and then adjusts the information provision method and topic selection based on that information. This system is composed of a glasses-type device (hereinafter referred to as the "terminal") and a cloud server (hereinafter referred to as the "server").

[0794] System Components

[0795] 1. Terminal

[0796] The device is a glasses-type device that incorporates a camera, microphone, speaker, processor, memory, wireless communication module, etc. This device performs facial recognition, emotion recognition, and acts as a voice assistant for the user.

[0797] 2. Server

[0798] The server is located in the cloud and functions as a database and data analysis system, managing and analyzing information about users and their conversation partners, social media data, and past conversation history.

[0799] 3. Emotion Engine

[0800] The emotion engine analyzes the user's facial expressions and tone of voice through the built-in camera and microphone, recognizing emotions in real time.

[0801] Program processing

[0802] This system performs the following main processes:

[0803] 1. Initial Setup and User Authentication

[0804] The user puts on the glasses-type device and turns it on. The device uses the built-in camera to capture the user's face and activates a facial recognition system (e.g., OpenCV or TensorFlow) to authenticate the user. If authentication is successful, the user can register information about business partners and friends on the settings screen. The device then sends this information to the server and stores it in a database.

[0805] 2. Recognizing others and acquiring information

[0806] When a business meeting or conversation begins, the device recognizes the face of the person it is talking to using its built-in camera. The recognized facial data is then sent to the server, which then searches past conversation history and social media data to obtain the necessary information. This communication uses the HTTP or HTTPS protocol.

[0807] 3. Emotion Recognition and Information Transmission

[0808] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine (e.g., Google Cloud Vision API or IBM Watson Tone Analyzer). The recognized emotion data is then sent to a server.

[0809] 4. Analysis and provision of information

[0810] The server analyzes the received emotional data, past conversation history, and social media data. NLP technology is used in the analysis to extract important keywords and topics. The analysis results are compiled in text format and sent to the device. The device then uses Google Assistant, Amazon Alexa, or other voice recognition to notify the user.

[0811] Specific examples

[0812] Example of initial settings

[0813] 1. The user puts on the glasses-type device and presses the power button to start it up.

[0814] 2. The device uses the built-in camera to capture the user's face and processes the facial image in real time.

[0815] 3. The device will use the OpenCV library to perform facial recognition, and if successful, will notify you by voice message saying "Authentication successful."

[0816] 4. The user opens the application's settings screen, enters information about business partners and friends, and connects their social media accounts. This information is sent from the device to the server and stored.

[0817] Examples of everyday use

[0818] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the face of the person they are talking to.

[0819] 2. The device sends the recognized facial data to the server, which then retrieves past conversation history and the latest social media posts from a database.

[0820] 3. The device uses a camera and microphone to recognize the user's emotions in real time and transmits the data to the server.

[0821] 4. The server uses natural language processing technology to analyze and select appropriate topics and information.

[0822] 5. The device uses Google Assistant to notify the user, "In our last conversation, we talked about ____" or "____ recently started a new project."

[0823] 6. Users can advance the conversation based on the information provided.

[0824] Examples of prompt statements

[0825] "Please explain how a user can recognize the emotions of the person they are talking to and offer appropriate topics of conversation when conducting business using a glasses-type device."

[0826] This system allows users to conduct business negotiations and conversations efficiently and smoothly, and provides adaptive information based on emotions.

[0827] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0828] Step 1:

[0829] The user puts on the glasses-type device and turns it on. The device captures the user's face using the built-in camera. It takes the facial image as input and starts a facial recognition system (e.g., OpenCV or TensorFlow) to authenticate the user. If facial recognition is successful, it outputs a message indicating successful authentication. Specifically, the camera takes a picture of the user's face, the image is sent to the processor, and a deep learning model performs facial recognition.

[0830] Step 2:

[0831] The user registers information about business partners and friends on a setup screen. The information entered on the setup screen includes a photo, name, and social media accounts. The device sends this information to the server and stores it in a database. An output confirmation message is displayed to confirm that the input data has been saved in the database. Specifically, the device formats the data to be sent and sends it to the server using an HTTP request.

[0832] Step 3:

[0833] When a user initiates a business meeting or conversation, the device uses its built-in camera to recognize the face of the person it is talking to. The captured facial image is sent as input to a server. The server then analyzes past conversation history and social media information based on this facial data to obtain the necessary data. The analysis results are sent as output to the device. Specifically, the server runs a facial recognition algorithm to search a database and extract relevant information.

[0834] Step 4:

[0835] The device uses a built-in camera and microphone to collect the user's facial expressions and tone of voice in real time. This data is sent to an emotion engine (e.g., Google Cloud Vision API or IBM Watson Tone Analyzer) for analysis. The input data is facial expression and tone of voice information, and the output data is recognized emotion information. Specifically, facial images captured by the camera and audio recorded by the microphone are sent to a processor and analyzed by the emotion recognition engine.

[0836] Step 5:

[0837] The recognized emotion data is sent to a server. The input data includes emotion information, and the server performs further data analysis based on this. The analysis results are sent to the device, and the output includes appropriate topics and keywords. Specifically, the server compares the received emotion data with a database and uses natural language processing technology to extract important keywords and topics.

[0838] Step 6:

[0839] Based on the received data, the device notifies the user using a voice assistant (e.g., Google Assistant or Amazon Alexa). The input includes the analysis results and appropriate topic information, and the output is a voice notification. Specifically, the device converts the analysis results into a voice format and notifies the user, for example, "In our last conversation, we talked about XX," or "X recently started a new project."

[0840] These steps realize a system that supports users in efficiently and smoothly conducting business negotiations and conversations.

[0841] (Application example 2)

[0842] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0843] Conventional customer service systems provide uniform responses without considering customer emotions, making it difficult to improve customer satisfaction or provide appropriate responses. While some systems utilize conversation content and social media information, they are unable to obtain customer emotional information in real time and dynamically provide appropriate information based on this information. Therefore, there was a need for a system that enables individual responses based on customer emotions and supports smoother communication.

[0844] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's face, means for storing the content of the previous conversation, means for recognizing the face of the conversation partner, means for acquiring the stored content of the previous conversation, means for acquiring the content of the conversation partner's recent social media posts, means for notifying the user of the acquired information by voice, means for analyzing the user's facial expression to recognize emotions, and means for providing appropriate information based on the emotions. This makes it possible to provide appropriate information and select topics according to the customer's emotions, thereby improving the quality of customer service.

[0845] "Means for recognizing a user's face" refers to the technology that the system uses to detect and identify the user's face.

[0846] The "means for storing the contents of the previous conversation" is a technique for saving the contents of a previous conversation between the user and the conversation partner.

[0847] The "means for recognizing the face of the interlocutor" is a technique that the system uses to detect and identify the face of the user's interlocutor.

[0848] The "means for acquiring stored previous conversation content" is a technique for calling up and making available the saved content of a previous conversation.

[0849] The "means for acquiring the latest social media posts of the conversation partner" is a technique for collecting the latest information posted on social media by the user's conversation partner.

[0850] The "means for notifying the user of acquired information by voice" is a technique for transmitting collected information to the user by voice.

[0851] "Means for analyzing a user's facial expression for emotion recognition" refers to technology that uses cameras and sensors to analyze a user's facial expression and identify the emotion.

[0852] "Means for providing appropriate information based on emotions" refers to technology that selects and provides optimal topics and information to users in response to recognized emotions.

[0853] MODE FOR CARRYING OUT THE INVENTION

[0854] This invention is a system that uses smart glasses to recognize customer emotions in real time when a user is serving customers in a physical store, and provides appropriate information and topics. The main components of the system are smart glasses (terminals), a cloud server (server), and an emotion engine. Detailed embodiments of the system are described below.

[0855] System Configuration

[0856] 1. Terminal

[0857] The smart glasses contain the following hardware:

[0858] Camera: A device used to capture images of users and customers' faces.

[0859] Microphone: A device for capturing the voice of users and customers.

[0860] Speaker: A device that notifies the user of acquired information by voice.

[0861] Processor: A central processing unit for processing data.

[0862] Memory: Storage for temporarily storing data.

[0863] Wireless communication module: A module for communicating with the server.

[0864] 2. Server

[0865] The servers in the cloud contain the following software and databases:

[0866] Database: Manages user and customer information, past conversation history, and social media data.

[0867] Emotion engine: Algorithms that analyze facial expressions and tone of voice to recognize emotions in users and customers.

[0868] Analysis system: A system that analyzes past conversation history and social media data to extract important keywords and topics.

[0869] Program processing overview

[0870] 1. User authentication and initial setup

[0871] When a user puts on the smart glasses, facial recognition is performed using the device's built-in camera. When using the glasses for the first time, the user registers information about business partners and customers on the settings screen and links their social media accounts. The device then sends the registered information to a server and stores it.

[0872] 2. Customer awareness and information acquisition

[0873] When a user interacts with a customer in a physical store, the device uses a camera to recognize the face of the person they are talking to, and sends the recognized facial information to a server, requesting past conversation history and the latest data from social media.

[0874] 3. Emotion Recognition Processing

[0875] The device uses a built-in camera and microphone to analyze the customer's facial expressions and tone of voice, and then uses an emotion engine to recognize emotions, which are then sent to a server along with other data.

[0876] 4. Analysis and provision of information

[0877] The server analyzes past conversation history and social media data to extract important keywords and topics. Based on the emotions recognized by the emotion engine, it adjusts the way information is presented and the selection of appropriate topics. The extracted data is sent to the device in text format, and the device then provides this information to the user via the voice assistant.

[0878] Specific examples

[0879] Example 1: Smart Customer Service Assistant

[0880] When a customer service staff member wears smart glasses and serves them in a physical store, the camera recognizes the face of the customer as they enter the store. Based on the recognized facial information, the system obtains the customer's past purchase history and preferences, and then suggests appropriate products based on this. If the customer's facial expression shows "satisfaction" or "excitement" regarding a product they are interested in, the system proactively approaches them, and if the customer looks "confused" or "questioning," it provides a detailed explanation.

[0881] Prompt Sentence Examples

[0882] Describe a system that analyzes how a customer reacts in real time using smart glasses that recognize your emotions and provide the most appropriate conversational topic based on that information. For example, if the customer is smiling, the system might provide a positive conversational topic, or if the customer looks anxious, the system might provide words of encouragement. Also describe the facial and emotion recognition algorithms that enable this functionality.

[0883] This system will enable more personalized customer service in physical stores, which is expected to improve customer satisfaction.

[0884] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0885] Step 1:

[0886] The user puts on the smart glasses and turns them on. The device's built-in camera activates and captures a picture of the user's face. Using the captured facial image, a facial recognition algorithm authenticates the user. The input is the user's facial image, and the output is the authentication result. If this authentication process is successful, the smart glasses can be used.

[0887] Step 2:

[0888] When a user starts interacting with a customer in a store, the camera scans the surroundings and recognizes the face of the person they are interacting with. The input is the facial image of the customer in the store, and the output is the customer's identification information. The recognized customer information is sent to the server via the wireless communication module.

[0889] Step 3:

[0890] The server searches a database based on the received customer identification information to retrieve the customer's past conversation history and latest social media posts. The input is customer identification information, and the output is past conversation history and social media data. This data is analyzed within the server to extract important keywords and topics.

[0891] Step 4:

[0892] The device uses a built-in camera and microphone to capture the customer's facial expressions and tone of voice in real time. The input is the customer's facial image and voice data, and the output is the emotion recognition result by the emotion engine. The recognized emotion information is sent to the server.

[0893] Step 5:

[0894] The server combines and analyzes the emotion recognition results with the customer's past data to select topics and information that fit the customer's current emotions. The input is the emotion recognition results, past conversation history, and social media data, and the output is the appropriate topics and information. The results are converted into text format and sent to the device.

[0895] Step 6:

[0896] The device notifies the user of the received information via the voice assistant. The input is text information sent from the server, and the output is a voice message. The user can then proceed with the conversation with the customer based on the information received. This adaptive topic provision improves the quality of customer service and increases customer satisfaction.

[0897] By performing data processing and calculations based on input data at each step and using it as output data for the next step, a system is created in which a series of processes can be carried out seamlessly.

[0898] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0899] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0900] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0901] [Third embodiment]

[0902] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0903] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0904] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0905] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0906] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0907] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0908] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0909] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0910] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0911] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0912] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0913] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0914] The present invention relates to an eyeglasses-type device that efficiently acquires stored information and new information and notifies the user by voice when the user is engaged in a business negotiation or conversation. Hereinafter, embodiments and specific examples of the present invention will be described.

[0915] System Configuration

[0916] The system mainly includes the following components:

[0917] 1. Terminal

[0918] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[0919] 2. Server

[0920] It is a database and processing system that exists on the cloud and manages and analyzes information on users and business partners, as well as social media data.

[0921] Program processing overview

[0922] 1. User authentication and initial setup

[0923] The user wears the glasses-type device.

[0924] The device uses a built-in camera to perform facial recognition and authenticate the user.

[0925] When using the service for the first time, the user registers information about business partners and friends on the settings screen.

[0926] 2. Recognizing others and acquiring information

[0927] When a user is at a business meeting or dinner, the device uses the camera to recognize the face of the person they are talking to.

[0928] The device sends the recognized facial information to a server and requests data from past conversation history and social media.

[0929] 3. Analysis and provision of information

[0930] The server analyzes past conversation history and social media data to extract important keywords and topics.

[0931] The extracted data is sent to the terminal in text format.

[0932] The device provides this information to the user via a voice assistant.

[0933] Specific examples

[0934] Initial Setup

[0935] 1. The user puts on the glasses-type device and turns it on.

[0936] 2. The device uses a camera to capture the user's face and perform facial authentication.

[0937] 3. When launching the application for the first time, the user opens the application settings screen, registers photos and names of business partners and friends, and links their social media accounts.

[0938] 4. The device sends the registered information to the server and stores it.

[0939] Daily use

[0940] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the other person's face.

[0941] 2. The device sends the recognized facial information to the server and requests past conversation history and the latest social media posts.

[0942] 3. The server searches the database for the corresponding information and performs the analysis.

[0943] 4. The server summarizes the analysis results in text format and sends them to the terminal.

[0944] 5. The device uses the voice assistant to notify the user, "In our last conversation, we talked about ____" or "It seems that ____ has recently started a new project."

[0945] 6. Users can smoothly advance the conversation based on the information provided.

[0946] By implementing the present invention in this manner, users can efficiently prepare for and manage business negotiations and conversations.

[0947] The processing flow will be explained below.

[0948] Program processing steps

[0949] Initial Setup

[0950] Step 1:

[0951] The user wears the glasses-type device.

[0952] The device powers up and starts the system.

[0953] Step 2:

[0954] The device uses the built-in camera to recognize the user's face and complete user authentication.

[0955] Recognized facial data is matched with an internal profile.

[0956] Step 3:

[0957] On first use, the user opens the device's settings screen.

[0958] Users register photos and names of business partners and friends.

[0959] The user configures their social media account access settings.

[0960] Step 4:

[0961] The terminal sends the registration information to the server.

[0962] The server stores the information in a database.

[0963] During a conversation

[0964] Step 1:

[0965] The user initiates a conversation.

[0966] The device's built-in camera captures the face of the person you're talking to and performs facial recognition.

[0967] Step 2:

[0968] The device sends the recognized face data to the server.

[0969] The device requests past conversation history and the latest social media posts.

[0970] Step 3:

[0971] The server searches the database for the corresponding conversation history and social media data.

[0972] The server extracts the relevant information and performs analysis.

[0973] Step 4:

[0974] The server compiles the analysis results in text format and sends them to the terminal.

[0975] The information includes keywords from past conversations and the latest social media posts.

[0976] Step 5:

[0977] The device notifies the user of the received information through the voice assistant.

[0978] The audio will provide information such as, "In our last conversation, we talked about XX," or "X recently started a new project."

[0979] Step 6:

[0980] The user can proceed with the conversation based on the notified information.

[0981] Users can use the provided topics to smoothly conduct business negotiations and dinner meetings.

[0982] By using the above specific processing steps, the present invention can improve the efficiency of preparation and management in business negotiations and conversation scenes, and increase convenience for users.

[0983] Example 1

[0984] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0985] In the past, when preparing for and conducting business negotiations or conversations, it was difficult for users to efficiently obtain the content of past conversations or the latest information on the other party. Furthermore, manually managing a large amount of information placed a heavy burden on users. The present invention aims to solve these problems and provide a system that allows users to quickly and efficiently obtain the information they need for business negotiations or conversations.

[0986] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0987] In this invention, the server includes means for transmitting face information to an external database system and requesting past conversation history and related information, means for analyzing the past conversation history and online information to extract important keywords and topics, and means for notifying the user of information using a voice assistant, thereby enabling the user to quickly and efficiently obtain information necessary for business negotiations and conversations, and to proceed smoothly.

[0988] "Means for recognizing the user's face" refers to the device's built-in camera and facial recognition software that identifies and authenticates the user's face.

[0989] The "means for storing the contents of the previous conversation" is a function that stores the contents of a conversation that has been held in the past and makes it possible to retrieve the contents again as needed.

[0990] "Means for recognizing the face of the person being spoken to" refers to the device's ability to use a camera and facial recognition software to identify the face of the person with whom the user is speaking.

[0991] The "means for acquiring the stored previous conversation content" is a function for searching and acquiring the saved past conversation content again.

[0992] "Means for obtaining recent posts from the public online information of the interlocutor" refers to a function for obtaining recent posts from the interlocutor's social media or other online platforms.

[0993] The "means for notifying the user of the acquired information by voice" is a function for notifying the user of the acquired information by voice using a voice assistant.

[0994] "Means for sending facial information to an external database system and requesting past conversation history and related information" refers to a function that sends recognized facial information to an external database such as a cloud server and requests past conversation history and related information.

[0995] "Means of extracting important keywords and topics by analyzing past conversation history and online information" refers to a function that uses natural language processing, etc. to identify and extract important keywords and topics based on saved conversation history and acquired online information.

[0996] A "voice assistant" is software that responds to voice instructions from a user and provides information.

[0997] The present invention relates to a glasses-type device that efficiently acquires stored information and new information when a user is engaged in business negotiations or conversations, and notifies the user of the information by voice.

[0998] System Configuration

[0999] The system includes the following components:

[1000] 1. Terminal: The main body of the glasses-type device, which contains a camera, microphone, speaker, processor, memory, and wireless communication module.

[1001] 2. Server: A database and processing system that exists on the cloud and manages and analyzes information on users, business partners, and social media data.

[1002] Program processing overview

[1003] The present invention provides support for users to smoothly conduct business negotiations and conversations through the following program processing.

[1004] 1. User authentication and initialization:

[1005] The user puts on the glasses-type device and turns it on.

[1006] The device takes a picture of the user's face with the built-in camera, performs facial recognition, and authenticates the user. Facial recognition uses technologies such as OpenCV.

[1007] When using the app for the first time, users register photos and names of business partners or friends on the device's application settings screen and link their social media accounts.

[1008] This information is sent from the terminal to the server and stored.

[1009] 2. Recognizing others and obtaining information:

[1010] When a user begins a business negotiation or conversation, the device activates its built-in camera and recognizes the other person's face.

[1011] The device sends the recognized facial information to a server and requests information from past conversation history and social media.

[1012] 3. Analysis and provision of information:

[1013] The server searches and retrieves past conversation history and the latest information from social media from a database, and analyzes it using natural language processing technology (e.g., spaCy).

[1014] As a result of the analysis, important keywords and topics are extracted and sent to the device in text format.

[1015] The device will use a voice assistant such as Google Cloud Text-to-Speech to notify the user via voice.

[1016] Specific examples

[1017] 1. Initial Setup:

[1018] The user puts on the glasses-type device and turns it on. The device then performs facial recognition using a camera.

[1019] Users register information about business partners and friends on the application's settings screen and link their social media accounts.

[1020] The registered information is sent to the server and stored.

[1021] 2. Daily use:

[1022] When a user starts a business negotiation or conversation, the device recognizes the other person's face using the built-in camera and sends that facial information to the server.

[1023] The server searches the database for past conversation history and social media information and analyzes it.

[1024] The analysis results are sent to the device in text format, and the device uses a voice assistant to notify the user, saying things like, "In our last conversation, we talked about XX," or "It seems like XX has recently started a new project."

[1025] Prompt Sentence Examples

[1026] "Know who your next sales call is and show them their latest social media posts."

[1027] "Please extract information from past conversation history that is appropriate for the next topic and tell me."

[1028] In this way, the system of the present invention provides the user with the necessary information quickly and efficiently, and supports smooth progress of business negotiations and conversations.

[1029] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1030] Step 1:

[1031] User authentication and initialization

[1032] The user puts on the glasses-type device and turns it on.

[1033] Input: Video of the user's face.

[1034] The device captures a picture of the user's face with its built-in camera and uses facial recognition software (e.g., OpenCV) to authenticate the user.

[1035] Output: Authentication complete flag.

[1036] How it works: If authentication is successful, the initial setup process continues: On first use, the user opens the application setup screen on the device.

[1037] Step 2:

[1038] Information registration and linking

[1039] Users operate the application to register photos and names of business partners and friends and link their social media accounts.

[1040] Input: Contact and friend information, social media accounts.

[1041] The device collects this information and sends it to a server in the cloud.

[1042] Output: The registered information is saved on the server.

[1043] How it works: The server stores the received information in a database.

[1044] Step 3:

[1045] Recognizing others and acquiring information

[1046] When a user starts a business meeting or conversation, the device activates the camera and scans the face of the person they are talking to.

[1047] Input: Video of the person you're talking to.

[1048] The device uses facial recognition software to recognize the face of the person you are talking to and sends the facial information to a server.

[1049] Output: ID of the person whose face was recognized.

[1050] What it does: Requests past conversation history and social media data from the server.

[1051] Step 4:

[1052] Searching and retrieving data

[1053] The server searches and retrieves from the database the past conversation history and the latest online postings of the corresponding user or business partner.

[1054] Input: facial recognition data, database query.

[1055] Output: Past conversation history and social media posts.

[1056] Operation: The server retrieves these data and sends them to the device.

[1057] Step 5:

[1058] Analysis of information

[1059] The server analyzes the acquired data and extracts important keywords and topics using natural language processing technology (e.g., spaCy).

[1060] Input: Past conversation history and social media posts.

[1061] Output: Extracted important keywords and topics.

[1062] Operation: The server sends the analysis results to the terminal in text format.

[1063] Step 6:

[1064] Execute voice notification

[1065] The device uses voice assistants such as Google Cloud Text-to-Speech to provide information to the user via voice.

[1066] Input: Analysis results in text format.

[1067] Output: Audio notification.

[1068] What it does: The device notifies the user, "In our last conversation, we talked about ____" or "It seems that ____ has recently started a new project."

[1069] (Application example 1)

[1070] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1071] Systems already exist that use facial recognition technology and social media information to assist in business negotiations and conversations, but applying this to security services would enable patrolling security personnel to efficiently obtain information on the status of facilities and past security incidents in real time.However, current systems do not adequately link real-time facial recognition with the acquisition of past data, making it difficult for security personnel to respond immediately.

[1072] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1073] In this invention, the server includes means for recognizing a user's face, means for storing the content of a previous conversation, means for recognizing the face of the person being talked to, means for acquiring the stored content of the previous conversation, means for acquiring the content of the person being talked to's recent social media posts, means for notifying the user of the acquired information by voice, means for recognizing faces during patrol and acquiring information about past security incidents, and means for notifying the user of the acquired past security information by voice. This enables security personnel to effectively acquire information about the status of a facility and past security incidents in real time during patrol, enabling them to respond quickly.

[1074] "User" means a person who uses the system.

[1075] A "means for recognizing a face" is a device or method that uses a camera or software to detect a person's face and analyze its features.

[1076] The "means for storing the contents of the previous conversation" is a method or device for storing the contents of a previous conversation in a storage device.

[1077] The "means for recognizing the face of the person being spoken to" refers to a device or method for detecting the face of the person being spoken to during the current conversation and analyzing its features.

[1078] The "means for acquiring the stored content of the previous conversation" is a method or device for retrieving the content of the previous conversation from a storage device.

[1079] A "means for obtaining recent social media posts" is a device or method for automatically obtaining the latest social media posts of a target person.

[1080] The "means for notifying the user of the acquired information by voice" refers to a device or method for conveying the acquired information to the user using voice synthesis technology.

[1081] "Means for recognizing faces during patrol and obtaining information on past security incidents" refers to a device or method for recognizing the faces of people seen during patrol and obtaining information on security incidents in which the people have been involved in the past.

[1082] The "means for notifying the user of acquired past security information by voice" refers to a device or method for conveying acquired security-related information to the user using voice synthesis technology.

[1083] The system of the present invention has the function of recognizing faces in real time while the user is on patrol, acquiring information about past security incidents, and notifying the user of the information by voice. Hereinafter, an embodiment of the present invention will be described.

[1084] System Configuration

[1085] The system mainly includes the following components:

[1086] 1. Terminal

[1087] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[1088] 2. Server

[1089] It is a database and processing system that exists on the cloud and manages and analyzes information about users and their conversation partners.

[1090] Program processing overview

[1091] 1. User Authentication

[1092] The device uses a built-in camera to perform facial recognition and authenticate the user.

[1093] 2. Facial Recognition and Information Acquisition

[1094] The device uses a camera to recognize the faces of people it sees while patrolling.

[1095] The device sends the recognized facial information to a server and requests data on past security incidents.

[1096] 3. Analysis and provision of information

[1097] The server searches the database for the corresponding information and performs the analysis.

[1098] The server compiles the analysis results in text format and sends them to the terminal.

[1099] The device provides this information to the user via a voice assistant.

[1100] Technology used

[1101] Hardware:

[1102] Glasses-type device (camera, microphone, speaker, processor)

[1103] software:

[1104] OpenCV: Camera image acquisition and image processing

[1105] face_recognition: Face recognition library

[1106] speech_recognition: Speech recognition library

[1107] pyttsx3: Text-to-speech library

[1108] requests: Communicating with the server

[1109] Specific examples

[1110] For example, if a user spots a suspicious person in a blind spot while patrolling a facility, the system will immediately recognize the person and provide a voice notification with information about their involvement in past security incidents, allowing the user to take immediate action.

[1111] Prompt Sentence Examples

[1112] Design a security assistant application that uses a glasses-type device to obtain real-time information about the facility's status and past security incidents while the user is patrolling, and provides voice notifications.

[1113] 1. User authentication through facial recognition.

[1114] 2. Camera-based facial recognition during patrols.

[1115] 3. Send your facial image to the server and request past security information.

[1116] 4. Voice notification of retrieved information.

[1117] Design your application to include the above functions and generate the appropriate program code.

[1118] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1119] Step 1:

[1120] When a user wears the glasses-type device, the device uses the built-in camera to perform facial recognition and authenticate the user. The input is the camera image and the output is the authentication result. Authentication is performed using OpenCV and the face_recognition library.

[1121] Step 2:

[1122] While the user is patrolling the facility, the device continuously uses the camera to recognize the faces of people around them. The input is the surrounding image, and the output is the face recognition result. The face_recognition library is used for face recognition.

[1123] Step 3:

[1124] The device sends the recognized facial information to the server and requests data about past security incidents. The input is the recognized facial feature data, and the output is a request to the server. The requests library is used to send the data.

[1125] Step 4:

[1126] The server searches a database based on the received facial information to obtain information on related security incidents. The input is facial feature data, and the output is past security information. The search and analysis are performed using a database system within the server.

[1127] Step 5:

[1128] The server compiles the analysis results in text format and sends them to the terminal. The input is the analyzed security information, and the output is text format data. The requests library is used to send the data.

[1129] Step 6:

[1130] The device notifies the user of the received security information using a voice assistant. The input is text-formatted security information, and the output is voice. The voice notification uses the pyttsx3 library.

[1131] Step 7:

[1132] The user takes the necessary action based on the provided information. The input is the security information notified by voice, and the output is the user's response. This allows for a quick response.

[1133] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1134] The present invention relates to a glasses-type device that uses an emotion engine to recognize emotions when a user is engaged in a business negotiation or conversation, and further adjusts the information provision method and topic selection based on the emotion recognition information. The following describes embodiments of the present invention and specific examples thereof.

[1135] System Configuration

[1136] The system mainly includes the following components:

[1137] 1. Terminal

[1138] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[1139] 2. Server

[1140] It is a database and processing system that exists on the cloud and manages and analyzes information on users and business partners, as well as social media data.

[1141] 3. Emotion Engine

[1142] The system uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, recognizing emotions in real time.

[1143] Program processing overview

[1144] 1. User authentication and initial setup

[1145] The user wears the glasses-type device.

[1146] The device uses a built-in camera to perform facial recognition and authenticate the user.

[1147] When using the service for the first time, the user registers information about business partners and friends on the settings screen.

[1148] 2. Recognizing others and acquiring information

[1149] When a user is at a business meeting or dinner, the device uses the camera to recognize the face of the person they are talking to.

[1150] The device sends the recognized facial information to a server and requests data from past conversation history and social media.

[1151] 3. Emotion Recognition Processing

[1152] The device uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, and uses an emotion engine to recognize emotions.

[1153] The recognized emotion information is sent to a server along with other data.

[1154] 4. Analysis and provision of information

[1155] The server analyzes past conversation history and social media data to extract important keywords and topics.

[1156] Based on the emotions recognized by the emotion engine, the way information is presented and the selection of appropriate topics are adjusted.

[1157] The extracted data is sent to the terminal in text format.

[1158] The device provides this information to the user via a voice assistant.

[1159] Specific examples

[1160] Initial Setup

[1161] 1. The user puts on the glasses-type device and turns it on.

[1162] 2. The device uses a camera to capture the user's face and perform facial authentication.

[1163] 3. When launching the application for the first time, the user opens the application settings screen, registers photos and names of business partners and friends, and links their social media accounts.

[1164] 4. The device sends the registered information to the server and stores it.

[1165] Daily use

[1166] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the other person's face.

[1167] 2. The device sends the recognized facial data to the server and requests past conversation history and the latest social media posts.

[1168] 3. The device uses a camera and microphone to recognize the user's emotions in real time and transmits the emotion data to the server.

[1169] 4. The server searches the database for the corresponding information and performs analysis. The analysis results include keywords from past conversations, the latest social media posts, and topics that take into account the user's emotions.

[1170] 5. The server summarizes the analysis results in text format and sends them to the terminal.

[1171] 6. The device uses a voice assistant to notify the user, saying things like, "In our last conversation, we talked about XX," or "X recently started a new project," providing appropriate topics that reflect the user's emotions.

[1172] 7. The user can then proceed with the conversation based on the information provided. The topics and information provided adapt to the user's emotions, allowing for smooth business negotiations and dinner meetings.

[1173] Through the above-described specific processing steps, the present invention enables users to conduct business negotiations and conversations efficiently and smoothly, and in particular, provides adaptive information that takes emotions into consideration.

[1174] The processing flow will be explained below.

[1175] The processing steps of the program (if it includes an emotion engine that recognizes the user's emotions)

[1176] Initial Setup

[1177] Step 1:

[1178] The user wears the glasses-type device.

[1179] The device powers up and starts the system.

[1180] Step 2:

[1181] The device uses the built-in camera to recognize the user's face and complete user authentication.

[1182] Recognized facial data is matched with an internal profile.

[1183] Step 3:

[1184] On first use, the user opens the device's settings screen.

[1185] Users register photos and names of business partners and friends.

[1186] The user configures their social media account access settings.

[1187] Step 4:

[1188] The terminal sends the registration information to the server.

[1189] The server stores the information in a database.

[1190] During a conversation

[1191] Step 1:

[1192] The user initiates a conversation.

[1193] The device's built-in camera captures the face of the person you're talking to and performs facial recognition.

[1194] Step 2:

[1195] The device sends the recognized face data to the server.

[1196] The device requests past conversation history and the latest social media posts.

[1197] Step 3:

[1198] The device uses its built-in camera and microphone to analyze the user's facial expressions and tone of voice, and uses an emotion engine to recognize real-time emotions.

[1199] The recognized emotion data is sent to a server along with other data.

[1200] Step 4:

[1201] The server searches the database for the corresponding conversation history and social media data.

[1202] The server analyzes the relevant information and identifies important keywords and topics.

[1203] Step 5:

[1204] The server adjusts the method and content of the information it provides based on the user's emotional information.

[1205] If the sentiment is negative, adjust it to add supportive information or encouraging messages.

[1206] Step 6:

[1207] The server compiles the analysis results in text format and sends them to the terminal.

[1208] The information includes keywords from past conversations, recent social media posts, and relevant topics based on emotional information.

[1209] Step 7:

[1210] The device notifies the user of the received information through the voice assistant.

[1211] The audio will provide information such as, "In our last conversation, we talked about XX," or "X recently started a new project."

[1212] Depending on the user's emotions, it suggests appropriate topics such as "How about talking about ○○ to relax?"

[1213] Step 8:

[1214] The user can proceed with the conversation based on the notified information.

[1215] The topics and information provided are adjusted to match the user's emotions, making business negotiations and dinner meetings go more smoothly.

[1216] By using the above specific processing steps, the present invention enables the user to efficiently acquire information in business negotiations and conversation scenes, and provides information adaptively according to emotions.

[1217] Example 2

[1218] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1219] Conventional conversation assistance systems provide information without considering the emotions of the user or the person they are talking to, making it difficult to select the right topic at the right time. Furthermore, they lack the functionality to efficiently collect and analyze past conversation content and social media information to provide useful information. This reduces the user's ability to adapt to smoothly progress business negotiations and conversations.

[1220] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1221] In this invention, the server includes means for recognizing the user's face, means for storing the content of the previous conversation, means for recognizing the face of the conversation partner, means for acquiring the stored content of the previous conversation, means for acquiring the content of the conversation partner's recent social media posts, means for recognizing the user's emotions in real time, means for analyzing the acquired information and the recognized emotion information to identify related topics, and means for notifying the user of the acquired information by voice. This allows the user to receive appropriate information based on the emotion of the conversation partner, thereby enabling business negotiations and conversations to proceed more smoothly.

[1222] A "user" is an individual who uses the system to conduct business or conversations.

[1223] "Facial recognition means" refers to technology that captures facial images with a camera and identifies specific people from those images.

[1224] "Means for storing the content of previous conversations" refers to technology that has the function of saving the content of past conversations in a database.

[1225] "Means for recognizing the face of the person being conversed with" refers to technology that uses a camera to capture and identify the face image of the person being conversed with the user.

[1226] "Means for retrieving stored content of previous conversations" refers to technology that has the function of searching and retrieving past conversation data stored in a database.

[1227] "Means of obtaining recent social media posts" refers to technology that has the function of collecting the latest post data from social media such as SNS.

[1228] "Means for recognizing emotions in real time" refers to technology that uses a camera or microphone to analyze a user's facial expressions and tone of voice to identify emotions in real time.

[1229] "Means for identifying relevant topics by analyzing acquired information and recognized emotional information" refers to technology that automatically selects appropriate topics for conversation based on collected data and emotional information.

[1230] "Means of notifying the user of acquired information by voice" refers to technology that uses voice assistants or other means to notify the user of analysis results and related information.

[1231] This invention relates to a system that uses an emotion engine to recognize emotions when a user is engaged in business negotiations or conversations, and then adjusts the information provision method and topic selection based on that information. This system is composed of a glasses-type device (hereinafter referred to as the "terminal") and a cloud server (hereinafter referred to as the "server").

[1232] System Components

[1233] 1. Terminal

[1234] The device is a glasses-type device that incorporates a camera, microphone, speaker, processor, memory, wireless communication module, etc. This device performs facial recognition, emotion recognition, and acts as a voice assistant for the user.

[1235] 2. Server

[1236] The server is located in the cloud and functions as a database and data analysis system, managing and analyzing information about users and their conversation partners, social media data, and past conversation history.

[1237] 3. Emotion Engine

[1238] The emotion engine analyzes the user's facial expressions and tone of voice through the built-in camera and microphone, recognizing emotions in real time.

[1239] Program processing

[1240] This system performs the following main processes:

[1241] 1. Initial Setup and User Authentication

[1242] The user puts on the glasses-type device and turns it on. The device uses the built-in camera to capture the user's face and activates a facial recognition system (e.g., OpenCV or TensorFlow) to authenticate the user. If authentication is successful, the user can register information about business partners and friends on the settings screen. The device then sends this information to the server and stores it in a database.

[1243] 2. Recognizing others and acquiring information

[1244] When a business meeting or conversation begins, the device recognizes the face of the person it is talking to using its built-in camera. The recognized facial data is then sent to the server, which then searches past conversation history and social media data to obtain the necessary information. This communication uses the HTTP or HTTPS protocol.

[1245] 3. Emotion Recognition and Information Transmission

[1246] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine (e.g., Google Cloud Vision API or IBM Watson Tone Analyzer). The recognized emotion data is then sent to a server.

[1247] 4. Analysis and provision of information

[1248] The server analyzes the received emotional data, past conversation history, and social media data. NLP technology is used in the analysis to extract important keywords and topics. The analysis results are compiled in text format and sent to the device. The device then uses Google Assistant, Amazon Alexa, or other voice recognition to notify the user.

[1249] Specific examples

[1250] Example of initial settings

[1251] 1. The user puts on the glasses-type device and presses the power button to start it up.

[1252] 2. The device uses the built-in camera to capture the user's face and processes the facial image in real time.

[1253] 3. The device will use the OpenCV library to perform facial recognition, and if successful, will notify you by voice message saying "Authentication successful."

[1254] 4. The user opens the application's settings screen, enters information about business partners and friends, and connects their social media accounts. This information is sent from the device to the server and stored.

[1255] Examples of everyday use

[1256] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the face of the person they are talking to.

[1257] 2. The device sends the recognized facial data to the server, which then retrieves past conversation history and the latest social media posts from a database.

[1258] 3. The device uses a camera and microphone to recognize the user's emotions in real time and transmits the data to the server.

[1259] 4. The server uses natural language processing technology to analyze and select appropriate topics and information.

[1260] 5. The device uses Google Assistant to notify the user, "In our last conversation, we talked about ____" or "____ recently started a new project."

[1261] 6. Users can advance the conversation based on the information provided.

[1262] Examples of prompt statements

[1263] "Please explain how a user can recognize the emotions of the person they are talking to and offer appropriate topics of conversation when conducting business using a glasses-type device."

[1264] This system allows users to conduct business negotiations and conversations efficiently and smoothly, and provides adaptive information based on emotions.

[1265] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1266] Step 1:

[1267] The user puts on the glasses-type device and turns it on. The device captures the user's face using the built-in camera. It takes the facial image as input and starts a facial recognition system (e.g., OpenCV or TensorFlow) to authenticate the user. If facial recognition is successful, it outputs a message indicating successful authentication. Specifically, the camera takes a picture of the user's face, the image is sent to the processor, and a deep learning model performs facial recognition.

[1268] Step 2:

[1269] The user registers information about business partners and friends on a setup screen. The information entered on the setup screen includes a photo, name, and social media accounts. The device sends this information to the server and stores it in a database. An output confirmation message is displayed to confirm that the input data has been saved in the database. Specifically, the device formats the data to be sent and sends it to the server using an HTTP request.

[1270] Step 3:

[1271] When a user initiates a business meeting or conversation, the device uses its built-in camera to recognize the face of the person it is talking to. The captured facial image is sent as input to a server. The server then analyzes past conversation history and social media information based on this facial data to obtain the necessary data. The analysis results are sent as output to the device. Specifically, the server runs a facial recognition algorithm to search a database and extract relevant information.

[1272] Step 4:

[1273] The device uses a built-in camera and microphone to collect the user's facial expressions and tone of voice in real time. This data is sent to an emotion engine (e.g., Google Cloud Vision API or IBM Watson Tone Analyzer) for analysis. The input data is facial expression and tone of voice information, and the output data is recognized emotion information. Specifically, facial images captured by the camera and audio recorded by the microphone are sent to a processor and analyzed by the emotion recognition engine.

[1274] Step 5:

[1275] The recognized emotion data is sent to a server. The input data includes emotion information, and the server performs further data analysis based on this. The analysis results are sent to the device, and the output includes appropriate topics and keywords. Specifically, the server compares the received emotion data with a database and uses natural language processing technology to extract important keywords and topics.

[1276] Step 6:

[1277] Based on the received data, the device notifies the user using a voice assistant (e.g., Google Assistant or Amazon Alexa). The input includes the analysis results and appropriate topic information, and the output is a voice notification. Specifically, the device converts the analysis results into a voice format and notifies the user, for example, "In our last conversation, we talked about XX," or "X recently started a new project."

[1278] These steps realize a system that supports users in efficiently and smoothly conducting business negotiations and conversations.

[1279] (Application example 2)

[1280] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1281] Conventional customer service systems provide uniform responses without considering customer emotions, making it difficult to improve customer satisfaction or provide appropriate responses. While some systems utilize conversation content and social media information, they are unable to obtain customer emotional information in real time and dynamically provide appropriate information based on this information. Therefore, there was a need for a system that enables individual responses based on customer emotions and supports smoother communication.

[1282] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's face, means for storing the content of the previous conversation, means for recognizing the face of the conversation partner, means for acquiring the stored content of the previous conversation, means for acquiring the content of the conversation partner's recent social media posts, means for notifying the user of the acquired information by voice, means for analyzing the user's facial expression to recognize emotions, and means for providing appropriate information based on the emotions. This makes it possible to provide appropriate information and select topics according to the customer's emotions, thereby improving the quality of customer service.

[1283] "Means for recognizing a user's face" refers to the technology that the system uses to detect and identify the user's face.

[1284] The "means for storing the contents of the previous conversation" is a technique for saving the contents of a previous conversation between the user and the conversation partner.

[1285] The "means for recognizing the face of the interlocutor" is a technique that the system uses to detect and identify the face of the user's interlocutor.

[1286] The "means for acquiring stored previous conversation content" is a technique for calling up and making available the saved content of a previous conversation.

[1287] The "means for acquiring the latest social media posts of the conversation partner" is a technique for collecting the latest information posted on social media by the user's conversation partner.

[1288] The "means for notifying the user of acquired information by voice" is a technique for transmitting collected information to the user by voice.

[1289] "Means for analyzing a user's facial expression for emotion recognition" refers to technology that uses cameras and sensors to analyze a user's facial expression and identify the emotion.

[1290] "Means for providing appropriate information based on emotions" refers to technology that selects and provides optimal topics and information to users in response to recognized emotions.

[1291] MODE FOR CARRYING OUT THE INVENTION

[1292] This invention is a system that uses smart glasses to recognize customer emotions in real time when a user is serving customers in a physical store, and provides appropriate information and topics. The main components of the system are smart glasses (terminals), a cloud server (server), and an emotion engine. Detailed embodiments of the system are described below.

[1293] System Configuration

[1294] 1. Terminal

[1295] The smart glasses contain the following hardware:

[1296] Camera: A device used to capture images of users and customers' faces.

[1297] Microphone: A device for capturing the voice of users and customers.

[1298] Speaker: A device that notifies the user of acquired information by voice.

[1299] Processor: A central processing unit for processing data.

[1300] Memory: Storage for temporarily storing data.

[1301] Wireless communication module: A module for communicating with the server.

[1302] 2. Server

[1303] The servers in the cloud contain the following software and databases:

[1304] Database: Manages user and customer information, past conversation history, and social media data.

[1305] Emotion engine: Algorithms that analyze facial expressions and tone of voice to recognize emotions in users and customers.

[1306] Analysis system: A system that analyzes past conversation history and social media data to extract important keywords and topics.

[1307] Program processing overview

[1308] 1. User authentication and initial setup

[1309] When a user puts on the smart glasses, facial recognition is performed using the device's built-in camera. When using the glasses for the first time, the user registers information about business partners and customers on the settings screen and links their social media accounts. The device then sends the registered information to a server and stores it.

[1310] 2. Customer awareness and information acquisition

[1311] When a user interacts with a customer in a physical store, the device uses a camera to recognize the face of the person they are talking to, and sends the recognized facial information to a server, requesting past conversation history and the latest data from social media.

[1312] 3. Emotion Recognition Processing

[1313] The device uses a built-in camera and microphone to analyze the customer's facial expressions and tone of voice, and then uses an emotion engine to recognize emotions, which are then sent to a server along with other data.

[1314] 4. Analysis and provision of information

[1315] The server analyzes past conversation history and social media data to extract important keywords and topics. Based on the emotions recognized by the emotion engine, it adjusts the way information is presented and the selection of appropriate topics. The extracted data is sent to the device in text format, and the device then provides this information to the user via the voice assistant.

[1316] Specific examples

[1317] Example 1: Smart Customer Service Assistant

[1318] When a customer service staff member wears smart glasses and serves them in a physical store, the camera recognizes the face of the customer as they enter the store. Based on the recognized facial information, the system obtains the customer's past purchase history and preferences, and then suggests appropriate products based on this. If the customer's facial expression shows "satisfaction" or "excitement" regarding a product they are interested in, the system proactively approaches them, and if the customer looks "confused" or "questioning," it provides a detailed explanation.

[1319] Prompt Sentence Examples

[1320] Describe a system that analyzes how a customer reacts in real time using smart glasses that recognize your emotions and provide the most appropriate conversational topic based on that information. For example, if the customer is smiling, the system might provide a positive conversational topic, or if the customer looks anxious, the system might provide words of encouragement. Also describe the facial and emotion recognition algorithms that enable this functionality.

[1321] This system will enable more personalized customer service in physical stores, which is expected to improve customer satisfaction.

[1322] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1323] Step 1:

[1324] The user puts on the smart glasses and turns them on. The device's built-in camera activates and captures a picture of the user's face. Using the captured facial image, a facial recognition algorithm authenticates the user. The input is the user's facial image, and the output is the authentication result. If this authentication process is successful, the smart glasses can be used.

[1325] Step 2:

[1326] When a user starts interacting with a customer in a store, the camera scans the surroundings and recognizes the face of the person they are interacting with. The input is the facial image of the customer in the store, and the output is the customer's identification information. The recognized customer information is sent to the server via the wireless communication module.

[1327] Step 3:

[1328] The server searches a database based on the received customer identification information to retrieve the customer's past conversation history and latest social media posts. The input is customer identification information, and the output is past conversation history and social media data. This data is analyzed within the server to extract important keywords and topics.

[1329] Step 4:

[1330] The device uses a built-in camera and microphone to capture the customer's facial expressions and tone of voice in real time. The input is the customer's facial image and voice data, and the output is the emotion recognition result by the emotion engine. The recognized emotion information is sent to the server.

[1331] Step 5:

[1332] The server combines and analyzes the emotion recognition results with the customer's past data to select topics and information that fit the customer's current emotions. The input is the emotion recognition results, past conversation history, and social media data, and the output is the appropriate topics and information. The results are converted into text format and sent to the device.

[1333] Step 6:

[1334] The device notifies the user of the received information via the voice assistant. The input is text information sent from the server, and the output is a voice message. The user can then proceed with the conversation with the customer based on the information received. This adaptive topic provision improves the quality of customer service and increases customer satisfaction.

[1335] By performing data processing and calculations based on input data at each step and using it as output data for the next step, a system is created in which a series of processes can be carried out seamlessly.

[1336] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1337] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1338] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1339] [Fourth embodiment]

[1340] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1341] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1342] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1343] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1344] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1345] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1346] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1347] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1348] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1349] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1350] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1351] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1352] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1353] The present invention relates to an eyeglasses-type device that efficiently acquires stored information and new information and notifies the user by voice when the user is engaged in a business negotiation or conversation. Hereinafter, embodiments and specific examples of the present invention will be described.

[1354] System Configuration

[1355] The system mainly includes the following components:

[1356] 1. Terminal

[1357] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[1358] 2. Server

[1359] It is a database and processing system that exists on the cloud and manages and analyzes information on users and business partners, as well as social media data.

[1360] Program processing overview

[1361] 1. User authentication and initial setup

[1362] The user wears the glasses-type device.

[1363] The device uses a built-in camera to perform facial recognition and authenticate the user.

[1364] When using the service for the first time, the user registers information about business partners and friends on the settings screen.

[1365] 2. Recognizing others and acquiring information

[1366] When a user is at a business meeting or dinner, the device uses the camera to recognize the face of the person they are talking to.

[1367] The device sends the recognized facial information to a server and requests data from past conversation history and social media.

[1368] 3. Analysis and provision of information

[1369] The server analyzes past conversation history and social media data to extract important keywords and topics.

[1370] The extracted data is sent to the terminal in text format.

[1371] The device provides this information to the user via a voice assistant.

[1372] Specific examples

[1373] Initial Setup

[1374] 1. The user puts on the glasses-type device and turns it on.

[1375] 2. The device uses a camera to capture the user's face and perform facial authentication.

[1376] 3. When launching the application for the first time, the user opens the application settings screen, registers photos and names of business partners and friends, and links their social media accounts.

[1377] 4. The device sends the registered information to the server and stores it.

[1378] Daily use

[1379] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the other person's face.

[1380] 2. The device sends the recognized facial information to the server and requests past conversation history and the latest social media posts.

[1381] 3. The server searches the database for the corresponding information and performs the analysis.

[1382] 4. The server summarizes the analysis results in text format and sends them to the terminal.

[1383] 5. The device uses the voice assistant to notify the user, "In our last conversation, we talked about ____" or "It seems that ____ has recently started a new project."

[1384] 6. Users can smoothly advance the conversation based on the information provided.

[1385] By implementing the present invention in this manner, users can efficiently prepare for and manage business negotiations and conversations.

[1386] The processing flow will be explained below.

[1387] Program processing steps

[1388] Initial Setup

[1389] Step 1:

[1390] The user wears the glasses-type device.

[1391] The device powers up and starts the system.

[1392] Step 2:

[1393] The device uses the built-in camera to recognize the user's face and complete user authentication.

[1394] Recognized facial data is matched with an internal profile.

[1395] Step 3:

[1396] On first use, the user opens the device's settings screen.

[1397] Users register photos and names of business partners and friends.

[1398] The user configures their social media account access settings.

[1399] Step 4:

[1400] The terminal sends the registration information to the server.

[1401] The server stores the information in a database.

[1402] During a conversation

[1403] Step 1:

[1404] The user initiates a conversation.

[1405] The device's built-in camera captures the face of the person you're talking to and performs facial recognition.

[1406] Step 2:

[1407] The device sends the recognized face data to the server.

[1408] The device requests past conversation history and the latest social media posts.

[1409] Step 3:

[1410] The server searches the database for the corresponding conversation history and social media data.

[1411] The server extracts the relevant information and performs analysis.

[1412] Step 4:

[1413] The server compiles the analysis results in text format and sends them to the terminal.

[1414] The information includes keywords from past conversations and the latest social media posts.

[1415] Step 5:

[1416] The device notifies the user of the received information through the voice assistant.

[1417] The audio will provide information such as, "In our last conversation, we talked about XX," or "X recently started a new project."

[1418] Step 6:

[1419] The user can proceed with the conversation based on the notified information.

[1420] Users can use the provided topics to smoothly conduct business negotiations and dinner meetings.

[1421] By using the above specific processing steps, the present invention can improve the efficiency of preparation and management in business negotiations and conversation scenes, and increase convenience for users.

[1422] Example 1

[1423] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1424] In the past, when preparing for and conducting business negotiations or conversations, it was difficult for users to efficiently obtain the content of past conversations or the latest information on the other party. Furthermore, manually managing a large amount of information placed a heavy burden on users. The present invention aims to solve these problems and provide a system that allows users to quickly and efficiently obtain the information they need for business negotiations or conversations.

[1425] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1426] In this invention, the server includes means for transmitting face information to an external database system and requesting past conversation history and related information, means for analyzing the past conversation history and online information to extract important keywords and topics, and means for notifying the user of information using a voice assistant, thereby enabling the user to quickly and efficiently obtain information necessary for business negotiations and conversations, and to proceed smoothly.

[1427] "Means for recognizing the user's face" refers to the device's built-in camera and facial recognition software that identifies and authenticates the user's face.

[1428] The "means for storing the contents of the previous conversation" is a function that stores the contents of a conversation that has been held in the past and makes it possible to retrieve the contents again as needed.

[1429] "Means for recognizing the face of the person being spoken to" refers to the device's ability to use a camera and facial recognition software to identify the face of the person with whom the user is speaking.

[1430] The "means for acquiring the stored previous conversation content" is a function for searching and acquiring the saved past conversation content again.

[1431] "Means for obtaining recent posts from the public online information of the interlocutor" refers to a function for obtaining recent posts from the interlocutor's social media or other online platforms.

[1432] The "means for notifying the user of the acquired information by voice" is a function for notifying the user of the acquired information by voice using a voice assistant.

[1433] "Means for sending facial information to an external database system and requesting past conversation history and related information" refers to a function that sends recognized facial information to an external database such as a cloud server and requests past conversation history and related information.

[1434] "Means of extracting important keywords and topics by analyzing past conversation history and online information" refers to a function that uses natural language processing, etc. to identify and extract important keywords and topics based on saved conversation history and acquired online information.

[1435] A "voice assistant" is software that responds to voice instructions from a user and provides information.

[1436] The present invention relates to a glasses-type device that efficiently acquires stored information and new information when a user is engaged in business negotiations or conversations, and notifies the user of the information by voice.

[1437] System Configuration

[1438] The system includes the following components:

[1439] 1. Terminal: The main body of the glasses-type device, which contains a camera, microphone, speaker, processor, memory, and wireless communication module.

[1440] 2. Server: A database and processing system that exists on the cloud and manages and analyzes information on users, business partners, and social media data.

[1441] Program processing overview

[1442] The present invention provides support for users to smoothly conduct business negotiations and conversations through the following program processing.

[1443] 1. User authentication and initialization:

[1444] The user puts on the glasses-type device and turns it on.

[1445] The device takes a picture of the user's face with the built-in camera, performs facial recognition, and authenticates the user. Facial recognition uses technologies such as OpenCV.

[1446] When using the app for the first time, users register photos and names of business partners or friends on the device's application settings screen and link their social media accounts.

[1447] This information is sent from the terminal to the server and stored.

[1448] 2. Recognizing others and obtaining information:

[1449] When a user begins a business negotiation or conversation, the device activates its built-in camera and recognizes the other person's face.

[1450] The device sends the recognized facial information to a server and requests information from past conversation history and social media.

[1451] 3. Analysis and provision of information:

[1452] The server searches and retrieves past conversation history and the latest information from social media from a database, and analyzes it using natural language processing technology (e.g., spaCy).

[1453] As a result of the analysis, important keywords and topics are extracted and sent to the device in text format.

[1454] The device will use a voice assistant such as Google Cloud Text-to-Speech to notify the user via voice.

[1455] Specific examples

[1456] 1. Initial Setup:

[1457] The user puts on the glasses-type device and turns it on. The device then performs facial recognition using a camera.

[1458] Users register information about business partners and friends on the application's settings screen and link their social media accounts.

[1459] The registered information is sent to the server and stored.

[1460] 2. Daily use:

[1461] When a user starts a business negotiation or conversation, the device recognizes the other person's face using the built-in camera and sends that facial information to the server.

[1462] The server searches the database for past conversation history and social media information and analyzes it.

[1463] The analysis results are sent to the device in text format, and the device uses a voice assistant to notify the user, saying things like, "In our last conversation, we talked about XX," or "It seems like XX has recently started a new project."

[1464] Prompt Sentence Examples

[1465] "Know who your next sales call is and show them their latest social media posts."

[1466] "Please extract information from past conversation history that is appropriate for the next topic and tell me."

[1467] In this way, the system of the present invention provides the user with the necessary information quickly and efficiently, and supports smooth progress of business negotiations and conversations.

[1468] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1469] Step 1:

[1470] User authentication and initialization

[1471] The user puts on the glasses-type device and turns it on.

[1472] Input: Video of the user's face.

[1473] The device captures a picture of the user's face with its built-in camera and uses facial recognition software (e.g., OpenCV) to authenticate the user.

[1474] Output: Authentication complete flag.

[1475] How it works: If authentication is successful, the initial setup process continues: On first use, the user opens the application setup screen on the device.

[1476] Step 2:

[1477] Information registration and linking

[1478] Users operate the application to register photos and names of business partners and friends and link their social media accounts.

[1479] Input: Contact and friend information, social media accounts.

[1480] The device collects this information and sends it to a server in the cloud.

[1481] Output: The registered information is saved on the server.

[1482] How it works: The server stores the received information in a database.

[1483] Step 3:

[1484] Recognizing others and acquiring information

[1485] When a user starts a business meeting or conversation, the device activates the camera and scans the face of the person they are talking to.

[1486] Input: Video of the person you're talking to.

[1487] The device uses facial recognition software to recognize the face of the person you are talking to and sends the facial information to a server.

[1488] Output: ID of the person whose face was recognized.

[1489] What it does: Requests past conversation history and social media data from the server.

[1490] Step 4:

[1491] Searching and retrieving data

[1492] The server searches and retrieves from the database the past conversation history and the latest online postings of the corresponding user or business partner.

[1493] Input: facial recognition data, database query.

[1494] Output: Past conversation history and social media posts.

[1495] Operation: The server retrieves these data and sends them to the device.

[1496] Step 5:

[1497] Analysis of information

[1498] The server analyzes the acquired data and extracts important keywords and topics using natural language processing technology (e.g., spaCy).

[1499] Input: Past conversation history and social media posts.

[1500] Output: Extracted important keywords and topics.

[1501] Operation: The server sends the analysis results to the terminal in text format.

[1502] Step 6:

[1503] Execute voice notification

[1504] The device uses voice assistants such as Google Cloud Text-to-Speech to provide information to the user via voice.

[1505] Input: Analysis results in text format.

[1506] Output: Audio notification.

[1507] What it does: The device notifies the user, "In our last conversation, we talked about ____" or "It seems that ____ has recently started a new project."

[1508] (Application example 1)

[1509] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1510] Systems already exist that use facial recognition technology and social media information to assist in business negotiations and conversations, but applying this to security services would enable patrolling security personnel to efficiently obtain information on the status of facilities and past security incidents in real time.However, current systems do not adequately link real-time facial recognition with the acquisition of past data, making it difficult for security personnel to respond immediately.

[1511] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1512] In this invention, the server includes means for recognizing a user's face, means for storing the content of a previous conversation, means for recognizing the face of the person being talked to, means for acquiring the stored content of the previous conversation, means for acquiring the content of the person being talked to's recent social media posts, means for notifying the user of the acquired information by voice, means for recognizing faces during patrol and acquiring information about past security incidents, and means for notifying the user of the acquired past security information by voice. This enables security personnel to effectively acquire information about the status of a facility and past security incidents in real time during patrol, enabling them to respond quickly.

[1513] "User" means a person who uses the system.

[1514] A "means for recognizing a face" is a device or method that uses a camera or software to detect a person's face and analyze its features.

[1515] The "means for storing the contents of the previous conversation" is a method or device for storing the contents of a previous conversation in a storage device.

[1516] The "means for recognizing the face of the person being spoken to" refers to a device or method for detecting the face of the person being spoken to during the current conversation and analyzing its features.

[1517] The "means for acquiring the stored content of the previous conversation" is a method or device for retrieving the content of the previous conversation from a storage device.

[1518] A "means for obtaining recent social media posts" is a device or method for automatically obtaining the latest social media posts of a target person.

[1519] The "means for notifying the user of the acquired information by voice" refers to a device or method for conveying the acquired information to the user using voice synthesis technology.

[1520] "Means for recognizing faces during patrol and obtaining information on past security incidents" refers to a device or method for recognizing the faces of people seen during patrol and obtaining information on security incidents in which the people have been involved in the past.

[1521] The "means for notifying the user of acquired past security information by voice" refers to a device or method for conveying acquired security-related information to the user using voice synthesis technology.

[1522] The system of the present invention has the function of recognizing faces in real time while the user is on patrol, acquiring information about past security incidents, and notifying the user of the information by voice. Hereinafter, an embodiment of the present invention will be described.

[1523] System Configuration

[1524] The system mainly includes the following components:

[1525] 1. Terminal

[1526] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[1527] 2. Server

[1528] It is a database and processing system that exists on the cloud and manages and analyzes information about users and their conversation partners.

[1529] Program processing overview

[1530] 1. User Authentication

[1531] The device uses a built-in camera to perform facial recognition and authenticate the user.

[1532] 2. Facial Recognition and Information Acquisition

[1533] The device uses a camera to recognize the faces of people it sees while patrolling.

[1534] The device sends the recognized facial information to a server and requests data on past security incidents.

[1535] 3. Analysis and provision of information

[1536] The server searches the database for the corresponding information and performs the analysis.

[1537] The server compiles the analysis results in text format and sends them to the terminal.

[1538] The device provides this information to the user via a voice assistant.

[1539] Technology used

[1540] Hardware:

[1541] Glasses-type device (camera, microphone, speaker, processor)

[1542] software:

[1543] OpenCV: Camera image acquisition and image processing

[1544] face_recognition: Face recognition library

[1545] speech_recognition: Speech recognition library

[1546] pyttsx3: Text-to-speech library

[1547] requests: Communicating with the server

[1548] Specific examples

[1549] For example, if a user spots a suspicious person in a blind spot while patrolling a facility, the system will immediately recognize the person and provide a voice notification with information about their involvement in past security incidents, allowing the user to take immediate action.

[1550] Prompt Sentence Examples

[1551] Design a security assistant application that uses a glasses-type device to obtain real-time information about the facility's status and past security incidents while the user is patrolling, and provides voice notifications.

[1552] 1. User authentication through facial recognition.

[1553] 2. Camera-based facial recognition during patrols.

[1554] 3. Send your facial image to the server and request past security information.

[1555] 4. Voice notification of retrieved information.

[1556] Design your application to include the above functions and generate the appropriate program code.

[1557] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1558] Step 1:

[1559] When a user wears the glasses-type device, the device uses the built-in camera to perform facial recognition and authenticate the user. The input is the camera image and the output is the authentication result. Authentication is performed using OpenCV and the face_recognition library.

[1560] Step 2:

[1561] While the user is patrolling the facility, the device continuously uses the camera to recognize the faces of people around them. The input is the surrounding image, and the output is the face recognition result. The face_recognition library is used for face recognition.

[1562] Step 3:

[1563] The device sends the recognized facial information to the server and requests data about past security incidents. The input is the recognized facial feature data, and the output is a request to the server. The requests library is used to send the data.

[1564] Step 4:

[1565] The server searches a database based on the received facial information to obtain information on related security incidents. The input is facial feature data, and the output is past security information. The search and analysis are performed using a database system within the server.

[1566] Step 5:

[1567] The server compiles the analysis results in text format and sends them to the terminal. The input is the analyzed security information, and the output is text format data. The requests library is used to send the data.

[1568] Step 6:

[1569] The device notifies the user of the received security information using a voice assistant. The input is text-formatted security information, and the output is voice. The voice notification uses the pyttsx3 library.

[1570] Step 7:

[1571] The user takes the necessary action based on the provided information. The input is the security information notified by voice, and the output is the user's response. This allows for a quick response.

[1572] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1573] The present invention relates to a glasses-type device that uses an emotion engine to recognize emotions when a user is engaged in a business negotiation or conversation, and further adjusts the information provision method and topic selection based on the emotion recognition information. The following describes embodiments of the present invention and specific examples thereof.

[1574] System Configuration

[1575] The system mainly includes the following components:

[1576] 1. Terminal

[1577] This is the main body of the glasses-type device, which has a built-in camera, microphone, speaker, processor, memory, and wireless communication module.

[1578] 2. Server

[1579] It is a database and processing system that exists on the cloud and manages and analyzes information on users and business partners, as well as social media data.

[1580] 3. Emotion Engine

[1581] The system uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, recognizing emotions in real time.

[1582] Program processing overview

[1583] 1. User authentication and initial setup

[1584] The user wears the glasses-type device.

[1585] The device uses a built-in camera to perform facial recognition and authenticate the user.

[1586] When using the service for the first time, the user registers information about business partners and friends on the settings screen.

[1587] 2. Recognizing others and acquiring information

[1588] When a user is at a business meeting or dinner, the device uses the camera to recognize the face of the person they are talking to.

[1589] The device sends the recognized facial information to a server and requests data from past conversation history and social media.

[1590] 3. Emotion Recognition Processing

[1591] The device uses a built-in camera and microphone to analyze the user's facial expressions and tone of voice, and uses an emotion engine to recognize emotions.

[1592] The recognized emotion information is sent to a server along with other data.

[1593] 4. Analysis and provision of information

[1594] The server analyzes past conversation history and social media data to extract important keywords and topics.

[1595] Based on the emotions recognized by the emotion engine, the way information is presented and the selection of appropriate topics are adjusted.

[1596] The extracted data is sent to the terminal in text format.

[1597] The device provides this information to the user via a voice assistant.

[1598] Specific examples

[1599] Initial Setup

[1600] 1. The user puts on the glasses-type device and turns it on.

[1601] 2. The device uses a camera to capture the user's face and perform facial authentication.

[1602] 3. When launching the application for the first time, the user opens the application settings screen, registers photos and names of business partners and friends, and links their social media accounts.

[1603] 4. The device sends the registered information to the server and stores it.

[1604] Daily use

[1605] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the other person's face.

[1606] 2. The device sends the recognized facial data to the server and requests past conversation history and the latest social media posts.

[1607] 3. The device uses a camera and microphone to recognize the user's emotions in real time and transmits the emotion data to the server.

[1608] 4. The server searches the database for the corresponding information and performs analysis. The analysis results include keywords from past conversations, the latest social media posts, and topics that take into account the user's emotions.

[1609] 5. The server summarizes the analysis results in text format and sends them to the terminal.

[1610] 6. The device uses a voice assistant to notify the user, saying things like, "In our last conversation, we talked about XX," or "X recently started a new project," providing appropriate topics that reflect the user's emotions.

[1611] 7. The user can then proceed with the conversation based on the information provided. The topics and information provided adapt to the user's emotions, allowing for smooth business negotiations and dinner meetings.

[1612] Through the above-described specific processing steps, the present invention enables users to conduct business negotiations and conversations efficiently and smoothly, and in particular, provides adaptive information that takes emotions into consideration.

[1613] The processing flow will be explained below.

[1614] The processing steps of the program (if it includes an emotion engine that recognizes the user's emotions)

[1615] Initial Setup

[1616] Step 1:

[1617] The user wears the glasses-type device.

[1618] The device powers up and starts the system.

[1619] Step 2:

[1620] The device uses the built-in camera to recognize the user's face and complete user authentication.

[1621] Recognized facial data is matched with an internal profile.

[1622] Step 3:

[1623] On first use, the user opens the device's settings screen.

[1624] Users register photos and names of business partners and friends.

[1625] The user configures their social media account access settings.

[1626] Step 4:

[1627] The terminal sends the registration information to the server.

[1628] The server stores the information in a database.

[1629] During a conversation

[1630] Step 1:

[1631] The user initiates a conversation.

[1632] The device's built-in camera captures the face of the person you're talking to and performs facial recognition.

[1633] Step 2:

[1634] The device sends the recognized face data to the server.

[1635] The device requests past conversation history and the latest social media posts.

[1636] Step 3:

[1637] The device uses its built-in camera and microphone to analyze the user's facial expressions and tone of voice, and uses an emotion engine to recognize real-time emotions.

[1638] The recognized emotion data is sent to a server along with other data.

[1639] Step 4:

[1640] The server searches the database for the corresponding conversation history and social media data.

[1641] The server analyzes the relevant information and identifies important keywords and topics.

[1642] Step 5:

[1643] The server adjusts the method and content of the information it provides based on the user's emotional information.

[1644] If the sentiment is negative, adjust it to add supportive information or encouraging messages.

[1645] Step 6:

[1646] The server compiles the analysis results in text format and sends them to the terminal.

[1647] The information includes keywords from past conversations, recent social media posts, and relevant topics based on emotional information.

[1648] Step 7:

[1649] The device notifies the user of the received information through the voice assistant.

[1650] The audio will provide information such as, "In our last conversation, we talked about XX," or "X recently started a new project."

[1651] Depending on the user's emotions, it suggests appropriate topics such as "How about talking about ○○ to relax?"

[1652] Step 8:

[1653] The user can proceed with the conversation based on the notified information.

[1654] The topics and information provided are adjusted to match the user's emotions, making business negotiations and dinner meetings go more smoothly.

[1655] By using the above specific processing steps, the present invention enables the user to efficiently acquire information in business negotiations and conversation scenes, and provides information adaptively according to emotions.

[1656] Example 2

[1657] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1658] Conventional conversation assistance systems provide information without considering the emotions of the user or the person they are talking to, making it difficult to select the right topic at the right time. Furthermore, they lack the functionality to efficiently collect and analyze past conversation content and social media information to provide useful information. This reduces the user's ability to adapt to smoothly progress business negotiations and conversations.

[1659] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1660] In this invention, the server includes means for recognizing the user's face, means for storing the content of the previous conversation, means for recognizing the face of the conversation partner, means for acquiring the stored content of the previous conversation, means for acquiring the content of the conversation partner's recent social media posts, means for recognizing the user's emotions in real time, means for analyzing the acquired information and the recognized emotion information to identify related topics, and means for notifying the user of the acquired information by voice. This allows the user to receive appropriate information based on the emotion of the conversation partner, thereby enabling business negotiations and conversations to proceed more smoothly.

[1661] A "user" is an individual who uses the system to conduct business or conversations.

[1662] "Facial recognition means" refers to technology that captures facial images with a camera and identifies specific people from those images.

[1663] "Means for storing the content of previous conversations" refers to technology that has the function of saving the content of past conversations in a database.

[1664] "Means for recognizing the face of the person being conversed with" refers to technology that uses a camera to capture and identify the face image of the person being conversed with the user.

[1665] "Means for retrieving stored content of previous conversations" refers to technology that has the function of searching and retrieving past conversation data stored in a database.

[1666] "Means of obtaining recent social media posts" refers to technology that has the function of collecting the latest post data from social media such as SNS.

[1667] "Means for recognizing emotions in real time" refers to technology that uses a camera or microphone to analyze a user's facial expressions and tone of voice to identify emotions in real time.

[1668] "Means for identifying relevant topics by analyzing acquired information and recognized emotional information" refers to technology that automatically selects appropriate topics for conversation based on collected data and emotional information.

[1669] "Means of notifying the user of acquired information by voice" refers to technology that uses voice assistants or other means to notify the user of analysis results and related information.

[1670] This invention relates to a system that uses an emotion engine to recognize emotions when a user is engaged in business negotiations or conversations, and then adjusts the information provision method and topic selection based on that information. This system is composed of a glasses-type device (hereinafter referred to as the "terminal") and a cloud server (hereinafter referred to as the "server").

[1671] System Components

[1672] 1. Terminal

[1673] The device is a glasses-type device that incorporates a camera, microphone, speaker, processor, memory, wireless communication module, etc. This device performs facial recognition, emotion recognition, and acts as a voice assistant for the user.

[1674] 2. Server

[1675] The server is located in the cloud and functions as a database and data analysis system, managing and analyzing information about users and their conversation partners, social media data, and past conversation history.

[1676] 3. Emotion Engine

[1677] The emotion engine analyzes the user's facial expressions and tone of voice through the built-in camera and microphone, recognizing emotions in real time.

[1678] Program processing

[1679] This system performs the following main processes:

[1680] 1. Initial Setup and User Authentication

[1681] The user puts on the glasses-type device and turns it on. The device uses the built-in camera to capture the user's face and activates a facial recognition system (e.g., OpenCV or TensorFlow) to authenticate the user. If authentication is successful, the user can register information about business partners and friends on the settings screen. The device then sends this information to the server and stores it in a database.

[1682] 2. Recognizing others and acquiring information

[1683] When a business meeting or conversation begins, the device recognizes the face of the person it is talking to using its built-in camera. The recognized facial data is then sent to the server, which then searches past conversation history and social media data to obtain the necessary information. This communication uses the HTTP or HTTPS protocol.

[1684] 3. Emotion Recognition and Information Transmission

[1685] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine (e.g., Google Cloud Vision API or IBM Watson Tone Analyzer). The recognized emotion data is then sent to a server.

[1686] 4. Analysis and provision of information

[1687] The server analyzes the received emotional data, past conversation history, and social media data. NLP technology is used in the analysis to extract important keywords and topics. The analysis results are compiled in text format and sent to the device. The device then uses Google Assistant, Amazon Alexa, or other voice recognition to notify the user.

[1688] Specific examples

[1689] Example of initial settings

[1690] 1. The user puts on the glasses-type device and presses the power button to start it up.

[1691] 2. The device uses the built-in camera to capture the user's face and processes the facial image in real time.

[1692] 3. The device will use the OpenCV library to perform facial recognition, and if successful, will notify you by voice message saying "Authentication successful."

[1693] 4. The user opens the application's settings screen, enters information about business partners and friends, and connects their social media accounts. This information is sent from the device to the server and stored.

[1694] Examples of everyday use

[1695] 1. When a user begins a business negotiation or conversation, the device's built-in camera recognizes the face of the person they are talking to.

[1696] 2. The device sends the recognized facial data to the server, which then retrieves past conversation history and the latest social media posts from a database.

[1697] 3. The device uses a camera and microphone to recognize the user's emotions in real time and transmits the data to the server.

[1698] 4. The server uses natural language processing technology to analyze and select appropriate topics and information.

[1699] 5. The device uses Google Assistant to notify the user, "In our last conversation, we talked about ____" or "____ recently started a new project."

[1700] 6. Users can advance the conversation based on the information provided.

[1701] Examples of prompt statements

[1702] "Please explain how a user can recognize the emotions of the person they are talking to and offer appropriate topics of conversation when conducting business using a glasses-type device."

[1703] This system allows users to conduct business negotiations and conversations efficiently and smoothly, and provides adaptive information based on emotions.

[1704] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1705] Step 1:

[1706] The user puts on the glasses-type device and turns it on. The device captures the user's face using the built-in camera. It takes the facial image as input and starts a facial recognition system (e.g., OpenCV or TensorFlow) to authenticate the user. If facial recognition is successful, it outputs a message indicating successful authentication. Specifically, the camera takes a picture of the user's face, the image is sent to the processor, and a deep learning model performs facial recognition.

[1707] Step 2:

[1708] The user registers information about business partners and friends on a setup screen. The information entered on the setup screen includes a photo, name, and social media accounts. The device sends this information to the server and stores it in a database. An output confirmation message is displayed to confirm that the input data has been saved in the database. Specifically, the device formats the data to be sent and sends it to the server using an HTTP request.

[1709] Step 3:

[1710] When a user initiates a business meeting or conversation, the device uses its built-in camera to recognize the face of the person it is talking to. The captured facial image is sent as input to a server. The server then analyzes past conversation history and social media information based on this facial data to obtain the necessary data. The analysis results are sent as output to the device. Specifically, the server runs a facial recognition algorithm to search a database and extract relevant information.

[1711] Step 4:

[1712] The device uses a built-in camera and microphone to collect the user's facial expressions and tone of voice in real time. This data is sent to an emotion engine (e.g., Google Cloud Vision API or IBM Watson Tone Analyzer) for analysis. The input data is facial expression and tone of voice information, and the output data is recognized emotion information. Specifically, facial images captured by the camera and audio recorded by the microphone are sent to a processor and analyzed by the emotion recognition engine.

[1713] Step 5:

[1714] The recognized emotion data is sent to a server. The input data includes emotion information, and the server performs further data analysis based on this. The analysis results are sent to the device, and the output includes appropriate topics and keywords. Specifically, the server compares the received emotion data with a database and uses natural language processing technology to extract important keywords and topics.

[1715] Step 6:

[1716] Based on the received data, the device notifies the user using a voice assistant (e.g., Google Assistant or Amazon Alexa). The input includes the analysis results and appropriate topic information, and the output is a voice notification. Specifically, the device converts the analysis results into a voice format and notifies the user, for example, "In our last conversation, we talked about XX," or "X recently started a new project."

[1717] These steps realize a system that supports users in efficiently and smoothly conducting business negotiations and conversations.

[1718] (Application example 2)

[1719] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1720] Conventional customer service systems provide uniform responses without considering customer emotions, making it difficult to improve customer satisfaction or provide appropriate responses. While some systems utilize conversation content and social media information, they are unable to obtain customer emotional information in real time and dynamically provide appropriate information based on this information. Therefore, there was a need for a system that enables individual responses based on customer emotions and supports smoother communication.

[1721] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the user's face, means for storing the content of the previous conversation, means for recognizing the face of the conversation partner, means for acquiring the stored content of the previous conversation, means for acquiring the content of the conversation partner's recent social media posts, means for notifying the user of the acquired information by voice, means for analyzing the user's facial expression to recognize emotions, and means for providing appropriate information based on the emotions. This makes it possible to provide appropriate information and select topics according to the customer's emotions, thereby improving the quality of customer service.

[1722] "Means for recognizing a user's face" refers to the technology that the system uses to detect and identify the user's face.

[1723] The "means for storing the contents of the previous conversation" is a technique for saving the contents of a previous conversation between the user and the conversation partner.

[1724] The "means for recognizing the face of the interlocutor" is a technique that the system uses to detect and identify the face of the user's interlocutor.

[1725] The "means for acquiring stored previous conversation content" is a technique for calling up and making available the saved content of a previous conversation.

[1726] The "means for acquiring the latest social media posts of the conversation partner" is a technique for collecting the latest information posted on social media by the user's conversation partner.

[1727] The "means for notifying the user of acquired information by voice" is a technique for transmitting collected information to the user by voice.

[1728] "Means for analyzing a user's facial expression for emotion recognition" refers to technology that uses cameras and sensors to analyze a user's facial expression and identify the emotion.

[1729] "Means for providing appropriate information based on emotions" refers to technology that selects and provides optimal topics and information to users in response to recognized emotions.

[1730] MODE FOR CARRYING OUT THE INVENTION

[1731] This invention is a system that uses smart glasses to recognize customer emotions in real time when a user is serving customers in a physical store, and provides appropriate information and topics. The main components of the system are smart glasses (terminals), a cloud server (server), and an emotion engine. Detailed embodiments of the system are described below.

[1732] System Configuration

[1733] 1. Terminal

[1734] The smart glasses contain the following hardware:

[1735] Camera: A device used to capture images of users and customers' faces.

[1736] Microphone: A device for capturing the voice of users and customers.

[1737] Speaker: A device that notifies the user of acquired information by voice.

[1738] Processor: A central processing unit for processing data.

[1739] Memory: Storage for temporarily storing data.

[1740] Wireless communication module: A module for communicating with the server.

[1741] 2. Server

[1742] The servers in the cloud contain the following software and databases:

[1743] Database: Manages user and customer information, past conversation history, and social media data.

[1744] Emotion engine: Algorithms that analyze facial expressions and tone of voice to recognize emotions in users and customers.

[1745] Analysis system: A system that analyzes past conversation history and social media data to extract important keywords and topics.

[1746] Program processing overview

[1747] 1. User authentication and initial setup

[1748] When a user puts on the smart glasses, facial recognition is performed using the device's built-in camera. When using the glasses for the first time, the user registers information about business partners and customers on the settings screen and links their social media accounts. The device then sends the registered information to a server and stores it.

[1749] 2. Customer awareness and information acquisition

[1750] When a user interacts with a customer in a physical store, the device uses a camera to recognize the face of the person they are talking to, and sends the recognized facial information to a server, requesting past conversation history and the latest data from social media.

[1751] 3. Emotion Recognition Processing

[1752] The device uses a built-in camera and microphone to analyze the customer's facial expressions and tone of voice, and then uses an emotion engine to recognize emotions, which are then sent to a server along with other data.

[1753] 4. Analysis and provision of information

[1754] The server analyzes past conversation history and social media data to extract important keywords and topics. Based on the emotions recognized by the emotion engine, it adjusts the way information is presented and the selection of appropriate topics. The extracted data is sent to the device in text format, and the device then provides this information to the user via the voice assistant.

[1755] Specific examples

[1756] Example 1: Smart Customer Service Assistant

[1757] When a customer service staff member wears smart glasses and serves them in a physical store, the camera recognizes the face of the customer as they enter the store. Based on the recognized facial information, the system obtains the customer's past purchase history and preferences, and then suggests appropriate products based on this. If the customer's facial expression shows "satisfaction" or "excitement" regarding a product they are interested in, the system proactively approaches them, and if the customer looks "confused" or "questioning," it provides a detailed explanation.

[1758] Prompt Sentence Examples

[1759] Describe a system that analyzes how a customer reacts in real time using smart glasses that recognize your emotions and provide the most appropriate conversational topic based on that information. For example, if the customer is smiling, the system might provide a positive conversational topic, or if the customer looks anxious, the system might provide words of encouragement. Also describe the facial and emotion recognition algorithms that enable this functionality.

[1760] This system will enable more personalized customer service in physical stores, which is expected to improve customer satisfaction.

[1761] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1762] Step 1:

[1763] The user puts on the smart glasses and turns them on. The device's built-in camera activates and captures a picture of the user's face. Using the captured facial image, a facial recognition algorithm authenticates the user. The input is the user's facial image, and the output is the authentication result. If this authentication process is successful, the smart glasses can be used.

[1764] Step 2:

[1765] When a user starts interacting with a customer in a store, the camera scans the surroundings and recognizes the face of the person they are interacting with. The input is the facial image of the customer in the store, and the output is the customer's identification information. The recognized customer information is sent to the server via the wireless communication module.

[1766] Step 3:

[1767] The server searches a database based on the received customer identification information to retrieve the customer's past conversation history and latest social media posts. The input is customer identification information, and the output is past conversation history and social media data. This data is analyzed within the server to extract important keywords and topics.

[1768] Step 4:

[1769] The device uses a built-in camera and microphone to capture the customer's facial expressions and tone of voice in real time. The input is the customer's facial image and voice data, and the output is the emotion recognition result by the emotion engine. The recognized emotion information is sent to the server.

[1770] Step 5:

[1771] The server combines and analyzes the emotion recognition results with the customer's past data to select topics and information that fit the customer's current emotions. The input is the emotion recognition results, past conversation history, and social media data, and the output is the appropriate topics and information. The results are converted into text format and sent to the device.

[1772] Step 6:

[1773] The device notifies the user of the received information via the voice assistant. The input is text information sent from the server, and the output is a voice message. The user can then proceed with the conversation with the customer based on the information received. This adaptive topic provision improves the quality of customer service and increases customer satisfaction.

[1774] By performing data processing and calculations based on input data at each step and using it as output data for the next step, a system is created in which a series of processes can be carried out seamlessly.

[1775] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1776] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1777] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1778] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1779] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1780] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1781] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1782] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1783] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1784] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1785] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1786] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1787] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1788] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1789] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1790] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1791] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1792] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1793] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1794] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1795] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1796] The following is further disclosed regarding the above embodiment.

[1797] (Claim 1)

[1798] means for recognizing a user's face;

[1799] A means for remembering the contents of previous conversations;

[1800] A means of recognizing the face of the person you are speaking to;

[1801] A means for retrieving the stored previous conversation content;

[1802] A means of obtaining the latest social media posts of the person you are speaking with;

[1803] means for notifying the user of the acquired information by voice;

[1804] A system including:

[1805] (Claim 2)

[1806] 2. The system according to claim 1, further comprising means for analyzing the previous conversation content to identify important keywords.

[1807] (Claim 3)

[1808] 10. The system of claim 1, further comprising means for analyzing social media posts to identify related topics.

[1809] (Claim 4)

[1810] A means for storing the face photos and names of business partners and friends that the user has registered in advance;

[1811] 10. The system of claim 1, further comprising means for linking social media account information.

[1812] (Claim 5)

[1813] 10. The system of claim 1, wherein the system is a glasses-type device that includes facial recognition and audio notifications.

[1814] "Example 1"

[1815] (Claim 1)

[1816] means for recognizing a user's face;

[1817] A means for remembering the contents of previous conversations;

[1818] A means of recognizing the face of the person you are speaking to;

[1819] A means for retrieving the stored previous conversation content;

[1820] a means for obtaining recent public online postings of the person being spoken to;

[1821] means for notifying the user of the acquired information by voice;

[1822] means for transmitting the facial information to an external database system and requesting past conversation history and related information;

[1823] A means of extracting important keywords and topics by analyzing past conversation history and online information;

[1824] A system including:

[1825] (Claim 2)

[1826] 10. The system of claim 1, further comprising means for analyzing the conversation content and online information to identify important keywords and related topics.

[1827] (Claim 3)

[1828] 10. The system of claim 1, further comprising means for notifying the user of information using a voice assistant.

[1829] "Application Example 1"

[1830] (Claim 1)

[1831] means for recognizing a user's face;

[1832] A means for remembering the contents of previous conversations;

[1833] A means of recognizing the face of the person you are speaking to;

[1834] A means for retrieving the stored previous conversation content;

[1835] A means of obtaining the latest social media posts of the person you are speaking with;

[1836] means for notifying the user of the acquired information by voice;

[1837] A means of recognizing faces while patrolling and obtaining information about past security incidents;

[1838] means for notifying the user of the acquired past security information by voice;

[1839] A system including:

[1840] (Claim 2)

[1841] 2. The system according to claim 1, further comprising means for analyzing the previous conversation content to identify important keywords.

[1842] (Claim 3)

[1843] 10. The system of claim 1, further comprising means for analyzing social media posts to identify related topics.

[1844] "Example 2: Combining Emotion Engines"

[1845] (Claim 1)

[1846] means for recognizing a user's face;

[1847] A means for remembering the contents of previous conversations;

[1848] A means of recognizing the face of the person you are speaking to;

[1849] A means for retrieving the stored previous conversation content;

[1850] A means of obtaining the latest social media posts of the person you are speaking with;

[1851] means for recognizing user emotions in real time;

[1852] means for analyzing the acquired information and the recognized emotion information to identify related topics;

[1853] means for notifying the user of the acquired information by voice;

[1854] A system including:

[1855] (Claim 2)

[1856] 2. The system according to claim 1, further comprising means for analyzing the previous conversation content to identify important keywords.

[1857] (Claim 3)

[1858] 10. The system of claim 1, further comprising means for analyzing the social media posts and the recognized sentiment information to identify related topics.

[1859] "Application example 2 when combining emotion engines"

[1860] (Claim 1)

[1861] means for recognizing a user's face;

[1862] A means for remembering the contents of previous conversations;

[1863] A means of recognizing the face of the person you are speaking to;

[1864] A means for retrieving the stored previous conversation content;

[1865] A means of obtaining the latest social media posts of the person you are speaking with;

[1866] means for notifying the user of the acquired information by voice;

[1867] means for analyzing a user's facial expression for emotion recognition;

[1868] A means of providing appropriate information based on emotions;

[1869] A system including:

[1870] (Claim 2)

[1871] 2. The system according to claim 1, further comprising means for analyzing the previous conversation content to identify important keywords.

[1872] (Claim 3)

[1873] 10. The system of claim 1, further comprising means for analyzing social media posts to identify related topics. [Explanation of symbols]

[1874] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for recognizing a user's face; A means for remembering the contents of previous conversations; A means of recognizing the face of the person you are speaking to; A means for retrieving the stored previous conversation content; A means of obtaining the latest social media posts of the person you are speaking with; means for notifying the user of the acquired information by voice; A system including:

2. 2. The system according to claim 1, further comprising means for analyzing the previous conversation content to identify important keywords.

3. The system of claim 1 , further comprising means for analyzing social media posts to identify related topics.

4. A means for storing the face photos and names of business partners and friends that the user has registered in advance; The system of claim 1 , further comprising means for linking social media account information.

5. 10. The system of claim 1, wherein the system is a glasses-type device that includes facial recognition and audio notifications.

Images