system

Abstract

Provide a system. [Solution] means for obtaining facial recognition data of a user; means for transmitting the acquired facial recognition data to a server; A means for matching the facial recognition data with a database on a server, searching for matching profile information, and transmitting the matching profile information to the terminal; means for providing the profile information sent from the server to the user; A means for collecting voice data and extracting keywords in real time; A means for obtaining related conversation history and common interests from the server based on the extracted keywords; means for notifying the user of the acquired information; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, interactions with a variety of people are on the rise, making it difficult to grasp the content of individual conversations and the preferences of others. Furthermore, in business and social situations, smooth communication requires the ability to instantly recall the content of past conversations and the interests of others. However, doing this manually takes time and effort and can be inaccurate. Furthermore, from the perspective of privacy protection, it is necessary to manage information in a way that prevents the leakage of personal information. [Means for solving the problem]

[0005] The present invention provides a means for acquiring a user's facial recognition data and transmitting the data to a server. The server then compares the data with a database to search for matching profile information and transmits it to a terminal. The terminal then provides the user with the provided profile information, past conversation history, and preference information. The system also includes a means for collecting voice data and extracting keywords in real time, and based on the keywords, obtains related conversation history and common interests from the server and notifies the user. Furthermore, the system includes a function for periodically collecting and analyzing the user's public SNS information, generating appropriate suggested information based on the extracted hobbies and interests, and transmitting the information to the terminal. This allows the user to receive reminders and suggested information at the appropriate time, enabling smooth communication.

[0006] "Facial recognition data" is digital data representing a user's facial features that is acquired using a device such as a camera.

[0007] A "server" is a central processing unit that receives data from multiple terminals via a network, processes, stores, and returns the data.

[0008] "Profile Information" is personal information including a user's name, contact information, interests and preferences.

[0009] A "terminal" is a computing device that is directly operated by a user and has functions such as facial recognition, voice collection, and information display.

[0010] A "database" is a system that efficiently stores large amounts of data and allows it to be searched and retrieved when needed.

[0011] "Audio data" refers to digital audio recordings of a user's speech, etc., collected through a microphone.

[0012] "Keywords" are important words or phrases extracted from speech or text data.

[0013] "SNS public information" refers to information such as posts and profiles that users make public on social networking services.

[0014] "Privacy Settings" are settings that allow users to choose and manage with whom they share their information.

[0015] "Reminder information" is information for alerting or reminding users that is provided based on past conversation history and user interests.

[0016] "Suggested Information" means recommended information provided to help users take action based on collected and analyzed data.

[0017] "Collection" is the act of acquiring and storing data.

[0018] "Analysis" is the process of evaluating collected data and extracting useful information. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention is a system for supporting user communication and deepening human relationships. This system combines the following various devices and means to achieve smooth interaction with users.

[0041] System Components

[0042] 1. Facial Recognition and Profile Management

[0043] This system acquires a user's facial recognition data and manages profile information based on it. When a user uses the system, facial recognition data is acquired through the device's camera. The acquired data is sent to a server, which checks it against a database to identify matching profile information. This profile information includes the user's name, contact information, past conversation history, interests, and preferences. This information is then sent to the device as appropriate and provided to the user.

[0044] For example, when User A starts the system, the device's camera recognizes User A's face and sends the facial recognition data to the server. The server searches the database, identifies User A's profile information, and sends it to the device. The device then provides that information to User A by displaying it or by voice.

[0045] 2. Audio collection and real-time analysis

[0046] This system has the ability to collect and analyze conversational audio in real time. The device collects the user's conversation content as audio data and extracts important keywords through a voice analysis module. This data is sent to a server, which then searches for past conversation history and related information and sends it to the device. The device notifies the user of this information to help the conversation progress.

[0047] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will then search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[0048] 3. Collecting and suggesting SNS information

[0049] Furthermore, this system collects information publicly shared on social media by users and makes suggestions based on that information. The server periodically collects users' social media information and stores it in their profiles. The collected information is analyzed to extract the user's hobbies and interests. The extracted information is then provided to the user as suggested information at the appropriate time.

[0050] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[0051] 4. User Privacy Management

[0052] This system allows users to configure settings to protect their privacy. When using the system for the first time, users select who they want to share their information with and set permission information. This prevents information from being shared without permission.

[0053] For example, when User D starts the system for the first time, he or she sets permission for information sharing. User D sets the system so that information is shared only with family members and specific business partners.

[0054] In this way, this system combines multiple functions such as facial recognition, voice analysis, collection and analysis of SNS information, and privacy management to support user communication and help deepen human relationships.

[0055] The processing flow will be explained below.

[0056] Program processing steps for face recognition and conversation reminder system

[0057] 1. User registration and license information settings

[0058] Step 1: Register a basic user profile

[0059] On the device: The application is launched and a screen is displayed that captures the user's facial recognition data using the camera. The user then enters basic profile information such as name, contact details, and social media accounts.

[0060] Server: Receives facial recognition data and basic profile information sent from the device and stores it in a database.

[0061] Step 2: Set up your license information

[0062] On your device: Display the privacy settings screen and give the option to choose who you want to share information with.

[0063] Users: Choose who you share your information with and set your privacy preferences.

[0064] Server: Receives the permission information, associates it with the profile, and stores it in a database.

[0065] 2. Facial Recognition and Information Extraction

[0066] Step 1: Obtaining facial recognition data

[0067] Device: Uses the camera to recognize the user's face and collect facial recognition data.

[0068] Step 2: Send and match facial recognition data

[0069] Device: Sends the acquired facial recognition data to the server.

[0070] Server: Matches the facial recognition data against a database to find matching profile information.

[0071] Step 3: Provide profile information

[0072] Server: Generates relevant profile information (such as name, past conversation history, hobbies, etc.) and sends it to the device.

[0073] Terminal: Provides profile information to the user by display or audio.

[0074] 3. Real-time conversation reminder function

[0075] Step 1: Recognizing voice input

[0076] Device: Uses a microphone to collect audio of conversations in real time.

[0077] Step 2: Keyword extraction

[0078] On the device: The speech analysis module extracts important keywords and conversation topics from the collected audio.

[0079] Step 3: View your conversation history

[0080] Device: Sends the extracted keywords to the server to retrieve related past conversation history and common interests.

[0081] Server: Searches for relevant information from a database based on the received keywords and sends it to the device.

[0082] Step 4: View Reminder Information

[0083] Device: Displays reminders to users in real time, such as information based on past topics or the other person's interests.

[0084] 4. Referencing and suggesting public information on social media

[0085] Step 1: Collect social media information

[0086] Server: Periodically collects users' public social media information via API with their permission and stores it in their profile.

[0087] Step 2: Analyze social media information

[0088] Server: Analyzes collected SNS information and extracts users' latest interests and hobbies.

[0089] Step 3: Generate proposals

[0090] Server: Based on the analysis results, generate appropriate proposal information and send it to the device.

[0091] On the device: Notify users of suggested information, such as a reminder to "Talk about a new interest you recently shared on social media."

[0092] 5. Privacy Management

[0093] Step 1: Provide privacy settings

[0094] Device: Provide users with a privacy settings screen and let them choose who they want to share information with.

[0095] User: Set up sharing of information with specific people (for example, family members or coworkers).

[0096] Step 2: Save your permission information

[0097] Server: Receives the configured permission information, stores it in a database, and associates it with a profile.

[0098] These steps of the present system allow users to obtain information that is appropriate for them and achieve smooth communication.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] Conventional communication support systems are unable to fully utilize the individual user's information, resulting in insufficient personalized support. Furthermore, from the perspective of privacy protection, there are problems with managing and sharing information. Furthermore, it is difficult to provide real-time conversation support or suggest appropriate information, making it difficult to improve the user experience.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for acquiring biometric authentication data of a user, means for transmitting the acquired biometric authentication data to a data processing device, means for matching the biometric authentication data with a data storage area in the data processing device to search for matching identification information and transmitting the matching identification information to a terminal device, means for providing the user with the identification information transmitted from the data processing device, means for collecting acoustic data and extracting keywords in real time, means for acquiring related conversation histories and common interests from the data processing device based on the extracted keywords, means for notifying the user of the acquired information, means for periodically collecting the user's electronic information disclosure data and storing it in the identification information, means for analyzing the collected electronic information disclosure data to extract the user's hobbies and interests, means for generating appropriate proposal information based on the extracted information and transmitting it to the terminal device, means for notifying the user of the transmitted proposal information, means for the user to input basic identification information when using the system for the first time, means for setting privacy and selecting partners with whom to share information, and means for storing the set permission information in the data storage area and associating it with the identification information. This enables personalized communication support for the user, efficient information proposals, and privacy management.

[0104] "Biometric data" is data used to identify an individual based on physical characteristics such as the user's face, fingerprint, or iris.

[0105] "Data Processing Device" means a computer system for processing and analyzing data obtained from a user.

[0106] The "data storage area" is a storage device for storing data such as user identification information and past conversation history.

[0107] An "end device" is a device that can be directly operated by a user, specifically a terminal such as a smartphone, tablet, or PC.

[0108] "Identification information" is data that includes a user's name, contact information, past conversation history, interests, preferences, etc.

[0109] "Acoustic data" is data that collects audio information such as the user's voice and surrounding sounds.

[0110] "Keywords" are important words or phrases extracted from the acoustic data.

[0111] "Electronic information public data" refers to online information that users have made public, such as on social media sites and blogs.

[0112] "Suggested information" is information that is generated by the data processing device based on the user's hobbies and interests, and is useful information for the user.

[0113] "Privacy settings" are settings that determine the scope of information sharing and permission information, which are set by the user when using the system for the first time.

[0114] "Permission information" is setting information that allows a user to permit information sharing with specific people or organizations.

[0115] The present invention provides a system for supporting user communication and deepening interpersonal relationships. This system integrates functions such as face recognition, voice analysis, collection and analysis of social networking site information, and privacy management. Specific embodiments are described below.

[0116] Facial Recognition and Profile Management

[0117] First, the biometric data of the user is acquired. The camera of the terminal is used to capture the user's face in real time, and facial feature points are extracted using a facial recognition algorithm (e.g., OpenCV). The acquired biometric data is then sent from the terminal to a data processing device.

[0118] The data processing device matches the received biometric data with its data storage to find matching identifying information, which may include the user's name, contact details, past conversation history, interests, and preferences. The identifying information is then sent from the data processing device to the end device and provided to the user.

[0119] As a specific example, when user A starts face recognition using a terminal, the camera recognizes user A's face and sends biometric authentication data to the data processing device. The data processing device compares it with the data storage area, identifies user A's identification information, sends it to the terminal, and displays it to user A.

[0120] Audio collection and real-time analysis

[0121] Next, the device collects the user's speech in real time through a microphone, and the collected acoustic data is converted into text through a speech analysis module (e.g., Google® Cloud Speech-to-Text) to extract important keywords.

[0122] The extracted keywords are sent from the terminal to a data processing device, which searches the data storage area for past conversation history and related information. The search results are sent to the terminal and notified to the user.

[0123] For example, if User B starts talking about "project progress" during a conversation, the device recognizes this keyword and sends it to the data processing device. The data processing device searches for past conversation history related to "project progress" and sends it to the device to notify User B.

[0124] SNS information collection and suggestions

[0125] The data processing device also periodically collects the user's public electronic information data using an API (e.g., Twitter API or Facebook Graph API). The collected public electronic information data is stored in a data storage area and analyzed. The analysis allows the user's hobbies and interests to be extracted.

[0126] The extracted information is sent from the data processing device to the terminal device at a time appropriate for the device and provided to the user as suggested information. For example, if User C posts on SNS that he or she has recently become interested in sports, the data processing device collects and analyzes this information. The next time a topic related to sports comes up during User C's conversation, the device will remind him or her by saying, "You've been posting about sports recently."

[0127] User Privacy Management

[0128] The system also places great importance on protecting users' privacy. When users first use the system, they can enter basic identification information and configure their privacy settings, which allow them to choose with whom they want to share their information.

[0129] The permission information is saved in the data storage area and associated with the profile. This prevents information from being shared without permission. For example, when User D starts the system for the first time, he or she can set it up so that information is shared only with family members and specific business partners.

[0130] Example of a generated AI prompt

[0131] Using a generative AI model, we can generate prompts like this:

[0132] Facial Recognition and Profile Management:

[0133] "Please explain in detail how your system works to obtain a user's facial recognition data and provide profile information."

[0134] Audio collection and real-time analysis:

[0135] "Please explain in detail how the system works, collecting audio in real time, analyzing important keywords, and notifying users."

[0136] Social media information collection and suggestions:

[0137] "Please explain in detail how the system works to collect social media information and suggest appropriate information to users."

[0138] User Privacy Controls:

[0139] "Please explain in detail how your system manages user privacy."

[0140] As described above, the present invention provides specific means for supporting communication between users and deepening human relationships.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1:

[0143] The device captures the user's biometric authentication data using a camera. At this time, it uses a facial recognition algorithm (e.g., OpenCV) to extract facial feature points. The input to this process is real-time video captured by the camera, and the output is facial recognition data with extracted feature points.

[0144] Specifically, the device's camera is activated, the user's face is detected, and feature points are extracted. This facial recognition data is then stored in temporary memory.

[0145] Step 2:

[0146] The device transmits the acquired biometric authentication data, including facial recognition data, to the server.

[0147] Specifically, the device sends facial recognition data to a server via the Internet, and once the transmission is complete, the data is stored on the server.

[0148] Step 3:

[0149] The server compares the received biometric data against a database to find matching identification information. The input to this process is the facial recognition data sent by the device, and the output is the matching profile information.

[0150] Specifically, the server matches the facial recognition data with a data store to identify corresponding identifying information, including the user's name, contact details, past conversation history, interests, and preferences.

[0151] Step 4:

[0152] The server transmits the acquired identification information to the terminal, and the transmitted data includes the user's profile information.

[0153] Specifically, the server converts the identification information into packets and sends them to the terminal via the Internet. The terminal receives this data.

[0154] Step 5:

[0155] The terminal provides the user with the identification information sent by the server. The input to this process is the profile information sent by the server, and the output is the display of the information to the user.

[0156] Specifically, the device displays profile information on the screen, allowing users to check their own name and past conversation history.

[0157] Step 6:

[0158] The device collects the user's conversation content in real time through a microphone. The input of this process is the user's voice, and the output is acoustic data.

[0159] Specifically, the device's microphone captures the user's voice and stores it as audio data, which is then temporarily stored in the device's internal memory.

[0160] Step 7:

[0161] The terminal uses a speech analysis module to convert the collected acoustic data into text and extract important keywords. The input of this process is the collected acoustic data, and the output is the extracted keywords.

[0162] Specifically, the device calls a speech analysis module such as Google Cloud Speech-to-Text to convert the voice data into text, and then extracts important keywords from the converted text.

[0163] Step 8:

[0164] The terminal transmits the extracted keywords to the server. The input of this process is the extracted keywords, and the output is data transmission to the server.

[0165] Specifically, the device sends the keyword to a server via the Internet, where the data is temporarily stored.

[0166] Step 9:

[0167] The server searches the database for related conversation histories and common interests based on the received keywords. The input to this process is the extracted keywords, and the output is related information.

[0168] Specifically, the server uses keywords to search the data storage for relevant conversation histories and common interests, and once the search results are identified, they are stored in temporary memory.

[0169] Step 10:

[0170] The server transmits the retrieved related information to the terminal. The input of this process is the retrieved related information, and the output is data transmission to the terminal.

[0171] Specifically, the server converts the relevant information into packets and sends them to the terminal via the Internet, which then receives the data.

[0172] Step 11:

[0173] The terminal notifies the user of the relevant information sent from the server. The input of this process is the relevant information sent from the server, and the output is a notification to the user.

[0174] Specifically, the device will display relevant information on the screen or notify the user via voice, allowing the user to check this information in real time.

[0175] (Application example 1)

[0176] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0177] Conventional advertising display systems have had difficulty effectively providing personalized advertisements based on users' interests. Furthermore, they have been unable to properly utilize users' profile information and social networking site activity to display highly relevant advertisements in real time, making it difficult to attract users' attention and reducing advertising effectiveness. The present invention aims to solve these problems.

[0178] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0179] In this invention, the server includes means for acquiring facial recognition data of a user, means for transmitting the acquired facial recognition data to the server, means for the server to compare the facial recognition data with a database and search for matching profile information and transmit the matching profile information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation history and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, means for personalizing advertisement content displayed on the terminal based on the user's profile information and SNS information, and means for analyzing the user's interests using the acquired profile information and SNS information and displaying relevant advertisements. This enables effective provision of personalized advertisements based on the user's interests.

[0180] - "User's facial recognition data" means data that captures the user's facial features as digital information and enables identification.

[0181] The "server" is a computer system that manages users' facial recognition data and profile information, and collates and searches data in real time.

[0182] "Profile information" is data that includes personal information such as a user's name, contact details, past conversation history, interests and preferences.

[0183] A "terminal" is a device that has an interface with a user and displays or notifies information.

[0184] "Voice data" refers to data that records the contents of a user's conversation in digital format.

[0185] "Keyword extraction" is the process of analyzing and extracting important words and phrases from audio data in real time.

[0186] "Related conversation history" is data obtained by searching for past conversation content and related information based on keywords.

[0187] "SNS information" refers to information that a user makes public on a social networking service.

[0188] "Personalization" is the process of customizing advertisements and information based on individual user profile information and social media information.

[0189] "Advertising display" is the process of displaying advertisements on a device based on the user's interests.

[0190] To implement this invention, the following steps are required: The user installs a dedicated smartphone application, enters basic profile information when the application is first launched, and faces the camera to obtain facial recognition data. The application collects the user's facial recognition data and sends it to a server. At this time, it performs facial recognition using a library such as OpenCV. The server compares the received facial recognition data with a database, searches for matching profile information, and sends it to the terminal.

[0191] The device then displays the profile information and begins collecting voice data. As the user begins to talk, voice data is collected in real time and keyword analysis is performed. This analysis uses a voice recognition library and natural language processing tools. The analyzed keywords are sent to the server, which searches for relevant information based on past conversation history and the user's interests and sends it to the device. The device notifies the user of this and helps the conversation proceed smoothly.

[0192] The server also periodically collects social media information and stores it in the user's profile. A data analysis module is used to analyze this social media information and extract the user's hobbies and interests. The analysis results are sent to the device, and suggestions tailored to the user are generated. This suggestion information is displayed as personalized advertisements based on the user's profile information and social media information.

[0193] For example, if a user posts on social media that they are looking for new running shoes, the server will retrieve and analyze this information. The next time the user uses the app, an advertisement for related running shoes will be displayed. In this way, information tailored to the user's interests is provided in real time.

[0194] Example prompt sentence:

[0195] "The user posted on social media that they were interested in new running shoes, so please show them an appropriate ad."

[0196] The hardware required to realize this system is a smartphone camera and microphone, and the software used includes the OpenCV library, speech recognition library, natural language processing tools, data analysis modules, server APIs, etc. This makes it possible to accurately capture user interests and display personalized advertisements.

[0197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0198] Step 1:

[0199] A user launches a smartphone application and enters profile information. When the user launches the application for the first time, they enter basic information such as their name, contact details, and interests, and then face the camera. The device acquires facial recognition data and sends it to the server along with the acquired profile information. The input is the user's basic information and facial recognition data, and the output is the profile data sent to the server.

[0200] Step 2:

[0201] The server checks the received facial recognition data against a database to find matching profile information. It compares the facial recognition data with existing data in the database to identify the best matching profile information. The input is the facial recognition data, and the output is the matching profile information. This information is then sent back to the device.

[0202] Step 3:

[0203] The device displays the profile information received from the server to the user. This profile information includes the user's name, interests, past conversation history, etc. Based on the displayed profile information, the user can check for more detailed information. The input is the matching profile information, and the output is the information display to the user.

[0204] Step 4:

[0205] When a user starts a conversation, the device collects the conversation content as voice data in real time. Using a voice recognition library, important keywords are extracted from the collected voice data. The input is the user's voice data, and the output is the extracted keywords.

[0206] Step 5:

[0207] The extracted keywords are sent to the server, which searches for related information based on past conversation history and related interests. The server searches a database based on the extracted keywords to obtain related conversation history and common interests. The input is the keywords, and the output is related information. This related information is sent to the terminal and notified to the user.

[0208] Step 6:

[0209] The server periodically collects SNS information and stores it in the user's profile. The server obtains SNS information and analyzes the user's current interests and concerns from that information. The input is SNS information, and the output is analyzed interest and concern information.

[0210] Step 7:

[0211] The server generates personalized suggestion information based on the acquired interest information and sends it to the device. The device receives the suggestion information and displays it to the user. The input is the analyzed interest information, and the output is personalized suggestion information provided to the user.

[0212] Step 8:

[0213] Based on the user's profile information and social media information, the device generates and displays personalized advertisements. The advertisement display system selects the most suitable advertisement and provides it to the user in real time. The input is the profile information and social media information, and the output is the displayed advertisement.

[0214] In this way, the invention utilizes a user's facial recognition data, voice data, and social media information to provide a personalized experience based on their individual profile information.

[0215] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0216] This invention is a system that combines facial recognition, voice analysis, collection and analysis of SNS information, privacy management, and an emotion engine that recognizes user emotions in order to support user communication and deepen human relationships.

[0217] System Components

[0218] 1. Facial Recognition and Profile Management

[0219] The system acquires a user's facial recognition data and manages profile information based on it. When a user uses the system, facial recognition data is acquired through the device's camera and sent to the server. The server compares the data with a database to identify matching profile information and sends it to the device. This profile information includes the user's name, contact information, past conversation history, interests, and preferences. The device then provides this information to the user.

[0220] As a concrete example, when User A starts the system, the device's camera recognizes User A's face and sends the facial recognition data to the server. The server searches the database, identifies the profile information of "User A," and sends it to the device. The device then provides the information to User A by displaying it or by voice.

[0221] 2. Audio collection and real-time analysis

[0222] This system has the ability to collect and analyze conversational audio in real time. The device collects the user's conversation content as audio data and extracts important keywords through the audio analysis module. This data is sent to a server, which then searches for past conversation history and related information and sends it to the device. The device then notifies the user of this information and supports the progress of the conversation.

[0223] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will then search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[0224] 3. Collecting and suggesting SNS information

[0225] Furthermore, this system collects information publicly shared by users on social media and uses that information to make suggestions that are appropriate for the user. The server periodically collects social media information and saves it in a profile. The collected information is analyzed to extract the user's hobbies and interests, which are then sent to the device as suggested information at the appropriate time.

[0226] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[0227] 4. Privacy Management

[0228] This system allows users to configure settings to protect their privacy. When using the system for the first time, users select with whom they want to share their information and set permission information. This prevents information from being shared without permission.

[0229] For example, when User D starts the system for the first time, he or she sets permission for information sharing. User D sets the system to share information only with family members and specific business partners.

[0230] 5. Introducing the Emotion Engine

[0231] This system incorporates an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's voice and facial expressions in real time and collects the data. The collected emotion data is sent to a server and associated with the user's profile. The server analyzes the emotion data and provides appropriate information and responses based on the user's emotions.

[0232] As a concrete example, if user E expresses negative emotions during a conversation with a friend, the emotion engine recognizes the emotion and sends emotional data to the server. Based on that data, the server sends appropriate suggestions to the device, such as "switch to a more relaxing topic" or "send a sympathetic message." The device then reminds user E of these suggestions, helping to keep the conversation flowing smoothly.

[0233] In this way, this system not only incorporates facial recognition, voice analysis, collection and analysis of SNS information, and privacy management, but also an emotion engine, allowing it to respond to a variety of communication situations for users and help build deeper human relationships.

[0234] The processing flow will be explained below.

[0235] Processing steps of the facial recognition and conversation reminder system (including emotion engine)

[0236] 1. User registration and license information settings

[0237] Step 1: Register a basic user profile

[0238] Device: The application launches and displays a screen that captures the user's facial recognition data using the camera. The user then enters basic profile information such as name, contact details, and social media accounts.

[0239] Server: Receives facial recognition data and basic profile information sent from the device and stores it in a database.

[0240] Step 2: Set up your license information

[0241] On your device: Display the privacy settings screen and give you the option to choose who you want to share information with.

[0242] Users: Choose who you share information with and set your privacy preferences.

[0243] Server: Receives the permission information, associates it with the profile, and stores it in a database.

[0244] 2. Facial Recognition and Information Extraction

[0245] Step 1: Obtaining facial recognition data

[0246] Device: Uses the camera to recognize the user's face and collect facial recognition data.

[0247] Step 2: Send and match facial recognition data

[0248] Terminal: Sends the acquired facial recognition data to the server.

[0249] Server: Matches the facial recognition data against a database to find matching profile information.

[0250] Step 3: Provide profile information

[0251] Server: Generates relevant profile information (such as name, past conversation history, hobbies, etc.) and sends it to the device.

[0252] Terminal: Provides profile information to the user by display or audio.

[0253] 3. Real-time conversation reminder function

[0254] Step 1: Recognizing voice input

[0255] Device: Uses a microphone to collect audio of conversations in real time.

[0256] Step 2: Keyword extraction

[0257] Terminal: The speech analysis module extracts important keywords and conversation topics from the collected speech.

[0258] Step 3: View your conversation history

[0259] Device: Sends the extracted keywords to the server to obtain related past conversation history and common interests.

[0260] Server: Searches for relevant information from a database based on the received keywords and sends it to the terminal.

[0261] Step 4: View Reminder Information

[0262] Device: Displays reminder information to the user in real time. For example, it can present information based on topics that have been discussed in the past or the other person's hobbies and preferences.

[0263] 4. Emotion Recognition by Emotion Engine

[0264] Step 1: Collecting emotion data

[0265] Device: Uses a camera and microphone to collect emotional data from the user's facial expressions and voice.

[0266] Step 2: Analyze the emotion data

[0267] Terminal: The emotion engine analyzes the user's emotional state from the collected data.

[0268] Step 3: Send and store emotion data

[0269] Terminal: Transmits the analyzed emotion data to the server.

[0270] Server: Stores the received emotion data in a database and associates it with a profile.

[0271] Step 4: Emotionally informed

[0272] Server: Generates appropriate information and suggestions based on the user's emotional state and sends them to the device.

[0273] Device: Displays and notifies users of emotion-based reminders and appropriate responses.

[0274] 5. Referencing and suggesting public information on social media

[0275] Step 1: Collect social media information

[0276] Server: Periodically collects users' public social media information via API with their permission and stores it in their profile.

[0277] Step 2: Analyze social media information

[0278] Server: Analyzes collected SNS information and extracts users' latest interests and hobbies.

[0279] Step 3: Generate proposals

[0280] Server: Based on the analysis results, it generates appropriate proposal information and sends it to the device.

[0281] On your device: Providing suggested information to you, such as a reminder to "Talk about a new interest you recently shared on social media."

[0282] 6. Privacy Management

[0283] Step 1: Provide privacy settings

[0284] Device: Provide users with a privacy settings screen and allow them to select who they want to share information with.

[0285] User: Set up sharing of information with specific people (for example, family members or coworkers).

[0286] Step 2: Save your permission information

[0287] Server: Receives the configured permission information, stores it in a database, and associates it with the profile.

[0288] Through these steps, this system integrates a wide range of functions, including facial recognition, voice analysis, emotion recognition, collection and analysis of SNS information, and privacy management, thereby providing users with valuable information and enabling smooth communication.

[0289] Example 2

[0290] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0291] Current communication technologies and interaction systems have difficulty accurately understanding and responding appropriately to users' individuality and emotions. Therefore, systems that can improve the quality of communication with users and build deeper human relationships are needed. Furthermore, existing systems lack the functionality to reflect users' social media information and real-time emotions and make personalized suggestions. Furthermore, privacy protection settings are difficult to understand, and data permission management is insufficient.

[0292] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring face recognition data of a user, means for transmitting the acquired face recognition data to the server, means for comparing the face recognition data with a database in the server and searching for matching profile information and transmitting the matching profile information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation histories and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, emotion recognition means for analyzing the user's voice and facial expressions in real time, means for transmitting the analyzed emotion data to the server, and means for providing appropriate information and reactions based on the emotion data. This makes it possible to provide and suggest appropriate information to individual users in real time according to their emotions and interests, thereby realizing deeper communication with users.

[0293] "Facial recognition data" is feature data obtained by analyzing a facial image of a user captured using a device such as a camera.

[0294] A "server" is a computer system that processes, stores, and manages data over a network.

[0295] A "database" is an information system that allows data to be stored, managed, and searched in an organized manner.

[0296] "Profile information" is a collection of data that includes personal information, history, interests, etc. about a user.

[0297] "Voice data" refers to the recording and analysis of user-generated voice as digital data.

[0298] "Keywords" are important words or phrases extracted from the user's conversation.

[0299] "Conversation history" is a record of past conversations and statements.

[0300] "SNS public information" refers to information that a user has set to be public on a social networking service.

[0301] "Privacy settings" are settings that allow users to define the scope and permission conditions for sharing personal information.

[0302] "Emotion recognition means" refers to technology or devices that analyze and recognize the user's emotional state from their voice and facial expressions.

[0303] "Notification means" is a function for notifying the user of information or suggestions through a terminal or device.

[0304] This invention is a system that supports user communication and deepens human relationships. This system combines face recognition, voice analysis, collection and analysis of SNS information, privacy management, and an emotion engine that recognizes user emotions.

[0305] composition

[0306] This system mainly consists of the following components: The device is equipped with a camera and microphone, which collect data. The data is processed on the server, and appropriate feedback is provided to the user using the information stored in the database.

[0307] Hardware and software used

[0308] 1. Terminal: A device such as a smartphone, tablet, or PC. It uses a camera module to take pictures and a microphone to collect audio.

[0309] 2. Server: A computer system that processes data and manages databases.

[0310] 3. Facial recognition software: Facial recognition library such as OpenCV, dlib.

[0311] 4. Speech analysis software: A speech recognition library such as the Google Speech-to-Text API.

[0312] 5. Emotion recognition software: Use IBM Watson® Tone Analyzer, Microsoft® Azure® Face API, etc.

[0313] 6. Database: A relational database such as MySQL (registered trademark).

[0314] Program processing

[0315] 1. Facial Recognition and Profile Management

[0316] When a user starts the system, a facial image is captured using the device's camera.

[0317] The device inputs the captured facial image into a facial recognition library (e.g., OpenCV) to generate facial recognition data.

[0318] The recognized face data is sent to a server, which searches a database to identify the corresponding profile information.

[0319] The server sends profile information, which the terminal then displays or audibly presents to the user.

[0320] As a concrete example, when User A starts the system, the device's camera recognizes User A's face and sends the data to the server. The server searches the database, identifies the profile information of "User A," and sends it to the device. The device then provides the information to User A by displaying it or by voice.

[0321] 2. Audio collection and real-time analysis

[0322] The terminal uses a microphone to collect the user's conversational voice.

[0323] The collected voice data is converted into text using a voice analysis module (e.g., Google Speech-to-Text API).

[0324] Important keywords are extracted from the converted text and sent to the server.

[0325] The server searches past conversation history and related information and sends it to the terminal.

[0326] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[0327] 3. Collecting and suggesting SNS information

[0328] The server periodically collects public information from the social media accounts authorized by the user.

[0329] The collected SNS information is analyzed using analytical tools (e.g., TENSORFLOW (registered trademark) or PyTorch) to extract users' hobbies and interests.

[0330] The extracted information is stored in a profile and suggested information is sent to the device at the appropriate time.

[0331] The terminal notifies the user of the suggestion information.

[0332] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[0333] 4. Privacy Management

[0334] When users first use the system, they set their privacy preferences and choose who they want to share their information with.

[0335] The terminal transmits the set permission information to the server.

[0336] The server stores the permission information in a database and associates it with a profile.

[0337] As a specific example, when User D starts the system for the first time, he / she sets permission for information sharing. User D sets the system so that information is shared only with family members and specific business partners.

[0338] 5. Introducing the Emotion Engine

[0339] The device recognizes emotions in real time from the user's voice and facial expressions (for example, using IBM Watson Tone Analyzer or Microsoft Azure Face API).

[0340] The recognized emotion data is sent to the server.

[0341] The server analyzes the emotion data and associates it with the user's profile.

[0342] Data is sent to the terminal that provides appropriate information and responses based on the user's emotions.

[0343] As a concrete example, if user E expresses negative emotions during a conversation with a friend, the emotion engine recognizes the emotion and sends emotional data to the server. Based on that data, the server sends appropriate suggestions to the device, such as "switch to a more relaxing topic" or "send a sympathetic message." The device then reminds user E of these suggestions, helping to keep the conversation flowing smoothly.

[0344] In this way, this system not only incorporates facial recognition, voice analysis, collection and analysis of SNS information, and privacy management, but also an emotion engine, allowing it to respond to a variety of communication situations for users and help build deeper human relationships.

[0345] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0346] Step 1:

[0347] The user starts the system. The device camera captures the user's facial image. Input: User's facial image. Output: Facial recognition data.

[0348] The device uses a facial recognition library (e.g., OpenCV or dlib) to analyze the captured facial image and generate facial recognition data, which is represented as template data with unique features.

[0349] Step 2:

[0350] The facial recognition data generated by the device is sent to the server. Input: Facial recognition data. Output: Identified profile information.

[0351] The server then matches the received facial recognition data with a database containing user profile information (such as name, contact details, past conversation history, interests, and preferences) to find matching profiles.

[0352] Step 3:

[0353] The server sends the profile information to the device. Input: Identified profile information. Output: Provided profile information.

[0354] The device provides the profile information sent from the server to the user, which can be displayed on the screen or read aloud by a voice assistant.

[0355] Step 4:

[0356] The device uses a microphone to collect the user's conversational voice in real time. Input: User's voice data. Output: Text-converted voice data.

[0357] The voice data is converted into text through a voice analysis module (e.g., Google Speech-to-Text API), and the converted result is used as text.

[0358] Step 5:

[0359] The device extracts important keywords from the text-converted voice data. Input: Text-converted voice data. Output: Extracted keywords.

[0360] The natural language processing function of the speech analysis module is used to extract important keywords from the text. For example, "project progress" is extracted as an important keyword.

[0361] Step 6:

[0362] Based on important keywords, related conversation history and common interests are retrieved from the server. Input: Extracted keywords. Output: Related information.

[0363] The server searches the database based on the extracted keywords to retrieve information related to past conversation history and common interests, which is implemented using a text search algorithm.

[0364] Step 7:

[0365] Notify the user of the acquired information. Input: Relevant information. Output: Notification to the user.

[0366] The device notifies the user of the acquired related information, which can be displayed on the screen or given verbally by a voice assistant.

[0367] Step 8:

[0368] The server periodically collects the user's SNS public information and saves it in the profile. Input: SNS public information. Output: Updated profile information.

[0369] We use collection tools to automatically retrieve public information from users' social media accounts, using the social media APIs (e.g., Twitter API and Facebook Graph API).

[0370] Step 9:

[0371] Analyze collected SNS information to extract user hobbies and interests. Input: SNS information. Output: Extracted hobbies and interests.

[0372] Analyze social media information using analytical tools (e.g., TensorFlow or PyTorch), thereby probabilistically extracting users' hobbies and interests.

[0373] Step 10:

[0374] Based on the extracted information, appropriate suggestion information is generated and sent to the device. Input: Extracted information on hobbies and interests. Output: Suggestion information.

[0375] The server generates suggested information based on the extracted hobbies and interests. For example, if the user is interested in "sports," it will suggest related events and news.

[0376] Step 11:

[0377] Notify the user of the submitted proposal information. Input: Proposal information. Output: Notification to the user.

[0378] The device notifies the user of the received suggestion information by means of a screen display, a pop-up message, or guidance via a voice assistant.

[0379] Step 12:

[0380] When a user first uses the system, they set their privacy preferences. Input: User's privacy preferences information. Output: Privacy preferences data.

[0381] The device receives user input and generates privacy setting data, allowing the user to select the scope of information sharing and the people with whom the information is to be shared.

[0382] Step 13:

[0383] The device sends the generated privacy setting data to the server. Input: Privacy setting data. Output: Saved permission information.

[0384] The server stores the received privacy setting data in a database and associates it with the profile.

[0385] Step 14:

[0386] The device analyzes the user's voice and facial expressions in real time. Input: User's voice and facial expression data. Output: Recognized emotion data.

[0387] Emotion recognition software (e.g., IBM Watson Tone Analyzer or Microsoft Azure Face API) is used to analyze the user's emotions from the collected voice and facial expressions.

[0388] Step 15:

[0389] The analyzed emotion data is sent to the server. Input: Recognized emotion data. Output: User's emotion information.

[0390] The server receives the emotion data and processes it to associate it with a profile.

[0391] Step 16:

[0392] Implement measures to provide appropriate information and responses based on emotional data. Input: User's emotional information. Output: Appropriate responses and suggestions.

[0393] The server generates appropriate information and responses based on the user's emotional data and sends them to the device, which then notifies the user, improving the quality of communication.

[0394] (Application example 2)

[0395] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0396] Existing communication support systems do not take into account the user's emotions or mood state, and therefore provide uniform information and suggestions, making it impossible to provide a personalized experience. Furthermore, there is a need for systems that not only collect and analyze social media information and profile information, but also perform real-time emotion recognition to suggest optimal content according to the user's emotions. This has led to challenges in improving entertainment experiences and deepening human relationships.

[0397] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0398] In this invention, the server includes means for acquiring facial recognition data of a user, means for transmitting the acquired facial recognition data to the server, means for the server to compare the facial recognition data with a database and search for matching profile information and transmit the searched information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation history and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, means for recognizing emotion data from the user's voice and facial expressions using an emotion engine and transmitting the emotion data to the server, and means for the server to analyze the emotion data, associate the emotion data with the profile, generate appropriate information and reactions, and transmit the information and reactions to the terminal. This enables personalized entertainment content to be proposed that takes the user's emotions into consideration.

[0399] "Facial recognition data" is data that captures a user's face in digital form and analyzes its features.

[0400] A "server" is a computer system that provides and manages data on a network and exchanges information with client terminals.

[0401] "Profile information" is a collection of information related to a particular user, including data such as name, contact information, conversation history, hobbies and interests, etc.

[0402] A "terminal" is a device that is directly operated by a user and displays or provides information via audio, such as a smartphone or smart glasses.

[0403] "Voice data" refers to data that is used to record and analyze user utterances and conversation content in digital form.

[0404] "Keywords" are important words or phrases extracted from audio data and used to identify themes and key points of a conversation.

[0405] "Conversation history" is data that records the contents of past conversations, and is used to understand the flow of communication between users.

[0406] "SNS information" refers to information about posts and activities made public by users on social networking services.

[0407] An "emotion engine" is a system or software that analyzes emotional data from a user's voice and facial expressions and recognizes their state in real time.

[0408] "Suggested information" refers to information or content suggestions that the system provides to the user, and is generated based on the user's profile information and emotional state.

[0409] "Privacy settings" are settings that allow users to decide with whom they want to share their information and to what extent they want to make it public.

[0410] "Permission information" is data that records the content and scope of information handling that a user has approved.

[0411] This invention supports user communication and appropriately suggests entertainment content by building a system that uses an emotion engine in addition to facial recognition, voice analysis, collection and analysis of SNS information, and privacy management.

[0412] Hardware and Software Configuration

[0413] 1. Obtaining the user's facial recognition data

[0414] Hardware: Smart glasses and smartphone cameras

[0415] Software: Facial recognition API (e.g., Amazon Rekognition)

[0416] Description: When a user turns on the device, the camera captures facial recognition data and sends it to a server, which then matches the data with a database to find matching profile information and send it to the device.

[0417] 2. Collection and analysis of audio data

[0418] Hardware: Smart glasses and smartphone microphones

[0419] Software: Speech analysis API (e.g., Google Cloud Speech-to-Text)

[0420] Description: When a user starts talking, the microphone collects voice data and extracts keywords in real time. The extracted keywords are sent to a server, which searches past conversation history and related information and provides them to the device.

[0421] 3. Introducing an Emotion Engine

[0422] Hardware: Smart glasses and smartphone cameras and microphones

[0423] Software: Sentiment analysis API (e.g. IBM Watson Tone Analyzer)

[0424] Description: Analyzes emotional data from voice and facial expressions in real time. Emotional data is associated with a profile and generates appropriate information and responses based on the user's state.

[0425] 4. Collecting and suggesting SNS information

[0426] Software: Social Media API (e.g. Twitter API, Facebook Graph API)

[0427] Description: The server periodically collects the user's social media information and saves it in the user's profile. It analyzes the collected information to extract hobbies and interests. Based on this, it generates suitable suggestions and sends them to the device.

[0428] 5. Privacy Settings

[0429] Software: Privacy Management API (e.g., OAuth 2.0)

[0430] Description: When a user first uses the system, they set their preferences regarding who they want to share their information with and how their emotional data is handled, and this information is stored in a database and associated with their profile.

[0431] 6. Content Suggestion

[0432] Description: The server suggests entertainment content based on the user's profile information and emotional data. For example, if the user is feeling stressed, it will suggest a list of relaxing movies. This is done by analyzing the user's state from social media information and past data.

[0433] Specific examples

[0434] When a user puts on the smart glasses and launches the content recommendation system, their profile is loaded using facial recognition. Based on the user's conversation and facial expressions, the system analyzes emotional data in real time and combines it with social media information to suggest appropriate entertainment content.

[0435] Prompt Sentence Examples

[0436] Generate a list of relaxing movies to suggest when the user is feeling stressed. Read the user's recent emotions from their social media posts and refer to their profile information. Choose the latest comedy movies as suggestions.

[0437] The above is an embodiment of the present invention, which allows users to have a personalized entertainment experience based on their emotions and interests.

[0438] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0439] Step 1:

[0440] Acquiring facial recognition data and loading profile information

[0441] When a user turns on their smart glasses or smartphone, the device's camera captures the user's facial recognition data.

[0442] Input: Video of user's face

[0443] Data processing: The raw data collected by the camera is processed by a facial recognition API (e.g., Amazon Rekognition) to produce quantified facial recognition data.

[0444] Output: Facial recognition data

[0445] The server receives the facial recognition data sent from the device and compares it with a database.

[0446] Input: Facial recognition data

[0447] Data calculation: Compare with existing facial recognition data in the database to find matching profile information.

[0448] Output: Profile information

[0449] The profile information is transmitted to the terminal and provided to the user.

[0450] Step 2:

[0451] Audio data collection and analysis

[0452] When a user starts talking, the device's microphone collects audio data.

[0453] Input: User's voice

[0454] Data processing: The audio data is converted into text format using a speech analysis API (e.g., Google Cloud Speech-to-Text).

[0455] Output: Speech-to-text data

[0456] The server analyzes the text data and extracts keywords.

[0457] Input: Speech-to-text data

[0458] Data calculations: The text analysis module identifies important keywords.

[0459] Output: Keywords

[0460] Based on the extracted keywords, the server obtains past conversation history and common interests and sends them to the terminal.

[0461] Input: keyword

[0462] Data calculation: Search the conversation history database and retrieve relevant information.

[0463] Output: Related information

[0464] Step 3:

[0465] Emotional Data Recognition and Analysis

[0466] Emotional data is collected from the user's facial expressions and voice using the device's camera and microphone.

[0467] Input: User's facial expression and audio

[0468] Data processing: Video and audio data are processed using an emotion analysis API (e.g., IBM Watson Tone Analyzer) and converted into emotion data.

[0469] Output: Emotion data

[0470] The server associates the emotional data with a profile and analyzes it.

[0471] Input: Emotion data

[0472] Data calculations: Emotional data is linked to profile information to analyze current emotional state.

[0473] Output: Emotional state analysis results

[0474] The server generates appropriate information and responses based on the emotional state and sends them to the device.

[0475] Input: Emotional state analysis results

[0476] Data calculation: Generate responses and suggestions based on the user's emotions.

[0477] Output: Proposal information

[0478] Step 4:

[0479] SNS information collection and suggestions

[0480] The server periodically collects the user's social media information and stores it in the profile.

[0481] Input: SNS public information

[0482] Data processing: Collect information using SNS APIs and add it to your profile.

[0483] Output: Updated profile information

[0484] The server analyzes the collected SNS information and extracts the user's hobbies and interests.

[0485] Input: Social Media Information

[0486] Data calculation: Hobbies and interests are identified through an information analysis module.

[0487] Output: Hobbies and interests analysis results

[0488] The server generates proposal information based on this and transmits it to the terminal.

[0489] Input: Hobbies and Interests Analysis Results

[0490] Data calculation: The personalized suggestion generation module makes appropriate content suggestions.

[0491] Output: Proposal information

[0492] Step 5:

[0493] Content suggestion notifications

[0494] The terminal receives the proposal information from the server and notifies the user.

[0495] Input: Proposal information

[0496] Data calculation: Processing is performed to present the received proposal information visually or audibly.

[0497] Output: Notify user of content suggestion

[0498] For example, consider a situation where a user is wearing smart glasses and browsing a list of movies. If the emotion engine detects that the user is stressed, the prompt to suggest "relaxing comedy movies" might be:

[0499] Generate a list of relaxing movies to suggest when the user is feeling stressed. Read the user's recent emotions from their social media posts and refer to their profile information. Choose the latest comedy movies as suggestions.

[0500] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0501] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0502] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0503] [Second embodiment]

[0504] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0505] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0506] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0507] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0508] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0509] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0510] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0511] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0512] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0513] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0514] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0515] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0516] The present invention is a system for supporting user communication and deepening human relationships. This system combines the following various devices and means to achieve smooth interaction with users.

[0517] System Components

[0518] 1. Facial Recognition and Profile Management

[0519] This system acquires a user's facial recognition data and manages profile information based on it. When a user uses the system, facial recognition data is acquired through the device's camera. The acquired data is sent to a server, which checks it against a database to identify matching profile information. This profile information includes the user's name, contact information, past conversation history, interests, and preferences. This information is then sent to the device as appropriate and provided to the user.

[0520] For example, when User A starts the system, the device's camera recognizes User A's face and sends the facial recognition data to the server. The server searches the database, identifies User A's profile information, and sends it to the device. The device then provides that information to User A by displaying it or by voice.

[0521] 2. Audio collection and real-time analysis

[0522] This system has the ability to collect and analyze conversational audio in real time. The device collects the user's conversation content as audio data and extracts important keywords through a voice analysis module. This data is sent to a server, which then searches for past conversation history and related information and sends it to the device. The device notifies the user of this information to help the conversation progress.

[0523] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will then search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[0524] 3. Collecting and suggesting SNS information

[0525] Furthermore, this system collects information publicly shared on social media by users and makes suggestions based on that information. The server periodically collects users' social media information and stores it in their profiles. The collected information is analyzed to extract the user's hobbies and interests. The extracted information is then provided to the user as suggested information at the appropriate time.

[0526] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[0527] 4. User Privacy Management

[0528] This system allows users to configure settings to protect their privacy. When using the system for the first time, users select who they want to share their information with and set permission information. This prevents information from being shared without permission.

[0529] For example, when User D starts the system for the first time, he or she sets permission for information sharing. User D sets the system so that information is shared only with family members and specific business partners.

[0530] In this way, this system combines multiple functions such as facial recognition, voice analysis, collection and analysis of SNS information, and privacy management to support user communication and help deepen human relationships.

[0531] The processing flow will be explained below.

[0532] Program processing steps for face recognition and conversation reminder system

[0533] 1. User registration and license information settings

[0534] Step 1: Register a basic user profile

[0535] On the device: The application is launched and a screen is displayed that captures the user's facial recognition data using the camera. The user then enters basic profile information such as name, contact details, and social media accounts.

[0536] Server: Receives facial recognition data and basic profile information sent from the device and stores it in a database.

[0537] Step 2: Set up your license information

[0538] On your device: Display the privacy settings screen and give the option to choose who you want to share information with.

[0539] Users: Choose who you share your information with and set your privacy preferences.

[0540] Server: Receives the permission information, associates it with the profile, and stores it in a database.

[0541] 2. Facial Recognition and Information Extraction

[0542] Step 1: Obtaining facial recognition data

[0543] Device: Uses the camera to recognize the user's face and collect facial recognition data.

[0544] Step 2: Send and match facial recognition data

[0545] Device: Sends the acquired facial recognition data to the server.

[0546] Server: Matches the facial recognition data against a database to find matching profile information.

[0547] Step 3: Provide profile information

[0548] Server: Generates relevant profile information (such as name, past conversation history, hobbies, etc.) and sends it to the device.

[0549] Terminal: Provides profile information to the user by display or audio.

[0550] 3. Real-time conversation reminder function

[0551] Step 1: Recognizing voice input

[0552] Device: Uses a microphone to collect audio of conversations in real time.

[0553] Step 2: Keyword extraction

[0554] On the device: The speech analysis module extracts important keywords and conversation topics from the collected audio.

[0555] Step 3: View your conversation history

[0556] Device: Sends the extracted keywords to the server to retrieve related past conversation history and common interests.

[0557] Server: Searches for relevant information from a database based on the received keywords and sends it to the device.

[0558] Step 4: View Reminder Information

[0559] Device: Displays reminders to users in real time, such as information based on past topics or the other person's interests.

[0560] 4. Referencing and suggesting public information on social media

[0561] Step 1: Collect social media information

[0562] Server: Periodically collects users' public social media information via API with their permission and stores it in their profile.

[0563] Step 2: Analyze social media information

[0564] Server: Analyzes collected SNS information and extracts users' latest interests and hobbies.

[0565] Step 3: Generate proposals

[0566] Server: Based on the analysis results, generate appropriate proposal information and send it to the device.

[0567] On the device: Notify users of suggested information, such as a reminder to "Talk about a new interest you recently shared on social media."

[0568] 5. Privacy Management

[0569] Step 1: Provide privacy settings

[0570] Device: Provide users with a privacy settings screen and let them choose who they want to share information with.

[0571] User: Set up sharing of information with specific people (for example, family members or coworkers).

[0572] Step 2: Save your permission information

[0573] Server: Receives the configured permission information, stores it in a database, and associates it with a profile.

[0574] These steps of the present system allow users to obtain information that is appropriate for them and achieve smooth communication.

[0575] Example 1

[0576] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0577] Conventional communication support systems are unable to fully utilize the individual user's information, resulting in insufficient personalized support. Furthermore, from the perspective of privacy protection, there are problems with managing and sharing information. Furthermore, it is difficult to provide real-time conversation support or suggest appropriate information, making it difficult to improve the user experience.

[0578] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0579] In this invention, the server includes means for acquiring biometric authentication data of a user, means for transmitting the acquired biometric authentication data to a data processing device, means for matching the biometric authentication data with a data storage area in the data processing device to search for matching identification information and transmitting the matching identification information to a terminal device, means for providing the user with the identification information transmitted from the data processing device, means for collecting acoustic data and extracting keywords in real time, means for acquiring related conversation histories and common interests from the data processing device based on the extracted keywords, means for notifying the user of the acquired information, means for periodically collecting the user's electronic information disclosure data and storing it in the identification information, means for analyzing the collected electronic information disclosure data to extract the user's hobbies and interests, means for generating appropriate proposal information based on the extracted information and transmitting it to the terminal device, means for notifying the user of the transmitted proposal information, means for the user to input basic identification information when using the system for the first time, means for setting privacy and selecting partners with whom to share information, and means for storing the set permission information in the data storage area and associating it with the identification information. This enables personalized communication support for the user, efficient information proposals, and privacy management.

[0580] "Biometric data" is data used to identify an individual based on physical characteristics such as the user's face, fingerprint, or iris.

[0581] "Data Processing Device" means a computer system for processing and analyzing data obtained from a user.

[0582] The "data storage area" is a storage device for storing data such as user identification information and past conversation history.

[0583] An "end device" is a device that can be directly operated by a user, specifically a terminal such as a smartphone, tablet, or PC.

[0584] "Identification information" is data that includes a user's name, contact information, past conversation history, interests, preferences, etc.

[0585] "Acoustic data" is data that collects audio information such as the user's voice and surrounding sounds.

[0586] "Keywords" are important words or phrases extracted from the acoustic data.

[0587] "Electronic information public data" refers to online information that users have made public, such as on social media sites and blogs.

[0588] "Suggested information" is information that is generated by the data processing device based on the user's hobbies and interests, and is useful information for the user.

[0589] "Privacy settings" are settings that determine the scope of information sharing and permission information, which are set by the user when using the system for the first time.

[0590] "Permission information" is setting information that allows a user to permit information sharing with specific people or organizations.

[0591] The present invention provides a system for supporting user communication and deepening interpersonal relationships. This system integrates functions such as face recognition, voice analysis, collection and analysis of social networking site information, and privacy management. Specific embodiments are described below.

[0592] Facial Recognition and Profile Management

[0593] First, the biometric data of the user is acquired. The camera of the terminal is used to capture the user's face in real time, and facial feature points are extracted using a facial recognition algorithm (e.g., OpenCV). The acquired biometric data is then sent from the terminal to a data processing device.

[0594] The data processing device matches the received biometric data with its data storage to find matching identifying information, which may include the user's name, contact details, past conversation history, interests, and preferences. The identifying information is then sent from the data processing device to the end device and provided to the user.

[0595] As a specific example, when user A starts face recognition using a terminal, the camera recognizes user A's face and sends biometric authentication data to the data processing device. The data processing device compares it with the data storage area, identifies user A's identification information, sends it to the terminal, and displays it to user A.

[0596] Audio collection and real-time analysis

[0597] Next, the device collects the user's speech in real time through a microphone, and the collected acoustic data is converted into text through a speech analysis module (e.g., Google Cloud Speech-to-Text) to extract important keywords.

[0598] The extracted keywords are sent from the terminal to a data processing device, which searches the data storage area for past conversation history and related information. The search results are sent to the terminal and notified to the user.

[0599] For example, if User B starts talking about "project progress" during a conversation, the device recognizes this keyword and sends it to the data processing device. The data processing device searches for past conversation history related to "project progress" and sends it to the device to notify User B.

[0600] SNS information collection and suggestions

[0601] The data processing device also periodically collects the user's public electronic information data using an API (e.g., Twitter API or Facebook Graph API). The collected public electronic information data is stored in a data storage area and analyzed. The analysis allows the user's hobbies and interests to be extracted.

[0602] The extracted information is sent from the data processing device to the terminal device at a time appropriate for the device and provided to the user as suggested information. For example, if User C posts on SNS that he or she has recently become interested in sports, the data processing device collects and analyzes this information. The next time a topic related to sports comes up during User C's conversation, the device will remind him or her by saying, "You've been posting about sports recently."

[0603] User Privacy Management

[0604] The system also places great importance on protecting users' privacy. When users first use the system, they can enter basic identification information and configure their privacy settings, which allow them to choose with whom they want to share their information.

[0605] The permission information is saved in the data storage area and associated with the profile. This prevents information from being shared without permission. For example, when User D starts the system for the first time, he or she can set it up so that information is shared only with family members and specific business partners.

[0606] Example of a generated AI prompt

[0607] Using a generative AI model, we can generate prompts like this:

[0608] Facial Recognition and Profile Management:

[0609] "Please explain in detail how your system works to obtain a user's facial recognition data and provide profile information."

[0610] Audio collection and real-time analysis:

[0611] "Please explain in detail how the system works, collecting audio in real time, analyzing important keywords, and notifying users."

[0612] Social media information collection and suggestions:

[0613] "Please explain in detail how the system works to collect social media information and suggest appropriate information to users."

[0614] User Privacy Controls:

[0615] "Please explain in detail how your system manages user privacy."

[0616] As described above, the present invention provides specific means for supporting communication between users and deepening human relationships.

[0617] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0618] Step 1:

[0619] The device captures the user's biometric authentication data using a camera. At this time, it uses a facial recognition algorithm (e.g., OpenCV) to extract facial feature points. The input to this process is real-time video captured by the camera, and the output is facial recognition data with extracted feature points.

[0620] Specifically, the device's camera is activated, the user's face is detected, and feature points are extracted. This facial recognition data is then stored in temporary memory.

[0621] Step 2:

[0622] The device transmits the acquired biometric authentication data, including facial recognition data, to the server.

[0623] Specifically, the device sends facial recognition data to a server via the Internet, and once the transmission is complete, the data is stored on the server.

[0624] Step 3:

[0625] The server compares the received biometric data against a database to find matching identification information. The input to this process is the facial recognition data sent by the device, and the output is the matching profile information.

[0626] Specifically, the server matches the facial recognition data with a data store to identify corresponding identifying information, including the user's name, contact details, past conversation history, interests, and preferences.

[0627] Step 4:

[0628] The server transmits the acquired identification information to the terminal, and the transmitted data includes the user's profile information.

[0629] Specifically, the server converts the identification information into packets and sends them to the terminal via the Internet. The terminal receives this data.

[0630] Step 5:

[0631] The terminal provides the user with the identification information sent by the server. The input to this process is the profile information sent by the server, and the output is the display of the information to the user.

[0632] Specifically, the device displays profile information on the screen, allowing users to check their own name and past conversation history.

[0633] Step 6:

[0634] The device collects the user's conversation content in real time through a microphone. The input of this process is the user's voice, and the output is acoustic data.

[0635] Specifically, the device's microphone captures the user's voice and stores it as audio data, which is then temporarily stored in the device's internal memory.

[0636] Step 7:

[0637] The terminal uses a speech analysis module to convert the collected acoustic data into text and extract important keywords. The input of this process is the collected acoustic data, and the output is the extracted keywords.

[0638] Specifically, the device calls a speech analysis module such as Google Cloud Speech-to-Text to convert the voice data into text, and then extracts important keywords from the converted text.

[0639] Step 8:

[0640] The terminal transmits the extracted keywords to the server. The input of this process is the extracted keywords, and the output is data transmission to the server.

[0641] Specifically, the device sends the keyword to a server via the Internet, where the data is temporarily stored.

[0642] Step 9:

[0643] The server searches the database for related conversation histories and common interests based on the received keywords. The input to this process is the extracted keywords, and the output is related information.

[0644] Specifically, the server uses keywords to search the data storage for relevant conversation histories and common interests, and once the search results are identified, they are stored in temporary memory.

[0645] Step 10:

[0646] The server transmits the retrieved related information to the terminal. The input of this process is the retrieved related information, and the output is data transmission to the terminal.

[0647] Specifically, the server converts the relevant information into packets and sends them to the terminal via the Internet, which then receives the data.

[0648] Step 11:

[0649] The terminal notifies the user of the relevant information sent from the server. The input of this process is the relevant information sent from the server, and the output is a notification to the user.

[0650] Specifically, the device will display relevant information on the screen or notify the user via voice, allowing the user to check this information in real time.

[0651] (Application example 1)

[0652] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0653] Conventional advertising display systems have had difficulty effectively providing personalized advertisements based on users' interests. Furthermore, they have been unable to properly utilize users' profile information and social networking site activity to display highly relevant advertisements in real time, making it difficult to attract users' attention and reducing advertising effectiveness. The present invention aims to solve these problems.

[0654] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0655] In this invention, the server includes means for acquiring facial recognition data of a user, means for transmitting the acquired facial recognition data to the server, means for the server to compare the facial recognition data with a database and search for matching profile information and transmit the matching profile information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation history and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, means for personalizing advertisement content displayed on the terminal based on the user's profile information and SNS information, and means for analyzing the user's interests using the acquired profile information and SNS information and displaying relevant advertisements. This enables effective provision of personalized advertisements based on the user's interests.

[0656] - "User's facial recognition data" means data that captures the user's facial features as digital information and enables identification.

[0657] The "server" is a computer system that manages users' facial recognition data and profile information, and collates and searches data in real time.

[0658] "Profile information" is data that includes personal information such as a user's name, contact details, past conversation history, interests and preferences.

[0659] A "terminal" is a device that has an interface with a user and displays or notifies information.

[0660] "Voice data" refers to data that records the contents of a user's conversation in digital format.

[0661] "Keyword extraction" is the process of analyzing and extracting important words and phrases from audio data in real time.

[0662] "Related conversation history" is data obtained by searching for past conversation content and related information based on keywords.

[0663] "SNS information" refers to information that a user makes public on a social networking service.

[0664] "Personalization" is the process of customizing advertisements and information based on individual user profile information and social media information.

[0665] "Advertising display" is the process of displaying advertisements on a device based on the user's interests.

[0666] To implement this invention, the following steps are required: The user installs a dedicated smartphone application, enters basic profile information when the application is first launched, and faces the camera to obtain facial recognition data. The application collects the user's facial recognition data and sends it to a server. At this time, it performs facial recognition using a library such as OpenCV. The server compares the received facial recognition data with a database, searches for matching profile information, and sends it to the terminal.

[0667] The device then displays the profile information and begins collecting voice data. As the user begins to talk, voice data is collected in real time and keyword analysis is performed. This analysis uses a voice recognition library and natural language processing tools. The analyzed keywords are sent to the server, which searches for relevant information based on past conversation history and the user's interests and sends it to the device. The device notifies the user of this and helps the conversation proceed smoothly.

[0668] The server also periodically collects social media information and stores it in the user's profile. A data analysis module is used to analyze this social media information and extract the user's hobbies and interests. The analysis results are sent to the device, and suggestions tailored to the user are generated. This suggestion information is displayed as personalized advertisements based on the user's profile information and social media information.

[0669] For example, if a user posts on social media that they are looking for new running shoes, the server will retrieve and analyze this information. The next time the user uses the app, an advertisement for related running shoes will be displayed. In this way, information tailored to the user's interests is provided in real time.

[0670] Example prompt sentence:

[0671] "The user posted on social media that they were interested in new running shoes, so please show them an appropriate ad."

[0672] The hardware required to realize this system is a smartphone camera and microphone, and the software used includes the OpenCV library, speech recognition library, natural language processing tools, data analysis modules, server APIs, etc. This makes it possible to accurately capture user interests and display personalized advertisements.

[0673] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0674] Step 1:

[0675] A user launches a smartphone application and enters profile information. When the user launches the application for the first time, they enter basic information such as their name, contact details, and interests, and then face the camera. The device acquires facial recognition data and sends it to the server along with the acquired profile information. The input is the user's basic information and facial recognition data, and the output is the profile data sent to the server.

[0676] Step 2:

[0677] The server checks the received facial recognition data against a database to find matching profile information. It compares the facial recognition data with existing data in the database to identify the best matching profile information. The input is the facial recognition data, and the output is the matching profile information. This information is then sent back to the device.

[0678] Step 3:

[0679] The device displays the profile information received from the server to the user. This profile information includes the user's name, interests, past conversation history, etc. Based on the displayed profile information, the user can check for more detailed information. The input is the matching profile information, and the output is the information display to the user.

[0680] Step 4:

[0681] When a user starts a conversation, the device collects the conversation content as voice data in real time. Using a voice recognition library, important keywords are extracted from the collected voice data. The input is the user's voice data, and the output is the extracted keywords.

[0682] Step 5:

[0683] The extracted keywords are sent to the server, which searches for related information based on past conversation history and related interests. The server searches a database based on the extracted keywords to obtain related conversation history and common interests. The input is the keywords, and the output is related information. This related information is sent to the terminal and notified to the user.

[0684] Step 6:

[0685] The server periodically collects SNS information and stores it in the user's profile. The server obtains SNS information and analyzes the user's current interests and concerns from that information. The input is SNS information, and the output is analyzed interest and concern information.

[0686] Step 7:

[0687] The server generates personalized suggestion information based on the acquired interest information and sends it to the device. The device receives the suggestion information and displays it to the user. The input is the analyzed interest information, and the output is personalized suggestion information provided to the user.

[0688] Step 8:

[0689] Based on the user's profile information and social media information, the device generates and displays personalized advertisements. The advertisement display system selects the most suitable advertisement and provides it to the user in real time. The input is the profile information and social media information, and the output is the displayed advertisement.

[0690] In this way, the invention utilizes a user's facial recognition data, voice data, and social media information to provide a personalized experience based on their individual profile information.

[0691] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0692] This invention is a system that combines facial recognition, voice analysis, collection and analysis of SNS information, privacy management, and an emotion engine that recognizes user emotions in order to support user communication and deepen human relationships.

[0693] System Components

[0694] 1. Facial Recognition and Profile Management

[0695] The system acquires a user's facial recognition data and manages profile information based on it. When a user uses the system, facial recognition data is acquired through the device's camera and sent to the server. The server compares the data with a database to identify matching profile information and sends it to the device. This profile information includes the user's name, contact information, past conversation history, interests, and preferences. The device then provides this information to the user.

[0696] As a concrete example, when User A starts the system, the device's camera recognizes User A's face and sends the facial recognition data to the server. The server searches the database, identifies the profile information of "User A," and sends it to the device. The device then provides the information to User A by displaying it or by voice.

[0697] 2. Audio collection and real-time analysis

[0698] This system has the ability to collect and analyze conversational audio in real time. The device collects the user's conversation content as audio data and extracts important keywords through the audio analysis module. This data is sent to a server, which then searches for past conversation history and related information and sends it to the device. The device then notifies the user of this information and supports the progress of the conversation.

[0699] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will then search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[0700] 3. Collecting and suggesting SNS information

[0701] Furthermore, this system collects information publicly shared by users on social media and uses that information to make suggestions that are appropriate for the user. The server periodically collects social media information and saves it in a profile. The collected information is analyzed to extract the user's hobbies and interests, which are then sent to the device as suggested information at the appropriate time.

[0702] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[0703] 4. Privacy Management

[0704] This system allows users to configure settings to protect their privacy. When using the system for the first time, users select with whom they want to share their information and set permission information. This prevents information from being shared without permission.

[0705] For example, when User D starts the system for the first time, he or she sets permission for information sharing. User D sets the system to share information only with family members and specific business partners.

[0706] 5. Introducing the Emotion Engine

[0707] This system incorporates an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's voice and facial expressions in real time and collects the data. The collected emotion data is sent to a server and associated with the user's profile. The server analyzes the emotion data and provides appropriate information and responses based on the user's emotions.

[0708] As a concrete example, if user E expresses negative emotions during a conversation with a friend, the emotion engine recognizes the emotion and sends emotional data to the server. Based on that data, the server sends appropriate suggestions to the device, such as "switch to a more relaxing topic" or "send a sympathetic message." The device then reminds user E of these suggestions, helping to keep the conversation flowing smoothly.

[0709] In this way, this system not only incorporates facial recognition, voice analysis, collection and analysis of SNS information, and privacy management, but also an emotion engine, allowing it to respond to a variety of communication situations for users and help build deeper human relationships.

[0710] The processing flow will be explained below.

[0711] Processing steps of the facial recognition and conversation reminder system (including emotion engine)

[0712] 1. User registration and license information settings

[0713] Step 1: Register a basic user profile

[0714] Device: The application launches and displays a screen that captures the user's facial recognition data using the camera. The user then enters basic profile information such as name, contact details, and social media accounts.

[0715] Server: Receives facial recognition data and basic profile information sent from the device and stores it in a database.

[0716] Step 2: Set up your license information

[0717] On your device: Display the privacy settings screen and give you the option to choose who you want to share information with.

[0718] Users: Choose who you share information with and set your privacy preferences.

[0719] Server: Receives the permission information, associates it with the profile, and stores it in a database.

[0720] 2. Facial Recognition and Information Extraction

[0721] Step 1: Obtaining facial recognition data

[0722] Device: Uses the camera to recognize the user's face and collect facial recognition data.

[0723] Step 2: Send and match facial recognition data

[0724] Terminal: Sends the acquired facial recognition data to the server.

[0725] Server: Matches the facial recognition data against a database to find matching profile information.

[0726] Step 3: Provide profile information

[0727] Server: Generates relevant profile information (such as name, past conversation history, hobbies, etc.) and sends it to the device.

[0728] Terminal: Provides profile information to the user by display or audio.

[0729] 3. Real-time conversation reminder function

[0730] Step 1: Recognizing voice input

[0731] Device: Uses a microphone to collect audio of conversations in real time.

[0732] Step 2: Keyword extraction

[0733] Terminal: The speech analysis module extracts important keywords and conversation topics from the collected speech.

[0734] Step 3: View your conversation history

[0735] Device: Sends the extracted keywords to the server to obtain related past conversation history and common interests.

[0736] Server: Searches for relevant information from a database based on the received keywords and sends it to the terminal.

[0737] Step 4: View Reminder Information

[0738] Device: Displays reminder information to the user in real time. For example, it can present information based on topics that have been discussed in the past or the other person's hobbies and preferences.

[0739] 4. Emotion Recognition by Emotion Engine

[0740] Step 1: Collecting emotion data

[0741] Device: Uses a camera and microphone to collect emotional data from the user's facial expressions and voice.

[0742] Step 2: Analyze the emotion data

[0743] Terminal: The emotion engine analyzes the user's emotional state from the collected data.

[0744] Step 3: Send and store emotion data

[0745] Terminal: Transmits the analyzed emotion data to the server.

[0746] Server: Stores the received emotion data in a database and associates it with a profile.

[0747] Step 4: Emotionally informed

[0748] Server: Generates appropriate information and suggestions based on the user's emotional state and sends them to the device.

[0749] Device: Displays and notifies users of emotion-based reminders and appropriate responses.

[0750] 5. Referencing and suggesting public information on social media

[0751] Step 1: Collect social media information

[0752] Server: Periodically collects users' public social media information via API with their permission and stores it in their profile.

[0753] Step 2: Analyze social media information

[0754] Server: Analyzes collected SNS information and extracts users' latest interests and hobbies.

[0755] Step 3: Generate proposals

[0756] Server: Based on the analysis results, it generates appropriate proposal information and sends it to the device.

[0757] On your device: Providing suggested information to you, such as a reminder to "Talk about a new interest you recently shared on social media."

[0758] 6. Privacy Management

[0759] Step 1: Provide privacy settings

[0760] Device: Provide users with a privacy settings screen and allow them to select who they want to share information with.

[0761] User: Set up sharing of information with specific people (for example, family members or coworkers).

[0762] Step 2: Save your permission information

[0763] Server: Receives the configured permission information, stores it in a database, and associates it with the profile.

[0764] Through these steps, this system integrates a wide range of functions, including facial recognition, voice analysis, emotion recognition, collection and analysis of SNS information, and privacy management, thereby providing users with valuable information and enabling smooth communication.

[0765] Example 2

[0766] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0767] Current communication technologies and interaction systems have difficulty accurately understanding and responding appropriately to users' individuality and emotions. Therefore, systems that can improve the quality of communication with users and build deeper human relationships are needed. Furthermore, existing systems lack the functionality to reflect users' social media information and real-time emotions and make personalized suggestions. Furthermore, privacy protection settings are difficult to understand, and data permission management is insufficient.

[0768] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring face recognition data of a user, means for transmitting the acquired face recognition data to the server, means for comparing the face recognition data with a database in the server and searching for matching profile information and transmitting the matching profile information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation histories and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, emotion recognition means for analyzing the user's voice and facial expressions in real time, means for transmitting the analyzed emotion data to the server, and means for providing appropriate information and reactions based on the emotion data. This makes it possible to provide and suggest appropriate information to individual users in real time according to their emotions and interests, thereby realizing deeper communication with users.

[0769] "Facial recognition data" is feature data obtained by analyzing a facial image of a user captured using a device such as a camera.

[0770] A "server" is a computer system that processes, stores, and manages data over a network.

[0771] A "database" is an information system that allows data to be stored, managed, and searched in an organized manner.

[0772] "Profile information" is a collection of data that includes personal information, history, interests, etc. about a user.

[0773] "Voice data" refers to the recording and analysis of user-generated voice as digital data.

[0774] "Keywords" are important words or phrases extracted from the user's conversation.

[0775] "Conversation history" is a record of past conversations and statements.

[0776] "SNS public information" refers to information that a user has set to be public on a social networking service.

[0777] "Privacy settings" are settings that allow users to define the scope and permission conditions for sharing personal information.

[0778] "Emotion recognition means" refers to technology or devices that analyze and recognize the user's emotional state from their voice and facial expressions.

[0779] "Notification means" is a function for notifying the user of information or suggestions through a terminal or device.

[0780] This invention is a system that supports user communication and deepens human relationships. This system combines face recognition, voice analysis, collection and analysis of SNS information, privacy management, and an emotion engine that recognizes user emotions.

[0781] composition

[0782] This system mainly consists of the following components: The device is equipped with a camera and microphone, which collect data. The data is processed on the server, and appropriate feedback is provided to the user using the information stored in the database.

[0783] Hardware and software used

[0784] 1. Terminal: A device such as a smartphone, tablet, or PC. It uses a camera module to take pictures and a microphone to collect audio.

[0785] 2. Server: A computer system that processes data and manages databases.

[0786] 3. Facial recognition software: Facial recognition library such as OpenCV, dlib.

[0787] 4. Speech analysis software: A speech recognition library such as the Google Speech-to-Text API.

[0788] 5. Emotion recognition software: Use IBM Watson Tone Analyzer, Microsoft Azure Face API, etc.

[0789] 6. Database: A relational database such as MySQL.

[0790] Program processing

[0791] 1. Facial Recognition and Profile Management

[0792] When a user starts the system, a facial image is captured using the device's camera.

[0793] The device inputs the captured facial image into a facial recognition library (e.g., OpenCV) to generate facial recognition data.

[0794] The recognized face data is sent to a server, which searches a database to identify the corresponding profile information.

[0795] The server sends profile information, which the terminal then displays or audibly presents to the user.

[0796] As a concrete example, when User A starts the system, the device's camera recognizes User A's face and sends the data to the server. The server searches the database, identifies the profile information of "User A," and sends it to the device. The device then provides the information to User A by displaying it or by voice.

[0797] 2. Audio collection and real-time analysis

[0798] The terminal uses a microphone to collect the user's conversational voice.

[0799] The collected voice data is converted into text using a voice analysis module (e.g., Google Speech-to-Text API).

[0800] Important keywords are extracted from the converted text and sent to the server.

[0801] The server searches past conversation history and related information and sends it to the terminal.

[0802] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[0803] 3. Collecting and suggesting SNS information

[0804] The server periodically collects public information from the social media accounts authorized by the user.

[0805] The collected SNS information is analyzed using analytical tools (e.g., TensorFlow or PyTorch) to extract users' hobbies and interests.

[0806] The extracted information is stored in a profile and suggested information is sent to the device at the appropriate time.

[0807] The terminal notifies the user of the suggestion information.

[0808] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[0809] 4. Privacy Management

[0810] When users first use the system, they set their privacy preferences and choose who they want to share their information with.

[0811] The terminal transmits the set permission information to the server.

[0812] The server stores the permission information in a database and associates it with a profile.

[0813] As a specific example, when User D starts the system for the first time, he / she sets permission for information sharing. User D sets the system so that information is shared only with family members and specific business partners.

[0814] 5. Introducing the Emotion Engine

[0815] The device recognizes emotions in real time from the user's voice and facial expressions (for example, using IBM Watson Tone Analyzer or Microsoft Azure Face API).

[0816] The recognized emotion data is sent to the server.

[0817] The server analyzes the emotion data and associates it with the user's profile.

[0818] Data is sent to the terminal that provides appropriate information and responses based on the user's emotions.

[0819] As a concrete example, if user E expresses negative emotions during a conversation with a friend, the emotion engine recognizes the emotion and sends emotional data to the server. Based on that data, the server sends appropriate suggestions to the device, such as "switch to a more relaxing topic" or "send a sympathetic message." The device then reminds user E of these suggestions, helping to keep the conversation flowing smoothly.

[0820] In this way, this system not only incorporates facial recognition, voice analysis, collection and analysis of SNS information, and privacy management, but also an emotion engine, allowing it to respond to a variety of communication situations for users and help build deeper human relationships.

[0821] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0822] Step 1:

[0823] The user starts the system. The device camera captures the user's facial image. Input: User's facial image. Output: Facial recognition data.

[0824] The device uses a facial recognition library (e.g., OpenCV or dlib) to analyze the captured facial image and generate facial recognition data, which is represented as template data with unique features.

[0825] Step 2:

[0826] The facial recognition data generated by the device is sent to the server. Input: Facial recognition data. Output: Identified profile information.

[0827] The server then matches the received facial recognition data with a database containing user profile information (such as name, contact details, past conversation history, interests, and preferences) to find matching profiles.

[0828] Step 3:

[0829] The server sends the profile information to the device. Input: Identified profile information. Output: Provided profile information.

[0830] The device provides the profile information sent from the server to the user, which can be displayed on the screen or read aloud by a voice assistant.

[0831] Step 4:

[0832] The device uses a microphone to collect the user's conversational voice in real time. Input: User's voice data. Output: Text-converted voice data.

[0833] The voice data is converted into text through a voice analysis module (e.g., Google Speech-to-Text API), and the converted result is used as text.

[0834] Step 5:

[0835] The device extracts important keywords from the text-converted voice data. Input: Text-converted voice data. Output: Extracted keywords.

[0836] The natural language processing function of the speech analysis module is used to extract important keywords from the text. For example, "project progress" is extracted as an important keyword.

[0837] Step 6:

[0838] Based on important keywords, related conversation history and common interests are retrieved from the server. Input: Extracted keywords. Output: Related information.

[0839] The server searches the database based on the extracted keywords to retrieve information related to past conversation history and common interests, which is implemented using a text search algorithm.

[0840] Step 7:

[0841] Notify the user of the acquired information. Input: Relevant information. Output: Notification to the user.

[0842] The device notifies the user of the acquired related information, which can be displayed on the screen or given verbally by a voice assistant.

[0843] Step 8:

[0844] The server periodically collects the user's SNS public information and saves it in the profile. Input: SNS public information. Output: Updated profile information.

[0845] We use collection tools to automatically retrieve public information from users' social media accounts, using the social media APIs (e.g., Twitter API and Facebook Graph API).

[0846] Step 9:

[0847] Analyze collected SNS information to extract user hobbies and interests. Input: SNS information. Output: Extracted hobbies and interests.

[0848] Analyze social media information using analytical tools (e.g., TensorFlow or PyTorch), thereby probabilistically extracting users' hobbies and interests.

[0849] Step 10:

[0850] Based on the extracted information, appropriate suggestion information is generated and sent to the device. Input: Extracted information on hobbies and interests. Output: Suggestion information.

[0851] The server generates suggested information based on the extracted hobbies and interests. For example, if the user is interested in "sports," it will suggest related events and news.

[0852] Step 11:

[0853] Notify the user of the submitted proposal information. Input: Proposal information. Output: Notification to the user.

[0854] The device notifies the user of the received suggestion information by means of a screen display, a pop-up message, or guidance via a voice assistant.

[0855] Step 12:

[0856] When a user first uses the system, they set their privacy preferences. Input: User's privacy preferences information. Output: Privacy preferences data.

[0857] The device receives user input and generates privacy setting data, allowing the user to select the scope of information sharing and the people with whom the information is to be shared.

[0858] Step 13:

[0859] The device sends the generated privacy setting data to the server. Input: Privacy setting data. Output: Saved permission information.

[0860] The server stores the received privacy setting data in a database and associates it with the profile.

[0861] Step 14:

[0862] The device analyzes the user's voice and facial expressions in real time. Input: User's voice and facial expression data. Output: Recognized emotion data.

[0863] Emotion recognition software (e.g., IBM Watson Tone Analyzer or Microsoft Azure Face API) is used to analyze the user's emotions from the collected voice and facial expressions.

[0864] Step 15:

[0865] The analyzed emotion data is sent to the server. Input: Recognized emotion data. Output: User's emotion information.

[0866] The server receives the emotion data and processes it to associate it with a profile.

[0867] Step 16:

[0868] Implement measures to provide appropriate information and responses based on emotional data. Input: User's emotional information. Output: Appropriate responses and suggestions.

[0869] The server generates appropriate information and responses based on the user's emotional data and sends them to the device, which then notifies the user, improving the quality of communication.

[0870] (Application example 2)

[0871] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0872] Existing communication support systems do not take into account the user's emotions or mood state, and therefore provide uniform information and suggestions, making it impossible to provide a personalized experience. Furthermore, there is a need for systems that not only collect and analyze social media information and profile information, but also perform real-time emotion recognition to suggest optimal content according to the user's emotions. This has led to challenges in improving entertainment experiences and deepening human relationships.

[0873] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0874] In this invention, the server includes means for acquiring facial recognition data of a user, means for transmitting the acquired facial recognition data to the server, means for the server to compare the facial recognition data with a database and search for matching profile information and transmit the searched information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation history and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, means for recognizing emotion data from the user's voice and facial expressions using an emotion engine and transmitting the emotion data to the server, and means for the server to analyze the emotion data, associate the emotion data with the profile, generate appropriate information and reactions, and transmit the information and reactions to the terminal. This enables personalized entertainment content to be proposed that takes the user's emotions into consideration.

[0875] "Facial recognition data" is data that captures a user's face in digital form and analyzes its features.

[0876] A "server" is a computer system that provides and manages data on a network and exchanges information with client terminals.

[0877] "Profile information" is a collection of information related to a particular user, including data such as name, contact information, conversation history, hobbies and interests, etc.

[0878] A "terminal" is a device that is directly operated by a user and displays or provides information via audio, such as a smartphone or smart glasses.

[0879] "Voice data" refers to data that is used to record and analyze user utterances and conversation content in digital form.

[0880] "Keywords" are important words or phrases extracted from audio data and used to identify themes and key points of a conversation.

[0881] "Conversation history" is data that records the contents of past conversations, and is used to understand the flow of communication between users.

[0882] "SNS information" refers to information about posts and activities made public by users on social networking services.

[0883] An "emotion engine" is a system or software that analyzes emotional data from a user's voice and facial expressions and recognizes their state in real time.

[0884] "Suggested information" refers to information or content suggestions that the system provides to the user, and is generated based on the user's profile information and emotional state.

[0885] "Privacy settings" are settings that allow users to decide with whom they want to share their information and to what extent they want to make it public.

[0886] "Permission information" is data that records the content and scope of information handling that a user has approved.

[0887] This invention supports user communication and appropriately suggests entertainment content by building a system that uses an emotion engine in addition to facial recognition, voice analysis, collection and analysis of SNS information, and privacy management.

[0888] Hardware and Software Configuration

[0889] 1. Obtaining the user's facial recognition data

[0890] Hardware: Smart glasses and smartphone cameras

[0891] Software: Facial recognition API (e.g., Amazon Rekognition)

[0892] Description: When a user turns on the device, the camera captures facial recognition data and sends it to a server, which then matches the data with a database to find matching profile information and send it to the device.

[0893] 2. Collection and analysis of audio data

[0894] Hardware: Smart glasses and smartphone microphones

[0895] Software: Speech analysis API (e.g., Google Cloud Speech-to-Text)

[0896] Description: When a user starts talking, the microphone collects voice data and extracts keywords in real time. The extracted keywords are sent to a server, which searches past conversation history and related information and provides them to the device.

[0897] 3. Introducing an Emotion Engine

[0898] Hardware: Smart glasses and smartphone cameras and microphones

[0899] Software: Sentiment analysis API (e.g. IBM Watson Tone Analyzer)

[0900] Description: Analyzes emotional data from voice and facial expressions in real time. Emotional data is associated with a profile and generates appropriate information and responses based on the user's state.

[0901] 4. Collecting and suggesting SNS information

[0902] Software: Social Media API (e.g. Twitter API, Facebook Graph API)

[0903] Description: The server periodically collects the user's social media information and saves it in the user's profile. It analyzes the collected information to extract hobbies and interests. Based on this, it generates suitable suggestions and sends them to the device.

[0904] 5. Privacy Settings

[0905] Software: Privacy Management API (e.g., OAuth 2.0)

[0906] Description: When a user first uses the system, they set their preferences regarding who they want to share their information with and how their emotional data is handled, and this information is stored in a database and associated with their profile.

[0907] 6. Content Suggestion

[0908] Description: The server suggests entertainment content based on the user's profile information and emotional data. For example, if the user is feeling stressed, it will suggest a list of relaxing movies. This is done by analyzing the user's state from social media information and past data.

[0909] Specific examples

[0910] When a user puts on the smart glasses and launches the content recommendation system, their profile is loaded using facial recognition. Based on the user's conversation and facial expressions, the system analyzes emotional data in real time and combines it with social media information to suggest appropriate entertainment content.

[0911] Prompt Sentence Examples

[0912] Generate a list of relaxing movies to suggest when the user is feeling stressed. Read the user's recent emotions from their social media posts and refer to their profile information. Choose the latest comedy movies as suggestions.

[0913] The above is an embodiment of the present invention, which allows users to have a personalized entertainment experience based on their emotions and interests.

[0914] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0915] Step 1:

[0916] Acquiring facial recognition data and loading profile information

[0917] When a user turns on their smart glasses or smartphone, the device's camera captures the user's facial recognition data.

[0918] Input: Video of user's face

[0919] Data processing: The raw data collected by the camera is processed by a facial recognition API (e.g., Amazon Rekognition) to produce quantified facial recognition data.

[0920] Output: Facial recognition data

[0921] The server receives the facial recognition data sent from the device and compares it with a database.

[0922] Input: Facial recognition data

[0923] Data calculation: Compare with existing facial recognition data in the database to find matching profile information.

[0924] Output: Profile information

[0925] The profile information is transmitted to the terminal and provided to the user.

[0926] Step 2:

[0927] Audio data collection and analysis

[0928] When a user starts talking, the device's microphone collects audio data.

[0929] Input: User's voice

[0930] Data processing: The audio data is converted into text format using a speech analysis API (e.g., Google Cloud Speech-to-Text).

[0931] Output: Speech-to-text data

[0932] The server analyzes the text data and extracts keywords.

[0933] Input: Speech-to-text data

[0934] Data calculations: The text analysis module identifies important keywords.

[0935] Output: Keywords

[0936] Based on the extracted keywords, the server obtains past conversation history and common interests and sends them to the terminal.

[0937] Input: keyword

[0938] Data calculation: Search the conversation history database and retrieve relevant information.

[0939] Output: Related information

[0940] Step 3:

[0941] Emotional Data Recognition and Analysis

[0942] Emotional data is collected from the user's facial expressions and voice using the device's camera and microphone.

[0943] Input: User's facial expression and audio

[0944] Data processing: Video and audio data are processed using an emotion analysis API (e.g., IBM Watson Tone Analyzer) and converted into emotion data.

[0945] Output: Emotion data

[0946] The server associates the emotional data with a profile and analyzes it.

[0947] Input: Emotion data

[0948] Data calculations: Emotional data is linked to profile information to analyze current emotional state.

[0949] Output: Emotional state analysis results

[0950] The server generates appropriate information and responses based on the emotional state and sends them to the device.

[0951] Input: Emotional state analysis results

[0952] Data calculation: Generate responses and suggestions based on the user's emotions.

[0953] Output: Proposal information

[0954] Step 4:

[0955] SNS information collection and suggestions

[0956] The server periodically collects the user's social media information and stores it in the profile.

[0957] Input: SNS public information

[0958] Data processing: Collect information using SNS APIs and add it to your profile.

[0959] Output: Updated profile information

[0960] The server analyzes the collected SNS information and extracts the user's hobbies and interests.

[0961] Input: Social Media Information

[0962] Data calculation: Hobbies and interests are identified through an information analysis module.

[0963] Output: Hobbies and interests analysis results

[0964] The server generates proposal information based on this and transmits it to the terminal.

[0965] Input: Hobbies and Interests Analysis Results

[0966] Data calculation: The personalized suggestion generation module makes appropriate content suggestions.

[0967] Output: Proposal information

[0968] Step 5:

[0969] Content suggestion notifications

[0970] The terminal receives the proposal information from the server and notifies the user.

[0971] Input: Proposal information

[0972] Data calculation: Processing is performed to present the received proposal information visually or audibly.

[0973] Output: Notify user of content suggestion

[0974] For example, consider a situation where a user is wearing smart glasses and browsing a list of movies. If the emotion engine detects that the user is stressed, the prompt to suggest "relaxing comedy movies" might be:

[0975] Generate a list of relaxing movies to suggest when the user is feeling stressed. Read the user's recent emotions from their social media posts and refer to their profile information. Choose the latest comedy movies as suggestions.

[0976] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0977] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0978] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0979] [Third embodiment]

[0980] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0981] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0982] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0983] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0984] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0985] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0986] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0987] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0988] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0989] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0990] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0991] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0992] The present invention is a system for supporting user communication and deepening human relationships. This system combines the following various devices and means to achieve smooth interaction with users.

[0993] System Components

[0994] 1. Facial Recognition and Profile Management

[0995] This system acquires a user's facial recognition data and manages profile information based on it. When a user uses the system, facial recognition data is acquired through the device's camera. The acquired data is sent to a server, which checks it against a database to identify matching profile information. This profile information includes the user's name, contact information, past conversation history, interests, and preferences. This information is then sent to the device as appropriate and provided to the user.

[0996] For example, when User A starts the system, the device's camera recognizes User A's face and sends the facial recognition data to the server. The server searches the database, identifies User A's profile information, and sends it to the device. The device then provides that information to User A by displaying it or by voice.

[0997] 2. Audio collection and real-time analysis

[0998] This system has the ability to collect and analyze conversational audio in real time. The device collects the user's conversation content as audio data and extracts important keywords through a voice analysis module. This data is sent to a server, which then searches for past conversation history and related information and sends it to the device. The device notifies the user of this information to help the conversation progress.

[0999] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will then search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[1000] 3. Collecting and suggesting SNS information

[1001] Furthermore, this system collects information publicly shared on social media by users and makes suggestions based on that information. The server periodically collects users' social media information and stores it in their profiles. The collected information is analyzed to extract the user's hobbies and interests. The extracted information is then provided to the user as suggested information at the appropriate time.

[1002] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[1003] 4. User Privacy Management

[1004] This system allows users to configure settings to protect their privacy. When using the system for the first time, users select who they want to share their information with and set permission information. This prevents information from being shared without permission.

[1005] For example, when User D starts the system for the first time, he or she sets permission for information sharing. User D sets the system so that information is shared only with family members and specific business partners.

[1006] In this way, this system combines multiple functions such as facial recognition, voice analysis, collection and analysis of SNS information, and privacy management to support user communication and help deepen human relationships.

[1007] The processing flow will be explained below.

[1008] Program processing steps for face recognition and conversation reminder system

[1009] 1. User registration and license information settings

[1010] Step 1: Register a basic user profile

[1011] On the device: The application is launched and a screen is displayed that captures the user's facial recognition data using the camera. The user then enters basic profile information such as name, contact details, and social media accounts.

[1012] Server: Receives facial recognition data and basic profile information sent from the device and stores it in a database.

[1013] Step 2: Set up your license information

[1014] On your device: Display the privacy settings screen and give the option to choose who you want to share information with.

[1015] Users: Choose who you share your information with and set your privacy preferences.

[1016] Server: Receives the permission information, associates it with the profile, and stores it in a database.

[1017] 2. Facial Recognition and Information Extraction

[1018] Step 1: Obtaining facial recognition data

[1019] Device: Uses the camera to recognize the user's face and collect facial recognition data.

[1020] Step 2: Send and match facial recognition data

[1021] Device: Sends the acquired facial recognition data to the server.

[1022] Server: Matches the facial recognition data against a database to find matching profile information.

[1023] Step 3: Provide profile information

[1024] Server: Generates relevant profile information (such as name, past conversation history, hobbies, etc.) and sends it to the device.

[1025] Terminal: Provides profile information to the user by display or audio.

[1026] 3. Real-time conversation reminder function

[1027] Step 1: Recognizing voice input

[1028] Device: Uses a microphone to collect audio of conversations in real time.

[1029] Step 2: Keyword extraction

[1030] On the device: The speech analysis module extracts important keywords and conversation topics from the collected audio.

[1031] Step 3: View your conversation history

[1032] Device: Sends the extracted keywords to the server to retrieve related past conversation history and common interests.

[1033] Server: Searches for relevant information from a database based on the received keywords and sends it to the device.

[1034] Step 4: View Reminder Information

[1035] Device: Displays reminders to users in real time, such as information based on past topics or the other person's interests.

[1036] 4. Referencing and suggesting public information on social media

[1037] Step 1: Collect social media information

[1038] Server: Periodically collects users' public social media information via API with their permission and stores it in their profile.

[1039] Step 2: Analyze social media information

[1040] Server: Analyzes collected SNS information and extracts users' latest interests and hobbies.

[1041] Step 3: Generate proposals

[1042] Server: Based on the analysis results, generate appropriate proposal information and send it to the device.

[1043] On the device: Notify users of suggested information, such as a reminder to "Talk about a new interest you recently shared on social media."

[1044] 5. Privacy Management

[1045] Step 1: Provide privacy settings

[1046] Device: Provide users with a privacy settings screen and let them choose who they want to share information with.

[1047] User: Set up sharing of information with specific people (for example, family members or coworkers).

[1048] Step 2: Save your permission information

[1049] Server: Receives the configured permission information, stores it in a database, and associates it with a profile.

[1050] These steps of the present system allow users to obtain information that is appropriate for them and achieve smooth communication.

[1051] Example 1

[1052] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1053] Conventional communication support systems are unable to fully utilize the individual user's information, resulting in insufficient personalized support. Furthermore, from the perspective of privacy protection, there are problems with managing and sharing information. Furthermore, it is difficult to provide real-time conversation support or suggest appropriate information, making it difficult to improve the user experience.

[1054] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1055] In this invention, the server includes means for acquiring biometric authentication data of a user, means for transmitting the acquired biometric authentication data to a data processing device, means for matching the biometric authentication data with a data storage area in the data processing device to search for matching identification information and transmitting the matching identification information to a terminal device, means for providing the user with the identification information transmitted from the data processing device, means for collecting acoustic data and extracting keywords in real time, means for acquiring related conversation histories and common interests from the data processing device based on the extracted keywords, means for notifying the user of the acquired information, means for periodically collecting the user's electronic information disclosure data and storing it in the identification information, means for analyzing the collected electronic information disclosure data to extract the user's hobbies and interests, means for generating appropriate proposal information based on the extracted information and transmitting it to the terminal device, means for notifying the user of the transmitted proposal information, means for the user to input basic identification information when using the system for the first time, means for setting privacy and selecting partners with whom to share information, and means for storing the set permission information in the data storage area and associating it with the identification information. This enables personalized communication support for the user, efficient information proposals, and privacy management.

[1056] "Biometric data" is data used to identify an individual based on physical characteristics such as the user's face, fingerprint, or iris.

[1057] "Data Processing Device" means a computer system for processing and analyzing data obtained from a user.

[1058] The "data storage area" is a storage device for storing data such as user identification information and past conversation history.

[1059] An "end device" is a device that can be directly operated by a user, specifically a terminal such as a smartphone, tablet, or PC.

[1060] "Identification information" is data that includes a user's name, contact information, past conversation history, interests, preferences, etc.

[1061] "Acoustic data" is data that collects audio information such as the user's voice and surrounding sounds.

[1062] "Keywords" are important words or phrases extracted from the acoustic data.

[1063] "Electronic information public data" refers to online information that users have made public, such as on social media sites and blogs.

[1064] "Suggested information" is information that is generated by the data processing device based on the user's hobbies and interests, and is useful information for the user.

[1065] "Privacy settings" are settings that determine the scope of information sharing and permission information, which are set by the user when using the system for the first time.

[1066] "Permission information" is setting information that allows a user to permit information sharing with specific people or organizations.

[1067] The present invention provides a system for supporting user communication and deepening interpersonal relationships. This system integrates functions such as face recognition, voice analysis, collection and analysis of social networking site information, and privacy management. Specific embodiments are described below.

[1068] Facial Recognition and Profile Management

[1069] First, the biometric data of the user is acquired. The camera of the terminal is used to capture the user's face in real time, and facial feature points are extracted using a facial recognition algorithm (e.g., OpenCV). The acquired biometric data is then sent from the terminal to a data processing device.

[1070] The data processing device matches the received biometric data with its data storage to find matching identifying information, which may include the user's name, contact details, past conversation history, interests, and preferences. The identifying information is then sent from the data processing device to the end device and provided to the user.

[1071] As a specific example, when user A starts face recognition using a terminal, the camera recognizes user A's face and sends biometric authentication data to the data processing device. The data processing device compares it with the data storage area, identifies user A's identification information, sends it to the terminal, and displays it to user A.

[1072] Audio collection and real-time analysis

[1073] Next, the device collects the user's speech in real time through a microphone, and the collected acoustic data is converted into text through a speech analysis module (e.g., Google Cloud Speech-to-Text) to extract important keywords.

[1074] The extracted keywords are sent from the terminal to a data processing device, which searches the data storage area for past conversation history and related information. The search results are sent to the terminal and notified to the user.

[1075] For example, if User B starts talking about "project progress" during a conversation, the device recognizes this keyword and sends it to the data processing device. The data processing device searches for past conversation history related to "project progress" and sends it to the device to notify User B.

[1076] SNS information collection and suggestions

[1077] The data processing device also periodically collects the user's public electronic information data using an API (e.g., Twitter API or Facebook Graph API). The collected public electronic information data is stored in a data storage area and analyzed. The analysis allows the user's hobbies and interests to be extracted.

[1078] The extracted information is sent from the data processing device to the terminal device at a time appropriate for the device and provided to the user as suggested information. For example, if User C posts on SNS that he or she has recently become interested in sports, the data processing device collects and analyzes this information. The next time a topic related to sports comes up during User C's conversation, the device will remind him or her by saying, "You've been posting about sports recently."

[1079] User Privacy Management

[1080] The system also places great importance on protecting users' privacy. When users first use the system, they can enter basic identification information and configure their privacy settings, which allow them to choose with whom they want to share their information.

[1081] The permission information is saved in the data storage area and associated with the profile. This prevents information from being shared without permission. For example, when User D starts the system for the first time, he or she can set it up so that information is shared only with family members and specific business partners.

[1082] Example of a generated AI prompt

[1083] Using a generative AI model, we can generate prompts like this:

[1084] Facial Recognition and Profile Management:

[1085] "Please explain in detail how your system works to obtain a user's facial recognition data and provide profile information."

[1086] Audio collection and real-time analysis:

[1087] "Please explain in detail how the system works, collecting audio in real time, analyzing important keywords, and notifying users."

[1088] Social media information collection and suggestions:

[1089] "Please explain in detail how the system works to collect social media information and suggest appropriate information to users."

[1090] User Privacy Controls:

[1091] "Please explain in detail how your system manages user privacy."

[1092] As described above, the present invention provides specific means for supporting communication between users and deepening human relationships.

[1093] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1094] Step 1:

[1095] The device captures the user's biometric authentication data using a camera. At this time, it uses a facial recognition algorithm (e.g., OpenCV) to extract facial feature points. The input to this process is real-time video captured by the camera, and the output is facial recognition data with extracted feature points.

[1096] Specifically, the device's camera is activated, the user's face is detected, and feature points are extracted. This facial recognition data is then stored in temporary memory.

[1097] Step 2:

[1098] The device transmits the acquired biometric authentication data, including facial recognition data, to the server.

[1099] Specifically, the device sends facial recognition data to a server via the Internet, and once the transmission is complete, the data is stored on the server.

[1100] Step 3:

[1101] The server compares the received biometric data against a database to find matching identification information. The input to this process is the facial recognition data sent by the device, and the output is the matching profile information.

[1102] Specifically, the server matches the facial recognition data with a data store to identify corresponding identifying information, including the user's name, contact details, past conversation history, interests, and preferences.

[1103] Step 4:

[1104] The server transmits the acquired identification information to the terminal, and the transmitted data includes the user's profile information.

[1105] Specifically, the server converts the identification information into packets and sends them to the terminal via the Internet. The terminal receives this data.

[1106] Step 5:

[1107] The terminal provides the user with the identification information sent by the server. The input to this process is the profile information sent by the server, and the output is the display of the information to the user.

[1108] Specifically, the device displays profile information on the screen, allowing users to check their own name and past conversation history.

[1109] Step 6:

[1110] The device collects the user's conversation content in real time through a microphone. The input of this process is the user's voice, and the output is acoustic data.

[1111] Specifically, the device's microphone captures the user's voice and stores it as audio data, which is then temporarily stored in the device's internal memory.

[1112] Step 7:

[1113] The terminal uses a speech analysis module to convert the collected acoustic data into text and extract important keywords. The input of this process is the collected acoustic data, and the output is the extracted keywords.

[1114] Specifically, the device calls a speech analysis module such as Google Cloud Speech-to-Text to convert the voice data into text, and then extracts important keywords from the converted text.

[1115] Step 8:

[1116] The terminal transmits the extracted keywords to the server. The input of this process is the extracted keywords, and the output is data transmission to the server.

[1117] Specifically, the device sends the keyword to a server via the Internet, where the data is temporarily stored.

[1118] Step 9:

[1119] The server searches the database for related conversation histories and common interests based on the received keywords. The input to this process is the extracted keywords, and the output is related information.

[1120] Specifically, the server uses keywords to search the data storage for relevant conversation histories and common interests, and once the search results are identified, they are stored in temporary memory.

[1121] Step 10:

[1122] The server transmits the retrieved related information to the terminal. The input of this process is the retrieved related information, and the output is data transmission to the terminal.

[1123] Specifically, the server converts the relevant information into packets and sends them to the terminal via the Internet, which then receives the data.

[1124] Step 11:

[1125] The terminal notifies the user of the relevant information sent from the server. The input of this process is the relevant information sent from the server, and the output is a notification to the user.

[1126] Specifically, the device will display relevant information on the screen or notify the user via voice, allowing the user to check this information in real time.

[1127] (Application example 1)

[1128] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1129] Conventional advertising display systems have had difficulty effectively providing personalized advertisements based on users' interests. Furthermore, they have been unable to properly utilize users' profile information and social networking site activity to display highly relevant advertisements in real time, making it difficult to attract users' attention and reducing advertising effectiveness. The present invention aims to solve these problems.

[1130] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1131] In this invention, the server includes means for acquiring facial recognition data of a user, means for transmitting the acquired facial recognition data to the server, means for the server to compare the facial recognition data with a database and search for matching profile information and transmit the matching profile information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation history and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, means for personalizing advertisement content displayed on the terminal based on the user's profile information and SNS information, and means for analyzing the user's interests using the acquired profile information and SNS information and displaying relevant advertisements. This enables effective provision of personalized advertisements based on the user's interests.

[1132] - "User's facial recognition data" means data that captures the user's facial features as digital information and enables identification.

[1133] The "server" is a computer system that manages users' facial recognition data and profile information, and collates and searches data in real time.

[1134] "Profile information" is data that includes personal information such as a user's name, contact details, past conversation history, interests and preferences.

[1135] A "terminal" is a device that has an interface with a user and displays or notifies information.

[1136] "Voice data" refers to data that records the contents of a user's conversation in digital format.

[1137] "Keyword extraction" is the process of analyzing and extracting important words and phrases from audio data in real time.

[1138] "Related conversation history" is data obtained by searching for past conversation content and related information based on keywords.

[1139] "SNS information" refers to information that a user makes public on a social networking service.

[1140] "Personalization" is the process of customizing advertisements and information based on individual user profile information and social media information.

[1141] "Advertising display" is the process of displaying advertisements on a device based on the user's interests.

[1142] To implement this invention, the following steps are required: The user installs a dedicated smartphone application, enters basic profile information when the application is first launched, and faces the camera to obtain facial recognition data. The application collects the user's facial recognition data and sends it to a server. At this time, it performs facial recognition using a library such as OpenCV. The server compares the received facial recognition data with a database, searches for matching profile information, and sends it to the terminal.

[1143] The device then displays the profile information and begins collecting voice data. As the user begins to talk, voice data is collected in real time and keyword analysis is performed. This analysis uses a voice recognition library and natural language processing tools. The analyzed keywords are sent to the server, which searches for relevant information based on past conversation history and the user's interests and sends it to the device. The device notifies the user of this and helps the conversation proceed smoothly.

[1144] The server also periodically collects social media information and stores it in the user's profile. A data analysis module is used to analyze this social media information and extract the user's hobbies and interests. The analysis results are sent to the device, and suggestions tailored to the user are generated. This suggestion information is displayed as personalized advertisements based on the user's profile information and social media information.

[1145] For example, if a user posts on social media that they are looking for new running shoes, the server will retrieve and analyze this information. The next time the user uses the app, an advertisement for related running shoes will be displayed. In this way, information tailored to the user's interests is provided in real time.

[1146] Example prompt sentence:

[1147] "The user posted on social media that they were interested in new running shoes, so please show them an appropriate ad."

[1148] The hardware required to realize this system is a smartphone camera and microphone, and the software used includes the OpenCV library, speech recognition library, natural language processing tools, data analysis modules, server APIs, etc. This makes it possible to accurately capture user interests and display personalized advertisements.

[1149] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1150] Step 1:

[1151] A user launches a smartphone application and enters profile information. When the user launches the application for the first time, they enter basic information such as their name, contact details, and interests, and then face the camera. The device acquires facial recognition data and sends it to the server along with the acquired profile information. The input is the user's basic information and facial recognition data, and the output is the profile data sent to the server.

[1152] Step 2:

[1153] The server checks the received facial recognition data against a database to find matching profile information. It compares the facial recognition data with existing data in the database to identify the best matching profile information. The input is the facial recognition data, and the output is the matching profile information. This information is then sent back to the device.

[1154] Step 3:

[1155] The device displays the profile information received from the server to the user. This profile information includes the user's name, interests, past conversation history, etc. Based on the displayed profile information, the user can check for more detailed information. The input is the matching profile information, and the output is the information display to the user.

[1156] Step 4:

[1157] When a user starts a conversation, the device collects the conversation content as voice data in real time. Using a voice recognition library, important keywords are extracted from the collected voice data. The input is the user's voice data, and the output is the extracted keywords.

[1158] Step 5:

[1159] The extracted keywords are sent to the server, which searches for related information based on past conversation history and related interests. The server searches a database based on the extracted keywords to obtain related conversation history and common interests. The input is the keywords, and the output is related information. This related information is sent to the terminal and notified to the user.

[1160] Step 6:

[1161] The server periodically collects SNS information and stores it in the user's profile. The server obtains SNS information and analyzes the user's current interests and concerns from that information. The input is SNS information, and the output is analyzed interest and concern information.

[1162] Step 7:

[1163] The server generates personalized suggestion information based on the acquired interest information and sends it to the device. The device receives the suggestion information and displays it to the user. The input is the analyzed interest information, and the output is personalized suggestion information provided to the user.

[1164] Step 8:

[1165] Based on the user's profile information and social media information, the device generates and displays personalized advertisements. The advertisement display system selects the most suitable advertisement and provides it to the user in real time. The input is the profile information and social media information, and the output is the displayed advertisement.

[1166] In this way, the invention utilizes a user's facial recognition data, voice data, and social media information to provide a personalized experience based on their individual profile information.

[1167] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1168] This invention is a system that combines facial recognition, voice analysis, collection and analysis of SNS information, privacy management, and an emotion engine that recognizes user emotions in order to support user communication and deepen human relationships.

[1169] System Components

[1170] 1. Facial Recognition and Profile Management

[1171] The system acquires a user's facial recognition data and manages profile information based on it. When a user uses the system, facial recognition data is acquired through the device's camera and sent to the server. The server compares the data with a database to identify matching profile information and sends it to the device. This profile information includes the user's name, contact information, past conversation history, interests, and preferences. The device then provides this information to the user.

[1172] As a concrete example, when User A starts the system, the device's camera recognizes User A's face and sends the facial recognition data to the server. The server searches the database, identifies the profile information of "User A," and sends it to the device. The device then provides the information to User A by displaying it or by voice.

[1173] 2. Audio collection and real-time analysis

[1174] This system has the ability to collect and analyze conversational audio in real time. The device collects the user's conversation content as audio data and extracts important keywords through the audio analysis module. This data is sent to a server, which then searches for past conversation history and related information and sends it to the device. The device then notifies the user of this information and supports the progress of the conversation.

[1175] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will then search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[1176] 3. Collecting and suggesting SNS information

[1177] Furthermore, this system collects information publicly shared by users on social media and uses that information to make suggestions that are appropriate for the user. The server periodically collects social media information and saves it in a profile. The collected information is analyzed to extract the user's hobbies and interests, which are then sent to the device as suggested information at the appropriate time.

[1178] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[1179] 4. Privacy Management

[1180] This system allows users to configure settings to protect their privacy. When using the system for the first time, users select with whom they want to share their information and set permission information. This prevents information from being shared without permission.

[1181] For example, when User D starts the system for the first time, he or she sets permission for information sharing. User D sets the system to share information only with family members and specific business partners.

[1182] 5. Introducing the Emotion Engine

[1183] This system incorporates an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's voice and facial expressions in real time and collects the data. The collected emotion data is sent to a server and associated with the user's profile. The server analyzes the emotion data and provides appropriate information and responses based on the user's emotions.

[1184] As a concrete example, if user E expresses negative emotions during a conversation with a friend, the emotion engine recognizes the emotion and sends emotional data to the server. Based on that data, the server sends appropriate suggestions to the device, such as "switch to a more relaxing topic" or "send a sympathetic message." The device then reminds user E of these suggestions, helping to keep the conversation flowing smoothly.

[1185] In this way, this system not only incorporates facial recognition, voice analysis, collection and analysis of SNS information, and privacy management, but also an emotion engine, allowing it to respond to a variety of communication situations for users and help build deeper human relationships.

[1186] The processing flow will be explained below.

[1187] Processing steps of the facial recognition and conversation reminder system (including emotion engine)

[1188] 1. User registration and license information settings

[1189] Step 1: Register a basic user profile

[1190] Device: The application launches and displays a screen that captures the user's facial recognition data using the camera. The user then enters basic profile information such as name, contact details, and social media accounts.

[1191] Server: Receives facial recognition data and basic profile information sent from the device and stores it in a database.

[1192] Step 2: Set up your license information

[1193] On your device: Display the privacy settings screen and give you the option to choose who you want to share information with.

[1194] Users: Choose who you share information with and set your privacy preferences.

[1195] Server: Receives the permission information, associates it with the profile, and stores it in a database.

[1196] 2. Facial Recognition and Information Extraction

[1197] Step 1: Obtaining facial recognition data

[1198] Device: Uses the camera to recognize the user's face and collect facial recognition data.

[1199] Step 2: Send and match facial recognition data

[1200] Terminal: Sends the acquired facial recognition data to the server.

[1201] Server: Matches the facial recognition data against a database to find matching profile information.

[1202] Step 3: Provide profile information

[1203] Server: Generates relevant profile information (such as name, past conversation history, hobbies, etc.) and sends it to the device.

[1204] Terminal: Provides profile information to the user by display or audio.

[1205] 3. Real-time conversation reminder function

[1206] Step 1: Recognizing voice input

[1207] Device: Uses a microphone to collect audio of conversations in real time.

[1208] Step 2: Keyword extraction

[1209] Terminal: The speech analysis module extracts important keywords and conversation topics from the collected speech.

[1210] Step 3: View your conversation history

[1211] Device: Sends the extracted keywords to the server to obtain related past conversation history and common interests.

[1212] Server: Searches for relevant information from a database based on the received keywords and sends it to the terminal.

[1213] Step 4: View Reminder Information

[1214] Device: Displays reminder information to the user in real time. For example, it can present information based on topics that have been discussed in the past or the other person's hobbies and preferences.

[1215] 4. Emotion Recognition by Emotion Engine

[1216] Step 1: Collecting emotion data

[1217] Device: Uses a camera and microphone to collect emotional data from the user's facial expressions and voice.

[1218] Step 2: Analyze the emotion data

[1219] Terminal: The emotion engine analyzes the user's emotional state from the collected data.

[1220] Step 3: Send and store emotion data

[1221] Terminal: Transmits the analyzed emotion data to the server.

[1222] Server: Stores the received emotion data in a database and associates it with a profile.

[1223] Step 4: Emotionally informed

[1224] Server: Generates appropriate information and suggestions based on the user's emotional state and sends them to the device.

[1225] Device: Displays and notifies users of emotion-based reminders and appropriate responses.

[1226] 5. Referencing and suggesting public information on social media

[1227] Step 1: Collect social media information

[1228] Server: Periodically collects users' public social media information via API with their permission and stores it in their profile.

[1229] Step 2: Analyze social media information

[1230] Server: Analyzes collected SNS information and extracts users' latest interests and hobbies.

[1231] Step 3: Generate proposals

[1232] Server: Based on the analysis results, it generates appropriate proposal information and sends it to the device.

[1233] On your device: Providing suggested information to you, such as a reminder to "Talk about a new interest you recently shared on social media."

[1234] 6. Privacy Management

[1235] Step 1: Provide privacy settings

[1236] Device: Provide users with a privacy settings screen and allow them to select who they want to share information with.

[1237] User: Set up sharing of information with specific people (for example, family members or coworkers).

[1238] Step 2: Save your permission information

[1239] Server: Receives the configured permission information, stores it in a database, and associates it with the profile.

[1240] Through these steps, this system integrates a wide range of functions, including facial recognition, voice analysis, emotion recognition, collection and analysis of SNS information, and privacy management, thereby providing users with valuable information and enabling smooth communication.

[1241] Example 2

[1242] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1243] Current communication technologies and interaction systems have difficulty accurately understanding and responding appropriately to users' individuality and emotions. Therefore, systems that can improve the quality of communication with users and build deeper human relationships are needed. Furthermore, existing systems lack the functionality to reflect users' social media information and real-time emotions and make personalized suggestions. Furthermore, privacy protection settings are difficult to understand, and data permission management is insufficient.

[1244] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring face recognition data of a user, means for transmitting the acquired face recognition data to the server, means for comparing the face recognition data with a database in the server and searching for matching profile information and transmitting the matching profile information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation histories and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, emotion recognition means for analyzing the user's voice and facial expressions in real time, means for transmitting the analyzed emotion data to the server, and means for providing appropriate information and reactions based on the emotion data. This makes it possible to provide and suggest appropriate information to individual users in real time according to their emotions and interests, thereby realizing deeper communication with users.

[1245] "Facial recognition data" is feature data obtained by analyzing a facial image of a user captured using a device such as a camera.

[1246] A "server" is a computer system that processes, stores, and manages data over a network.

[1247] A "database" is an information system that allows data to be stored, managed, and searched in an organized manner.

[1248] "Profile information" is a collection of data that includes personal information, history, interests, etc. about a user.

[1249] "Voice data" refers to the recording and analysis of user-generated voice as digital data.

[1250] "Keywords" are important words or phrases extracted from the user's conversation.

[1251] "Conversation history" is a record of past conversations and statements.

[1252] "SNS public information" refers to information that a user has set to be public on a social networking service.

[1253] "Privacy settings" are settings that allow users to define the scope and permission conditions for sharing personal information.

[1254] "Emotion recognition means" refers to technology or devices that analyze and recognize the user's emotional state from their voice and facial expressions.

[1255] "Notification means" is a function for notifying the user of information or suggestions through a terminal or device.

[1256] This invention is a system that supports user communication and deepens human relationships. This system combines face recognition, voice analysis, collection and analysis of SNS information, privacy management, and an emotion engine that recognizes user emotions.

[1257] composition

[1258] This system mainly consists of the following components: The device is equipped with a camera and microphone, which collect data. The data is processed on the server, and appropriate feedback is provided to the user using the information stored in the database.

[1259] Hardware and software used

[1260] 1. Terminal: A device such as a smartphone, tablet, or PC. It uses a camera module to take pictures and a microphone to collect audio.

[1261] 2. Server: A computer system that processes data and manages databases.

[1262] 3. Facial recognition software: Facial recognition library such as OpenCV, dlib.

[1263] 4. Speech analysis software: A speech recognition library such as the Google Speech-to-Text API.

[1264] 5. Emotion recognition software: Use IBM Watson Tone Analyzer, Microsoft Azure Face API, etc.

[1265] 6. Database: A relational database such as MySQL.

[1266] Program processing

[1267] 1. Facial Recognition and Profile Management

[1268] When a user starts the system, a facial image is captured using the device's camera.

[1269] The device inputs the captured facial image into a facial recognition library (e.g., OpenCV) to generate facial recognition data.

[1270] The recognized face data is sent to a server, which searches a database to identify the corresponding profile information.

[1271] The server sends profile information, which the terminal then displays or audibly presents to the user.

[1272] As a concrete example, when User A starts the system, the device's camera recognizes User A's face and sends the data to the server. The server searches the database, identifies the profile information of "User A," and sends it to the device. The device then provides the information to User A by displaying it or by voice.

[1273] 2. Audio collection and real-time analysis

[1274] The terminal uses a microphone to collect the user's conversational voice.

[1275] The collected voice data is converted into text using a voice analysis module (e.g., Google Speech-to-Text API).

[1276] Important keywords are extracted from the converted text and sent to the server.

[1277] The server searches past conversation history and related information and sends it to the terminal.

[1278] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[1279] 3. Collecting and suggesting SNS information

[1280] The server periodically collects public information from the social media accounts authorized by the user.

[1281] The collected SNS information is analyzed using analytical tools (e.g., TensorFlow or PyTorch) to extract users' hobbies and interests.

[1282] The extracted information is stored in a profile and suggested information is sent to the device at the appropriate time.

[1283] The terminal notifies the user of the suggestion information.

[1284] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[1285] 4. Privacy Management

[1286] When users first use the system, they set their privacy preferences and choose who they want to share their information with.

[1287] The terminal transmits the set permission information to the server.

[1288] The server stores the permission information in a database and associates it with a profile.

[1289] As a specific example, when User D starts the system for the first time, he / she sets permission for information sharing. User D sets the system so that information is shared only with family members and specific business partners.

[1290] 5. Introducing the Emotion Engine

[1291] The device recognizes emotions in real time from the user's voice and facial expressions (for example, using IBM Watson Tone Analyzer or Microsoft Azure Face API).

[1292] The recognized emotion data is sent to the server.

[1293] The server analyzes the emotion data and associates it with the user's profile.

[1294] Data is sent to the terminal that provides appropriate information and responses based on the user's emotions.

[1295] As a concrete example, if user E expresses negative emotions during a conversation with a friend, the emotion engine recognizes the emotion and sends emotional data to the server. Based on that data, the server sends appropriate suggestions to the device, such as "switch to a more relaxing topic" or "send a sympathetic message." The device then reminds user E of these suggestions, helping to keep the conversation flowing smoothly.

[1296] In this way, this system not only incorporates facial recognition, voice analysis, collection and analysis of SNS information, and privacy management, but also an emotion engine, allowing it to respond to a variety of communication situations for users and help build deeper human relationships.

[1297] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1298] Step 1:

[1299] The user starts the system. The device camera captures the user's facial image. Input: User's facial image. Output: Facial recognition data.

[1300] The device uses a facial recognition library (e.g., OpenCV or dlib) to analyze the captured facial image and generate facial recognition data, which is represented as template data with unique features.

[1301] Step 2:

[1302] The facial recognition data generated by the device is sent to the server. Input: Facial recognition data. Output: Identified profile information.

[1303] The server then matches the received facial recognition data with a database containing user profile information (such as name, contact details, past conversation history, interests, and preferences) to find matching profiles.

[1304] Step 3:

[1305] The server sends the profile information to the device. Input: Identified profile information. Output: Provided profile information.

[1306] The device provides the profile information sent from the server to the user, which can be displayed on the screen or read aloud by a voice assistant.

[1307] Step 4:

[1308] The device uses a microphone to collect the user's conversational voice in real time. Input: User's voice data. Output: Text-converted voice data.

[1309] The voice data is converted into text through a voice analysis module (e.g., Google Speech-to-Text API), and the converted result is used as text.

[1310] Step 5:

[1311] The device extracts important keywords from the text-converted voice data. Input: Text-converted voice data. Output: Extracted keywords.

[1312] The natural language processing function of the speech analysis module is used to extract important keywords from the text. For example, "project progress" is extracted as an important keyword.

[1313] Step 6:

[1314] Based on important keywords, related conversation history and common interests are retrieved from the server. Input: Extracted keywords. Output: Related information.

[1315] The server searches the database based on the extracted keywords to retrieve information related to past conversation history and common interests, which is implemented using a text search algorithm.

[1316] Step 7:

[1317] Notify the user of the acquired information. Input: Relevant information. Output: Notification to the user.

[1318] The device notifies the user of the acquired related information, which can be displayed on the screen or given verbally by a voice assistant.

[1319] Step 8:

[1320] The server periodically collects the user's SNS public information and saves it in the profile. Input: SNS public information. Output: Updated profile information.

[1321] We use collection tools to automatically retrieve public information from users' social media accounts, using the social media APIs (e.g., Twitter API and Facebook Graph API).

[1322] Step 9:

[1323] Analyze collected SNS information to extract user hobbies and interests. Input: SNS information. Output: Extracted hobbies and interests.

[1324] Analyze social media information using analytical tools (e.g., TensorFlow or PyTorch), thereby probabilistically extracting users' hobbies and interests.

[1325] Step 10:

[1326] Based on the extracted information, appropriate suggestion information is generated and sent to the device. Input: Extracted information on hobbies and interests. Output: Suggestion information.

[1327] The server generates suggested information based on the extracted hobbies and interests. For example, if the user is interested in "sports," it will suggest related events and news.

[1328] Step 11:

[1329] Notify the user of the submitted proposal information. Input: Proposal information. Output: Notification to the user.

[1330] The device notifies the user of the received suggestion information by means of a screen display, a pop-up message, or guidance via a voice assistant.

[1331] Step 12:

[1332] When a user first uses the system, they set their privacy preferences. Input: User's privacy preferences information. Output: Privacy preferences data.

[1333] The device receives user input and generates privacy setting data, allowing the user to select the scope of information sharing and the people with whom the information is to be shared.

[1334] Step 13:

[1335] The device sends the generated privacy setting data to the server. Input: Privacy setting data. Output: Saved permission information.

[1336] The server stores the received privacy setting data in a database and associates it with the profile.

[1337] Step 14:

[1338] The device analyzes the user's voice and facial expressions in real time. Input: User's voice and facial expression data. Output: Recognized emotion data.

[1339] Emotion recognition software (e.g., IBM Watson Tone Analyzer or Microsoft Azure Face API) is used to analyze the user's emotions from the collected voice and facial expressions.

[1340] Step 15:

[1341] The analyzed emotion data is sent to the server. Input: Recognized emotion data. Output: User's emotion information.

[1342] The server receives the emotion data and processes it to associate it with a profile.

[1343] Step 16:

[1344] Implement measures to provide appropriate information and responses based on emotional data. Input: User's emotional information. Output: Appropriate responses and suggestions.

[1345] The server generates appropriate information and responses based on the user's emotional data and sends them to the device, which then notifies the user, improving the quality of communication.

[1346] (Application example 2)

[1347] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1348] Existing communication support systems do not take into account the user's emotions or mood state, and therefore provide uniform information and suggestions, making it impossible to provide a personalized experience. Furthermore, there is a need for systems that not only collect and analyze social media information and profile information, but also perform real-time emotion recognition to suggest optimal content according to the user's emotions. This has led to challenges in improving entertainment experiences and deepening human relationships.

[1349] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1350] In this invention, the server includes means for acquiring facial recognition data of a user, means for transmitting the acquired facial recognition data to the server, means for the server to compare the facial recognition data with a database and search for matching profile information and transmit the searched information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation history and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, means for recognizing emotion data from the user's voice and facial expressions using an emotion engine and transmitting the emotion data to the server, and means for the server to analyze the emotion data, associate the emotion data with the profile, generate appropriate information and reactions, and transmit the information and reactions to the terminal. This enables personalized entertainment content to be proposed that takes the user's emotions into consideration.

[1351] "Facial recognition data" is data that captures a user's face in digital form and analyzes its features.

[1352] A "server" is a computer system that provides and manages data on a network and exchanges information with client terminals.

[1353] "Profile information" is a collection of information related to a particular user, including data such as name, contact information, conversation history, hobbies and interests, etc.

[1354] A "terminal" is a device that is directly operated by a user and displays or provides information via audio, such as a smartphone or smart glasses.

[1355] "Voice data" refers to data that is used to record and analyze user utterances and conversation content in digital form.

[1356] "Keywords" are important words or phrases extracted from audio data and used to identify themes and key points of a conversation.

[1357] "Conversation history" is data that records the contents of past conversations, and is used to understand the flow of communication between users.

[1358] "SNS information" refers to information about posts and activities made public by users on social networking services.

[1359] An "emotion engine" is a system or software that analyzes emotional data from a user's voice and facial expressions and recognizes their state in real time.

[1360] "Suggested information" refers to information or content suggestions that the system provides to the user, and is generated based on the user's profile information and emotional state.

[1361] "Privacy settings" are settings that allow users to decide with whom they want to share their information and to what extent they want to make it public.

[1362] "Permission information" is data that records the content and scope of information handling that a user has approved.

[1363] This invention supports user communication and appropriately suggests entertainment content by building a system that uses an emotion engine in addition to facial recognition, voice analysis, collection and analysis of SNS information, and privacy management.

[1364] Hardware and Software Configuration

[1365] 1. Obtaining the user's facial recognition data

[1366] Hardware: Smart glasses and smartphone cameras

[1367] Software: Facial recognition API (e.g., Amazon Rekognition)

[1368] Description: When a user turns on the device, the camera captures facial recognition data and sends it to a server, which then matches the data with a database to find matching profile information and send it to the device.

[1369] 2. Collection and analysis of audio data

[1370] Hardware: Smart glasses and smartphone microphones

[1371] Software: Speech analysis API (e.g., Google Cloud Speech-to-Text)

[1372] Description: When a user starts talking, the microphone collects voice data and extracts keywords in real time. The extracted keywords are sent to a server, which searches past conversation history and related information and provides them to the device.

[1373] 3. Introducing an Emotion Engine

[1374] Hardware: Smart glasses and smartphone cameras and microphones

[1375] Software: Sentiment analysis API (e.g. IBM Watson Tone Analyzer)

[1376] Description: Analyzes emotional data from voice and facial expressions in real time. Emotional data is associated with a profile and generates appropriate information and responses based on the user's state.

[1377] 4. Collecting and suggesting SNS information

[1378] Software: Social Media API (e.g. Twitter API, Facebook Graph API)

[1379] Description: The server periodically collects the user's social media information and saves it in the user's profile. It analyzes the collected information to extract hobbies and interests. Based on this, it generates suitable suggestions and sends them to the device.

[1380] 5. Privacy Settings

[1381] Software: Privacy Management API (e.g., OAuth 2.0)

[1382] Description: When a user first uses the system, they set their preferences regarding who they want to share their information with and how their emotional data is handled, and this information is stored in a database and associated with their profile.

[1383] 6. Content Suggestion

[1384] Description: The server suggests entertainment content based on the user's profile information and emotional data. For example, if the user is feeling stressed, it will suggest a list of relaxing movies. This is done by analyzing the user's state from social media information and past data.

[1385] Specific examples

[1386] When a user puts on the smart glasses and launches the content recommendation system, their profile is loaded using facial recognition. Based on the user's conversation and facial expressions, the system analyzes emotional data in real time and combines it with social media information to suggest appropriate entertainment content.

[1387] Prompt Sentence Examples

[1388] Generate a list of relaxing movies to suggest when the user is feeling stressed. Read the user's recent emotions from their social media posts and refer to their profile information. Choose the latest comedy movies as suggestions.

[1389] The above is an embodiment of the present invention, which allows users to have a personalized entertainment experience based on their emotions and interests.

[1390] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1391] Step 1:

[1392] Acquiring facial recognition data and loading profile information

[1393] When a user turns on their smart glasses or smartphone, the device's camera captures the user's facial recognition data.

[1394] Input: Video of user's face

[1395] Data processing: The raw data collected by the camera is processed by a facial recognition API (e.g., Amazon Rekognition) to produce quantified facial recognition data.

[1396] Output: Facial recognition data

[1397] The server receives the facial recognition data sent from the device and compares it with a database.

[1398] Input: Facial recognition data

[1399] Data calculation: Compare with existing facial recognition data in the database to find matching profile information.

[1400] Output: Profile information

[1401] The profile information is transmitted to the terminal and provided to the user.

[1402] Step 2:

[1403] Audio data collection and analysis

[1404] When a user starts talking, the device's microphone collects audio data.

[1405] Input: User's voice

[1406] Data processing: The audio data is converted into text format using a speech analysis API (e.g., Google Cloud Speech-to-Text).

[1407] Output: Speech-to-text data

[1408] The server analyzes the text data and extracts keywords.

[1409] Input: Speech-to-text data

[1410] Data calculations: The text analysis module identifies important keywords.

[1411] Output: Keywords

[1412] Based on the extracted keywords, the server obtains past conversation history and common interests and sends them to the terminal.

[1413] Input: keyword

[1414] Data calculation: Search the conversation history database and retrieve relevant information.

[1415] Output: Related information

[1416] Step 3:

[1417] Emotional Data Recognition and Analysis

[1418] Emotional data is collected from the user's facial expressions and voice using the device's camera and microphone.

[1419] Input: User's facial expression and audio

[1420] Data processing: Video and audio data are processed using an emotion analysis API (e.g., IBM Watson Tone Analyzer) and converted into emotion data.

[1421] Output: Emotion data

[1422] The server associates the emotional data with a profile and analyzes it.

[1423] Input: Emotion data

[1424] Data calculations: Emotional data is linked to profile information to analyze current emotional state.

[1425] Output: Emotional state analysis results

[1426] The server generates appropriate information and responses based on the emotional state and sends them to the device.

[1427] Input: Emotional state analysis results

[1428] Data calculation: Generate responses and suggestions based on the user's emotions.

[1429] Output: Proposal information

[1430] Step 4:

[1431] SNS information collection and suggestions

[1432] The server periodically collects the user's social media information and stores it in the profile.

[1433] Input: SNS public information

[1434] Data processing: Collect information using SNS APIs and add it to your profile.

[1435] Output: Updated profile information

[1436] The server analyzes the collected SNS information and extracts the user's hobbies and interests.

[1437] Input: Social Media Information

[1438] Data calculation: Hobbies and interests are identified through an information analysis module.

[1439] Output: Hobbies and interests analysis results

[1440] The server generates proposal information based on this and transmits it to the terminal.

[1441] Input: Hobbies and Interests Analysis Results

[1442] Data calculation: The personalized suggestion generation module makes appropriate content suggestions.

[1443] Output: Proposal information

[1444] Step 5:

[1445] Content suggestion notifications

[1446] The terminal receives the proposal information from the server and notifies the user.

[1447] Input: Proposal information

[1448] Data calculation: Processing is performed to present the received proposal information visually or audibly.

[1449] Output: Notify user of content suggestion

[1450] For example, consider a situation where a user is wearing smart glasses and browsing a list of movies. If the emotion engine detects that the user is stressed, the prompt to suggest "relaxing comedy movies" might be:

[1451] Generate a list of relaxing movies to suggest when the user is feeling stressed. Read the user's recent emotions from their social media posts and refer to their profile information. Choose the latest comedy movies as suggestions.

[1452] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1453] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1454] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1455] [Fourth embodiment]

[1456] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1457] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1458] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1459] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1460] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1461] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1462] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1463] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1464] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1465] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1466] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1467] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1468] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1469] The present invention is a system for supporting user communication and deepening human relationships. This system combines the following various devices and means to achieve smooth interaction with users.

[1470] System Components

[1471] 1. Facial Recognition and Profile Management

[1472] This system acquires a user's facial recognition data and manages profile information based on it. When a user uses the system, facial recognition data is acquired through the device's camera. The acquired data is sent to a server, which checks it against a database to identify matching profile information. This profile information includes the user's name, contact information, past conversation history, interests, and preferences. This information is then sent to the device as appropriate and provided to the user.

[1473] For example, when User A starts the system, the device's camera recognizes User A's face and sends the facial recognition data to the server. The server searches the database, identifies User A's profile information, and sends it to the device. The device then provides that information to User A by displaying it or by voice.

[1474] 2. Audio collection and real-time analysis

[1475] This system has the ability to collect and analyze conversational audio in real time. The device collects the user's conversation content as audio data and extracts important keywords through a voice analysis module. This data is sent to a server, which then searches for past conversation history and related information and sends it to the device. The device notifies the user of this information to help the conversation progress.

[1476] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will then search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[1477] 3. Collecting and suggesting SNS information

[1478] Furthermore, this system collects information publicly shared on social media by users and makes suggestions based on that information. The server periodically collects users' social media information and stores it in their profiles. The collected information is analyzed to extract the user's hobbies and interests. The extracted information is then provided to the user as suggested information at the appropriate time.

[1479] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[1480] 4. User Privacy Management

[1481] This system allows users to configure settings to protect their privacy. When using the system for the first time, users select who they want to share their information with and set permission information. This prevents information from being shared without permission.

[1482] For example, when User D starts the system for the first time, he or she sets permission for information sharing. User D sets the system so that information is shared only with family members and specific business partners.

[1483] In this way, this system combines multiple functions such as facial recognition, voice analysis, collection and analysis of SNS information, and privacy management to support user communication and help deepen human relationships.

[1484] The processing flow will be explained below.

[1485] Program processing steps for face recognition and conversation reminder system

[1486] 1. User registration and license information settings

[1487] Step 1: Register a basic user profile

[1488] On the device: The application is launched and a screen is displayed that captures the user's facial recognition data using the camera. The user then enters basic profile information such as name, contact details, and social media accounts.

[1489] Server: Receives facial recognition data and basic profile information sent from the device and stores it in a database.

[1490] Step 2: Set up your license information

[1491] On your device: Display the privacy settings screen and give the option to choose who you want to share information with.

[1492] Users: Choose who you share your information with and set your privacy preferences.

[1493] Server: Receives the permission information, associates it with the profile, and stores it in a database.

[1494] 2. Facial Recognition and Information Extraction

[1495] Step 1: Obtaining facial recognition data

[1496] Device: Uses the camera to recognize the user's face and collect facial recognition data.

[1497] Step 2: Send and match facial recognition data

[1498] Device: Sends the acquired facial recognition data to the server.

[1499] Server: Matches the facial recognition data against a database to find matching profile information.

[1500] Step 3: Provide profile information

[1501] Server: Generates relevant profile information (such as name, past conversation history, hobbies, etc.) and sends it to the device.

[1502] Terminal: Provides profile information to the user by display or audio.

[1503] 3. Real-time conversation reminder function

[1504] Step 1: Recognizing voice input

[1505] Device: Uses a microphone to collect audio of conversations in real time.

[1506] Step 2: Keyword extraction

[1507] On the device: The speech analysis module extracts important keywords and conversation topics from the collected audio.

[1508] Step 3: View your conversation history

[1509] Device: Sends the extracted keywords to the server to retrieve related past conversation history and common interests.

[1510] Server: Searches for relevant information from a database based on the received keywords and sends it to the device.

[1511] Step 4: View Reminder Information

[1512] Device: Displays reminders to users in real time, such as information based on past topics or the other person's interests.

[1513] 4. Referencing and suggesting public information on social media

[1514] Step 1: Collect social media information

[1515] Server: Periodically collects users' public social media information via API with their permission and stores it in their profile.

[1516] Step 2: Analyze social media information

[1517] Server: Analyzes collected SNS information and extracts users' latest interests and hobbies.

[1518] Step 3: Generate proposals

[1519] Server: Based on the analysis results, generate appropriate proposal information and send it to the device.

[1520] On the device: Notify users of suggested information, such as a reminder to "Talk about a new interest you recently shared on social media."

[1521] 5. Privacy Management

[1522] Step 1: Provide privacy settings

[1523] Device: Provide users with a privacy settings screen and let them choose who they want to share information with.

[1524] User: Set up sharing of information with specific people (for example, family members or coworkers).

[1525] Step 2: Save your permission information

[1526] Server: Receives the configured permission information, stores it in a database, and associates it with a profile.

[1527] These steps of the present system allow users to obtain information that is appropriate for them and achieve smooth communication.

[1528] Example 1

[1529] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1530] Conventional communication support systems are unable to fully utilize the individual user's information, resulting in insufficient personalized support. Furthermore, from the perspective of privacy protection, there are problems with managing and sharing information. Furthermore, it is difficult to provide real-time conversation support or suggest appropriate information, making it difficult to improve the user experience.

[1531] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1532] In this invention, the server includes means for acquiring biometric authentication data of a user, means for transmitting the acquired biometric authentication data to a data processing device, means for matching the biometric authentication data with a data storage area in the data processing device to search for matching identification information and transmitting the matching identification information to a terminal device, means for providing the user with the identification information transmitted from the data processing device, means for collecting acoustic data and extracting keywords in real time, means for acquiring related conversation histories and common interests from the data processing device based on the extracted keywords, means for notifying the user of the acquired information, means for periodically collecting the user's electronic information disclosure data and storing it in the identification information, means for analyzing the collected electronic information disclosure data to extract the user's hobbies and interests, means for generating appropriate proposal information based on the extracted information and transmitting it to the terminal device, means for notifying the user of the transmitted proposal information, means for the user to input basic identification information when using the system for the first time, means for setting privacy and selecting partners with whom to share information, and means for storing the set permission information in the data storage area and associating it with the identification information. This enables personalized communication support for the user, efficient information proposals, and privacy management.

[1533] "Biometric data" is data used to identify an individual based on physical characteristics such as the user's face, fingerprint, or iris.

[1534] "Data Processing Device" means a computer system for processing and analyzing data obtained from a user.

[1535] The "data storage area" is a storage device for storing data such as user identification information and past conversation history.

[1536] An "end device" is a device that can be directly operated by a user, specifically a terminal such as a smartphone, tablet, or PC.

[1537] "Identification information" is data that includes a user's name, contact information, past conversation history, interests, preferences, etc.

[1538] "Acoustic data" is data that collects audio information such as the user's voice and surrounding sounds.

[1539] "Keywords" are important words or phrases extracted from the acoustic data.

[1540] "Electronic information public data" refers to online information that users have made public, such as on social media sites and blogs.

[1541] "Suggested information" is information that is generated by the data processing device based on the user's hobbies and interests, and is useful information for the user.

[1542] "Privacy settings" are settings that determine the scope of information sharing and permission information, which are set by the user when using the system for the first time.

[1543] "Permission information" is setting information that allows a user to permit information sharing with specific people or organizations.

[1544] The present invention provides a system for supporting user communication and deepening interpersonal relationships. This system integrates functions such as face recognition, voice analysis, collection and analysis of social networking site information, and privacy management. Specific embodiments are described below.

[1545] Facial Recognition and Profile Management

[1546] First, the biometric data of the user is acquired. The camera of the terminal is used to capture the user's face in real time, and facial feature points are extracted using a facial recognition algorithm (e.g., OpenCV). The acquired biometric data is then sent from the terminal to a data processing device.

[1547] The data processing device matches the received biometric data with its data storage to find matching identifying information, which may include the user's name, contact details, past conversation history, interests, and preferences. The identifying information is then sent from the data processing device to the end device and provided to the user.

[1548] As a specific example, when user A starts face recognition using a terminal, the camera recognizes user A's face and sends biometric authentication data to the data processing device. The data processing device compares it with the data storage area, identifies user A's identification information, sends it to the terminal, and displays it to user A.

[1549] Audio collection and real-time analysis

[1550] Next, the device collects the user's speech in real time through a microphone, and the collected acoustic data is converted into text through a speech analysis module (e.g., Google Cloud Speech-to-Text) to extract important keywords.

[1551] The extracted keywords are sent from the terminal to a data processing device, which searches the data storage area for past conversation history and related information. The search results are sent to the terminal and notified to the user.

[1552] For example, if User B starts talking about "project progress" during a conversation, the device recognizes this keyword and sends it to the data processing device. The data processing device searches for past conversation history related to "project progress" and sends it to the device to notify User B.

[1553] SNS information collection and suggestions

[1554] The data processing device also periodically collects the user's public electronic information data using an API (e.g., Twitter API or Facebook Graph API). The collected public electronic information data is stored in a data storage area and analyzed. The analysis allows the user's hobbies and interests to be extracted.

[1555] The extracted information is sent from the data processing device to the terminal device at a time appropriate for the device and provided to the user as suggested information. For example, if User C posts on SNS that he or she has recently become interested in sports, the data processing device collects and analyzes this information. The next time a topic related to sports comes up during User C's conversation, the device will remind him or her by saying, "You've been posting about sports recently."

[1556] User Privacy Management

[1557] The system also places great importance on protecting users' privacy. When users first use the system, they can enter basic identification information and configure their privacy settings, which allow them to choose with whom they want to share their information.

[1558] The permission information is saved in the data storage area and associated with the profile. This prevents information from being shared without permission. For example, when User D starts the system for the first time, he or she can set it up so that information is shared only with family members and specific business partners.

[1559] Example of a generated AI prompt

[1560] Using a generative AI model, we can generate prompts like this:

[1561] Facial Recognition and Profile Management:

[1562] "Please explain in detail how your system works to obtain a user's facial recognition data and provide profile information."

[1563] Audio collection and real-time analysis:

[1564] "Please explain in detail how the system works, collecting audio in real time, analyzing important keywords, and notifying users."

[1565] Social media information collection and suggestions:

[1566] "Please explain in detail how the system works to collect social media information and suggest appropriate information to users."

[1567] User Privacy Controls:

[1568] "Please explain in detail how your system manages user privacy."

[1569] As described above, the present invention provides specific means for supporting communication between users and deepening human relationships.

[1570] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1571] Step 1:

[1572] The device captures the user's biometric authentication data using a camera. At this time, it uses a facial recognition algorithm (e.g., OpenCV) to extract facial feature points. The input to this process is real-time video captured by the camera, and the output is facial recognition data with extracted feature points.

[1573] Specifically, the device's camera is activated, the user's face is detected, and feature points are extracted. This facial recognition data is then stored in temporary memory.

[1574] Step 2:

[1575] The device transmits the acquired biometric authentication data, including facial recognition data, to the server.

[1576] Specifically, the device sends facial recognition data to a server via the Internet, and once the transmission is complete, the data is stored on the server.

[1577] Step 3:

[1578] The server compares the received biometric data against a database to find matching identification information. The input to this process is the facial recognition data sent by the device, and the output is the matching profile information.

[1579] Specifically, the server matches the facial recognition data with a data store to identify corresponding identifying information, including the user's name, contact details, past conversation history, interests, and preferences.

[1580] Step 4:

[1581] The server transmits the acquired identification information to the terminal, and the transmitted data includes the user's profile information.

[1582] Specifically, the server converts the identification information into packets and sends them to the terminal via the Internet. The terminal receives this data.

[1583] Step 5:

[1584] The terminal provides the user with the identification information sent by the server. The input to this process is the profile information sent by the server, and the output is the display of the information to the user.

[1585] Specifically, the device displays profile information on the screen, allowing users to check their own name and past conversation history.

[1586] Step 6:

[1587] The device collects the user's conversation content in real time through a microphone. The input of this process is the user's voice, and the output is acoustic data.

[1588] Specifically, the device's microphone captures the user's voice and stores it as audio data, which is then temporarily stored in the device's internal memory.

[1589] Step 7:

[1590] The terminal uses a speech analysis module to convert the collected acoustic data into text and extract important keywords. The input of this process is the collected acoustic data, and the output is the extracted keywords.

[1591] Specifically, the device calls a speech analysis module such as Google Cloud Speech-to-Text to convert the voice data into text, and then extracts important keywords from the converted text.

[1592] Step 8:

[1593] The terminal transmits the extracted keywords to the server. The input of this process is the extracted keywords, and the output is data transmission to the server.

[1594] Specifically, the device sends the keyword to a server via the Internet, where the data is temporarily stored.

[1595] Step 9:

[1596] The server searches the database for related conversation histories and common interests based on the received keywords. The input to this process is the extracted keywords, and the output is related information.

[1597] Specifically, the server uses keywords to search the data storage for relevant conversation histories and common interests, and once the search results are identified, they are stored in temporary memory.

[1598] Step 10:

[1599] The server transmits the retrieved related information to the terminal. The input of this process is the retrieved related information, and the output is data transmission to the terminal.

[1600] Specifically, the server converts the relevant information into packets and sends them to the terminal via the Internet, which then receives the data.

[1601] Step 11:

[1602] The terminal notifies the user of the relevant information sent from the server. The input of this process is the relevant information sent from the server, and the output is a notification to the user.

[1603] Specifically, the device will display relevant information on the screen or notify the user via voice, allowing the user to check this information in real time.

[1604] (Application example 1)

[1605] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1606] Conventional advertising display systems have had difficulty effectively providing personalized advertisements based on users' interests. Furthermore, they have been unable to properly utilize users' profile information and social networking site activity to display highly relevant advertisements in real time, making it difficult to attract users' attention and reducing advertising effectiveness. The present invention aims to solve these problems.

[1607] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1608] In this invention, the server includes means for acquiring facial recognition data of a user, means for transmitting the acquired facial recognition data to the server, means for the server to compare the facial recognition data with a database and search for matching profile information and transmit the matching profile information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation history and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, means for personalizing advertisement content displayed on the terminal based on the user's profile information and SNS information, and means for analyzing the user's interests using the acquired profile information and SNS information and displaying relevant advertisements. This enables effective provision of personalized advertisements based on the user's interests.

[1609] - "User's facial recognition data" means data that captures the user's facial features as digital information and enables identification.

[1610] The "server" is a computer system that manages users' facial recognition data and profile information, and collates and searches data in real time.

[1611] "Profile information" is data that includes personal information such as a user's name, contact details, past conversation history, interests and preferences.

[1612] A "terminal" is a device that has an interface with a user and displays or notifies information.

[1613] "Voice data" refers to data that records the contents of a user's conversation in digital format.

[1614] "Keyword extraction" is the process of analyzing and extracting important words and phrases from audio data in real time.

[1615] "Related conversation history" is data obtained by searching for past conversation content and related information based on keywords.

[1616] "SNS information" refers to information that a user makes public on a social networking service.

[1617] "Personalization" is the process of customizing advertisements and information based on individual user profile information and social media information.

[1618] "Advertising display" is the process of displaying advertisements on a device based on the user's interests.

[1619] To implement this invention, the following steps are required: The user installs a dedicated smartphone application, enters basic profile information when the application is first launched, and faces the camera to obtain facial recognition data. The application collects the user's facial recognition data and sends it to a server. At this time, it performs facial recognition using a library such as OpenCV. The server compares the received facial recognition data with a database, searches for matching profile information, and sends it to the terminal.

[1620] The device then displays the profile information and begins collecting voice data. As the user begins to talk, voice data is collected in real time and keyword analysis is performed. This analysis uses a voice recognition library and natural language processing tools. The analyzed keywords are sent to the server, which searches for relevant information based on past conversation history and the user's interests and sends it to the device. The device notifies the user of this and helps the conversation proceed smoothly.

[1621] The server also periodically collects social media information and stores it in the user's profile. A data analysis module is used to analyze this social media information and extract the user's hobbies and interests. The analysis results are sent to the device, and suggestions tailored to the user are generated. This suggestion information is displayed as personalized advertisements based on the user's profile information and social media information.

[1622] For example, if a user posts on social media that they are looking for new running shoes, the server will retrieve and analyze this information. The next time the user uses the app, an advertisement for related running shoes will be displayed. In this way, information tailored to the user's interests is provided in real time.

[1623] Example prompt sentence:

[1624] "The user posted on social media that they were interested in new running shoes, so please show them an appropriate ad."

[1625] The hardware required to realize this system is a smartphone camera and microphone, and the software used includes the OpenCV library, speech recognition library, natural language processing tools, data analysis modules, server APIs, etc. This makes it possible to accurately capture user interests and display personalized advertisements.

[1626] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1627] Step 1:

[1628] A user launches a smartphone application and enters profile information. When the user launches the application for the first time, they enter basic information such as their name, contact details, and interests, and then face the camera. The device acquires facial recognition data and sends it to the server along with the acquired profile information. The input is the user's basic information and facial recognition data, and the output is the profile data sent to the server.

[1629] Step 2:

[1630] The server checks the received facial recognition data against a database to find matching profile information. It compares the facial recognition data with existing data in the database to identify the best matching profile information. The input is the facial recognition data, and the output is the matching profile information. This information is then sent back to the device.

[1631] Step 3:

[1632] The device displays the profile information received from the server to the user. This profile information includes the user's name, interests, past conversation history, etc. Based on the displayed profile information, the user can check for more detailed information. The input is the matching profile information, and the output is the information display to the user.

[1633] Step 4:

[1634] When a user starts a conversation, the device collects the conversation content as voice data in real time. Using a voice recognition library, important keywords are extracted from the collected voice data. The input is the user's voice data, and the output is the extracted keywords.

[1635] Step 5:

[1636] The extracted keywords are sent to the server, which searches for related information based on past conversation history and related interests. The server searches a database based on the extracted keywords to obtain related conversation history and common interests. The input is the keywords, and the output is related information. This related information is sent to the terminal and notified to the user.

[1637] Step 6:

[1638] The server periodically collects SNS information and stores it in the user's profile. The server obtains SNS information and analyzes the user's current interests and concerns from that information. The input is SNS information, and the output is analyzed interest and concern information.

[1639] Step 7:

[1640] The server generates personalized suggestion information based on the acquired interest information and sends it to the device. The device receives the suggestion information and displays it to the user. The input is the analyzed interest information, and the output is personalized suggestion information provided to the user.

[1641] Step 8:

[1642] Based on the user's profile information and social media information, the device generates and displays personalized advertisements. The advertisement display system selects the most suitable advertisement and provides it to the user in real time. The input is the profile information and social media information, and the output is the displayed advertisement.

[1643] In this way, the invention utilizes a user's facial recognition data, voice data, and social media information to provide a personalized experience based on their individual profile information.

[1644] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1645] This invention is a system that combines facial recognition, voice analysis, collection and analysis of SNS information, privacy management, and an emotion engine that recognizes user emotions in order to support user communication and deepen human relationships.

[1646] System Components

[1647] 1. Facial Recognition and Profile Management

[1648] The system acquires a user's facial recognition data and manages profile information based on it. When a user uses the system, facial recognition data is acquired through the device's camera and sent to the server. The server compares the data with a database to identify matching profile information and sends it to the device. This profile information includes the user's name, contact information, past conversation history, interests, and preferences. The device then provides this information to the user.

[1649] As a concrete example, when User A starts the system, the device's camera recognizes User A's face and sends the facial recognition data to the server. The server searches the database, identifies the profile information of "User A," and sends it to the device. The device then provides the information to User A by displaying it or by voice.

[1650] 2. Audio collection and real-time analysis

[1651] This system has the ability to collect and analyze conversational audio in real time. The device collects the user's conversation content as audio data and extracts important keywords through the audio analysis module. This data is sent to a server, which then searches for past conversation history and related information and sends it to the device. The device then notifies the user of this information and supports the progress of the conversation.

[1652] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will then search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[1653] 3. Collecting and suggesting SNS information

[1654] Furthermore, this system collects information publicly shared by users on social media and uses that information to make suggestions that are appropriate for the user. The server periodically collects social media information and saves it in a profile. The collected information is analyzed to extract the user's hobbies and interests, which are then sent to the device as suggested information at the appropriate time.

[1655] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[1656] 4. Privacy Management

[1657] This system allows users to configure settings to protect their privacy. When using the system for the first time, users select with whom they want to share their information and set permission information. This prevents information from being shared without permission.

[1658] For example, when User D starts the system for the first time, he or she sets permission for information sharing. User D sets the system to share information only with family members and specific business partners.

[1659] 5. Introducing the Emotion Engine

[1660] This system incorporates an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's voice and facial expressions in real time and collects the data. The collected emotion data is sent to a server and associated with the user's profile. The server analyzes the emotion data and provides appropriate information and responses based on the user's emotions.

[1661] As a concrete example, if user E expresses negative emotions during a conversation with a friend, the emotion engine recognizes the emotion and sends emotional data to the server. Based on that data, the server sends appropriate suggestions to the device, such as "switch to a more relaxing topic" or "send a sympathetic message." The device then reminds user E of these suggestions, helping to keep the conversation flowing smoothly.

[1662] In this way, this system not only incorporates facial recognition, voice analysis, collection and analysis of SNS information, and privacy management, but also an emotion engine, allowing it to respond to a variety of communication situations for users and help build deeper human relationships.

[1663] The processing flow will be explained below.

[1664] Processing steps of the facial recognition and conversation reminder system (including emotion engine)

[1665] 1. User registration and license information settings

[1666] Step 1: Register a basic user profile

[1667] Device: The application launches and displays a screen that captures the user's facial recognition data using the camera. The user then enters basic profile information such as name, contact details, and social media accounts.

[1668] Server: Receives facial recognition data and basic profile information sent from the device and stores it in a database.

[1669] Step 2: Set up your license information

[1670] On your device: Display the privacy settings screen and give you the option to choose who you want to share information with.

[1671] Users: Choose who you share information with and set your privacy preferences.

[1672] Server: Receives the permission information, associates it with the profile, and stores it in a database.

[1673] 2. Facial Recognition and Information Extraction

[1674] Step 1: Obtaining facial recognition data

[1675] Device: Uses the camera to recognize the user's face and collect facial recognition data.

[1676] Step 2: Send and match facial recognition data

[1677] Terminal: Sends the acquired facial recognition data to the server.

[1678] Server: Matches the facial recognition data against a database to find matching profile information.

[1679] Step 3: Provide profile information

[1680] Server: Generates relevant profile information (such as name, past conversation history, hobbies, etc.) and sends it to the device.

[1681] Terminal: Provides profile information to the user by display or audio.

[1682] 3. Real-time conversation reminder function

[1683] Step 1: Recognizing voice input

[1684] Device: Uses a microphone to collect audio of conversations in real time.

[1685] Step 2: Keyword extraction

[1686] Terminal: The speech analysis module extracts important keywords and conversation topics from the collected speech.

[1687] Step 3: View your conversation history

[1688] Device: Sends the extracted keywords to the server to obtain related past conversation history and common interests.

[1689] Server: Searches for relevant information from a database based on the received keywords and sends it to the terminal.

[1690] Step 4: View Reminder Information

[1691] Device: Displays reminder information to the user in real time. For example, it can present information based on topics that have been discussed in the past or the other person's hobbies and preferences.

[1692] 4. Emotion Recognition by Emotion Engine

[1693] Step 1: Collecting emotion data

[1694] Device: Uses a camera and microphone to collect emotional data from the user's facial expressions and voice.

[1695] Step 2: Analyze the emotion data

[1696] Terminal: The emotion engine analyzes the user's emotional state from the collected data.

[1697] Step 3: Send and store emotion data

[1698] Terminal: Transmits the analyzed emotion data to the server.

[1699] Server: Stores the received emotion data in a database and associates it with a profile.

[1700] Step 4: Emotionally informed

[1701] Server: Generates appropriate information and suggestions based on the user's emotional state and sends them to the device.

[1702] Device: Displays and notifies users of emotion-based reminders and appropriate responses.

[1703] 5. Referencing and suggesting public information on social media

[1704] Step 1: Collect social media information

[1705] Server: Periodically collects users' public social media information via API with their permission and stores it in their profile.

[1706] Step 2: Analyze social media information

[1707] Server: Analyzes collected SNS information and extracts users' latest interests and hobbies.

[1708] Step 3: Generate proposals

[1709] Server: Based on the analysis results, it generates appropriate proposal information and sends it to the device.

[1710] On your device: Providing suggested information to you, such as a reminder to "Talk about a new interest you recently shared on social media."

[1711] 6. Privacy Management

[1712] Step 1: Provide privacy settings

[1713] Device: Provide users with a privacy settings screen and allow them to select who they want to share information with.

[1714] User: Set up sharing of information with specific people (for example, family members or coworkers).

[1715] Step 2: Save your permission information

[1716] Server: Receives the configured permission information, stores it in a database, and associates it with the profile.

[1717] Through these steps, this system integrates a wide range of functions, including facial recognition, voice analysis, emotion recognition, collection and analysis of SNS information, and privacy management, thereby providing users with valuable information and enabling smooth communication.

[1718] Example 2

[1719] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1720] Current communication technologies and interaction systems have difficulty accurately understanding and responding appropriately to users' individuality and emotions. Therefore, systems that can improve the quality of communication with users and build deeper human relationships are needed. Furthermore, existing systems lack the functionality to reflect users' social media information and real-time emotions and make personalized suggestions. Furthermore, privacy protection settings are difficult to understand, and data permission management is insufficient.

[1721] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring face recognition data of a user, means for transmitting the acquired face recognition data to the server, means for comparing the face recognition data with a database in the server and searching for matching profile information and transmitting the matching profile information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation histories and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, emotion recognition means for analyzing the user's voice and facial expressions in real time, means for transmitting the analyzed emotion data to the server, and means for providing appropriate information and reactions based on the emotion data. This makes it possible to provide and suggest appropriate information to individual users in real time according to their emotions and interests, thereby realizing deeper communication with users.

[1722] "Facial recognition data" is feature data obtained by analyzing a facial image of a user captured using a device such as a camera.

[1723] A "server" is a computer system that processes, stores, and manages data over a network.

[1724] A "database" is an information system that allows data to be stored, managed, and searched in an organized manner.

[1725] "Profile information" is a collection of data that includes personal information, history, interests, etc. about a user.

[1726] "Voice data" refers to the recording and analysis of user-generated voice as digital data.

[1727] "Keywords" are important words or phrases extracted from the user's conversation.

[1728] "Conversation history" is a record of past conversations and statements.

[1729] "SNS public information" refers to information that a user has set to be public on a social networking service.

[1730] "Privacy settings" are settings that allow users to define the scope and permission conditions for sharing personal information.

[1731] "Emotion recognition means" refers to technology or devices that analyze and recognize the user's emotional state from their voice and facial expressions.

[1732] "Notification means" is a function for notifying the user of information or suggestions through a terminal or device.

[1733] This invention is a system that supports user communication and deepens human relationships. This system combines face recognition, voice analysis, collection and analysis of SNS information, privacy management, and an emotion engine that recognizes user emotions.

[1734] composition

[1735] This system mainly consists of the following components: The device is equipped with a camera and microphone, which collect data. The data is processed on the server, and appropriate feedback is provided to the user using the information stored in the database.

[1736] Hardware and software used

[1737] 1. Terminal: A device such as a smartphone, tablet, or PC. It uses a camera module to take pictures and a microphone to collect audio.

[1738] 2. Server: A computer system that processes data and manages databases.

[1739] 3. Facial recognition software: Facial recognition library such as OpenCV, dlib.

[1740] 4. Speech analysis software: A speech recognition library such as the Google Speech-to-Text API.

[1741] 5. Emotion recognition software: Use IBM Watson Tone Analyzer, Microsoft Azure Face API, etc.

[1742] 6. Database: A relational database such as MySQL.

[1743] Program processing

[1744] 1. Facial Recognition and Profile Management

[1745] When a user starts the system, a facial image is captured using the device's camera.

[1746] The device inputs the captured facial image into a facial recognition library (e.g., OpenCV) to generate facial recognition data.

[1747] The recognized face data is sent to a server, which searches a database to identify the corresponding profile information.

[1748] The server sends profile information, which the terminal then displays or audibly presents to the user.

[1749] As a concrete example, when User A starts the system, the device's camera recognizes User A's face and sends the data to the server. The server searches the database, identifies the profile information of "User A," and sends it to the device. The device then provides the information to User A by displaying it or by voice.

[1750] 2. Audio collection and real-time analysis

[1751] The terminal uses a microphone to collect the user's conversational voice.

[1752] The collected voice data is converted into text using a voice analysis module (e.g., Google Speech-to-Text API).

[1753] Important keywords are extracted from the converted text and sent to the server.

[1754] The server searches past conversation history and related information and sends it to the terminal.

[1755] For example, if User B starts talking about "project progress" during a conversation with a business partner, the device will recognize this keyword and send it to the server. The server will search for related information, such as "the last time we talked about project progress," and send it to the device. The device will then notify User B of this information.

[1756] 3. Collecting and suggesting SNS information

[1757] The server periodically collects public information from the social media accounts authorized by the user.

[1758] The collected SNS information is analyzed using analytical tools (e.g., TensorFlow or PyTorch) to extract users' hobbies and interests.

[1759] The extracted information is stored in a profile and suggested information is sent to the device at the appropriate time.

[1760] The terminal notifies the user of the suggestion information.

[1761] For example, if User C posts on social media that he or she has recently become interested in sports, the server collects and analyzes this information. The next time User C talks about sports, the device will provide a reminder that he or she has recently posted about sports.

[1762] 4. Privacy Management

[1763] When users first use the system, they set their privacy preferences and choose who they want to share their information with.

[1764] The terminal transmits the set permission information to the server.

[1765] The server stores the permission information in a database and associates it with a profile.

[1766] As a specific example, when User D starts the system for the first time, he / she sets permission for information sharing. User D sets the system so that information is shared only with family members and specific business partners.

[1767] 5. Introducing the Emotion Engine

[1768] The device recognizes emotions in real time from the user's voice and facial expressions (for example, using IBM Watson Tone Analyzer or Microsoft Azure Face API).

[1769] The recognized emotion data is sent to the server.

[1770] The server analyzes the emotion data and associates it with the user's profile.

[1771] Data is sent to the terminal that provides appropriate information and responses based on the user's emotions.

[1772] As a concrete example, if user E expresses negative emotions during a conversation with a friend, the emotion engine recognizes the emotion and sends emotional data to the server. Based on that data, the server sends appropriate suggestions to the device, such as "switch to a more relaxing topic" or "send a sympathetic message." The device then reminds user E of these suggestions, helping to keep the conversation flowing smoothly.

[1773] In this way, this system not only incorporates facial recognition, voice analysis, collection and analysis of SNS information, and privacy management, but also an emotion engine, allowing it to respond to a variety of communication situations for users and help build deeper human relationships.

[1774] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1775] Step 1:

[1776] The user starts the system. The device camera captures the user's facial image. Input: User's facial image. Output: Facial recognition data.

[1777] The device uses a facial recognition library (e.g., OpenCV or dlib) to analyze the captured facial image and generate facial recognition data, which is represented as template data with unique features.

[1778] Step 2:

[1779] The facial recognition data generated by the device is sent to the server. Input: Facial recognition data. Output: Identified profile information.

[1780] The server then matches the received facial recognition data with a database containing user profile information (such as name, contact details, past conversation history, interests, and preferences) to find matching profiles.

[1781] Step 3:

[1782] The server sends the profile information to the device. Input: Identified profile information. Output: Provided profile information.

[1783] The device provides the profile information sent from the server to the user, which can be displayed on the screen or read aloud by a voice assistant.

[1784] Step 4:

[1785] The device uses a microphone to collect the user's conversational voice in real time. Input: User's voice data. Output: Text-converted voice data.

[1786] The voice data is converted into text through a voice analysis module (e.g., Google Speech-to-Text API), and the converted result is used as text.

[1787] Step 5:

[1788] The device extracts important keywords from the text-converted voice data. Input: Text-converted voice data. Output: Extracted keywords.

[1789] The natural language processing function of the speech analysis module is used to extract important keywords from the text. For example, "project progress" is extracted as an important keyword.

[1790] Step 6:

[1791] Based on important keywords, related conversation history and common interests are retrieved from the server. Input: Extracted keywords. Output: Related information.

[1792] The server searches the database based on the extracted keywords to retrieve information related to past conversation history and common interests, which is implemented using a text search algorithm.

[1793] Step 7:

[1794] Notify the user of the acquired information. Input: Relevant information. Output: Notification to the user.

[1795] The device notifies the user of the acquired related information, which can be displayed on the screen or given verbally by a voice assistant.

[1796] Step 8:

[1797] The server periodically collects the user's SNS public information and saves it in the profile. Input: SNS public information. Output: Updated profile information.

[1798] We use collection tools to automatically retrieve public information from users' social media accounts, using the social media APIs (e.g., Twitter API and Facebook Graph API).

[1799] Step 9:

[1800] Analyze collected SNS information to extract user hobbies and interests. Input: SNS information. Output: Extracted hobbies and interests.

[1801] Analyze social media information using analytical tools (e.g., TensorFlow or PyTorch), thereby probabilistically extracting users' hobbies and interests.

[1802] Step 10:

[1803] Based on the extracted information, appropriate suggestion information is generated and sent to the device. Input: Extracted information on hobbies and interests. Output: Suggestion information.

[1804] The server generates suggested information based on the extracted hobbies and interests. For example, if the user is interested in "sports," it will suggest related events and news.

[1805] Step 11:

[1806] Notify the user of the submitted proposal information. Input: Proposal information. Output: Notification to the user.

[1807] The device notifies the user of the received suggestion information by means of a screen display, a pop-up message, or guidance via a voice assistant.

[1808] Step 12:

[1809] When a user first uses the system, they set their privacy preferences. Input: User's privacy preferences information. Output: Privacy preferences data.

[1810] The device receives user input and generates privacy setting data, allowing the user to select the scope of information sharing and the people with whom the information is to be shared.

[1811] Step 13:

[1812] The device sends the generated privacy setting data to the server. Input: Privacy setting data. Output: Saved permission information.

[1813] The server stores the received privacy setting data in a database and associates it with the profile.

[1814] Step 14:

[1815] The device analyzes the user's voice and facial expressions in real time. Input: User's voice and facial expression data. Output: Recognized emotion data.

[1816] Emotion recognition software (e.g., IBM Watson Tone Analyzer or Microsoft Azure Face API) is used to analyze the user's emotions from the collected voice and facial expressions.

[1817] Step 15:

[1818] The analyzed emotion data is sent to the server. Input: Recognized emotion data. Output: User's emotion information.

[1819] The server receives the emotion data and processes it to associate it with a profile.

[1820] Step 16:

[1821] Implement measures to provide appropriate information and responses based on emotional data. Input: User's emotional information. Output: Appropriate responses and suggestions.

[1822] The server generates appropriate information and responses based on the user's emotional data and sends them to the device, which then notifies the user, improving the quality of communication.

[1823] (Application example 2)

[1824] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1825] Existing communication support systems do not take into account the user's emotions or mood state, and therefore provide uniform information and suggestions, making it impossible to provide a personalized experience. Furthermore, there is a need for systems that not only collect and analyze social media information and profile information, but also perform real-time emotion recognition to suggest optimal content according to the user's emotions. This has led to challenges in improving entertainment experiences and deepening human relationships.

[1826] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1827] In this invention, the server includes means for acquiring facial recognition data of a user, means for transmitting the acquired facial recognition data to the server, means for the server to compare the facial recognition data with a database and search for matching profile information and transmit the searched information to the terminal, means for providing the profile information transmitted from the server to the user, means for collecting voice data and extracting keywords in real time, means for acquiring related conversation history and common interests from the server based on the extracted keywords, means for notifying the user of the acquired information, means for recognizing emotion data from the user's voice and facial expressions using an emotion engine and transmitting the emotion data to the server, and means for the server to analyze the emotion data, associate the emotion data with the profile, generate appropriate information and reactions, and transmit the information and reactions to the terminal. This enables personalized entertainment content to be proposed that takes the user's emotions into consideration.

[1828] "Facial recognition data" is data that captures a user's face in digital form and analyzes its features.

[1829] A "server" is a computer system that provides and manages data on a network and exchanges information with client terminals.

[1830] "Profile information" is a collection of information related to a particular user, including data such as name, contact information, conversation history, hobbies and interests, etc.

[1831] A "terminal" is a device that is directly operated by a user and displays or provides information via audio, such as a smartphone or smart glasses.

[1832] "Voice data" refers to data that is used to record and analyze user utterances and conversation content in digital form.

[1833] "Keywords" are important words or phrases extracted from audio data and used to identify themes and key points of a conversation.

[1834] "Conversation history" is data that records the contents of past conversations, and is used to understand the flow of communication between users.

[1835] "SNS information" refers to information about posts and activities made public by users on social networking services.

[1836] An "emotion engine" is a system or software that analyzes emotional data from a user's voice and facial expressions and recognizes their state in real time.

[1837] "Suggested information" refers to information or content suggestions that the system provides to the user, and is generated based on the user's profile information and emotional state.

[1838] "Privacy settings" are settings that allow users to decide with whom they want to share their information and to what extent they want to make it public.

[1839] "Permission information" is data that records the content and scope of information handling that a user has approved.

[1840] This invention supports user communication and appropriately suggests entertainment content by building a system that uses an emotion engine in addition to facial recognition, voice analysis, collection and analysis of SNS information, and privacy management.

[1841] Hardware and Software Configuration

[1842] 1. Obtaining the user's facial recognition data

[1843] Hardware: Smart glasses and smartphone cameras

[1844] Software: Facial recognition API (e.g., Amazon Rekognition)

[1845] Description: When a user turns on the device, the camera captures facial recognition data and sends it to a server, which then matches the data with a database to find matching profile information and send it to the device.

[1846] 2. Collection and analysis of audio data

[1847] Hardware: Smart glasses and smartphone microphones

[1848] Software: Speech analysis API (e.g., Google Cloud Speech-to-Text)

[1849] ...

Claims

1. means for obtaining facial recognition data of a user; means for transmitting the acquired facial recognition data to a server; A means for matching the facial recognition data with a database on a server, searching for matching profile information, and transmitting the matching profile information to the terminal; means for providing the profile information sent from the server to the user; A means for collecting voice data and extracting keywords in real time; A means for obtaining related conversation history and common interests from the server based on the extracted keywords; means for notifying the user of the acquired information; A system including:

2. A method for periodically collecting public information about users' SNS accounts and saving it in their profiles; A means of analyzing collected SNS information to extract users' hobbies and interests, means for generating suitable proposal information based on the extracted information and transmitting the proposal information to the terminal; means for notifying the user of the transmitted proposal information; The system of claim 1 further comprising:

3. a means for a user to enter basic profile information when using the system for the first time; A means to set your privacy preferences and choose who you share information with; A means for storing the set permission information in a database and associating it with a profile; The system of claim 1 further comprising:

Images