system

Abstract

Provide a system. A means for storing personal information of a user in a storage means; A means for recognizing a user's face by a facial recognition means; a means for recognizing a user's voice by a voice recognition means; means for providing appropriate assistance based on the storage means; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, busy businessmen and the elderly are increasingly in need of assistance with schedule management and memory. However, existing systems have not been able to adequately address these issues. In particular, the lack of systems that can effectively assist with memory and daily life is a major problem for elderly people suffering from memory decline and those who feel lonely. Therefore, there is a need for a system that can recognize the user's face and voice, remember personal information, and provide appropriate support. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means. First, a system is used that includes a means for storing personal information of a user in a storage means, a means for recognizing the user's face using a face recognition means, and a means for recognizing the user's voice using a voice recognition means. Furthermore, by providing a means for providing appropriate assistance based on the storage means, it is possible to provide effective assistance to the user. Furthermore, by adding a means for managing the user's schedule and a means for providing information about the user's hobbies and preferences, it is possible to provide more diverse assistance. In this way, a personal assistant system that can provide assistance customized to a specific individual is realized.

[0006] "Storage means" refers to a device or method for storing personal information of a user and retrieving and using that information as needed.

[0007] "Facial recognition means" refers to a device or method for analyzing a user's facial image and identifying the user based on its features.

[0008] "Speech recognition means" refers to a device or method for analyzing a user's voice and identifying the user based on its characteristics.

[0009] "Appropriate support" refers to the act of providing information and taking action according to the user's needs and condition based on the user's personal information stored in the storage means.

[0010] A "schedule management means" is a device or method for recording and managing a user's schedules and events.

[0011] "Information about hobbies and preferences" is data about a user's areas of interest and preferred activities, and is information that reflects that person's personal preferences. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. This system has the following functions.

[0034] First, the server has the function of receiving a user's facial image. This image is uploaded to the server from the device. The server analyzes the facial image using a facial recognition algorithm and extracts its features. The obtained features are compared with an existing database to identify the user's ID. This allows the server to recognize the user's face.

[0035] Next, the user provides their voice to the server, which is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. Based on these voice characteristics, the server compares them with an existing database to identify the user's ID, making it possible to recognize the user through their voice.

[0036] Once the user is identified, the server uses the storage means to store the user's personal information. This information includes the user's name, age, hobbies, preferences, etc. The server can also store the user's schedule information. When the user's information is updated, the server immediately updates the storage means.

[0037] As a concrete example, consider the case where a user wants to add a new appointment. The user communicates the appointment to the server by voice. For example, the user might say, "Add a meeting for next Tuesday." The server first performs voice recognition to identify the user's ID. Next, it uses the storage means to retrieve the user's schedule and adds the new appointment. The user is reminded of this appointment at the appropriate time. This allows the user to manage their schedule efficiently.

[0038] Furthermore, if the user wants to remember something, the server can refer to the storage means and provide the user with past information. For example, if the user instructs the server to "tell me a list of past birthdays," the server will provide the user with stored birthday information. In this way, the server provides appropriate assistance to support the user's daily life.

[0039] This system combines facial and voice recognition to effectively manage users' personal information and provide appropriate assistance, making it extremely useful for busy businessmen, elderly people suffering from memory loss, and people who feel lonely.

[0040] The processing flow will be explained below.

[0041] Step 1:

[0042] Server: The system is initialized and an instance of the PersonalAssistantAI class is created. At this time, user_data is passed as an argument and the internal memory is initialized.

[0043] Step 2:

[0044] Terminal: The user captures a facial image and uploads it to the server, which provides the facial image to the system.

[0045] Step 3:

[0046] Server: The recognize_face method is called, receiving the uploaded face image as an argument. The server analyzes the face image using a facial recognition algorithm and extracts facial features. The extracted features are then compared with an existing database to identify the user's ID.

[0047] Step 4:

[0048] User: Records audio and sends it to the server. This audio includes the user's statements and commands.

[0049] Step 5:

[0050] Server: The recognize_voice method is called to accept a voice sample. The speech is analyzed using a speech recognition algorithm to extract voice features. These features are then matched with an existing database to identify the user's ID.

[0051] Step 6:

[0052] Device: Sends new information (e.g., name, age, etc.) to the server. Sends data containing the user's updated information.

[0053] Step 7:

[0054] Server: Calls the update_memory method, receiving the user ID and new information as arguments. This information is saved or updated in internal memory.

[0055] Step 8:

[0056] Server: Provides appropriate support based on the user's individual information. For example, it manages the user's schedule and stores new events in a storage device.

[0057] Step 9:

[0058] Terminal: When a user wants to obtain specific information, they send a request to the server by voice or text.

[0059] Step 10:

[0060] Server: Calls the get_memory method to retrieve the requested information from internal memory and provide it to the user. For example, it returns a list of past events or birthdays.

[0061] Step 11:

[0062] Server: It also stores information about the user's hobbies and preferences and provides that information upon request. For example, it can provide information about the user's favorite movies and music.

[0063] These steps allow the server to use facial and voice recognition to identify users, and then provide individual memories, schedule management, and appropriate information.

[0064] Example 1

[0065] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0066] In today's world, users must manage a large amount of information in their daily lives and work, placing a significant burden on them. Furthermore, when using various devices or services, users must authenticate each time, resulting in inconvenient usage. Furthermore, declining memory and information overload can make management cumbersome. There is a need for technology that can resolve these issues, efficiently and intuitively manage users' personal information, and provide appropriate support.

[0067] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0068] In this invention, the server includes means for receiving a user's facial image, analyzing the facial image using a facial recognition algorithm, and extracting features, means for comparing the features of the received facial image with an existing database to identify the user's ID, means for receiving the user's voice, analyzing the voice using a voice recognition algorithm, and extracting features of the voice, means for comparing the features of the received voice with an existing database to identify the user's ID, means for storing the user's personal information using a storage means when the user is identified and updating the information as necessary, and means for providing appropriate support based on the stored user information. This makes it possible to efficiently manage the user's personal information and provide appropriate support by intuitively authenticating the user using their face or voice.

[0069] "User's facial image" refers to a still image or video frame of the user's face.

[0070] "Facial recognition algorithm" is a general term for computational methods and programs that extract feature points from facial images and identify individual people.

[0071] "Extracting features" refers to the process of extracting important parameters, shapes, and patterns necessary for identification from facial images and voice.

[0072] "Existing database" refers to a collection of information that has been stored up to now, including personal information about users (such as facial features and voice features).

[0073] "Identifying the user's ID" means uniquely identifying the user by matching the extracted features with an existing database.

[0074] "Speech recognition algorithm" is a general term for computational methods and programs used to analyze speech signals and recognize language and spoken content.

[0075] "Storage means" refers to a storage device or data storage for storing a user's personal information.

[0076] "User's personal information" refers to individual data and information about the user, such as name, age, hobbies, preferences, and schedule.

[0077] "Providing appropriate assistance" means providing necessary information and performing tasks according to the user's requests and circumstances.

[0078] This invention is a personal assistant system that recognizes a user's face and voice, memorizes personal information, and provides appropriate assistance. This system is implemented using the following hardware and software.

[0079] Hardware and software used

[0080] Hardware: Servers, devices (smartphones, tablets, PCs, etc.), microphones, cameras

[0081] Software: Facial recognition algorithms, voice recognition algorithms, database management systems, storage means (memory)

[0082] First, the server has a means to receive the user's facial image. The user captures the facial image using a camera and uploads it to the server from their device. Next, the server uses a facial recognition algorithm to analyze the received facial image and extract its features. The facial recognition algorithm identifies the contours of the face and the relative positions of each feature, and extracts feature points. These feature points are then compared with facial data in an existing database to identify the user's ID.

[0083] Next, the user provides their voice to the server via a microphone. The server is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. The speech recognition algorithm recognizes language-specific patterns and individual voice characteristics from the sound waveforms and extracts features. These voice characteristics are then compared with an existing database to identify the user's ID.

[0084] Once the user's ID is identified, the server uses a storage device to store the user's personal information, including the user's name, age, hobbies, preferences, and schedule information. The server updates this storage device whenever it receives new information from the user.

[0085] For example, if a user wants to add a new appointment, they can send a voice command to the server. The user commands, "Add a meeting for next Tuesday." The server performs voice recognition to identify the user's ID, then retrieves the user's schedule using the storage means and adds the new appointment. The user is reminded of this new appointment at the appropriate time.

[0086] If the user wants to recall past information, the server can provide that information by referring to the storage means. For example, if the user gives a voice command such as "Tell me a list of past birthdays," the server will provide the user with the birthday information it has stored.

[0087] Prompt Sentence Examples

[0088] "Please describe a scenario where a user wants to add a new appointment."

[0089] "Please specify how the system responds when a user retrieves a list of past birthdays."

[0090] By combining facial and voice recognition, this system effectively manages users' personal information and supports their daily lives. It is particularly useful for busy business people and elderly people who suffer from memory loss. This technology frees users from the hassle of managing their information, enabling them to live more efficient lives.

[0091] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0092] Step 1:

[0093] Input: The user activates the device's camera and captures a facial image.

[0094] Specific operation: A facial image is captured on the device.

[0095] Output: The device uploads the captured face image to the server.

[0096] Step 2:

[0097] Input: The server receives the face image sent from the device.

[0098] Specific operation: The server runs a facial recognition algorithm and extracts feature points from the facial image.

[0099] Output: The extracted features are obtained.

[0100] Step 3:

[0101] Input: The extracted features are matched against face data from an existing database.

[0102] What happens: The server uses a database management system to perform the matching process.

[0103] Output: The user's ID is identified.

[0104] Step 4:

[0105] Input: The user uses a microphone to input voice into the server.

[0106] What happens: Audio is captured by the device and sent to the server.

[0107] Output: Audio data arrives at the server.

[0108] Step 5:

[0109] Input: The server processes the received audio data.

[0110] What happens: The server runs a speech recognition algorithm to analyze and extract speech features.

[0111] Output: Extracted audio features are obtained.

[0112] Step 6:

[0113] Input: The extracted audio features are matched against existing audio data in a database.

[0114] What happens: The server uses a database management system to perform the matching process.

[0115] Output: The user's ID is identified.

[0116] Step 7:

[0117] Input: After the user's ID is identified, the server retrieves the user's personal information from the storage means.

[0118] Specific operations: The server reads the relevant data from the storage device.

[0119] Output: The user's personal information (name, age, hobbies, preferences, schedule information, etc.) is obtained.

[0120] Step 8:

[0121] Input: If a user wants to add a new event, they say, "Add a meeting for next Tuesday."

[0122] Specific operation: The server performs voice recognition, confirms the user's ID, retrieves existing schedule information, and adds new information.

[0123] Output: A new event is added to the schedule and a reminder is sent at the appropriate time.

[0124] Step 9:

[0125] Input: If the user wants to see past information, they can say, "Give me a list of past birthdays."

[0126] Specific operation: The server performs voice recognition, confirms the user's ID, and then retrieves the birthday information from the storage means.

[0127] Output: The birthdate information is provided to the user.

[0128] This system allows users to intuitively manage their personal information and receive appropriate support using only their face and voice.

[0129] (Application example 1)

[0130] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0131] In modern society, there is a demand for stronger security in office buildings and apartment complexes. In particular, the difficulty of quickly and accurately authenticating visitors and managing their access using conventional methods places an increased burden on administrators. Furthermore, there is a risk of incomplete records and misidentification, making it an urgent need to improve security levels.

[0132] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0133] In this invention, the server includes a means for storing personal information of users in a storage means, a means for recognizing the user's face using a facial recognition means, and a means for recognizing the user's voice using a voice recognition means. This enables a means for managing access based on facial recognition, a means for managing access based on voice recognition, a means for storing visitor records, and a means for providing appropriate support based on the storage means. This improves the security level and reduces the burden on administrators.

[0134] "Storage means" refers to a device or system that stores information about a user's personal information, schedule information, hobbies and preferences.

[0135] A "face recognition means" is a device or algorithm that analyzes a user's facial image, extracts its features, and identifies the individual.

[0136] "Voice recognition means" refers to a device or algorithm that analyzes a user's voice, extracts its characteristics, and identifies the individual.

[0137] The "means for providing appropriate support" is a device or system that provides the user with necessary information and services based on the storage means.

[0138] A "means for managing access based on facial recognition" is a device or system that uses facial recognition means to identify individuals and control access rights.

[0139] A "means for controlling access based on voice recognition" is a device or system that uses voice recognition means to identify individuals and control access rights.

[0140] "Means for storing visitor records" refers to a device or system that stores visitor access history and authentication results in a database or the like.

[0141] The system required to implement this invention is mainly composed of a server equipped with a storage means, a face recognition means, and a voice recognition means. The specific system configuration and its operation will be described below.

[0142] 1. Storage means

[0143] The server is equipped with a storage means for saving personal information of users (e.g., name, age, hobbies, preferences, schedule information). This storage means uses a database management system (DBMS). For example, a database such as SQLite or MySQL (registered trademark) is used.

[0144] 2. Facial Recognition Methods

[0145] The server receives the facial image sent from the device and identifies the individual using a facial recognition algorithm, such as the Python face_recognition library. The server then compares the features of the recognized facial image with a database to identify the user's ID.

[0146] 3. Voice Recognition Method

[0147] The server receives the voice data sent from the device and identifies the individual using a speech recognition algorithm. This algorithm uses the Python speech_recognition library and the Google® Speech Recognition API. The server compares the recognized voice characteristics with a database to identify the user's ID.

[0148] 4. Access Management

[0149] The server manages access rights for identified users based on facial and voice recognition. For example, it processes video and audio transmitted from cameras and microphones installed at the entrances of office buildings and apartment complexes to authenticate individuals. Authenticated users can then be given permission to unlock automatic doors and use elevators.

[0150] 5. Record-Keeping

[0151] The server stores the results of access control and visitor records in a database, including information such as the visitor's ID, authentication date and time, and authentication location.

[0152] As a concrete example, the following scenario can be imagined:

[0153] Use case: At the entrance of an office building, administrators use facial and voice recognition to recognize visitors and automatically manage access permissions.

[0154] input:

[0155] Image file (e.g. visitor_image.jpg)

[0156] Audio file (e.g. visitor_audio.wav)

[0157] An example prompt is:

[0158] "Upload a photo: visitor_image.jpg"

[0159] "Please upload audio for speech recognition: visitor_audio.wav"

[0160] This system allows administrators to quickly and accurately authenticate visitors, streamlining access management, and storing records in a database for future review and tracking of unauthorized access.

[0161] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0162] Step 1:

[0163] A user uses a terminal to provide a facial image to a camera at the entrance. The terminal captures this image and sends it to a server. The input is a facial image, and the output is image data.

[0164] Step 2:

[0165] The server passes the received facial image to a facial recognition algorithm to extract facial features. The extracted features are then matched against an existing database. The input is facial image data, and the output is the match result and the user's ID. Specifically, the Python face_recognition library is used to obtain the facial encoding and compare it with known encodings in the database.

[0166] Step 3:

[0167] The user uses a terminal to provide voice data to the microphone at the entrance. The terminal records this voice and sends it to the server. The input is voice data, and the output is an audio file.

[0168] Step 4:

[0169] The server passes the received audio data to a speech recognition algorithm to extract audio features. The extracted features are compared with an existing database. The input is the audio data, and the output is the speech recognition results and the user's ID. Specifically, the Python speech_recognition library and the Google Speech Recognition API are used to convert the audio into text and compare it with known audio data in the database.

[0170] Step 5:

[0171] The server determines the user's identity based on the results of facial and voice recognition. If both results match, the user is considered to have been successfully authenticated. The input is the facial and voice recognition results, and the output is the authentication result.

[0172] Step 6:

[0173] If the authentication is successful, the server sends an instruction to unlock the entrance door to the authorized user via the access management means. The input is the authentication result, and the output is a command to unlock the door. Specifically, the server sends a signal to the electromagnetic locking device.

[0174] Step 7:

[0175] The server stores the visitor's authentication result and the access date and time in a recording medium. The input is the user's ID, authentication result, and date and time information, and the output is the result stored in a database. Specifically, this information is stored using an SQLite or MySQL database.

[0176] Example prompt sentence:

[0177] "Upload a photo: visitor_image.jpg"

[0178] "Please upload audio for speech recognition: visitor_audio.wav"

[0179] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0180] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. Furthermore, by incorporating an emotion engine, it has the ability to recognize and respond to the user's emotional state.

[0181] First, the system has the following functions: The server has the function of receiving a user's facial image. This image is uploaded to the server from the terminal. The server analyzes the facial image using a facial recognition algorithm and extracts its features. The obtained features are compared with an existing database to identify the user's ID. This allows the server to recognize the user's face.

[0182] Next, the user provides their voice to the server, which is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. Based on these voice characteristics, the server compares them with an existing database to identify the user's ID, making it possible to recognize the user through their voice.

[0183] Once the user is identified, the server uses the storage means to store the user's personal information. This information includes the user's name, age, hobbies, preferences, etc. The server can also store the user's schedule information. When the user's information is updated, the server immediately updates the storage means.

[0184] Furthermore, the system incorporates an emotion engine that analyzes the user's facial expressions and vocal tone to recognize their current emotional state. For example, if the user shows signs of discomfort, the server can select the appropriate assistance method based on that information.

[0185] As a concrete example, consider the case where a user is feeling stressed. The user communicates their current feelings to the server by voice. For example, if the user says, "I'm feeling a little irritated today," the server performs voice recognition and identifies the user's ID. Next, the emotion engine analyzes the tone and content of the voice and determines that the user is feeling stressed. In response, the server suggests relaxing music and provides relaxation advice. It is also possible to record the user's stress level so that it can be reviewed later.

[0186] Furthermore, if a user has an important presentation coming up, they input their schedule into the server, which not only remembers the information and reminds them before the presentation, but also uses an emotion engine to detect tension before the presentation and send them relaxation advice or encouraging messages.

[0187] In this way, by combining facial recognition, voice recognition, memory means, and an emotion engine, this system provides customized assistance to users, making it extremely useful for busy businessmen, elderly people suffering from memory decline, and people who feel lonely.

[0188] The processing flow will be explained below.

[0189] Step 1:

[0190] Terminal: The user captures a facial image with a camera and uploads it to the server.

[0191] Step 2:

[0192] Server: Calls the recognize_face method to receive the uploaded face image. It analyzes the face image using a facial recognition algorithm and extracts its features. The extracted features are compared with an existing database to identify the user's ID.

[0193] Step 3:

[0194] Device: The user uses a microphone to record voice and sends the voice samples to the server.

[0195] Step 4:

[0196] Server: Calls the recognize_voice method to receive a voice sample. Analyzes the voice using a speech recognition algorithm and extracts voice features. Matches the extracted features with an existing database to identify the user's ID.

[0197] Step 5:

[0198] Terminal: The user enters their information (e.g., name, age, etc.) and sends that information to the server.

[0199] Step 6:

[0200] Server: Call the update_memory method to save or update the user's personal information, including the user's name, age, hobbies, and preferences.

[0201] Step 7:

[0202] Server: Updates the user's schedule information. When a user adds a new event (e.g., a meeting), the server records this information in memory.

[0203] Step 8:

[0204] Terminal: The user sends their current feelings and state to the server via voice or text.

[0205] Step 9:

[0206] Server: Analyzes the received voice and text and recognizes the user's emotional state using an emotion engine, for example by analyzing voice tone and expressed emotions.

[0207] Step 10:

[0208] Server: Provides appropriate support based on the user's recognized emotional state. For example, if the user is feeling stressed, it can provide relaxing music or relaxation advice.

[0209] Step 11:

[0210] Server: Uses an emotion engine to monitor the user's emotional state before a specific event (e.g., before a presentation) and sends encouraging messages or advice as needed.

[0211] Step 12:

[0212] Server: Records past support history and emotional state, and supports users in self-analysis and reflection in the future.

[0213] Through these steps, the server can combine face recognition, voice recognition, memory means, and an emotion engine to provide customized assistance to users, making it an extremely useful service for busy businessmen, elderly people suffering from memory loss, or people who feel lonely.

[0214] Example 2

[0215] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0216] Conventional personal assistant systems have basic functions such as recognizing the user's face and voice and remembering personal information, but they are unable to recognize the user's emotional state and respond based on that, making it difficult to provide truly beneficial assistance to the user. Furthermore, they lack advanced support such as managing the user's schedule and suggesting ways to reduce stress based on the user's emotional state.

[0217] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing a face image and transmitting the face image, a means for analyzing the face image using a face recognition algorithm and extracting features, and a means for matching the features with an existing database and identifying the user's ID. This enables face recognition of the user.

[0218] The server further includes means for recording and transmitting the voice, means for analyzing the voice using a voice recognition algorithm to extract characteristics, and means for matching the voice characteristics with an existing database to identify the user's identity, thereby enabling voice recognition of the user.

[0219] The server also includes a means for storing personal information of the user, a means for providing appropriate support based on the stored information, a means for recognizing the emotional state of the user by analyzing facial expressions and voice tones, and a means for providing appropriate responses based on the emotional state, thereby making it possible to provide support suited to the emotional state of the user.

[0220] Finally, the server includes a means for storing the user's schedule and providing timely reminders, and a means for providing stress reduction suggestions based on the user's emotional state, so that the user can avoid forgetting important plans and receive appropriate support when feeling stressed.

[0221] A "face image" is still image data of a photograph of the user's face.

[0222] A "server" is a computer system that operates on a network and receives, processes, stores, and transmits data.

[0223] A "facial recognition algorithm" is a mathematical model or computational method for extracting specific features from a facial image and identifying an individual based on those features.

[0224] Extracting "features" is a process of obtaining specific patterns or attributes as numerical data from facial images or voice data.

[0225] A "database" is a data structure that systematically stores and makes searchable personal information, facial feature data, voice feature data, and other information about users.

[0226] "ID identification" is the process of matching characteristic data with a database to identify matching personal identifiers.

[0227] "Voice" is waveform data of the voice uttered by the user.

[0228] A "speech recognition algorithm" is a mathematical model or computational method that analyzes linguistic information and specific features from speech data and converts them into text data or identification information.

[0229] "Personal information" refers to data relating to an individual, such as a user's name, age, hobbies, preferences, schedule information, etc.

[0230] "Storage" is the process of saving information in a storage medium such as a database.

[0231] An "emotion engine" is a computational model or system that analyzes facial expressions and vocal tone to recognize a user's emotional state.

[0232] "Appropriate support" is the process of providing advice and services that are tailored to the user's needs and condition.

[0233] "Schedule" is information that describes the user's plans and plans.

[0234] "Remind" is the process of notifying users of specific events or information in advance.

[0235] "Stress reduction suggestions" is the process of recommending advice or activities aimed at reducing stress when a user is feeling stressed.

[0236] MODE FOR CARRYING OUT THE INVENTION

[0237] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. By incorporating an emotion engine, this system is also capable of recognizing and responding to the user's emotional state.

[0238] Hardware and software used

[0239] Hardware: Smartphones, PCs, servers

[0240] software:

[0241] Face Recognition Algorithm: OpenCV, FaceNet

[0242] Speech recognition algorithm: Google Speech-to-Text, IBM Watson(R)

[0243] Emotion engine: Microsoft® Azure® Emotion API, Affectiva

[0244] Facial Recognition Processing

[0245] 1. Image capture

[0246] The user takes a picture of their face using a device (such as a smartphone or PC camera).

[0247] The device temporarily stores the captured image.

[0248] 2. Upload an image

[0249] The device uploads the stored facial image to the server using an HTTP request.

[0250] 3. Image Analysis

[0251] The server analyzes the received image using a facial recognition algorithm (e.g., OpenCV's face detection library or FaceNet model) and extracts facial feature points.

[0252] 4. ID identification

[0253] The server compares the extracted feature data with an existing database to identify the user's ID.

[0254] Speech Recognition Processing

[0255] 1. Voice input

[0256] The user records the voice using the microphone of the device.

[0257] The device temporarily stores the recorded audio.

[0258] 2. Upload audio

[0259] The device uploads the saved audio data to the server using an HTTP request.

[0260] 3. Audio Analysis

[0261] The server analyzes the received audio using a speech recognition algorithm (e.g., Google Speech-to-Text API or IBM Watson Speech to Text) and extracts features from the audio data.

[0262] 4. ID identification

[0263] The server matches the voice characteristics with information in a user database to determine the user's ID.

[0264] Information storage processing

[0265] 1. Information Record

[0266] The server stores the identified user's personal information (such as name, age, hobbies, preferences, schedule information, etc.) in a database.

[0267] 2. Information update

[0268] The server immediately updates the database when user information changes.

[0269] 3. Information provision

[0270] The server uses the recorded information as needed to provide appropriate assistance to the user.

[0271] Emotion Recognition Processing

[0272] 1. Sentiment analysis

[0273] The server uses an emotion engine (Microsoft Azure Emotion API or Affectiva) to analyze facial expressions and vocal tone.

[0274] 2. Providing support

[0275] The server provides an appropriate response based on the results of the sentiment analysis.

[0276] Specific examples

[0277] Dealing with stressed users

[0278] The user uses the device to voice input, "I'm feeling a bit frustrated today."

[0279] The terminal uploads the audio data to the server.

[0280] The server performs voice recognition to identify the user's identity and analyzes the voice tone with an emotion engine.

[0281] The server recognizes the user's emotional state as "stress" and suggests ways to relax.

[0282] What to do if you have an important presentation coming up

[0283] The user uses the terminal to enter schedule information (date and time of presentation).

[0284] The terminal uploads the schedule information to the server.

[0285] The server stores the information and sends a reminder the day before the presentation.

[0286] On the day of the presentation, the emotion engine is used to recognize the user's state of tension and send relaxation advice and encouraging messages.

[0287] Prompt Sentence Examples

[0288] User Emotion Recognition

[0289] If a user verbally tells a server, "I'm feeling a bit frustrated today," explain how the server will recognize the user and what action to take.

[0290] User schedule management

[0291] If a user enters the schedule for an important presentation, explain how the system will support the user and what assistance it will provide.

[0292] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0293] Facial Recognition Processing

[0294] Step 1: Image capture

[0295] The user takes a picture of his or her face using the camera on the terminal.

[0296] Input: A still image of a face.

[0297] Output: Facial image files temporarily saved on your device.

[0298] Specific operation: The device's camera app is launched, and the user faces the camera and takes a picture of their face. The device saves the captured image to its internal storage.

[0299] Step 2: Upload an image

[0300] The device uploads the stored facial image to the server.

[0301] Input: A face image file stored on the device.

[0302] Output: Facial image data sent to the server.

[0303] Specific operation: Using an HTTP request, the device sends facial image data to the server. The server receives the data and stores it in storage.

[0304] Step 3: Image analysis

[0305] The server analyzes the received facial images using a facial recognition algorithm (e.g., OpenCV or FaceNet).

[0306] Input: Facial image data stored on the server.

[0307] Output: Extracted facial feature data.

[0308] Specific operation: Runs a facial recognition algorithm to extract feature points such as eyes, nose, and mouth from the image as numerical data.

[0309] Step 4: Identify ID

[0310] The server compares the extracted feature data with an existing database to identify the user's ID.

[0311] Input: Extracted facial feature data.

[0312] Output: The matched user's ID.

[0313] Specific behavior: Compares the user data and feature data in the database to identify matching users, and stores the identified user IDs for further processing.

[0314] Speech Recognition Processing

[0315] Step 1: Voice Input

[0316] The user records their own voice using the device's microphone.

[0317] Input: Audio as voice.

[0318] Output: An audio file temporarily saved on your device.

[0319] Specific operation: Launch the device's recording app to record the user's voice. After recording, the device will save the audio file to its internal storage.

[0320] Step 2: Upload your audio

[0321] The device uploads the saved audio data to the server.

[0322] Input: An audio file stored on your device.

[0323] Output: The audio data sent to the server.

[0324] Specific operation: Using an HTTP request, the device sends audio data to the server, which receives it and stores it in storage.

[0325] Step 3: Audio analysis

[0326] The server analyzes the received voice data using a speech recognition algorithm (e.g., Google Speech-to-Text API or IBM Watson Speech to Text).

[0327] Input: Audio data stored on the server.

[0328] Output: Extracted audio feature data.

[0329] What it does: Runs a speech recognition algorithm to extract linguistic information and specific features from speech as numerical data.

[0330] Step 4: Identify ID

[0331] The server matches the voice characteristics data with an existing database to determine the user's ID.

[0332] Input: Extracted audio feature data.

[0333] Output: The matched user's ID.

[0334] Specific behavior: Compares user data in the database with the voice feature data to identify matching users, and stores the identified user's ID for further processing.

[0335] Information storage processing

[0336] Step 1: Record information

[0337] The server stores the identified user's personal information in a database.

[0338] Input: User's personal information (name, age, hobbies, preferences, schedule, etc.).

[0339] Output: User information recorded in the database.

[0340] What it does: Adds new information about the user to the database and updates existing information as needed.

[0341] Step 2: Update your information

[0342] The server immediately updates the database when user information changes.

[0343] Input: New user information.

[0344] Output: The updated database.

[0345] Specific operation: When new information is received, it is compared with the existing information, and if it differs, an update process is performed.

[0346] Step 3: Provide information

[0347] The server uses the recorded information as needed to provide appropriate assistance to the user.

[0348] Input: User information recorded in the database.

[0349] Output: The assistance provided to the user.

[0350] What it does: Set reminders based on the user's schedule information and suggest activities based on their preferences.

[0351] Emotion Recognition Processing

[0352] Step 1: Sentiment analysis

[0353] The server uses an emotion engine (Microsoft Azure Emotion API or Affectiva) to analyze facial expressions and vocal tone.

[0354] Input: Facial image data, audio data.

[0355] Output: User's emotional state data.

[0356] Specific operation: Executes the emotion engine, extracts feature points indicating emotions from facial images and tone changes from audio, and determines the current emotional state.

[0357] Step 2: Provide support

[0358] The server provides an appropriate response based on the results of the sentiment analysis.

[0359] Input: Emotional state data.

[0360] Output: Emotion-based support content.

[0361] Specific behavior: If the emotional state is determined to be "stressed," the system will suggest relaxing activities and provide advice to reduce stress.

[0362] (Application example 2)

[0363] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0364] Elderly people need daily support in their daily lives, but conventional personal assistant systems lack the ability to properly recognize the emotional state of the elderly and provide individually customized support. There is also a need for systems that can provide relaxation advice and encouraging messages tailored to the elderly's emotional state. The purpose of this invention is to solve these problems.

[0365] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing personal information of the user in storage means, means for recognizing the user's face by face recognition means, means for recognizing the user's voice by voice recognition means, means for analyzing the user's emotional state by an emotion engine, and means for providing support customized according to the user's emotional state. This makes it possible to provide individually customized support based on the user's emotional state and support the elderly in their daily lives.

[0366] "Storage means" refers to a technical element for storing personal information of a user.

[0367] "Facial recognition means" refers to a technical element that recognizes the face of a user, and is a means of analyzing facial images to identify individuals.

[0368] "Voice recognition means" refers to a technical element that recognizes the user's voice, extracts voice characteristics, and performs personal identification and analysis of voice instructions.

[0369] The "means for providing appropriate support" refers to a technical element that provides optimal support to a user based on the storage means.

[0370] The "emotion engine" is a technology element that analyzes the user's facial expressions and tone of voice to recognize their emotional state.

[0371] The "means for providing support customized according to the user's emotional state" is a technical element that provides optimal support individually based on the user's emotional state recognized by the emotion engine.

[0372] The "schedule management means" is a technical element that manages the user's schedule and important events based on the storage means, and sends reminders and notifications at appropriate times.

[0373] "Means for providing reminders and notifications" refers to technical elements that provide necessary reminders and notifications according to the user's emotional state and schedule.

[0374] The "means for providing relaxation advice and encouraging messages" refers to a technical element that determines the emotional state of the user and provides appropriate relaxation methods and encouraging messages.

[0375] The system of the present invention includes a memory means, a face recognition means, a voice recognition means, an emotion engine, and a means for providing customized assistance according to the user's emotional state in order to support the elderly in their daily lives. The system is configured and implemented as follows.

[0376] First, the storage means stores the user's personal information, including name, age, hobbies, preferences, schedule, etc. The device (smartphone or head-mounted display) periodically transmits this information to the server, which stores it in a database.

[0377] Next, the facial recognition means uses the device's built-in camera to recognize the user's face, and the recognized facial features are sent to the server, where a facial recognition algorithm (e.g., OpenCV) analyzes them and matches them with a database to identify the user.

[0378] The speech recognition method uses the device's microphone to transmit the user's voice input to the server, where a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API) analyzes it and identifies the user's voice commands and emotions.

[0379] The emotion engine is a technology component that analyzes the user's facial expressions and vocal tone to recognize their current emotional state. The emotion engine can use, for example, the Microsoft Azure Emotion API. The results of this analysis are sent to a server to help guide the next steps.

[0380] The server uses this information to provide customized support according to the user's emotional state. For example, if it determines that an elderly person is feeling stressed or anxious, the server can send relaxation advice or encouraging messages to the device. Examples of such messages include videos of breathing exercises to help the elderly person relax or the playing of relaxing music.

[0381] Furthermore, the server uses the storage means to manage the user's schedule and provide reminders and notifications as needed, such as timely reminders for important medical appointments or medication times, and can provide relaxation advice if the emotion engine determines that the user's emotional state is tense.

[0382] Examples:

[0383] If an elderly person says, "I'm feeling a little anxious today,"

[0384] The server performs voice recognition to identify the user's ID, and the emotion engine analyzes the voice tone and content to identify the user's anxiety. Based on this information, the server sends a "relaxation breathing exercise video" to the device. In this way, the system can provide assistance tailored to the user's emotional state and improve the quality of life for the elderly.

[0385] Example prompt sentence:

[0386] If a senior is feeling anxious, "offer these next steps: Play a breathing exercise video followed by 10 minutes of relaxing music."

[0387] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0388] Step 1:

[0389] The user shows their face to the device (smartphone or head-mounted display). The device's camera captures a matching facial image and sends it to the server. The server uses a facial recognition algorithm to match the received facial image with existing facial data in a database and identify the user ID. This allows the server to recognize the user.

[0390] Input: User's face image

[0391] Data processing / data calculation: Analysis of facial images using face recognition algorithms and matching with databases

[0392] Output: User ID

[0393] Step 2:

[0394] The user enters voice commands into the device. The device's microphone captures the voice and sends it to the server. The server uses a speech recognition algorithm to convert the voice into text data and identify the user's instructions and emotional state.

[0395] Input: User's voice

[0396] Data processing / data calculation: Analysis of voice data using speech recognition algorithms and conversion to text

[0397] Output: Text data (user instructions, emotional state)

[0398] Step 3:

[0399] An emotion engine analyzes the text data and voice tone to determine the user's current emotional state. The server uses the emotion engine to identify the user's emotional state (e.g., stress, anxiety, joy, etc.).

[0400] Input: Text data and audio tones

[0401] Data processing / data calculation: Emotional state analysis using emotion engine

[0402] Output: User's emotional state

[0403] Step 4:

[0404] The server checks the user's personal information and schedule based on the storage means, thereby understanding the user's hobbies, preferences, plans, etc., and determines appropriate support content based on the user's emotional state.

[0405] Input: User's emotional state, personal information, schedule

[0406] Data processing / data calculation: Referencing and analyzing information using memory means

[0407] Output: Support content

[0408] Step 5:

[0409] The server provides customized support to the user based on the corresponding support content. For example, if the user feels anxious, the server will send a relaxing breathing exercise video to the device and instruct it to play. The server will also record the support content and the user's emotional state in a database for future reference.

[0410] Input: Support content

[0411] Data processing / data calculation: Deciding on support content and instructing distribution

[0412] Output: Support provided to the user (e.g., relaxation video)

[0413] Step 6:

[0414] The server monitors the user's emotional state and the effectiveness of the assistance provided, offering additional support as needed, or if it is ineffective, offering alternative advice or messages.

[0415] Input: User state after assistance

[0416] Data processing / data calculation: monitoring and analysis data feedback

[0417] Output: Additional support or ongoing monitoring

[0418] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0419] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0420] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0421] [Second embodiment]

[0422] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0423] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0424] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0425] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0426] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0427] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0428] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0429] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0430] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0431] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0432] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0433] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0434] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. This system has the following functions.

[0435] First, the server has the function of receiving a user's facial image. This image is uploaded to the server from the device. The server analyzes the facial image using a facial recognition algorithm and extracts its features. The obtained features are compared with an existing database to identify the user's ID. This allows the server to recognize the user's face.

[0436] Next, the user provides their voice to the server, which is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. Based on these voice characteristics, the server compares them with an existing database to identify the user's ID, making it possible to recognize the user through their voice.

[0437] Once the user is identified, the server uses the storage means to store the user's personal information. This information includes the user's name, age, hobbies, preferences, etc. The server can also store the user's schedule information. When the user's information is updated, the server immediately updates the storage means.

[0438] As a concrete example, consider the case where a user wants to add a new appointment. The user communicates the appointment to the server by voice. For example, the user might say, "Add a meeting for next Tuesday." The server first performs voice recognition to identify the user's ID. Next, it uses the storage means to retrieve the user's schedule and adds the new appointment. The user is reminded of this appointment at the appropriate time. This allows the user to manage their schedule efficiently.

[0439] Furthermore, if the user wants to remember something, the server can refer to the storage means and provide the user with past information. For example, if the user instructs the server to "tell me a list of past birthdays," the server will provide the user with stored birthday information. In this way, the server provides appropriate assistance to support the user's daily life.

[0440] This system combines facial and voice recognition to effectively manage users' personal information and provide appropriate assistance, making it extremely useful for busy businessmen, elderly people suffering from memory loss, and people who feel lonely.

[0441] The processing flow will be explained below.

[0442] Step 1:

[0443] Server: The system is initialized and an instance of the PersonalAssistantAI class is created. At this time, user_data is passed as an argument and the internal memory is initialized.

[0444] Step 2:

[0445] Terminal: The user captures a facial image and uploads it to the server, which provides the facial image to the system.

[0446] Step 3:

[0447] Server: The recognize_face method is called, receiving the uploaded face image as an argument. The server analyzes the face image using a facial recognition algorithm and extracts facial features. The extracted features are then compared with an existing database to identify the user's ID.

[0448] Step 4:

[0449] User: Records audio and sends it to the server. This audio includes the user's statements and commands.

[0450] Step 5:

[0451] Server: The recognize_voice method is called to accept a voice sample. The speech is analyzed using a speech recognition algorithm to extract voice features. These features are then matched with an existing database to identify the user's ID.

[0452] Step 6:

[0453] Device: Sends new information (e.g., name, age, etc.) to the server. Sends data containing the user's updated information.

[0454] Step 7:

[0455] Server: Calls the update_memory method, receiving the user ID and new information as arguments. This information is saved or updated in internal memory.

[0456] Step 8:

[0457] Server: Provides appropriate support based on the user's individual information. For example, it manages the user's schedule and stores new events in a storage device.

[0458] Step 9:

[0459] Terminal: When a user wants to obtain specific information, they send a request to the server by voice or text.

[0460] Step 10:

[0461] Server: Calls the get_memory method to retrieve the requested information from internal memory and provide it to the user. For example, it returns a list of past events or birthdays.

[0462] Step 11:

[0463] Server: It also stores information about the user's hobbies and preferences and provides that information upon request. For example, it can provide information about the user's favorite movies and music.

[0464] These steps allow the server to use facial and voice recognition to identify users, and then provide individual memories, schedule management, and appropriate information.

[0465] Example 1

[0466] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0467] In today's world, users must manage a large amount of information in their daily lives and work, placing a significant burden on them. Furthermore, when using various devices or services, users must authenticate each time, resulting in inconvenient usage. Furthermore, declining memory and information overload can make management cumbersome. There is a need for technology that can resolve these issues, efficiently and intuitively manage users' personal information, and provide appropriate support.

[0468] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0469] In this invention, the server includes means for receiving a user's facial image, analyzing the facial image using a facial recognition algorithm, and extracting features, means for comparing the features of the received facial image with an existing database to identify the user's ID, means for receiving the user's voice, analyzing the voice using a voice recognition algorithm, and extracting features of the voice, means for comparing the features of the received voice with an existing database to identify the user's ID, means for storing the user's personal information using a storage means when the user is identified and updating the information as necessary, and means for providing appropriate support based on the stored user information. This makes it possible to efficiently manage the user's personal information and provide appropriate support by intuitively authenticating the user using their face or voice.

[0470] "User's facial image" refers to a still image or video frame of the user's face.

[0471] "Facial recognition algorithm" is a general term for computational methods and programs that extract feature points from facial images and identify individual people.

[0472] "Extracting features" refers to the process of extracting important parameters, shapes, and patterns necessary for identification from facial images and voice.

[0473] "Existing database" refers to a collection of information that has been stored up to now, including personal information about users (such as facial features and voice features).

[0474] "Identifying the user's ID" means uniquely identifying the user by matching the extracted features with an existing database.

[0475] "Speech recognition algorithm" is a general term for computational methods and programs used to analyze speech signals and recognize language and spoken content.

[0476] "Storage means" refers to a storage device or data storage for storing a user's personal information.

[0477] "User's personal information" refers to individual data and information about the user, such as name, age, hobbies, preferences, and schedule.

[0478] "Providing appropriate assistance" means providing necessary information and performing tasks according to the user's requests and circumstances.

[0479] This invention is a personal assistant system that recognizes a user's face and voice, memorizes personal information, and provides appropriate assistance. This system is implemented using the following hardware and software.

[0480] Hardware and software used

[0481] Hardware: Servers, devices (smartphones, tablets, PCs, etc.), microphones, cameras

[0482] Software: Facial recognition algorithms, voice recognition algorithms, database management systems, storage means (memory)

[0483] First, the server has a means to receive the user's facial image. The user captures the facial image using a camera and uploads it to the server from their device. Next, the server uses a facial recognition algorithm to analyze the received facial image and extract its features. The facial recognition algorithm identifies the contours of the face and the relative positions of each feature, and extracts feature points. These feature points are then compared with facial data in an existing database to identify the user's ID.

[0484] Next, the user provides their voice to the server via a microphone. The server is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. The speech recognition algorithm recognizes language-specific patterns and individual voice characteristics from the sound waveforms and extracts features. These voice characteristics are then compared with an existing database to identify the user's ID.

[0485] Once the user's ID is identified, the server uses a storage device to store the user's personal information, including the user's name, age, hobbies, preferences, and schedule information. The server updates this storage device whenever it receives new information from the user.

[0486] For example, if a user wants to add a new appointment, they can send a voice command to the server. The user commands, "Add a meeting for next Tuesday." The server performs voice recognition to identify the user's ID, then retrieves the user's schedule using the storage means and adds the new appointment. The user is reminded of this new appointment at the appropriate time.

[0487] If the user wants to recall past information, the server can provide that information by referring to the storage means. For example, if the user gives a voice command such as "Tell me a list of past birthdays," the server will provide the user with the birthday information it has stored.

[0488] Prompt Sentence Examples

[0489] "Please describe a scenario where a user wants to add a new appointment."

[0490] "Please specify how the system responds when a user retrieves a list of past birthdays."

[0491] By combining facial and voice recognition, this system effectively manages users' personal information and supports their daily lives. It is particularly useful for busy business people and elderly people who suffer from memory loss. This technology frees users from the hassle of managing their information, enabling them to live more efficient lives.

[0492] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0493] Step 1:

[0494] Input: The user activates the device's camera and captures a facial image.

[0495] Specific operation: A facial image is captured on the device.

[0496] Output: The device uploads the captured face image to the server.

[0497] Step 2:

[0498] Input: The server receives the face image sent from the device.

[0499] Specific operation: The server runs a facial recognition algorithm and extracts feature points from the facial image.

[0500] Output: The extracted features are obtained.

[0501] Step 3:

[0502] Input: The extracted features are matched against face data from an existing database.

[0503] What happens: The server uses a database management system to perform the matching process.

[0504] Output: The user's ID is identified.

[0505] Step 4:

[0506] Input: The user uses a microphone to input voice into the server.

[0507] What happens: Audio is captured by the device and sent to the server.

[0508] Output: Audio data arrives at the server.

[0509] Step 5:

[0510] Input: The server processes the received audio data.

[0511] What happens: The server runs a speech recognition algorithm to analyze and extract speech features.

[0512] Output: Extracted audio features are obtained.

[0513] Step 6:

[0514] Input: The extracted audio features are matched against existing audio data in a database.

[0515] What happens: The server uses a database management system to perform the matching process.

[0516] Output: The user's ID is identified.

[0517] Step 7:

[0518] Input: After the user's ID is identified, the server retrieves the user's personal information from the storage means.

[0519] Specific operations: The server reads the relevant data from the storage device.

[0520] Output: The user's personal information (name, age, hobbies, preferences, schedule information, etc.) is obtained.

[0521] Step 8:

[0522] Input: If a user wants to add a new event, they say, "Add a meeting for next Tuesday."

[0523] Specific operation: The server performs voice recognition, confirms the user's ID, retrieves existing schedule information, and adds new information.

[0524] Output: A new event is added to the schedule and a reminder is sent at the appropriate time.

[0525] Step 9:

[0526] Input: If the user wants to see past information, they can say, "Give me a list of past birthdays."

[0527] Specific operation: The server performs voice recognition, confirms the user's ID, and then retrieves the birthday information from the storage means.

[0528] Output: The birthdate information is provided to the user.

[0529] This system allows users to intuitively manage their personal information and receive appropriate support using only their face and voice.

[0530] (Application example 1)

[0531] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0532] In modern society, there is a demand for stronger security in office buildings and apartment complexes. In particular, the difficulty of quickly and accurately authenticating visitors and managing their access using conventional methods places an increased burden on administrators. Furthermore, there is a risk of incomplete records and misidentification, making it an urgent need to improve security levels.

[0533] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0534] In this invention, the server includes a means for storing personal information of users in a storage means, a means for recognizing the user's face using a facial recognition means, and a means for recognizing the user's voice using a voice recognition means. This enables a means for managing access based on facial recognition, a means for managing access based on voice recognition, a means for storing visitor records, and a means for providing appropriate support based on the storage means. This improves the security level and reduces the burden on administrators.

[0535] "Storage means" refers to a device or system that stores information about a user's personal information, schedule information, hobbies and preferences.

[0536] A "face recognition means" is a device or algorithm that analyzes a user's facial image, extracts its features, and identifies the individual.

[0537] "Voice recognition means" refers to a device or algorithm that analyzes a user's voice, extracts its characteristics, and identifies the individual.

[0538] The "means for providing appropriate support" is a device or system that provides the user with necessary information and services based on the storage means.

[0539] A "means for managing access based on facial recognition" is a device or system that uses facial recognition means to identify individuals and control access rights.

[0540] A "means for controlling access based on voice recognition" is a device or system that uses voice recognition means to identify individuals and control access rights.

[0541] "Means for storing visitor records" refers to a device or system that stores visitor access history and authentication results in a database or the like.

[0542] The system required to implement this invention is mainly composed of a server equipped with a storage means, a face recognition means, and a voice recognition means. The specific system configuration and its operation will be described below.

[0543] 1. Storage means

[0544] The server is equipped with a storage means for saving personal information of users (e.g., name, age, hobbies, preferences, schedule information). This storage means uses a database management system (DBMS). For example, a database such as SQLite or MySQL is used.

[0545] 2. Facial Recognition Methods

[0546] The server receives the facial image sent from the device and identifies the individual using a facial recognition algorithm, such as the Python face_recognition library. The server then compares the features of the recognized facial image with a database to identify the user's ID.

[0547] 3. Voice Recognition Method

[0548] The server receives the voice data sent from the device and identifies the individual using a speech recognition algorithm. This algorithm uses the Python speech_recognition library and the Google Speech Recognition API. The server compares the recognized voice characteristics with a database to determine the user's ID.

[0549] 4. Access Management

[0550] The server manages access rights for identified users based on facial and voice recognition. For example, it processes video and audio transmitted from cameras and microphones installed at the entrances of office buildings and apartment complexes to authenticate individuals. Authenticated users can then be given permission to unlock automatic doors and use elevators.

[0551] 5. Record-Keeping

[0552] The server stores the results of access control and visitor records in a database, including information such as the visitor's ID, authentication date and time, and authentication location.

[0553] As a concrete example, the following scenario can be imagined:

[0554] Use case: At the entrance of an office building, administrators use facial and voice recognition to recognize visitors and automatically manage access permissions.

[0555] input:

[0556] Image file (e.g. visitor_image.jpg)

[0557] Audio file (e.g. visitor_audio.wav)

[0558] An example prompt is:

[0559] "Upload a photo: visitor_image.jpg"

[0560] "Please upload audio for speech recognition: visitor_audio.wav"

[0561] This system allows administrators to quickly and accurately authenticate visitors, streamlining access management, and storing records in a database for future review and tracking of unauthorized access.

[0562] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0563] Step 1:

[0564] A user uses a terminal to provide a facial image to a camera at the entrance. The terminal captures this image and sends it to a server. The input is a facial image, and the output is image data.

[0565] Step 2:

[0566] The server passes the received facial image to a facial recognition algorithm to extract facial features. The extracted features are then matched against an existing database. The input is facial image data, and the output is the match result and the user's ID. Specifically, the Python face_recognition library is used to obtain the facial encoding and compare it with known encodings in the database.

[0567] Step 3:

[0568] The user uses a terminal to provide voice data to the microphone at the entrance. The terminal records this voice and sends it to the server. The input is voice data, and the output is an audio file.

[0569] Step 4:

[0570] The server passes the received audio data to a speech recognition algorithm to extract audio features. The extracted features are compared with an existing database. The input is the audio data, and the output is the speech recognition results and the user's ID. Specifically, the Python speech_recognition library and the Google Speech Recognition API are used to convert the audio into text and compare it with known audio data in the database.

[0571] Step 5:

[0572] The server determines the user's identity based on the results of facial and voice recognition. If both results match, the user is considered to have been successfully authenticated. The input is the facial and voice recognition results, and the output is the authentication result.

[0573] Step 6:

[0574] If the authentication is successful, the server sends an instruction to unlock the entrance door to the authorized user via the access management means. The input is the authentication result, and the output is a command to unlock the door. Specifically, the server sends a signal to the electromagnetic locking device.

[0575] Step 7:

[0576] The server stores the visitor's authentication result and the access date and time in a recording medium. The input is the user's ID, authentication result, and date and time information, and the output is the result stored in a database. Specifically, this information is stored using an SQLite or MySQL database.

[0577] Example prompt sentence:

[0578] "Upload a photo: visitor_image.jpg"

[0579] "Please upload audio for speech recognition: visitor_audio.wav"

[0580] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0581] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. Furthermore, by incorporating an emotion engine, it has the ability to recognize and respond to the user's emotional state.

[0582] First, the system has the following functions: The server has the function of receiving a user's facial image. This image is uploaded to the server from the terminal. The server analyzes the facial image using a facial recognition algorithm and extracts its features. The obtained features are compared with an existing database to identify the user's ID. This allows the server to recognize the user's face.

[0583] Next, the user provides their voice to the server, which is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. Based on these voice characteristics, the server compares them with an existing database to identify the user's ID, making it possible to recognize the user through their voice.

[0584] Once the user is identified, the server uses the storage means to store the user's personal information. This information includes the user's name, age, hobbies, preferences, etc. The server can also store the user's schedule information. When the user's information is updated, the server immediately updates the storage means.

[0585] Furthermore, the system incorporates an emotion engine that analyzes the user's facial expressions and vocal tone to recognize their current emotional state. For example, if the user shows signs of discomfort, the server can select the appropriate assistance method based on that information.

[0586] As a concrete example, consider the case where a user is feeling stressed. The user communicates their current feelings to the server by voice. For example, if the user says, "I'm feeling a little irritated today," the server performs voice recognition and identifies the user's ID. Next, the emotion engine analyzes the tone and content of the voice and determines that the user is feeling stressed. In response, the server suggests relaxing music and provides relaxation advice. It is also possible to record the user's stress level so that it can be reviewed later.

[0587] Furthermore, if a user has an important presentation coming up, they input their schedule into the server, which not only remembers the information and reminds them before the presentation, but also uses an emotion engine to detect tension before the presentation and send them relaxation advice or encouraging messages.

[0588] In this way, by combining facial recognition, voice recognition, memory means, and an emotion engine, this system provides customized assistance to users, making it extremely useful for busy businessmen, elderly people suffering from memory decline, and people who feel lonely.

[0589] The processing flow will be explained below.

[0590] Step 1:

[0591] Terminal: The user captures a facial image with a camera and uploads it to the server.

[0592] Step 2:

[0593] Server: Calls the recognize_face method to receive the uploaded face image. It analyzes the face image using a facial recognition algorithm and extracts its features. The extracted features are compared with an existing database to identify the user's ID.

[0594] Step 3:

[0595] Device: The user uses a microphone to record voice and sends the voice samples to the server.

[0596] Step 4:

[0597] Server: Calls the recognize_voice method to receive a voice sample. Analyzes the voice using a speech recognition algorithm and extracts voice features. Matches the extracted features with an existing database to identify the user's ID.

[0598] Step 5:

[0599] Terminal: The user enters their information (e.g., name, age, etc.) and sends that information to the server.

[0600] Step 6:

[0601] Server: Call the update_memory method to save or update the user's personal information, including the user's name, age, hobbies, and preferences.

[0602] Step 7:

[0603] Server: Updates the user's schedule information. When a user adds a new event (e.g., a meeting), the server records this information in memory.

[0604] Step 8:

[0605] Terminal: The user sends their current feelings and state to the server via voice or text.

[0606] Step 9:

[0607] Server: Analyzes the received voice and text and recognizes the user's emotional state using an emotion engine, for example by analyzing voice tone and expressed emotions.

[0608] Step 10:

[0609] Server: Provides appropriate support based on the user's recognized emotional state. For example, if the user is feeling stressed, it can provide relaxing music or relaxation advice.

[0610] Step 11:

[0611] Server: Uses an emotion engine to monitor the user's emotional state before a specific event (e.g., before a presentation) and sends encouraging messages or advice as needed.

[0612] Step 12:

[0613] Server: Records past support history and emotional state, and supports users in self-analysis and reflection in the future.

[0614] Through these steps, the server can combine face recognition, voice recognition, memory means, and an emotion engine to provide customized assistance to users, making it an extremely useful service for busy businessmen, elderly people suffering from memory loss, or people who feel lonely.

[0615] Example 2

[0616] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0617] Conventional personal assistant systems have basic functions such as recognizing the user's face and voice and remembering personal information, but they are unable to recognize the user's emotional state and respond based on that, making it difficult to provide truly beneficial assistance to the user. Furthermore, they lack advanced support such as managing the user's schedule and suggesting ways to reduce stress based on the user's emotional state.

[0618] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing a face image and transmitting the face image, a means for analyzing the face image using a face recognition algorithm and extracting features, and a means for matching the features with an existing database and identifying the user's ID. This enables face recognition of the user.

[0619] The server further includes means for recording and transmitting the voice, means for analyzing the voice using a voice recognition algorithm to extract characteristics, and means for matching the voice characteristics with an existing database to identify the user's identity, thereby enabling voice recognition of the user.

[0620] The server also includes a means for storing personal information of the user, a means for providing appropriate support based on the stored information, a means for recognizing the emotional state of the user by analyzing facial expressions and voice tones, and a means for providing appropriate responses based on the emotional state, thereby making it possible to provide support suited to the emotional state of the user.

[0621] Finally, the server includes a means for storing the user's schedule and providing timely reminders, and a means for providing stress reduction suggestions based on the user's emotional state, so that the user can avoid forgetting important plans and receive appropriate support when feeling stressed.

[0622] A "face image" is still image data of a photograph of the user's face.

[0623] A "server" is a computer system that operates on a network and receives, processes, stores, and transmits data.

[0624] A "facial recognition algorithm" is a mathematical model or computational method for extracting specific features from a facial image and identifying an individual based on those features.

[0625] Extracting "features" is a process of obtaining specific patterns or attributes as numerical data from facial images or voice data.

[0626] A "database" is a data structure that systematically stores and makes searchable personal information, facial feature data, voice feature data, and other information about users.

[0627] "ID identification" is the process of matching characteristic data with a database to identify matching personal identifiers.

[0628] "Voice" is waveform data of the voice uttered by the user.

[0629] A "speech recognition algorithm" is a mathematical model or computational method that analyzes linguistic information and specific features from speech data and converts them into text data or identification information.

[0630] "Personal information" refers to data relating to an individual, such as a user's name, age, hobbies, preferences, schedule information, etc.

[0631] "Storage" is the process of saving information in a storage medium such as a database.

[0632] An "emotion engine" is a computational model or system that analyzes facial expressions and vocal tone to recognize a user's emotional state.

[0633] "Appropriate support" is the process of providing advice and services that are tailored to the user's needs and condition.

[0634] "Schedule" is information that describes the user's plans and plans.

[0635] "Remind" is the process of notifying users of specific events or information in advance.

[0636] "Stress reduction suggestions" is the process of recommending advice or activities aimed at reducing stress when a user is feeling stressed.

[0637] MODE FOR CARRYING OUT THE INVENTION

[0638] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. By incorporating an emotion engine, this system is also capable of recognizing and responding to the user's emotional state.

[0639] Hardware and software used

[0640] Hardware: Smartphones, PCs, servers

[0641] software:

[0642] Face Recognition Algorithm: OpenCV, FaceNet

[0643] Speech recognition algorithms: Google Speech-to-Text, IBM Watson

[0644] Emotion engine: Microsoft Azure Emotion API, Affectiva

[0645] Facial Recognition Processing

[0646] 1. Image capture

[0647] The user takes a picture of their face using a device (such as a smartphone or PC camera).

[0648] The device temporarily stores the captured image.

[0649] 2. Upload an image

[0650] The device uploads the stored facial image to the server using an HTTP request.

[0651] 3. Image Analysis

[0652] The server analyzes the received image using a facial recognition algorithm (e.g., OpenCV's face detection library or FaceNet model) and extracts facial feature points.

[0653] 4. ID identification

[0654] The server compares the extracted feature data with an existing database to identify the user's ID.

[0655] Speech Recognition Processing

[0656] 1. Voice input

[0657] The user records the voice using the microphone of the device.

[0658] The device temporarily stores the recorded audio.

[0659] 2. Upload audio

[0660] The device uploads the saved audio data to the server using an HTTP request.

[0661] 3. Audio Analysis

[0662] The server analyzes the received audio using a speech recognition algorithm (e.g., Google Speech-to-Text API or IBM Watson Speech to Text) and extracts features from the audio data.

[0663] 4. ID identification

[0664] The server matches the voice characteristics with information in a user database to determine the user's ID.

[0665] Information storage processing

[0666] 1. Information Record

[0667] The server stores the identified user's personal information (such as name, age, hobbies, preferences, schedule information, etc.) in a database.

[0668] 2. Information update

[0669] The server immediately updates the database when user information changes.

[0670] 3. Information provision

[0671] The server uses the recorded information as needed to provide appropriate assistance to the user.

[0672] Emotion Recognition Processing

[0673] 1. Sentiment analysis

[0674] The server uses an emotion engine (Microsoft Azure Emotion API or Affectiva) to analyze facial expressions and vocal tone.

[0675] 2. Providing support

[0676] The server provides an appropriate response based on the results of the sentiment analysis.

[0677] Specific examples

[0678] Dealing with stressed users

[0679] The user uses the device to voice input, "I'm feeling a bit frustrated today."

[0680] The terminal uploads the audio data to the server.

[0681] The server performs voice recognition to identify the user's identity and analyzes the voice tone with an emotion engine.

[0682] The server recognizes the user's emotional state as "stress" and suggests ways to relax.

[0683] What to do if you have an important presentation coming up

[0684] The user uses the terminal to enter schedule information (date and time of presentation).

[0685] The terminal uploads the schedule information to the server.

[0686] The server stores the information and sends a reminder the day before the presentation.

[0687] On the day of the presentation, the emotion engine is used to recognize the user's state of tension and send relaxation advice and encouraging messages.

[0688] Prompt Sentence Examples

[0689] User Emotion Recognition

[0690] If a user verbally tells a server, "I'm feeling a bit frustrated today," explain how the server will recognize the user and what action to take.

[0691] User schedule management

[0692] If a user enters the schedule for an important presentation, explain how the system will support the user and what assistance it will provide.

[0693] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0694] Facial Recognition Processing

[0695] Step 1: Image capture

[0696] The user takes a picture of his or her face using the camera on the terminal.

[0697] Input: A still image of a face.

[0698] Output: Facial image files temporarily saved on your device.

[0699] Specific operation: The device's camera app is launched, and the user faces the camera and takes a picture of their face. The device saves the captured image to its internal storage.

[0700] Step 2: Upload an image

[0701] The device uploads the stored facial image to the server.

[0702] Input: A face image file stored on the device.

[0703] Output: Facial image data sent to the server.

[0704] Specific operation: Using an HTTP request, the device sends facial image data to the server. The server receives the data and stores it in storage.

[0705] Step 3: Image analysis

[0706] The server analyzes the received facial images using a facial recognition algorithm (e.g., OpenCV or FaceNet).

[0707] Input: Facial image data stored on the server.

[0708] Output: Extracted facial feature data.

[0709] Specific operation: Runs a facial recognition algorithm to extract feature points such as eyes, nose, and mouth from the image as numerical data.

[0710] Step 4: Identify ID

[0711] The server compares the extracted feature data with an existing database to identify the user's ID.

[0712] Input: Extracted facial feature data.

[0713] Output: The matched user's ID.

[0714] Specific behavior: Compares the user data and feature data in the database to identify matching users, and stores the identified user IDs for further processing.

[0715] Speech Recognition Processing

[0716] Step 1: Voice Input

[0717] The user records their own voice using the device's microphone.

[0718] Input: Audio as voice.

[0719] Output: An audio file temporarily saved on your device.

[0720] Specific operation: Launch the device's recording app to record the user's voice. After recording, the device will save the audio file to its internal storage.

[0721] Step 2: Upload your audio

[0722] The device uploads the saved audio data to the server.

[0723] Input: An audio file stored on your device.

[0724] Output: The audio data sent to the server.

[0725] Specific operation: Using an HTTP request, the device sends audio data to the server, which receives it and stores it in storage.

[0726] Step 3: Audio analysis

[0727] The server analyzes the received voice data using a speech recognition algorithm (e.g., Google Speech-to-Text API or IBM Watson Speech to Text).

[0728] Input: Audio data stored on the server.

[0729] Output: Extracted audio feature data.

[0730] What it does: Runs a speech recognition algorithm to extract linguistic information and specific features from speech as numerical data.

[0731] Step 4: Identify ID

[0732] The server matches the voice characteristics data with an existing database to determine the user's ID.

[0733] Input: Extracted audio feature data.

[0734] Output: The matched user's ID.

[0735] Specific behavior: Compares user data in the database with the voice feature data to identify matching users, and stores the identified user's ID for further processing.

[0736] Information storage processing

[0737] Step 1: Record information

[0738] The server stores the identified user's personal information in a database.

[0739] Input: User's personal information (name, age, hobbies, preferences, schedule, etc.).

[0740] Output: User information recorded in the database.

[0741] What it does: Adds new information about the user to the database and updates existing information as needed.

[0742] Step 2: Update your information

[0743] The server immediately updates the database when user information changes.

[0744] Input: New user information.

[0745] Output: The updated database.

[0746] Specific operation: When new information is received, it is compared with the existing information, and if it differs, an update process is performed.

[0747] Step 3: Provide information

[0748] The server uses the recorded information as needed to provide appropriate assistance to the user.

[0749] Input: User information recorded in the database.

[0750] Output: The assistance provided to the user.

[0751] What it does: Set reminders based on the user's schedule information and suggest activities based on their preferences.

[0752] Emotion Recognition Processing

[0753] Step 1: Sentiment analysis

[0754] The server uses an emotion engine (Microsoft Azure Emotion API or Affectiva) to analyze facial expressions and vocal tone.

[0755] Input: Facial image data, audio data.

[0756] Output: User's emotional state data.

[0757] Specific operation: Executes the emotion engine, extracts feature points indicating emotions from facial images and tone changes from audio, and determines the current emotional state.

[0758] Step 2: Provide support

[0759] The server provides an appropriate response based on the results of the sentiment analysis.

[0760] Input: Emotional state data.

[0761] Output: Emotion-based support content.

[0762] Specific behavior: If the emotional state is determined to be "stressed," the system will suggest relaxing activities and provide advice to reduce stress.

[0763] (Application example 2)

[0764] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0765] Elderly people need daily support in their daily lives, but conventional personal assistant systems lack the ability to properly recognize the emotional state of the elderly and provide individually customized support. There is also a need for systems that can provide relaxation advice and encouraging messages tailored to the elderly's emotional state. The purpose of this invention is to solve these problems.

[0766] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing personal information of the user in storage means, means for recognizing the user's face by face recognition means, means for recognizing the user's voice by voice recognition means, means for analyzing the user's emotional state by an emotion engine, and means for providing support customized according to the user's emotional state. This makes it possible to provide individually customized support based on the user's emotional state and support the elderly in their daily lives.

[0767] "Storage means" refers to a technical element for storing personal information of a user.

[0768] "Facial recognition means" refers to a technical element that recognizes the face of a user, and is a means of analyzing facial images to identify individuals.

[0769] "Voice recognition means" refers to a technical element that recognizes the user's voice, extracts voice characteristics, and performs personal identification and analysis of voice instructions.

[0770] The "means for providing appropriate support" refers to a technical element that provides optimal support to a user based on the storage means.

[0771] The "emotion engine" is a technology element that analyzes the user's facial expressions and tone of voice to recognize their emotional state.

[0772] The "means for providing support customized according to the user's emotional state" is a technical element that provides optimal support individually based on the user's emotional state recognized by the emotion engine.

[0773] The "schedule management means" is a technical element that manages the user's schedule and important events based on the storage means, and sends reminders and notifications at appropriate times.

[0774] "Means for providing reminders and notifications" refers to technical elements that provide necessary reminders and notifications according to the user's emotional state and schedule.

[0775] The "means for providing relaxation advice and encouraging messages" refers to a technical element that determines the emotional state of the user and provides appropriate relaxation methods and encouraging messages.

[0776] The system of the present invention includes a memory means, a face recognition means, a voice recognition means, an emotion engine, and a means for providing customized assistance according to the user's emotional state in order to support the elderly in their daily lives. The system is configured and implemented as follows.

[0777] First, the storage means stores the user's personal information, including name, age, hobbies, preferences, schedule, etc. The device (smartphone or head-mounted display) periodically transmits this information to the server, which stores it in a database.

[0778] Next, the facial recognition means uses the device's built-in camera to recognize the user's face, and the recognized facial features are sent to the server, where a facial recognition algorithm (e.g., OpenCV) analyzes them and matches them with a database to identify the user.

[0779] The speech recognition method uses the device's microphone to transmit the user's voice input to the server, where a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API) analyzes it and identifies the user's voice commands and emotions.

[0780] The emotion engine is a technology component that analyzes the user's facial expressions and vocal tone to recognize their current emotional state. The emotion engine can use, for example, the Microsoft Azure Emotion API. The results of this analysis are sent to a server to help guide the next steps.

[0781] The server uses this information to provide customized support according to the user's emotional state. For example, if it determines that an elderly person is feeling stressed or anxious, the server can send relaxation advice or encouraging messages to the device. Examples of such messages include videos of breathing exercises to help the elderly person relax or the playing of relaxing music.

[0782] Furthermore, the server uses the storage means to manage the user's schedule and provide reminders and notifications as needed, such as timely reminders for important medical appointments or medication times, and can provide relaxation advice if the emotion engine determines that the user's emotional state is tense.

[0783] Examples:

[0784] If an elderly person says, "I'm feeling a little anxious today,"

[0785] The server performs voice recognition to identify the user's ID, and the emotion engine analyzes the voice tone and content to identify the user's anxiety. Based on this information, the server sends a "relaxation breathing exercise video" to the device. In this way, the system can provide assistance tailored to the user's emotional state and improve the quality of life for the elderly.

[0786] Example prompt sentence:

[0787] If a senior is feeling anxious, "offer these next steps: Play a breathing exercise video followed by 10 minutes of relaxing music."

[0788] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0789] Step 1:

[0790] The user shows their face to the device (smartphone or head-mounted display). The device's camera captures a matching facial image and sends it to the server. The server uses a facial recognition algorithm to match the received facial image with existing facial data in a database and identify the user ID. This allows the server to recognize the user.

[0791] Input: User's face image

[0792] Data processing / data calculation: Analysis of facial images using face recognition algorithms and matching with databases

[0793] Output: User ID

[0794] Step 2:

[0795] The user enters voice commands into the device. The device's microphone captures the voice and sends it to the server. The server uses a speech recognition algorithm to convert the voice into text data and identify the user's instructions and emotional state.

[0796] Input: User's voice

[0797] Data processing / data calculation: Analysis of voice data using speech recognition algorithms and conversion to text

[0798] Output: Text data (user instructions, emotional state)

[0799] Step 3:

[0800] An emotion engine analyzes the text data and voice tone to determine the user's current emotional state. The server uses the emotion engine to identify the user's emotional state (e.g., stress, anxiety, joy, etc.).

[0801] Input: Text data and audio tones

[0802] Data processing / data calculation: Emotional state analysis using emotion engine

[0803] Output: User's emotional state

[0804] Step 4:

[0805] The server checks the user's personal information and schedule based on the storage means, thereby understanding the user's hobbies, preferences, plans, etc., and determines appropriate support content based on the user's emotional state.

[0806] Input: User's emotional state, personal information, schedule

[0807] Data processing / data calculation: Referencing and analyzing information using memory means

[0808] Output: Support content

[0809] Step 5:

[0810] The server provides customized support to the user based on the corresponding support content. For example, if the user feels anxious, the server will send a relaxing breathing exercise video to the device and instruct it to play. The server will also record the support content and the user's emotional state in a database for future reference.

[0811] Input: Support content

[0812] Data processing / data calculation: Deciding on support content and instructing distribution

[0813] Output: Support provided to the user (e.g., relaxation video)

[0814] Step 6:

[0815] The server monitors the user's emotional state and the effectiveness of the assistance provided, offering additional support as needed, or if it is ineffective, offering alternative advice or messages.

[0816] Input: User state after assistance

[0817] Data processing / data calculation: monitoring and analysis data feedback

[0818] Output: Additional support or ongoing monitoring

[0819] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0820] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0821] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0822] [Third embodiment]

[0823] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0824] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0825] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0826] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0827] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0828] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0829] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0830] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0831] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0832] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0833] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0834] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0835] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. This system has the following functions.

[0836] First, the server has the function of receiving a user's facial image. This image is uploaded to the server from the device. The server analyzes the facial image using a facial recognition algorithm and extracts its features. The obtained features are compared with an existing database to identify the user's ID. This allows the server to recognize the user's face.

[0837] Next, the user provides their voice to the server, which is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. Based on these voice characteristics, the server compares them with an existing database to identify the user's ID, making it possible to recognize the user through their voice.

[0838] Once the user is identified, the server uses the storage means to store the user's personal information. This information includes the user's name, age, hobbies, preferences, etc. The server can also store the user's schedule information. When the user's information is updated, the server immediately updates the storage means.

[0839] As a concrete example, consider the case where a user wants to add a new appointment. The user communicates the appointment to the server by voice. For example, the user might say, "Add a meeting for next Tuesday." The server first performs voice recognition to identify the user's ID. Next, it uses the storage means to retrieve the user's schedule and adds the new appointment. The user is reminded of this appointment at the appropriate time. This allows the user to manage their schedule efficiently.

[0840] Furthermore, if the user wants to remember something, the server can refer to the storage means and provide the user with past information. For example, if the user instructs the server to "tell me a list of past birthdays," the server will provide the user with stored birthday information. In this way, the server provides appropriate assistance to support the user's daily life.

[0841] This system combines facial and voice recognition to effectively manage users' personal information and provide appropriate assistance, making it extremely useful for busy businessmen, elderly people suffering from memory loss, and people who feel lonely.

[0842] The processing flow will be explained below.

[0843] Step 1:

[0844] Server: The system is initialized and an instance of the PersonalAssistantAI class is created. At this time, user_data is passed as an argument and the internal memory is initialized.

[0845] Step 2:

[0846] Terminal: The user captures a facial image and uploads it to the server, which provides the facial image to the system.

[0847] Step 3:

[0848] Server: The recognize_face method is called, receiving the uploaded face image as an argument. The server analyzes the face image using a facial recognition algorithm and extracts facial features. The extracted features are then compared with an existing database to identify the user's ID.

[0849] Step 4:

[0850] User: Records audio and sends it to the server. This audio includes the user's statements and commands.

[0851] Step 5:

[0852] Server: The recognize_voice method is called to accept a voice sample. The speech is analyzed using a speech recognition algorithm to extract voice features. These features are then matched with an existing database to identify the user's ID.

[0853] Step 6:

[0854] Device: Sends new information (e.g., name, age, etc.) to the server. Sends data containing the user's updated information.

[0855] Step 7:

[0856] Server: Calls the update_memory method, receiving the user ID and new information as arguments. This information is saved or updated in internal memory.

[0857] Step 8:

[0858] Server: Provides appropriate support based on the user's individual information. For example, it manages the user's schedule and stores new events in a storage device.

[0859] Step 9:

[0860] Terminal: When a user wants to obtain specific information, they send a request to the server by voice or text.

[0861] Step 10:

[0862] Server: Calls the get_memory method to retrieve the requested information from internal memory and provide it to the user. For example, it returns a list of past events or birthdays.

[0863] Step 11:

[0864] Server: It also stores information about the user's hobbies and preferences and provides that information upon request. For example, it can provide information about the user's favorite movies and music.

[0865] These steps allow the server to use facial and voice recognition to identify users, and then provide individual memories, schedule management, and appropriate information.

[0866] Example 1

[0867] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0868] In today's world, users must manage a large amount of information in their daily lives and work, placing a significant burden on them. Furthermore, when using various devices or services, users must authenticate each time, resulting in inconvenient usage. Furthermore, declining memory and information overload can make management cumbersome. There is a need for technology that can resolve these issues, efficiently and intuitively manage users' personal information, and provide appropriate support.

[0869] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0870] In this invention, the server includes means for receiving a user's facial image, analyzing the facial image using a facial recognition algorithm, and extracting features, means for comparing the features of the received facial image with an existing database to identify the user's ID, means for receiving the user's voice, analyzing the voice using a voice recognition algorithm, and extracting features of the voice, means for comparing the features of the received voice with an existing database to identify the user's ID, means for storing the user's personal information using a storage means when the user is identified and updating the information as necessary, and means for providing appropriate support based on the stored user information. This makes it possible to efficiently manage the user's personal information and provide appropriate support by intuitively authenticating the user using their face or voice.

[0871] "User's facial image" refers to a still image or video frame of the user's face.

[0872] "Facial recognition algorithm" is a general term for computational methods and programs that extract feature points from facial images and identify individual people.

[0873] "Extracting features" refers to the process of extracting important parameters, shapes, and patterns necessary for identification from facial images and voice.

[0874] "Existing database" refers to a collection of information that has been stored up to now, including personal information about users (such as facial features and voice features).

[0875] "Identifying the user's ID" means uniquely identifying the user by matching the extracted features with an existing database.

[0876] "Speech recognition algorithm" is a general term for computational methods and programs used to analyze speech signals and recognize language and spoken content.

[0877] "Storage means" refers to a storage device or data storage for storing a user's personal information.

[0878] "User's personal information" refers to individual data and information about the user, such as name, age, hobbies, preferences, and schedule.

[0879] "Providing appropriate assistance" means providing necessary information and performing tasks according to the user's requests and circumstances.

[0880] This invention is a personal assistant system that recognizes a user's face and voice, memorizes personal information, and provides appropriate assistance. This system is implemented using the following hardware and software.

[0881] Hardware and software used

[0882] Hardware: Servers, devices (smartphones, tablets, PCs, etc.), microphones, cameras

[0883] Software: Facial recognition algorithms, voice recognition algorithms, database management systems, storage means (memory)

[0884] First, the server has a means to receive the user's facial image. The user captures the facial image using a camera and uploads it to the server from their device. Next, the server uses a facial recognition algorithm to analyze the received facial image and extract its features. The facial recognition algorithm identifies the contours of the face and the relative positions of each feature, and extracts feature points. These feature points are then compared with facial data in an existing database to identify the user's ID.

[0885] Next, the user provides their voice to the server via a microphone. The server is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. The speech recognition algorithm recognizes language-specific patterns and individual voice characteristics from the sound waveforms and extracts features. These voice characteristics are then compared with an existing database to identify the user's ID.

[0886] Once the user's ID is identified, the server uses a storage device to store the user's personal information, including the user's name, age, hobbies, preferences, and schedule information. The server updates this storage device whenever it receives new information from the user.

[0887] For example, if a user wants to add a new appointment, they can send a voice command to the server. The user commands, "Add a meeting for next Tuesday." The server performs voice recognition to identify the user's ID, then retrieves the user's schedule using the storage means and adds the new appointment. The user is reminded of this new appointment at the appropriate time.

[0888] If the user wants to recall past information, the server can provide that information by referring to the storage means. For example, if the user gives a voice command such as "Tell me a list of past birthdays," the server will provide the user with the birthday information it has stored.

[0889] Prompt Sentence Examples

[0890] "Please describe a scenario where a user wants to add a new appointment."

[0891] "Please specify how the system responds when a user retrieves a list of past birthdays."

[0892] By combining facial and voice recognition, this system effectively manages users' personal information and supports their daily lives. It is particularly useful for busy business people and elderly people who suffer from memory loss. This technology frees users from the hassle of managing their information, enabling them to live more efficient lives.

[0893] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0894] Step 1:

[0895] Input: The user activates the device's camera and captures a facial image.

[0896] Specific operation: A facial image is captured on the device.

[0897] Output: The device uploads the captured face image to the server.

[0898] Step 2:

[0899] Input: The server receives the face image sent from the device.

[0900] Specific operation: The server runs a facial recognition algorithm and extracts feature points from the facial image.

[0901] Output: The extracted features are obtained.

[0902] Step 3:

[0903] Input: The extracted features are matched against face data from an existing database.

[0904] What happens: The server uses a database management system to perform the matching process.

[0905] Output: The user's ID is identified.

[0906] Step 4:

[0907] Input: The user uses a microphone to input voice into the server.

[0908] What happens: Audio is captured by the device and sent to the server.

[0909] Output: Audio data arrives at the server.

[0910] Step 5:

[0911] Input: The server processes the received audio data.

[0912] What happens: The server runs a speech recognition algorithm to analyze and extract speech features.

[0913] Output: Extracted audio features are obtained.

[0914] Step 6:

[0915] Input: The extracted audio features are matched against existing audio data in a database.

[0916] What happens: The server uses a database management system to perform the matching process.

[0917] Output: The user's ID is identified.

[0918] Step 7:

[0919] Input: After the user's ID is identified, the server retrieves the user's personal information from the storage means.

[0920] Specific operations: The server reads the relevant data from the storage device.

[0921] Output: The user's personal information (name, age, hobbies, preferences, schedule information, etc.) is obtained.

[0922] Step 8:

[0923] Input: If a user wants to add a new event, they say, "Add a meeting for next Tuesday."

[0924] Specific operation: The server performs voice recognition, confirms the user's ID, retrieves existing schedule information, and adds new information.

[0925] Output: A new event is added to the schedule and a reminder is sent at the appropriate time.

[0926] Step 9:

[0927] Input: If the user wants to see past information, they can say, "Give me a list of past birthdays."

[0928] Specific operation: The server performs voice recognition, confirms the user's ID, and then retrieves the birthday information from the storage means.

[0929] Output: The birthdate information is provided to the user.

[0930] This system allows users to intuitively manage their personal information and receive appropriate support using only their face and voice.

[0931] (Application example 1)

[0932] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0933] In modern society, there is a demand for stronger security in office buildings and apartment complexes. In particular, the difficulty of quickly and accurately authenticating visitors and managing their access using conventional methods places an increased burden on administrators. Furthermore, there is a risk of incomplete records and misidentification, making it an urgent need to improve security levels.

[0934] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0935] In this invention, the server includes a means for storing personal information of users in a storage means, a means for recognizing the user's face using a facial recognition means, and a means for recognizing the user's voice using a voice recognition means. This enables a means for managing access based on facial recognition, a means for managing access based on voice recognition, a means for storing visitor records, and a means for providing appropriate support based on the storage means. This improves the security level and reduces the burden on administrators.

[0936] "Storage means" refers to a device or system that stores information about a user's personal information, schedule information, hobbies and preferences.

[0937] A "face recognition means" is a device or algorithm that analyzes a user's facial image, extracts its features, and identifies the individual.

[0938] "Voice recognition means" refers to a device or algorithm that analyzes a user's voice, extracts its characteristics, and identifies the individual.

[0939] The "means for providing appropriate support" is a device or system that provides the user with necessary information and services based on the storage means.

[0940] A "means for managing access based on facial recognition" is a device or system that uses facial recognition means to identify individuals and control access rights.

[0941] A "means for controlling access based on voice recognition" is a device or system that uses voice recognition means to identify individuals and control access rights.

[0942] "Means for storing visitor records" refers to a device or system that stores visitor access history and authentication results in a database or the like.

[0943] The system required to implement this invention is mainly composed of a server equipped with a storage means, a face recognition means, and a voice recognition means. The specific system configuration and its operation will be described below.

[0944] 1. Storage means

[0945] The server is equipped with a storage means for saving personal information of users (e.g., name, age, hobbies, preferences, schedule information). This storage means uses a database management system (DBMS). For example, a database such as SQLite or MySQL is used.

[0946] 2. Facial Recognition Methods

[0947] The server receives the facial image sent from the device and identifies the individual using a facial recognition algorithm, such as the Python face_recognition library. The server then compares the features of the recognized facial image with a database to identify the user's ID.

[0948] 3. Voice Recognition Method

[0949] The server receives the voice data sent from the device and identifies the individual using a speech recognition algorithm. This algorithm uses the Python speech_recognition library and the Google Speech Recognition API. The server compares the recognized voice characteristics with a database to determine the user's ID.

[0950] 4. Access Management

[0951] The server manages access rights for identified users based on facial and voice recognition. For example, it processes video and audio transmitted from cameras and microphones installed at the entrances of office buildings and apartment complexes to authenticate individuals. Authenticated users can then be given permission to unlock automatic doors and use elevators.

[0952] 5. Record-Keeping

[0953] The server stores the results of access control and visitor records in a database, including information such as the visitor's ID, authentication date and time, and authentication location.

[0954] As a concrete example, the following scenario can be imagined:

[0955] Use case: At the entrance of an office building, administrators use facial and voice recognition to recognize visitors and automatically manage access permissions.

[0956] input:

[0957] Image file (e.g. visitor_image.jpg)

[0958] Audio file (e.g. visitor_audio.wav)

[0959] An example prompt is:

[0960] "Upload a photo: visitor_image.jpg"

[0961] "Please upload audio for speech recognition: visitor_audio.wav"

[0962] This system allows administrators to quickly and accurately authenticate visitors, streamlining access management, and storing records in a database for future review and tracking of unauthorized access.

[0963] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0964] Step 1:

[0965] A user uses a terminal to provide a facial image to a camera at the entrance. The terminal captures this image and sends it to a server. The input is a facial image, and the output is image data.

[0966] Step 2:

[0967] The server passes the received facial image to a facial recognition algorithm to extract facial features. The extracted features are then matched against an existing database. The input is facial image data, and the output is the match result and the user's ID. Specifically, the Python face_recognition library is used to obtain the facial encoding and compare it with known encodings in the database.

[0968] Step 3:

[0969] The user uses a terminal to provide voice data to the microphone at the entrance. The terminal records this voice and sends it to the server. The input is voice data, and the output is an audio file.

[0970] Step 4:

[0971] The server passes the received audio data to a speech recognition algorithm to extract audio features. The extracted features are compared with an existing database. The input is the audio data, and the output is the speech recognition results and the user's ID. Specifically, the Python speech_recognition library and the Google Speech Recognition API are used to convert the audio into text and compare it with known audio data in the database.

[0972] Step 5:

[0973] The server determines the user's identity based on the results of facial and voice recognition. If both results match, the user is considered to have been successfully authenticated. The input is the facial and voice recognition results, and the output is the authentication result.

[0974] Step 6:

[0975] If the authentication is successful, the server sends an instruction to unlock the entrance door to the authorized user via the access management means. The input is the authentication result, and the output is a command to unlock the door. Specifically, the server sends a signal to the electromagnetic locking device.

[0976] Step 7:

[0977] The server stores the visitor's authentication result and the access date and time in a recording medium. The input is the user's ID, authentication result, and date and time information, and the output is the result stored in a database. Specifically, this information is stored using an SQLite or MySQL database.

[0978] Example prompt sentence:

[0979] "Upload a photo: visitor_image.jpg"

[0980] "Please upload audio for speech recognition: visitor_audio.wav"

[0981] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0982] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. Furthermore, by incorporating an emotion engine, it has the ability to recognize and respond to the user's emotional state.

[0983] First, the system has the following functions: The server has the function of receiving a user's facial image. This image is uploaded to the server from the terminal. The server analyzes the facial image using a facial recognition algorithm and extracts its features. The obtained features are compared with an existing database to identify the user's ID. This allows the server to recognize the user's face.

[0984] Next, the user provides their voice to the server, which is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. Based on these voice characteristics, the server compares them with an existing database to identify the user's ID, making it possible to recognize the user through their voice.

[0985] Once the user is identified, the server uses the storage means to store the user's personal information. This information includes the user's name, age, hobbies, preferences, etc. The server can also store the user's schedule information. When the user's information is updated, the server immediately updates the storage means.

[0986] Furthermore, the system incorporates an emotion engine that analyzes the user's facial expressions and vocal tone to recognize their current emotional state. For example, if the user shows signs of discomfort, the server can select the appropriate assistance method based on that information.

[0987] As a concrete example, consider the case where a user is feeling stressed. The user communicates their current feelings to the server by voice. For example, if the user says, "I'm feeling a little irritated today," the server performs voice recognition and identifies the user's ID. Next, the emotion engine analyzes the tone and content of the voice and determines that the user is feeling stressed. In response, the server suggests relaxing music and provides relaxation advice. It is also possible to record the user's stress level so that it can be reviewed later.

[0988] Furthermore, if a user has an important presentation coming up, they input their schedule into the server, which not only remembers the information and reminds them before the presentation, but also uses an emotion engine to detect tension before the presentation and send them relaxation advice or encouraging messages.

[0989] In this way, by combining facial recognition, voice recognition, memory means, and an emotion engine, this system provides customized assistance to users, making it extremely useful for busy businessmen, elderly people suffering from memory decline, and people who feel lonely.

[0990] The processing flow will be explained below.

[0991] Step 1:

[0992] Terminal: The user captures a facial image with a camera and uploads it to the server.

[0993] Step 2:

[0994] Server: Calls the recognize_face method to receive the uploaded face image. It analyzes the face image using a facial recognition algorithm and extracts its features. The extracted features are compared with an existing database to identify the user's ID.

[0995] Step 3:

[0996] Device: The user uses a microphone to record voice and sends the voice samples to the server.

[0997] Step 4:

[0998] Server: Calls the recognize_voice method to receive a voice sample. Analyzes the voice using a speech recognition algorithm and extracts voice features. Matches the extracted features with an existing database to identify the user's ID.

[0999] Step 5:

[1000] Terminal: The user enters their information (e.g., name, age, etc.) and sends that information to the server.

[1001] Step 6:

[1002] Server: Call the update_memory method to save or update the user's personal information, including the user's name, age, hobbies, and preferences.

[1003] Step 7:

[1004] Server: Updates the user's schedule information. When a user adds a new event (e.g., a meeting), the server records this information in memory.

[1005] Step 8:

[1006] Terminal: The user sends their current feelings and state to the server via voice or text.

[1007] Step 9:

[1008] Server: Analyzes the received voice and text and recognizes the user's emotional state using an emotion engine, for example by analyzing voice tone and expressed emotions.

[1009] Step 10:

[1010] Server: Provides appropriate support based on the user's recognized emotional state. For example, if the user is feeling stressed, it can provide relaxing music or relaxation advice.

[1011] Step 11:

[1012] Server: Uses an emotion engine to monitor the user's emotional state before a specific event (e.g., before a presentation) and sends encouraging messages or advice as needed.

[1013] Step 12:

[1014] Server: Records past support history and emotional state, and supports users in self-analysis and reflection in the future.

[1015] Through these steps, the server can combine face recognition, voice recognition, memory means, and an emotion engine to provide customized assistance to users, making it an extremely useful service for busy businessmen, elderly people suffering from memory loss, or people who feel lonely.

[1016] Example 2

[1017] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1018] Conventional personal assistant systems have basic functions such as recognizing the user's face and voice and remembering personal information, but they are unable to recognize the user's emotional state and respond based on that, making it difficult to provide truly beneficial assistance to the user. Furthermore, they lack advanced support such as managing the user's schedule and suggesting ways to reduce stress based on the user's emotional state.

[1019] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing a face image and transmitting the face image, a means for analyzing the face image using a face recognition algorithm and extracting features, and a means for matching the features with an existing database and identifying the user's ID. This enables face recognition of the user.

[1020] The server further includes means for recording and transmitting the voice, means for analyzing the voice using a voice recognition algorithm to extract characteristics, and means for matching the voice characteristics with an existing database to identify the user's identity, thereby enabling voice recognition of the user.

[1021] The server also includes a means for storing personal information of the user, a means for providing appropriate support based on the stored information, a means for recognizing the emotional state of the user by analyzing facial expressions and voice tones, and a means for providing appropriate responses based on the emotional state, thereby making it possible to provide support suited to the emotional state of the user.

[1022] Finally, the server includes a means for storing the user's schedule and providing timely reminders, and a means for providing stress reduction suggestions based on the user's emotional state, so that the user can avoid forgetting important plans and receive appropriate support when feeling stressed.

[1023] A "face image" is still image data of a photograph of the user's face.

[1024] A "server" is a computer system that operates on a network and receives, processes, stores, and transmits data.

[1025] A "facial recognition algorithm" is a mathematical model or computational method for extracting specific features from a facial image and identifying an individual based on those features.

[1026] Extracting "features" is a process of obtaining specific patterns or attributes as numerical data from facial images or voice data.

[1027] A "database" is a data structure that systematically stores and makes searchable personal information, facial feature data, voice feature data, and other information about users.

[1028] "ID identification" is the process of matching characteristic data with a database to identify matching personal identifiers.

[1029] "Voice" is waveform data of the voice uttered by the user.

[1030] A "speech recognition algorithm" is a mathematical model or computational method that analyzes linguistic information and specific features from speech data and converts them into text data or identification information.

[1031] "Personal information" refers to data relating to an individual, such as a user's name, age, hobbies, preferences, schedule information, etc.

[1032] "Storage" is the process of saving information in a storage medium such as a database.

[1033] An "emotion engine" is a computational model or system that analyzes facial expressions and vocal tone to recognize a user's emotional state.

[1034] "Appropriate support" is the process of providing advice and services that are tailored to the user's needs and condition.

[1035] "Schedule" is information that describes the user's plans and plans.

[1036] "Remind" is the process of notifying users of specific events or information in advance.

[1037] "Stress reduction suggestions" is the process of recommending advice or activities aimed at reducing stress when a user is feeling stressed.

[1038] MODE FOR CARRYING OUT THE INVENTION

[1039] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. By incorporating an emotion engine, this system is also capable of recognizing and responding to the user's emotional state.

[1040] Hardware and software used

[1041] Hardware: Smartphones, PCs, servers

[1042] software:

[1043] Face Recognition Algorithm: OpenCV, FaceNet

[1044] Speech recognition algorithms: Google Speech-to-Text, IBM Watson

[1045] Emotion engine: Microsoft Azure Emotion API, Affectiva

[1046] Facial Recognition Processing

[1047] 1. Image capture

[1048] The user takes a picture of their face using a device (such as a smartphone or PC camera).

[1049] The device temporarily stores the captured image.

[1050] 2. Upload an image

[1051] The device uploads the stored facial image to the server using an HTTP request.

[1052] 3. Image Analysis

[1053] The server analyzes the received image using a facial recognition algorithm (e.g., OpenCV's face detection library or FaceNet model) and extracts facial feature points.

[1054] 4. ID identification

[1055] The server compares the extracted feature data with an existing database to identify the user's ID.

[1056] Speech Recognition Processing

[1057] 1. Voice input

[1058] The user records the voice using the microphone of the device.

[1059] The device temporarily stores the recorded audio.

[1060] 2. Upload audio

[1061] The device uploads the saved audio data to the server using an HTTP request.

[1062] 3. Audio Analysis

[1063] The server analyzes the received audio using a speech recognition algorithm (e.g., Google Speech-to-Text API or IBM Watson Speech to Text) and extracts features from the audio data.

[1064] 4. ID identification

[1065] The server matches the voice characteristics with information in a user database to determine the user's ID.

[1066] Information storage processing

[1067] 1. Information Record

[1068] The server stores the identified user's personal information (such as name, age, hobbies, preferences, schedule information, etc.) in a database.

[1069] 2. Information update

[1070] The server immediately updates the database when user information changes.

[1071] 3. Information provision

[1072] The server uses the recorded information as needed to provide appropriate assistance to the user.

[1073] Emotion Recognition Processing

[1074] 1. Sentiment analysis

[1075] The server uses an emotion engine (Microsoft Azure Emotion API or Affectiva) to analyze facial expressions and vocal tone.

[1076] 2. Providing support

[1077] The server provides an appropriate response based on the results of the sentiment analysis.

[1078] Specific examples

[1079] Dealing with stressed users

[1080] The user uses the device to voice input, "I'm feeling a bit frustrated today."

[1081] The terminal uploads the audio data to the server.

[1082] The server performs voice recognition to identify the user's identity and analyzes the voice tone with an emotion engine.

[1083] The server recognizes the user's emotional state as "stress" and suggests ways to relax.

[1084] What to do if you have an important presentation coming up

[1085] The user uses the terminal to enter schedule information (date and time of presentation).

[1086] The terminal uploads the schedule information to the server.

[1087] The server stores the information and sends a reminder the day before the presentation.

[1088] On the day of the presentation, the emotion engine is used to recognize the user's state of tension and send relaxation advice and encouraging messages.

[1089] Prompt Sentence Examples

[1090] User Emotion Recognition

[1091] If a user verbally tells a server, "I'm feeling a bit frustrated today," explain how the server will recognize the user and what action to take.

[1092] User schedule management

[1093] If a user enters the schedule for an important presentation, explain how the system will support the user and what assistance it will provide.

[1094] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1095] Facial Recognition Processing

[1096] Step 1: Image capture

[1097] The user takes a picture of his or her face using the camera on the terminal.

[1098] Input: A still image of a face.

[1099] Output: Facial image files temporarily saved on your device.

[1100] Specific operation: The device's camera app is launched, and the user faces the camera and takes a picture of their face. The device saves the captured image to its internal storage.

[1101] Step 2: Upload an image

[1102] The device uploads the stored facial image to the server.

[1103] Input: A face image file stored on the device.

[1104] Output: Facial image data sent to the server.

[1105] Specific operation: Using an HTTP request, the device sends facial image data to the server. The server receives the data and stores it in storage.

[1106] Step 3: Image analysis

[1107] The server analyzes the received facial images using a facial recognition algorithm (e.g., OpenCV or FaceNet).

[1108] Input: Facial image data stored on the server.

[1109] Output: Extracted facial feature data.

[1110] Specific operation: Runs a facial recognition algorithm to extract feature points such as eyes, nose, and mouth from the image as numerical data.

[1111] Step 4: Identify ID

[1112] The server compares the extracted feature data with an existing database to identify the user's ID.

[1113] Input: Extracted facial feature data.

[1114] Output: The matched user's ID.

[1115] Specific behavior: Compares the user data and feature data in the database to identify matching users, and stores the identified user IDs for further processing.

[1116] Speech Recognition Processing

[1117] Step 1: Voice Input

[1118] The user records their own voice using the device's microphone.

[1119] Input: Audio as voice.

[1120] Output: An audio file temporarily saved on your device.

[1121] Specific operation: Launch the device's recording app to record the user's voice. After recording, the device will save the audio file to its internal storage.

[1122] Step 2: Upload your audio

[1123] The device uploads the saved audio data to the server.

[1124] Input: An audio file stored on your device.

[1125] Output: The audio data sent to the server.

[1126] Specific operation: Using an HTTP request, the device sends audio data to the server, which receives it and stores it in storage.

[1127] Step 3: Audio analysis

[1128] The server analyzes the received voice data using a speech recognition algorithm (e.g., Google Speech-to-Text API or IBM Watson Speech to Text).

[1129] Input: Audio data stored on the server.

[1130] Output: Extracted audio feature data.

[1131] What it does: Runs a speech recognition algorithm to extract linguistic information and specific features from speech as numerical data.

[1132] Step 4: Identify ID

[1133] The server matches the voice characteristics data with an existing database to determine the user's ID.

[1134] Input: Extracted audio feature data.

[1135] Output: The matched user's ID.

[1136] Specific behavior: Compares user data in the database with the voice feature data to identify matching users, and stores the identified user's ID for further processing.

[1137] Information storage processing

[1138] Step 1: Record information

[1139] The server stores the identified user's personal information in a database.

[1140] Input: User's personal information (name, age, hobbies, preferences, schedule, etc.).

[1141] Output: User information recorded in the database.

[1142] What it does: Adds new information about the user to the database and updates existing information as needed.

[1143] Step 2: Update your information

[1144] The server immediately updates the database when user information changes.

[1145] Input: New user information.

[1146] Output: The updated database.

[1147] Specific operation: When new information is received, it is compared with the existing information, and if it differs, an update process is performed.

[1148] Step 3: Provide information

[1149] The server uses the recorded information as needed to provide appropriate assistance to the user.

[1150] Input: User information recorded in the database.

[1151] Output: The assistance provided to the user.

[1152] What it does: Set reminders based on the user's schedule information and suggest activities based on their preferences.

[1153] Emotion Recognition Processing

[1154] Step 1: Sentiment analysis

[1155] The server uses an emotion engine (Microsoft Azure Emotion API or Affectiva) to analyze facial expressions and vocal tone.

[1156] Input: Facial image data, audio data.

[1157] Output: User's emotional state data.

[1158] Specific operation: Executes the emotion engine, extracts feature points indicating emotions from facial images and tone changes from audio, and determines the current emotional state.

[1159] Step 2: Provide support

[1160] The server provides an appropriate response based on the results of the sentiment analysis.

[1161] Input: Emotional state data.

[1162] Output: Emotion-based support content.

[1163] Specific behavior: If the emotional state is determined to be "stressed," the system will suggest relaxing activities and provide advice to reduce stress.

[1164] (Application example 2)

[1165] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1166] Elderly people need daily support in their daily lives, but conventional personal assistant systems lack the ability to properly recognize the emotional state of the elderly and provide individually customized support. There is also a need for systems that can provide relaxation advice and encouraging messages tailored to the elderly's emotional state. The purpose of this invention is to solve these problems.

[1167] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing personal information of the user in storage means, means for recognizing the user's face by face recognition means, means for recognizing the user's voice by voice recognition means, means for analyzing the user's emotional state by an emotion engine, and means for providing support customized according to the user's emotional state. This makes it possible to provide individually customized support based on the user's emotional state and support the elderly in their daily lives.

[1168] "Storage means" refers to a technical element for storing personal information of a user.

[1169] "Facial recognition means" refers to a technical element that recognizes the face of a user, and is a means of analyzing facial images to identify individuals.

[1170] "Voice recognition means" refers to a technical element that recognizes the user's voice, extracts voice characteristics, and performs personal identification and analysis of voice instructions.

[1171] The "means for providing appropriate support" refers to a technical element that provides optimal support to a user based on the storage means.

[1172] The "emotion engine" is a technology element that analyzes the user's facial expressions and tone of voice to recognize their emotional state.

[1173] The "means for providing support customized according to the user's emotional state" is a technical element that provides optimal support individually based on the user's emotional state recognized by the emotion engine.

[1174] The "schedule management means" is a technical element that manages the user's schedule and important events based on the storage means, and sends reminders and notifications at appropriate times.

[1175] "Means for providing reminders and notifications" refers to technical elements that provide necessary reminders and notifications according to the user's emotional state and schedule.

[1176] The "means for providing relaxation advice and encouraging messages" refers to a technical element that determines the emotional state of the user and provides appropriate relaxation methods and encouraging messages.

[1177] The system of the present invention includes a memory means, a face recognition means, a voice recognition means, an emotion engine, and a means for providing customized assistance according to the user's emotional state in order to support the elderly in their daily lives. The system is configured and implemented as follows.

[1178] First, the storage means stores the user's personal information, including name, age, hobbies, preferences, schedule, etc. The device (smartphone or head-mounted display) periodically transmits this information to the server, which stores it in a database.

[1179] Next, the facial recognition means uses the device's built-in camera to recognize the user's face, and the recognized facial features are sent to the server, where a facial recognition algorithm (e.g., OpenCV) analyzes them and matches them with a database to identify the user.

[1180] The speech recognition method uses the device's microphone to transmit the user's voice input to the server, where a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API) analyzes it and identifies the user's voice commands and emotions.

[1181] The emotion engine is a technology component that analyzes the user's facial expressions and vocal tone to recognize their current emotional state. The emotion engine can use, for example, the Microsoft Azure Emotion API. The results of this analysis are sent to a server to help guide the next steps.

[1182] The server uses this information to provide customized support according to the user's emotional state. For example, if it determines that an elderly person is feeling stressed or anxious, the server can send relaxation advice or encouraging messages to the device. Examples of such messages include videos of breathing exercises to help the elderly person relax or the playing of relaxing music.

[1183] Furthermore, the server uses the storage means to manage the user's schedule and provide reminders and notifications as needed, such as timely reminders for important medical appointments or medication times, and can provide relaxation advice if the emotion engine determines that the user's emotional state is tense.

[1184] Examples:

[1185] If an elderly person says, "I'm feeling a little anxious today,"

[1186] The server performs voice recognition to identify the user's ID, and the emotion engine analyzes the voice tone and content to identify the user's anxiety. Based on this information, the server sends a "relaxation breathing exercise video" to the device. In this way, the system can provide assistance tailored to the user's emotional state and improve the quality of life for the elderly.

[1187] Example prompt sentence:

[1188] If a senior is feeling anxious, "offer these next steps: Play a breathing exercise video followed by 10 minutes of relaxing music."

[1189] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1190] Step 1:

[1191] The user shows their face to the device (smartphone or head-mounted display). The device's camera captures a matching facial image and sends it to the server. The server uses a facial recognition algorithm to match the received facial image with existing facial data in a database and identify the user ID. This allows the server to recognize the user.

[1192] Input: User's face image

[1193] Data processing / data calculation: Analysis of facial images using face recognition algorithms and matching with databases

[1194] Output: User ID

[1195] Step 2:

[1196] The user enters voice commands into the device. The device's microphone captures the voice and sends it to the server. The server uses a speech recognition algorithm to convert the voice into text data and identify the user's instructions and emotional state.

[1197] Input: User's voice

[1198] Data processing / data calculation: Analysis of voice data using speech recognition algorithms and conversion to text

[1199] Output: Text data (user instructions, emotional state)

[1200] Step 3:

[1201] An emotion engine analyzes the text data and voice tone to determine the user's current emotional state. The server uses the emotion engine to identify the user's emotional state (e.g., stress, anxiety, joy, etc.).

[1202] Input: Text data and audio tones

[1203] Data processing / data calculation: Emotional state analysis using emotion engine

[1204] Output: User's emotional state

[1205] Step 4:

[1206] The server checks the user's personal information and schedule based on the storage means, thereby understanding the user's hobbies, preferences, plans, etc., and determines appropriate support content based on the user's emotional state.

[1207] Input: User's emotional state, personal information, schedule

[1208] Data processing / data calculation: Referencing and analyzing information using memory means

[1209] Output: Support content

[1210] Step 5:

[1211] The server provides customized support to the user based on the corresponding support content. For example, if the user feels anxious, the server will send a relaxing breathing exercise video to the device and instruct it to play. The server will also record the support content and the user's emotional state in a database for future reference.

[1212] Input: Support content

[1213] Data processing / data calculation: Deciding on support content and instructing distribution

[1214] Output: Support provided to the user (e.g., relaxation video)

[1215] Step 6:

[1216] The server monitors the user's emotional state and the effectiveness of the assistance provided, offering additional support as needed, or if it is ineffective, offering alternative advice or messages.

[1217] Input: User state after assistance

[1218] Data processing / data calculation: monitoring and analysis data feedback

[1219] Output: Additional support or ongoing monitoring

[1220] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1221] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1222] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1223] [Fourth embodiment]

[1224] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1225] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1226] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1227] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1228] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1230] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1231] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1232] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1233] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1234] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1235] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1236] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1237] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. This system has the following functions.

[1238] First, the server has the function of receiving a user's facial image. This image is uploaded to the server from the device. The server analyzes the facial image using a facial recognition algorithm and extracts its features. The obtained features are compared with an existing database to identify the user's ID. This allows the server to recognize the user's face.

[1239] Next, the user provides their voice to the server, which is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. Based on these voice characteristics, the server compares them with an existing database to identify the user's ID, making it possible to recognize the user through their voice.

[1240] Once the user is identified, the server uses the storage means to store the user's personal information. This information includes the user's name, age, hobbies, preferences, etc. The server can also store the user's schedule information. When the user's information is updated, the server immediately updates the storage means.

[1241] As a concrete example, consider the case where a user wants to add a new appointment. The user communicates the appointment to the server by voice. For example, the user might say, "Add a meeting for next Tuesday." The server first performs voice recognition to identify the user's ID. Next, it uses the storage means to retrieve the user's schedule and adds the new appointment. The user is reminded of this appointment at the appropriate time. This allows the user to manage their schedule efficiently.

[1242] Furthermore, if the user wants to remember something, the server can refer to the storage means and provide the user with past information. For example, if the user instructs the server to "tell me a list of past birthdays," the server will provide the user with stored birthday information. In this way, the server provides appropriate assistance to support the user's daily life.

[1243] This system combines facial and voice recognition to effectively manage users' personal information and provide appropriate assistance, making it extremely useful for busy businessmen, elderly people suffering from memory loss, and people who feel lonely.

[1244] The processing flow will be explained below.

[1245] Step 1:

[1246] Server: The system is initialized and an instance of the PersonalAssistantAI class is created. At this time, user_data is passed as an argument and the internal memory is initialized.

[1247] Step 2:

[1248] Terminal: The user captures a facial image and uploads it to the server, which provides the facial image to the system.

[1249] Step 3:

[1250] Server: The recognize_face method is called, receiving the uploaded face image as an argument. The server analyzes the face image using a facial recognition algorithm and extracts facial features. The extracted features are then compared with an existing database to identify the user's ID.

[1251] Step 4:

[1252] User: Records audio and sends it to the server. This audio includes the user's statements and commands.

[1253] Step 5:

[1254] Server: The recognize_voice method is called to accept a voice sample. The speech is analyzed using a speech recognition algorithm to extract voice features. These features are then matched with an existing database to identify the user's ID.

[1255] Step 6:

[1256] Device: Sends new information (e.g., name, age, etc.) to the server. Sends data containing the user's updated information.

[1257] Step 7:

[1258] Server: Calls the update_memory method, receiving the user ID and new information as arguments. This information is saved or updated in internal memory.

[1259] Step 8:

[1260] Server: Provides appropriate support based on the user's individual information. For example, it manages the user's schedule and stores new events in a storage device.

[1261] Step 9:

[1262] Terminal: When a user wants to obtain specific information, they send a request to the server by voice or text.

[1263] Step 10:

[1264] Server: Calls the get_memory method to retrieve the requested information from internal memory and provide it to the user. For example, it returns a list of past events or birthdays.

[1265] Step 11:

[1266] Server: It also stores information about the user's hobbies and preferences and provides that information upon request. For example, it can provide information about the user's favorite movies and music.

[1267] These steps allow the server to use facial and voice recognition to identify users, and then provide individual memories, schedule management, and appropriate information.

[1268] Example 1

[1269] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1270] In today's world, users must manage a large amount of information in their daily lives and work, placing a significant burden on them. Furthermore, when using various devices or services, users must authenticate each time, resulting in inconvenient usage. Furthermore, declining memory and information overload can make management cumbersome. There is a need for technology that can resolve these issues, efficiently and intuitively manage users' personal information, and provide appropriate support.

[1271] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1272] In this invention, the server includes means for receiving a user's facial image, analyzing the facial image using a facial recognition algorithm, and extracting features, means for comparing the features of the received facial image with an existing database to identify the user's ID, means for receiving the user's voice, analyzing the voice using a voice recognition algorithm, and extracting features of the voice, means for comparing the features of the received voice with an existing database to identify the user's ID, means for storing the user's personal information using a storage means when the user is identified and updating the information as necessary, and means for providing appropriate support based on the stored user information. This makes it possible to efficiently manage the user's personal information and provide appropriate support by intuitively authenticating the user using their face or voice.

[1273] "User's facial image" refers to a still image or video frame of the user's face.

[1274] "Facial recognition algorithm" is a general term for computational methods and programs that extract feature points from facial images and identify individual people.

[1275] "Extracting features" refers to the process of extracting important parameters, shapes, and patterns necessary for identification from facial images and voice.

[1276] "Existing database" refers to a collection of information that has been stored up to now, including personal information about users (such as facial features and voice features).

[1277] "Identifying the user's ID" means uniquely identifying the user by matching the extracted features with an existing database.

[1278] "Speech recognition algorithm" is a general term for computational methods and programs used to analyze speech signals and recognize language and spoken content.

[1279] "Storage means" refers to a storage device or data storage for storing a user's personal information.

[1280] "User's personal information" refers to individual data and information about the user, such as name, age, hobbies, preferences, and schedule.

[1281] "Providing appropriate assistance" means providing necessary information and performing tasks according to the user's requests and circumstances.

[1282] This invention is a personal assistant system that recognizes a user's face and voice, memorizes personal information, and provides appropriate assistance. This system is implemented using the following hardware and software.

[1283] Hardware and software used

[1284] Hardware: Servers, devices (smartphones, tablets, PCs, etc.), microphones, cameras

[1285] Software: Facial recognition algorithms, voice recognition algorithms, database management systems, storage means (memory)

[1286] First, the server has a means to receive the user's facial image. The user captures the facial image using a camera and uploads it to the server from their device. Next, the server uses a facial recognition algorithm to analyze the received facial image and extract its features. The facial recognition algorithm identifies the contours of the face and the relative positions of each feature, and extracts feature points. These feature points are then compared with facial data in an existing database to identify the user's ID.

[1287] Next, the user provides their voice to the server via a microphone. The server is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. The speech recognition algorithm recognizes language-specific patterns and individual voice characteristics from the sound waveforms and extracts features. These voice characteristics are then compared with an existing database to identify the user's ID.

[1288] Once the user's ID is identified, the server uses a storage device to store the user's personal information, including the user's name, age, hobbies, preferences, and schedule information. The server updates this storage device whenever it receives new information from the user.

[1289] For example, if a user wants to add a new appointment, they can send a voice command to the server. The user commands, "Add a meeting for next Tuesday." The server performs voice recognition to identify the user's ID, then retrieves the user's schedule using the storage means and adds the new appointment. The user is reminded of this new appointment at the appropriate time.

[1290] If the user wants to recall past information, the server can provide that information by referring to the storage means. For example, if the user gives a voice command such as "Tell me a list of past birthdays," the server will provide the user with the birthday information it has stored.

[1291] Prompt Sentence Examples

[1292] "Please describe a scenario where a user wants to add a new appointment."

[1293] "Please specify how the system responds when a user retrieves a list of past birthdays."

[1294] By combining facial and voice recognition, this system effectively manages users' personal information and supports their daily lives. It is particularly useful for busy business people and elderly people who suffer from memory loss. This technology frees users from the hassle of managing their information, enabling them to live more efficient lives.

[1295] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1296] Step 1:

[1297] Input: The user activates the device's camera and captures a facial image.

[1298] Specific operation: A facial image is captured on the device.

[1299] Output: The device uploads the captured face image to the server.

[1300] Step 2:

[1301] Input: The server receives the face image sent from the device.

[1302] Specific operation: The server runs a facial recognition algorithm and extracts feature points from the facial image.

[1303] Output: The extracted features are obtained.

[1304] Step 3:

[1305] Input: The extracted features are matched against face data from an existing database.

[1306] What happens: The server uses a database management system to perform the matching process.

[1307] Output: The user's ID is identified.

[1308] Step 4:

[1309] Input: The user uses a microphone to input voice into the server.

[1310] What happens: Audio is captured by the device and sent to the server.

[1311] Output: Audio data arrives at the server.

[1312] Step 5:

[1313] Input: The server processes the received audio data.

[1314] What happens: The server runs a speech recognition algorithm to analyze and extract speech features.

[1315] Output: Extracted audio features are obtained.

[1316] Step 6:

[1317] Input: The extracted audio features are matched against existing audio data in a database.

[1318] What happens: The server uses a database management system to perform the matching process.

[1319] Output: The user's ID is identified.

[1320] Step 7:

[1321] Input: After the user's ID is identified, the server retrieves the user's personal information from the storage means.

[1322] Specific operations: The server reads the relevant data from the storage device.

[1323] Output: The user's personal information (name, age, hobbies, preferences, schedule information, etc.) is obtained.

[1324] Step 8:

[1325] Input: If a user wants to add a new event, they say, "Add a meeting for next Tuesday."

[1326] Specific operation: The server performs voice recognition, confirms the user's ID, retrieves existing schedule information, and adds new information.

[1327] Output: A new event is added to the schedule and a reminder is sent at the appropriate time.

[1328] Step 9:

[1329] Input: If the user wants to see past information, they can say, "Give me a list of past birthdays."

[1330] Specific operation: The server performs voice recognition, confirms the user's ID, and then retrieves the birthday information from the storage means.

[1331] Output: The birthdate information is provided to the user.

[1332] This system allows users to intuitively manage their personal information and receive appropriate support using only their face and voice.

[1333] (Application example 1)

[1334] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1335] In modern society, there is a demand for stronger security in office buildings and apartment complexes. In particular, the difficulty of quickly and accurately authenticating visitors and managing their access using conventional methods places an increased burden on administrators. Furthermore, there is a risk of incomplete records and misidentification, making it an urgent need to improve security levels.

[1336] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1337] In this invention, the server includes a means for storing personal information of users in a storage means, a means for recognizing the user's face using a facial recognition means, and a means for recognizing the user's voice using a voice recognition means. This enables a means for managing access based on facial recognition, a means for managing access based on voice recognition, a means for storing visitor records, and a means for providing appropriate support based on the storage means. This improves the security level and reduces the burden on administrators.

[1338] "Storage means" refers to a device or system that stores information about a user's personal information, schedule information, hobbies and preferences.

[1339] A "face recognition means" is a device or algorithm that analyzes a user's facial image, extracts its features, and identifies the individual.

[1340] "Voice recognition means" refers to a device or algorithm that analyzes a user's voice, extracts its characteristics, and identifies the individual.

[1341] The "means for providing appropriate support" is a device or system that provides the user with necessary information and services based on the storage means.

[1342] A "means for managing access based on facial recognition" is a device or system that uses facial recognition means to identify individuals and control access rights.

[1343] A "means for controlling access based on voice recognition" is a device or system that uses voice recognition means to identify individuals and control access rights.

[1344] "Means for storing visitor records" refers to a device or system that stores visitor access history and authentication results in a database or the like.

[1345] The system required to implement this invention is mainly composed of a server equipped with a storage means, a face recognition means, and a voice recognition means. The specific system configuration and its operation will be described below.

[1346] 1. Storage means

[1347] The server is equipped with a storage means for saving personal information of users (e.g., name, age, hobbies, preferences, schedule information). This storage means uses a database management system (DBMS). For example, a database such as SQLite or MySQL is used.

[1348] 2. Facial Recognition Methods

[1349] The server receives the facial image sent from the device and identifies the individual using a facial recognition algorithm, such as the Python face_recognition library. The server then compares the features of the recognized facial image with a database to identify the user's ID.

[1350] 3. Voice Recognition Method

[1351] The server receives the voice data sent from the device and identifies the individual using a speech recognition algorithm. This algorithm uses the Python speech_recognition library and the Google Speech Recognition API. The server compares the recognized voice characteristics with a database to determine the user's ID.

[1352] 4. Access Management

[1353] The server manages access rights for identified users based on facial and voice recognition. For example, it processes video and audio transmitted from cameras and microphones installed at the entrances of office buildings and apartment complexes to authenticate individuals. Authenticated users can then be given permission to unlock automatic doors and use elevators.

[1354] 5. Record-Keeping

[1355] The server stores the results of access control and visitor records in a database, including information such as the visitor's ID, authentication date and time, and authentication location.

[1356] As a concrete example, the following scenario can be imagined:

[1357] Use case: At the entrance of an office building, administrators use facial and voice recognition to recognize visitors and automatically manage access permissions.

[1358] input:

[1359] Image file (e.g. visitor_image.jpg)

[1360] Audio file (e.g. visitor_audio.wav)

[1361] An example prompt is:

[1362] "Upload a photo: visitor_image.jpg"

[1363] "Please upload audio for speech recognition: visitor_audio.wav"

[1364] This system allows administrators to quickly and accurately authenticate visitors, streamlining access management, and storing records in a database for future review and tracking of unauthorized access.

[1365] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1366] Step 1:

[1367] A user uses a terminal to provide a facial image to a camera at the entrance. The terminal captures this image and sends it to a server. The input is a facial image, and the output is image data.

[1368] Step 2:

[1369] The server passes the received facial image to a facial recognition algorithm to extract facial features. The extracted features are then matched against an existing database. The input is facial image data, and the output is the match result and the user's ID. Specifically, the Python face_recognition library is used to obtain the facial encoding and compare it with known encodings in the database.

[1370] Step 3:

[1371] The user uses a terminal to provide voice data to the microphone at the entrance. The terminal records this voice and sends it to the server. The input is voice data, and the output is an audio file.

[1372] Step 4:

[1373] The server passes the received audio data to a speech recognition algorithm to extract audio features. The extracted features are compared with an existing database. The input is the audio data, and the output is the speech recognition results and the user's ID. Specifically, the Python speech_recognition library and the Google Speech Recognition API are used to convert the audio into text and compare it with known audio data in the database.

[1374] Step 5:

[1375] The server determines the user's identity based on the results of facial and voice recognition. If both results match, the user is considered to have been successfully authenticated. The input is the facial and voice recognition results, and the output is the authentication result.

[1376] Step 6:

[1377] If the authentication is successful, the server sends an instruction to unlock the entrance door to the authorized user via the access management means. The input is the authentication result, and the output is a command to unlock the door. Specifically, the server sends a signal to the electromagnetic locking device.

[1378] Step 7:

[1379] The server stores the visitor's authentication result and the access date and time in a recording medium. The input is the user's ID, authentication result, and date and time information, and the output is the result stored in a database. Specifically, this information is stored using an SQLite or MySQL database.

[1380] Example prompt sentence:

[1381] "Upload a photo: visitor_image.jpg"

[1382] "Please upload audio for speech recognition: visitor_audio.wav"

[1383] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1384] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. Furthermore, by incorporating an emotion engine, it has the ability to recognize and respond to the user's emotional state.

[1385] First, the system has the following functions: The server has the function of receiving a user's facial image. This image is uploaded to the server from the terminal. The server analyzes the facial image using a facial recognition algorithm and extracts its features. The obtained features are compared with an existing database to identify the user's ID. This allows the server to recognize the user's face.

[1386] Next, the user provides their voice to the server, which is equipped with a speech recognition algorithm that analyzes the received voice and extracts voice characteristics. Based on these voice characteristics, the server compares them with an existing database to identify the user's ID, making it possible to recognize the user through their voice.

[1387] Once the user is identified, the server uses the storage means to store the user's personal information. This information includes the user's name, age, hobbies, preferences, etc. The server can also store the user's schedule information. When the user's information is updated, the server immediately updates the storage means.

[1388] Furthermore, the system incorporates an emotion engine that analyzes the user's facial expressions and vocal tone to recognize their current emotional state. For example, if the user shows signs of discomfort, the server can select the appropriate assistance method based on that information.

[1389] As a concrete example, consider the case where a user is feeling stressed. The user communicates their current feelings to the server by voice. For example, if the user says, "I'm feeling a little irritated today," the server performs voice recognition and identifies the user's ID. Next, the emotion engine analyzes the tone and content of the voice and determines that the user is feeling stressed. In response, the server suggests relaxing music and provides relaxation advice. It is also possible to record the user's stress level so that it can be reviewed later.

[1390] Furthermore, if a user has an important presentation coming up, they input their schedule into the server, which not only remembers the information and reminds them before the presentation, but also uses an emotion engine to detect tension before the presentation and send them relaxation advice or encouraging messages.

[1391] In this way, by combining facial recognition, voice recognition, memory means, and an emotion engine, this system provides customized assistance to users, making it extremely useful for busy businessmen, elderly people suffering from memory decline, and people who feel lonely.

[1392] The processing flow will be explained below.

[1393] Step 1:

[1394] Terminal: The user captures a facial image with a camera and uploads it to the server.

[1395] Step 2:

[1396] Server: Calls the recognize_face method to receive the uploaded face image. It analyzes the face image using a facial recognition algorithm and extracts its features. The extracted features are compared with an existing database to identify the user's ID.

[1397] Step 3:

[1398] Device: The user uses a microphone to record voice and sends the voice samples to the server.

[1399] Step 4:

[1400] Server: Calls the recognize_voice method to receive a voice sample. Analyzes the voice using a speech recognition algorithm and extracts voice features. Matches the extracted features with an existing database to identify the user's ID.

[1401] Step 5:

[1402] Terminal: The user enters their information (e.g., name, age, etc.) and sends that information to the server.

[1403] Step 6:

[1404] Server: Call the update_memory method to save or update the user's personal information, including the user's name, age, hobbies, and preferences.

[1405] Step 7:

[1406] Server: Updates the user's schedule information. When a user adds a new event (e.g., a meeting), the server records this information in memory.

[1407] Step 8:

[1408] Terminal: The user sends their current feelings and state to the server via voice or text.

[1409] Step 9:

[1410] Server: Analyzes the received voice and text and recognizes the user's emotional state using an emotion engine, for example by analyzing voice tone and expressed emotions.

[1411] Step 10:

[1412] Server: Provides appropriate support based on the user's recognized emotional state. For example, if the user is feeling stressed, it can provide relaxing music or relaxation advice.

[1413] Step 11:

[1414] Server: Uses an emotion engine to monitor the user's emotional state before a specific event (e.g., before a presentation) and sends encouraging messages or advice as needed.

[1415] Step 12:

[1416] Server: Records past support history and emotional state, and supports users in self-analysis and reflection in the future.

[1417] Through these steps, the server can combine face recognition, voice recognition, memory means, and an emotion engine to provide customized assistance to users, making it an extremely useful service for busy businessmen, elderly people suffering from memory loss, or people who feel lonely.

[1418] Example 2

[1419] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1420] Conventional personal assistant systems have basic functions such as recognizing the user's face and voice and remembering personal information, but they are unable to recognize the user's emotional state and respond based on that, making it difficult to provide truly beneficial assistance to the user. Furthermore, they lack advanced support such as managing the user's schedule and suggesting ways to reduce stress based on the user's emotional state.

[1421] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing a face image and transmitting the face image, a means for analyzing the face image using a face recognition algorithm and extracting features, and a means for matching the features with an existing database and identifying the user's ID. This enables face recognition of the user.

[1422] The server further includes means for recording and transmitting the voice, means for analyzing the voice using a voice recognition algorithm to extract characteristics, and means for matching the voice characteristics with an existing database to identify the user's identity, thereby enabling voice recognition of the user.

[1423] The server also includes a means for storing personal information of the user, a means for providing appropriate support based on the stored information, a means for recognizing the emotional state of the user by analyzing facial expressions and voice tones, and a means for providing appropriate responses based on the emotional state, thereby making it possible to provide support suited to the emotional state of the user.

[1424] Finally, the server includes a means for storing the user's schedule and providing timely reminders, and a means for providing stress reduction suggestions based on the user's emotional state, so that the user can avoid forgetting important plans and receive appropriate support when feeling stressed.

[1425] A "face image" is still image data of a photograph of the user's face.

[1426] A "server" is a computer system that operates on a network and receives, processes, stores, and transmits data.

[1427] A "facial recognition algorithm" is a mathematical model or computational method for extracting specific features from a facial image and identifying an individual based on those features.

[1428] Extracting "features" is a process of obtaining specific patterns or attributes as numerical data from facial images or voice data.

[1429] A "database" is a data structure that systematically stores and makes searchable personal information, facial feature data, voice feature data, and other information about users.

[1430] "ID identification" is the process of matching characteristic data with a database to identify matching personal identifiers.

[1431] "Voice" is waveform data of the voice uttered by the user.

[1432] A "speech recognition algorithm" is a mathematical model or computational method that analyzes linguistic information and specific features from speech data and converts them into text data or identification information.

[1433] "Personal information" refers to data relating to an individual, such as a user's name, age, hobbies, preferences, schedule information, etc.

[1434] "Storage" is the process of saving information in a storage medium such as a database.

[1435] An "emotion engine" is a computational model or system that analyzes facial expressions and vocal tone to recognize a user's emotional state.

[1436] "Appropriate support" is the process of providing advice and services that are tailored to the user's needs and condition.

[1437] "Schedule" is information that describes the user's plans and plans.

[1438] "Remind" is the process of notifying users of specific events or information in advance.

[1439] "Stress reduction suggestions" is the process of recommending advice or activities aimed at reducing stress when a user is feeling stressed.

[1440] MODE FOR CARRYING OUT THE INVENTION

[1441] This invention is a personal assistant system that recognizes the user's face and voice, memorizes personal information, and provides appropriate assistance. By incorporating an emotion engine, this system is also capable of recognizing and responding to the user's emotional state.

[1442] Hardware and software used

[1443] Hardware: Smartphones, PCs, servers

[1444] software:

[1445] Face Recognition Algorithm: OpenCV, FaceNet

[1446] Speech recognition algorithms: Google Speech-to-Text, IBM Watson

[1447] Emotion engine: Microsoft Azure Emotion API, Affectiva

[1448] Facial Recognition Processing

[1449] 1. Image capture

[1450] The user takes a picture of their face using a device (such as a smartphone or PC camera).

[1451] The device temporarily stores the captured image.

[1452] 2. Upload an image

[1453] The device uploads the stored facial image to the server using an HTTP request.

[1454] 3. Image Analysis

[1455] The server analyzes the received image using a facial recognition algorithm (e.g., OpenCV's face detection library or FaceNet model) and extracts facial feature points.

[1456] 4. ID identification

[1457] The server compares the extracted feature data with an existing database to identify the user's ID.

[1458] Speech Recognition Processing

[1459] 1. Voice input

[1460] The user records the voice using the microphone of the device.

[1461] The device temporarily stores the recorded audio.

[1462] 2. Upload audio

[1463] The device uploads the saved audio data to the server using an HTTP request.

[1464] 3. Audio Analysis

[1465] The server analyzes the received audio using a speech recognition algorithm (e.g., Google Speech-to-Text API or IBM Watson Speech to Text) and extracts features from the audio data.

[1466] 4. ID identification

[1467] The server matches the voice characteristics with information in a user database to determine the user's ID.

[1468] Information storage processing

[1469] 1. Information Record

[1470] The server stores the identified user's personal information (such as name, age, hobbies, preferences, schedule information, etc.) in a database.

[1471] 2. Information update

[1472] The server immediately updates the database when user information changes.

[1473] 3. Information provision

[1474] The server uses the recorded information as needed to provide appropriate assistance to the user.

[1475] Emotion Recognition Processing

[1476] 1. Sentiment analysis

[1477] The server uses an emotion engine (Microsoft Azure Emotion API or Affectiva) to analyze facial expressions and vocal tone.

[1478] 2. Providing support

[1479] The server provides an appropriate response based on the results of the sentiment analysis.

[1480] Specific examples

[1481] Dealing with stressed users

[1482] The user uses the device to voice input, "I'm feeling a bit frustrated today."

[1483] The terminal uploads the audio data to the server.

[1484] The server performs voice recognition to identify the user's identity and analyzes the voice tone with an emotion engine.

[1485] The server recognizes the user's emotional state as "stress" and suggests ways to relax.

[1486] What to do if you have an important presentation coming up

[1487] The user uses the terminal to enter schedule information (date and time of presentation).

[1488] The terminal uploads the schedule information to the server.

[1489] The server stores the information and sends a reminder the day before the presentation.

[1490] On the day of the presentation, the emotion engine is used to recognize the user's state of tension and send relaxation advice and encouraging messages.

[1491] Prompt Sentence Examples

[1492] User Emotion Recognition

[1493] If a user verbally tells a server, "I'm feeling a bit frustrated today," explain how the server will recognize the user and what action to take.

[1494] User schedule management

[1495] If a user enters the schedule for an important presentation, explain how the system will support the user and what assistance it will provide.

[1496] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1497] Facial Recognition Processing

[1498] Step 1: Image capture

[1499] The user takes a picture of his or her face using the camera on the terminal.

[1500] Input: A still image of a face.

[1501] Output: Facial image files temporarily saved on your device.

[1502] Specific operation: The device's camera app is launched, and the user faces the camera and takes a picture of their face. The device saves the captured image to its internal storage.

[1503] Step 2: Upload an image

[1504] The device uploads the stored facial image to the server.

[1505] Input: A face image file stored on the device.

[1506] Output: Facial image data sent to the server.

[1507] Specific operation: Using an HTTP request, the device sends facial image data to the server. The server receives the data and stores it in storage.

[1508] Step 3: Image analysis

[1509] The server analyzes the received facial images using a facial recognition algorithm (e.g., OpenCV or FaceNet).

[1510] Input: Facial image data stored on the server.

[1511] Output: Extracted facial feature data.

[1512] Specific operation: Runs a facial recognition algorithm to extract feature points such as eyes, nose, and mouth from the image as numerical data.

[1513] Step 4: Identify ID

[1514] The server compares the extracted feature data with an existing database to identify the user's ID.

[1515] Input: Extracted facial feature data.

[1516] Output: The matched user's ID.

[1517] Specific behavior: Compares the user data and feature data in the database to identify matching users, and stores the identified user IDs for further processing.

[1518] Speech Recognition Processing

[1519] Step 1: Voice Input

[1520] The user records their own voice using the device's microphone.

[1521] Input: Audio as voice.

[1522] Output: An audio file temporarily saved on your device.

[1523] Specific operation: Launch the device's recording app to record the user's voice. After recording, the device will save the audio file to its internal storage.

[1524] Step 2: Upload your audio

[1525] The device uploads the saved audio data to the server.

[1526] Input: An audio file stored on your device.

[1527] Output: The audio data sent to the server.

[1528] Specific operation: Using an HTTP request, the device sends audio data to the server, which receives it and stores it in storage.

[1529] Step 3: Audio analysis

[1530] The server analyzes the received voice data using a speech recognition algorithm (e.g., Google Speech-to-Text API or IBM Watson Speech to Text).

[1531] Input: Audio data stored on the server.

[1532] Output: Extracted audio feature data.

[1533] What it does: Runs a speech recognition algorithm to extract linguistic information and specific features from speech as numerical data.

[1534] Step 4: Identify ID

[1535] The server matches the voice characteristics data with an existing database to determine the user's ID.

[1536] Input: Extracted audio feature data.

[1537] Output: The matched user's ID.

[1538] Specific behavior: Compares user data in the database with the voice feature data to identify matching users, and stores the identified user's ID for further processing.

[1539] Information storage processing

[1540] Step 1: Record information

[1541] The server stores the identified user's personal information in a database.

[1542] Input: User's personal information (name, age, hobbies, preferences, schedule, etc.).

[1543] Output: User information recorded in the database.

[1544] What it does: Adds new information about the user to the database and updates existing information as needed.

[1545] Step 2: Update your information

[1546] The server immediately updates the database when user information changes.

[1547] Input: New user information.

[1548] Output: The updated database.

[1549] Specific operation: When new information is received, it is compared with the existing information, and if it differs, an update process is performed.

[1550] Step 3: Provide information

[1551] The server uses the recorded information as needed to provide appropriate assistance to the user.

[1552] Input: User information recorded in the database.

[1553] Output: The assistance provided to the user.

[1554] What it does: Set reminders based on the user's schedule information and suggest activities based on their preferences.

[1555] Emotion Recognition Processing

[1556] Step 1: Sentiment analysis

[1557] The server uses an emotion engine (Microsoft Azure Emotion API or Affectiva) to analyze facial expressions and vocal tone.

[1558] Input: Facial image data, audio data.

[1559] Output: User's emotional state data.

[1560] Specific operation: Executes the emotion engine, extracts feature points indicating emotions from facial images and tone changes from audio, and determines the current emotional state.

[1561] Step 2: Provide support

[1562] The server provides an appropriate response based on the results of the sentiment analysis.

[1563] Input: Emotional state data.

[1564] Output: Emotion-based support content.

[1565] Specific behavior: If the emotional state is determined to be "stressed," the system will suggest relaxing activities and provide advice to reduce stress.

[1566] (Application example 2)

[1567] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1568] Elderly people need daily support in their daily lives, but conventional personal assistant systems lack the ability to properly recognize the emotional state of the elderly and provide individually customized support. There is also a need for systems that can provide relaxation advice and encouraging messages tailored to the elderly's emotional state. The purpose of this invention is to solve these problems.

[1569] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for storing personal information of the user in storage means, means for recognizing the user's face by face recognition means, means for recognizing the user's voice by voice recognition means, means for analyzing the user's emotional state by an emotion engine, and means for providing support customized according to the user's emotional state. This makes it possible to provide individually customized support based on the user's emotional state and support the elderly in their daily lives.

[1570] "Storage means" refers to a technical element for storing personal information of a user.

[1571] "Facial recognition means" refers to a technical element that recognizes the face of a user, and is a means of analyzing facial images to identify individuals.

[1572] "Voice recognition means" refers to a technical element that recognizes the user's voice, extracts voice characteristics, and performs personal identification and analysis of voice instructions.

[1573] The "means for providing appropriate support" refers to a technical element that provides optimal support to a user based on the storage means.

[1574] The "emotion engine" is a technology element that analyzes the user's facial expressions and tone of voice to recognize their emotional state.

[1575] The "means for providing support customized according to the user's emotional state" is a technical element that provides optimal support individually based on the user's emotional state recognized by the emotion engine.

[1576] The "schedule management means" is a technical element that manages the user's schedule and important events based on the storage means, and sends reminders and notifications at appropriate times.

[1577] "Means for providing reminders and notifications" refers to technical elements that provide necessary reminders and notifications according to the user's emotional state and schedule.

[1578] The "means for providing relaxation advice and encouraging messages" refers to a technical element that determines the emotional state of the user and provides appropriate relaxation methods and encouraging messages.

[1579] The system of the present invention includes a memory means, a face recognition means, a voice recognition means, an emotion engine, and a means for providing customized assistance according to the user's emotional state in order to support the elderly in their daily lives. The system is configured and implemented as follows.

[1580] First, the storage means stores the user's personal information, including name, age, hobbies, preferences, schedule, etc. The device (smartphone or head-mounted display) periodically transmits this information to the server, which stores it in a database.

[1581] Next, the facial recognition means uses the device's built-in camera to recognize the user's face, and the recognized facial features are sent to the server, where a facial recognition algorithm (e.g., OpenCV) analyzes them and matches them with a database to identify the user.

[1582] The speech recognition method uses the device's microphone to transmit the user's voice input to the server, where a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API) analyzes it and identifies the user's voice commands and emotions.

[1583] The emotion engine is a technology component that analyzes the user's facial expressions and vocal tone to recognize their current emotional state. The emotion engine can use, for example, the Microsoft Azure Emotion API. The results of this analysis are sent to a server to help guide the next steps.

[1584] The server uses this information to provide customized support according to the user's emotional state. For example, if it determines that an elderly person is feeling stressed or anxious, the server can send relaxation advice or encouraging messages to the device. Examples of such messages include videos of breathing exercises to help the elderly person relax or the playing of relaxing music.

[1585] Furthermore, the server uses the storage means to manage the user's schedule and provide reminders and notifications as needed, such as timely reminders for important medical appointments or medication times, and can provide relaxation advice if the emotion engine determines that the user's emotional state is tense.

[1586] Examples:

[1587] If an elderly person says, "I'm feeling a little anxious today,"

[1588] The server performs voice recognition to identify the user's ID, and the emotion engine analyzes the voice tone and content to identify the user's anxiety. Based on this information, the server sends a "relaxation breathing exercise video" to the device. In this way, the system can provide assistance tailored to the user's emotional state and improve the quality of life for the elderly.

[1589] Example prompt sentence:

[1590] If a senior is feeling anxious, "offer these next steps: Play a breathing exercise video followed by 10 minutes of relaxing music."

[1591] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1592] Step 1:

[1593] The user shows their face to the device (smartphone or head-mounted display). The device's camera captures a matching facial image and sends it to the server. The server uses a facial recognition algorithm to match the received facial image with existing facial data in a database and identify the user ID. This allows the server to recognize the user.

[1594] Input: User's face image

[1595] Data processing / data calculation: Analysis of facial images using face recognition algorithms and matching with databases

[1596] Output: User ID

[1597] Step 2:

[1598] The user enters voice commands into the device. The device's microphone captures the voice and sends it to the server. The server uses a speech recognition algorithm to convert the voice into text data and identify the user's instructions and emotional state.

[1599] Input: User's voice

[1600] Data processing / data calculation: Analysis of voice data using speech recognition algorithms and conversion to text

[1601] Output: Text data (user instructions, emotional state)

[1602] Step 3:

[1603] An emotion engine analyzes the text data and voice tone to determine the user's current emotional state. The server uses the emotion engine to identify the user's emotional state (e.g., stress, anxiety, joy, etc.).

[1604] Input: Text data and audio tones

[1605] Data processing / data calculation: Emotional state analysis using emotion engine

[1606] Output: User's emotional state

[1607] Step 4:

[1608] The server checks the user's personal information and schedule based on the storage means, thereby understanding the user's hobbies, preferences, plans, etc., and determines appropriate support content based on the user's emotional state.

[1609] Input: User's emotional state, personal information, schedule

[1610] Data processing / data calculation: Referencing and analyzing information using memory means

[1611] Output: Support content

[1612] Step 5:

[1613] The server provides customized support to the user based on the corresponding support content. For example, if the user feels anxious, the server will send a relaxing breathing exercise video to the device and instruct it to play. The server will also record the support content and the user's emotional state in a database for future reference.

[1614] Input: Support content

[1615] Data processing / data calculation: Deciding on support content and instructing distribution

[1616] Output: Support provided to the user (e.g., relaxation video)

[1617] Step 6:

[1618] The server monitors the user's emotional state and the effectiveness of the assistance provided, offering additional support as needed, or if it is ineffective, offering alternative advice or messages.

[1619] Input: User state after assistance

[1620] Data processing / data calculation: monitoring and analysis data feedback

[1621] Output: Additional support or ongoing monitoring

[1622] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1623] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1624] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1625] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1626] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1627] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1628] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1629] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1630] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1631] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1632] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1633] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1634] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1635] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1636] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1637] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1638] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1639] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1640] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1641] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1642] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1643] The following is further disclosed regarding the above embodiment.

[1644] (Claim 1)

[1645] means for storing personal information of a user in a storage means;

[1646] A means for recognizing a user's face by a facial recognition means;

[1647] a means for recognizing a user's voice by a voice recognition means;

[1648] means for providing appropriate assistance based on the storage means;

[1649] A system including:

[1650] (Claim 2)

[1651] 2. The system according to claim 1, further comprising means for managing a schedule of a user based on the storage means.

[1652] (Claim 3)

[1653] 10. The system of claim 1, further comprising means for providing information about the user's hobbies and preferences based on the storage means.

[1654] "Example 1"

[1655] (Claim 1)

[1656] means for receiving a facial image of a user and analyzing the facial image using a facial recognition algorithm to extract facial features;

[1657] A means for identifying the user's ID by comparing the characteristics of the received facial image with an existing database;

[1658] means for receiving a user's voice, analyzing the voice using a voice recognition algorithm, and extracting voice characteristics;

[1659] A means for identifying the user's ID by comparing the characteristics of the received voice with an existing database;

[1660] means for storing personal information of the user using a storage means when the user is identified and updating the information as necessary;

[1661] means for providing appropriate assistance based on the stored user information;

[1662] A system including:

[1663] (Claim 2)

[1664] 2. The system according to claim 1, further comprising means for managing a user's schedule, adding necessary appointments, and reminding the user at appropriate times based on the stored user information.

[1665] (Claim 3)

[1666] 10. The system of claim 1, further comprising means for providing information about the user's hobbies and preferences based on the stored user information.

[1667] "Application Example 1"

[1668] (Claim 1)

[1669] means for storing personal information of a user in a storage means;

[1670] A means for recognizing a user's face by a facial recognition means;

[1671] a means for recognizing a user's voice by a voice recognition means;

[1672] means for providing appropriate assistance based on the storage means;

[1673] a means of controlling access based on facial recognition;

[1674] a means for controlling access based on voice recognition;

[1675] a means of keeping a record of visitors;

[1676] A system including:

[1677] (Claim 2)

[1678] 2. The system according to claim 1, further comprising means for managing a schedule of a user based on the storage means.

[1679] (Claim 3)

[1680] 10. The system of claim 1, further comprising means for providing information about the user's hobbies and preferences based on the storage means.

[1681] "Example 2: Combining Emotion Engines"

[1682] (Claim 1)

[1683] means for capturing a facial image and transmitting the facial image;

[1684] means for analyzing the facial image and extracting features using a facial recognition algorithm;

[1685] A means of matching the characteristics with an existing database to identify the user's identity;

[1686] a means for recording audio and transmitting the audio;

[1687] means for analyzing the speech using a speech recognition algorithm to extract features;

[1688] A means of identifying the user's ID by matching the voice characteristics with an existing database;

[1689] means for storing personal information of the user;

[1690] means for providing appropriate assistance based on the stored information;

[1691] means for recognizing a user's emotional state by analyzing facial expressions and vocal tone;

[1692] a means of providing appropriate responses based on emotional state;

[1693] A system including:

[1694] (Claim 2)

[1695] 10. The system according to claim 1, further comprising means for storing the user's schedule and for providing timely reminders.

[1696] (Claim 3)

[1697] 10. The system of claim 1, further comprising means for providing stress reduction suggestions based on emotional state.

[1698] "Application example 2 when combining emotion engines"

[1699] (Claim 1)

[1700] means for storing personal information of a user in a storage means;

[1701] A means for recognizing a user's face by a facial recognition means;

[1702] a means for recognizing a user's voice by a voice recognition means;

[1703] means for providing appropriate assistance based on the storage means;

[1704] means for analyzing the emotional state of a user using an emotion engine;

[1705] A means for providing customized assistance according to the user's emotional state;

[1706] A system including:

[1707] (Claim 2)

[1708] a means for managing a user's schedule based on the storage means;

[1709] 10. The system of claim 1, further comprising means for providing reminders and notifications according to the user's emotional state.

[1710] (Claim 3)

[1711] means for providing information relating to the user's hobbies and preferences based on the storage means;

[1712] 10. The system of claim 1, further comprising means for providing relaxation advice and encouraging messages according to the user's emotional state. [Explanation of symbols]

[1713] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for storing personal information of a user in a storage means; A means for recognizing a user's face by a facial recognition means; a means for recognizing a user's voice by a voice recognition means; means for providing appropriate assistance based on the storage means; A system including:

2. 2. The system according to claim 1, further comprising means for managing a user's schedule based on the storage means.

3. 2. The system of claim 1, further comprising means for providing information about the user's hobbies and preferences based on the storage means.

Images