system

Abstract

We provide the system. [Solution] A facial recognition method for collecting user identification information, A speech recognition means for converting acoustic information into text data, An automated generation method for generating necessary documents for users, An interactive response system that enables question-based responses, Security management measures for ensuring the security of data communications using encryption technology, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern times, there is a demand for a system that allows users with busy lives to perform procedures and inquiries even outside business hours. However, conventional systems have many deficiencies in terms of security and user support, and in particular, secure identity verification using face recognition and voice recognition has not been achieved. Also, during time periods when manned support is not available, there is a lack of means to address users' anxieties and questions. There is a need to solve these problems.

Means for Solving the Problems

[0005] This invention provides a system that achieves high security by using user facial recognition and voice recognition. Specifically, it solves these problems by configuring a system that includes a facial recognition means for collecting user identification information, a voice recognition means for converting acoustic information into text data, a means for automatically generating necessary documents for the user, an interactive response means for enabling responses based on questions, and a data communication security management means using encryption technology. As a result, users can perform procedures safely and easily at any time.

[0006] "User" refers to an individual who operates the system and performs various procedures or makes inquiries.

[0007] "Identification information" refers to data that includes facial features and voice characteristics used to identify an individual user.

[0008] "Facial recognition means" refers to technology or devices that use a camera to detect and identify a user's face.

[0009] "Voice recognition means" refers to technology or devices for converting a user's voice into text data.

[0010] "Automatic generation means" refers to a technology or system that automatically creates necessary documents based on information provided by the user.

[0011] "Interactive response means" refers to a technology or system that has the function of providing appropriate information and engaging in dialogue in response to questions and requests from users.

[0012] "Security management measures" refer to encryption and data management technologies used to ensure the security of communications and data.

[0013] An "intertabible computing environment" refers to flexible computing resources and system architectures that provide system functionality to users at any given time.

[0014] "Information retrieval means" refers to a technology or system that retrieves and provides appropriate information by referring to past data. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] The present invention is a highly secure system that allows users to perform procedures and make inquiries at any time, and has the following configuration as a specific embodiment.

[0037] When a user uses the system, the terminal first detects the user's face and obtains the user's identification information using facial recognition. The terminal then sends this identification information to the server, which performs authentication by referring to a database. This ensures accurate authentication while securely protecting the user's personal information.

[0038] After successful registration, the user can give voice commands to the device. The device collects the user's audio information through its microphone and sends it to the server. The server uses speech recognition to convert this into text data and understand the user's requests. Through this process, the device can be operated even when the user's hands are occupied.

[0039] Next, depending on the required procedure, the server uses an automated generation mechanism to create the necessary documents. Based on the information provided by the user, the server inserts the appropriate data into the document template and automatically generates the documents. The generated documents are returned to the terminal, where the user can review and approve them.

[0040] Furthermore, when a user asks a question to the system, the terminal sends the question to the server. The server uses an interactive response mechanism to generate an appropriate answer by referring to past FAQ databases and support history. This answer is then communicated to the user via the terminal, resolving any concerns or questions the user may have.

[0041] Furthermore, all communications are encrypted using security management measures, and the server verifies and manages the security of each session to prevent unauthorized access from external sources. This ensures a high level of protection for personal information and procedural data.

[0042] Through this series of operations, this system provides a safe and efficient user experience, creating an environment where users can complete necessary procedures and inquiries 24 hours a day.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The user brings their face close to the device's camera. The device captures the camera image and uses an algorithm to detect the face area.

[0046] Step 2:

[0047] The device extracts facial feature points and sends this data to the server. The server matches this data against facial data in its database to authenticate the user.

[0048] Step 3:

[0049] After successful authentication, the user gives voice commands to the device. The device records the voice through its microphone and converts it into a digital signal.

[0050] Step 4:

[0051] The terminal sends the converted audio information to the server, which then uses a speech recognition engine to convert the speech into text.

[0052] Step 5:

[0053] The server analyzes the user's request based on the converted text data. For the necessary procedures, the server selects the appropriate document template.

[0054] Step 6:

[0055] The server inserts user information into a selected template and automatically generates the necessary documents. The generated documents are sent to the terminal for user confirmation.

[0056] Step 7:

[0057] The user enters a question using the device's chat function. The device then sends that message to the server.

[0058] Step 8:

[0059] The server analyzes the received message and searches the database. Based on past FAQs and support history, it generates an appropriate response.

[0060] Step 9:

[0061] The server sends the generated response to the terminal, and the terminal displays the answer to the user. If the user has any further questions, the process is repeated.

[0062] In this way, the entire system works together to provide users with efficient and secure procedures.

[0063] (Example 1)

[0064] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0065] The problem that this invention aims to solve is to provide a system that allows users to perform various procedures and make inquiries safely and efficiently, 24 hours a day. Conventional systems have security issues in authentication and data transmission, and are not sufficiently convenient because users have to perform manual operations when performing procedures or making inquiries. Against this backdrop, there has been a need for a method that balances improved user experience with security.

[0066] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0067] In this invention, the server includes authentication means for obtaining the user's biometric authentication information using a face detection device, acoustic analysis means for converting acoustic input into symbolic data, and document formation means for generating a document based on the information received from the user. This makes it possible for the user to securely access the system using biometric authentication, easily give instructions via voice input, and quickly generate the necessary documents.

[0068] A "face detection device" is a device used to identify a user's face and to obtain biometric authentication information.

[0069] "Authentication method" refers to a method for verifying the user's identity based on acquired biometric authentication information.

[0070] "Audio input" refers to information based on the voice or sound emitted by the user, which is then converted into symbolic data and processed.

[0071] "Acoustic analysis means" refers to the process of converting collected acoustic input into text data.

[0072] "Symbolic data" refers to text and other character information obtained by converting audio or sound information.

[0073] "Document formation means" refers to a process or device that creates necessary documents based on user instructions and information.

[0074] "Encryption techniques" are technologies that make information invisible in order to enhance data security.

[0075] "Protection and management measures" refer to mechanisms designed to ensure the security of data communications and prevent the unauthorized acquisition or alteration of information.

[0076] A "flexible computing environment" is a computing environment that is designed to allow users to access services at any time.

[0077] An "information acquisition device" is a system or device that refers to past documents and data to provide appropriate information in response to current inquiries.

[0078] The present invention provides a system that allows users to perform procedures and make inquiries securely and efficiently using biometric authentication, voice input, and automated generation technologies. This system includes a face detection device, acoustic analysis means, a document formation device, a dialogue response device, and protection and management means using encryption technology. This section describes specific embodiments of the system.

[0079] Hardware and software configuration:

[0080] 1. Terminal:

[0081] It is equipped with a high-resolution camera to capture the user's face. Face recognition uses a face detection device and a biometric authentication algorithm.

[0082] It has a built-in microphone to collect user voice. Advanced acoustic analysis software is used for speech recognition.

[0083] 2. Server:

[0084] It will have a database to store user information. The database will use an SQL-based system.

[0085] The speech recognition system incorporates an acoustic analysis engine to convert speech into text and utilizes services such as Google® Speech-to-Text.

[0086] Template-based automated document generation software is used for document creation.

[0087] For dialogue responses, we use a generative AI model and leverage a response generation engine such as ChatGPT®.

[0088] Specific example:

[0089] When a user uses the system, the terminal detects their face and verifies the user's identity through authentication methods.

[0090] In the voice command system, when a user says to the terminal, "I want to check my electricity bill," the acoustic analysis device converts this voice into text data and sends it to the server.

[0091] The server uses a document formatting mechanism to insert the latest billing information into the invoice template and automatically generates an invoice in PDF format.

[0092] If necessary, when a user asks a question to the terminal, such as "I want to change the billing address," the conversational response system refers to the FAQ database and past inquiry records and generates an answer.

[0093] As an example of a prompt, by inputting a request such as "Tell me how to check my new invoice" into the AI ​​model, the server can provide appropriate information based on the user's inquiry.

[0094] As described above, this system combines biometric authentication and voice input technology to provide a fast and secure user experience.

[0095] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0096] Step 1:

[0097] The device uses its camera to capture the user's face in order to authenticate the user. The input is the user's face image, and the output is the user's identification information. The facial recognition software within the system uses a biometric authentication algorithm to extract identification features from the face image and sends them to the server.

[0098] Step 2:

[0099] The server receives identification information and verifies it against the database. The input is the identification information received from the terminal, and the output is whether authentication was successful or not. The server searches the database using SQL queries and returns the authentication result to the terminal.

[0100] Step 3:

[0101] After authentication is complete, the user gives instructions to the system by voice. The input is the user's voice instructions, and the output is text data. The terminal uses a microphone to collect the voice data, packets this data, and sends it to the server.

[0102] Step 4:

[0103] The server uses an acoustic analysis engine to convert audio data into text. The input is audio data sent from the terminal, and the output is the converted text data. The server runs acoustic analysis software, extracts text from the audio data, and saves the analysis results.

[0104] Step 5:

[0105] Based on user instructions, the server automatically generates the necessary documents. The input is converted text data, and the output is the generated document. The server, referencing user information, uses a template engine to insert appropriate data into the document template and outputs the completed document in PDF format.

[0106] Step 6:

[0107] When a user asks a question within the system, the terminal sends that question to the server. The input is the user's question in string format, and the output is the answer generated by the server. The server activates the conversational response system, uses a generative AI model to generate an answer that matches the question, and returns it to the terminal.

[0108] Step 7:

[0109] The terminal displays information returned by the server to the user. Input is the document or response data received from the server, and output is information that the user can visually verify. The terminal uses an appropriate interface to present information to the user, enabling the user to complete the procedure.

[0110] (Application Example 1)

[0111] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0112] Conventional systems had cumbersome user authentication procedures within facilities, and because various authentication processes were manual, they were inefficient and raised security concerns. Furthermore, there was the challenge of maintaining facility security while ensuring visitor convenience.

[0113] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0114] In this invention, the server includes biometric authentication means for collecting user identification information, language processing means for converting acoustic information into text data, and dynamic response means for receiving data from a communication device in real time and generating a response based on that data. This enables users to access the facility quickly and securely, and also improves operational efficiency.

[0115] "Biometric authentication methods" refer to technologies that use a user's biometric information to authenticate an individual, and involve identifying individuals using methods such as facial features, fingerprints, or irises.

[0116] "Language processing means" refers to technologies that process, analyze, and convert natural language information such as speech and text, enabling functions such as speech recognition, translation, and summarization.

[0117] An "information generation means" is a system that automatically creates necessary data and documents based on the user's requests and environment.

[0118] A "dynamic response means" is a technology that receives data in real time and immediately generates an appropriate response in response to the received information.

[0119] A "data management system" is a system that manages data storage and communication using methods such as encryption to ensure the safety and accuracy of the data.

[0120] An "identification management system" is a mechanism for verifying and controlling access rights and authentication status for specific users.

[0121] An "on-demand computing environment" is a system that dynamically supplies the necessary computing resources according to the user's requests and executes processing efficiently.

[0122] This system is designed to allow users to complete authentication procedures smoothly and securely within the facility.

[0123] The server first uses biometric authentication to acquire a facial image of the user using the camera built into the device, such as a smartphone or tablet, and then performs authentication based on this image. OpenCV and facial recognition APIs are used for image processing to ensure reliable identity verification.

[0124] Next, the language processing system acquires acoustic information. It collects the user's voice instructions through the device's microphone and converts them into text data using the Google Cloud Speech-to-Text API to analyze what the user's request is.

[0125] The server uses dynamic response mechanisms to instantly generate responses based on user requests. Historical data is stored in MongoDB, and an AI chatbot utilizing natural language processing provides appropriate answers.

[0126] As a data management method, SSL / TLS technology is used to encrypt communications and enhance security. This ensures that users' personal information and confidential facility data are securely protected.

[0127] Furthermore, the identification and management system checks the user's access rights in real time and controls entry to the facility based on those rights.

[0128] For example, when a business person visits a designated company building, they can quickly complete the entry process through facial recognition and voice recognition using their smartphone. In this case, they can obtain company-related information before arrival and begin work without having to wait in line at reception.

[0129] An example of a prompt to input into the generating AI model might be: "Write code to create a security assistant app for facility visitors. Include the following features: facial recognition, voice command processing, automatic generation of entry documents, inquiry handling, and encrypted communication (SSL)."

[0130] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0131] Step 1:

[0132] The device activates its camera and acquires an image of the user's face. Using this image data as input, a facial recognition API is used to extract and match feature points, and the facial recognition result is obtained as output. The server receives this result and determines whether the user is authenticated or not.

[0133] Step 2:

[0134] The device's microphone waits for voice input and acquires audio information from the user. Using this audio data as input, the Google Cloud Speech-to-Text API is used to convert the speech into text data. This text data is sent to the server, where the request is parsed.

[0135] Step 3:

[0136] The server uses the received text data to refer to past visit history and FAQs stored in MongoDB and generates a response to the user's request. Here, a natural language processing algorithm is used to output an appropriate answer to the user's question.

[0137] Step 4:

[0138] The server sends the generated response data to the terminal, and the terminal interactively provides information to the user. In this process, it is also possible to use a speech synthesis engine to convert text into speech and deliver the answer to the user verbally.

[0139] Step 5:

[0140] The server uses SSL / TLS to encrypt all data communications and ensure security. Session management is performed at the start and end of data communication to guarantee the confidentiality of all information.

[0141] Step 6:

[0142] Based on the information the user provides, access to the facility is granted. At this time, the terminal displays dynamically generated admission documents to the user as needed, and the server further verifies the user's access rights and checks again whether they are authorized.

[0143] Step 7:

[0144] The terminal provides a user interface to facilitate these procedures, helping users achieve their goals with minimal steps. All user operations are logged on the server and saved as data for later analysis.

[0145] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0146] The present invention is a system that recognizes user emotions and enables the provision of services based on those emotions, and specific embodiments are described below.

[0147] This system features an emotion engine that analyzes the user's emotions from their voice and facial expressions. When a user speaks into the device, the device acquires the audio and simultaneously captures their facial expressions via the camera. This information is sent to a server, which uses the emotion engine to perform analysis. For example, it comprehensively analyzes the user's voice tone and speed, as well as facial muscle movements, to identify the user's emotions. Emotions that can be identified include joy, sadness, anger, and anxiety.

[0148] After detecting an emotion, the server uses interactive response mechanisms to generate an appropriate response tailored to the user. For example, if the user is feeling anxious, it will provide detailed information and a reassuring response. If the user is feeling happy, it can respond in a more friendly tone.

[0149] Users can receive these emotion-based responses through their devices. This allows users to receive support tailored to their emotions, resulting in a better customer experience. Furthermore, the server can record and analyze the history of emotion recognition, which can be used to improve future services and build user profiles.

[0150] This system enables service providers to respond flexibly based on user emotions. For example, in a customer support scenario, by appropriately addressing not only the content of the inquiry but also the user's emotions, it becomes possible to provide more reliable support. Thus, this invention provides a means to improve the user experience and enhance customer satisfaction in services.

[0151] The following describes the processing flow.

[0152] Step 1:

[0153] The user faces the device's camera and speaks into the microphone. The device simultaneously captures the camera video and audio.

[0154] Step 2:

[0155] The device converts the captured audio data into a digital signal and extracts feature points from the facial video data.

[0156] Step 3:

[0157] The device sends the converted audio data and facial feature data to the server. The server receives this data and begins analysis using its emotion engine.

[0158] Step 4:

[0159] The server analyzes parameters such as tone, speed, and emphasis in the speech, and estimates emotions by detecting muscle movements from facial expressions. This process typically utilizes known machine learning models.

[0160] Step 5:

[0161] Once the emotion engine identifies an emotion, the server uses interactive response mechanisms to generate a response appropriate to that emotion. For example, if the server determines that the user is feeling stressed, it will prepare a response that offers encouragement and reassurance.

[0162] Step 6:

[0163] The server sends the generated response to the terminal, which then displays or reads it aloud to the user. This allows the user to receive emotionally empathetic support.

[0164] Step 7:

[0165] The server records the results of emotion analysis and the history of responses with users, accumulating data to help improve future services.

[0166] This series of processes enables the system to provide users with appropriate responses that take their emotions into account in real time, thereby improving the quality of service.

[0167] (Example 2)

[0168] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0169] In modern digital services, accurately understanding and appropriately responding to users' emotions is crucial. However, existing systems do not fully utilize users' biometric information, resulting in insufficient accuracy and appropriateness in emotion-based responses. This leads to decreased user satisfaction and a failure to improve service quality.

[0170] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0171] In this invention, the server includes an analysis means for analyzing biometric information and identifying emotions, a generation means for generating an appropriate response based on the identified emotions, and a presentation means for displaying or playing the generated response to the user. This enables flexible and accurate responses based on emotions.

[0172] "Analysis means" refers to a technological device that analyzes biometric information obtained from a user and identifies states such as emotions.

[0173] A "generation means" is a technical device that creates a response to the user based on emotions identified by an analysis means.

[0174] "Presentation means" refers to a technical device for providing the generated response to the user visually or audibly.

[0175] "Protective measures" refer to technical devices used to ensure security in data communications.

[0176] A "recording means" is a technological device that stores the results of emotion identification and response history in a storage device, making them available for subsequent analysis and service improvement.

[0177] This system is designed to utilize the user's biometric information to provide flexible, emotion-based responses. Specifically, the terminal and server work together to analyze the user's voice and facial expressions to provide appropriate services.

[0178] Device functions and roles

[0179] The device first captures the user's voice using a microphone. Simultaneously, it captures the user's facial expressions using a camera built into the device. For example, using the latest model's voice microphone and high-resolution camera allows for accurate data collection.

[0180] Server analysis and generation function

[0181] The data acquired by the device is sent to the server. The server has analytical means for analyzing biometric information, processing voice data and facial expression data to identify the user's emotions. The server inputs information about changes in acoustic characteristics and facial muscle movements into a generating AI model to perform emotion identification.

[0182] The server then uses a generation mechanism to generate a response that corresponds to the identified emotion. Typical response generation utilizes a generation AI model and uses appropriate prompts based on the user's emotion. An example of a prompt might be, "If the user is feeling anxious, generate a response that provides detailed information to reassure them."

[0183] Presentation of responses and recording of history

[0184] The generated response is sent back from the server to the terminal, which then uses its display and speakers to provide the response to the user. This allows the user to receive appropriate support that matches their emotions.

[0185] Furthermore, the server uses recording mechanisms to save the sentiment identification results and response history. This history can be used to improve future services and build new user profiles. For example, by referring to past sentiment data, it is possible to analyze user behavior and provide more personalized services.

[0186] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0187] Step 1:

[0188] The user enters information.

[0189] When a user begins speaking into the device, it activates its built-in microphone to acquire audio data. Simultaneously, it activates the camera to capture the user's facial expressions. The input data consists of audio signals and image data. Based on this, the device captures audio and video information.

[0190] Step 2:

[0191] The device sends data to the server.

[0192] The terminal sends the captured audio data (audio signal conversion data) and facial expression data (image data) to the server as a data package. The output at this stage is the data package received by the server. Since secure communication methods are used for transmission, the process includes the execution of an encryption protocol.

[0193] Step 3:

[0194] The server analyzes the biometric information.

[0195] The server inputs the received data package into an analysis device and uses a generative AI model to analyze the voice and facial expressions. The server extracts features such as tone and pitch from the voice data and analyzes the movement of facial muscles from the image data. The output of this analysis is identified emotional information. For example, it comprehensively evaluates the intonation of the voice and changes in facial expressions to recognize the user's emotions.

[0196] Step 4:

[0197] The server generates a response.

[0198] Based on the identified sentiment information, the server generates prompts using a generation mechanism and uses a generation AI model to produce responses. A specific prompt might be, "If the user is happy, generate a more friendly response." The output at this stage is either text or audio data for the user to receive.

[0199] Step 5:

[0200] The server sends a response to the terminal.

[0201] The server sends the generated response data back to the terminal. This communication is also encrypted using a secure protocol. The receiving terminal then displays or plays the response data to the user as audio or text.

[0202] Step 6:

[0203] The server records history.

[0204] The server records the analyzed sentiment information and response content in a database. This recorded data will be used for future user characteristic analysis and service improvement. This includes the operation of properly saving data to the database.

[0205] (Application Example 2)

[0206] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0207] Modern service delivery demands flexible and personalized responses based on user emotions. However, conventional systems struggle to accurately recognize emotions from users' facial expressions and voices and generate immediate and appropriate responses accordingly. This results in a lack of effective means to improve user satisfaction and alleviate anxiety. In particular, there is a need for the development of systems that provide information and interactive responses tailored to user emotions.

[0208] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0209] In this invention, the server includes emotion detection means for analyzing the user's emotions, information provision means for providing optimal information, and security management means for ensuring the security of data communication. This enables the generation of appropriate responses based on the user's emotions and the provision of information safely and effectively.

[0210] "User identification information" refers to information used by a system to recognize and authenticate a specific user.

[0211] "Facial recognition technology" refers to technology used to identify individuals based on their facial features.

[0212] "Acoustic information" refers to information perceived through hearing, including human speech and ambient sounds.

[0213] "Speech recognition means" refers to technology for converting acoustic information into text data.

[0214] "Emotion detection means" refers to technology for identifying a user's emotional state from their voice and facial expressions.

[0215] "Automatic generation means" refers to technology for automatically creating necessary documents and data in response to user requests.

[0216] "Interactive response methods" are technologies for responding to user questions in a natural, conversational format.

[0217] "Information provision means" refers to technologies for searching for and presenting appropriate information in response to user requests.

[0218] "Security management measures" refer to encryption technologies and other protective measures to ensure the security of data communications.

[0219] An "intertabible computing environment" is the arrangement of computing resources to provide users with appropriate functions and services based on their authentication.

[0220] "Past data" refers to a collection of recorded information accumulated based on user requests and behavioral history.

[0221] This invention is a system that provides flexible and personalized services based on the user's emotions. The system consists of the following elements:

[0222] First, the user's device uses its camera and microphone to capture facial expressions and audio information. This information is collected in real time and sent to the server. Image processing libraries such as OpenCV are used for the camera, and the Google Speech-to-Text API is used for processing the audio information.

[0223] The server analyzes emotions based on the received data using emotion detection means. These emotion detection means estimate emotions using models built with machine learning libraries such as Keras and TENSORFLOW®. Based on the analyzed emotions, information provision means are utilized to provide optimal information. Interactive response means generate natural conversational responses in response to prompts.

[0224] Security management measures include applying encryption technology to data communications to ensure data security and protect user information.

[0225] As a concrete example, suppose a user says, "I'm worried about security." In this case, the emotion detection system senses the user's anxiety, the information provision system presents detailed security measures, and the interactive response system provides appropriate advice in voice. The prompt used in this example is, "When a user is feeling anxious, please suggest how to provide them with the latest security information as a concrete example."

[0226] In this way, the system provides sophisticated services tailored to the user's emotions, improving the user experience.

[0227] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0228] Step 1:

[0229] The device activates its camera and microphone to capture the user's facial expressions and voice data. This allows for real-time capture of the user's facial expressions and voice input.

[0230] Step 2:

[0231] The device converts captured audio data into text data using speech recognition technology. Using the Google Speech-to-Text API, this text data is sent to a server, providing the user's spoken content in an analyzable format.

[0232] Step 3:

[0233] The server receives facial expression data and speech-to-text data sent from the terminal and performs emotion analysis using emotion detection methods. Using Keras or TensorFlow, it analyzes speech tone, speaking speed, and facial expression patterns to identify the user's emotional state. This analysis determines whether the user is experiencing a specific emotion such as joy or anxiety.

[0234] Step 4:

[0235] The server collects data to generate the optimal response using informational tools based on the identified emotion. For example, if the emotion of anxiety is identified, it searches the database for information regarding security improvements and prepares it.

[0236] Step 5:

[0237] The server generates prompts using an interactive response mechanism based on the collected data, and creates natural-sounding conversational responses using a generative AI model. The prompt used here is, "When a user is feeling anxious, please suggest how to provide them with the latest security information as a concrete example."

[0238] Step 6:

[0239] The server encrypts the response it generates using security management measures and sends it to the terminal. This enables the provision of safe and personalized information tailored to each user.

[0240] Step 7:

[0241] The terminal decrypts the encrypted response received from the server and presents it to the user in audio or text format. This allows the user to receive advice with peace of mind.

[0242] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0243] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0244] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0245] [Second Embodiment]

[0246] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0247] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0248] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0249] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0250] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0251] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0252] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0253] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0254] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0255] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0256] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0257] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0258] The present invention is a highly secure system that allows users to perform procedures and make inquiries at any time, and has the following configuration as a specific embodiment.

[0259] When a user uses the system, the terminal first detects the user's face and obtains the user's identification information using facial recognition. The terminal then sends this identification information to the server, which performs authentication by referring to a database. This ensures accurate authentication while securely protecting the user's personal information.

[0260] After successful registration, the user can give voice commands to the device. The device collects the user's audio information through its microphone and sends it to the server. The server uses speech recognition to convert this into text data and understand the user's requests. Through this process, the device can be operated even when the user's hands are occupied.

[0261] Next, depending on the required procedure, the server uses an automated generation mechanism to create the necessary documents. Based on the information provided by the user, the server inserts the appropriate data into the document template and automatically generates the documents. The generated documents are returned to the terminal, where the user can review and approve them.

[0262] Furthermore, when a user asks a question to the system, the terminal sends that question to the server. The server uses an interactive response mechanism to refer to past FAQ databases and support history to generate an appropriate answer. This answer is then communicated to the user via the terminal, resolving any concerns or questions the user may have.

[0263] Furthermore, all communications are encrypted using security management measures, and the server verifies and manages the security of each session to prevent unauthorized access from external sources. This ensures a high level of protection for personal information and procedural data.

[0264] Through this series of operations, this system provides a safe and efficient user experience, creating an environment where users can complete necessary procedures and inquiries 24 hours a day.

[0265] The following describes the processing flow.

[0266] Step 1:

[0267] The user brings their face close to the device's camera. The device captures the camera image and uses an algorithm to detect the face area.

[0268] Step 2:

[0269] The device extracts facial feature points and sends this data to the server. The server matches this data against facial data in its database to authenticate the user.

[0270] Step 3:

[0271] After successful authentication, the user gives voice commands to the device. The device records the voice through its microphone and converts it into a digital signal.

[0272] Step 4:

[0273] The terminal sends the converted audio information to the server, which then uses a speech recognition engine to convert the speech into text.

[0274] Step 5:

[0275] The server analyzes the user's request based on the converted text data. For the necessary procedures, the server selects the appropriate document template.

[0276] Step 6:

[0277] The server inserts user information into a selected template and automatically generates the necessary documents. The generated documents are sent to the terminal for user confirmation.

[0278] Step 7:

[0279] The user enters a question using the device's chat function. The device then sends that message to the server.

[0280] Step 8:

[0281] The server analyzes the received message and searches the database. Based on past FAQs and support history, it generates an appropriate response.

[0282] Step 9:

[0283] The server transmits the response generated to the terminal, and the terminal displays the answer to the user. If there are additional questions from the user, the process is repeated.

[0284] In this way, the entire system cooperates to provide an efficient and secure procedure for the user.

[0285] (Example 1)

[0286] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0287] The problem to be solved by the present invention is to provide a system that enables a user to perform various procedures and inquiries safely and efficiently at any time of the day. In the conventional system, there are security issues in authentication and data transmission, and in addition, when a user performs a procedure or an inquiry, manual operation is required, so the convenience is not sufficient. Against such a background, a method that achieves both improvement of the user experience and security has been demanded.

[0288] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0289] In this invention, the server includes an authentication means for acquiring biometric authentication information of a user using a face detection device, an acoustic analysis means for converting acoustic input into symbol data, and a document formation means for generating a document based on the information received from the user. Thereby, while the user performs secure access by biometric authentication, it is possible to easily give instructions by voice input and quickly generate the necessary document.

[0290] The "face detection device" is a device for identifying the face of a user and is used to acquire biometric authentication information.

[0291] "Authentication method" refers to a method for verifying the user's identity based on acquired biometric authentication information.

[0292] "Audio input" refers to information based on the voice or sound emitted by the user, which is then converted into symbolic data and processed.

[0293] "Acoustic analysis means" refers to the process of converting collected acoustic input into text data.

[0294] "Symbolic data" refers to text and other character information obtained by converting audio or sound information.

[0295] "Document formation means" refers to a process or device that creates necessary documents based on user instructions and information.

[0296] "Encryption techniques" are technologies that make information invisible in order to enhance data security.

[0297] "Protection and management measures" refer to mechanisms designed to ensure the security of data communications and prevent the unauthorized acquisition or alteration of information.

[0298] A "flexible computing environment" is a computing environment that is designed to allow users to access services at any time.

[0299] An "information acquisition device" is a system or device that refers to past documents and data to provide appropriate information in response to current inquiries.

[0300] The present invention provides a system that allows users to perform procedures and make inquiries securely and efficiently using biometric authentication, voice input, and automated generation technologies. This system includes a face detection device, acoustic analysis means, a document formation device, a dialogue response device, and protection and management means using encryption technology. This section describes specific embodiments of the system.

[0301] Hardware and Software Configuration:

[0302] 1. Terminal:

[0303] It is equipped with a high-resolution camera to capture the user's face. For face recognition, a face detection device and a biometric algorithm are used.

[0304] It has a built-in microphone to collect the user's voice. For voice recognition, advanced acoustic analysis software is used.

[0305] 2. Server:

[0306] It is equipped with a database to store user information. The database uses a SQL-based system.

[0307] For voice recognition, it is equipped with an acoustic analysis engine to convert voice into text and utilizes services such as Google Speech-to-Text.

[0308] For document formation, template-based automatic document generation software is used.

[0309] For dialogue response, a generative AI model is used, and a response generation engine such as ChatGPT is utilized.

[0310] Specific Example:

[0311] When the user uses the system, the terminal detects the face and verifies the user's identity by authentication means.

[0312] In a voice instruction, when the user says "I want to check the electricity bill" to the terminal, the acoustic analysis means converts this voice into character data and sends it to the server.

[0313] The server uses the document formation means to insert the latest billing information into the bill template and automatically generates a PDF-formatted bill.

[0314] If necessary, when a user asks a question to the terminal, such as "I want to change the billing address," the conversational response system refers to the FAQ database and past inquiry records and generates an answer.

[0315] As an example of a prompt, by inputting a request such as "Tell me how to check my new invoice" into the AI ​​model, the server can provide appropriate information based on the user's inquiry.

[0316] As described above, this system combines biometric authentication and voice input technology to provide a fast and secure user experience.

[0317] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0318] Step 1:

[0319] The device uses its camera to capture the user's face in order to authenticate the user. The input is the user's face image, and the output is the user's identification information. The facial recognition software within the system uses a biometric authentication algorithm to extract identifying features from the face image and sends them to the server.

[0320] Step 2:

[0321] The server receives identification information and verifies it against the database. The input is the identification information received from the terminal, and the output is whether authentication was successful or not. The server searches the database using SQL queries and returns the authentication result to the terminal.

[0322] Step 3:

[0323] After authentication is complete, the user gives instructions to the system by voice. The input is the user's voice instructions, and the output is text data. The terminal uses a microphone to collect the voice data, packets this data, and sends it to the server.

[0324] Step 4:

[0325] The server uses an acoustic analysis engine to convert audio data into text. The input is audio data sent from the terminal, and the output is the converted text data. The server runs acoustic analysis software, extracts text from the audio data, and saves the analysis results.

[0326] Step 5:

[0327] Based on user instructions, the server automatically generates the necessary documents. The input is converted text data, and the output is the generated document. The server, referencing user information, uses a template engine to insert appropriate data into the document template and outputs the completed document in PDF format.

[0328] Step 6:

[0329] When a user asks a question within the system, the terminal sends that question to the server. The input is the user's question in string format, and the output is the answer generated by the server. The server activates the conversational response system, uses a generative AI model to generate an answer that matches the question, and returns it to the terminal.

[0330] Step 7:

[0331] The terminal displays information returned by the server to the user. Input is the document or response data received from the server, and output is information that the user can visually verify. The terminal uses an appropriate interface to present information to the user, enabling the user to complete the procedure.

[0332] (Application Example 1)

[0333] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0334] Conventional systems had cumbersome user authentication procedures within facilities, and because various authentication processes were manual, they were inefficient and raised security concerns. Furthermore, there was the challenge of maintaining facility security while ensuring visitor convenience.

[0335] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0336] In this invention, the server includes biometric authentication means for collecting user identification information, language processing means for converting acoustic information into text data, and dynamic response means for receiving data from a communication device in real time and generating a response based on that data. This enables users to access the facility quickly and securely, and also improves operational efficiency.

[0337] "Biometric authentication methods" are technologies used to authenticate individuals using their biometric information, and they perform individual identification using methods such as facial recognition, fingerprints, and iris recognition.

[0338] "Language processing means" refers to technologies that process, analyze, and convert natural language information such as speech and text, enabling functions such as speech recognition, translation, and summarization.

[0339] An "information generation means" is a system that automatically creates necessary data and documents based on the user's requests and environment.

[0340] A "dynamic response means" is a technology that receives data in real time and immediately generates an appropriate response in response to the received information.

[0341] A "data management system" is a system that manages data storage and communication using methods such as encryption to ensure the safety and accuracy of the data.

[0342] An "identification management system" is a mechanism for verifying and controlling access rights and authentication status for specific users.

[0343] An "on-demand computing environment" is a system that dynamically supplies the necessary computing resources according to the user's requests and executes processing efficiently.

[0344] This system is designed to allow users to complete authentication procedures smoothly and securely within the facility.

[0345] The server first uses biometric authentication to acquire a facial image of the user using the camera built into the device, such as a smartphone or tablet, and then performs authentication based on this image. OpenCV and facial recognition APIs are used for image processing to ensure reliable identity verification.

[0346] Next, the language processing system acquires acoustic information. It collects the user's voice instructions through the device's microphone and converts them into text data using the Google Cloud Speech-to-Text API to analyze what the user's request is.

[0347] The server uses dynamic response mechanisms to instantly generate responses based on user requests. Historical data is stored in MongoDB, and an AI chatbot utilizing natural language processing provides appropriate answers.

[0348] As a data management method, SSL / TLS technology is used to encrypt communications and enhance security. This ensures that users' personal information and confidential facility data are securely protected.

[0349] Furthermore, the identification and management system checks the user's access rights in real time and controls entry to the facility based on those rights.

[0350] For example, when a business person visits a designated company building, they can quickly complete the entry process through facial recognition and voice recognition using their smartphone. In this case, they can obtain company-related information before arrival and begin work without having to wait in line at reception.

[0351] An example of a prompt to input into the generating AI model might be: "Write code to create a security assistant app for facility visitors. Include the following features: facial recognition, voice command processing, automatic generation of entry documents, inquiry handling, and encrypted communication (SSL)."

[0352] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0353] Step 1:

[0354] The device activates its camera and acquires an image of the user's face. Using this image data as input, the device uses a facial recognition API to extract and match feature points, and obtains the facial recognition result as output. The server receives this result and determines whether the user is authenticated or not.

[0355] Step 2:

[0356] The device's microphone waits for voice input and acquires audio information from the user. Using this audio data as input, the Google Cloud Speech-to-Text API is used to convert the speech into text data. This text data is sent to the server, where the request is parsed.

[0357] Step 3:

[0358] The server uses the received text data to refer to past visit history and FAQs stored in MongoDB and generates a response to the user's request. Here, a natural language processing algorithm is used to output an appropriate answer to the user's question.

[0359] Step 4:

[0360] The server sends the generated response data to the terminal, and the terminal interactively provides information to the user. In this process, it is also possible to use a speech synthesis engine to convert text into speech and deliver the answer to the user verbally.

[0361] Step 5:

[0362] The server uses SSL / TLS to encrypt all data communications and ensure security. Session management is performed at the start and end of data communication to guarantee the confidentiality of all information.

[0363] Step 6:

[0364] Based on the information obtained by the user, access to the facility is granted. At this time, the terminal displays dynamically generated admission documents to the user as needed, and the server further verifies the user's access rights and checks again whether they are authorized.

[0365] Step 7:

[0366] The terminal provides a user interface to facilitate these procedures, helping users achieve their goals with minimal steps. All user operations are logged on the server and saved as data for later analysis.

[0367] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0368] The present invention is a system that recognizes user emotions and enables the provision of services based on those emotions, and specific embodiments are described below.

[0369] This system features an emotion engine that analyzes the user's emotions from their voice and facial expressions. When a user speaks into the device, the device acquires the audio and simultaneously captures their facial expressions via the camera. This information is sent to a server, which uses the emotion engine to perform analysis. For example, it comprehensively analyzes the user's voice tone and speed, as well as facial muscle movements, to identify the user's emotions. Emotions that can be identified include joy, sadness, anger, and anxiety.

[0370] After detecting an emotion, the server uses interactive response mechanisms to generate an appropriate response tailored to the user. For example, if the user is feeling anxious, it will provide detailed information and a reassuring response. If the user is feeling happy, it can respond in a more friendly tone.

[0371] Users can receive these emotion-based responses through their devices. This allows users to receive support tailored to their emotions, resulting in a better customer experience. Furthermore, the server can record and analyze the history of emotion recognition, which can be used to improve future services and build user profiles.

[0372] This system enables service providers to respond flexibly based on user emotions. For example, in a customer support scenario, by appropriately addressing not only the content of the inquiry but also the user's emotions, it becomes possible to provide more reliable support. Thus, this invention provides a means to improve the user experience and enhance customer satisfaction in services.

[0373] The following describes the processing flow.

[0374] Step 1:

[0375] The user faces the device's camera and speaks into the microphone. The device simultaneously captures the camera video and audio.

[0376] Step 2:

[0377] The device converts the captured audio data into a digital signal and extracts feature points from the facial video data.

[0378] Step 3:

[0379] The device sends the converted audio data and facial feature data to the server. The server receives this data and begins analysis using its emotion engine.

[0380] Step 4:

[0381] The server analyzes parameters such as tone, speed, and emphasis in the speech, and estimates emotions by detecting muscle movements from facial expressions. This process typically utilizes known machine learning models.

[0382] Step 5:

[0383] Once the emotion engine identifies an emotion, the server uses interactive response mechanisms to generate a response appropriate to that emotion. For example, if the server determines that the user is feeling stressed, it will prepare a response that offers encouragement and reassurance.

[0384] Step 6:

[0385] The server sends the generated response to the terminal, which then displays or reads it aloud to the user. This allows the user to receive emotionally empathetic support.

[0386] Step 7:

[0387] The server records the results of emotion analysis and the history of responses with users, accumulating data to help improve future services.

[0388] This series of processes enables the system to provide users with appropriate responses that take their emotions into account in real time, thereby improving the quality of service.

[0389] (Example 2)

[0390] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0391] In modern digital services, accurately understanding and appropriately responding to users' emotions is crucial. However, existing systems do not fully utilize users' biometric information, resulting in insufficient accuracy and appropriateness in emotion-based responses. This leads to decreased user satisfaction and a failure to improve service quality.

[0392] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0393] In this invention, the server includes an analysis means for analyzing biometric information and identifying emotions, a generation means for generating an appropriate response based on the identified emotions, and a presentation means for displaying or playing the generated response to the user. This enables flexible and accurate responses based on emotions.

[0394] "Analysis means" refers to a technological device that analyzes biometric information obtained from a user and identifies states such as emotions.

[0395] A "generation means" is a technical device that creates a response to the user based on emotions identified by an analysis means.

[0396] "Presentation means" refers to a technical device for providing the generated response to the user visually or audibly.

[0397] "Protective measures" refer to technical devices used to ensure security in data communications.

[0398] A "recording means" is a technological device that stores the results of emotion identification and response history in a storage device, making them available for subsequent analysis and service improvement.

[0399] This system is designed to utilize the user's biometric information to provide flexible, emotion-based responses. Specifically, the terminal and server work together to analyze the user's voice and facial expressions to provide appropriate services.

[0400] Device functions and roles

[0401] The device first captures the user's voice using a microphone. Simultaneously, it captures the user's facial expressions using a camera built into the device. For example, using the latest model's voice microphone and high-resolution camera allows for accurate data collection.

[0402] Server analysis and generation function

[0403] The data acquired by the device is sent to the server. The server has analytical means to analyze biometric information, processing voice data and facial expression data to identify the user's emotions. The server inputs information about changes in acoustic characteristics and facial muscle movements into a generating AI model to perform emotion identification.

[0404] The server then uses a generation mechanism to generate a response that corresponds to the identified emotion. Typical response generation utilizes a generation AI model and uses appropriate prompts based on the user's emotion. An example of a prompt might be, "If the user is feeling anxious, generate a response that provides detailed information to reassure them."

[0405] Presentation of responses and recording of history

[0406] The generated response is sent back from the server to the terminal, which then uses its display and speakers to provide the response to the user. This allows the user to receive appropriate support that matches their emotions.

[0407] Furthermore, the server uses recording mechanisms to save the sentiment identification results and response history. This history can be used to improve future services and build new user profiles. For example, by referring to past sentiment data, it is possible to analyze user behavior and provide more personalized services.

[0408] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0409] Step 1:

[0410] The user enters information.

[0411] When a user begins speaking into the device, it activates its built-in microphone to acquire audio data. Simultaneously, it activates the camera to capture the user's facial expressions. The input data consists of audio signals and image data. Based on this, the device captures audio and video information.

[0412] Step 2:

[0413] The device sends data to the server.

[0414] The terminal sends the captured audio data (audio signal conversion data) and facial expression data (image data) to the server as a data package. The output at this stage is the data package received by the server. Since secure communication methods are used for transmission, the process includes the execution of an encryption protocol.

[0415] Step 3:

[0416] The server analyzes the biometric information.

[0417] The server inputs the received data package into an analysis device and uses a generative AI model to analyze the voice and facial expressions. The server extracts features such as tone and pitch from the voice data and analyzes the movement of facial muscles from the image data. The output of this analysis is identified emotional information. For example, it comprehensively evaluates the intonation of the voice and changes in facial expressions to recognize the user's emotions.

[0418] Step 4:

[0419] The server generates a response.

[0420] Based on the identified sentiment information, the server generates prompts using a generation mechanism and uses a generation AI model to produce responses. A specific prompt might be, "If the user is happy, generate a more friendly response." The output at this stage is either text or audio data for the user to receive.

[0421] Step 5:

[0422] The server sends a response to the terminal.

[0423] The server sends the generated response data back to the terminal. This communication is also encrypted using a secure protocol. The receiving terminal then displays or plays the response data to the user as audio or text.

[0424] Step 6:

[0425] The server records history.

[0426] The server records the analyzed sentiment information and response content in a database. This recorded data will be used for future user characteristic analysis and service improvement. This includes the operation of properly saving data to the database.

[0427] (Application Example 2)

[0428] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0429] Modern service delivery demands flexible and personalized responses based on user emotions. However, conventional systems struggle to accurately recognize emotions from users' facial expressions and voices and generate immediate and appropriate responses accordingly. This results in a lack of effective means to improve user satisfaction and alleviate anxiety. In particular, there is a need for the development of systems that provide information and interactive responses tailored to user emotions.

[0430] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0431] In this invention, the server includes emotion detection means for analyzing the user's emotions, information provision means for providing optimal information, and security management means for ensuring the security of data communication. This enables the generation of appropriate responses based on the user's emotions and the provision of information safely and effectively.

[0432] "User identification information" refers to information used by a system to recognize and authenticate a specific user.

[0433] "Facial recognition technology" refers to technology used to identify individuals based on their facial features.

[0434] "Acoustic information" refers to information perceived through hearing, including human speech and ambient sounds.

[0435] "Speech recognition means" refers to technology for converting acoustic information into text data.

[0436] "Emotion detection means" refers to technology for identifying a user's emotional state from their voice and facial expressions.

[0437] "Automatic generation means" refers to technology for automatically creating necessary documents and data in response to user requests.

[0438] "Interactive response methods" are technologies for responding to user questions in a natural, conversational format.

[0439] "Information provision means" refers to technologies for searching for and presenting appropriate information in response to user requests.

[0440] "Security management measures" refer to encryption technologies and other protective measures to ensure the security of data communications.

[0441] An "intertabible computing environment" is the arrangement of computing resources to provide users with appropriate functions and services based on their authentication.

[0442] "Past data" refers to a collection of recorded information accumulated based on user requests and behavioral history.

[0443] This invention is a system that provides flexible and personalized services based on the user's emotions. The system consists of the following elements:

[0444] First, the user's device uses its camera and microphone to capture facial expressions and audio information. This information is collected in real time and sent to the server. Image processing libraries such as OpenCV are used for the camera, and the Google Speech-to-Text API is used for processing the audio information.

[0445] The server analyzes emotions based on the received data using emotion detection means. These emotion detection means estimate emotions using models built with machine learning libraries such as Keras and TensorFlow. Based on the analyzed emotions, information provision means are utilized to provide optimal information. Interactive response means generate natural conversational responses in response to prompts.

[0446] Security management measures include applying encryption technology to data communications to ensure data security and protect user information.

[0447] As a concrete example, suppose a user says, "I'm worried about security." In this case, the emotion detection system senses the user's anxiety, the information provision system presents detailed security measures, and the interactive response system provides appropriate advice in voice. The prompt used in this example is, "When a user is feeling anxious, please suggest how to provide them with the latest security information as a concrete example."

[0448] In this way, the system provides sophisticated services tailored to the user's emotions, improving the user experience.

[0449] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0450] Step 1:

[0451] The device activates its camera and microphone to capture the user's facial expressions and voice data. This allows for real-time capture of the user's facial expressions and voice input.

[0452] Step 2:

[0453] The device converts captured audio data into text data using speech recognition technology. Using the Google Speech-to-Text API, this text data is sent to a server, providing the user's spoken content in an analyzable format.

[0454] Step 3:

[0455] The server receives facial expression data and speech-to-text data sent from the terminal and performs emotion analysis using emotion detection methods. Using Keras or TensorFlow, it analyzes speech tone, speaking speed, and facial expression patterns to identify the user's emotional state. This analysis determines whether the user is experiencing a specific emotion such as joy or anxiety.

[0456] Step 4:

[0457] Based on the identified emotion, the server collects data using informational tools to generate the optimal response. For example, if the emotion of anxiety is identified, it searches the database for and prepares information regarding security improvements.

[0458] Step 5:

[0459] The server generates prompts using an interactive response mechanism based on the collected data, and creates natural-sounding conversational responses using a generative AI model. The prompt used here is, "When a user is feeling anxious, please suggest how to provide them with the latest security information as a concrete example."

[0460] Step 6:

[0461] The server encrypts the response it generates using security management measures and sends it to the terminal. This enables the provision of safe and personalized information tailored to each user.

[0462] Step 7:

[0463] The terminal decrypts the encrypted response received from the server and presents it to the user in audio or text format. This allows the user to receive advice with peace of mind.

[0464] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0465] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0466] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0467] [Third Embodiment]

[0468] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0469] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0470] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0471] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0472] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0473] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0474] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0475] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0476] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0477] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0478] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0479] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0480] The present invention is a highly secure system that allows users to perform procedures and make inquiries at any time, and has the following configuration as a specific embodiment.

[0481] When a user uses the system, the terminal first detects the user's face and obtains the user's identification information using facial recognition. The terminal then sends this identification information to the server, which performs authentication by referring to a database. This ensures accurate authentication while securely protecting the user's personal information.

[0482] After successful registration, the user can give voice commands to the device. The device collects the user's audio information through its microphone and sends it to the server. The server uses speech recognition to convert this into text data and understand the user's requests. Through this process, the device can be operated even when the user's hands are occupied.

[0483] Next, depending on the required procedure, the server uses an automated generation mechanism to create the necessary documents. Based on the information provided by the user, the server inserts the appropriate data into the document template and automatically generates the documents. The generated documents are returned to the terminal, where the user can review and approve them.

[0484] Furthermore, when a user asks a question to the system, the terminal sends that question to the server. The server uses an interactive response mechanism to refer to past FAQ databases and support history to generate an appropriate answer. This answer is then communicated to the user via the terminal, resolving any concerns or questions the user may have.

[0485] Furthermore, all communications are encrypted using security management measures, and the server verifies and manages the security of each session to prevent unauthorized access from external sources. This ensures a high level of protection for personal information and procedural data.

[0486] Through this series of operations, this system provides a safe and efficient user experience, creating an environment where users can complete necessary procedures and inquiries 24 hours a day.

[0487] The following describes the processing flow.

[0488] Step 1:

[0489] The user brings their face close to the device's camera. The device captures the camera image and uses an algorithm to detect the face area.

[0490] Step 2:

[0491] The device extracts facial feature points and sends this data to the server. The server matches this data against facial data in its database to authenticate the user.

[0492] Step 3:

[0493] After successful authentication, the user gives voice commands to the device. The device records the voice through its microphone and converts it into a digital signal.

[0494] Step 4:

[0495] The terminal sends the converted audio information to the server, which then uses a speech recognition engine to convert the speech into text.

[0496] Step 5:

[0497] The server analyzes the user's request based on the converted text data. For the necessary procedures, the server selects the appropriate document template.

[0498] Step 6:

[0499] The server inserts user information into a selected template and automatically generates the necessary documents. The generated documents are sent to the terminal for user confirmation.

[0500] Step 7:

[0501] The user enters a question using the device's chat function. The device then sends that message to the server.

[0502] Step 8:

[0503] The server analyzes the received message and searches the database. Based on past FAQs and support history, it generates an appropriate response.

[0504] Step 9:

[0505] The server sends the generated response to the terminal, and the terminal displays the answer to the user. If the user has any further questions, the process is repeated.

[0506] In this way, the entire system works together to provide users with efficient and secure procedures.

[0507] (Example 1)

[0508] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0509] The problem that this invention aims to solve is to provide a system that allows users to perform various procedures and make inquiries safely and efficiently, 24 hours a day. Conventional systems have security issues in authentication and data transmission, and are not sufficiently convenient because users have to perform manual operations when performing procedures or making inquiries. Against this backdrop, there has been a need for a method that balances improved user experience with security.

[0510] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0511] In this invention, the server includes authentication means for obtaining the user's biometric authentication information using a face detection device, acoustic analysis means for converting acoustic input into symbolic data, and document formation means for generating a document based on the information received from the user. This makes it possible for the user to securely access the system using biometric authentication, easily give instructions via voice input, and quickly generate the necessary documents.

[0512] A "face detection device" is a device used to identify a user's face and to obtain biometric authentication information.

[0513] "Authentication method" refers to a method for verifying the user's identity based on acquired biometric authentication information.

[0514] "Audio input" refers to information based on the voice or sound emitted by the user, which is then converted into symbolic data and processed.

[0515] "Acoustic analysis means" refers to the process of converting collected acoustic input into text data.

[0516] "Symbolic data" refers to text and other character information obtained by converting audio or sound information.

[0517] "Document formation means" refers to a process or device that creates necessary documents based on user instructions and information.

[0518] "Encryption techniques" are technologies that make information invisible in order to enhance data security.

[0519] "Protection and management measures" refer to mechanisms designed to ensure the security of data communications and prevent the unauthorized acquisition or alteration of information.

[0520] A "flexible computing environment" is a computing environment that is designed to allow users to access services at any time.

[0521] An "information acquisition device" is a system or device that refers to past documents and data to provide appropriate information in response to current inquiries.

[0522] The present invention provides a system that allows users to perform procedures and make inquiries securely and efficiently using biometric authentication, voice input, and automated generation technologies. This system includes a face detection device, acoustic analysis means, a document formation device, a dialogue response device, and protection and management means using encryption technology. This section describes specific embodiments of the system.

[0523] Hardware and software configuration:

[0524] 1. Terminal:

[0525] It is equipped with a high-resolution camera to capture the user's face. Face recognition uses a face detection device and a biometric authentication algorithm.

[0526] It has a built-in microphone to collect user voice. Advanced acoustic analysis software is used for speech recognition.

[0527] 2. Server:

[0528] It will have a database to store user information. The database will use an SQL-based system.

[0529] The speech recognition system incorporates an acoustic analysis engine to convert speech into text, utilizing services like Google Speech-to-Text.

[0530] Template-based automated document generation software is used for document creation.

[0531] For dialogue responses, we use generative AI models and leverage response generation engines like ChatGPT.

[0532] Specific example:

[0533] When a user uses the system, the terminal detects their face and verifies the user's identity through authentication methods.

[0534] In the voice command system, when a user says to the terminal, "I want to check my electricity bill," the acoustic analysis device converts this voice into text data and sends it to the server.

[0535] The server uses a document formatting mechanism to insert the latest billing information into the invoice template and automatically generates an invoice in PDF format.

[0536] If necessary, when a user asks a question to the terminal, such as "I want to change the billing address," the conversational response system refers to the FAQ database and past inquiry records and generates an answer.

[0537] As an example of a prompt, by inputting a request such as "Tell me how to check my new invoice" into the AI ​​model, the server can provide appropriate information based on the user's inquiry.

[0538] As described above, this system combines biometric authentication and voice input technology to provide a fast and secure user experience.

[0539] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0540] Step 1:

[0541] The device uses its camera to capture the user's face in order to authenticate the user. The input is the user's face image, and the output is the user's identification information. The facial recognition software within the system uses a biometric authentication algorithm to extract identifying features from the face image and sends them to the server.

[0542] Step 2:

[0543] The server receives identification information and verifies it against the database. The input is the identification information received from the terminal, and the output is whether authentication was successful or not. The server searches the database using SQL queries and returns the authentication result to the terminal.

[0544] Step 3:

[0545] After authentication is complete, the user gives instructions to the system by voice. The input is the user's voice instructions, and the output is text data. The terminal uses a microphone to collect the voice data, packets this data, and sends it to the server.

[0546] Step 4:

[0547] The server uses an acoustic analysis engine to convert audio data into text. The input is audio data sent from the terminal, and the output is the converted text data. The server runs acoustic analysis software, extracts text from the audio data, and saves the analysis results.

[0548] Step 5:

[0549] Based on user instructions, the server automatically generates the necessary documents. The input is converted text data, and the output is the generated document. The server, referencing user information, uses a template engine to insert appropriate data into the document template and outputs the completed document in PDF format.

[0550] Step 6:

[0551] When a user asks a question within the system, the terminal sends that question to the server. The input is the user's question in string format, and the output is the answer generated by the server. The server activates the conversational response system, uses a generative AI model to generate an answer that matches the question, and returns it to the terminal.

[0552] Step 7:

[0553] The terminal displays information returned by the server to the user. Input is the document or response data received from the server, and output is information that the user can visually verify. The terminal uses an appropriate interface to present information to the user, enabling the user to complete the procedure.

[0554] (Application Example 1)

[0555] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0556] Conventional systems had cumbersome user authentication procedures within facilities, and because various authentication processes were manual, they were inefficient and raised security concerns. Furthermore, there was the challenge of maintaining facility security while ensuring visitor convenience.

[0557] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0558] In this invention, the server includes biometric authentication means for collecting user identification information, language processing means for converting acoustic information into text data, and dynamic response means for receiving data from a communication device in real time and generating a response based on that data. This enables users to access the facility quickly and securely, and also improves operational efficiency.

[0559] "Biometric authentication methods" are technologies used to authenticate individuals using their biometric information, and they perform individual identification using methods such as facial recognition, fingerprints, and iris recognition.

[0560] "Language processing means" refers to technologies that process, analyze, and convert natural language information such as speech and text, enabling functions such as speech recognition, translation, and summarization.

[0561] An "information generation means" is a system that automatically creates necessary data and documents based on the user's requests and environment.

[0562] A "dynamic response means" is a technology that receives data in real time and immediately generates an appropriate response in response to the received information.

[0563] A "data management system" is a system that manages data storage and communication using methods such as encryption to ensure the safety and accuracy of the data.

[0564] An "identification management system" is a mechanism for verifying and controlling access rights and authentication status for specific users.

[0565] An "on-demand computing environment" is a system that dynamically supplies the necessary computing resources according to the user's requests and executes processing efficiently.

[0566] This system is designed to allow users to complete authentication procedures smoothly and securely within the facility.

[0567] The server first uses biometric authentication to acquire a facial image of the user using the camera built into the device, such as a smartphone or tablet, and then performs authentication based on this image. OpenCV and facial recognition APIs are used for image processing to ensure reliable identity verification.

[0568] Next, the language processing system acquires acoustic information. It collects the user's voice instructions through the device's microphone and converts them into text data using the Google Cloud Speech-to-Text API to analyze what the user's request is.

[0569] The server uses dynamic response mechanisms to instantly generate responses based on user requests. Historical data is stored in MongoDB, and an AI chatbot utilizing natural language processing provides appropriate answers.

[0570] As a data management method, SSL / TLS technology is used to encrypt communications and enhance security. This ensures that users' personal information and confidential facility data are securely protected.

[0571] Furthermore, the identification and management system checks the user's access rights in real time and controls entry to the facility based on those rights.

[0572] For example, when a business person visits a designated company building, they can quickly complete the entry process through facial recognition and voice recognition using their smartphone. In this case, they can obtain company-related information before arrival and begin work without having to wait in line at reception.

[0573] An example of a prompt to input into the generating AI model might be: "Write code to create a security assistant app for facility visitors. Include the following features: facial recognition, voice command processing, automatic generation of entry documents, inquiry handling, and encrypted communication (SSL)."

[0574] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0575] Step 1:

[0576] The device activates its camera and acquires an image of the user's face. Using this image data as input, the device uses a facial recognition API to extract and match feature points, and obtains the facial recognition result as output. The server receives this result and determines whether the user is authenticated or not.

[0577] Step 2:

[0578] The device's microphone waits for voice input and acquires audio information from the user. Using this audio data as input, the Google Cloud Speech-to-Text API is used to convert the speech into text data. This text data is sent to the server, where the request is parsed.

[0579] Step 3:

[0580] The server uses the received text data to refer to past visit history and FAQs stored in MongoDB and generates a response to the user's request. Here, a natural language processing algorithm is used to output an appropriate answer to the user's question.

[0581] Step 4:

[0582] The server sends the generated response data to the terminal, and the terminal interactively provides information to the user. In this process, it is also possible to use a speech synthesis engine to convert text into speech and deliver the answer to the user verbally.

[0583] Step 5:

[0584] The server uses SSL / TLS to encrypt all data communications and ensure security. Session management is performed at the start and end of data communication to guarantee the confidentiality of all information.

[0585] Step 6:

[0586] Based on the information obtained by the user, access to the facility is granted. At this time, the terminal displays dynamically generated admission documents to the user as needed, and the server further verifies the user's access rights and checks again whether they are authorized.

[0587] Step 7:

[0588] The terminal provides a user interface to facilitate these procedures, helping users achieve their goals with minimal steps. All user operations are logged on the server and saved as data for later analysis.

[0589] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0590] The present invention is a system that recognizes user emotions and enables the provision of services based on those emotions, and specific embodiments are described below.

[0591] This system features an emotion engine that analyzes the user's emotions from their voice and facial expressions. When a user speaks into the device, the device acquires the audio and simultaneously captures their facial expressions via the camera. This information is sent to a server, which uses the emotion engine to perform analysis. For example, it comprehensively analyzes the user's voice tone and speed, as well as facial muscle movements, to identify the user's emotions. Emotions that can be identified include joy, sadness, anger, and anxiety.

[0592] After detecting an emotion, the server uses interactive response mechanisms to generate an appropriate response tailored to the user. For example, if the user is feeling anxious, it will provide detailed information and a reassuring response. If the user is feeling happy, it can respond in a more friendly tone.

[0593] Users can receive these emotion-based responses through their devices. This allows users to receive support tailored to their emotions, resulting in a better customer experience. Furthermore, the server can record and analyze the history of emotion recognition, which can be used to improve future services and build user profiles.

[0594] This system enables service providers to respond flexibly based on user emotions. For example, in a customer support scenario, by appropriately addressing not only the content of the inquiry but also the user's emotions, it becomes possible to provide more reliable support. Thus, this invention provides a means to improve the user experience and enhance customer satisfaction in services.

[0595] The following describes the processing flow.

[0596] Step 1:

[0597] The user faces the device's camera and speaks into the microphone. The device simultaneously captures the camera video and audio.

[0598] Step 2:

[0599] The device converts the captured audio data into a digital signal and extracts feature points from the facial video data.

[0600] Step 3:

[0601] The device sends the converted audio data and facial feature data to the server. The server receives this data and begins analysis using its emotion engine.

[0602] Step 4:

[0603] The server analyzes parameters such as tone, speed, and emphasis in the speech, and estimates emotions by detecting muscle movements from facial expressions. This process typically utilizes known machine learning models.

[0604] Step 5:

[0605] Once the emotion engine identifies an emotion, the server uses interactive response mechanisms to generate a response appropriate to that emotion. For example, if the server determines that the user is feeling stressed, it will prepare a response that offers encouragement and reassurance.

[0606] Step 6:

[0607] The server sends the generated response to the terminal, which then displays or reads it aloud to the user. This allows the user to receive emotionally empathetic support.

[0608] Step 7:

[0609] The server records the results of emotion analysis and the history of responses with users, accumulating data to help improve future services.

[0610] This series of processes enables the system to provide users with appropriate responses that take their emotions into account in real time, thereby improving the quality of service.

[0611] (Example 2)

[0612] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0613] In modern digital services, accurately understanding and appropriately responding to users' emotions is crucial. However, existing systems do not fully utilize users' biometric information, resulting in insufficient accuracy and appropriateness in emotion-based responses. This leads to decreased user satisfaction and a failure to improve service quality.

[0614] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0615] In this invention, the server includes an analysis means for analyzing biometric information and identifying emotions, a generation means for generating an appropriate response based on the identified emotions, and a presentation means for displaying or playing the generated response to the user. This enables flexible and accurate responses based on emotions.

[0616] "Analysis means" refers to a technological device that analyzes biometric information obtained from a user and identifies states such as emotions.

[0617] A "generation means" is a technical device that creates a response to the user based on emotions identified by an analysis means.

[0618] "Presentation means" refers to a technical device for providing the generated response to the user visually or audibly.

[0619] "Protective measures" refer to technical devices used to ensure security in data communications.

[0620] A "recording means" is a technological device that stores the results of emotion identification and response history in a storage device, making them available for subsequent analysis and service improvement.

[0621] This system is designed to utilize the user's biometric information to provide flexible, emotion-based responses. Specifically, the terminal and server work together to analyze the user's voice and facial expressions to provide appropriate services.

[0622] Device functions and roles

[0623] The device first captures the user's voice using a microphone. Simultaneously, it captures the user's facial expressions using a camera built into the device. For example, using the latest model's voice microphone and high-resolution camera allows for accurate data collection.

[0624] Server analysis and generation function

[0625] The data acquired by the device is sent to the server. The server has analytical means to analyze biometric information, processing voice data and facial expression data to identify the user's emotions. The server inputs information about changes in acoustic characteristics and facial muscle movements into a generating AI model to perform emotion identification.

[0626] The server then uses a generation mechanism to generate a response that corresponds to the identified emotion. Typical response generation utilizes a generation AI model and uses appropriate prompts based on the user's emotion. An example of a prompt might be, "If the user is feeling anxious, generate a response that provides detailed information to reassure them."

[0627] Presentation of responses and recording of history

[0628] The generated response is sent back from the server to the terminal, which then uses its display and speakers to provide the response to the user. This allows the user to receive appropriate support that matches their emotions.

[0629] Furthermore, the server uses recording mechanisms to save the sentiment identification results and response history. This history can be used to improve future services and build new user profiles. For example, by referring to past sentiment data, it is possible to analyze user behavior and provide more personalized services.

[0630] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0631] Step 1:

[0632] The user enters information.

[0633] When a user begins speaking into the device, it activates its built-in microphone to acquire audio data. Simultaneously, it activates the camera to capture the user's facial expressions. The input data consists of audio signals and image data. Based on this, the device captures audio and video information.

[0634] Step 2:

[0635] The device sends data to the server.

[0636] The terminal sends the captured audio data (audio signal conversion data) and facial expression data (image data) to the server as a data package. The output at this stage is the data package received by the server. Since secure communication methods are used for transmission, the process includes the execution of an encryption protocol.

[0637] Step 3:

[0638] The server analyzes the biometric information.

[0639] The server inputs the received data package into an analysis device and uses a generative AI model to analyze the voice and facial expressions. The server extracts features such as tone and pitch from the voice data and analyzes the movement of facial muscles from the image data. The output of this analysis is identified emotional information. For example, it comprehensively evaluates the intonation of the voice and changes in facial expressions to recognize the user's emotions.

[0640] Step 4:

[0641] The server generates a response.

[0642] Based on the identified sentiment information, the server generates prompts using a generation mechanism and uses a generation AI model to produce responses. A specific prompt might be, "If the user is happy, generate a more friendly response." The output at this stage is either text or audio data for the user to receive.

[0643] Step 5:

[0644] The server sends a response to the terminal.

[0645] The server sends the generated response data back to the terminal. This communication is also encrypted using a secure protocol. The receiving terminal then displays or plays the response data to the user as audio or text.

[0646] Step 6:

[0647] The server records history.

[0648] The server records the analyzed sentiment information and response content in a database. This recorded data will be used for future user characteristic analysis and service improvement. This includes the operation of properly saving data to the database.

[0649] (Application Example 2)

[0650] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0651] Modern service delivery demands flexible and personalized responses based on user emotions. However, conventional systems struggle to accurately recognize emotions from users' facial expressions and voices and generate immediate and appropriate responses accordingly. This results in a lack of effective means to improve user satisfaction and alleviate anxiety. In particular, there is a need for the development of systems that provide information and interactive responses tailored to user emotions.

[0652] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0653] In this invention, the server includes emotion detection means for analyzing the user's emotions, information provision means for providing optimal information, and security management means for ensuring the security of data communication. This enables the generation of appropriate responses based on the user's emotions and the provision of information safely and effectively.

[0654] "User identification information" refers to information used by a system to recognize and authenticate a specific user.

[0655] "Facial recognition technology" refers to technology used to identify individuals based on their facial features.

[0656] "Acoustic information" refers to information perceived through hearing, including human speech and ambient sounds.

[0657] "Speech recognition means" refers to technology for converting acoustic information into text data.

[0658] "Emotion detection means" refers to technology for identifying a user's emotional state from their voice and facial expressions.

[0659] "Automatic generation means" refers to technology for automatically creating necessary documents and data in response to user requests.

[0660] "Interactive response methods" are technologies for responding to user questions in a natural, conversational format.

[0661] "Information provision means" refers to technologies for searching for and presenting appropriate information in response to user requests.

[0662] "Security management measures" refer to encryption technologies and other protective measures to ensure the security of data communications.

[0663] An "intertabible computing environment" is the arrangement of computing resources to provide users with appropriate functions and services based on their authentication.

[0664] "Past data" refers to a collection of recorded information accumulated based on user requests and behavioral history.

[0665] This invention is a system that provides flexible and personalized services based on the user's emotions. The system consists of the following elements:

[0666] First, the user's device uses its camera and microphone to capture facial expressions and audio information. This information is collected in real time and sent to the server. Image processing libraries such as OpenCV are used for the camera, and the Google Speech-to-Text API is used for processing the audio information.

[0667] The server analyzes emotions based on the received data using emotion detection means. These emotion detection means estimate emotions using models built with machine learning libraries such as Keras and TensorFlow. Based on the analyzed emotions, information provision means are utilized to provide optimal information. Interactive response means generate natural conversational responses in response to prompts.

[0668] Security management measures include applying encryption technology to data communications to ensure data security and protect user information.

[0669] As a concrete example, suppose a user says, "I'm worried about security." In this case, the emotion detection system senses the user's anxiety, the information provision system presents detailed security measures, and the interactive response system provides appropriate advice in voice. The prompt used in this example is, "When a user is feeling anxious, please suggest how to provide them with the latest security information as a concrete example."

[0670] In this way, the system provides sophisticated services tailored to the user's emotions, improving the user experience.

[0671] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0672] Step 1:

[0673] The device activates its camera and microphone to capture the user's facial expressions and voice data. This allows for real-time capture of the user's facial expressions and voice input.

[0674] Step 2:

[0675] The device converts captured audio data into text data using speech recognition technology. Using the Google Speech-to-Text API, this text data is sent to a server, providing the user's spoken content in an analyzable format.

[0676] Step 3:

[0677] The server receives facial expression data and speech-to-text data sent from the terminal and performs emotion analysis using emotion detection methods. Using Keras or TensorFlow, it analyzes speech tone, speaking speed, and facial expression patterns to identify the user's emotional state. This analysis determines whether the user is experiencing a specific emotion such as joy or anxiety.

[0678] Step 4:

[0679] Based on the identified emotion, the server collects data using informational tools to generate the optimal response. For example, if the emotion of anxiety is identified, it searches the database for and prepares information regarding security improvements.

[0680] Step 5:

[0681] The server generates prompts using an interactive response mechanism based on the collected data, and creates natural-sounding conversational responses using a generative AI model. The prompt used here is, "When a user is feeling anxious, please suggest how to provide them with the latest security information as a concrete example."

[0682] Step 6:

[0683] The server encrypts the response it generates using security management measures and sends it to the terminal. This enables the provision of safe and personalized information tailored to each user.

[0684] Step 7:

[0685] The terminal decrypts the encrypted response received from the server and presents it to the user in audio or text format. This allows the user to receive advice with peace of mind.

[0686] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0687] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0688] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0689] [Fourth Embodiment]

[0690] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0691] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0692] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0693] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0694] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0695] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0696] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0697] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0698] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0699] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0700] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0701] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0702] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0703] The present invention is a highly secure system that allows users to perform procedures and make inquiries at any time, and has the following configuration as a specific embodiment.

[0704] When a user uses the system, the terminal first detects the user's face and obtains the user's identification information using facial recognition. The terminal then sends this identification information to the server, which performs authentication by referring to a database. This ensures accurate authentication while securely protecting the user's personal information.

[0705] After successful registration, the user can give voice commands to the device. The device collects the user's audio information through its microphone and sends it to the server. The server uses speech recognition to convert this into text data and understand the user's requests. Through this process, the device can be operated even when the user's hands are occupied.

[0706] Next, depending on the required procedure, the server uses an automated generation mechanism to create the necessary documents. Based on the information provided by the user, the server inserts the appropriate data into the document template and automatically generates the documents. The generated documents are returned to the terminal, where the user can review and approve them.

[0707] Furthermore, when a user asks a question to the system, the terminal sends that question to the server. The server uses an interactive response mechanism to refer to past FAQ databases and support history to generate an appropriate answer. This answer is then communicated to the user via the terminal, resolving any concerns or questions the user may have.

[0708] Furthermore, all communications are encrypted using security management measures, and the server verifies and manages the security of each session to prevent unauthorized access from external sources. This ensures a high level of protection for personal information and procedural data.

[0709] Through this series of operations, this system provides a safe and efficient user experience, creating an environment where users can complete necessary procedures and inquiries 24 hours a day.

[0710] The following describes the processing flow.

[0711] Step 1:

[0712] The user brings their face close to the device's camera. The device captures the camera image and uses an algorithm to detect the face area.

[0713] Step 2:

[0714] The device extracts facial feature points and sends this data to the server. The server matches this data against facial data in its database to authenticate the user.

[0715] Step 3:

[0716] After successful authentication, the user gives voice commands to the device. The device records the voice through its microphone and converts it into a digital signal.

[0717] Step 4:

[0718] The terminal sends the converted audio information to the server, which then uses a speech recognition engine to convert the speech into text.

[0719] Step 5:

[0720] The server analyzes the user's request based on the converted text data. For the necessary procedures, the server selects the appropriate document template.

[0721] Step 6:

[0722] The server inserts user information into a selected template and automatically generates the necessary documents. The generated documents are sent to the terminal for user confirmation.

[0723] Step 7:

[0724] The user enters a question using the device's chat function. The device then sends that message to the server.

[0725] Step 8:

[0726] The server analyzes the received message and searches the database. Based on past FAQs and support history, it generates an appropriate response.

[0727] Step 9:

[0728] The server sends the generated response to the terminal, and the terminal displays the answer to the user. If the user has any further questions, the process is repeated.

[0729] In this way, the entire system works together to provide users with efficient and secure procedures.

[0730] (Example 1)

[0731] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0732] The problem that this invention aims to solve is to provide a system that allows users to perform various procedures and make inquiries safely and efficiently, 24 hours a day. Conventional systems have security issues in authentication and data transmission, and are not sufficiently convenient because users have to perform manual operations when performing procedures or making inquiries. Against this backdrop, there has been a need for a method that balances improved user experience with security.

[0733] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0734] In this invention, the server includes authentication means for obtaining the user's biometric authentication information using a face detection device, acoustic analysis means for converting acoustic input into symbolic data, and document formation means for generating a document based on the information received from the user. This makes it possible for the user to securely access the system using biometric authentication, easily give instructions via voice input, and quickly generate the necessary documents.

[0735] A "face detection device" is a device used to identify a user's face and to obtain biometric authentication information.

[0736] "Authentication method" refers to a method for verifying the user's identity based on acquired biometric authentication information.

[0737] "Audio input" refers to information based on the voice or sound emitted by the user, which is then converted into symbolic data and processed.

[0738] "Acoustic analysis means" refers to the process of converting collected acoustic input into text data.

[0739] "Symbolic data" refers to text and other character information obtained by converting audio or sound information.

[0740] "Document formation means" refers to a process or device that creates necessary documents based on user instructions and information.

[0741] "Encryption techniques" are technologies that make information invisible in order to enhance data security.

[0742] "Protection and management measures" refer to mechanisms designed to ensure the security of data communications and prevent the unauthorized acquisition or alteration of information.

[0743] A "flexible computing environment" is a computing environment that is designed to allow users to access services at any time.

[0744] An "information acquisition device" is a system or device that refers to past documents and data to provide appropriate information in response to current inquiries.

[0745] The present invention provides a system that allows users to perform procedures and make inquiries securely and efficiently using biometric authentication, voice input, and automated generation technologies. This system includes a face detection device, acoustic analysis means, a document formation device, a dialogue response device, and protection and management means using encryption technology. This section describes specific embodiments of the system.

[0746] Hardware and software configuration:

[0747] 1. Terminal:

[0748] It is equipped with a high-resolution camera to capture the user's face. Face recognition uses a face detection device and a biometric authentication algorithm.

[0749] It has a built-in microphone to collect user voice. Advanced acoustic analysis software is used for speech recognition.

[0750] 2. Server:

[0751] It will have a database to store user information. The database will use an SQL-based system.

[0752] The speech recognition system incorporates an acoustic analysis engine to convert speech into text, utilizing services like Google Speech-to-Text.

[0753] Template-based automated document generation software is used for document creation.

[0754] For dialogue responses, we use generative AI models and leverage response generation engines like ChatGPT.

[0755] Specific example:

[0756] When a user uses the system, the terminal detects their face and verifies the user's identity through authentication methods.

[0757] In the voice command system, when a user says to the terminal, "I want to check my electricity bill," the acoustic analysis device converts this voice into text data and sends it to the server.

[0758] The server uses a document formatting mechanism to insert the latest billing information into the invoice template and automatically generates an invoice in PDF format.

[0759] If necessary, when a user asks a question to the terminal, such as "I want to change the billing address," the conversational response system refers to the FAQ database and past inquiry records and generates an answer.

[0760] As an example of a prompt, by inputting a request such as "Tell me how to check my new invoice" into the AI ​​model, the server can provide appropriate information based on the user's inquiry.

[0761] As described above, this system combines biometric authentication and voice input technology to provide a fast and secure user experience.

[0762] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0763] Step 1:

[0764] The device uses its camera to capture the user's face in order to authenticate the user. The input is the user's face image, and the output is the user's identification information. The facial recognition software within the system uses a biometric authentication algorithm to extract identifying features from the face image and sends them to the server.

[0765] Step 2:

[0766] The server receives identification information and verifies it against the database. The input is the identification information received from the terminal, and the output is whether authentication was successful or not. The server searches the database using SQL queries and returns the authentication result to the terminal.

[0767] Step 3:

[0768] After authentication is complete, the user gives instructions to the system by voice. The input is the user's voice instructions, and the output is text data. The terminal uses a microphone to collect the voice data, packets this data, and sends it to the server.

[0769] Step 4:

[0770] The server uses an acoustic analysis engine to convert audio data into text. The input is audio data sent from the terminal, and the output is the converted text data. The server runs acoustic analysis software, extracts text from the audio data, and saves the analysis results.

[0771] Step 5:

[0772] Based on user instructions, the server automatically generates the necessary documents. The input is converted text data, and the output is the generated document. The server, referencing user information, uses a template engine to insert appropriate data into the document template and outputs the completed document in PDF format.

[0773] Step 6:

[0774] When a user asks a question within the system, the terminal sends that question to the server. The input is the user's question in string format, and the output is the answer generated by the server. The server activates the conversational response system, uses a generative AI model to generate an answer that matches the question, and returns it to the terminal.

[0775] Step 7:

[0776] The terminal displays information returned by the server to the user. Input is the document or response data received from the server, and output is information that the user can visually verify. The terminal uses an appropriate interface to present information to the user, enabling the user to complete the procedure.

[0777] (Application Example 1)

[0778] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0779] Conventional systems had cumbersome user authentication procedures within facilities, and because various authentication processes were manual, they were inefficient and raised security concerns. Furthermore, there was the challenge of maintaining facility security while ensuring visitor convenience.

[0780] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0781] In this invention, the server includes biometric authentication means for collecting user identification information, language processing means for converting acoustic information into text data, and dynamic response means for receiving data from a communication device in real time and generating a response based on that data. This enables users to access the facility quickly and securely, and also improves operational efficiency.

[0782] "Biometric authentication methods" are technologies used to authenticate individuals using their biometric information, and they perform individual identification using methods such as facial recognition, fingerprints, and iris recognition.

[0783] "Language processing means" refers to technologies that process, analyze, and convert natural language information such as speech and text, enabling functions such as speech recognition, translation, and summarization.

[0784] An "information generation means" is a system that automatically creates necessary data and documents based on the user's requests and environment.

[0785] A "dynamic response means" is a technology that receives data in real time and immediately generates an appropriate response in response to the received information.

[0786] A "data management system" is a system that manages data storage and communication using methods such as encryption to ensure the safety and accuracy of the data.

[0787] An "identification management system" is a mechanism for verifying and controlling access rights and authentication status for specific users.

[0788] An "on-demand computing environment" is a system that dynamically supplies the necessary computing resources according to the user's requests and executes processing efficiently.

[0789] This system is designed to allow users to complete authentication procedures smoothly and securely within the facility.

[0790] The server first uses biometric authentication to acquire a facial image of the user using the camera built into the device, such as a smartphone or tablet, and then performs authentication based on this image. OpenCV and facial recognition APIs are used for image processing to ensure reliable identity verification.

[0791] Next, the language processing system acquires acoustic information. It collects the user's voice instructions through the device's microphone and converts them into text data using the Google Cloud Speech-to-Text API to analyze what the user's request is.

[0792] The server uses dynamic response mechanisms to instantly generate responses based on user requests. Historical data is stored in MongoDB, and an AI chatbot utilizing natural language processing provides appropriate answers.

[0793] As a data management method, SSL / TLS technology is used to encrypt communications and enhance security. This ensures that users' personal information and confidential facility data are securely protected.

[0794] Furthermore, the identification and management system checks the user's access rights in real time and controls entry to the facility based on those rights.

[0795] For example, when a business person visits a designated company building, they can quickly complete the entry process through facial recognition and voice recognition using their smartphone. In this case, they can obtain company-related information before arrival and begin work without having to wait in line at reception.

[0796] An example of a prompt to input into the generating AI model might be: "Write code to create a security assistant app for facility visitors. Include the following features: facial recognition, voice command processing, automatic generation of entry documents, inquiry handling, and encrypted communication (SSL)."

[0797] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0798] Step 1:

[0799] The device activates its camera and acquires an image of the user's face. Using this image data as input, the device uses a facial recognition API to extract and match feature points, and obtains the facial recognition result as output. The server receives this result and determines whether the user is authenticated or not.

[0800] Step 2:

[0801] The device's microphone waits for voice input and acquires audio information from the user. Using this audio data as input, the Google Cloud Speech-to-Text API is used to convert the speech into text data. This text data is sent to the server, where the request is parsed.

[0802] Step 3:

[0803] The server uses the received text data to refer to past visit history and FAQs stored in MongoDB and generates a response to the user's request. Here, a natural language processing algorithm is used to output an appropriate answer to the user's question.

[0804] Step 4:

[0805] The server sends the generated response data to the terminal, and the terminal interactively provides information to the user. In this process, it is also possible to use a speech synthesis engine to convert text into speech and deliver the answer to the user verbally.

[0806] Step 5:

[0807] The server uses SSL / TLS to encrypt all data communications and ensure security. Session management is performed at the start and end of data communication to guarantee the confidentiality of all information.

[0808] Step 6:

[0809] Based on the information obtained by the user, access to the facility is granted. At this time, the terminal displays dynamically generated admission documents to the user as needed, and the server further verifies the user's access rights and checks again whether they are authorized.

[0810] Step 7:

[0811] The terminal provides a user interface to facilitate these procedures, helping users achieve their goals with minimal steps. All user operations are logged on the server and saved as data for later analysis.

[0812] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0813] The present invention is a system that recognizes user emotions and enables the provision of services based on those emotions, and specific embodiments are described below.

[0814] This system features an emotion engine that analyzes the user's emotions from their voice and facial expressions. When a user speaks into the device, the device acquires the audio and simultaneously captures their facial expressions via the camera. This information is sent to a server, which uses the emotion engine to perform analysis. For example, it comprehensively analyzes the user's voice tone and speed, as well as facial muscle movements, to identify the user's emotions. Emotions that can be identified include joy, sadness, anger, and anxiety.

[0815] After detecting an emotion, the server uses interactive response mechanisms to generate an appropriate response tailored to the user. For example, if the user is feeling anxious, it will provide detailed information and a reassuring response. If the user is feeling happy, it can respond in a more friendly tone.

[0816] Users can receive these emotion-based responses through their devices. This allows users to receive support tailored to their emotions, resulting in a better customer experience. Furthermore, the server can record and analyze the history of emotion recognition, which can be used to improve future services and build user profiles.

[0817] This system enables service providers to respond flexibly based on user emotions. For example, in a customer support scenario, by appropriately addressing not only the content of the inquiry but also the user's emotions, it becomes possible to provide more reliable support. Thus, this invention provides a means to improve the user experience and enhance customer satisfaction in services.

[0818] The following describes the processing flow.

[0819] Step 1:

[0820] The user faces the device's camera and speaks into the microphone. The device simultaneously captures the camera video and audio.

[0821] Step 2:

[0822] The device converts the captured audio data into a digital signal and extracts feature points from the facial video data.

[0823] Step 3:

[0824] The device sends the converted audio data and facial feature data to the server. The server receives this data and begins analysis using its emotion engine.

[0825] Step 4:

[0826] The server analyzes parameters such as tone, speed, and emphasis in the speech, and estimates emotions by detecting muscle movements from facial expressions. This process typically utilizes known machine learning models.

[0827] Step 5:

[0828] Once the emotion engine identifies an emotion, the server uses interactive response mechanisms to generate a response appropriate to that emotion. For example, if the server determines that the user is feeling stressed, it will prepare a response that offers encouragement and reassurance.

[0829] Step 6:

[0830] The server sends the generated response to the terminal, which then displays or reads it aloud to the user. This allows the user to receive emotionally empathetic support.

[0831] Step 7:

[0832] The server records the results of emotion analysis and the history of responses with users, accumulating data to help improve future services.

[0833] This series of processes enables the system to provide users with appropriate responses that take their emotions into account in real time, thereby improving the quality of service.

[0834] (Example 2)

[0835] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0836] In modern digital services, accurately understanding and appropriately responding to users' emotions is crucial. However, existing systems do not fully utilize users' biometric information, resulting in insufficient accuracy and appropriateness in emotion-based responses. This leads to decreased user satisfaction and a failure to improve service quality.

[0837] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0838] In this invention, the server includes an analysis means for analyzing biometric information and identifying emotions, a generation means for generating an appropriate response based on the identified emotions, and a presentation means for displaying or playing the generated response to the user. This enables flexible and accurate responses based on emotions.

[0839] "Analysis means" refers to a technological device that analyzes biometric information obtained from a user and identifies states such as emotions.

[0840] A "generation means" is a technical device that creates a response to the user based on emotions identified by an analysis means.

[0841] "Presentation means" refers to a technical device for providing the generated response to the user visually or audibly.

[0842] "Protective measures" refer to technical devices used to ensure security in data communications.

[0843] A "recording means" is a technological device that stores the results of emotion identification and response history in a storage device, making them available for subsequent analysis and service improvement.

[0844] This system is designed to utilize the user's biometric information to provide flexible, emotion-based responses. Specifically, the terminal and server work together to analyze the user's voice and facial expressions to provide appropriate services.

[0845] Device functions and roles

[0846] The device first captures the user's voice using a microphone. Simultaneously, it captures the user's facial expressions using a camera built into the device. For example, using the latest model's voice microphone and high-resolution camera allows for accurate data collection.

[0847] Server analysis and generation function

[0848] The data acquired by the device is sent to the server. The server has analytical means to analyze biometric information, processing voice data and facial expression data to identify the user's emotions. The server inputs information about changes in acoustic characteristics and facial muscle movements into a generating AI model to perform emotion identification.

[0849] The server then uses a generation mechanism to generate a response that corresponds to the identified emotion. Typical response generation utilizes a generation AI model and uses appropriate prompts based on the user's emotion. An example of a prompt might be, "If the user is feeling anxious, generate a response that provides detailed information to reassure them."

[0850] Presentation of responses and recording of history

[0851] The generated response is sent back from the server to the terminal, which then uses its display and speakers to provide the response to the user. This allows the user to receive appropriate support that matches their emotions.

[0852] Furthermore, the server uses recording mechanisms to save the sentiment identification results and response history. This history can be used to improve future services and build new user profiles. For example, by referring to past sentiment data, it is possible to analyze user behavior and provide more personalized services.

[0853] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0854] Step 1:

[0855] The user enters information.

[0856] When a user begins speaking into the device, it activates its built-in microphone to acquire audio data. Simultaneously, it activates the camera to capture the user's facial expressions. The input data consists of audio signals and image data. Based on this, the device captures audio and video information.

[0857] Step 2:

[0858] The device sends data to the server.

[0859] The terminal sends the captured audio data (audio signal conversion data) and facial expression data (image data) to the server as a data package. The output at this stage is the data package received by the server. Since secure communication methods are used for transmission, the process includes the execution of an encryption protocol.

[0860] Step 3:

[0861] The server analyzes the biometric information.

[0862] The server inputs the received data package into an analysis device and uses a generative AI model to analyze the voice and facial expressions. The server extracts features such as tone and pitch from the voice data and analyzes the movement of facial muscles from the image data. The output of this analysis is identified emotional information. For example, it comprehensively evaluates the intonation of the voice and changes in facial expressions to recognize the user's emotions.

[0863] Step 4:

[0864] The server generates a response.

[0865] Based on the identified sentiment information, the server generates prompts using a generation mechanism and uses a generation AI model to produce responses. A specific prompt might be, "If the user is happy, generate a more friendly response." The output at this stage is either text or audio data for the user to receive.

[0866] Step 5:

[0867] The server sends a response to the terminal.

[0868] The server sends the generated response data back to the terminal. This communication is also encrypted using a secure protocol. The receiving terminal then displays or plays the response data to the user as audio or text.

[0869] Step 6:

[0870] The server records history.

[0871] The server records the analyzed sentiment information and response content in a database. This recorded data will be used for future user characteristic analysis and service improvement. This includes the operation of properly saving data to the database.

[0872] (Application Example 2)

[0873] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0874] Modern service delivery demands flexible and personalized responses based on user emotions. However, conventional systems struggle to accurately recognize emotions from users' facial expressions and voices and generate immediate and appropriate responses accordingly. This results in a lack of effective means to improve user satisfaction and alleviate anxiety. In particular, there is a need for the development of systems that provide information and interactive responses tailored to user emotions.

[0875] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0876] In this invention, the server includes emotion detection means for analyzing the user's emotions, information provision means for providing optimal information, and security management means for ensuring the security of data communication. This enables the generation of appropriate responses based on the user's emotions and the provision of information safely and effectively.

[0877] "User identification information" refers to information used by a system to recognize and authenticate a specific user.

[0878] "Facial recognition technology" refers to technology used to identify individuals based on their facial features.

[0879] "Acoustic information" refers to information perceived through hearing, including human speech and ambient sounds.

[0880] "Speech recognition means" refers to technology for converting acoustic information into text data.

[0881] "Emotion detection means" refers to technology for identifying a user's emotional state from their voice and facial expressions.

[0882] "Automatic generation means" refers to technology for automatically creating necessary documents and data in response to user requests.

[0883] "Interactive response methods" are technologies for responding to user questions in a natural, conversational format.

[0884] "Information provision means" refers to technologies for searching for and presenting appropriate information in response to user requests.

[0885] "Security management measures" refer to encryption technologies and other protective measures to ensure the security of data communications.

[0886] An "intertabible computing environment" is the arrangement of computing resources to provide users with appropriate functions and services based on their authentication.

[0887] "Past data" refers to a collection of recorded information accumulated based on user requests and behavioral history.

[0888] This invention is a system that provides flexible and personalized services based on the user's emotions. The system consists of the following elements:

[0889] First, the user's device uses its camera and microphone to capture facial expressions and audio information. This information is collected in real time and sent to the server. Image processing libraries such as OpenCV are used for the camera, and the Google Speech-to-Text API is used for processing the audio information.

[0890] The server analyzes emotions based on the received data using emotion detection means. These emotion detection means estimate emotions using models built with machine learning libraries such as Keras and TensorFlow. Based on the analyzed emotions, information provision means are utilized to provide optimal information. Interactive response means generate natural conversational responses in response to prompts.

[0891] Security management measures include applying encryption technology to data communications to ensure data security and protect user information.

[0892] As a concrete example, suppose a user says, "I'm worried about security." In this case, the emotion detection system senses the user's anxiety, the information provision system presents detailed security measures, and the interactive response system provides appropriate advice in voice. The prompt used in this example is, "When a user is feeling anxious, please suggest how to provide them with the latest security information as a concrete example."

[0893] In this way, the system provides sophisticated services tailored to the user's emotions, improving the user experience.

[0894] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0895] Step 1:

[0896] The device activates its camera and microphone to capture the user's facial expressions and voice data. This allows for real-time capture of the user's facial expressions and voice input.

[0897] Step 2:

[0898] The device converts captured audio data into text data using speech recognition technology. Using the Google Speech-to-Text API, this text data is sent to a server, providing the user's spoken content in an analyzable format.

[0899] Step 3:

[0900] The server receives facial expression data and speech-to-text data sent from the terminal and performs emotion analysis using emotion detection methods. Using Keras or TensorFlow, it analyzes speech tone, speaking speed, and facial expression patterns to identify the user's emotional state. This analysis determines whether the user is experiencing a specific emotion such as joy or anxiety.

[0901] Step 4:

[0902] Based on the identified emotion, the server collects data using informational tools to generate the optimal response. For example, if the emotion of anxiety is identified, it searches the database for and prepares information regarding security improvements.

[0903] Step 5:

[0904] The server generates prompts using an interactive response mechanism based on the collected data, and creates natural-sounding conversational responses using a generative AI model. The prompt used here is, "When a user is feeling anxious, please suggest how to provide them with the latest security information as a concrete example."

[0905] Step 6:

[0906] The server encrypts the response it generates using security management measures and sends it to the terminal. This enables the provision of safe and personalized information tailored to each user.

[0907] Step 7:

[0908] The terminal decrypts the encrypted response received from the server and presents it to the user in audio or text format. This allows the user to receive advice with peace of mind.

[0909] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0910] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0911] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0912] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0913] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0914] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0915] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0916] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0917] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0918] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0919] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0920] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0921] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0922] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0923] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0924] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0925] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0926] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0927] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0928] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0929] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0930] The following is further disclosed regarding the embodiments described above.

[0931] (Claim 1)

[0932] A facial recognition method for collecting user identification information,

[0933] A speech recognition means for converting acoustic information into text data,

[0934] An automated generation method for generating necessary documents for users,

[0935] An interactive response system that enables question-based responses,

[0936] Security management measures for ensuring the security of data communications using encryption technology,

[0937] A system that includes this.

[0938] (Claim 2)

[0939] The system according to claim 1, comprising an intertabulated computing environment for supplying the provided functions to authenticated users at any given time.

[0940] (Claim 3)

[0941] The system according to claim 1, comprising an information retrieval means that refers to past data in response to the user's inquiry and provides the most appropriate information.

[0942] "Example 1"

[0943] (Claim 1)

[0944] An authentication means for obtaining the user's biometric authentication information using a facial detection device,

[0945] An acoustic analysis means for converting acoustic input into symbolic data,

[0946] A document formation means for generating a document based on information received from a user,

[0947] A dialogue response device for creating responses to inquiries,

[0948] A protection management means for protecting information transmission using encryption methods,

[0949] A system that includes this.

[0950] (Claim 2)

[0951] The system according to claim 1, comprising a flexible computing environment for providing functions to authenticated users within any given time.

[0952] (Claim 3)

[0953] The system according to claim 1, comprising an information acquisition device for referring to past materials that match the content of the user's question and providing appropriate information.

[0954] "Application Example 1"

[0955] (Claim 1)

[0956] A biometric authentication method for collecting user identification information,

[0957] A language processing means for converting acoustic information into text data,

[0958] Information generation means for generating necessary information for users,

[0959] A dynamic response means that receives data in real time from a communication device and generates a response based on that data,

[0960] A data management method for ensuring the security of data communications using encryption technology,

[0961] An identification management means for managing access rights within an environment authorized to authenticated users,

[0962] A system that includes this.

[0963] (Claim 2)

[0964] The system according to claim 1, comprising an on-demand computing environment for supplying the provided functions to authenticated users at any given time.

[0965] (Claim 3)

[0966] The system according to claim 1, comprising an information provision means that refers to past historical data in response to the user's inquiry and provides the most appropriate information.

[0967] "Example 2 of combining an emotion engine"

[0968] (Claim 1)

[0969] An analytical means for analyzing the user's biometric information and identifying emotions,

[0970] A generation means for generating an appropriate response based on identified emotions,

[0971] A presentation means for displaying or playing back the generated response to the user,

[0972] Protective measures to ensure the security of data communications,

[0973] A recording method for recording and analyzing the emotional history of users,

[0974] A system that includes this.

[0975] (Claim 2)

[0976] The system according to claim 1, comprising a computing environment for supplying the provided functions to authenticated users at any given time.

[0977] (Claim 3)

[0978] The system according to claim 1, comprising a search means that refers to past data in accordance with the user's inquiry and identified emotions, and provides optimal information.

[0979] "Application example 2 when combining with an emotional engine"

[0980] (Claim 1)

[0981] A facial recognition method for collecting user identification information,

[0982] A speech recognition means for converting acoustic information into text data,

[0983] An emotion detection method for analyzing the user's emotions,

[0984] An automated generation method for generating necessary documents for users,

[0985] An interactive response system that enables question-based responses,

[0986] Information provision means for providing optimal information,

[0987] Security management measures to ensure the safety of data communications,

[0988] A system that includes this.

[0989] (Claim 2)

[0990] The system according to claim 1, comprising an interactable computing environment for supplying the provided functions to authenticated users at any given time and for generating responses corresponding to the user's emotional state.

[0991] (Claim 3)

[0992] The system according to claim 1, comprising an information retrieval means that refers to past data based on the user's inquiry and detected emotional state, and provides optimal information. [Explanation of symbols]

[0993] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A facial recognition method for collecting user identification information, A speech recognition means for converting acoustic information into text data, An automated generation method for generating necessary documents for users, An interactive response system that enables question-based responses, Security management measures for ensuring the security of data communications using encryption technology, A system that includes this.

2. The system according to claim 1, comprising an intertabible computing environment for supplying the provided functions to authenticated users at any given time.

3. The system according to claim 1, comprising an information retrieval means that refers to past data in response to the user's inquiry and provides the most appropriate information.

Images