System

Abstract

A system is provided.SOLUTION: A system for generating and displaying a facial expression for a subject whose facial expression is restricted due to a disorder of specific nerves or muscles, the system comprising: means for uploading and storing facial photograph of the subject; means for learning a AI model based on the uploaded facial photograph; means for capturing an image in real time based on an emotion selected by the subject and generating a facial expression corresponding to the emotion by the AI model; and means for displaying the generated facial expression on a tablet or AR glasses.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia have difficulty expressing their emotions appropriately, which can lead to misunderstandings and feelings of loneliness. It is hoped that a solution can be found to resolve communication barriers between such patients and those around them, thereby improving their quality of life (QoL). [Means for solving the problem]

[0005] The present invention provides the following means for patients whose facial expressions are limited due to specific nerve or muscle disorders. First, it provides a means for uploading and saving the patient's facial photo data. Second, it provides a means for an AI model to learn based on the uploaded facial photo data. It also provides a means for capturing images in real time based on an emotion selected by the patient and generating an expression corresponding to that emotion using an AI model. It also provides a means for displaying the generated expressions on a tablet or AR glasses, improving communication between the patient and others and featuring a feedback function for visually conveying the patient's emotions. It also provides a means for the AI ​​model to re-train based on feedback to improve the accuracy of the generated expressions. This enables patients to express their emotions appropriately and achieve smooth communication with those around them.

[0006] "Patient" refers to a person whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia.

[0007] "Facial photo data" refers to digital photographs of patients' faces that are used to train AI models.

[0008] "Upload" refers to the operation of sending a patient's facial photo data from the terminal to the server.

[0009] An "AI model" is a computational model that uses artificial intelligence technology to analyze facial photo data and generate facial expressions that correspond to selected emotions.

[0010] "Emotion" refers to the specific psychological state that the patient wants to express, such as feelings of "happy," "sad," or "joyful."

[0011] "Real-time image" refers to image data captured through a camera at the current time.

[0012] "Capture" refers to the act of obtaining image data using a camera.

[0013] "Generation" refers to the operation in which the AI ​​model creates new facial expression images based on the analysis results.

[0014] "Facial expression" refers to emotions or moods visually expressed through the movement and position of facial muscles.

[0015] A "tablet" is a flat mobile information terminal that allows touch operation on the screen.

[0016] "AR glasses" are eyeglass-type devices that use augmented reality technology and can overlay digital information onto real-world scenery.

[0017] "Display" refers to the operation of visually displaying the generated facial expression image on the screen of a tablet or AR glasses.

[0018] "Feedback" refers to the operation of providing opinions from the patient or caregiver to evaluate and instruct corrections to the generated facial expression.

[0019] "Retraining" refers to the process by which an AI model repeats the learning process using new data based on feedback. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to express their emotions appropriately. This system uses an AI model to learn from photographic data of the patient's face, generates facial expressions corresponding to the patient's emotions in real time, and displays them on a tablet or AR glasses, thereby supporting communication between the patient and others.

[0042] Overall system configuration

[0043] Uploading and saving face photo data

[0044] 1. The user launches the application and captures a facial photo.

[0045] Click the "Upload Face Photo" button on the application.

[0046] Open your device's photo library and select a photo of your face.

[0047] 2. The terminal compresses the selected facial photo data and sends it to the server.

[0048] Send the face photo data to the server using an HTTP POST request.

[0049] 3. The server stores the received facial photo data in its database.

[0050] The image data is converted into an appropriate format and saved.

[0051] Training an AI model

[0052] 1. The server trains the AI ​​model based on the stored facial photo data.

[0053] Feature points are extracted from facial photographs and various facial expressions corresponding to emotions are learned.

[0054] Real-time image capture and facial expression generation

[0055] 1. The user launches the application and selects their current emotion.

[0056] Select an emotion such as "happy" or "sad" from the "Emotion Selection" menu.

[0057] Review your selection and confirm.

[0058] 2. The device activates the camera and captures images in real time.

[0059] Use the camera to recognize faces.

[0060] 3. The device sends the captured real-time image to the server.

[0061] A real-time image is sent to the server using an HTTP POST request.

[0062] 4. The server analyzes the received real-time images and generates facial expressions using an AI model based on the selected emotion.

[0063] Extract facial feature points and generate the required facial expression.

[0064] 5. The server sends the generated facial expression image to the terminal.

[0065] The generated facial expression image is returned to the terminal as an HTTP response.

[0066] Facial Expressions and Feedback

[0067] 1. The device displays the generated facial expression image on a tablet or AR glasses.

[0068] A display component is used to visually display the facial expression image.

[0069] 2. The user and caregiver review the displayed facial expressions and provide feedback.

[0070] Check whether the image displayed matches the emotion.

[0071] If necessary, use the feedback form to provide your feedback.

[0072] 3. The device sends the feedback to the server and uses it to retrain the AI ​​model.

[0073] Send the feedback data to the server via an HTTP POST request.

[0074] 4. The server retrains the AI ​​model based on the received feedback.

[0075] The feedback data is analyzed, and the AI ​​model is retrained to improve accuracy.

[0076] Specific examples

[0077] 1. The user launches the app on their tablet and uploads a photo of their face.

[0078] 2. The device sends the photo data to the server and begins training the AI ​​model.

[0079] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[0080] 4. The device sends the captured image to the server, and the AI ​​model generates an expression corresponding to the emotion "happy."

[0081] 5. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[0082] 6. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[0083] 7. The device sends feedback to the server, which retrains the AI ​​model.

[0084] The above-described embodiment enables patients to express their emotions appropriately and supports smooth communication with others.

[0085] The processing flow will be explained below.

[0086] Step 1:

[0087] The user launches the application and uploads a photo of their face.

[0088] Specifically, click the "Upload Face Photo" button in the application and select the relevant face photo from your device's photo library.

[0089] Step 2:

[0090] The terminal compresses the facial photograph data selected by the user and transmits it to the server.

[0091] Specifically, the facial photo data is converted into an appropriate format and sent to the server using an HTTP POST request.

[0092] Step 3:

[0093] The server stores the received facial photo data and uses it as learning data for the AI ​​model.

[0094] Specifically, image data is stored in a database and initial learning is performed using an AI model inside the server.

[0095] Step 4:

[0096] The user selects an emotion from the application's emotion selection menu.

[0097] Specifically, open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[0098] Step 5:

[0099] The device activates the user's camera and captures images in real time.

[0100] Specifically, it activates the camera function, runs a facial recognition algorithm to detect a face, and captures an image of it.

[0101] Step 6:

[0102] The device transmits the captured real-time image to the server.

[0103] Specifically, the captured image is converted into an appropriate format and sent to the server using an HTTP POST request.

[0104] Step 7:

[0105] The server analyzes the received real-time images and generates facial expressions using an AI model based on the emotions selected by the user.

[0106] Specifically, facial features are extracted and an AI model is used to generate an expression that corresponds to the selected emotion.

[0107] Step 8:

[0108] The server transmits the generated facial expression image to the terminal.

[0109] Specifically, the generated facial expression image is encoded and sent back to the terminal as an HTTP response.

[0110] Step 9:

[0111] The device displays the facial expression images received from the server on a tablet or AR glasses.

[0112] Specifically, the response from the server is decoded and the facial expression image is displayed in the display component.

[0113] Step 10:

[0114] The user and caregiver review the displayed facial expressions and provide feedback as needed.

[0115] Specifically, the user checks whether the displayed facial expression image matches the emotion, and then uses the feedback form to input their evaluation and suggestions for correction.

[0116] Step 11:

[0117] The terminal transmits the provided feedback data to the server.

[0118] Specifically, the feedback data is sent to the server using an HTTP POST request.

[0119] Step 12:

[0120] The server retrains the AI ​​model based on the received feedback data, improving generation accuracy.

[0121] Specifically, the feedback data is analyzed and the AI ​​model undergoes additional learning.

[0122] Example 1

[0123] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0124] Patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia have difficulty appropriately conveying their emotions to others. As a result, communication between the patient and their caregivers or family members can be hindered. The objective of the present invention is to provide a system that enables such patients to effectively express their emotions and communicate smoothly with others.

[0125] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0126] In this invention, the server includes means for uploading and saving facial photo data of patients using a terminal operated by a user, means for the server to train an AI model based on the saved facial photo data, means for capturing images in real time based on an emotion selected by the patient using the terminal operated by the user and sending them to the server, means for generating facial expressions based on the real-time images received by the server and the selected emotion, and means for sending the generated facial image from the server to the terminal operated by the user and displaying it on a tablet or an augmented reality device. This allows patients to show others facial expressions that match their emotions, thereby realizing smooth communication.

[0127] A "user" is an individual or caregiver who operates the system.

[0128] A "terminal" is an electronic device operated by a user, and primarily includes smartphones, tablets, and augmented reality devices.

[0129] A "server" is a computer system that processes data and trains AI models.

[0130] "Facial image data" means image data of a patient's face that is stored and processed in digital format.

[0131] An "AI model" is a computer model that uses machine learning and deep learning techniques to generate emotion-based facial expressions.

[0132] "Means for storing" refers to a method or device for storing facial photo data in a storage device such as a database.

[0133] "Means for learning" refers to a method or apparatus for feeding data to an AI model and training the model for emotion-based facial expression generation.

[0134] "Capturing means" refers to a method or device that uses a camera function to acquire images in real time.

[0135] "Means for generating" refers to a method or device that uses an AI model to generate facial expressions based on a selected emotion.

[0136] "Transmitting means" refers to a method or apparatus for transferring data or images to another device or system.

[0137] The "display means" refers to a method or device for displaying the generated facial expression image on the screen of a terminal operated by the user or on an augmented reality device.

[0138] The "feedback function" refers to a method or device that allows the user and caregiver to provide their opinions on the displayed facial expression image to the system.

[0139] "Means for relearning" refers to methods or devices that retrain the AI ​​model based on feedback information to improve the accuracy of facial expression generation.

[0140] This invention is a system that allows patients whose facial expressions are limited due to nerve or muscle disorders or the progression of dementia to express their emotions appropriately. This system uses an AI model that learns from photographic data of the patient's face, generates facial expressions corresponding to the patient's emotions in real time, and displays them on a tablet or augmented reality device, thereby supporting communication between the patient and others.

[0141] System components include devices operated by users (e.g., smartphones, tablets, augmented reality devices), servers, databases, AI models, etc.

[0142] First, the user uploads the patient's facial photo data using a device with the application installed. Specifically, the user clicks the "Upload Face Photo" button in the application, selects a facial photo from the device's photo library, and uploads it. At this time, the device compresses the selected facial photo data and sends it to the server using an HTTP POST request.

[0143] The server stores the received facial photo data in a database. The server converts the image data into an appropriate format (e.g., PNG, JPEG) and inserts it into the database.

[0144] The server then trains an AI model based on the stored facial photo data. Specific software used includes OpenCV for detecting facial feature points and TensorFlow for building deep learning models. This process involves retrieving facial photos from the database, feeding the data to the AI ​​model, and training it to generate facial expressions corresponding to emotions.

[0145] Once the AI ​​model has completed its training, the next step is to generate the patient's facial expression in real time. The user launches the application and selects their current emotion. From the "Emotion Selection" menu, they choose an emotion such as "Happy" or "Sad" and press the confirmation button. At this time, the device's camera function is activated to capture an image in real time.

[0146] The device sends the captured real-time image to the server using an HTTP POST request. The server analyzes the received real-time image and generates a facial expression based on the selected emotion. The server extracts feature points from the received image and generates a facial expression corresponding to the selected emotion using an AI model.

[0147] The generated facial image is sent from the server to the device, which displays it on a tablet or augmented reality device. The user and caregiver can then review the displayed facial expression and provide feedback. This feedback is sent from the device to the server via an HTTP POST request. The server then retrains the AI ​​model based on the received feedback to improve the accuracy of the generated facial expressions.

[0148] As a specific example, the following procedure can be considered.

[0149] 1. The user launches the app on their tablet and uploads a photo of their face.

[0150] 2. The device sends the photo data to the server, and the server begins training the AI ​​model.

[0151] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[0152] 4. The device sends the captured image to the server, and the server's AI model generates an expression corresponding to the emotion "happy."

[0153] 5. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[0154] 6. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[0155] 7. The device sends feedback to the server, which retrains the AI ​​model.

[0156] In this way, users can effectively express their feelings and facilitate communication with others.

[0157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0158] Step 1:

[0159] The user launches the application and uploads a face photo. The user clicks the "Upload Face Photo" button and selects a face photo from the photo library on the device.

[0160] Input: User's face photo data (e.g., an image file selected from a smartphone's photo library)

[0161] Output: Compressed face photo data

[0162] Specific actions include the user selecting a photo of their face from the smartphone gallery and pressing the upload button.

[0163] Step 2:

[0164] The device compresses the selected facial photo data and sends it to the server. It reduces the image size using a data compression algorithm and sends the data to the server using an HTTP POST request.

[0165] Input: Selected face photo data

[0166] Output: Compressed face photo data sent to the server

[0167] Specific operations include compressing the image in JPEG format and sending it in the HTTP request body.

[0168] Step 3:

[0169] The server stores the received facial photo data in a database. The server first converts the image data into an appropriate format and then inserts the data into the database.

[0170] Input: Compressed facial photo data

[0171] Output: Facial photo data stored in the database

[0172] Specific operations include converting the image format to PNG and inserting the data into the image table of the database.

[0173] Step 4:

[0174] The server trains the AI ​​model based on the stored facial photo data, extracting feature points from the facial photos and learning various facial expressions that correspond to emotions.

[0175] Input: Facial photo data stored in the database

[0176] Output: Trained AI model

[0177] Specific operations include extracting facial landmarks using OpenCV and training them with a deep learning model using TensorFlow.

[0178] Step 5:

[0179] The user starts the application and selects their current emotion by choosing an emotion such as "happy" or "sad" from the "Emotion Selection" menu and pressing the confirmation button.

[0180] Input: User-selected emotion

[0181] Output: Settings based on selected emotion

[0182] Specific actions include selecting "Happy" from the menu and tapping the confirmation button.

[0183] Step 6:

[0184] The device will activate the camera to capture images in real time. Turn on the device's camera and recognize the user's face in real time.

[0185] Input: Real-time image capture

[0186] Output: Captured face photo

[0187] Specific operations include activating the front camera and applying a facial recognition algorithm to detect the user's face.

[0188] Step 7:

[0189] The device sends the captured real-time image to the server by sending the image data to the server via an HTTP POST request.

[0190] Input: Captured face photo

[0191] Output: Real-time images sent to the server

[0192] Specific operations include converting the captured image into JPEG format and sending it in the HTTP request body.

[0193] Step 8:

[0194] The server analyzes the received real-time images and generates facial expressions based on the selected emotion. It extracts feature points from the received images and generates facial expressions corresponding to the emotion using an AI model.

[0195] Input: Real-time image, selected emotion

[0196] Output: Generated facial expression image

[0197] Specific operations include feeding the received image to the model and generating a smile corresponding to the emotion 'happy'.

[0198] Step 9:

[0199] The server sends the generated facial expression image to the terminal, and returns the generated facial expression image to the terminal as an HTTP response.

[0200] Input: Generated facial expression image

[0201] Output: Facial expression image sent to the device

[0202] Specific operations include encoding the generated facial expression image in PNG format and including it in the HTTP response body.

[0203] Step 10:

[0204] The terminal displays the generated facial expression image on a tablet or an augmented reality device. A display component is used to show the generated facial expression image to the user.

[0205] Input: Facial expression image sent from the server

[0206] Output: Displayed facial expression image

[0207] Specific operations include displaying the generated facial expression image full screen in the center of the tablet screen.

[0208] Step 11:

[0209] The user and caregiver review the displayed facial expressions and provide feedback: check whether the displayed image matches the emotion and enter their opinion in the feedback form.

[0210] Input: Displayed facial expression

[0211] Output: Feedback data

[0212] Specific actions include evaluating whether the displayed 'happy' facial expression looks natural and writing your opinion in a feedback form.

[0213] Step 12:

[0214] The device sends the feedback to the server by compressing the feedback data and sending it to the server via an HTTP POST request.

[0215] Input: Feedback data

[0216] Output: Feedback data sent to the server

[0217] Specific operations include serializing the feedback data in JSON format and sending it in the HTTP request body.

[0218] Step 13:

[0219] The server retrains the AI ​​model based on the received feedback, analyzing the feedback data and retraining the model to reflect the results.

[0220] Input: Feedback data

[0221] Output: Retrained AI model

[0222] Specific operations include analyzing the feedback data, extracting important features, adding them to the learning model, and retraining.

[0223] (Application example 1)

[0224] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0225] Individuals with limited facial expressions due to specific nerve or muscle disorders, or individuals with difficulty expressing emotions due to the progression of dementia, may find it difficult to properly convey their emotions to others, resulting in communication difficulties. This can make it difficult to provide appropriate customer service and services in physical stores. Conventional technology has difficulty generating and conveying facial expressions corresponding to emotions in real time. Another problem is that feedback functions to improve the accuracy of generated facial expressions are not sufficient.

[0226] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0227] In this invention, the server includes means for uploading and storing facial photo data of individuals, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on emotions selected by the individuals and generating facial expressions corresponding to those emotions using the AI ​​model, means for displaying the generated facial expressions on a tablet or head-mounted display, and means for analyzing the emotions received in real time and providing feedback to help the individuals appropriately express the emotions they are feeling. This enables smooth communication in physical stores even for individuals with limited facial expressions, enabling appropriate customer service and provision of services.

[0228] An "individual" is a living organism whose facial expressions may be restricted due to specific nerve or muscle disorders or the progression of dementia.

[0229] "Facial photo data" is image data used to capture an individual's facial features, and is the basic data for AI models to learn from.

[0230] An "AI model" is an artificial intelligence algorithm that uses machine learning to generate facial expressions corresponding to emotions from facial photo data.

[0231] "Real-time image capture" is the process of using the device's camera to capture current facial expressions in real-time.

[0232] "Generating facial expressions" means using an AI model to create facial expressions that correspond to emotions from captured facial expression data.

[0233] A "tablet" is a portable computing device with a flat display for visually displaying generated facial expressions.

[0234] A "head-mounted display" is a display device worn on the head, used to visually display generated facial expressions.

[0235] "Feedback" refers to evaluations and opinions provided on the generated facial expressions and emotion analysis, and is data used to retrain the AI ​​model.

[0236] This invention is a system that enables individuals whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. This system uses an AI model to learn from photographic data of the individual's face, generates facial expressions corresponding to the individual's emotions in real time, and displays them on a tablet or head-mounted display, thereby supporting communication between the individual and others.

[0237] First, the user accesses the system using a tablet or smart glasses, obtains facial photo data of the individual, and uploads it through the tablet or smart glasses. The uploaded facial photo data is stored on the server and used to train the AI ​​model.

[0238] The server uses the stored facial photo data to train the AI ​​model. It extracts facial feature points and learns facial expressions that correspond to emotions. This training process uses software such as machine learning libraries TensorFlow and PyTorch. Once training is complete, the AI ​​model analyzes the image data received in real time and generates facial expressions based on the emotions felt by the individual.

[0239] Next, the user selects an emotion in real time. The camera on the tablet or smart glasses captures the individual's facial expression in real time. This captured image data is sent to a server where it is analyzed. Based on the analyzed image data, an AI model generates an expression corresponding to the emotion. This generated expression is displayed on the tablet or head-mounted display (e.g., Google Glass, Microsoft HoloLens, etc.).

[0240] Additionally, there is a function to provide feedback on the displayed facial expressions and emotion analysis. Users and others can enter their ratings and opinions through a feedback form, which is then sent to the server. This feedback is used to retrain the AI ​​model and improve the accuracy of the generated facial expressions. This feedback loop allows the system to continuously learn and improve.

[0241] As a concrete example, the following scenario can be considered.

[0242] 1. An individual visits a physical store and a staff member is wearing the smart glasses.

[0243] 2. Staff will be able to capture the individual's facial expressions through smart glasses and see their emotions in real time.

[0244] 3. For example, if an individual looks anxious, the smart glasses' display will show "Anxious." Based on this information, the staff member will ask the individual, "Is there something that bothers you?"

[0245] 4. The staff member fills in the feedback form, stating that "the anxious individual has been dealt with," and sends it to the server. Based on this feedback, the AI ​​model undergoes retraining.

[0246] Examples of prompts for generative AI models include:

[0247] "What are your suggestions for dealing with a customer who appears anxious? Please base your suggestions on the following:

[0248] 1. They appear stressed.

[0249] 2. What was said was not clear.

[0250] 3. The tone of the voice is low.

[0251] Please explain what action you recommend and why."

[0252] This system allows individuals with limited facial expressions to express their emotions appropriately and communicate smoothly with others.

[0253] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0254] Step 1:

[0255] The user takes a facial photo of an individual using a tablet or smart glasses and uploads it through the application. As input, the user selects the facial photo data of the individual and clicks the upload button. As output, the facial photo data is compressed and then sent to the server. At this stage, image preprocessing (e.g., compression, color correction) is performed.

[0256] Step 2:

[0257] The server receives the uploaded facial photo data and stores it in a database. As input, compressed facial photo data is sent to the server. As output, the saved facial photo data is stored in the database. During this process, the image data is converted into an appropriate format.

[0258] Step 3:

[0259] The server trains the AI ​​model based on the stored facial photo data. Facial photo data retrieved from the database is provided as input to the AI ​​model. The output is a trained AI model for generating facial expressions corresponding to emotions. During this process, facial feature points are extracted and emotion tags are assigned, and the model is trained using TensorFlow, PyTorch, or similar.

[0260] Step 4:

[0261] The user selects an emotion in an application on a tablet or smart glasses. As input, they select an emotion such as "happy" or "sad" from an emotion selection menu. As output, the selected emotion is passed to the next step. This selection is made through a UI (User Interface).

[0262] Step 5:

[0263] A user uses a camera on a tablet or smart glasses to capture an individual's facial expression in real time. As input, real-time facial images are acquired by the camera. As output, the captured facial images are sent to a server. At this stage, image preprocessing (e.g., noise removal, facial region extraction) is performed.

[0264] Step 6:

[0265] The server analyzes the captured facial images in real time and generates facial expressions using an AI model based on the selected emotion. The captured facial images and the selected emotion are given to the AI ​​model as input. As output, an expression image corresponding to the selected emotion is generated. During this process, facial feature points are analyzed, and template matching and emotion mapping are performed.

[0266] Step 7:

[0267] The server sends the generated facial image to the terminal. The generated facial image is stored on the server as input. The facial image is returned to the terminal as output. Image data is sent as an HTTP response.

[0268] Step 8:

[0269] The device displays the generated facial expression images on a tablet or head-mounted display. As input, the facial expression images sent from the server are stored on the device. As output, the images are visually displayed using a display component. The user or others can check the facial expressions.

[0270] Step 9:

[0271] The user and others provide feedback based on the displayed facial expressions. As input, evaluations and opinions on the displayed facial expressions are entered into a feedback form. As output, the feedback is sent to a server. The feedback data is collected.

[0272] Step 10:

[0273] The server retrains the AI ​​model based on the received feedback. The server receives the feedback data as input. The output is a retrained AI model. The feedback is analyzed and the parameters of the AI ​​model are updated to improve accuracy.

[0274] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0275] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. By combining this invention with an emotion engine that recognizes the user's emotions, the accuracy of the generated facial expressions can be further improved. This system uses an AI model that learns from photographic data of the patient's face, generates facial expressions corresponding to the emotions in real time, and displays them on a tablet or AR glasses, thereby supporting communication between the patient and others.

[0276] Overall system configuration

[0277] Uploading and saving face photo data

[0278] 1. The user launches the application and captures a facial photo.

[0279] Click the "Upload Face Photo" button in the application and select the appropriate face photo from your device's photo library.

[0280] 2. The terminal compresses the selected facial photo data and sends it to the server.

[0281] Send the face photo data to the server using an HTTP POST request.

[0282] 3. The server stores the received facial photo data and uses it as training data for the AI ​​model.

[0283] The image data is converted into an appropriate format and stored in a database.

[0284] Training an AI model

[0285] 1. The server trains the AI ​​model based on the stored facial photo data.

[0286] Feature points are extracted from facial photographs and various facial expressions corresponding to emotions are learned.

[0287] Real-time image capture and facial expression generation

[0288] 1. The user selects an emotion from the emotion selection menu in the application.

[0289] Open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[0290] 2. The device activates the camera and captures images in real time.

[0291] The camera function is used to run a facial recognition algorithm to detect a face and capture an image of it.

[0292] 3. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions.

[0293] Voice recognition technology is used to analyze the tone and content of a user's voice.

[0294] Sensors collect biometric information (heart rate, sweat volume, etc.) and analyze emotions.

[0295] 4. The device sends the captured real-time image and the emotion data obtained by the emotion engine to the server.

[0296] Real-time images and emotion data are sent to the server using HTTP POST requests.

[0297] 5. The server analyzes the received real-time images and emotional data and generates facial expressions using an AI model.

[0298] Based on facial feature points and emotion data, the AI ​​model generates facial expressions that correspond to the selected emotion.

[0299] 6. The server sends the generated facial expression image to the terminal.

[0300] The generated facial expression image is returned to the terminal as an HTTP response.

[0301] Facial Expressions and Feedback

[0302] 1. The device displays the generated facial expression image on a tablet or AR glasses.

[0303] A display component is used to visually display the facial expression image.

[0304] 2. The user and caregiver review the displayed facial expressions and provide feedback.

[0305] Check whether the displayed image matches the emotion and use the feedback form to enter your rating and suggestions for correction.

[0306] 3. The device sends the provided feedback data to the server.

[0307] Send the feedback data to the server via an HTTP POST request.

[0308] 4. The server retrains the AI ​​model based on the received feedback data to improve generation accuracy.

[0309] The feedback data is analyzed and the AI ​​model undergoes additional learning.

[0310] Specific examples

[0311] 1. The user launches the app on their tablet and uploads a photo of their face.

[0312] 2. The device sends the photo data to the server and begins training the AI ​​model.

[0313] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[0314] 4. The emotion engine analyzes the user's voice and confirms that the user is feeling "happy."

[0315] 5. The device sends the captured image and emotional data to the server, and the AI ​​model generates an expression corresponding to the emotion "happy."

[0316] 6. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[0317] 7. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[0318] 8. The device sends feedback to the server, which retrains the AI ​​model.

[0319] The above-described embodiments enable patients to express their emotions appropriately and support smooth communication with others. By combining the emotion engine, the accuracy of facial expression generation is further improved, realizing more accurate emotional expression.

[0320] The processing flow will be explained below.

[0321] Step 1:

[0322] The user launches the application and uploads a photo of their face.

[0323] Specifically, click the "Upload Face Photo" button in the application and select the relevant face photo from your device's photo library.

[0324] Step 2:

[0325] The terminal compresses the facial photograph data selected by the user and transmits it to the server.

[0326] Specifically, the facial photo data is converted into an appropriate format and sent to the server using an HTTP POST request.

[0327] Step 3:

[0328] The server stores the received facial photo data and uses it as learning data for the AI ​​model.

[0329] Specifically, the image data is converted into an appropriate format and stored in a database.

[0330] Step 4:

[0331] The server trains an AI model based on the stored facial photo data.

[0332] Specifically, feature points are extracted from facial photographs and facial expressions corresponding to various emotions are learned.

[0333] Step 5:

[0334] The user selects an emotion from the application's emotion selection menu.

[0335] Specifically, open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[0336] Step 6:

[0337] The device will activate the camera and capture images in real time.

[0338] Specifically, it activates the camera function, runs a facial recognition algorithm to detect a face, and captures an image of it.

[0339] Step 7:

[0340] The emotion engine analyzes the user's voice and biometric information and recognizes the user's emotions (although the subject here is "emotion engine," it is included for convenience).

[0341] Specifically, it uses voice recognition technology to analyze the tone and content of the user's voice, and uses sensors to obtain biometric information (heart rate, amount of sweat, etc.) to analyze emotions.

[0342] Step 8:

[0343] The device transmits the captured real-time images and the emotion data obtained by the emotion engine to the server.

[0344] Specifically, real-time images and the emotion data from the analysis results are sent to the server using an HTTP POST request.

[0345] Step 9:

[0346] The server analyzes the received real-time images and emotional data and generates facial expressions using an AI model.

[0347] Specifically, the AI ​​model generates facial expressions corresponding to the selected emotion based on facial feature points and emotional data.

[0348] Step 10:

[0349] The server transmits the generated facial expression image to the terminal.

[0350] Specifically, the generated facial expression image is encoded and sent back to the terminal as an HTTP response.

[0351] Step 11:

[0352] The device displays the facial expression images received from the server on a tablet or AR glasses.

[0353] Specifically, the response from the server is decoded and the facial expression image is displayed in the display component.

[0354] Step 12:

[0355] The user and caregiver view the displayed facial expressions and provide feedback.

[0356] Specifically, the user checks whether the displayed facial expression image matches the emotion, and then uses the feedback form to input their evaluation and suggestions for correction.

[0357] Step 13:

[0358] The terminal transmits the provided feedback data to the server.

[0359] Specifically, the feedback data is sent to the server using an HTTP POST request.

[0360] Step 14:

[0361] The server retrains the AI ​​model based on the received feedback data, improving generation accuracy.

[0362] Specifically, the feedback data is analyzed and the AI ​​model undergoes additional learning.

[0363] Example 2

[0364] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0365] With conventional methods, patients who have limited facial expression due to specific nerve or muscle disorders or the progression of dementia have difficulty expressing their emotions appropriately. Furthermore, there is a lack of technology to generate facial expressions corresponding to emotions in real time, which makes it difficult for patients to communicate smoothly with others.

[0366] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0367] In this invention, the server includes means for users to upload and store facial photo data, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on emotions selected by the user and generating facial expressions corresponding to the emotions using the AI ​​model, means for displaying the generated facial expressions on a tablet or augmented reality glasses, means for analyzing the user's voice and biometric information in real time and using an emotion engine to recognize emotions, and means for collecting feedback on the generated facial expressions and relearning the AI ​​model based on the feedback. This allows patients to appropriately express their emotions and communicate smoothly with others.

[0368] A "user" is an individual who uses the system to upload facial photo data and select an emotion.

[0369] "Facial photo data" refers to image data of a user's face that is uploaded and used as training data for the AI ​​model.

[0370] An "artificial intelligence model" is an algorithm within the system that uses machine learning techniques to generate facial expressions corresponding to emotions from facial photo data.

[0371] A "real-time image" is an image of a user's face captured instantaneously using a camera.

[0372] The "emotion engine" is part of a system that analyzes the user's voice and biometric information to recognize emotions.

[0373] A "tablet" is a portable computing device with a touch screen that is used to display generated facial expressions.

[0374] "Augmented reality glasses" are wearable devices that overlay digital information onto the user's field of vision.

[0375] "Feedback" refers to evaluations and suggestions for corrections from the user or caregiver regarding the generated facial expression.

[0376] "Retraining" is the process of updating an artificial intelligence model and improving its facial expression generation accuracy based on collected feedback.

[0377] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. This system uses facial photograph data to train an artificial intelligence model, generates facial expressions corresponding to emotions in real time, and displays them on a tablet or augmented reality glasses, thereby supporting communication between the patient and others.

[0378] Uploading and saving face photo data:

[0379] 1. The user launches the application and imports a face photo. Specifically, the user clicks the "Upload Face Photo" button in the application and selects the relevant face photo from the photo library of their device.

[0380] 2. The device compresses the selected facial photo data and sends it to the server using an HTTP POST request. For example, the destination URL is "http: / / example.com / upload."

[0381] 3. The server stores the received facial photo data and uses it as training data for the AI ​​model. Here, the received image data is converted into an appropriate format (e.g., PNG format) and stored in a database (e.g., an Amazon S3 bucket).

[0382] Training the AI ​​model:

[0383] 1. The server trains an AI model based on the stored facial photo data. At this stage, it extracts facial feature points (e.g., the position of the eyes, nose, and mouth) from the facial photo and learns various facial expressions corresponding to emotions. Specifically, it trains a CNN (convolutional neural network) model using TensorFlow and PyTorch.

[0384] Real-time image capture and facial expression generation:

[0385] 1. The user selects an emotion from the application's emotion selection menu. This is an operation in which the user opens the "Emotion Selection" menu, selects from a list of emotions such as "Happy" or "Sad," and confirms the emotion.

[0386] 2. The device starts the camera and captures images in real time. The device's camera function is started, and a face recognition algorithm (for example, OpenCV's face recognition function) is executed to detect the face and generate a captured image.

[0387] 3. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions. It uses voice recognition technology (e.g., Google Speech-to-Text) to analyze the tone and content of the user's voice. It also acquires biometric information (e.g., heart rate, sweat volume) using sensors (e.g., smartwatches) to analyze emotions.

[0388] 4. The device sends the captured real-time image and the emotion data obtained by the emotion engine to the server. The URL for sending the image and emotion data to the server via an HTTP POST request is "http: / / example.com / analyze".

[0389] 5. The server analyzes the received real-time images and emotion data and generates facial expressions using an artificial intelligence model. Specifically, it uses a pre-trained AI model (e.g., a generative adversarial network) to generate facial expressions based on facial feature points and emotion data.

[0390] 6. The server sends the generated facial image to the terminal. At this time, it returns the generated facial image to the terminal as an HTTP response. For example, it sends the image data as Content-Type: image / png.

[0391] Facial Expressions and Feedback:

[0392] 1. The device displays the generated facial image on a tablet or augmented reality glasses. A display component (e.g., React Native's Image component) is used to visually display the facial image.

[0393] 2. The user and caregiver review the displayed facial expressions and provide feedback. They confirm that the displayed image matches the emotion and use the in-app feedback form to enter their rating and suggestions for correction.

[0394] 3. The device sends the provided feedback data to the server, using the URL "http: / / example.com / feedback".

[0395] 4. The server retrains the AI ​​model based on the received feedback data to improve generation accuracy. It analyzes the feedback data and has the AI ​​model undergo additional learning. Specifically, it retrains the model using backpropagation with the additional data.

[0396] Examples:

[0397] The user launches the app on a tablet at home and uploads a recent facial photo. The device compresses the photo data and sends it to the server, which stores the data and begins training the AI ​​model. The user selects the emotion "happy" in the application, and the tablet's camera captures a facial image in real time. The emotion engine analyzes the user's voice data and confirms that the user is feeling "happy." The device sends the captured image and emotion data to the server, and the AI ​​model generates a facial expression corresponding to the emotion "happy." The server sends the generated facial image back to the device, which displays it on the tablet screen. The user's family or caregiver can review the displayed "happy" facial expression and provide feedback. The device sends the feedback to the server, which retrains the AI ​​model based on the feedback data, further improving the accuracy of facial expression generation.

[0398] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0399] Step 1: The user uploads a photo of their face.

[0400] Specific operation: The user launches the application and clicks the "Upload Face Photo" button. The user selects the appropriate face photo from the device's photo library.

[0401] Input: Facial photo data stored on the device.

[0402] Output: The selected face photo data is made available to the application.

[0403] Step 2: The device compresses the facial photo data and sends it to the server.

[0404] Specific operation: The device compresses the selected facial photo data into a format such as JPEG, and sends the facial photo data to the server using an HTTP POST request.

[0405] Input: Selected and compressed facial photo data.

[0406] Output: The compressed data is sent to the server.

[0407] Step 3: The server stores the received facial photo data.

[0408] Specific operation: The server converts the received image data into an appropriate format (e.g. PNG format) and stores the converted data in a database (e.g. Amazon S3 bucket).

[0409] Input: Compressed image data sent to the server.

[0410] Output: Image data converted and saved in the appropriate format.

[0411] Step 4: The server trains the AI ​​model based on the facial photo data.

[0412] Specific operation: The server extracts facial feature points from the stored facial photo data and trains a CNN (convolutional neural network) model using TensorFlow or PyTorch.

[0413] Input: Stored facial photo data.

[0414] Output: An AI model that has learned various facial expressions that correspond to emotions.

[0415] Step 5: The user selects an emotion.

[0416] Specific behavior: The user opens the application's "Emotion Selection" menu, selects from a list of emotions such as "Happy" or "Sad," and confirms the emotion.

[0417] Input: User emotion selection.

[0418] Output: Selected emotion data.

[0419] Step 6: The device captures the face image in real time.

[0420] Specific operation: Activate the device's camera function, run a face recognition algorithm (e.g., OpenCV) to detect the face, and generate a captured image.

[0421] Input: Real-time video.

[0422] Output: Real-time captured image.

[0423] Step 7: The emotion engine recognizes the user's emotion.

[0424] Specific operation: Using voice recognition technology (e.g., Google Speech-to-Text), the system analyzes the tone and content of the user's voice. It also uses sensors to acquire biometric information (e.g., heart rate, sweat volume, etc.) and analyzes emotions.

[0425] Input: User's voice data, biometric data.

[0426] Output: Parsed emotion data.

[0427] Step 8: The device sends the captured image and emotion data to the server.

[0428] Specific operation: The captured image and emotion data are sent to the server via an HTTP POST request. The destination URL is "http: / / example.com / analyze".

[0429] Input: Captcha image, emotion data.

[0430] Output: The data sent to the server.

[0431] Step 9: The server analyzes the received data and generates facial expressions using an AI model.

[0432] Specific operation: Based on the received image and emotion data, facial expressions are generated using a pre-trained AI model (e.g., Generative Adversarial Network).

[0433] Input: Captured image, parsed emotion data.

[0434] Output: The generated facial expression image.

[0435] Step 10: The server sends the generated facial expression image to the terminal.

[0436] Specific operation: The generated facial image is sent back to the device as an HTTP response. When sending, specify Content-Type: image / png.

[0437] Input: Generated facial expression images.

[0438] Output: Facial expression image sent back to the device.

[0439] Step 11: The device displays the generated facial expression image.

[0440] Specific behavior: Use a display component (e.g., React Native's Image component) to display the facial image on a tablet or augmented reality glasses.

[0441] Input: The returned facial expression image.

[0442] Output: Facial expression images displayed on a tablet or augmented reality glasses.

[0443] Step 12: User and caregiver provide feedback.

[0444] What it does: Check whether the displayed image matches the emotion and enter your rating and suggestions for corrections using the in-app feedback form.

[0445] Input: Feedback rating, correction suggestions.

[0446] Output: Feedback data.

[0447] Step 13: The terminal sends the feedback data to the server.

[0448] Specific operation: The feedback data is sent to the server via an HTTP POST request, with the destination URL being "http: / / example.com / feedback".

[0449] Input: Feedback data.

[0450] Output: Feedback data sent to the server.

[0451] Step 14: The server retrains the AI ​​model based on the feedback.

[0452] Specific operation: Analyzes the feedback data and allows the AI ​​model to perform additional learning. Specifically, the model is retrained using backpropagation with the additional data.

[0453] Input: Feedback data.

[0454] Output: An AI model with improved accuracy.

[0455] (Application example 2)

[0456] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0457] Patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia have difficulty expressing their emotions appropriately. As a result, communication between the patient and those around them is hindered, and the patient themselves suffers psychological stress. To solve this problem, an efficient system is needed that can generate facial expressions corresponding to the patient's emotions in real time based on photographic data of the patient's face and visually convey them to others.

[0458] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading and saving patient facial photo data, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on an emotion selected by the patient and generating a facial expression corresponding to that emotion using the AI ​​model, means for displaying the generated facial expression on a visual display device, means for analyzing voice and biometric information using an emotion engine to recognize the patient's emotion, and means for wearing an assistance device integrating a camera function and emotion analysis means for use in a specific facility. This enables the patient's emotion to be accurately expressed in real time and facilitates smooth communication with others.

[0459] "Specific nerve or muscle disorders" refers to a condition in which abnormalities in the nervous or muscular system make it difficult to move parts of the body normally.

[0460] A "system for generating and displaying facial expressions" is a set of devices and technologies that uses a computer to recognize a patient's emotions, generate a facial expression corresponding to that emotion in real time, and visually display it.

[0461] "Patient facial photograph data" refers to image data of a patient's face used for emotion analysis and facial expression generation.

[0462] An "AI model" is an artificial intelligence model that uses machine learning algorithms to learn specific patterns from data and recognize emotions and generate facial expressions.

[0463] "Real-time image capture means" refers to devices and algorithms that instantly capture and process images of the current moment.

[0464] "Visual display device" refers to a display or screen that visually presents generated expressions and information to a user, including tablets and AR glasses.

[0465] An "emotion engine" is a software or hardware component that analyzes voice and biometric information to recognize a user's emotions.

[0466] "Assistive devices" refer to devices that integrate various sensors and display devices to provide specific functions. Specifically, they include devices with camera functions and emotion analysis functions.

[0467] The present invention provides a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders to express their emotions appropriately. This system recognizes the patient's emotions, generates facial expressions in real time based on the emotions, and displays them on a visual display device to support communication. An embodiment of the system is described in detail below.

[0468] Overall system configuration

[0469] 1. Upload and save face photo data

[0470] The user uploads face photo data to the application and sends it to the server. The server saves the received face photo data and uses it as training data for the AI ​​model. Specifically, the user clicks the "Upload Face Photo" button in the application and selects the relevant face photo from the device's photo library. The selected face photo data is sent to the server as compressed data using an HTTP POST request. This process can be performed using the Python requests library or Flask as a framework.

[0471] 2. Training the AI ​​model

[0472] The server trains the AI ​​model based on the stored facial photo data. The AI ​​model used for training uses machine learning frameworks such as TensorFlow and PyTorch. It extracts feature points from the facial photos and trains the AI ​​model to recognize various facial expressions that correspond to emotions.

[0473] 3. Real-time image capture and facial expression generation

[0474] The user selects an emotion from the emotion selection menu, and the device activates the camera to capture an image in real time. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotion. The analysis uses the Emotion API and a custom machine learning model. The captured real-time image and emotional data are sent to a server, where the AI ​​model generates the corresponding facial expression.

[0475] 4. Facial Expression Display and Feedback

[0476] The generated facial images are displayed on a visual display device, specifically a tablet or AR glasses. The generated facial images are visually confirmed and the user or caregiver provides feedback. The feedback is sent to the server through a form in the application, and the AI ​​model is retrained. This improves the accuracy of facial expression generation.

[0477] System operation example

[0478] As a specific example of how it works, let's consider its use in a nursing home. A patient wears a head-mounted display, and the facility manager launches the application. When the manager selects "happy" as the patient's emotion, a real-time image of the patient is captured by a camera and analyzed by an emotion engine. The AI ​​model generates a "happy" facial expression, which is displayed on the head-mounted display. The manager checks the displayed expression and provides feedback. This process facilitates smooth communication.

[0479] Prompt Sentence Examples

[0480] For example, you can use the following prompt:

[0481] Using the "Emotion Communication Assist" app, the system analyzes the patient's emotions in real time and displays them on the head-mounted display. For example, if the patient is feeling happy, a facial expression corresponding to that emotion is displayed.

[0482] In this way, the present invention overcomes the limitations of facial expression caused by specific nerve or muscle disorders and provides a useful tool to assist communication with others.

[0483] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0484] Step 1: Upload your face photo data

[0485] A user launches the application and selects and uploads a facial photo from the device's photo library. Specifically, when the user clicks the "Upload Face Photo" button in the application and selects a facial photo, the device compresses the facial photo data and sends it to the server via an HTTP POST request. The input of this process is the facial photo data selected by the user, and the output is the compressed facial photo data stored on the server.

[0486] Step 2: Training the AI ​​model

[0487] The server trains the AI ​​model based on the uploaded facial photo data. Specifically, it uses Python's TensorFlow and PyTorch to extract feature points from the facial photos and learn various facial expressions that correspond to emotions. The input is the facial photo data and existing training data, and the output is an AI model that can recognize facial expression patterns that correspond to emotions.

[0488] Step 3: Select emotion and activate camera

[0489] The user selects an emotion from the application's emotion selection menu. The selected emotion data is input to the device. After selecting the emotion, the device activates the camera and captures an image in real time. The captured image data becomes input for real-time processing and is sent to the server.

[0490] Step 4: Recognize emotions

[0491] The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions. Voice data and biometric information such as heart rate are acquired from sensors and analyzed by the emotion engine (such as EmotionAPI). The input is the user's voice data and biometric information data, and the output is recognized emotion data.

[0492] Step 5: Facial Expression Generation

[0493] The server uses an AI model to generate facial expressions corresponding to the selected emotion based on the captured image data and emotional data. The AI ​​model utilizes previously learned patterns to convert the captured facial photo into an expression corresponding to the emotion. The input is the real-time image and recognized emotional data, and the output is the generated facial expression image.

[0494] Step 6: Displaying facial expressions

[0495] The generated facial image is sent from the server to the device. The device displays the received facial image on a display device (tablet or AR glasses). Specifically, the generated facial image is displayed immediately on the device screen. The input is the generated facial image data, and the output is the facial expression displayed on the visual display device.

[0496] Step 7: Getting feedback

[0497] The user and caregiver can check the displayed facial expressions and provide feedback. The feedback is entered using a feedback form within the application, and the device sends the data to the server. The input is the feedback data provided by the user or caregiver, and the output is the feedback data stored on the server.

[0498] Step 8: Retraining the AI ​​model

[0499] The server retrains the AI ​​model based on the received feedback data. Specifically, it analyzes the feedback data and provides additional training data to the AI ​​model to improve the accuracy of facial expression generation. The input is the feedback data and the existing AI model, and the output is the updated AI model.

[0500] Through the above steps, the present invention enables the patient's emotions to be accurately expressed in real time, enabling smooth communication with others.

[0501] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0502] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0503] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0504] [Second embodiment]

[0505] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0506] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0507] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0508] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0509] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0510] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0511] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0512] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0513] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0514] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0515] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0516] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0517] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to express their emotions appropriately. This system uses an AI model to learn from photographic data of the patient's face, generates facial expressions corresponding to the patient's emotions in real time, and displays them on a tablet or AR glasses, thereby supporting communication between the patient and others.

[0518] Overall system configuration

[0519] Uploading and saving face photo data

[0520] 1. The user launches the application and captures a facial photo.

[0521] Click the "Upload Face Photo" button on the application.

[0522] Open your device's photo library and select a photo of your face.

[0523] 2. The terminal compresses the selected facial photo data and sends it to the server.

[0524] Send the face photo data to the server using an HTTP POST request.

[0525] 3. The server stores the received facial photo data in its database.

[0526] The image data is converted into an appropriate format and saved.

[0527] Training an AI model

[0528] 1. The server trains the AI ​​model based on the stored facial photo data.

[0529] Feature points are extracted from facial photographs and various facial expressions corresponding to emotions are learned.

[0530] Real-time image capture and facial expression generation

[0531] 1. The user launches the application and selects their current emotion.

[0532] Select an emotion such as "happy" or "sad" from the "Emotion Selection" menu.

[0533] Review your selection and confirm.

[0534] 2. The device activates the camera and captures images in real time.

[0535] Use the camera to recognize faces.

[0536] 3. The device sends the captured real-time image to the server.

[0537] A real-time image is sent to the server using an HTTP POST request.

[0538] 4. The server analyzes the received real-time images and generates facial expressions using an AI model based on the selected emotion.

[0539] Extract facial feature points and generate the required facial expression.

[0540] 5. The server sends the generated facial expression image to the terminal.

[0541] The generated facial expression image is returned to the terminal as an HTTP response.

[0542] Facial Expressions and Feedback

[0543] 1. The device displays the generated facial expression image on a tablet or AR glasses.

[0544] A display component is used to visually display the facial expression image.

[0545] 2. The user and caregiver review the displayed facial expressions and provide feedback.

[0546] Check whether the image displayed matches the emotion.

[0547] If necessary, use the feedback form to provide your feedback.

[0548] 3. The device sends the feedback to the server and uses it to retrain the AI ​​model.

[0549] Send the feedback data to the server via an HTTP POST request.

[0550] 4. The server retrains the AI ​​model based on the received feedback.

[0551] The feedback data is analyzed, and the AI ​​model is retrained to improve accuracy.

[0552] Specific examples

[0553] 1. The user launches the app on their tablet and uploads a photo of their face.

[0554] 2. The device sends the photo data to the server and begins training the AI ​​model.

[0555] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[0556] 4. The device sends the captured image to the server, and the AI ​​model generates an expression corresponding to the emotion "happy."

[0557] 5. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[0558] 6. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[0559] 7. The device sends feedback to the server, which retrains the AI ​​model.

[0560] The above-described embodiment enables patients to express their emotions appropriately and supports smooth communication with others.

[0561] The processing flow will be explained below.

[0562] Step 1:

[0563] The user launches the application and uploads a photo of their face.

[0564] Specifically, click the "Upload Face Photo" button in the application and select the relevant face photo from your device's photo library.

[0565] Step 2:

[0566] The terminal compresses the facial photograph data selected by the user and transmits it to the server.

[0567] Specifically, the facial photo data is converted into an appropriate format and sent to the server using an HTTP POST request.

[0568] Step 3:

[0569] The server stores the received facial photo data and uses it as learning data for the AI ​​model.

[0570] Specifically, image data is stored in a database and initial learning is performed using an AI model inside the server.

[0571] Step 4:

[0572] The user selects an emotion from the application's emotion selection menu.

[0573] Specifically, open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[0574] Step 5:

[0575] The device activates the user's camera and captures images in real time.

[0576] Specifically, it activates the camera function, runs a facial recognition algorithm to detect a face, and captures an image of it.

[0577] Step 6:

[0578] The device transmits the captured real-time image to the server.

[0579] Specifically, the captured image is converted into an appropriate format and sent to the server using an HTTP POST request.

[0580] Step 7:

[0581] The server analyzes the received real-time images and generates facial expressions using an AI model based on the emotions selected by the user.

[0582] Specifically, facial features are extracted and an AI model is used to generate an expression that corresponds to the selected emotion.

[0583] Step 8:

[0584] The server transmits the generated facial expression image to the terminal.

[0585] Specifically, the generated facial expression image is encoded and sent back to the terminal as an HTTP response.

[0586] Step 9:

[0587] The device displays the facial expression images received from the server on a tablet or AR glasses.

[0588] Specifically, the response from the server is decoded and the facial expression image is displayed in the display component.

[0589] Step 10:

[0590] The user and caregiver review the displayed facial expressions and provide feedback as needed.

[0591] Specifically, the user checks whether the displayed facial expression image matches the emotion, and then uses the feedback form to input their evaluation and suggestions for correction.

[0592] Step 11:

[0593] The terminal transmits the provided feedback data to the server.

[0594] Specifically, the feedback data is sent to the server using an HTTP POST request.

[0595] Step 12:

[0596] The server retrains the AI ​​model based on the received feedback data, improving generation accuracy.

[0597] Specifically, the feedback data is analyzed and the AI ​​model undergoes additional learning.

[0598] Example 1

[0599] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] Patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia have difficulty appropriately conveying their emotions to others. As a result, communication between the patient and their caregivers or family members can be hindered. The objective of the present invention is to provide a system that enables such patients to effectively express their emotions and communicate smoothly with others.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0602] In this invention, the server includes means for uploading and saving facial photo data of patients using a terminal operated by a user, means for the server to train an AI model based on the saved facial photo data, means for capturing images in real time based on an emotion selected by the patient using the terminal operated by the user and sending them to the server, means for generating facial expressions based on the real-time images received by the server and the selected emotion, and means for sending the generated facial image from the server to the terminal operated by the user and displaying it on a tablet or an augmented reality device. This allows patients to show others facial expressions that match their emotions, thereby realizing smooth communication.

[0603] A "user" is an individual or caregiver who operates the system.

[0604] A "terminal" is an electronic device operated by a user, and primarily includes smartphones, tablets, and augmented reality devices.

[0605] A "server" is a computer system that processes data and trains AI models.

[0606] "Facial image data" means image data of a patient's face that is stored and processed in digital format.

[0607] An "AI model" is a computer model that uses machine learning and deep learning techniques to generate emotion-based facial expressions.

[0608] "Means for storing" refers to a method or device for storing facial photo data in a storage device such as a database.

[0609] "Means for learning" refers to a method or apparatus for feeding data to an AI model and training the model for emotion-based facial expression generation.

[0610] "Capturing means" refers to a method or device that uses a camera function to acquire images in real time.

[0611] "Means for generating" refers to a method or device that uses an AI model to generate facial expressions based on a selected emotion.

[0612] "Transmitting means" refers to a method or apparatus for transferring data or images to another device or system.

[0613] The "display means" refers to a method or device for displaying the generated facial expression image on the screen of a terminal operated by the user or on an augmented reality device.

[0614] The "feedback function" refers to a method or device that allows the user and caregiver to provide their opinions on the displayed facial expression image to the system.

[0615] "Means for relearning" refers to methods or devices that retrain the AI ​​model based on feedback information to improve the accuracy of facial expression generation.

[0616] This invention is a system that allows patients whose facial expressions are limited due to nerve or muscle disorders or the progression of dementia to express their emotions appropriately. This system uses an AI model that learns from photographic data of the patient's face, generates facial expressions corresponding to the patient's emotions in real time, and displays them on a tablet or augmented reality device, thereby supporting communication between the patient and others.

[0617] System components include devices operated by users (e.g., smartphones, tablets, augmented reality devices), servers, databases, AI models, etc.

[0618] First, the user uploads the patient's facial photo data using a device with the application installed. Specifically, the user clicks the "Upload Face Photo" button in the application, selects a facial photo from the device's photo library, and uploads it. At this time, the device compresses the selected facial photo data and sends it to the server using an HTTP POST request.

[0619] The server stores the received facial photo data in a database. The server converts the image data into an appropriate format (e.g., PNG, JPEG) and inserts it into the database.

[0620] The server then trains an AI model based on the stored facial photo data. Specific software used includes OpenCV for detecting facial feature points and TensorFlow for building deep learning models. This process involves retrieving facial photos from the database, feeding the data to the AI ​​model, and training it to generate facial expressions corresponding to emotions.

[0621] Once the AI ​​model has completed its training, the next step is to generate the patient's facial expression in real time. The user launches the application and selects their current emotion. From the "Emotion Selection" menu, they choose an emotion such as "Happy" or "Sad" and press the confirmation button. At this time, the device's camera function is activated to capture an image in real time.

[0622] The device sends the captured real-time image to the server using an HTTP POST request. The server analyzes the received real-time image and generates a facial expression based on the selected emotion. The server extracts feature points from the received image and generates a facial expression corresponding to the selected emotion using an AI model.

[0623] The generated facial image is sent from the server to the device, which displays it on a tablet or augmented reality device. The user and caregiver can then review the displayed facial expression and provide feedback. This feedback is sent from the device to the server via an HTTP POST request. The server then retrains the AI ​​model based on the received feedback to improve the accuracy of the generated facial expressions.

[0624] As a specific example, the following procedure can be considered.

[0625] 1. The user launches the app on their tablet and uploads a photo of their face.

[0626] 2. The device sends the photo data to the server, and the server begins training the AI ​​model.

[0627] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[0628] 4. The device sends the captured image to the server, and the server's AI model generates an expression corresponding to the emotion "happy."

[0629] 5. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[0630] 6. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[0631] 7. The device sends feedback to the server, which retrains the AI ​​model.

[0632] In this way, users can effectively express their feelings and facilitate communication with others.

[0633] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0634] Step 1:

[0635] The user launches the application and uploads a face photo. The user clicks the "Upload Face Photo" button and selects a face photo from the photo library on the device.

[0636] Input: User's face photo data (e.g., an image file selected from a smartphone's photo library)

[0637] Output: Compressed face photo data

[0638] Specific actions include the user selecting a photo of their face from the smartphone gallery and pressing the upload button.

[0639] Step 2:

[0640] The device compresses the selected facial photo data and sends it to the server. It reduces the image size using a data compression algorithm and sends the data to the server using an HTTP POST request.

[0641] Input: Selected face photo data

[0642] Output: Compressed face photo data sent to the server

[0643] Specific operations include compressing the image in JPEG format and sending it in the HTTP request body.

[0644] Step 3:

[0645] The server stores the received facial photo data in a database. The server first converts the image data into an appropriate format and then inserts the data into the database.

[0646] Input: Compressed facial photo data

[0647] Output: Facial photo data stored in the database

[0648] Specific operations include converting the image format to PNG and inserting the data into the image table of the database.

[0649] Step 4:

[0650] The server trains the AI ​​model based on the stored facial photo data, extracting feature points from the facial photos and learning various facial expressions that correspond to emotions.

[0651] Input: Facial photo data stored in the database

[0652] Output: Trained AI model

[0653] Specific operations include extracting facial landmarks using OpenCV and training them with a deep learning model using TensorFlow.

[0654] Step 5:

[0655] The user starts the application and selects their current emotion by choosing an emotion such as "happy" or "sad" from the "Emotion Selection" menu and pressing the confirmation button.

[0656] Input: User-selected emotion

[0657] Output: Settings based on selected emotion

[0658] Specific actions include selecting "Happy" from the menu and tapping the confirmation button.

[0659] Step 6:

[0660] The device will activate the camera to capture images in real time. Turn on the device's camera and recognize the user's face in real time.

[0661] Input: Real-time image capture

[0662] Output: Captured face photo

[0663] Specific operations include activating the front camera and applying a facial recognition algorithm to detect the user's face.

[0664] Step 7:

[0665] The device sends the captured real-time image to the server by sending the image data to the server via an HTTP POST request.

[0666] Input: Captured face photo

[0667] Output: Real-time images sent to the server

[0668] Specific operations include converting the captured image into JPEG format and sending it in the HTTP request body.

[0669] Step 8:

[0670] The server analyzes the received real-time images and generates facial expressions based on the selected emotion. It extracts feature points from the received images and generates facial expressions corresponding to the emotion using an AI model.

[0671] Input: Real-time image, selected emotion

[0672] Output: Generated facial expression image

[0673] Specific operations include feeding the received image to the model and generating a smile corresponding to the emotion 'happy'.

[0674] Step 9:

[0675] The server sends the generated facial expression image to the terminal, and returns the generated facial expression image to the terminal as an HTTP response.

[0676] Input: Generated facial expression image

[0677] Output: Facial expression image sent to the device

[0678] Specific operations include encoding the generated facial expression image in PNG format and including it in the HTTP response body.

[0679] Step 10:

[0680] The terminal displays the generated facial expression image on a tablet or an augmented reality device. A display component is used to show the generated facial expression image to the user.

[0681] Input: Facial expression image sent from the server

[0682] Output: Displayed facial expression image

[0683] Specific operations include displaying the generated facial expression image full screen in the center of the tablet screen.

[0684] Step 11:

[0685] The user and caregiver review the displayed facial expressions and provide feedback: check whether the displayed image matches the emotion and enter their opinion in the feedback form.

[0686] Input: Displayed facial expression

[0687] Output: Feedback data

[0688] Specific actions include evaluating whether the displayed 'happy' facial expression looks natural and writing your opinion in a feedback form.

[0689] Step 12:

[0690] The device sends the feedback to the server by compressing the feedback data and sending it to the server via an HTTP POST request.

[0691] Input: Feedback data

[0692] Output: Feedback data sent to the server

[0693] Specific operations include serializing the feedback data in JSON format and sending it in the HTTP request body.

[0694] Step 13:

[0695] The server retrains the AI ​​model based on the received feedback, analyzing the feedback data and retraining the model to reflect the results.

[0696] Input: Feedback data

[0697] Output: Retrained AI model

[0698] Specific operations include analyzing the feedback data, extracting important features, adding them to the learning model, and retraining.

[0699] (Application example 1)

[0700] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0701] Individuals with limited facial expressions due to specific nerve or muscle disorders, or individuals with difficulty expressing emotions due to the progression of dementia, may find it difficult to properly convey their emotions to others, resulting in communication difficulties. This can make it difficult to provide appropriate customer service and services in physical stores. Conventional technology has difficulty generating and conveying facial expressions corresponding to emotions in real time. Another problem is that feedback functions to improve the accuracy of generated facial expressions are not sufficient.

[0702] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0703] In this invention, the server includes means for uploading and storing facial photo data of individuals, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on emotions selected by the individuals and generating facial expressions corresponding to those emotions using the AI ​​model, means for displaying the generated facial expressions on a tablet or head-mounted display, and means for analyzing the emotions received in real time and providing feedback to help the individuals appropriately express the emotions they are feeling. This enables smooth communication in physical stores even for individuals with limited facial expressions, enabling appropriate customer service and provision of services.

[0704] An "individual" is a living organism whose facial expressions may be restricted due to specific nerve or muscle disorders or the progression of dementia.

[0705] "Facial photo data" is image data used to capture an individual's facial features, and is the basic data for AI models to learn from.

[0706] An "AI model" is an artificial intelligence algorithm that uses machine learning to generate facial expressions corresponding to emotions from facial photo data.

[0707] "Real-time image capture" is the process of using the device's camera to capture current facial expressions in real-time.

[0708] "Generating facial expressions" means using an AI model to create facial expressions that correspond to emotions from captured facial expression data.

[0709] A "tablet" is a portable computing device with a flat display for visually displaying generated facial expressions.

[0710] A "head-mounted display" is a display device worn on the head, used to visually display generated facial expressions.

[0711] "Feedback" refers to evaluations and opinions provided on the generated facial expressions and emotion analysis, and is data used to retrain the AI ​​model.

[0712] This invention is a system that enables individuals whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. This system uses an AI model to learn from photographic data of the individual's face, generates facial expressions corresponding to the individual's emotions in real time, and displays them on a tablet or head-mounted display, thereby supporting communication between the individual and others.

[0713] First, the user accesses the system using a tablet or smart glasses, obtains facial photo data of the individual, and uploads it through the tablet or smart glasses. The uploaded facial photo data is stored on the server and used to train the AI ​​model.

[0714] The server uses the stored facial photo data to train the AI ​​model. It extracts facial feature points and learns facial expressions that correspond to emotions. This training process uses software such as machine learning libraries TensorFlow and PyTorch. Once training is complete, the AI ​​model analyzes the image data received in real time and generates facial expressions based on the emotions felt by the individual.

[0715] Next, the user selects an emotion in real time. The camera on the tablet or smart glasses captures the individual's facial expression in real time. This captured image data is sent to a server where it is analyzed. Based on the analyzed image data, an AI model generates an expression corresponding to the emotion. This generated expression is displayed on the tablet or head-mounted display (e.g., Google Glass, Microsoft HoloLens, etc.).

[0716] Additionally, there is a function to provide feedback on the displayed facial expressions and emotion analysis. Users and others can enter their ratings and opinions through a feedback form, which is then sent to the server. This feedback is used to retrain the AI ​​model and improve the accuracy of the generated facial expressions. This feedback loop allows the system to continuously learn and improve.

[0717] As a concrete example, the following scenario can be considered.

[0718] 1. An individual visits a physical store and a staff member is wearing the smart glasses.

[0719] 2. Staff will be able to capture the individual's facial expressions through smart glasses and see their emotions in real time.

[0720] 3. For example, if an individual looks anxious, the smart glasses' display will show "Anxious." Based on this information, the staff member will ask the individual, "Is there something that bothers you?"

[0721] 4. The staff member fills in the feedback form, stating that "the anxious individual has been dealt with," and sends it to the server. Based on this feedback, the AI ​​model undergoes retraining.

[0722] Examples of prompts for generative AI models include:

[0723] "What are your suggestions for dealing with a customer who appears anxious? Please base your suggestions on the following:

[0724] 1. They appear stressed.

[0725] 2. What was said was not clear.

[0726] 3. The tone of the voice is low.

[0727] Please explain what action you recommend and why."

[0728] This system allows individuals with limited facial expressions to express their emotions appropriately and communicate smoothly with others.

[0729] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0730] Step 1:

[0731] The user takes a facial photo of an individual using a tablet or smart glasses and uploads it through the application. As input, the user selects the facial photo data of the individual and clicks the upload button. As output, the facial photo data is compressed and then sent to the server. At this stage, image preprocessing (e.g., compression, color correction) is performed.

[0732] Step 2:

[0733] The server receives the uploaded facial photo data and stores it in a database. As input, compressed facial photo data is sent to the server. As output, the saved facial photo data is stored in the database. During this process, the image data is converted into an appropriate format.

[0734] Step 3:

[0735] The server trains the AI ​​model based on the stored facial photo data. Facial photo data retrieved from the database is provided as input to the AI ​​model. The output is a trained AI model for generating facial expressions corresponding to emotions. During this process, facial feature points are extracted and emotion tags are assigned, and the model is trained using TensorFlow, PyTorch, or similar.

[0736] Step 4:

[0737] The user selects an emotion in an application on a tablet or smart glasses. As input, they select an emotion such as "happy" or "sad" from an emotion selection menu. As output, the selected emotion is passed to the next step. This selection is made through a UI (User Interface).

[0738] Step 5:

[0739] A user uses a camera on a tablet or smart glasses to capture an individual's facial expression in real time. As input, real-time facial images are acquired by the camera. As output, the captured facial images are sent to a server. At this stage, image preprocessing (e.g., noise removal, facial region extraction) is performed.

[0740] Step 6:

[0741] The server analyzes the captured facial images in real time and generates facial expressions using an AI model based on the selected emotion. The captured facial images and the selected emotion are given to the AI ​​model as input. As output, an expression image corresponding to the selected emotion is generated. During this process, facial feature points are analyzed, and template matching and emotion mapping are performed.

[0742] Step 7:

[0743] The server sends the generated facial image to the terminal. The generated facial image is stored on the server as input. The facial image is returned to the terminal as output. Image data is sent as an HTTP response.

[0744] Step 8:

[0745] The device displays the generated facial expression images on a tablet or head-mounted display. As input, the facial expression images sent from the server are stored on the device. As output, the images are visually displayed using a display component. The user or others can check the facial expressions.

[0746] Step 9:

[0747] The user and others provide feedback based on the displayed facial expressions. As input, evaluations and opinions on the displayed facial expressions are entered into a feedback form. As output, the feedback is sent to a server. The feedback data is collected.

[0748] Step 10:

[0749] The server retrains the AI ​​model based on the received feedback. The server receives the feedback data as input. The output is a retrained AI model. The feedback is analyzed and the parameters of the AI ​​model are updated to improve accuracy.

[0750] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0751] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. By combining this invention with an emotion engine that recognizes the user's emotions, the accuracy of the generated facial expressions can be further improved. This system uses an AI model that learns from photographic data of the patient's face, generates facial expressions corresponding to the emotions in real time, and displays them on a tablet or AR glasses, thereby supporting communication between the patient and others.

[0752] Overall system configuration

[0753] Uploading and saving face photo data

[0754] 1. The user launches the application and captures a facial photo.

[0755] Click the "Upload Face Photo" button in the application and select the appropriate face photo from your device's photo library.

[0756] 2. The terminal compresses the selected facial photo data and sends it to the server.

[0757] Send the face photo data to the server using an HTTP POST request.

[0758] 3. The server stores the received facial photo data and uses it as training data for the AI ​​model.

[0759] The image data is converted into an appropriate format and stored in a database.

[0760] Training an AI model

[0761] 1. The server trains the AI ​​model based on the stored facial photo data.

[0762] Feature points are extracted from facial photographs and various facial expressions corresponding to emotions are learned.

[0763] Real-time image capture and facial expression generation

[0764] 1. The user selects an emotion from the emotion selection menu in the application.

[0765] Open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[0766] 2. The device activates the camera and captures images in real time.

[0767] The camera function is used to run a facial recognition algorithm to detect a face and capture an image of it.

[0768] 3. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions.

[0769] Voice recognition technology is used to analyze the tone and content of a user's voice.

[0770] Sensors collect biometric information (heart rate, sweat volume, etc.) and analyze emotions.

[0771] 4. The device sends the captured real-time image and the emotion data obtained by the emotion engine to the server.

[0772] Real-time images and emotion data are sent to the server using HTTP POST requests.

[0773] 5. The server analyzes the received real-time images and emotional data and generates facial expressions using an AI model.

[0774] Based on facial feature points and emotion data, the AI ​​model generates facial expressions that correspond to the selected emotion.

[0775] 6. The server sends the generated facial expression image to the terminal.

[0776] The generated facial expression image is returned to the terminal as an HTTP response.

[0777] Facial Expressions and Feedback

[0778] 1. The device displays the generated facial expression image on a tablet or AR glasses.

[0779] A display component is used to visually display the facial expression image.

[0780] 2. The user and caregiver review the displayed facial expressions and provide feedback.

[0781] Check whether the displayed image matches the emotion and use the feedback form to enter your rating and suggestions for correction.

[0782] 3. The device sends the provided feedback data to the server.

[0783] Send the feedback data to the server via an HTTP POST request.

[0784] 4. The server retrains the AI ​​model based on the received feedback data to improve generation accuracy.

[0785] The feedback data is analyzed and the AI ​​model undergoes additional learning.

[0786] Specific examples

[0787] 1. The user launches the app on their tablet and uploads a photo of their face.

[0788] 2. The device sends the photo data to the server and begins training the AI ​​model.

[0789] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[0790] 4. The emotion engine analyzes the user's voice and confirms that the user is feeling "happy."

[0791] 5. The device sends the captured image and emotional data to the server, and the AI ​​model generates an expression corresponding to the emotion "happy."

[0792] 6. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[0793] 7. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[0794] 8. The device sends feedback to the server, which retrains the AI ​​model.

[0795] The above-described embodiments enable patients to express their emotions appropriately and support smooth communication with others. By combining the emotion engine, the accuracy of facial expression generation is further improved, realizing more accurate emotional expression.

[0796] The processing flow will be explained below.

[0797] Step 1:

[0798] The user launches the application and uploads a photo of their face.

[0799] Specifically, click the "Upload Face Photo" button in the application and select the relevant face photo from your device's photo library.

[0800] Step 2:

[0801] The terminal compresses the facial photograph data selected by the user and transmits it to the server.

[0802] Specifically, the facial photo data is converted into an appropriate format and sent to the server using an HTTP POST request.

[0803] Step 3:

[0804] The server stores the received facial photo data and uses it as learning data for the AI ​​model.

[0805] Specifically, the image data is converted into an appropriate format and stored in a database.

[0806] Step 4:

[0807] The server trains an AI model based on the stored facial photo data.

[0808] Specifically, feature points are extracted from facial photographs and facial expressions corresponding to various emotions are learned.

[0809] Step 5:

[0810] The user selects an emotion from the application's emotion selection menu.

[0811] Specifically, open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[0812] Step 6:

[0813] The device will activate the camera and capture images in real time.

[0814] Specifically, it activates the camera function, runs a facial recognition algorithm to detect a face, and captures an image of it.

[0815] Step 7:

[0816] The emotion engine analyzes the user's voice and biometric information and recognizes the user's emotions (although the subject here is "emotion engine," it is included for convenience).

[0817] Specifically, it uses voice recognition technology to analyze the tone and content of the user's voice, and uses sensors to obtain biometric information (heart rate, amount of sweat, etc.) to analyze emotions.

[0818] Step 8:

[0819] The device transmits the captured real-time images and the emotion data obtained by the emotion engine to the server.

[0820] Specifically, real-time images and the emotion data from the analysis results are sent to the server using an HTTP POST request.

[0821] Step 9:

[0822] The server analyzes the received real-time images and emotional data and generates facial expressions using an AI model.

[0823] Specifically, the AI ​​model generates facial expressions corresponding to the selected emotion based on facial feature points and emotional data.

[0824] Step 10:

[0825] The server transmits the generated facial expression image to the terminal.

[0826] Specifically, the generated facial expression image is encoded and sent back to the terminal as an HTTP response.

[0827] Step 11:

[0828] The device displays the facial expression images received from the server on a tablet or AR glasses.

[0829] Specifically, the response from the server is decoded and the facial expression image is displayed in the display component.

[0830] Step 12:

[0831] The user and caregiver view the displayed facial expressions and provide feedback.

[0832] Specifically, the user checks whether the displayed facial expression image matches the emotion, and then uses the feedback form to input their evaluation and suggestions for correction.

[0833] Step 13:

[0834] The terminal transmits the provided feedback data to the server.

[0835] Specifically, the feedback data is sent to the server using an HTTP POST request.

[0836] Step 14:

[0837] The server retrains the AI ​​model based on the received feedback data, improving generation accuracy.

[0838] Specifically, the feedback data is analyzed and the AI ​​model undergoes additional learning.

[0839] Example 2

[0840] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0841] With conventional methods, patients who have limited facial expression due to specific nerve or muscle disorders or the progression of dementia have difficulty expressing their emotions appropriately. Furthermore, there is a lack of technology to generate facial expressions corresponding to emotions in real time, which makes it difficult for patients to communicate smoothly with others.

[0842] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0843] In this invention, the server includes means for users to upload and store facial photo data, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on emotions selected by the user and generating facial expressions corresponding to the emotions using the AI ​​model, means for displaying the generated facial expressions on a tablet or augmented reality glasses, means for analyzing the user's voice and biometric information in real time and using an emotion engine to recognize emotions, and means for collecting feedback on the generated facial expressions and relearning the AI ​​model based on the feedback. This allows patients to appropriately express their emotions and communicate smoothly with others.

[0844] A "user" is an individual who uses the system to upload facial photo data and select an emotion.

[0845] "Facial photo data" refers to image data of a user's face that is uploaded and used as training data for the AI ​​model.

[0846] An "artificial intelligence model" is an algorithm within the system that uses machine learning techniques to generate facial expressions corresponding to emotions from facial photo data.

[0847] A "real-time image" is an image of a user's face captured instantaneously using a camera.

[0848] The "emotion engine" is part of a system that analyzes the user's voice and biometric information to recognize emotions.

[0849] A "tablet" is a portable computing device with a touch screen that is used to display generated facial expressions.

[0850] "Augmented reality glasses" are wearable devices that overlay digital information onto the user's field of vision.

[0851] "Feedback" refers to evaluations and suggestions for corrections from the user or caregiver regarding the generated facial expression.

[0852] "Retraining" is the process of updating an artificial intelligence model and improving its facial expression generation accuracy based on collected feedback.

[0853] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. This system uses facial photograph data to train an artificial intelligence model, generates facial expressions corresponding to emotions in real time, and displays them on a tablet or augmented reality glasses, thereby supporting communication between the patient and others.

[0854] Uploading and saving face photo data:

[0855] 1. The user launches the application and imports a face photo. Specifically, the user clicks the "Upload Face Photo" button in the application and selects the relevant face photo from the photo library of their device.

[0856] 2. The device compresses the selected facial photo data and sends it to the server using an HTTP POST request. For example, the destination URL is "http: / / example.com / upload."

[0857] 3. The server stores the received facial photo data and uses it as training data for the AI ​​model. Here, the received image data is converted into an appropriate format (e.g., PNG format) and stored in a database (e.g., an Amazon S3 bucket).

[0858] Training the AI ​​model:

[0859] 1. The server trains an AI model based on the stored facial photo data. At this stage, it extracts facial feature points (e.g., the position of the eyes, nose, and mouth) from the facial photo and learns various facial expressions corresponding to emotions. Specifically, it trains a CNN (convolutional neural network) model using TensorFlow and PyTorch.

[0860] Real-time image capture and facial expression generation:

[0861] 1. The user selects an emotion from the application's emotion selection menu. This is an operation in which the user opens the "Emotion Selection" menu, selects from a list of emotions such as "Happy" or "Sad," and confirms the emotion.

[0862] 2. The device starts the camera and captures images in real time. The device's camera function is started, and a face recognition algorithm (for example, OpenCV's face recognition function) is executed to detect the face and generate a captured image.

[0863] 3. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions. It uses voice recognition technology (e.g., Google Speech-to-Text) to analyze the tone and content of the user's voice. It also acquires biometric information (e.g., heart rate, sweat volume) using sensors (e.g., smartwatches) to analyze emotions.

[0864] 4. The device sends the captured real-time image and the emotion data obtained by the emotion engine to the server. The URL for sending the image and emotion data to the server via an HTTP POST request is "http: / / example.com / analyze".

[0865] 5. The server analyzes the received real-time images and emotion data and generates facial expressions using an artificial intelligence model. Specifically, it uses a pre-trained AI model (e.g., a generative adversarial network) to generate facial expressions based on facial feature points and emotion data.

[0866] 6. The server sends the generated facial image to the terminal. At this time, it returns the generated facial image to the terminal as an HTTP response. For example, it sends the image data as Content-Type: image / png.

[0867] Facial Expressions and Feedback:

[0868] 1. The device displays the generated facial image on a tablet or augmented reality glasses. A display component (e.g., React Native's Image component) is used to visually display the facial image.

[0869] 2. The user and caregiver review the displayed facial expressions and provide feedback. They confirm that the displayed image matches the emotion and use the in-app feedback form to enter their rating and suggestions for correction.

[0870] 3. The device sends the provided feedback data to the server, using the URL "http: / / example.com / feedback".

[0871] 4. The server retrains the AI ​​model based on the received feedback data to improve generation accuracy. It analyzes the feedback data and has the AI ​​model undergo additional learning. Specifically, it retrains the model using backpropagation with the additional data.

[0872] Examples:

[0873] The user launches the app on a tablet at home and uploads a recent facial photo. The device compresses the photo data and sends it to the server, which stores the data and begins training the AI ​​model. The user selects the emotion "happy" in the application, and the tablet's camera captures a facial image in real time. The emotion engine analyzes the user's voice data and confirms that the user is feeling "happy." The device sends the captured image and emotion data to the server, and the AI ​​model generates a facial expression corresponding to the emotion "happy." The server sends the generated facial image back to the device, which displays it on the tablet screen. The user's family or caregiver can review the displayed "happy" facial expression and provide feedback. The device sends the feedback to the server, which retrains the AI ​​model based on the feedback data, further improving the accuracy of facial expression generation.

[0874] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0875] Step 1: The user uploads a photo of their face.

[0876] Specific operation: The user launches the application and clicks the "Upload Face Photo" button. The user selects the appropriate face photo from the device's photo library.

[0877] Input: Facial photo data stored on the device.

[0878] Output: The selected face photo data is made available to the application.

[0879] Step 2: The device compresses the facial photo data and sends it to the server.

[0880] Specific operation: The device compresses the selected facial photo data into a format such as JPEG, and sends the facial photo data to the server using an HTTP POST request.

[0881] Input: Selected and compressed facial photo data.

[0882] Output: The compressed data is sent to the server.

[0883] Step 3: The server stores the received facial photo data.

[0884] Specific operation: The server converts the received image data into an appropriate format (e.g. PNG format) and stores the converted data in a database (e.g. Amazon S3 bucket).

[0885] Input: Compressed image data sent to the server.

[0886] Output: Image data converted and saved in the appropriate format.

[0887] Step 4: The server trains the AI ​​model based on the facial photo data.

[0888] Specific operation: The server extracts facial feature points from the stored facial photo data and trains a CNN (convolutional neural network) model using TensorFlow or PyTorch.

[0889] Input: Stored facial photo data.

[0890] Output: An AI model that has learned various facial expressions that correspond to emotions.

[0891] Step 5: The user selects an emotion.

[0892] Specific behavior: The user opens the application's "Emotion Selection" menu, selects from a list of emotions such as "Happy" or "Sad," and confirms the emotion.

[0893] Input: User emotion selection.

[0894] Output: Selected emotion data.

[0895] Step 6: The device captures the face image in real time.

[0896] Specific operation: Activate the device's camera function, run a face recognition algorithm (e.g., OpenCV) to detect the face, and generate a captured image.

[0897] Input: Real-time video.

[0898] Output: Real-time captured image.

[0899] Step 7: The emotion engine recognizes the user's emotion.

[0900] Specific operation: Using voice recognition technology (e.g., Google Speech-to-Text), the system analyzes the tone and content of the user's voice. It also uses sensors to acquire biometric information (e.g., heart rate, sweat volume, etc.) and analyzes emotions.

[0901] Input: User's voice data, biometric data.

[0902] Output: Parsed emotion data.

[0903] Step 8: The device sends the captured image and emotion data to the server.

[0904] Specific operation: The captured image and emotion data are sent to the server via an HTTP POST request. The destination URL is "http: / / example.com / analyze".

[0905] Input: Captcha image, emotion data.

[0906] Output: The data sent to the server.

[0907] Step 9: The server analyzes the received data and generates facial expressions using an AI model.

[0908] Specific operation: Based on the received image and emotion data, facial expressions are generated using a pre-trained AI model (e.g., Generative Adversarial Network).

[0909] Input: Captured image, parsed emotion data.

[0910] Output: The generated facial expression image.

[0911] Step 10: The server sends the generated facial expression image to the terminal.

[0912] Specific operation: The generated facial image is sent back to the device as an HTTP response. When sending, specify Content-Type: image / png.

[0913] Input: Generated facial expression images.

[0914] Output: Facial expression image sent back to the device.

[0915] Step 11: The device displays the generated facial expression image.

[0916] Specific behavior: Use a display component (e.g., React Native's Image component) to display the facial image on a tablet or augmented reality glasses.

[0917] Input: The returned facial expression image.

[0918] Output: Facial expression images displayed on a tablet or augmented reality glasses.

[0919] Step 12: User and caregiver provide feedback.

[0920] What it does: Check whether the displayed image matches the emotion and enter your rating and suggestions for corrections using the in-app feedback form.

[0921] Input: Feedback rating, correction suggestions.

[0922] Output: Feedback data.

[0923] Step 13: The terminal sends the feedback data to the server.

[0924] Specific operation: The feedback data is sent to the server via an HTTP POST request, with the destination URL being "http: / / example.com / feedback".

[0925] Input: Feedback data.

[0926] Output: Feedback data sent to the server.

[0927] Step 14: The server retrains the AI ​​model based on the feedback.

[0928] Specific operation: Analyzes the feedback data and allows the AI ​​model to perform additional learning. Specifically, the model is retrained using backpropagation with the additional data.

[0929] Input: Feedback data.

[0930] Output: An AI model with improved accuracy.

[0931] (Application example 2)

[0932] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0933] Patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia have difficulty expressing their emotions appropriately. As a result, communication between the patient and those around them is hindered, and the patient themselves suffers psychological stress. To solve this problem, an efficient system is needed that can generate facial expressions corresponding to the patient's emotions in real time based on photographic data of the patient's face and visually convey them to others.

[0934] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading and saving patient facial photo data, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on an emotion selected by the patient and generating a facial expression corresponding to that emotion using the AI ​​model, means for displaying the generated facial expression on a visual display device, means for analyzing voice and biometric information using an emotion engine to recognize the patient's emotion, and means for wearing an assistance device integrating a camera function and emotion analysis means for use in a specific facility. This enables the patient's emotion to be accurately expressed in real time and facilitates smooth communication with others.

[0935] "Specific nerve or muscle disorders" refers to a condition in which abnormalities in the nervous or muscular system make it difficult to move parts of the body normally.

[0936] A "system for generating and displaying facial expressions" is a set of devices and technologies that uses a computer to recognize a patient's emotions, generate a facial expression corresponding to that emotion in real time, and visually display it.

[0937] "Patient facial photograph data" refers to image data of a patient's face used for emotion analysis and facial expression generation.

[0938] An "AI model" is an artificial intelligence model that uses machine learning algorithms to learn specific patterns from data and recognize emotions and generate facial expressions.

[0939] "Real-time image capture means" refers to devices and algorithms that instantly capture and process images of the current moment.

[0940] "Visual display device" refers to a display or screen that visually presents generated expressions and information to a user, including tablets and AR glasses.

[0941] An "emotion engine" is a software or hardware component that analyzes voice and biometric information to recognize a user's emotions.

[0942] "Assistive devices" refer to devices that integrate various sensors and display devices to provide specific functions. Specifically, they include devices with camera functions and emotion analysis functions.

[0943] The present invention provides a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders to express their emotions appropriately. This system recognizes the patient's emotions, generates facial expressions in real time based on the emotions, and displays them on a visual display device to support communication. An embodiment of the system is described in detail below.

[0944] Overall system configuration

[0945] 1. Upload and save face photo data

[0946] The user uploads face photo data to the application and sends it to the server. The server saves the received face photo data and uses it as training data for the AI ​​model. Specifically, the user clicks the "Upload Face Photo" button in the application and selects the relevant face photo from the device's photo library. The selected face photo data is sent to the server as compressed data using an HTTP POST request. This process can be performed using the Python requests library or Flask as a framework.

[0947] 2. Training the AI ​​model

[0948] The server trains the AI ​​model based on the stored facial photo data. The AI ​​model used for training uses machine learning frameworks such as TensorFlow and PyTorch. It extracts feature points from the facial photos and trains the AI ​​model to recognize various facial expressions that correspond to emotions.

[0949] 3. Real-time image capture and facial expression generation

[0950] The user selects an emotion from the emotion selection menu, and the device activates the camera to capture an image in real time. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotion. The analysis uses the Emotion API and a custom machine learning model. The captured real-time image and emotional data are sent to a server, where the AI ​​model generates the corresponding facial expression.

[0951] 4. Facial Expression Display and Feedback

[0952] The generated facial images are displayed on a visual display device, specifically a tablet or AR glasses. The generated facial images are visually confirmed and the user or caregiver provides feedback. The feedback is sent to the server through a form in the application, and the AI ​​model is retrained. This improves the accuracy of facial expression generation.

[0953] System operation example

[0954] As a specific example of how it works, let's consider its use in a nursing home. A patient wears a head-mounted display, and the facility manager launches the application. When the manager selects "happy" as the patient's emotion, a real-time image of the patient is captured by a camera and analyzed by an emotion engine. The AI ​​model generates a "happy" facial expression, which is displayed on the head-mounted display. The manager checks the displayed expression and provides feedback. This process facilitates smooth communication.

[0955] Prompt Sentence Examples

[0956] For example, you can use the following prompt:

[0957] Using the "Emotion Communication Assist" app, the system analyzes the patient's emotions in real time and displays them on the head-mounted display. For example, if the patient is feeling happy, a facial expression corresponding to that emotion is displayed.

[0958] In this way, the present invention overcomes the limitations of facial expression caused by specific nerve or muscle disorders and provides a useful tool to assist communication with others.

[0959] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0960] Step 1: Upload your face photo data

[0961] A user launches the application and selects and uploads a facial photo from the device's photo library. Specifically, when the user clicks the "Upload Face Photo" button in the application and selects a facial photo, the device compresses the facial photo data and sends it to the server via an HTTP POST request. The input of this process is the facial photo data selected by the user, and the output is the compressed facial photo data stored on the server.

[0962] Step 2: Training the AI ​​model

[0963] The server trains the AI ​​model based on the uploaded facial photo data. Specifically, it uses Python's TensorFlow and PyTorch to extract feature points from the facial photos and learn various facial expressions that correspond to emotions. The input is the facial photo data and existing training data, and the output is an AI model that can recognize facial expression patterns that correspond to emotions.

[0964] Step 3: Select emotion and activate camera

[0965] The user selects an emotion from the application's emotion selection menu. The selected emotion data is input to the device. After selecting the emotion, the device activates the camera and captures an image in real time. The captured image data becomes input for real-time processing and is sent to the server.

[0966] Step 4: Recognize emotions

[0967] The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions. Voice data and biometric information such as heart rate are acquired from sensors and analyzed by the emotion engine (such as EmotionAPI). The input is the user's voice data and biometric information data, and the output is recognized emotion data.

[0968] Step 5: Facial Expression Generation

[0969] The server uses an AI model to generate facial expressions corresponding to the selected emotion based on the captured image data and emotional data. The AI ​​model utilizes previously learned patterns to convert the captured facial photo into an expression corresponding to the emotion. The input is the real-time image and recognized emotional data, and the output is the generated facial expression image.

[0970] Step 6: Displaying facial expressions

[0971] The generated facial image is sent from the server to the device. The device displays the received facial image on a display device (tablet or AR glasses). Specifically, the generated facial image is displayed immediately on the device screen. The input is the generated facial image data, and the output is the facial expression displayed on the visual display device.

[0972] Step 7: Getting feedback

[0973] The user and caregiver can check the displayed facial expressions and provide feedback. The feedback is entered using a feedback form within the application, and the device sends the data to the server. The input is the feedback data provided by the user or caregiver, and the output is the feedback data stored on the server.

[0974] Step 8: Retraining the AI ​​model

[0975] The server retrains the AI ​​model based on the received feedback data. Specifically, it analyzes the feedback data and provides additional training data to the AI ​​model to improve the accuracy of facial expression generation. The input is the feedback data and the existing AI model, and the output is the updated AI model.

[0976] Through the above steps, the present invention enables the patient's emotions to be accurately expressed in real time, enabling smooth communication with others.

[0977] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0978] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0979] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0980] [Third embodiment]

[0981] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0982] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0983] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0984] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0985] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0986] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0987] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0988] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0989] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0990] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0991] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0992] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0993] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to express their emotions appropriately. This system uses an AI model to learn from photographic data of the patient's face, generates facial expressions corresponding to the patient's emotions in real time, and displays them on a tablet or AR glasses, thereby supporting communication between the patient and others.

[0994] Overall system configuration

[0995] Uploading and saving face photo data

[0996] 1. The user launches the application and captures a facial photo.

[0997] Click the "Upload Face Photo" button on the application.

[0998] Open your device's photo library and select a photo of your face.

[0999] 2. The terminal compresses the selected facial photo data and sends it to the server.

[1000] Send the face photo data to the server using an HTTP POST request.

[1001] 3. The server stores the received facial photo data in its database.

[1002] The image data is converted into an appropriate format and saved.

[1003] Training an AI model

[1004] 1. The server trains the AI ​​model based on the stored facial photo data.

[1005] Feature points are extracted from facial photographs and various facial expressions corresponding to emotions are learned.

[1006] Real-time image capture and facial expression generation

[1007] 1. The user launches the application and selects their current emotion.

[1008] Select an emotion such as "happy" or "sad" from the "Emotion Selection" menu.

[1009] Review your selection and confirm.

[1010] 2. The device activates the camera and captures images in real time.

[1011] Use the camera to recognize faces.

[1012] 3. The device sends the captured real-time image to the server.

[1013] A real-time image is sent to the server using an HTTP POST request.

[1014] 4. The server analyzes the received real-time images and generates facial expressions using an AI model based on the selected emotion.

[1015] Extract facial feature points and generate the required facial expression.

[1016] 5. The server sends the generated facial expression image to the terminal.

[1017] The generated facial expression image is returned to the terminal as an HTTP response.

[1018] Facial Expressions and Feedback

[1019] 1. The device displays the generated facial expression image on a tablet or AR glasses.

[1020] A display component is used to visually display the facial expression image.

[1021] 2. The user and caregiver review the displayed facial expressions and provide feedback.

[1022] Check whether the image displayed matches the emotion.

[1023] If necessary, use the feedback form to provide your feedback.

[1024] 3. The device sends the feedback to the server and uses it to retrain the AI ​​model.

[1025] Send the feedback data to the server via an HTTP POST request.

[1026] 4. The server retrains the AI ​​model based on the received feedback.

[1027] The feedback data is analyzed, and the AI ​​model is retrained to improve accuracy.

[1028] Specific examples

[1029] 1. The user launches the app on their tablet and uploads a photo of their face.

[1030] 2. The device sends the photo data to the server and begins training the AI ​​model.

[1031] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[1032] 4. The device sends the captured image to the server, and the AI ​​model generates an expression corresponding to the emotion "happy."

[1033] 5. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[1034] 6. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[1035] 7. The device sends feedback to the server, which retrains the AI ​​model.

[1036] The above-described embodiment enables patients to express their emotions appropriately and supports smooth communication with others.

[1037] The processing flow will be explained below.

[1038] Step 1:

[1039] The user launches the application and uploads a photo of their face.

[1040] Specifically, click the "Upload Face Photo" button in the application and select the relevant face photo from your device's photo library.

[1041] Step 2:

[1042] The terminal compresses the facial photograph data selected by the user and transmits it to the server.

[1043] Specifically, the facial photo data is converted into an appropriate format and sent to the server using an HTTP POST request.

[1044] Step 3:

[1045] The server stores the received facial photo data and uses it as learning data for the AI ​​model.

[1046] Specifically, image data is stored in a database and initial learning is performed using an AI model inside the server.

[1047] Step 4:

[1048] The user selects an emotion from the application's emotion selection menu.

[1049] Specifically, open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[1050] Step 5:

[1051] The device activates the user's camera and captures images in real time.

[1052] Specifically, it activates the camera function, runs a facial recognition algorithm to detect a face, and captures an image of it.

[1053] Step 6:

[1054] The device transmits the captured real-time image to the server.

[1055] Specifically, the captured image is converted into an appropriate format and sent to the server using an HTTP POST request.

[1056] Step 7:

[1057] The server analyzes the received real-time images and generates facial expressions using an AI model based on the emotions selected by the user.

[1058] Specifically, facial features are extracted and an AI model is used to generate an expression that corresponds to the selected emotion.

[1059] Step 8:

[1060] The server transmits the generated facial expression image to the terminal.

[1061] Specifically, the generated facial expression image is encoded and sent back to the terminal as an HTTP response.

[1062] Step 9:

[1063] The device displays the facial expression images received from the server on a tablet or AR glasses.

[1064] Specifically, the response from the server is decoded and the facial expression image is displayed in the display component.

[1065] Step 10:

[1066] The user and caregiver review the displayed facial expressions and provide feedback as needed.

[1067] Specifically, the user checks whether the displayed facial expression image matches the emotion, and then uses the feedback form to input their evaluation and suggestions for correction.

[1068] Step 11:

[1069] The terminal transmits the provided feedback data to the server.

[1070] Specifically, the feedback data is sent to the server using an HTTP POST request.

[1071] Step 12:

[1072] The server retrains the AI ​​model based on the received feedback data, improving generation accuracy.

[1073] Specifically, the feedback data is analyzed and the AI ​​model undergoes additional learning.

[1074] Example 1

[1075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1076] Patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia have difficulty appropriately conveying their emotions to others. As a result, communication between the patient and their caregivers or family members can be hindered. The objective of the present invention is to provide a system that enables such patients to effectively express their emotions and communicate smoothly with others.

[1077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1078] In this invention, the server includes means for uploading and saving facial photo data of patients using a terminal operated by a user, means for the server to train an AI model based on the saved facial photo data, means for capturing images in real time based on an emotion selected by the patient using the terminal operated by the user and sending them to the server, means for generating facial expressions based on the real-time images received by the server and the selected emotion, and means for sending the generated facial image from the server to the terminal operated by the user and displaying it on a tablet or an augmented reality device. This allows patients to show others facial expressions that match their emotions, thereby realizing smooth communication.

[1079] A "user" is an individual or caregiver who operates the system.

[1080] A "terminal" is an electronic device operated by a user, and primarily includes smartphones, tablets, and augmented reality devices.

[1081] A "server" is a computer system that processes data and trains AI models.

[1082] "Facial image data" means image data of a patient's face that is stored and processed in digital format.

[1083] An "AI model" is a computer model that uses machine learning and deep learning techniques to generate emotion-based facial expressions.

[1084] "Means for storing" refers to a method or device for storing facial photo data in a storage device such as a database.

[1085] "Means for learning" refers to a method or apparatus for feeding data to an AI model and training the model for emotion-based facial expression generation.

[1086] "Capturing means" refers to a method or device that uses a camera function to acquire images in real time.

[1087] "Means for generating" refers to a method or device that uses an AI model to generate facial expressions based on a selected emotion.

[1088] "Transmitting means" refers to a method or apparatus for transferring data or images to another device or system.

[1089] The "display means" refers to a method or device for displaying the generated facial expression image on the screen of a terminal operated by the user or on an augmented reality device.

[1090] The "feedback function" refers to a method or device that allows the user and caregiver to provide their opinions on the displayed facial expression image to the system.

[1091] "Means for relearning" refers to methods or devices that retrain the AI ​​model based on feedback information to improve the accuracy of facial expression generation.

[1092] This invention is a system that allows patients whose facial expressions are limited due to nerve or muscle disorders or the progression of dementia to express their emotions appropriately. This system uses an AI model that learns from photographic data of the patient's face, generates facial expressions corresponding to the patient's emotions in real time, and displays them on a tablet or augmented reality device, thereby supporting communication between the patient and others.

[1093] System components include devices operated by users (e.g., smartphones, tablets, augmented reality devices), servers, databases, AI models, etc.

[1094] First, the user uploads the patient's facial photo data using a device with the application installed. Specifically, the user clicks the "Upload Face Photo" button in the application, selects a facial photo from the device's photo library, and uploads it. At this time, the device compresses the selected facial photo data and sends it to the server using an HTTP POST request.

[1095] The server stores the received facial photo data in a database. The server converts the image data into an appropriate format (e.g., PNG, JPEG) and inserts it into the database.

[1096] The server then trains an AI model based on the stored facial photo data. Specific software used includes OpenCV for detecting facial feature points and TensorFlow for building deep learning models. This process involves retrieving facial photos from the database, feeding the data to the AI ​​model, and training it to generate facial expressions corresponding to emotions.

[1097] Once the AI ​​model has completed its training, the next step is to generate the patient's facial expression in real time. The user launches the application and selects their current emotion. From the "Emotion Selection" menu, they choose an emotion such as "Happy" or "Sad" and press the confirmation button. At this time, the device's camera function is activated to capture an image in real time.

[1098] The device sends the captured real-time image to the server using an HTTP POST request. The server analyzes the received real-time image and generates a facial expression based on the selected emotion. The server extracts feature points from the received image and generates a facial expression corresponding to the selected emotion using an AI model.

[1099] The generated facial image is sent from the server to the device, which displays it on a tablet or augmented reality device. The user and caregiver can then review the displayed facial expression and provide feedback. This feedback is sent from the device to the server via an HTTP POST request. The server then retrains the AI ​​model based on the received feedback to improve the accuracy of the generated facial expressions.

[1100] As a specific example, the following procedure can be considered.

[1101] 1. The user launches the app on their tablet and uploads a photo of their face.

[1102] 2. The device sends the photo data to the server, and the server begins training the AI ​​model.

[1103] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[1104] 4. The device sends the captured image to the server, and the server's AI model generates an expression corresponding to the emotion "happy."

[1105] 5. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[1106] 6. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[1107] 7. The device sends feedback to the server, which retrains the AI ​​model.

[1108] In this way, users can effectively express their feelings and facilitate communication with others.

[1109] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1110] Step 1:

[1111] The user launches the application and uploads a face photo. The user clicks the "Upload Face Photo" button and selects a face photo from the photo library on the device.

[1112] Input: User's face photo data (e.g., an image file selected from a smartphone's photo library)

[1113] Output: Compressed face photo data

[1114] Specific actions include the user selecting a photo of their face from the smartphone gallery and pressing the upload button.

[1115] Step 2:

[1116] The device compresses the selected facial photo data and sends it to the server. It reduces the image size using a data compression algorithm and sends the data to the server using an HTTP POST request.

[1117] Input: Selected face photo data

[1118] Output: Compressed face photo data sent to the server

[1119] Specific operations include compressing the image in JPEG format and sending it in the HTTP request body.

[1120] Step 3:

[1121] The server stores the received facial photo data in a database. The server first converts the image data into an appropriate format and then inserts the data into the database.

[1122] Input: Compressed facial photo data

[1123] Output: Facial photo data stored in the database

[1124] Specific operations include converting the image format to PNG and inserting the data into the image table of the database.

[1125] Step 4:

[1126] The server trains the AI ​​model based on the stored facial photo data, extracting feature points from the facial photos and learning various facial expressions that correspond to emotions.

[1127] Input: Facial photo data stored in the database

[1128] Output: Trained AI model

[1129] Specific operations include extracting facial landmarks using OpenCV and training them with a deep learning model using TensorFlow.

[1130] Step 5:

[1131] The user starts the application and selects their current emotion by choosing an emotion such as "happy" or "sad" from the "Emotion Selection" menu and pressing the confirmation button.

[1132] Input: User-selected emotion

[1133] Output: Settings based on selected emotion

[1134] Specific actions include selecting "Happy" from the menu and tapping the confirmation button.

[1135] Step 6:

[1136] The device will activate the camera to capture images in real time. Turn on the device's camera and recognize the user's face in real time.

[1137] Input: Real-time image capture

[1138] Output: Captured face photo

[1139] Specific operations include activating the front camera and applying a facial recognition algorithm to detect the user's face.

[1140] Step 7:

[1141] The device sends the captured real-time image to the server by sending the image data to the server via an HTTP POST request.

[1142] Input: Captured face photo

[1143] Output: Real-time images sent to the server

[1144] Specific operations include converting the captured image into JPEG format and sending it in the HTTP request body.

[1145] Step 8:

[1146] The server analyzes the received real-time images and generates facial expressions based on the selected emotion. It extracts feature points from the received images and generates facial expressions corresponding to the emotion using an AI model.

[1147] Input: Real-time image, selected emotion

[1148] Output: Generated facial expression image

[1149] Specific operations include feeding the received image to the model and generating a smile corresponding to the emotion 'happy'.

[1150] Step 9:

[1151] The server sends the generated facial expression image to the terminal, and returns the generated facial expression image to the terminal as an HTTP response.

[1152] Input: Generated facial expression image

[1153] Output: Facial expression image sent to the device

[1154] Specific operations include encoding the generated facial expression image in PNG format and including it in the HTTP response body.

[1155] Step 10:

[1156] The terminal displays the generated facial expression image on a tablet or an augmented reality device. A display component is used to show the generated facial expression image to the user.

[1157] Input: Facial expression image sent from the server

[1158] Output: Displayed facial expression image

[1159] Specific operations include displaying the generated facial expression image full screen in the center of the tablet screen.

[1160] Step 11:

[1161] The user and caregiver review the displayed facial expressions and provide feedback: check whether the displayed image matches the emotion and enter their opinion in the feedback form.

[1162] Input: Displayed facial expression

[1163] Output: Feedback data

[1164] Specific actions include evaluating whether the displayed 'happy' facial expression looks natural and writing your opinion in a feedback form.

[1165] Step 12:

[1166] The device sends the feedback to the server by compressing the feedback data and sending it to the server via an HTTP POST request.

[1167] Input: Feedback data

[1168] Output: Feedback data sent to the server

[1169] Specific operations include serializing the feedback data in JSON format and sending it in the HTTP request body.

[1170] Step 13:

[1171] The server retrains the AI ​​model based on the received feedback, analyzing the feedback data and retraining the model to reflect the results.

[1172] Input: Feedback data

[1173] Output: Retrained AI model

[1174] Specific operations include analyzing the feedback data, extracting important features, adding them to the learning model, and retraining.

[1175] (Application example 1)

[1176] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1177] Individuals with limited facial expressions due to specific nerve or muscle disorders, or individuals with difficulty expressing emotions due to the progression of dementia, may find it difficult to properly convey their emotions to others, resulting in communication difficulties. This can make it difficult to provide appropriate customer service and services in physical stores. Conventional technology has difficulty generating and conveying facial expressions corresponding to emotions in real time. Another problem is that feedback functions to improve the accuracy of generated facial expressions are not sufficient.

[1178] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1179] In this invention, the server includes means for uploading and storing facial photo data of individuals, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on emotions selected by the individuals and generating facial expressions corresponding to those emotions using the AI ​​model, means for displaying the generated facial expressions on a tablet or head-mounted display, and means for analyzing the emotions received in real time and providing feedback to help the individuals appropriately express the emotions they are feeling. This enables smooth communication in physical stores even for individuals with limited facial expressions, enabling appropriate customer service and provision of services.

[1180] An "individual" is a living organism whose facial expressions may be restricted due to specific nerve or muscle disorders or the progression of dementia.

[1181] "Facial photo data" is image data used to capture an individual's facial features, and is the basic data for AI models to learn from.

[1182] An "AI model" is an artificial intelligence algorithm that uses machine learning to generate facial expressions corresponding to emotions from facial photo data.

[1183] "Real-time image capture" is the process of using the device's camera to capture current facial expressions in real-time.

[1184] "Generating facial expressions" means using an AI model to create facial expressions that correspond to emotions from captured facial expression data.

[1185] A "tablet" is a portable computing device with a flat display for visually displaying generated facial expressions.

[1186] A "head-mounted display" is a display device worn on the head, used to visually display generated facial expressions.

[1187] "Feedback" refers to evaluations and opinions provided on the generated facial expressions and emotion analysis, and is data used to retrain the AI ​​model.

[1188] This invention is a system that enables individuals whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. This system uses an AI model to learn from photographic data of the individual's face, generates facial expressions corresponding to the individual's emotions in real time, and displays them on a tablet or head-mounted display, thereby supporting communication between the individual and others.

[1189] First, the user accesses the system using a tablet or smart glasses, obtains facial photo data of the individual, and uploads it through the tablet or smart glasses. The uploaded facial photo data is stored on the server and used to train the AI ​​model.

[1190] The server uses the stored facial photo data to train the AI ​​model. It extracts facial feature points and learns facial expressions that correspond to emotions. This training process uses software such as machine learning libraries TensorFlow and PyTorch. Once training is complete, the AI ​​model analyzes the image data received in real time and generates facial expressions based on the emotions felt by the individual.

[1191] Next, the user selects an emotion in real time. The camera on the tablet or smart glasses captures the individual's facial expression in real time. This captured image data is sent to a server where it is analyzed. Based on the analyzed image data, an AI model generates an expression corresponding to the emotion. This generated expression is displayed on the tablet or head-mounted display (e.g., Google Glass, Microsoft HoloLens, etc.).

[1192] Additionally, there is a function to provide feedback on the displayed facial expressions and emotion analysis. Users and others can enter their ratings and opinions through a feedback form, which is then sent to the server. This feedback is used to retrain the AI ​​model and improve the accuracy of the generated facial expressions. This feedback loop allows the system to continuously learn and improve.

[1193] As a concrete example, the following scenario can be considered.

[1194] 1. An individual visits a physical store and a staff member is wearing the smart glasses.

[1195] 2. Staff will be able to capture the individual's facial expressions through smart glasses and see their emotions in real time.

[1196] 3. For example, if an individual looks anxious, the smart glasses' display will show "Anxious." Based on this information, the staff member will ask the individual, "Is there something that bothers you?"

[1197] 4. The staff member fills in the feedback form, stating that "the anxious individual has been dealt with," and sends it to the server. Based on this feedback, the AI ​​model undergoes retraining.

[1198] Examples of prompts for generative AI models include:

[1199] "What are your suggestions for dealing with a customer who appears anxious? Please base your suggestions on the following:

[1200] 1. They appear stressed.

[1201] 2. What was said was not clear.

[1202] 3. The tone of the voice is low.

[1203] Please explain what action you recommend and why."

[1204] This system allows individuals with limited facial expressions to express their emotions appropriately and communicate smoothly with others.

[1205] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1206] Step 1:

[1207] The user takes a facial photo of an individual using a tablet or smart glasses and uploads it through the application. As input, the user selects the facial photo data of the individual and clicks the upload button. As output, the facial photo data is compressed and then sent to the server. At this stage, image preprocessing (e.g., compression, color correction) is performed.

[1208] Step 2:

[1209] The server receives the uploaded facial photo data and stores it in a database. As input, compressed facial photo data is sent to the server. As output, the saved facial photo data is stored in the database. During this process, the image data is converted into an appropriate format.

[1210] Step 3:

[1211] The server trains the AI ​​model based on the stored facial photo data. Facial photo data retrieved from the database is provided as input to the AI ​​model. The output is a trained AI model for generating facial expressions corresponding to emotions. During this process, facial feature points are extracted and emotion tags are assigned, and the model is trained using TensorFlow, PyTorch, or similar.

[1212] Step 4:

[1213] The user selects an emotion in an application on a tablet or smart glasses. As input, they select an emotion such as "happy" or "sad" from an emotion selection menu. As output, the selected emotion is passed to the next step. This selection is made through a UI (User Interface).

[1214] Step 5:

[1215] A user uses a camera on a tablet or smart glasses to capture an individual's facial expression in real time. As input, real-time facial images are acquired by the camera. As output, the captured facial images are sent to a server. At this stage, image preprocessing (e.g., noise removal, facial region extraction) is performed.

[1216] Step 6:

[1217] The server analyzes the captured facial images in real time and generates facial expressions using an AI model based on the selected emotion. The captured facial images and the selected emotion are given to the AI ​​model as input. As output, an expression image corresponding to the selected emotion is generated. During this process, facial feature points are analyzed, and template matching and emotion mapping are performed.

[1218] Step 7:

[1219] The server sends the generated facial image to the terminal. The generated facial image is stored on the server as input. The facial image is returned to the terminal as output. Image data is sent as an HTTP response.

[1220] Step 8:

[1221] The device displays the generated facial expression images on a tablet or head-mounted display. As input, the facial expression images sent from the server are stored on the device. As output, the images are visually displayed using a display component. The user or others can check the facial expressions.

[1222] Step 9:

[1223] The user and others provide feedback based on the displayed facial expressions. As input, evaluations and opinions on the displayed facial expressions are entered into a feedback form. As output, the feedback is sent to a server. The feedback data is collected.

[1224] Step 10:

[1225] The server retrains the AI ​​model based on the received feedback. The server receives the feedback data as input. The output is a retrained AI model. The feedback is analyzed and the parameters of the AI ​​model are updated to improve accuracy.

[1226] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1227] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. By combining this invention with an emotion engine that recognizes the user's emotions, the accuracy of the generated facial expressions can be further improved. This system uses an AI model that learns from photographic data of the patient's face, generates facial expressions corresponding to the emotions in real time, and displays them on a tablet or AR glasses, thereby supporting communication between the patient and others.

[1228] Overall system configuration

[1229] Uploading and saving face photo data

[1230] 1. The user launches the application and captures a facial photo.

[1231] Click the "Upload Face Photo" button in the application and select the appropriate face photo from your device's photo library.

[1232] 2. The terminal compresses the selected facial photo data and sends it to the server.

[1233] Send the face photo data to the server using an HTTP POST request.

[1234] 3. The server stores the received facial photo data and uses it as training data for the AI ​​model.

[1235] The image data is converted into an appropriate format and stored in a database.

[1236] Training an AI model

[1237] 1. The server trains the AI ​​model based on the stored facial photo data.

[1238] Feature points are extracted from facial photographs and various facial expressions corresponding to emotions are learned.

[1239] Real-time image capture and facial expression generation

[1240] 1. The user selects an emotion from the emotion selection menu in the application.

[1241] Open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[1242] 2. The device activates the camera and captures images in real time.

[1243] The camera function is used to run a facial recognition algorithm to detect a face and capture an image of it.

[1244] 3. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions.

[1245] Voice recognition technology is used to analyze the tone and content of a user's voice.

[1246] Sensors collect biometric information (heart rate, sweat volume, etc.) and analyze emotions.

[1247] 4. The device sends the captured real-time image and the emotion data obtained by the emotion engine to the server.

[1248] Real-time images and emotion data are sent to the server using HTTP POST requests.

[1249] 5. The server analyzes the received real-time images and emotional data and generates facial expressions using an AI model.

[1250] Based on facial feature points and emotion data, the AI ​​model generates facial expressions that correspond to the selected emotion.

[1251] 6. The server sends the generated facial expression image to the terminal.

[1252] The generated facial expression image is returned to the terminal as an HTTP response.

[1253] Facial Expressions and Feedback

[1254] 1. The device displays the generated facial expression image on a tablet or AR glasses.

[1255] A display component is used to visually display the facial expression image.

[1256] 2. The user and caregiver review the displayed facial expressions and provide feedback.

[1257] Check whether the displayed image matches the emotion and use the feedback form to enter your rating and suggestions for correction.

[1258] 3. The device sends the provided feedback data to the server.

[1259] Send the feedback data to the server via an HTTP POST request.

[1260] 4. The server retrains the AI ​​model based on the received feedback data to improve generation accuracy.

[1261] The feedback data is analyzed and the AI ​​model undergoes additional learning.

[1262] Specific examples

[1263] 1. The user launches the app on their tablet and uploads a photo of their face.

[1264] 2. The device sends the photo data to the server and begins training the AI ​​model.

[1265] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[1266] 4. The emotion engine analyzes the user's voice and confirms that the user is feeling "happy."

[1267] 5. The device sends the captured image and emotional data to the server, and the AI ​​model generates an expression corresponding to the emotion "happy."

[1268] 6. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[1269] 7. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[1270] 8. The device sends feedback to the server, which retrains the AI ​​model.

[1271] The above-described embodiments enable patients to express their emotions appropriately and support smooth communication with others. By combining the emotion engine, the accuracy of facial expression generation is further improved, realizing more accurate emotional expression.

[1272] The processing flow will be explained below.

[1273] Step 1:

[1274] The user launches the application and uploads a photo of their face.

[1275] Specifically, click the "Upload Face Photo" button in the application and select the relevant face photo from your device's photo library.

[1276] Step 2:

[1277] The terminal compresses the facial photograph data selected by the user and transmits it to the server.

[1278] Specifically, the facial photo data is converted into an appropriate format and sent to the server using an HTTP POST request.

[1279] Step 3:

[1280] The server stores the received facial photo data and uses it as learning data for the AI ​​model.

[1281] Specifically, the image data is converted into an appropriate format and stored in a database.

[1282] Step 4:

[1283] The server trains an AI model based on the stored facial photo data.

[1284] Specifically, feature points are extracted from facial photographs and facial expressions corresponding to various emotions are learned.

[1285] Step 5:

[1286] The user selects an emotion from the application's emotion selection menu.

[1287] Specifically, open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[1288] Step 6:

[1289] The device will activate the camera and capture images in real time.

[1290] Specifically, it activates the camera function, runs a facial recognition algorithm to detect a face, and captures an image of it.

[1291] Step 7:

[1292] The emotion engine analyzes the user's voice and biometric information and recognizes the user's emotions (although the subject here is "emotion engine," it is included for convenience).

[1293] Specifically, it uses voice recognition technology to analyze the tone and content of the user's voice, and uses sensors to obtain biometric information (heart rate, amount of sweat, etc.) to analyze emotions.

[1294] Step 8:

[1295] The device transmits the captured real-time images and the emotion data obtained by the emotion engine to the server.

[1296] Specifically, real-time images and the emotion data from the analysis results are sent to the server using an HTTP POST request.

[1297] Step 9:

[1298] The server analyzes the received real-time images and emotional data and generates facial expressions using an AI model.

[1299] Specifically, the AI ​​model generates facial expressions corresponding to the selected emotion based on facial feature points and emotional data.

[1300] Step 10:

[1301] The server transmits the generated facial expression image to the terminal.

[1302] Specifically, the generated facial expression image is encoded and sent back to the terminal as an HTTP response.

[1303] Step 11:

[1304] The device displays the facial expression images received from the server on a tablet or AR glasses.

[1305] Specifically, the response from the server is decoded and the facial expression image is displayed in the display component.

[1306] Step 12:

[1307] The user and caregiver view the displayed facial expressions and provide feedback.

[1308] Specifically, the user checks whether the displayed facial expression image matches the emotion, and then uses the feedback form to input their evaluation and suggestions for correction.

[1309] Step 13:

[1310] The terminal transmits the provided feedback data to the server.

[1311] Specifically, the feedback data is sent to the server using an HTTP POST request.

[1312] Step 14:

[1313] The server retrains the AI ​​model based on the received feedback data, improving generation accuracy.

[1314] Specifically, the feedback data is analyzed and the AI ​​model undergoes additional learning.

[1315] Example 2

[1316] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1317] With conventional methods, patients who have limited facial expression due to specific nerve or muscle disorders or the progression of dementia have difficulty expressing their emotions appropriately. Furthermore, there is a lack of technology to generate facial expressions corresponding to emotions in real time, which makes it difficult for patients to communicate smoothly with others.

[1318] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1319] In this invention, the server includes means for users to upload and store facial photo data, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on emotions selected by the user and generating facial expressions corresponding to the emotions using the AI ​​model, means for displaying the generated facial expressions on a tablet or augmented reality glasses, means for analyzing the user's voice and biometric information in real time and using an emotion engine to recognize emotions, and means for collecting feedback on the generated facial expressions and relearning the AI ​​model based on the feedback. This allows patients to appropriately express their emotions and communicate smoothly with others.

[1320] A "user" is an individual who uses the system to upload facial photo data and select an emotion.

[1321] "Facial photo data" refers to image data of a user's face that is uploaded and used as training data for the AI ​​model.

[1322] An "artificial intelligence model" is an algorithm within the system that uses machine learning techniques to generate facial expressions corresponding to emotions from facial photo data.

[1323] A "real-time image" is an image of a user's face captured instantaneously using a camera.

[1324] The "emotion engine" is part of a system that analyzes the user's voice and biometric information to recognize emotions.

[1325] A "tablet" is a portable computing device with a touch screen that is used to display generated facial expressions.

[1326] "Augmented reality glasses" are wearable devices that overlay digital information onto the user's field of vision.

[1327] "Feedback" refers to evaluations and suggestions for corrections from the user or caregiver regarding the generated facial expression.

[1328] "Retraining" is the process of updating an artificial intelligence model and improving its facial expression generation accuracy based on collected feedback.

[1329] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. This system uses facial photograph data to train an artificial intelligence model, generates facial expressions corresponding to emotions in real time, and displays them on a tablet or augmented reality glasses, thereby supporting communication between the patient and others.

[1330] Uploading and saving face photo data:

[1331] 1. The user launches the application and imports a face photo. Specifically, the user clicks the "Upload Face Photo" button in the application and selects the relevant face photo from the photo library of their device.

[1332] 2. The device compresses the selected facial photo data and sends it to the server using an HTTP POST request. For example, the destination URL is "http: / / example.com / upload."

[1333] 3. The server stores the received facial photo data and uses it as training data for the AI ​​model. Here, the received image data is converted into an appropriate format (e.g., PNG format) and stored in a database (e.g., an Amazon S3 bucket).

[1334] Training the AI ​​model:

[1335] 1. The server trains an AI model based on the stored facial photo data. At this stage, it extracts facial feature points (e.g., the position of the eyes, nose, and mouth) from the facial photo and learns various facial expressions corresponding to emotions. Specifically, it trains a CNN (convolutional neural network) model using TensorFlow and PyTorch.

[1336] Real-time image capture and facial expression generation:

[1337] 1. The user selects an emotion from the application's emotion selection menu. This is an operation in which the user opens the "Emotion Selection" menu, selects from a list of emotions such as "Happy" or "Sad," and confirms the emotion.

[1338] 2. The device starts the camera and captures images in real time. The device's camera function is started, and a face recognition algorithm (for example, OpenCV's face recognition function) is executed to detect the face and generate a captured image.

[1339] 3. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions. It uses voice recognition technology (e.g., Google Speech-to-Text) to analyze the tone and content of the user's voice. It also acquires biometric information (e.g., heart rate, sweat volume) using sensors (e.g., smartwatches) to analyze emotions.

[1340] 4. The device sends the captured real-time image and the emotion data obtained by the emotion engine to the server. The URL for sending the image and emotion data to the server via an HTTP POST request is "http: / / example.com / analyze".

[1341] 5. The server analyzes the received real-time images and emotion data and generates facial expressions using an artificial intelligence model. Specifically, it uses a pre-trained AI model (e.g., a generative adversarial network) to generate facial expressions based on facial feature points and emotion data.

[1342] 6. The server sends the generated facial image to the terminal. At this time, it returns the generated facial image to the terminal as an HTTP response. For example, it sends the image data as Content-Type: image / png.

[1343] Facial Expressions and Feedback:

[1344] 1. The device displays the generated facial image on a tablet or augmented reality glasses. A display component (e.g., React Native's Image component) is used to visually display the facial image.

[1345] 2. The user and caregiver review the displayed facial expressions and provide feedback. They confirm that the displayed image matches the emotion and use the in-app feedback form to enter their rating and suggestions for correction.

[1346] 3. The device sends the provided feedback data to the server, using the URL "http: / / example.com / feedback".

[1347] 4. The server retrains the AI ​​model based on the received feedback data to improve generation accuracy. It analyzes the feedback data and has the AI ​​model undergo additional learning. Specifically, it retrains the model using backpropagation with the additional data.

[1348] Examples:

[1349] The user launches the app on a tablet at home and uploads a recent facial photo. The device compresses the photo data and sends it to the server, which stores the data and begins training the AI ​​model. The user selects the emotion "happy" in the application, and the tablet's camera captures a facial image in real time. The emotion engine analyzes the user's voice data and confirms that the user is feeling "happy." The device sends the captured image and emotion data to the server, and the AI ​​model generates a facial expression corresponding to the emotion "happy." The server sends the generated facial image back to the device, which displays it on the tablet screen. The user's family or caregiver can review the displayed "happy" facial expression and provide feedback. The device sends the feedback to the server, which retrains the AI ​​model based on the feedback data, further improving the accuracy of facial expression generation.

[1350] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1351] Step 1: The user uploads a photo of their face.

[1352] Specific operation: The user launches the application and clicks the "Upload Face Photo" button. The user selects the appropriate face photo from the device's photo library.

[1353] Input: Facial photo data stored on the device.

[1354] Output: The selected face photo data is made available to the application.

[1355] Step 2: The device compresses the facial photo data and sends it to the server.

[1356] Specific operation: The device compresses the selected facial photo data into a format such as JPEG, and sends the facial photo data to the server using an HTTP POST request.

[1357] Input: Selected and compressed facial photo data.

[1358] Output: The compressed data is sent to the server.

[1359] Step 3: The server stores the received facial photo data.

[1360] Specific operation: The server converts the received image data into an appropriate format (e.g. PNG format) and stores the converted data in a database (e.g. Amazon S3 bucket).

[1361] Input: Compressed image data sent to the server.

[1362] Output: Image data converted and saved in the appropriate format.

[1363] Step 4: The server trains the AI ​​model based on the facial photo data.

[1364] Specific operation: The server extracts facial feature points from the stored facial photo data and trains a CNN (convolutional neural network) model using TensorFlow or PyTorch.

[1365] Input: Stored facial photo data.

[1366] Output: An AI model that has learned various facial expressions that correspond to emotions.

[1367] Step 5: The user selects an emotion.

[1368] Specific behavior: The user opens the application's "Emotion Selection" menu, selects from a list of emotions such as "Happy" or "Sad," and confirms the emotion.

[1369] Input: User emotion selection.

[1370] Output: Selected emotion data.

[1371] Step 6: The device captures the face image in real time.

[1372] Specific operation: Activate the device's camera function, run a face recognition algorithm (e.g., OpenCV) to detect the face, and generate a captured image.

[1373] Input: Real-time video.

[1374] Output: Real-time captured image.

[1375] Step 7: The emotion engine recognizes the user's emotion.

[1376] Specific operation: Using voice recognition technology (e.g., Google Speech-to-Text), the system analyzes the tone and content of the user's voice. It also uses sensors to acquire biometric information (e.g., heart rate, sweat volume, etc.) and analyzes emotions.

[1377] Input: User's voice data, biometric data.

[1378] Output: Parsed emotion data.

[1379] Step 8: The device sends the captured image and emotion data to the server.

[1380] Specific operation: The captured image and emotion data are sent to the server via an HTTP POST request. The destination URL is "http: / / example.com / analyze".

[1381] Input: Captcha image, emotion data.

[1382] Output: The data sent to the server.

[1383] Step 9: The server analyzes the received data and generates facial expressions using an AI model.

[1384] Specific operation: Based on the received image and emotion data, facial expressions are generated using a pre-trained AI model (e.g., Generative Adversarial Network).

[1385] Input: Captured image, parsed emotion data.

[1386] Output: The generated facial expression image.

[1387] Step 10: The server sends the generated facial expression image to the terminal.

[1388] Specific operation: The generated facial image is sent back to the device as an HTTP response. When sending, specify Content-Type: image / png.

[1389] Input: Generated facial expression images.

[1390] Output: Facial expression image sent back to the device.

[1391] Step 11: The device displays the generated facial expression image.

[1392] Specific behavior: Use a display component (e.g., React Native's Image component) to display the facial image on a tablet or augmented reality glasses.

[1393] Input: The returned facial expression image.

[1394] Output: Facial expression images displayed on a tablet or augmented reality glasses.

[1395] Step 12: User and caregiver provide feedback.

[1396] What it does: Check whether the displayed image matches the emotion and enter your rating and suggestions for corrections using the in-app feedback form.

[1397] Input: Feedback rating, correction suggestions.

[1398] Output: Feedback data.

[1399] Step 13: The terminal sends the feedback data to the server.

[1400] Specific operation: The feedback data is sent to the server via an HTTP POST request, with the destination URL being "http: / / example.com / feedback".

[1401] Input: Feedback data.

[1402] Output: Feedback data sent to the server.

[1403] Step 14: The server retrains the AI ​​model based on the feedback.

[1404] Specific operation: Analyzes the feedback data and allows the AI ​​model to perform additional learning. Specifically, the model is retrained using backpropagation with the additional data.

[1405] Input: Feedback data.

[1406] Output: An AI model with improved accuracy.

[1407] (Application example 2)

[1408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1409] Patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia have difficulty expressing their emotions appropriately. As a result, communication between the patient and those around them is hindered, and the patient themselves suffers psychological stress. To solve this problem, an efficient system is needed that can generate facial expressions corresponding to the patient's emotions in real time based on photographic data of the patient's face and visually convey them to others.

[1410] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading and saving patient facial photo data, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on an emotion selected by the patient and generating a facial expression corresponding to that emotion using the AI ​​model, means for displaying the generated facial expression on a visual display device, means for analyzing voice and biometric information using an emotion engine to recognize the patient's emotion, and means for wearing an assistance device integrating a camera function and emotion analysis means for use in a specific facility. This enables the patient's emotion to be accurately expressed in real time and facilitates smooth communication with others.

[1411] "Specific nerve or muscle disorders" refers to a condition in which abnormalities in the nervous or muscular system make it difficult to move parts of the body normally.

[1412] A "system for generating and displaying facial expressions" is a set of devices and technologies that uses a computer to recognize a patient's emotions, generate a facial expression corresponding to that emotion in real time, and visually display it.

[1413] "Patient facial photograph data" refers to image data of a patient's face used for emotion analysis and facial expression generation.

[1414] An "AI model" is an artificial intelligence model that uses machine learning algorithms to learn specific patterns from data and recognize emotions and generate facial expressions.

[1415] "Real-time image capture means" refers to devices and algorithms that instantly capture and process images of the current moment.

[1416] "Visual display device" refers to a display or screen that visually presents generated expressions and information to a user, including tablets and AR glasses.

[1417] An "emotion engine" is a software or hardware component that analyzes voice and biometric information to recognize a user's emotions.

[1418] "Assistive devices" refer to devices that integrate various sensors and display devices to provide specific functions. Specifically, they include devices with camera functions and emotion analysis functions.

[1419] The present invention provides a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders to express their emotions appropriately. This system recognizes the patient's emotions, generates facial expressions in real time based on the emotions, and displays them on a visual display device to support communication. An embodiment of the system is described in detail below.

[1420] Overall system configuration

[1421] 1. Upload and save face photo data

[1422] The user uploads face photo data to the application and sends it to the server. The server saves the received face photo data and uses it as training data for the AI ​​model. Specifically, the user clicks the "Upload Face Photo" button in the application and selects the relevant face photo from the device's photo library. The selected face photo data is sent to the server as compressed data using an HTTP POST request. This process can be performed using the Python requests library or Flask as a framework.

[1423] 2. Training the AI ​​model

[1424] The server trains the AI ​​model based on the stored facial photo data. The AI ​​model used for training uses machine learning frameworks such as TensorFlow and PyTorch. It extracts feature points from the facial photos and trains the AI ​​model to recognize various facial expressions that correspond to emotions.

[1425] 3. Real-time image capture and facial expression generation

[1426] The user selects an emotion from the emotion selection menu, and the device activates the camera to capture an image in real time. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotion. The analysis uses the Emotion API and a custom machine learning model. The captured real-time image and emotional data are sent to a server, where the AI ​​model generates the corresponding facial expression.

[1427] 4. Facial Expression Display and Feedback

[1428] The generated facial images are displayed on a visual display device, specifically a tablet or AR glasses. The generated facial images are visually confirmed and the user or caregiver provides feedback. The feedback is sent to the server through a form in the application, and the AI ​​model is retrained. This improves the accuracy of facial expression generation.

[1429] System operation example

[1430] As a specific example of how it works, let's consider its use in a nursing home. A patient wears a head-mounted display, and the facility manager launches the application. When the manager selects "happy" as the patient's emotion, a real-time image of the patient is captured by a camera and analyzed by an emotion engine. The AI ​​model generates a "happy" facial expression, which is displayed on the head-mounted display. The manager checks the displayed expression and provides feedback. This process facilitates smooth communication.

[1431] Prompt Sentence Examples

[1432] For example, you can use the following prompt:

[1433] Using the "Emotion Communication Assist" app, the system analyzes the patient's emotions in real time and displays them on the head-mounted display. For example, if the patient is feeling happy, a facial expression corresponding to that emotion is displayed.

[1434] In this way, the present invention overcomes the limitations of facial expression caused by specific nerve or muscle disorders and provides a useful tool to assist communication with others.

[1435] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1436] Step 1: Upload your face photo data

[1437] A user launches the application and selects and uploads a facial photo from the device's photo library. Specifically, when the user clicks the "Upload Face Photo" button in the application and selects a facial photo, the device compresses the facial photo data and sends it to the server via an HTTP POST request. The input of this process is the facial photo data selected by the user, and the output is the compressed facial photo data stored on the server.

[1438] Step 2: Training the AI ​​model

[1439] The server trains the AI ​​model based on the uploaded facial photo data. Specifically, it uses Python's TensorFlow and PyTorch to extract feature points from the facial photos and learn various facial expressions that correspond to emotions. The input is the facial photo data and existing training data, and the output is an AI model that can recognize facial expression patterns that correspond to emotions.

[1440] Step 3: Select emotion and activate camera

[1441] The user selects an emotion from the application's emotion selection menu. The selected emotion data is input to the device. After selecting the emotion, the device activates the camera and captures an image in real time. The captured image data becomes input for real-time processing and is sent to the server.

[1442] Step 4: Recognize emotions

[1443] The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions. Voice data and biometric information such as heart rate are acquired from sensors and analyzed by the emotion engine (such as EmotionAPI). The input is the user's voice data and biometric information data, and the output is recognized emotion data.

[1444] Step 5: Facial Expression Generation

[1445] The server uses an AI model to generate facial expressions corresponding to the selected emotion based on the captured image data and emotional data. The AI ​​model utilizes previously learned patterns to convert the captured facial photo into an expression corresponding to the emotion. The input is the real-time image and recognized emotional data, and the output is the generated facial expression image.

[1446] Step 6: Displaying facial expressions

[1447] The generated facial image is sent from the server to the device. The device displays the received facial image on a display device (tablet or AR glasses). Specifically, the generated facial image is displayed immediately on the device screen. The input is the generated facial image data, and the output is the facial expression displayed on the visual display device.

[1448] Step 7: Getting feedback

[1449] The user and caregiver can check the displayed facial expressions and provide feedback. The feedback is entered using a feedback form within the application, and the device sends the data to the server. The input is the feedback data provided by the user or caregiver, and the output is the feedback data stored on the server.

[1450] Step 8: Retraining the AI ​​model

[1451] The server retrains the AI ​​model based on the received feedback data. Specifically, it analyzes the feedback data and provides additional training data to the AI ​​model to improve the accuracy of facial expression generation. The input is the feedback data and the existing AI model, and the output is the updated AI model.

[1452] Through the above steps, the present invention enables the patient's emotions to be accurately expressed in real time, enabling smooth communication with others.

[1453] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1454] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1455] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1456] [Fourth embodiment]

[1457] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1458] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1459] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1460] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1461] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1462] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1463] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1464] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1465] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1466] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1467] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1468] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1469] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1470] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to express their emotions appropriately. This system uses an AI model to learn from photographic data of the patient's face, generates facial expressions corresponding to the patient's emotions in real time, and displays them on a tablet or AR glasses, thereby supporting communication between the patient and others.

[1471] Overall system configuration

[1472] Uploading and saving face photo data

[1473] 1. The user launches the application and captures a facial photo.

[1474] Click the "Upload Face Photo" button on the application.

[1475] Open your device's photo library and select a photo of your face.

[1476] 2. The terminal compresses the selected facial photo data and sends it to the server.

[1477] Send the face photo data to the server using an HTTP POST request.

[1478] 3. The server stores the received facial photo data in its database.

[1479] The image data is converted into an appropriate format and saved.

[1480] Training an AI model

[1481] 1. The server trains the AI ​​model based on the stored facial photo data.

[1482] Feature points are extracted from facial photographs and various facial expressions corresponding to emotions are learned.

[1483] Real-time image capture and facial expression generation

[1484] 1. The user launches the application and selects their current emotion.

[1485] Select an emotion such as "happy" or "sad" from the "Emotion Selection" menu.

[1486] Review your selection and confirm.

[1487] 2. The device activates the camera and captures images in real time.

[1488] Use the camera to recognize faces.

[1489] 3. The device sends the captured real-time image to the server.

[1490] A real-time image is sent to the server using an HTTP POST request.

[1491] 4. The server analyzes the received real-time images and generates facial expressions using an AI model based on the selected emotion.

[1492] Extract facial feature points and generate the required facial expression.

[1493] 5. The server sends the generated facial expression image to the terminal.

[1494] The generated facial expression image is returned to the terminal as an HTTP response.

[1495] Facial Expressions and Feedback

[1496] 1. The device displays the generated facial expression image on a tablet or AR glasses.

[1497] A display component is used to visually display the facial expression image.

[1498] 2. The user and caregiver review the displayed facial expressions and provide feedback.

[1499] Check whether the image displayed matches the emotion.

[1500] If necessary, use the feedback form to provide your feedback.

[1501] 3. The device sends the feedback to the server and uses it to retrain the AI ​​model.

[1502] Send the feedback data to the server via an HTTP POST request.

[1503] 4. The server retrains the AI ​​model based on the received feedback.

[1504] The feedback data is analyzed, and the AI ​​model is retrained to improve accuracy.

[1505] Specific examples

[1506] 1. The user launches the app on their tablet and uploads a photo of their face.

[1507] 2. The device sends the photo data to the server and begins training the AI ​​model.

[1508] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[1509] 4. The device sends the captured image to the server, and the AI ​​model generates an expression corresponding to the emotion "happy."

[1510] 5. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[1511] 6. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[1512] 7. The device sends feedback to the server, which retrains the AI ​​model.

[1513] The above-described embodiment enables patients to express their emotions appropriately and supports smooth communication with others.

[1514] The processing flow will be explained below.

[1515] Step 1:

[1516] The user launches the application and uploads a photo of their face.

[1517] Specifically, click the "Upload Face Photo" button in the application and select the relevant face photo from your device's photo library.

[1518] Step 2:

[1519] The terminal compresses the facial photograph data selected by the user and transmits it to the server.

[1520] Specifically, the facial photo data is converted into an appropriate format and sent to the server using an HTTP POST request.

[1521] Step 3:

[1522] The server stores the received facial photo data and uses it as learning data for the AI ​​model.

[1523] Specifically, image data is stored in a database and initial learning is performed using an AI model inside the server.

[1524] Step 4:

[1525] The user selects an emotion from the application's emotion selection menu.

[1526] Specifically, open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[1527] Step 5:

[1528] The device activates the user's camera and captures images in real time.

[1529] Specifically, it activates the camera function, runs a facial recognition algorithm to detect a face, and captures an image of it.

[1530] Step 6:

[1531] The device transmits the captured real-time image to the server.

[1532] Specifically, the captured image is converted into an appropriate format and sent to the server using an HTTP POST request.

[1533] Step 7:

[1534] The server analyzes the received real-time images and generates facial expressions using an AI model based on the emotions selected by the user.

[1535] Specifically, facial features are extracted and an AI model is used to generate an expression that corresponds to the selected emotion.

[1536] Step 8:

[1537] The server transmits the generated facial expression image to the terminal.

[1538] Specifically, the generated facial expression image is encoded and sent back to the terminal as an HTTP response.

[1539] Step 9:

[1540] The device displays the facial expression images received from the server on a tablet or AR glasses.

[1541] Specifically, the response from the server is decoded and the facial expression image is displayed in the display component.

[1542] Step 10:

[1543] The user and caregiver review the displayed facial expressions and provide feedback as needed.

[1544] Specifically, the user checks whether the displayed facial expression image matches the emotion, and then uses the feedback form to input their evaluation and suggestions for correction.

[1545] Step 11:

[1546] The terminal transmits the provided feedback data to the server.

[1547] Specifically, the feedback data is sent to the server using an HTTP POST request.

[1548] Step 12:

[1549] The server retrains the AI ​​model based on the received feedback data, improving generation accuracy.

[1550] Specifically, the feedback data is analyzed and the AI ​​model undergoes additional learning.

[1551] Example 1

[1552] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1553] Patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia have difficulty appropriately conveying their emotions to others. As a result, communication between the patient and their caregivers or family members can be hindered. The objective of the present invention is to provide a system that enables such patients to effectively express their emotions and communicate smoothly with others.

[1554] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1555] In this invention, the server includes means for uploading and saving facial photo data of patients using a terminal operated by a user, means for the server to train an AI model based on the saved facial photo data, means for capturing images in real time based on an emotion selected by the patient using the terminal operated by the user and sending them to the server, means for generating facial expressions based on the real-time images received by the server and the selected emotion, and means for sending the generated facial image from the server to the terminal operated by the user and displaying it on a tablet or an augmented reality device. This allows patients to show others facial expressions that match their emotions, thereby realizing smooth communication.

[1556] A "user" is an individual or caregiver who operates the system.

[1557] A "terminal" is an electronic device operated by a user, and primarily includes smartphones, tablets, and augmented reality devices.

[1558] A "server" is a computer system that processes data and trains AI models.

[1559] "Facial image data" means image data of a patient's face that is stored and processed in digital format.

[1560] An "AI model" is a computer model that uses machine learning and deep learning techniques to generate emotion-based facial expressions.

[1561] "Means for storing" refers to a method or device for storing facial photo data in a storage device such as a database.

[1562] "Means for learning" refers to a method or apparatus for feeding data to an AI model and training the model for emotion-based facial expression generation.

[1563] "Capturing means" refers to a method or device that uses a camera function to acquire images in real time.

[1564] "Means for generating" refers to a method or device that uses an AI model to generate facial expressions based on a selected emotion.

[1565] "Transmitting means" refers to a method or apparatus for transferring data or images to another device or system.

[1566] The "display means" refers to a method or device for displaying the generated facial expression image on the screen of a terminal operated by the user or on an augmented reality device.

[1567] The "feedback function" refers to a method or device that allows the user and caregiver to provide their opinions on the displayed facial expression image to the system.

[1568] "Means for relearning" refers to methods or devices that retrain the AI ​​model based on feedback information to improve the accuracy of facial expression generation.

[1569] This invention is a system that allows patients whose facial expressions are limited due to nerve or muscle disorders or the progression of dementia to express their emotions appropriately. This system uses an AI model that learns from photographic data of the patient's face, generates facial expressions corresponding to the patient's emotions in real time, and displays them on a tablet or augmented reality device, thereby supporting communication between the patient and others.

[1570] System components include devices operated by users (e.g., smartphones, tablets, augmented reality devices), servers, databases, AI models, etc.

[1571] First, the user uploads the patient's facial photo data using a device with the application installed. Specifically, the user clicks the "Upload Face Photo" button in the application, selects a facial photo from the device's photo library, and uploads it. At this time, the device compresses the selected facial photo data and sends it to the server using an HTTP POST request.

[1572] The server stores the received facial photo data in a database. The server converts the image data into an appropriate format (e.g., PNG, JPEG) and inserts it into the database.

[1573] The server then trains an AI model based on the stored facial photo data. Specific software used includes OpenCV for detecting facial feature points and TensorFlow for building deep learning models. This process involves retrieving facial photos from the database, feeding the data to the AI ​​model, and training it to generate facial expressions corresponding to emotions.

[1574] Once the AI ​​model has completed its training, the next step is to generate the patient's facial expression in real time. The user launches the application and selects their current emotion. From the "Emotion Selection" menu, they choose an emotion such as "Happy" or "Sad" and press the confirmation button. At this time, the device's camera function is activated to capture an image in real time.

[1575] The device sends the captured real-time image to the server using an HTTP POST request. The server analyzes the received real-time image and generates a facial expression based on the selected emotion. The server extracts feature points from the received image and generates a facial expression corresponding to the selected emotion using an AI model.

[1576] The generated facial image is sent from the server to the device, which displays it on a tablet or augmented reality device. The user and caregiver can then review the displayed facial expression and provide feedback. This feedback is sent from the device to the server via an HTTP POST request. The server then retrains the AI ​​model based on the received feedback to improve the accuracy of the generated facial expressions.

[1577] As a specific example, the following procedure can be considered.

[1578] 1. The user launches the app on their tablet and uploads a photo of their face.

[1579] 2. The device sends the photo data to the server, and the server begins training the AI ​​model.

[1580] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[1581] 4. The device sends the captured image to the server, and the server's AI model generates an expression corresponding to the emotion "happy."

[1582] 5. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[1583] 6. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[1584] 7. The device sends feedback to the server, which retrains the AI ​​model.

[1585] In this way, users can effectively express their feelings and facilitate communication with others.

[1586] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1587] Step 1:

[1588] The user launches the application and uploads a face photo. The user clicks the "Upload Face Photo" button and selects a face photo from the photo library on the device.

[1589] Input: User's face photo data (e.g., an image file selected from a smartphone's photo library)

[1590] Output: Compressed face photo data

[1591] Specific actions include the user selecting a photo of their face from the smartphone gallery and pressing the upload button.

[1592] Step 2:

[1593] The device compresses the selected facial photo data and sends it to the server. It reduces the image size using a data compression algorithm and sends the data to the server using an HTTP POST request.

[1594] Input: Selected face photo data

[1595] Output: Compressed face photo data sent to the server

[1596] Specific operations include compressing the image in JPEG format and sending it in the HTTP request body.

[1597] Step 3:

[1598] The server stores the received facial photo data in a database. The server first converts the image data into an appropriate format and then inserts the data into the database.

[1599] Input: Compressed facial photo data

[1600] Output: Facial photo data stored in the database

[1601] Specific operations include converting the image format to PNG and inserting the data into the image table of the database.

[1602] Step 4:

[1603] The server trains the AI ​​model based on the stored facial photo data, extracting feature points from the facial photos and learning various facial expressions that correspond to emotions.

[1604] Input: Facial photo data stored in the database

[1605] Output: Trained AI model

[1606] Specific operations include extracting facial landmarks using OpenCV and training them with a deep learning model using TensorFlow.

[1607] Step 5:

[1608] The user starts the application and selects their current emotion by choosing an emotion such as "happy" or "sad" from the "Emotion Selection" menu and pressing the confirmation button.

[1609] Input: User-selected emotion

[1610] Output: Settings based on selected emotion

[1611] Specific actions include selecting "Happy" from the menu and tapping the confirmation button.

[1612] Step 6:

[1613] The device will activate the camera to capture images in real time. Turn on the device's camera and recognize the user's face in real time.

[1614] Input: Real-time image capture

[1615] Output: Captured face photo

[1616] Specific operations include activating the front camera and applying a facial recognition algorithm to detect the user's face.

[1617] Step 7:

[1618] The device sends the captured real-time image to the server by sending the image data to the server via an HTTP POST request.

[1619] Input: Captured face photo

[1620] Output: Real-time images sent to the server

[1621] Specific operations include converting the captured image into JPEG format and sending it in the HTTP request body.

[1622] Step 8:

[1623] The server analyzes the received real-time images and generates facial expressions based on the selected emotion. It extracts feature points from the received images and generates facial expressions corresponding to the emotion using an AI model.

[1624] Input: Real-time image, selected emotion

[1625] Output: Generated facial expression image

[1626] Specific operations include feeding the received image to the model and generating a smile corresponding to the emotion 'happy'.

[1627] Step 9:

[1628] The server sends the generated facial expression image to the terminal, and returns the generated facial expression image to the terminal as an HTTP response.

[1629] Input: Generated facial expression image

[1630] Output: Facial expression image sent to the device

[1631] Specific operations include encoding the generated facial expression image in PNG format and including it in the HTTP response body.

[1632] Step 10:

[1633] The terminal displays the generated facial expression image on a tablet or an augmented reality device. A display component is used to show the generated facial expression image to the user.

[1634] Input: Facial expression image sent from the server

[1635] Output: Displayed facial expression image

[1636] Specific operations include displaying the generated facial expression image full screen in the center of the tablet screen.

[1637] Step 11:

[1638] The user and caregiver review the displayed facial expressions and provide feedback: check whether the displayed image matches the emotion and enter their opinion in the feedback form.

[1639] Input: Displayed facial expression

[1640] Output: Feedback data

[1641] Specific actions include evaluating whether the displayed 'happy' facial expression looks natural and writing your opinion in a feedback form.

[1642] Step 12:

[1643] The device sends the feedback to the server by compressing the feedback data and sending it to the server via an HTTP POST request.

[1644] Input: Feedback data

[1645] Output: Feedback data sent to the server

[1646] Specific operations include serializing the feedback data in JSON format and sending it in the HTTP request body.

[1647] Step 13:

[1648] The server retrains the AI ​​model based on the received feedback, analyzing the feedback data and retraining the model to reflect the results.

[1649] Input: Feedback data

[1650] Output: Retrained AI model

[1651] Specific operations include analyzing the feedback data, extracting important features, adding them to the learning model, and retraining.

[1652] (Application example 1)

[1653] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1654] Individuals with limited facial expressions due to specific nerve or muscle disorders, or individuals with difficulty expressing emotions due to the progression of dementia, may find it difficult to properly convey their emotions to others, resulting in communication difficulties. This can make it difficult to provide appropriate customer service and services in physical stores. Conventional technology has difficulty generating and conveying facial expressions corresponding to emotions in real time. Another problem is that feedback functions to improve the accuracy of generated facial expressions are not sufficient.

[1655] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1656] In this invention, the server includes means for uploading and storing facial photo data of individuals, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on emotions selected by the individuals and generating facial expressions corresponding to those emotions using the AI ​​model, means for displaying the generated facial expressions on a tablet or head-mounted display, and means for analyzing the emotions received in real time and providing feedback to help the individuals appropriately express the emotions they are feeling. This enables smooth communication in physical stores even for individuals with limited facial expressions, enabling appropriate customer service and provision of services.

[1657] An "individual" is a living organism whose facial expressions may be restricted due to specific nerve or muscle disorders or the progression of dementia.

[1658] "Facial photo data" is image data used to capture an individual's facial features, and is the basic data for AI models to learn from.

[1659] An "AI model" is an artificial intelligence algorithm that uses machine learning to generate facial expressions corresponding to emotions from facial photo data.

[1660] "Real-time image capture" is the process of using the device's camera to capture current facial expressions in real-time.

[1661] "Generating facial expressions" means using an AI model to create facial expressions that correspond to emotions from captured facial expression data.

[1662] A "tablet" is a portable computing device with a flat display for visually displaying generated facial expressions.

[1663] A "head-mounted display" is a display device worn on the head, used to visually display generated facial expressions.

[1664] "Feedback" refers to evaluations and opinions provided on the generated facial expressions and emotion analysis, and is data used to retrain the AI ​​model.

[1665] This invention is a system that enables individuals whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. This system uses an AI model to learn from photographic data of the individual's face, generates facial expressions corresponding to the individual's emotions in real time, and displays them on a tablet or head-mounted display, thereby supporting communication between the individual and others.

[1666] First, the user accesses the system using a tablet or smart glasses, obtains facial photo data of the individual, and uploads it through the tablet or smart glasses. The uploaded facial photo data is stored on the server and used to train the AI ​​model.

[1667] The server uses the stored facial photo data to train the AI ​​model. It extracts facial feature points and learns facial expressions that correspond to emotions. This training process uses software such as machine learning libraries TensorFlow and PyTorch. Once training is complete, the AI ​​model analyzes the image data received in real time and generates facial expressions based on the emotions felt by the individual.

[1668] Next, the user selects an emotion in real time. The camera on the tablet or smart glasses captures the individual's facial expression in real time. This captured image data is sent to a server where it is analyzed. Based on the analyzed image data, an AI model generates an expression corresponding to the emotion. This generated expression is displayed on the tablet or head-mounted display (e.g., Google Glass, Microsoft HoloLens, etc.).

[1669] Additionally, there is a function to provide feedback on the displayed facial expressions and emotion analysis. Users and others can enter their ratings and opinions through a feedback form, which is then sent to the server. This feedback is used to retrain the AI ​​model and improve the accuracy of the generated facial expressions. This feedback loop allows the system to continuously learn and improve.

[1670] As a concrete example, the following scenario can be considered.

[1671] 1. An individual visits a physical store and a staff member is wearing the smart glasses.

[1672] 2. Staff will be able to capture the individual's facial expressions through smart glasses and see their emotions in real time.

[1673] 3. For example, if an individual looks anxious, the smart glasses' display will show "Anxious." Based on this information, the staff member will ask the individual, "Is there something that bothers you?"

[1674] 4. The staff member fills in the feedback form, stating that "the anxious individual has been dealt with," and sends it to the server. Based on this feedback, the AI ​​model undergoes retraining.

[1675] Examples of prompts for generative AI models include:

[1676] "What are your suggestions for dealing with a customer who appears anxious? Please base your suggestions on the following:

[1677] 1. They appear stressed.

[1678] 2. What was said was not clear.

[1679] 3. The tone of the voice is low.

[1680] Please explain what action you recommend and why."

[1681] This system allows individuals with limited facial expressions to express their emotions appropriately and communicate smoothly with others.

[1682] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1683] Step 1:

[1684] The user takes a facial photo of an individual using a tablet or smart glasses and uploads it through the application. As input, the user selects the facial photo data of the individual and clicks the upload button. As output, the facial photo data is compressed and then sent to the server. At this stage, image preprocessing (e.g., compression, color correction) is performed.

[1685] Step 2:

[1686] The server receives the uploaded facial photo data and stores it in a database. As input, compressed facial photo data is sent to the server. As output, the saved facial photo data is stored in the database. During this process, the image data is converted into an appropriate format.

[1687] Step 3:

[1688] The server trains the AI ​​model based on the stored facial photo data. Facial photo data retrieved from the database is provided as input to the AI ​​model. The output is a trained AI model for generating facial expressions corresponding to emotions. During this process, facial feature points are extracted and emotion tags are assigned, and the model is trained using TensorFlow, PyTorch, or similar.

[1689] Step 4:

[1690] The user selects an emotion in an application on a tablet or smart glasses. As input, they select an emotion such as "happy" or "sad" from an emotion selection menu. As output, the selected emotion is passed to the next step. This selection is made through a UI (User Interface).

[1691] Step 5:

[1692] A user uses a camera on a tablet or smart glasses to capture an individual's facial expression in real time. As input, real-time facial images are acquired by the camera. As output, the captured facial images are sent to a server. At this stage, image preprocessing (e.g., noise removal, facial region extraction) is performed.

[1693] Step 6:

[1694] The server analyzes the captured facial images in real time and generates facial expressions using an AI model based on the selected emotion. The captured facial images and the selected emotion are given to the AI ​​model as input. As output, an expression image corresponding to the selected emotion is generated. During this process, facial feature points are analyzed, and template matching and emotion mapping are performed.

[1695] Step 7:

[1696] The server sends the generated facial image to the terminal. The generated facial image is stored on the server as input. The facial image is returned to the terminal as output. Image data is sent as an HTTP response.

[1697] Step 8:

[1698] The device displays the generated facial expression images on a tablet or head-mounted display. As input, the facial expression images sent from the server are stored on the device. As output, the images are visually displayed using a display component. The user or others can check the facial expressions.

[1699] Step 9:

[1700] The user and others provide feedback based on the displayed facial expressions. As input, evaluations and opinions on the displayed facial expressions are entered into a feedback form. As output, the feedback is sent to a server. The feedback data is collected.

[1701] Step 10:

[1702] The server retrains the AI ​​model based on the received feedback. The server receives the feedback data as input. The output is a retrained AI model. The feedback is analyzed and the parameters of the AI ​​model are updated to improve accuracy.

[1703] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1704] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. By combining this invention with an emotion engine that recognizes the user's emotions, the accuracy of the generated facial expressions can be further improved. This system uses an AI model that learns from photographic data of the patient's face, generates facial expressions corresponding to the emotions in real time, and displays them on a tablet or AR glasses, thereby supporting communication between the patient and others.

[1705] Overall system configuration

[1706] Uploading and saving face photo data

[1707] 1. The user launches the application and captures a facial photo.

[1708] Click the "Upload Face Photo" button in the application and select the appropriate face photo from your device's photo library.

[1709] 2. The terminal compresses the selected facial photo data and sends it to the server.

[1710] Send the face photo data to the server using an HTTP POST request.

[1711] 3. The server stores the received facial photo data and uses it as training data for the AI ​​model.

[1712] The image data is converted into an appropriate format and stored in a database.

[1713] Training an AI model

[1714] 1. The server trains the AI ​​model based on the stored facial photo data.

[1715] Feature points are extracted from facial photographs and various facial expressions corresponding to emotions are learned.

[1716] Real-time image capture and facial expression generation

[1717] 1. The user selects an emotion from the emotion selection menu in the application.

[1718] Open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[1719] 2. The device activates the camera and captures images in real time.

[1720] The camera function is used to run a facial recognition algorithm to detect a face and capture an image of it.

[1721] 3. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions.

[1722] Voice recognition technology is used to analyze the tone and content of a user's voice.

[1723] Sensors collect biometric information (heart rate, sweat volume, etc.) and analyze emotions.

[1724] 4. The device sends the captured real-time image and the emotion data obtained by the emotion engine to the server.

[1725] Real-time images and emotion data are sent to the server using HTTP POST requests.

[1726] 5. The server analyzes the received real-time images and emotional data and generates facial expressions using an AI model.

[1727] Based on facial feature points and emotion data, the AI ​​model generates facial expressions that correspond to the selected emotion.

[1728] 6. The server sends the generated facial expression image to the terminal.

[1729] The generated facial expression image is returned to the terminal as an HTTP response.

[1730] Facial Expressions and Feedback

[1731] 1. The device displays the generated facial expression image on a tablet or AR glasses.

[1732] A display component is used to visually display the facial expression image.

[1733] 2. The user and caregiver review the displayed facial expressions and provide feedback.

[1734] Check whether the displayed image matches the emotion and use the feedback form to enter your rating and suggestions for correction.

[1735] 3. The device sends the provided feedback data to the server.

[1736] Send the feedback data to the server via an HTTP POST request.

[1737] 4. The server retrains the AI ​​model based on the received feedback data to improve generation accuracy.

[1738] The feedback data is analyzed and the AI ​​model undergoes additional learning.

[1739] Specific examples

[1740] 1. The user launches the app on their tablet and uploads a photo of their face.

[1741] 2. The device sends the photo data to the server and begins training the AI ​​model.

[1742] 3. The user selects the emotion "happy" and the tablet camera captures a facial image.

[1743] 4. The emotion engine analyzes the user's voice and confirms that the user is feeling "happy."

[1744] 5. The device sends the captured image and emotional data to the server, and the AI ​​model generates an expression corresponding to the emotion "happy."

[1745] 6. The server sends the generated facial expression image back to the device, which then displays the expression on the tablet screen.

[1746] 7. The user's family or caregiver can view the displayed "happy" facial expression and provide feedback.

[1747] 8. The device sends feedback to the server, which retrains the AI ​​model.

[1748] The above-described embodiments enable patients to express their emotions appropriately and support smooth communication with others. By combining the emotion engine, the accuracy of facial expression generation is further improved, realizing more accurate emotional expression.

[1749] The processing flow will be explained below.

[1750] Step 1:

[1751] The user launches the application and uploads a photo of their face.

[1752] Specifically, click the "Upload Face Photo" button in the application and select the relevant face photo from your device's photo library.

[1753] Step 2:

[1754] The terminal compresses the facial photograph data selected by the user and transmits it to the server.

[1755] Specifically, the facial photo data is converted into an appropriate format and sent to the server using an HTTP POST request.

[1756] Step 3:

[1757] The server stores the received facial photo data and uses it as learning data for the AI ​​model.

[1758] Specifically, the image data is converted into an appropriate format and stored in a database.

[1759] Step 4:

[1760] The server trains an AI model based on the stored facial photo data.

[1761] Specifically, feature points are extracted from facial photographs and facial expressions corresponding to various emotions are learned.

[1762] Step 5:

[1763] The user selects an emotion from the application's emotion selection menu.

[1764] Specifically, open the "Emotion Selection" menu, select an emotion such as "Happy" or "Sad," and confirm your selection.

[1765] Step 6:

[1766] The device will activate the camera and capture images in real time.

[1767] Specifically, it activates the camera function, runs a facial recognition algorithm to detect a face, and captures an image of it.

[1768] Step 7:

[1769] The emotion engine analyzes the user's voice and biometric information and recognizes the user's emotions (although the subject here is "emotion engine," it is included for convenience).

[1770] Specifically, it uses voice recognition technology to analyze the tone and content of the user's voice, and uses sensors to obtain biometric information (heart rate, amount of sweat, etc.) to analyze emotions.

[1771] Step 8:

[1772] The device transmits the captured real-time images and the emotion data obtained by the emotion engine to the server.

[1773] Specifically, real-time images and the emotion data from the analysis results are sent to the server using an HTTP POST request.

[1774] Step 9:

[1775] The server analyzes the received real-time images and emotional data and generates facial expressions using an AI model.

[1776] Specifically, the AI ​​model generates facial expressions corresponding to the selected emotion based on facial feature points and emotional data.

[1777] Step 10:

[1778] The server transmits the generated facial expression image to the terminal.

[1779] Specifically, the generated facial expression image is encoded and sent back to the terminal as an HTTP response.

[1780] Step 11:

[1781] The device displays the facial expression images received from the server on a tablet or AR glasses.

[1782] Specifically, the response from the server is decoded and the facial expression image is displayed in the display component.

[1783] Step 12:

[1784] The user and caregiver view the displayed facial expressions and provide feedback.

[1785] Specifically, the user checks whether the displayed facial expression image matches the emotion, and then uses the feedback form to input their evaluation and suggestions for correction.

[1786] Step 13:

[1787] The terminal transmits the provided feedback data to the server.

[1788] Specifically, the feedback data is sent to the server using an HTTP POST request.

[1789] Step 14:

[1790] The server retrains the AI ​​model based on the received feedback data, improving generation accuracy.

[1791] Specifically, the feedback data is analyzed and the AI ​​model undergoes additional learning.

[1792] Example 2

[1793] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1794] With conventional methods, patients who have limited facial expression due to specific nerve or muscle disorders or the progression of dementia have difficulty expressing their emotions appropriately. Furthermore, there is a lack of technology to generate facial expressions corresponding to emotions in real time, which makes it difficult for patients to communicate smoothly with others.

[1795] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1796] In this invention, the server includes means for users to upload and store facial photo data, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on emotions selected by the user and generating facial expressions corresponding to the emotions using the AI ​​model, means for displaying the generated facial expressions on a tablet or augmented reality glasses, means for analyzing the user's voice and biometric information in real time and using an emotion engine to recognize emotions, and means for collecting feedback on the generated facial expressions and relearning the AI ​​model based on the feedback. This allows patients to appropriately express their emotions and communicate smoothly with others.

[1797] A "user" is an individual who uses the system to upload facial photo data and select an emotion.

[1798] "Facial photo data" refers to image data of a user's face that is uploaded and used as training data for the AI ​​model.

[1799] An "artificial intelligence model" is an algorithm within the system that uses machine learning techniques to generate facial expressions corresponding to emotions from facial photo data.

[1800] A "real-time image" is an image of a user's face captured instantaneously using a camera.

[1801] The "emotion engine" is part of a system that analyzes the user's voice and biometric information to recognize emotions.

[1802] A "tablet" is a portable computing device with a touch screen that is used to display generated facial expressions.

[1803] "Augmented reality glasses" are wearable devices that overlay digital information onto the user's field of vision.

[1804] "Feedback" refers to evaluations and suggestions for corrections from the user or caregiver regarding the generated facial expression.

[1805] "Retraining" is the process of updating an artificial intelligence model and improving its facial expression generation accuracy based on collected feedback.

[1806] This invention is a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia to appropriately express their emotions. This system uses facial photograph data to train an artificial intelligence model, generates facial expressions corresponding to emotions in real time, and displays them on a tablet or augmented reality glasses, thereby supporting communication between the patient and others.

[1807] Uploading and saving face photo data:

[1808] 1. The user launches the application and imports a face photo. Specifically, the user clicks the "Upload Face Photo" button in the application and selects the relevant face photo from the photo library of their device.

[1809] 2. The device compresses the selected facial photo data and sends it to the server using an HTTP POST request. For example, the destination URL is "http: / / example.com / upload."

[1810] 3. The server stores the received facial photo data and uses it as training data for the AI ​​model. Here, the received image data is converted into an appropriate format (e.g., PNG format) and stored in a database (e.g., an Amazon S3 bucket).

[1811] Training the AI ​​model:

[1812] 1. The server trains an AI model based on the stored facial photo data. At this stage, it extracts facial feature points (e.g., the position of the eyes, nose, and mouth) from the facial photo and learns various facial expressions corresponding to emotions. Specifically, it trains a CNN (convolutional neural network) model using TensorFlow and PyTorch.

[1813] Real-time image capture and facial expression generation:

[1814] 1. The user selects an emotion from the application's emotion selection menu. This is an operation in which the user opens the "Emotion Selection" menu, selects from a list of emotions such as "Happy" or "Sad," and confirms the emotion.

[1815] 2. The device starts the camera and captures images in real time. The device's camera function is started, and a face recognition algorithm (for example, OpenCV's face recognition function) is executed to detect the face and generate a captured image.

[1816] 3. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions. It uses voice recognition technology (e.g., Google Speech-to-Text) to analyze the tone and content of the user's voice. It also acquires biometric information (e.g., heart rate, sweat volume) using sensors (e.g., smartwatches) to analyze emotions.

[1817] 4. The device sends the captured real-time image and the emotion data obtained by the emotion engine to the server. The URL for sending the image and emotion data to the server via an HTTP POST request is "http: / / example.com / analyze".

[1818] 5. The server analyzes the received real-time images and emotion data and generates facial expressions using an artificial intelligence model. Specifically, it uses a pre-trained AI model (e.g., a generative adversarial network) to generate facial expressions based on facial feature points and emotion data.

[1819] 6. The server sends the generated facial image to the terminal. At this time, it returns the generated facial image to the terminal as an HTTP response. For example, it sends the image data as Content-Type: image / png.

[1820] Facial Expressions and Feedback:

[1821] 1. The device displays the generated facial image on a tablet or augmented reality glasses. A display component (e.g., React Native's Image component) is used to visually display the facial image.

[1822] 2. The user and caregiver review the displayed facial expressions and provide feedback. They confirm that the displayed image matches the emotion and use the in-app feedback form to enter their rating and suggestions for correction.

[1823] 3. The device sends the provided feedback data to the server, using the URL "http: / / example.com / feedback".

[1824] 4. The server retrains the AI ​​model based on the received feedback data to improve generation accuracy. It analyzes the feedback data and has the AI ​​model undergo additional learning. Specifically, it retrains the model using backpropagation with the additional data.

[1825] Examples:

[1826] The user launches the app on a tablet at home and uploads a recent facial photo. The device compresses the photo data and sends it to the server, which stores the data and begins training the AI ​​model. The user selects the emotion "happy" in the application, and the tablet's camera captures a facial image in real time. The emotion engine analyzes the user's voice data and confirms that the user is feeling "happy." The device sends the captured image and emotion data to the server, and the AI ​​model generates a facial expression corresponding to the emotion "happy." The server sends the generated facial image back to the device, which displays it on the tablet screen. The user's family or caregiver can review the displayed "happy" facial expression and provide feedback. The device sends the feedback to the server, which retrains the AI ​​model based on the feedback data, further improving the accuracy of facial expression generation.

[1827] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1828] Step 1: The user uploads a photo of their face.

[1829] Specific operation: The user launches the application and clicks the "Upload Face Photo" button. The user selects the appropriate face photo from the device's photo library.

[1830] Input: Facial photo data stored on the device.

[1831] Output: The selected face photo data is made available to the application.

[1832] Step 2: The device compresses the facial photo data and sends it to the server.

[1833] Specific operation: The device compresses the selected facial photo data into a format such as JPEG, and sends the facial photo data to the server using an HTTP POST request.

[1834] Input: Selected and compressed facial photo data.

[1835] Output: The compressed data is sent to the server.

[1836] Step 3: The server stores the received facial photo data.

[1837] Specific operation: The server converts the received image data into an appropriate format (e.g. PNG format) and stores the converted data in a database (e.g. Amazon S3 bucket).

[1838] Input: Compressed image data sent to the server.

[1839] Output: Image data converted and saved in the appropriate format.

[1840] Step 4: The server trains the AI ​​model based on the facial photo data.

[1841] Specific operation: The server extracts facial feature points from the stored facial photo data and trains a CNN (convolutional neural network) model using TensorFlow or PyTorch.

[1842] Input: Stored facial photo data.

[1843] Output: An AI model that has learned various facial expressions that correspond to emotions.

[1844] Step 5: The user selects an emotion.

[1845] Specific behavior: The user opens the application's "Emotion Selection" menu, selects from a list of emotions such as "Happy" or "Sad," and confirms the emotion.

[1846] Input: User emotion selection.

[1847] Output: Selected emotion data.

[1848] Step 6: The device captures the face image in real time.

[1849] Specific operation: Activate the device's camera function, run a face recognition algorithm (e.g., OpenCV) to detect the face, and generate a captured image.

[1850] Input: Real-time video.

[1851] Output: Real-time captured image.

[1852] Step 7: The emotion engine recognizes the user's emotion.

[1853] Specific operation: Using voice recognition technology (e.g., Google Speech-to-Text), the system analyzes the tone and content of the user's voice. It also uses sensors to acquire biometric information (e.g., heart rate, sweat volume, etc.) and analyzes emotions.

[1854] Input: User's voice data, biometric data.

[1855] Output: Parsed emotion data.

[1856] Step 8: The device sends the captured image and emotion data to the server.

[1857] Specific operation: The captured image and emotion data are sent to the server via an HTTP POST request. The destination URL is "http: / / example.com / analyze".

[1858] Input: Captcha image, emotion data.

[1859] Output: The data sent to the server.

[1860] Step 9: The server analyzes the received data and generates facial expressions using an AI model.

[1861] Specific operation: Based on the received image and emotion data, facial expressions are generated using a pre-trained AI model (e.g., Generative Adversarial Network).

[1862] Input: Captured image, parsed emotion data.

[1863] Output: The generated facial expression image.

[1864] Step 10: The server sends the generated facial expression image to the terminal.

[1865] Specific operation: The generated facial image is sent back to the device as an HTTP response. When sending, specify Content-Type: image / png.

[1866] Input: Generated facial expression images.

[1867] Output: Facial expression image sent back to the device.

[1868] Step 11: The device displays the generated facial expression image.

[1869] Specific behavior: Use a display component (e.g., React Native's Image component) to display the facial image on a tablet or augmented reality glasses.

[1870] Input: The returned facial expression image.

[1871] Output: Facial expression images displayed on a tablet or augmented reality glasses.

[1872] Step 12: User and caregiver provide feedback.

[1873] What it does: Check whether the displayed image matches the emotion and enter your rating and suggestions for corrections using the in-app feedback form.

[1874] Input: Feedback rating, correction suggestions.

[1875] Output: Feedback data.

[1876] Step 13: The terminal sends the feedback data to the server.

[1877] Specific operation: The feedback data is sent to the server via an HTTP POST request, with the destination URL being "http: / / example.com / feedback".

[1878] Input: Feedback data.

[1879] Output: Feedback data sent to the server.

[1880] Step 14: The server retrains the AI ​​model based on the feedback.

[1881] Specific operation: Analyzes the feedback data and allows the AI ​​model to perform additional learning. Specifically, the model is retrained using backpropagation with the additional data.

[1882] Input: Feedback data.

[1883] Output: An AI model with improved accuracy.

[1884] (Application example 2)

[1885] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1886] Patients whose facial expressions are limited due to specific nerve or muscle disorders or the progression of dementia have difficulty expressing their emotions appropriately. As a result, communication between the patient and those around them is hindered, and the patient themselves suffers psychological stress. To solve this problem, an efficient system is needed that can generate facial expressions corresponding to the patient's emotions in real time based on photographic data of the patient's face and visually convey them to others.

[1887] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading and saving patient facial photo data, means for an AI model to learn based on the uploaded facial photo data, means for capturing images in real time based on an emotion selected by the patient and generating a facial expression corresponding to that emotion using the AI ​​model, means for displaying the generated facial expression on a visual display device, means for analyzing voice and biometric information using an emotion engine to recognize the patient's emotion, and means for wearing an assistance device integrating a camera function and emotion analysis means for use in a specific facility. This enables the patient's emotion to be accurately expressed in real time and facilitates smooth communication with others.

[1888] "Specific nerve or muscle disorders" refers to a condition in which abnormalities in the nervous or muscular system make it difficult to move parts of the body normally.

[1889] A "system for generating and displaying facial expressions" is a set of devices and technologies that uses a computer to recognize a patient's emotions, generate a facial expression corresponding to that emotion in real time, and visually display it.

[1890] "Patient facial photograph data" refers to image data of a patient's face used for emotion analysis and facial expression generation.

[1891] An "AI model" is an artificial intelligence model that uses machine learning algorithms to learn specific patterns from data and recognize emotions and generate facial expressions.

[1892] "Real-time image capture means" refers to devices and algorithms that instantly capture and process images of the current moment.

[1893] "Visual display device" refers to a display or screen that visually presents generated expressions and information to a user, including tablets and AR glasses.

[1894] An "emotion engine" is a software or hardware component that analyzes voice and biometric information to recognize a user's emotions.

[1895] "Assistive devices" refer to devices that integrate various sensors and display devices to provide specific functions. Specifically, they include devices with camera functions and emotion analysis functions.

[1896] The present invention provides a system that allows patients whose facial expressions are limited due to specific nerve or muscle disorders to express their emotions appropriately. This system recognizes the patient's emotions, generates facial expressions in real time based on the emotions, and displays them on a visual display device to support communication. An embodiment of the system is described in detail below.

[1897] Overall system configuration

[1898] 1. Upload and save face photo data

[1899] The user uploads face photo data to the application and sends it to the server. The server saves the received face photo data and uses it as training data for the AI ​​model. Specifically, the user clicks the "Upload Face Photo" button in the application and selects the relevant face photo from the device's photo library. The selected face photo data is sent to the server as compressed data using an HTTP POST request. This process can be performed using the Python requests library or Flask as a framework.

[1900] 2. Training the AI ​​model

[1901] The server trains the AI ​​model based on the stored facial photo data. The AI ​​model used for training uses machine learning frameworks such as TensorFlow and PyTorch. It extracts feature points from the facial photos and trains the AI ​​model to recognize various facial expressions that correspond to emotions.

[1902] 3. Real-time image capture and facial expression generation

[1903] The user selects an emotion from the emotion selection menu, and the device activates the camera to capture an image in real time. The emotion engine analyzes the user's voice and biometric information to recognize the user's emotion. The analysis uses the Emotion API and a custom machine learning model. The captured real-time image and emotional data are sent to a server, where the AI ​​model generates the corresponding facial expression.

[1904] 4. Facial Expression Display and Feedback

[1905] The generated facial images are displayed on a visual display device, specifically a tablet or AR glasses. The generated facial images are visually confirmed and the user or caregiver provides feedback. The feedback is sent to the server through a form in the application, and the AI ​​model is retrained. This improves the accuracy of facial expression generation.

[1906] System operation example

[1907] As a specific example of how it works, let's consider its use in a nursing home. A patient wears a head-mounted display, and the facility manager launches the application. When the manager selects "happy" as the patient's emotion, a real-time image of the patient is captured by a camera and analyzed by an emotion engine. The AI ​​model generates a "happy" facial expression, which is displayed on the head-mounted display. The manager checks the displayed expression and provides feedback. This process facilitates smooth communication.

[1908] Prompt Sentence Examples

[1909] For example, you can use the following prompt:

[1910] Using the "Emotion Communication Assist" app, the system analyzes the patient's emotions in real time and displays them on the head-mounted display. For example, if the patient is feeling happy, a facial expression corresponding to that emotion is displayed.

[1911] In this way, the present invention overcomes the limitations of facial expression caused by specific nerve or muscle disorders and provides a useful tool to assist communication with others.

[1912] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1913] Step 1: Upload your face photo data

[1914] A user launches the application and selects and uploads a facial photo from the device's photo library. Specifically, when the user clicks the "Upload Face Photo" button in the application and selects a facial photo, the device compresses the facial photo data and sends it to the server via an HTTP POST request. The input of this process is the facial photo data selected by the user, and the output is the compressed facial photo data stored on the server.

[1915] Step 2: Training the AI ​​model

[1916] The server trains the AI ​​model based on the uploaded facial photo data. Specifically, it uses Python's TensorFlow and PyTorch to extract feature points from the facial photos and learn various facial expressions that correspond to emotions. The input is the facial photo data and existing training data, and the output is an AI model that can recognize facial expression patterns that correspond to emotions.

[1917] Step 3: Select emotion and activate camera

[1918] The user selects an emotion from the application's emotion selection menu. The selected emotion data is input to the device. After selecting the emotion, the device activates the camera and captures an image in real time. The captured image data becomes input for real-time processing and is sent to the server.

[1919] Step 4: Recognize emotions

[1920] The emotion engine analyzes the user's voice and biometric information to recognize the user's emotions. Voice data and biometric information such as heart rate are acquired from sensors and analyzed by the emotion engine (such as EmotionAPI). The input is the user's voice data and biometric information data, and the output is recognized emotion data.

[1921] Step 5: Facial Expression Generation

[1922] The server uses an AI model to generate facial expressions corresponding to the selected emotion based on the captured image data and emotional data. The AI ​​model utilizes previously learned patterns to convert the captured facial photo into an expression corresponding to the emotion. The input is the real-time image and recognized emotional data, and the output is the generated facial expression image.

[1923] Step 6: Displaying facial expressions

[1924] The generated facial image is sent from the server to the device. The device displays the received facial image on a display device (tablet or AR glasses). Specifically, the generated facial image is displayed immediately on the device screen. The input is the generated facial image data, and the output is the facial expression displayed on the visual display device.

[1925] Step 7: Getting feedback

[1926] The user and caregiver can check the displayed facial expressions and provide feedback. The feedback is entered using a feedback form within the application, and the device sends the data to the server. The input is the feedback data provided by the user or caregiver, and the output is the feedback data stored on the server.

[1927] Step 8: Retraining the AI ​​model

[1928] The server retrains the AI ​​model based on the received feedback data. Specifically, it analyzes the feedback data and provides additional training data to the AI ​​model to improve the accuracy of facial expression generation. The input is the feedback data and the existing AI model, and the output is the updated AI model.

[1929] Through the above steps, the present invention enables the patient's emotions to be accurately expressed in real time, enabling smooth communication with others.

[1930] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1931] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1932] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1933] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1934] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1935] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1936] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1937] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1938] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1939] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1940] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1941] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1942] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1943] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1944] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1945] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, ...

Claims

1. A system for generating and displaying facial expressions for a patient whose facial expressions are limited due to a specific nerve or muscle disorder, comprising: A means for uploading and storing patient facial photograph data; A method for the AI ​​model to learn based on uploaded facial photo data, a means for capturing images in real time and generating facial expressions corresponding to the patient's selected emotion using an AI model; A means to display the generated facial expressions on a tablet or AR glasses, A system including:

2. 10. The system of claim 1, further comprising a feedback function for improving communication between the patient and others and visually conveying the patient's emotions.

3. 2. The system of claim 1, further comprising means for retraining the AI ​​model based on feedback to improve the accuracy of the generated facial expressions.

Images