System

Abstract

A system is provided.SOLUTION: The system includes a means for uploading image data and moving image data from a user terminal, a means for analyzing the image data and the moving image data on a server and diagnosing the personality of a user, a means for generating a 3D model based on the result of the personality diagnosis, a means for transferring the 3D model to the user terminal and displaying it, a means for generating a natural language conversation between the user and the 3D model via the user terminal, and a means for recording the conversation contents and transmitting a regular report to a registered family.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the number of elderly people living alone has been rapidly increasing with the progress of the aging society. As a result, elderly people living alone often experience a sense of isolation and mental stress due to the lack of conversation and social connections within the home. Therefore, there is a need to provide an environment where elderly people living alone can live safely. However, many current systems are limited to simple voice dialogue and the provision of limited information, making it difficult to realize the same kind of support as that provided by real family members. A method to overcome this issue is needed. [Means for solving the problem]

[0005] This invention targets elderly people living alone and provides a means for uploading image and video data from a user terminal, a means for analyzing the data on a server to assess their personality, and a means for generating a 3D model based on the assessment results. The 3D model is then transferred to the user terminal and displayed. The system also includes a means for generating a natural language dialogue between the user and the 3D model via the user terminal, and a means for recording the dialogue and sending periodic reports to family members. This allows elderly people living alone to constantly enjoy natural dialogue with someone who resembles a family member, reducing feelings of loneliness and providing emotional support. Furthermore, periodic reports to family members allow family members, even those in remote locations, to easily understand the user's condition.

[0006] A "user terminal" is an electronic device operated by a user, such as a smartphone, tablet, or personal computer.

[0007] "Image data" refers to digital image files, such as photographs and still images, that are made up of information in units of pixels.

[0008] "Video data" refers to a file in which a series of images are displayed consecutively along a time axis, and may also include audio.

[0009] A "server" is a computer system dedicated to storing, processing, and transmitting data over a network.

[0010] "Personality diagnosis" is the process of analyzing a user's characteristics and personality based on image and video data, and then creating a profile based on the results.

[0011] A "3D model" is digital data that reproduces the shape and appearance of an object or person in three-dimensional space.

[0012] "Transfer" is the process of moving data or information from one device to another.

[0013] "Natural language dialogue" is a form of communication in which a computer system and a user exchange words using a language that humans normally use.

[0014] "Dialogue content" refers to specific text and voice data of utterances and responses in a dialogue using natural language.

[0015] A "periodic report" is a document or data file that is generated at regular intervals and reports specific information in a consolidated manner. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that enables simulated conversations with family members using a user terminal, a server, and a generation AI to reduce the sense of isolation felt by elderly people. Specific embodiments for implementing this system are described below.

[0038] Uploading photos and videos

[0039] User:

[0040] A user selects and uploads family photos and videos using a user device such as a smartphone or tablet. As a specific example, a user selects five family photos and a 30-second video from the smartphone gallery.

[0041] Device:

[0042] The user's device temporarily stores the selected photos and videos in a buffer and then transmits them to the server via the Internet. Once the photos and videos have been transmitted, the device displays a confirmation message to the user.

[0043] Personality Assessment

[0044] server:

[0045] The server analyzes the received photos and videos. Using image recognition and voice analysis technology, it extracts the family members' physical features and personality traits. For example, it recognizes facial features from still images and analyzes the tone of voice and speaking style from videos. This allows it to extract traits such as a "gentle and kind personality."

[0046] 3D avatar generation

[0047] server:

[0048] The server uses a 3D modeling tool based on the analysis results to generate 3D avatars of the family. These avatars combine appearance information and personality assessment results, and resemble the appearance and behavior of the actual family. The generated avatars are then transferred to the user's device.

[0049] Device:

[0050] The user device decompresses the 3D avatar received from the server and displays it on the application screen. The user can move and change the orientation of the avatar by touching it.

[0051] Natural language dialogue

[0052] User:

[0053] The user initiates a dialogue with the 3D avatar through the application, for example by asking, "How was your day today?" This input is recorded as text and / or audio.

[0054] Device:

[0055] The user terminal converts the voice input into text data and sends it to the server, or if the input is text, sends it as is to the server.

[0056] server:

[0057] The server analyzes the received text data and uses a natural language processing (NLP) engine to understand its meaning. It generates an appropriate response and sends the text data back to the device. For example, it generates a response such as, "The weather was nice today, so I went for a walk."

[0058] Device:

[0059] The user device converts the response received from the server into voice output for the 3D avatar, which responds to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[0060] Generate and send scheduled reports

[0061] server:

[0062] The server periodically analyzes the conversations between users and their 3D avatars to identify specific topics and emotional trends. For example, it might discover from a week's worth of conversation logs that "health" and "weather" are frequently discussed topics.

[0063] server:

[0064] Based on the extracted information, periodic reports are generated that include information on the frequency of interactions, topic trends, and the user's emotional state.

[0065] server:

[0066] The generated regular reports are sent to the registered email addresses of family members, allowing family members in remote locations to easily keep track of the user's status.

[0067] This system allows elderly people living alone to enjoy natural conversation with someone who resembles a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to feel at ease and keep track of the user's condition.

[0068] The processing flow will be explained below.

[0069] Uploading photos and videos

[0070] Step 1: User Selects Media File

[0071] User:

[0072] Open the app on your smartphone or tablet and select photos and videos of your family: five photos and a 30-second video from your gallery.

[0073] Step 2: Upload your media files

[0074] Device:

[0075] The selected media file is temporarily stored in a buffer and then sent over the Internet to a server. Once the file transfer is complete, the device displays a confirmation message to the user saying "Upload Complete."

[0076] Personality Assessment

[0077] Step 3: Receive and save media files

[0078] server:

[0079] Receives uploaded photos and videos, stores them in the server's storage, records the storage path, and generates metadata for further processing.

[0080] Step 4: Analysis and personality assessment

[0081] server:

[0082] The system analyzes stored photos and videos. It uses image recognition and voice analysis technology to extract personality traits from family members' appearances and voices. For example, a "gentle and kind personality" can be extracted from the voice of a video.

[0083] 3D avatar generation

[0084] Step 5: Preparing for 3D modeling

[0085] server:

[0086] Based on the personality test results and the characteristics of the media file, a 3D modeling tool is used to generate 3D avatars of your family members, combining appearance and personality traits to create the perfect avatar.

[0087] Step 6: Saving and transferring your avatar

[0088] server:

[0089] The generated 3D avatar is stored on the server and transferred to the user's device. The avatar data is compressed to enable efficient transfer.

[0090] Step 7: Receiving and displaying your avatar

[0091] Device:

[0092] The app receives the 3D avatar transferred from the server, decompresses it, and displays it in the app. The user can move and rotate the avatar using touch controls.

[0093] Natural language dialogue

[0094] Step 8: Initiating a conversation

[0095] User:

[0096] The user initiates a conversation with the 3D avatar through the app, specifically by asking, "How was your day?"

[0097] Step 9: Submitting input data

[0098] Device:

[0099] The user's voice input is converted into text data and sent to the server. In addition, if the input is text, it is sent to the server as is.

[0100] Step 10: Analyzing the interaction data

[0101] server:

[0102] It analyzes the received text data, understands its meaning using a natural language processing (NLP) engine, generates an appropriate response, and sends the text data back to the device.

[0103] Step 11: View the response

[0104] Device:

[0105] The response received from the server is converted into voice output by the 3D avatar and returned to the user as a dialogue. For example, the avatar might respond with a natural expression, "The weather was nice today, so I went for a walk."

[0106] Generate and send scheduled reports

[0107] Step 12: Recording interaction history

[0108] server:

[0109] The interaction history between the user and the 3D avatar is periodically collected and stored in a database, and the frequency and content of the interactions are organized and recorded.

[0110] Step 13: Generate reports

[0111] server:

[0112] The system analyzes conversation history to extract specific topics and sentiment trends, summarizing major topics such as "health" and "weather" on a weekly basis.

[0113] Step 14: Submitting the report

[0114] server:

[0115] The generated periodic reports are sent to registered family members via email or other means, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[0116] This process allows elderly people living alone to enjoy natural conversation with people who resemble family members, reducing feelings of loneliness and providing emotional care. In addition, regular reports to families allow them to understand the user's condition with peace of mind.

[0117] Example 1

[0118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0119] Elderly people tend to feel isolated and require psychological care. Furthermore, families living far away have limited means of understanding the situation of their elderly relatives, making it difficult for them to provide support for their lives with peace of mind. Current technology does not allow for direct communication with family members, and there is a lack of mechanisms for providing detailed information about the elderly's living situation to family members.

[0120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0121] In this invention, the server includes means for analyzing the received image data and video data, means for extracting the user's appearance features and personality traits based on the analysis, and means for generating a 3D model based on the personality diagnosis results. This reduces the elderly person's sense of isolation through natural conversation with a presence that resembles a family member, and by recording the content of the conversation and sending regular reports to the family, it becomes easier for the family to understand the elderly person's condition even remotely.

[0122] A "user terminal" is an electronic device such as a smartphone, tablet, or personal computer that is operated by a user.

[0123] "Image data" refers to still image data uploaded from a user terminal.

[0124] "Video data" refers to dynamic video data uploaded from a user terminal.

[0125] A "buffer memory" is a temporary storage area for data, and is used to increase the efficiency of data transfer.

[0126] The "Internet" is a huge computer network that connects computer networks all over the world.

[0127] A "server" is a computer or software that provides services to other computers over a network.

[0128] "Analysis" is the act of processing image data and video data to extract their contents and characteristics.

[0129] "Appearance features" refer to visual features such as a person's face, posture, and clothing that are extracted from images and videos.

[0130] "Personality traits" refer to a person's personality characteristics obtained through voice analysis and behavioral analysis.

[0131] "3D Model" means a three-dimensional digital character generated by the server and simulated for the user.

[0132] A "generative AI model" is an artificial intelligence algorithm that generates new data based on specific patterns or data.

[0133] A "prompt" is a sentence or question that is input into a generative AI model, and is the input data that the AI ​​uses to generate responses and data.

[0134] A "natural language processing engine" is an algorithm or software that understands and appropriately processes human language.

[0135] "Voice input" refers to the process of converting words spoken by a user into digital data through a microphone.

[0136] "Text data" refers to data obtained by converting voice input into text information.

[0137] "Speech conversion" refers to the process of converting text data back into speech.

[0138] A "regular report" is a document that is generated periodically to inform family members of the user's conversation content and emotional tendencies.

[0139] The present invention is a system that enables simulated conversations with family members using a user terminal, a server, and a generation AI to reduce the sense of isolation felt by elderly people. Specific embodiments for implementing this system are described below.

[0140] Uploading photos and videos

[0141] User:

[0142] Users select and upload photos and videos of their family using their smartphone, tablet, or other device. Specifically, users access the gallery through the app interface and select five photos and a 30-second video.

[0143] Device:

[0144] The user device temporarily stores the selected photos and videos in a buffer memory. Then, it sends this data to the server via an HTTP POST request. During this process, the user device displays a progress bar to indicate the progress, and once the transfer is complete, it displays the message "Upload complete" to the user.

[0145] Personality Assessment

[0146] server:

[0147] The server receives the photo and video data sent from the device and stores it in cloud storage (e.g., AWS® S3). It then analyzes the data using image recognition (e.g., OpenCV) and voice analysis (e.g., Google® Cloud Speech-to-Text) technologies to extract the family's physical features and personality traits.

[0148] 3D avatar generation

[0149] server:

[0150] Based on the appearance and voice characteristics discovered from the analysis, the server applies a personality diagnosis algorithm (e.g., decision tree or neural network) to derive characteristics such as a "gentle and kind personality."

[0151] server:

[0152] Then, 3D avatars of the family are generated using 3D modeling software (e.g., Blender), recreating facial contours, hairstyles, and clothing styles, and adding facial expressions and movements based on their personalities. The resulting 3D avatars are then sent to the user's device in a compressed format, using the SSL / TLS protocol to ensure secure data transfer.

[0153] Device:

[0154] The user device decompresses the received 3D avatar data and displays it in the application's 3D view. The user can change the orientation of the avatar by touching or swiping the screen.

[0155] Natural language dialogue

[0156] User:

[0157] The user initiates a dialogue with the 3D avatar using voice or text input, for example, by asking, "How was your day today?"

[0158] Device:

[0159] The user device converts the voice input into text data via an API (e.g., Google Cloud Speech-to-Text). If the input is text, it is sent directly to the server.

[0160] server:

[0161] The server analyzes the received text data and generates an appropriate response using a generative AI model (e.g., GPT-3 (registered trademark)). For example, it generates a sentence such as, "The weather was nice today, so I went for a walk."

[0162] Device:

[0163] The user device converts the response text received from the server into speech using a speech synthesis API (e.g., Amazon Polly) and synchronizes it with the mouth movements of the 3D avatar, allowing the user to enjoy natural conversations through the avatar.

[0164] Generate and send scheduled reports

[0165] server:

[0166] The server periodically analyzes the conversation between the user and the 3D avatar to extract trends in topics and emotions. Specifically, it uses an NLP engine to classify the conversation logs by topic and evaluates the user's emotional state using an emotion analysis algorithm.

[0167] server:

[0168] Based on the analysis results, regular reports are generated that include information such as the frequency of conversations, main topics, and emotional state, and are displayed in easy-to-read graphs and charts.

[0169] server:

[0170] The generated periodic reports are automatically sent to the family's email addresses. The server uses the SMTP protocol to realize the email sending function.

[0171] For example, some input prompts for a generative AI model might include:

[0172] "Generate a 3D avatar of a very kind man based on the following dialogue: 'Hi, Grandpa! What were you doing today?' 'Hi, Grandma! It's very cool today, so I was reading in the garden.'"

[0173] This system allows elderly people to enjoy natural conversation with someone who resembles a family member, reducing feelings of isolation. It also makes it easier for family members to keep track of the user's condition through regular reports.

[0174] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0175] Step 1: User selection of photos and videos

[0176] User:

[0177] The user launches the app on a smartphone, tablet, or other device, and operates the app's interface to select five family photos and a 30-second video from the gallery.

[0178] Input: Five photos of your family and a 30-second video.

[0179] Output: Selected photos and videos.

[0180] Step 2: Upload photos and videos

[0181] Device:

[0182] The user's device temporarily stores the selected photos and videos in a buffer memory, then sends the data to the server via an HTTP POST request, displaying a progress bar during the transfer and a message indicating the upload is complete when the transfer is complete.

[0183] Input: Photos and videos stored in buffer memory.

[0184] Output: A completion message sent to the server.

[0185] Step 3: Receiving and analyzing image and video data

[0186] server:

[0187] The server receives image and video data sent from the device and stores it in cloud storage (e.g., AWS S3). It then analyzes the data using image recognition (e.g., OpenCV) and voice analysis (e.g., Google Cloud Speech-to-Text) technologies to extract the family's physical features and personality traits.

[0188] Input: Received image and video data.

[0189] Output: Analysis of family members' physical features and personality traits.

[0190] Step 4: Applying the personality assessment algorithm

[0191] server:

[0192] The server applies a personality diagnosis algorithm (e.g., decision tree or neural network) based on the appearance and voice characteristics obtained from the analysis results to derive characteristics such as a "gentle and kind personality."

[0193] Input: Physical and vocal characteristics of family members.

[0194] Output: Personality test results.

[0195] Step 5: Generate a 3D avatar

[0196] server:

[0197] The server uses 3D modeling software (e.g., Blender) to generate 3D avatars of the family based on the personality test results and appearance data, and the generated avatars are sent to the user's device in a compressed format.

[0198] Input: Personality test results and physical characteristics data.

[0199] Output: Generated 3D avatar data.

[0200] Step 6: Displaying the 3D Avatar

[0201] Device:

[0202] The user device decompresses the received 3D avatar data and displays it in the application's 3D view. The user can change the orientation of the avatar by touching or swiping the screen.

[0203] Input: Received 3D avatar data.

[0204] Output: 3D avatar displayed on the application screen.

[0205] Step 7: Initiating a natural language dialogue

[0206] User:

[0207] The user initiates a dialogue with the 3D avatar through voice or text input, for example, by asking, "How was your day today?"

[0208] Input: Voice or text input.

[0209] Output: The input audio or text data.

[0210] Step 8: Process and transmit voice / text data

[0211] Device:

[0212] The user device converts the voice input into text data via an API (e.g., Google Cloud Speech-to-Text). If the input is text, it is sent directly to the server.

[0213] Input: Audio or text data.

[0214] Output: The text data sent to the server.

[0215] Step 9: Generate a response

[0216] server:

[0217] The server analyzes the received text data and generates an appropriate response using a generative AI model (e.g., GPT-3), such as "The weather was nice today, so I went for a walk."

[0218] Input: Text data.

[0219] Output: The generated response text.

[0220] Step 10: Returning a response and converting it

[0221] Device:

[0222] The user device converts the response text received from the server into speech using a speech synthesis API (e.g., Amazon Polly) and synchronizes it with the mouth movements of the 3D avatar, allowing the user to enjoy natural conversations through the avatar.

[0223] Input: The response text received from the server.

[0224] Output: Voice response by avatar.

[0225] Step 11: Record and analyze the conversation

[0226] server:

[0227] The server periodically analyzes the conversation between the user and the 3D avatar to extract trends in topics and emotions. Specifically, it uses an NLP engine to classify the conversation logs by topic and evaluates the user's emotional state using an emotion analysis algorithm.

[0228] Input: Log data of the conversation.

[0229] Output: Trend analysis results of topics and sentiment.

[0230] Step 12: Generate and send scheduled reports

[0231] server:

[0232] The server generates periodic reports based on the analysis results and automatically sends them to the family's email addresses, including information on the frequency of conversations, main topics, and emotional state.

[0233] Input: Topic and sentiment trend analysis results.

[0234] Output: Regular reports sent to families.

[0235] This allows the elderly to enjoy natural conversation with people who resemble family members, reducing feelings of loneliness.Families can also keep track of the elderly's condition through regular reports, even when they are far away.

[0236] (Application example 1)

[0237] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0238] There are existing systems that provide simulated dialogue with family members to reduce the sense of isolation felt by elderly people. However, these systems make it difficult for users to enjoy a sophisticated dialogue experience, such as receiving advice from family members in more specific life situations. Furthermore, there is a lack of systems that can generate specific dialogue responses tailored to specific situations. There is a growing demand for solutions to this problem.

[0239] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0240] In this invention, the server includes means for uploading image data and video data from a user terminal, means for analyzing the image data and video data on the server and diagnosing the user's personality, means for generating a 3D model based on the personality diagnosis results, means for transferring the 3D model to a head-mounted display terminal and displaying it, means for generating a natural language dialogue between the user and the 3D model via the head-mounted display terminal, and means for recording the content of the dialogue and sending regular reports to registered family members. This allows elderly people to experience more realistic dialogue with their families and receive appropriate advice in specific life situations.

[0241] A "user terminal" is an electronic device that allows a user to take photos and videos and upload them to a server.

[0242] "Image data and video data" refers to photographs and video files taken or selected by a user using a user terminal.

[0243] A "server" is a remote computer system that receives image and video data sent from a user terminal and analyzes and processes it.

[0244] "Personality diagnosis" is the process of identifying the subject's personality and characteristics by analyzing image and video data.

[0245] A "3D model" is a three-dimensional avatar generated on a computer based on the results of analysis, and is a digital object that reflects a person's appearance and personality.

[0246] A "head-mounted display terminal" is a display device worn on the head, and is a device that provides VR (virtual reality) and AR (augmented reality) experiences.

[0247] "Natural language interaction" is communication between humans and computers in which the user speaks in spoken language and the system responds.

[0248] A "regular report" is a document that the system records and analyzes the content of conversations and periodically reports on the user's status and conversation trends.

[0249] The present invention provides a virtual family shop system that allows users to get family advice in specific situations. Specific embodiments are given below.

[0250] Hardware and Software Configuration

[0251] User device: An electronic device such as a smartphone or tablet that a user uses to take photos or videos.

[0252] Server: A computer system that runs on the cloud and analyzes image and video data, generates 3D models, and performs natural language processing.

[0253] Software used: Google Cloud Speech-to-Text, Google Dialogflow, Blender, GPT-4 (registered trademark)

[0254] Head-mounted display device: A display device worn by the user to interact with 3D models. An example of this is the Oculus Quest 2.

[0255] Data processing and calculation procedures

[0256] 1. Uploading photos and videos

[0257] The user wears a head-mounted display device and takes photos and videos of their family. This data is temporarily stored on the user's device and then sent to a server via the Internet. The server analyzes the received data and performs a personality diagnosis.

[0258] 2. Personality Test

[0259] The software used on the server (Google Cloud Speech-to-Text and Google Dialogflow) analyzes the appearance and personality of family members from the received photos and videos. For example, facial features are recognized from still images, and voice tone and speaking style are analyzed from videos. This allows for the identification of personalities and characteristics.

[0260] 3. 3D Avatar Generation

[0261] The server uses Blender to generate 3D avatars based on image data and personality assessments. The generated 3D avatars reflect the family members' appearance information and personality assessment results.

[0262] 4. Natural Language Interaction

[0263] When a user wearing a head-mounted display device starts a conversation with an avatar, their voice is converted into text data by Google Cloud Speech-to-Text, and then Google Dialogflow analyzes the text and generates an appropriate response. The response is then converted into natural-sounding speech using GPT-4 and played back on the head-mounted display device.

[0264] Specific examples

[0265] When choosing clothes, the user wears a head-mounted display device and asks, "Mom, which outfit looks good on me?" The generated 3D avatar responds, "I think the red dress looks great on you." This allows the user to experience a conversation with a family member and receive specific advice.

[0266] Prompt Sentence Examples

[0267] The user types the following into the system:

[0268] "Simulate a situation where a 3D avatar of a family member is giving product advice to a user. Code it to respond to the question 'Which dress do you like?' with 'A red dress would look good on you.'"

[0269] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0270] Step 1:

[0271] The user wears a head-mounted display device and takes photos and videos of their family. The captured data is temporarily stored on the user's device and then sent to a server via the Internet. The input is the photo or video data, and the output is the data sent to the server.

[0272] Step 2:

[0273] The server analyzes the received image and video data and uses image recognition and voice analysis technology to extract the family's physical features and personality traits. The input is image and video data, and the output is appearance information and personality assessment results. Specifically, Google Cloud Speech-to-Text and Google Dialogflow are used to recognize facial features from photos and analyze voice tone and speaking style from videos.

[0274] Step 3:

[0275] The server uses Blender to generate a 3D avatar based on image data and personality assessment. The input is appearance information and personality assessment results, and the output is a 3D avatar. Specifically, Blender is used to model the avatar based on appearance information and assessment results.

[0276] Step 4:

[0277] The generated 3D avatar is transferred from the server to the head-mounted display device and displayed. The input is the 3D avatar, and the output is the avatar displayed on the user's HMD. Specifically, the data is sent to the HMD using a data transfer protocol.

[0278] Step 5:

[0279] The user speaks to an avatar through a head-mounted display terminal. The user's voice input is captured by the HMD's microphone and sent to the user terminal. The input is the user's voice data, and the output is the text conversion result of the voice data.

[0280] Step 6:

[0281] The server receives the voice data and converts it to text data using Google Cloud Speech-to-Text. The input is voice data and the output is text data. Specifically, it calls an API that converts the voice data to text data.

[0282] Step 7:

[0283] The server analyzes the text data and generates an appropriate response using Google Dialogflow and GPT-4. The input is text data, and the output is response text data. Specifically, it uses Dialogflow to analyze the intent of the question and GPT-4 to generate natural dialogue.

[0284] Step 8:

[0285] The response text data generated by the server is converted into voice data and sent to the head-mounted display terminal. The input is the response text data, and the output is voice data. Specifically, the voice data is generated using a text-to-speech API and sent to the HMD.

[0286] Step 9:

[0287] The head-mounted display terminal plays back audio data and responds to the user as a dialogue. The input is audio data, and the output is an audio response that the user can hear. Specifically, the audio is played back through the HMD's speakers.

[0288] Step 10:

[0289] The server records the dialogue between the user and the 3D avatar, periodically analyzes it, and generates a report. The input is the dialogue log, and the output is the analysis results and a report. Specifically, the dialogue log is stored in a database, and an algorithm is executed to periodically analyze the content and generate a report.

[0290] Step 11:

[0291] The generated regular report is sent to the registered family member's email address. The input is the report, and the output is an email sent to the family member's email address. The specific operation is to use the email sending API to send the regular report.

[0292] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0293] This invention is a system that uses a user terminal, a server, and a generation AI to enable simulated conversations with family members, and further combines an emotion engine to recognize and respond to emotions in order to reduce feelings of isolation among the elderly. Specific embodiments for implementing this system are described below.

[0294] Uploading photos and videos

[0295] User:

[0296] A user selects and uploads family photos and videos using a user device such as a smartphone or tablet. As a specific example, a user selects five family photos and a 30-second video from the smartphone gallery.

[0297] Device:

[0298] The user's device temporarily stores the selected photos and videos in a buffer and then transmits them to the server via the Internet. Once the photos and videos have been transmitted, the device displays a confirmation message to the user.

[0299] Personality diagnosis and emotion recognition

[0300] server:

[0301] The server analyzes the received photos and videos. Using image recognition and voice analysis technology, it analyzes the family members' appearance and voice to extract personality traits and emotions. For example, it recognizes facial features from still images and analyzes the tone of voice and speaking style from videos. This results in a "gentle and kind personality" and "current emotional state."

[0302] 3D avatar generation and emotional reflection

[0303] server:

[0304] The server uses a 3D modeling tool based on the analysis results to generate 3D avatars of the family. These avatars combine appearance information, personality assessment results, and emotional states, and are designed to resemble the appearance and behavior of the actual family. The generated avatars are then transferred to the user's device.

[0305] Device:

[0306] The user's device decompresses the 3D avatar received from the server and displays it on the application screen. The user can move the avatar and change its orientation by touching it. The avatar's facial expression and attitude can change depending on the user's emotional state.

[0307] Natural language dialogue

[0308] User:

[0309] The user initiates a dialogue with the 3D avatar through the application, for example by asking, "How was your day today?" This input is recorded as text and / or audio.

[0310] Device:

[0311] The user terminal converts the voice input into text data and sends it to the server, or if the input is text, sends it as is to the server.

[0312] server:

[0313] The system analyzes the received text data and uses a natural language processing (NLP) engine to understand its meaning. It then uses an emotion engine to recognize the user's emotions and generates an appropriate response. For example, it could respond with a happy expression on the avatar, saying, "The weather was nice today, so I went for a walk."

[0314] Device:

[0315] The user device converts the response received from the server into the voice and facial expression output of the 3D avatar, and returns it to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[0316] Generate and send scheduled reports

[0317] server:

[0318] The conversation history between the user and the 3D avatar is periodically collected and stored in a database. The emotional engine is also used to analyze the user's emotional tendencies during the conversation.

[0319] server:

[0320] Based on the collected data, it extracts trends in specific topics and emotions and generates regular reports. For example, it can report from a week's worth of conversation logs that "health" and "weather" were frequently discussed, and that users were primarily "calm."

[0321] server:

[0322] The generated reports are sent to the registered family member's email address on a regular basis, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[0323] This system allows elderly people living alone to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. Furthermore, the introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[0324] The processing flow will be explained below.

[0325] MODE FOR CARRYING OUT THE INVENTION

[0326] Uploading photos and videos

[0327] Step 1:

[0328] User:

[0329] Open the app on your smartphone or tablet and select photos and videos of your family: five photos and a 30-second video from your gallery.

[0330] Step 2:

[0331] Device:

[0332] The selected photos and videos are temporarily stored in a buffer and then sent to a server over the Internet. Once the photos and videos have been sent, the device will display a confirmation message to the user saying "Upload complete."

[0333] Personality diagnosis and emotion recognition

[0334] Step 3:

[0335] server:

[0336] Receives uploaded photos and videos, stores them in the server's storage, records the storage path, and generates metadata for further processing.

[0337] Step 4:

[0338] server:

[0339] The system analyzes stored photos and videos. It uses image recognition and audio analysis technology to analyze the appearance and voice of family members to extract personality traits and emotions. For example, it can extract characteristics such as a "calm and gentle personality" and "current emotional state" from the audio in a video.

[0340] 3D avatar generation and emotional reflection

[0341] Step 5:

[0342] server:

[0343] Based on the analysis results, a 3D modeling tool is used to generate 3D avatars of the family members, combining appearance information with personality assessment results and emotional state to create the optimal avatar.

[0344] Step 6:

[0345] server:

[0346] The generated 3D avatar is stored on the server and transferred to the user's device. The avatar data is compressed to enable efficient transfer.

[0347] Step 7:

[0348] Device:

[0349] The app receives the 3D avatar transferred from the server, decompresses it, and displays it in the app. The user can move the avatar and change its orientation using touch controls. The avatar also has the ability to change its facial expression and attitude depending on the user's emotional state.

[0350] Natural language dialogue

[0351] Step 8:

[0352] User:

[0353] The user initiates a conversation with the 3D avatar through the app. For example, the user might say, "How was your day?"

[0354] Step 9:

[0355] Device:

[0356] The user's voice input is converted into text data and sent to the server. In addition, if the input is text, it is sent to the server as is.

[0357] Step 10:

[0358] server:

[0359] The system analyzes the received text data and uses a natural language processing (NLP) engine and an emotion engine to understand the meaning and emotion. It then generates an appropriate response and sends that text data back to the device. For example, it might respond with "The weather was nice today, so I went for a walk," along with a happy expression on the avatar.

[0360] Step 11:

[0361] Device:

[0362] The response received from the server is converted into the voice and facial expression output of the 3D avatar, and returned to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[0363] Generate and send scheduled reports

[0364] Step 12:

[0365] server:

[0366] The conversation history between the user and the 3D avatar is periodically collected and stored in a database. The emotional engine is also used to analyze the user's emotional tendencies during the conversation.

[0367] Step 13:

[0368] server:

[0369] Based on the collected data, it extracts trends in specific topics and emotions and generates regular reports. For example, it can report from a week's worth of conversation logs that "health" and "weather" were frequently discussed, and that users were primarily "calm."

[0370] Step 14:

[0371] server:

[0372] The generated reports are sent to the registered family member's email address on a regular basis, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[0373] This system allows elderly people living alone to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. Furthermore, the introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[0374] Example 2

[0375] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] There is a growing need for systems that can reduce the sense of isolation felt by the elderly and provide them with psychological care. However, existing systems lack natural dialogue and emotion recognition, making it difficult to provide appropriate responses and emotional care to users. Furthermore, it is difficult to communicate with family members and share the situation, and there is a lack of means to grasp the user's condition in a timely manner.

[0377] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0378] In this invention, the server includes means for uploading digital image data and video data from a user terminal, means for analyzing the digital image data and video data on the server and diagnosing the user's personality and emotions, means for generating a 3D character based on the personality and emotion diagnosis results, means for transferring the 3D character to the user terminal and displaying it, means for generating a natural language dialogue between the user and the 3D character via the user terminal, means for recording the dialogue and sending regular reports to registered family members, and means for generating responses according to the user's emotions using an emotion engine. This enables natural dialogue, allows appropriate care to be provided according to the user's emotional state, and enables family members to understand the user's condition in a timely manner.

[0379] A "user terminal" is an information processing device such as a smartphone, tablet, or personal computer operated by a user.

[0380] "Digital image data" refers to data of still images taken by a user, and is saved in a format such as JPEG or PNG.

[0381] "Video data" refers to video data shot by a user and saved in formats such as MP4 or AVI.

[0382] A "server" is a remote information processing device that provides functions such as data analysis, processing, and storage.

[0383] "Personality Assessment" is the process of identifying a user's personality traits through the analysis of image and audio data.

[0384] "Emotional diagnosis" is the process of identifying a user's current emotional state through the analysis of image and audio data.

[0385] A "3D character" is a three-dimensional virtual character generated based on the user's photo and video data.

[0386] "Natural language dialogue" refers to communication between a user and a system in natural language using voice or text.

[0387] "Regular reports" are reports generated based on the dialogue between the user and the 3D character and sent to family members.

[0388] An "emotion engine" is a software module that analyzes a user's emotions and generates an appropriate response.

[0389] A "natural language processing engine" is a software module that analyzes input text data and understands its meaning.

[0390] This invention is a system that uses a user terminal, a server, and a generation AI to enable simulated conversations with family members, and further combines an emotion engine to recognize and respond to emotions in order to reduce feelings of isolation among the elderly. Specific forms for realizing this system are shown below.

[0391] System Configuration

[0392] User terminal

[0393] The user terminals are smartphones, tablets, or personal computers. These devices provide the ability to take and store photos and videos of family members, and can transmit the data to a server via the Internet. They also provide an interface for displaying the generated 3D characters and interacting with them through applications.

[0394] server

[0395] The server receives and analyzes photo and video data uploaded from user devices and processes the data using generative AI models and emotion engines. The server has the following main functions:

[0396] 1. Data reception and storage function

[0397] 2. Image recognition and speech analysis functions (e.g., OpenCV, Google Cloud Speech-to-Text)

[0398] 3. Personality diagnosis and emotion recognition (e.g., NLP engines)

[0399] 4. 3D character generation function (e.g. Blender)

[0400] 5. Natural language processing engines (e.g., GPT-3, BERT)

[0401] 6. Emotion Engine

[0402] Explanation of specific processing contents

[0403] 1. Upload your data

[0404] A user selects and uploads family photos and videos using a user terminal such as a smartphone or tablet. For example, a user selects five family photos and a 30-second video from the smartphone gallery.

[0405] 2. Data Analysis

[0406] The server receives and analyzes photo and video data sent from the user's device. Image recognition is performed using OpenCV, and voice data is converted to text using the Google Cloud Speech-to-Text API for analysis. Personality diagnosis and emotion recognition are performed using an NLP engine, which extracts personality and emotional tendencies.

[0407] 3. 3D character generation

[0408] The server uses 3D modeling tools such as Blender to generate a 3D character based on the analysis results. This character reflects the appearance information obtained from the still images and the personality and emotional state extracted from the video. The generated character is then transferred to the user's device.

[0409] 4. Natural Language Interaction

[0410] The user initiates a natural language dialogue with the 3D character through the application. For example, they might say, "How was your day today?" This input is recorded as voice or text and sent from the user's device to the server. The server uses an NLP engine to analyze the input data and an emotion engine to generate an appropriate response. The generated response is converted into voice or facial expression and expressed by the 3D character on the user's device.

[0411] 5. Generate and send scheduled reports

[0412] The server periodically collects the user's interaction history with the 3D character and stores it in a database. It uses an emotion engine to analyze the user's emotional tendencies during the interaction, extracting specific topics and emotional patterns to generate periodic reports. These reports are sent to registered family members' email addresses, reporting the user's recent status and emotional tendencies.

[0413] Examples of concrete examples and prompts

[0414] Examples:

[0415] When a user asks, "How was your day today?", the server analyzes the question and generates a response such as, "The weather was nice today, so I went for a walk." The 3D character then displays a happy expression.

[0416] Example prompt sentence:

[0417] "The user asks, 'How was your day?' Generate a response that reflects a happy mood."

[0418] This system allows elderly people to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. The introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[0419] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0420] Step 1:

[0421] A user uses a smartphone or tablet to select and upload digital image data (photos) and video data (movies) of their family members. Specifically, the user opens the device's gallery app, selects five photos and a 30-second video, and taps the "Upload" button. This action saves the digital image data and video data in the device's buffer. The input is the selected image data and video data, and the output is the data saved in the buffer.

[0422] Step 2:

[0423] The device sends the image and video data stored in the buffer to the server. The device encrypts the data and uploads it to the server via the Internet. When the transmission is complete, the device displays a confirmation message to the user saying "Upload complete." The input is the data stored in the buffer, and the output is the data sent to the server.

[0424] Step 3:

[0425] The server receives image and video data from the user's device. The server checks the format of the data and prepares it for image recognition and voice analysis. The input is the data sent to the server, and the output is data in a format that can be analyzed.

[0426] Step 4:

[0427] The server uses OpenCV to analyze image data and extract the facial features of family members. It also uses the Google Cloud Speech-to-Text API to convert the video's audio data into text. An NLP engine is used to analyze speaking style and tone of voice from the text data to extract the family members' personalities and current emotional state. The input is data in an analyzable format, and the output is facial feature information, textual audio data, and personality and emotional diagnosis results.

[0428] Step 5:

[0429] The server uses a 3D modeling tool such as Blender to generate 3D characters of the family members based on the analysis results obtained in the previous step. These characters reflect facial feature information, personality diagnosis results, and emotional state. The input is facial feature information, text-converted voice data, and personality and emotional diagnosis results, and the output is the generated 3D characters.

[0430] Step 6:

[0431] The server transfers the data of the generated 3D character to the user terminal. The input is the data of the generated 3D character, and the output is the 3D character data sent to the user terminal.

[0432] Step 7:

[0433] The device decompresses the 3D character data received from the server and displays it on the application screen. The user can control the character by touching it. The avatar can change its facial expression and attitude depending on the user's emotional state. The input is the 3D character data sent from the server, and the output is the 3D character displayed in the application.

[0434] Step 8:

[0435] The user initiates a dialogue with the 3D character through the application, for example by asking, "How was your day today?" This input is recorded as voice or text. The input is what the user said, and the output is voice or text data.

[0436] Step 9:

[0437] The terminal converts the voice input into text data and sends it to the server. If the input is text, it sends it to the server as is. The input is voice or text data, and the output is the text data sent to the server.

[0438] Step 10:

[0439] The server analyzes the received text data and uses an NLP engine to understand the meaning of the input. It then uses an emotion engine to recognize the user's emotions and generates an appropriate response. For example, the response generated might be, "The weather was nice today, so I went for a walk." The input is text data, and the output is the generated response.

[0440] Step 11:

[0441] The device converts the response received from the server into voice and facial expressions, and returns it to the user in the form of a 3D character response. The avatar responds with natural facial expressions and movements. The input is the generated response, and the output is the response reflected in voice and facial expressions.

[0442] Step 12:

[0443] The server periodically collects the interaction history between the user and the 3D character and stores it in a database. The input is the interaction history, and the output is the stored data.

[0444] Step 13:

[0445] The server uses an emotion engine to analyze the user's emotional tendencies during the conversation. This analysis extracts specific topics and emotional patterns. The input is the saved conversation history data, and the output is the extracted emotional tendencies and topic patterns.

[0446] Step 14:

[0447] The server generates periodic reports based on the analysis results. For example, it reports that "health" and "weather" were frequently discussed based on a week's worth of conversation logs, and that the user was primarily "calm." The input is the analysis results, and the output is the generated periodic reports.

[0448] Step 15:

[0449] The server sends the generated periodic report to the registered family member's email address. The report includes information such as the frequency of conversations, topic trends, and the user's emotional state. The input is the generated periodic report, and the output is the sent report.

[0450] (Application example 2)

[0451] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0452] In addition to systems that provide psychological care through simulated conversations with family members for elderly people and users who feel isolated, there is a need for effective support in providing product explanations and customer service to customers in physical stores. Furthermore, it is necessary to improve the level of service in stores by using a system that uses avatars that can accurately convey product features during conversations with customers.

[0453] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data and video data from a user terminal, means for analyzing the image data and video data and diagnosing the user's personality, means for generating a 3D model based on the personality diagnosis results, means for uploading product photos and videos in the store and analyzing the product features, means for generating a 3D product avatar based on the features and displaying it on the user terminal, means for generating a natural language dialogue between the user and the 3D product avatar via the user terminal, and means for recording the content of the dialogue and generating a service report for the user. This enables psychological care for elderly users and users who feel isolated, and effective customer service support for customers in physical stores.

[0454] A "user terminal" is a device operated by a user, such as a computer, smartphone, or tablet.

[0455] "Image data" refers to digital image files such as photographs uploaded by users.

[0456] "Video data" refers to digital video files such as video clips uploaded by users.

[0457] A "server" is a computer system for receiving and processing data sent from a user terminal.

[0458] "Personality diagnosis" is a method of analyzing the personality traits and emotional state of a user or subject.

[0459] A "3D model" is a three-dimensional virtual object generated using computer graphics.

[0460] "Natural language dialogue" refers to communication using language that humans use on a daily basis.

[0461] "Recording the content of the dialogue" means saving the content of the dialogue between the user and the 3D model in a database or the like.

[0462] A "regular report" is a report that summarizes the content of conversations and the status of users over a certain period of time.

[0463] "Product features" refers to information about the physical properties and functions of a product.

[0464] A "3D product avatar" is a three-dimensional character that virtually recreates a product.

[0465] A "service report" is a report that summarizes the content of conversations with users and business negotiation history.

[0466] A "natural language processing engine" is software that analyzes input in natural language and generates appropriate responses.

[0467] An "emotion recognition engine" is software that analyzes and recognizes the emotional state of a user or target.

[0468] To implement this invention, the following system is used: The main components of the system are a user terminal, a server, a generative AI model, and an emotion recognition engine.

[0469] Users upload image and video data using user devices such as smartphones and tablets. This data is sent to a server via the Internet. The server analyzes the received image and video data to extract characteristics of the user and the product. For example, it analyzes the user's personality and the product's external characteristics. It also uses an emotion recognition engine to diagnose the user's emotions.

[0470] The server then uses a 3D modeling tool based on the analysis results to generate 3D models of users and products. For example, it generates avatars for "gentle and kind personalities" or "healthy products" based on facial features obtained from still images and voice tones obtained from videos. These 3D models are then transferred from the server to the user's device and displayed on the application screen. Users can then have natural conversations with the avatars.

[0471] User interaction input is recorded as voice or text and sent from the user's device to the server. The server then uses a natural language processing engine to analyze the input and generate an appropriate response. For example, if the user asks, "How was your day today?", the server might respond with, "The weather was nice today, so I went for a walk," and display a happy expression on the avatar.

[0472] Furthermore, the server records the content of conversations and generates periodic reports based on them. These reports are sent to registered family members and users and include information such as the frequency of conversations, topic trends, and emotional state. For example, a week's worth of conversation logs may be analyzed to determine that "health" and "weather" were frequently discussed, and the server may report to family members that the user was primarily "calm."

[0473] To illustrate, here is a prompt statement that generates the following program:

[0474] "Generate a Python program for a mobile application that uses uploaded product photos and videos to create a 3D avatar representing the products in a store. The avatar should be able to interact with users through natural language conversation. The program should use TENSORFLOW(R) for emotion recognition, BERT for natural language processing, and SpeechRecognition for converting audio input to text."

[0475] This system not only enables psychological care for elderly people and users who feel isolated, but also enables effective customer service support for physical stores.

[0476] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0477] Step 1:

[0478] A user selects and uploads image data and video data using a user device such as a smartphone or tablet. The user device temporarily stores the selected data in buffer memory and sends it to the server. The input is image data and video data, and the output is the data sent to the server. Specifically, the user selects five product photos and a 30-second promotional video, and clicks the upload button.

[0479] Step 2:

[0480] The server receives image and video data sent from the user's device and prepares it for analysis. The server analyzes the data using image recognition and audio analysis technology to extract product features. The input is data from the user's device, and the output is the analysis results (product features). Specifically, TensorFlow is used to extract product features such as color, shape, and logo from images, and to analyze audio tone from videos.

[0481] Step 3:

[0482] The server uses a 3D modeling tool based on the analysis results to generate a 3D avatar of the product. This avatar combines information about the product's appearance and features. The input is the analysis results, and the output is the generated 3D avatar. Specifically, the operation uses a modeling tool such as Blender to generate a 3D model that faithfully reproduces the product's appearance.

[0483] Step 4:

[0484] The server transfers the generated 3D avatar to the user's device. The user's device receives the avatar and displays it on the application screen. The input is the 3D avatar, and the output is the avatar displayed on the user's device. Specifically, the user opens the app on their smartphone, and the 3D avatar of the product appears on the screen.

[0485] Step 5:

[0486] The user initiates a natural language dialogue with a 3D avatar via the user terminal. The user's voice input is converted into text on the user terminal and sent to the server. The input is the user's voice, and the output is text data. Specifically, the action is to ask, "What are the features of this product?"

[0487] Step 6:

[0488] The server analyzes the received text data and understands its meaning using a natural language processing (NLP) engine. It also uses an emotion recognition engine to recognize the user's emotions and generate an appropriate response. The input is text data, and the output is a response text. Specifically, it uses BERT to analyze the user's question and generates the response, "This product uses organic ingredients."

[0489] Step 7:

[0490] The server sends the generated response to the user's device and converts it into the voice and facial expression output of the 3D avatar. The user's device receives the response, and the avatar responds with natural facial expressions and movements. The input is the response text, and the output is voice and facial expressions. Specifically, the user listens to the avatar's explanation on their smartphone, and the avatar explains with a smile.

[0491] Step 8:

[0492] The server records the content of the conversation between the user and the 3D avatar and periodically analyzes it. Specific topics and emotional trends are extracted from the conversation and compiled into a service report. The input is the conversation content, and the output is the analysis results and report. Specifically, the server analyzes the conversation log for a week and finds that there were many questions about "health," and compiles this into a report.

[0493] Step 9:

[0494] The server periodically sends service reports to registered family members and users. These reports include information such as the frequency of interactions, topic trends, and the user's emotional state. The input is the service report, and the output is email or notification. Specifically, this operation sends a "weekly interaction history report" to the family member's email address.

[0495] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0496] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0497] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0498] [Second embodiment]

[0499] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0500] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0501] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0502] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0503] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0504] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0505] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0506] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0507] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0508] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0509] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0510] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0511] This invention is a system that enables simulated conversations with family members using a user terminal, a server, and a generation AI to reduce the sense of isolation felt by elderly people. Specific embodiments for implementing this system are described below.

[0512] Uploading photos and videos

[0513] User:

[0514] A user selects and uploads family photos and videos using a user device such as a smartphone or tablet. As a specific example, a user selects five family photos and a 30-second video from the smartphone gallery.

[0515] Device:

[0516] The user's device temporarily stores the selected photos and videos in a buffer and then transmits them to the server via the Internet. Once the photos and videos have been transmitted, the device displays a confirmation message to the user.

[0517] Personality Assessment

[0518] server:

[0519] The server analyzes the received photos and videos. Using image recognition and voice analysis technology, it extracts the family members' physical features and personality traits. For example, it recognizes facial features from still images and analyzes the tone of voice and speaking style from videos. This allows it to extract traits such as a "gentle and kind personality."

[0520] 3D avatar generation

[0521] server:

[0522] The server uses a 3D modeling tool based on the analysis results to generate 3D avatars of the family. These avatars combine appearance information and personality assessment results, and resemble the appearance and behavior of the actual family. The generated avatars are then transferred to the user's device.

[0523] Device:

[0524] The user device decompresses the 3D avatar received from the server and displays it on the application screen. The user can move and change the orientation of the avatar by touching it.

[0525] Natural language dialogue

[0526] User:

[0527] The user initiates a dialogue with the 3D avatar through the application, for example by asking, "How was your day today?" This input is recorded as text and / or audio.

[0528] Device:

[0529] The user terminal converts the voice input into text data and sends it to the server, or if the input is text, sends it as is to the server.

[0530] server:

[0531] The server analyzes the received text data and uses a natural language processing (NLP) engine to understand its meaning. It generates an appropriate response and sends the text data back to the device. For example, it generates a response such as, "The weather was nice today, so I went for a walk."

[0532] Device:

[0533] The user device converts the response received from the server into voice output for the 3D avatar, which responds to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[0534] Generate and send scheduled reports

[0535] server:

[0536] The server periodically analyzes the conversations between users and their 3D avatars to identify specific topics and emotional trends. For example, it might discover from a week's worth of conversation logs that "health" and "weather" are frequently discussed topics.

[0537] server:

[0538] Based on the extracted information, periodic reports are generated that include information on the frequency of interactions, topic trends, and the user's emotional state.

[0539] server:

[0540] The generated regular reports are sent to the registered email addresses of family members, allowing family members in remote locations to easily keep track of the user's status.

[0541] This system allows elderly people living alone to enjoy natural conversation with someone who resembles a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to feel at ease and keep track of the user's condition.

[0542] The processing flow will be explained below.

[0543] Uploading photos and videos

[0544] Step 1: User Selects Media File

[0545] User:

[0546] Open the app on your smartphone or tablet and select photos and videos of your family: five photos and a 30-second video from your gallery.

[0547] Step 2: Upload your media files

[0548] Device:

[0549] The selected media file is temporarily stored in a buffer and then sent over the Internet to a server. Once the file transfer is complete, the device displays a confirmation message to the user saying "Upload Complete."

[0550] Personality Assessment

[0551] Step 3: Receive and save media files

[0552] server:

[0553] Receives uploaded photos and videos, stores them in the server's storage, records the storage path, and generates metadata for further processing.

[0554] Step 4: Analysis and personality assessment

[0555] server:

[0556] The system analyzes stored photos and videos. It uses image recognition and voice analysis technology to extract personality traits from family members' appearances and voices. For example, a "gentle and kind personality" can be extracted from the voice of a video.

[0557] 3D avatar generation

[0558] Step 5: Preparing for 3D modeling

[0559] server:

[0560] Based on the personality test results and the characteristics of the media file, a 3D modeling tool is used to generate 3D avatars of your family members, combining appearance and personality traits to create the perfect avatar.

[0561] Step 6: Saving and transferring your avatar

[0562] server:

[0563] The generated 3D avatar is stored on the server and transferred to the user's device. The avatar data is compressed to enable efficient transfer.

[0564] Step 7: Receiving and displaying your avatar

[0565] Device:

[0566] The app receives the 3D avatar transferred from the server, decompresses it, and displays it in the app. The user can move and rotate the avatar using touch controls.

[0567] Natural language dialogue

[0568] Step 8: Initiating a conversation

[0569] User:

[0570] The user initiates a conversation with the 3D avatar through the app, specifically by asking, "How was your day?"

[0571] Step 9: Submitting input data

[0572] Device:

[0573] The user's voice input is converted into text data and sent to the server. In addition, if the input is text, it is sent to the server as is.

[0574] Step 10: Analyzing the interaction data

[0575] server:

[0576] It analyzes the received text data, understands its meaning using a natural language processing (NLP) engine, generates an appropriate response, and sends the text data back to the device.

[0577] Step 11: View the response

[0578] Device:

[0579] The response received from the server is converted into voice output by the 3D avatar and returned to the user as a dialogue. For example, the avatar might respond with a natural expression, "The weather was nice today, so I went for a walk."

[0580] Generate and send scheduled reports

[0581] Step 12: Recording interaction history

[0582] server:

[0583] The interaction history between the user and the 3D avatar is periodically collected and stored in a database, and the frequency and content of the interactions are organized and recorded.

[0584] Step 13: Generate reports

[0585] server:

[0586] The system analyzes conversation history to extract specific topics and sentiment trends, summarizing major topics such as "health" and "weather" on a weekly basis.

[0587] Step 14: Submitting the report

[0588] server:

[0589] The generated periodic reports are sent to registered family members via email or other means, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[0590] This process allows elderly people living alone to enjoy natural conversation with people who resemble family members, reducing feelings of loneliness and providing emotional care. In addition, regular reports to families allow them to understand the user's condition with peace of mind.

[0591] Example 1

[0592] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0593] Elderly people tend to feel isolated and require psychological care. Furthermore, families living far away have limited means of understanding the situation of their elderly relatives, making it difficult for them to provide support for their lives with peace of mind. Current technology does not allow for direct communication with family members, and there is a lack of mechanisms for providing detailed information about the elderly's living situation to family members.

[0594] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0595] In this invention, the server includes means for analyzing the received image data and video data, means for extracting the user's appearance features and personality traits based on the analysis, and means for generating a 3D model based on the personality diagnosis results. This reduces the elderly person's sense of isolation through natural conversation with a presence that resembles a family member, and by recording the content of the conversation and sending regular reports to the family, it becomes easier for the family to understand the elderly person's condition even remotely.

[0596] A "user terminal" is an electronic device such as a smartphone, tablet, or personal computer that is operated by a user.

[0597] "Image data" refers to still image data uploaded from a user terminal.

[0598] "Video data" refers to dynamic video data uploaded from a user terminal.

[0599] A "buffer memory" is a temporary storage area for data, and is used to increase the efficiency of data transfer.

[0600] The "Internet" is a huge computer network that connects computer networks all over the world.

[0601] A "server" is a computer or software that provides services to other computers over a network.

[0602] "Analysis" is the act of processing image data and video data to extract their contents and characteristics.

[0603] "Appearance features" refer to visual features such as a person's face, posture, and clothing that are extracted from images and videos.

[0604] "Personality traits" refer to a person's personality characteristics obtained through voice analysis and behavioral analysis.

[0605] "3D Model" means a three-dimensional digital character generated by the server and simulated for the user.

[0606] A "generative AI model" is an artificial intelligence algorithm that generates new data based on specific patterns or data.

[0607] A "prompt" is a sentence or question that is input into a generative AI model, and is the input data that the AI ​​uses to generate responses and data.

[0608] A "natural language processing engine" is an algorithm or software that understands and appropriately processes human language.

[0609] "Voice input" refers to the process of converting words spoken by a user into digital data through a microphone.

[0610] "Text data" refers to data obtained by converting voice input into text information.

[0611] "Speech conversion" refers to the process of converting text data back into speech.

[0612] A "regular report" is a document that is generated periodically to inform family members of the user's conversation content and emotional tendencies.

[0613] The present invention is a system that enables simulated conversations with family members using a user terminal, a server, and a generation AI to reduce the sense of isolation felt by elderly people. Specific embodiments for implementing this system are described below.

[0614] Uploading photos and videos

[0615] User:

[0616] Users select and upload photos and videos of their family using their smartphone, tablet, or other device. Specifically, users access the gallery through the app interface and select five photos and a 30-second video.

[0617] Device:

[0618] The user device temporarily stores the selected photos and videos in a buffer memory. Then, it sends this data to the server via an HTTP POST request. During this process, the user device displays a progress bar to indicate the progress, and once the transfer is complete, it displays the message "Upload complete" to the user.

[0619] Personality Assessment

[0620] server:

[0621] The server receives the photos and videos sent from the devices and stores them in cloud storage (e.g., AWS S3). It then analyzes the data using image recognition (e.g., OpenCV) and voice analysis (e.g., Google Cloud Speech-to-Text) technologies to extract the family's physical features and personality traits.

[0622] 3D avatar generation

[0623] server:

[0624] Based on the appearance and voice characteristics discovered from the analysis, the server applies a personality diagnosis algorithm (e.g., decision tree or neural network) to derive characteristics such as a "gentle and kind personality."

[0625] server:

[0626] Then, 3D avatars of the family are generated using 3D modeling software (e.g., Blender), recreating facial contours, hairstyles, and clothing styles, and adding facial expressions and movements based on their personalities. The resulting 3D avatars are then sent to the user's device in a compressed format, using the SSL / TLS protocol to ensure secure data transfer.

[0627] Device:

[0628] The user device decompresses the received 3D avatar data and displays it in the application's 3D view. The user can change the orientation of the avatar by touching or swiping the screen.

[0629] Natural language dialogue

[0630] User:

[0631] The user initiates a dialogue with the 3D avatar using voice or text input, for example, by asking, "How was your day today?"

[0632] Device:

[0633] The user device converts the voice input into text data via an API (e.g., Google Cloud Speech-to-Text). If the input is text, it is sent directly to the server.

[0634] server:

[0635] The server analyzes the received text data and generates an appropriate response using a generative AI model (e.g., GPT-3), such as "The weather was nice today, so I went for a walk."

[0636] Device:

[0637] The user device converts the response text received from the server into speech using a speech synthesis API (e.g., Amazon Polly) and synchronizes it with the mouth movements of the 3D avatar, allowing the user to enjoy natural conversations through the avatar.

[0638] Generate and send scheduled reports

[0639] server:

[0640] The server periodically analyzes the conversation between the user and the 3D avatar to extract trends in topics and emotions. Specifically, it uses an NLP engine to classify the conversation logs by topic and evaluates the user's emotional state using an emotion analysis algorithm.

[0641] server:

[0642] Based on the analysis results, regular reports are generated that include information such as the frequency of conversations, main topics, and emotional state, and are displayed in easy-to-read graphs and charts.

[0643] server:

[0644] The generated periodic reports are automatically sent to the family's email addresses. The server uses the SMTP protocol to realize the email sending function.

[0645] For example, some input prompts for a generative AI model might include:

[0646] "Generate a 3D avatar of a very kind man based on the following dialogue: 'Hi, Grandpa! What were you doing today?' 'Hi, Grandma! It's very cool today, so I was reading in the garden.'"

[0647] This system allows elderly people to enjoy natural conversation with someone who resembles a family member, reducing feelings of isolation. It also makes it easier for family members to keep track of the user's condition through regular reports.

[0648] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0649] Step 1: User selection of photos and videos

[0650] User:

[0651] The user launches the app on a smartphone, tablet, or other device, and operates the app's interface to select five family photos and a 30-second video from the gallery.

[0652] Input: Five photos of your family and a 30-second video.

[0653] Output: Selected photos and videos.

[0654] Step 2: Upload photos and videos

[0655] Device:

[0656] The user's device temporarily stores the selected photos and videos in a buffer memory, then sends the data to the server via an HTTP POST request, displaying a progress bar during the transfer and a message indicating the upload is complete when the transfer is complete.

[0657] Input: Photos and videos stored in buffer memory.

[0658] Output: A completion message sent to the server.

[0659] Step 3: Receiving and analyzing image and video data

[0660] server:

[0661] The server receives image and video data sent from the device and stores it in cloud storage (e.g., AWS S3). It then analyzes the data using image recognition (e.g., OpenCV) and voice analysis (e.g., Google Cloud Speech-to-Text) technologies to extract the family's physical features and personality traits.

[0662] Input: Received image and video data.

[0663] Output: Analysis of family members' physical features and personality traits.

[0664] Step 4: Applying the personality assessment algorithm

[0665] server:

[0666] The server applies a personality diagnosis algorithm (e.g., decision tree or neural network) based on the appearance and voice characteristics obtained from the analysis results to derive characteristics such as a "gentle and kind personality."

[0667] Input: Physical and vocal characteristics of family members.

[0668] Output: Personality test results.

[0669] Step 5: Generate a 3D avatar

[0670] server:

[0671] The server uses 3D modeling software (e.g., Blender) to generate 3D avatars of the family based on the personality test results and appearance data, and the generated avatars are sent to the user's device in a compressed format.

[0672] Input: Personality test results and physical characteristics data.

[0673] Output: Generated 3D avatar data.

[0674] Step 6: Displaying the 3D Avatar

[0675] Device:

[0676] The user device decompresses the received 3D avatar data and displays it in the application's 3D view. The user can change the orientation of the avatar by touching or swiping the screen.

[0677] Input: Received 3D avatar data.

[0678] Output: 3D avatar displayed on the application screen.

[0679] Step 7: Initiating a natural language dialogue

[0680] User:

[0681] The user initiates a dialogue with the 3D avatar through voice or text input, for example, by asking, "How was your day today?"

[0682] Input: Voice or text input.

[0683] Output: The input audio or text data.

[0684] Step 8: Process and transmit voice / text data

[0685] Device:

[0686] The user device converts the voice input into text data via an API (e.g., Google Cloud Speech-to-Text). If the input is text, it is sent directly to the server.

[0687] Input: Audio or text data.

[0688] Output: The text data sent to the server.

[0689] Step 9: Generate a response

[0690] server:

[0691] The server analyzes the received text data and generates an appropriate response using a generative AI model (e.g., GPT-3), such as "The weather was nice today, so I went for a walk."

[0692] Input: Text data.

[0693] Output: The generated response text.

[0694] Step 10: Returning a response and converting it

[0695] Device:

[0696] The user device converts the response text received from the server into speech using a speech synthesis API (e.g., Amazon Polly) and synchronizes it with the mouth movements of the 3D avatar, allowing the user to enjoy natural conversations through the avatar.

[0697] Input: The response text received from the server.

[0698] Output: Voice response by avatar.

[0699] Step 11: Record and analyze the conversation

[0700] server:

[0701] The server periodically analyzes the conversation between the user and the 3D avatar to extract trends in topics and emotions. Specifically, it uses an NLP engine to classify the conversation logs by topic and evaluates the user's emotional state using an emotion analysis algorithm.

[0702] Input: Log data of the conversation.

[0703] Output: Trend analysis results of topics and sentiment.

[0704] Step 12: Generate and send scheduled reports

[0705] server:

[0706] The server generates periodic reports based on the analysis results and automatically sends them to the family's email addresses, including information on the frequency of conversations, main topics, and emotional state.

[0707] Input: Topic and sentiment trend analysis results.

[0708] Output: Regular reports sent to families.

[0709] This allows the elderly to enjoy natural conversation with people who resemble family members, reducing feelings of loneliness.Families can also keep track of the elderly's condition through regular reports, even when they are far away.

[0710] (Application example 1)

[0711] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0712] There are existing systems that provide simulated dialogue with family members to reduce the sense of isolation felt by elderly people. However, these systems make it difficult for users to enjoy a sophisticated dialogue experience, such as receiving advice from family members in more specific life situations. Furthermore, there is a lack of systems that can generate specific dialogue responses tailored to specific situations. There is a growing demand for solutions to this problem.

[0713] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0714] In this invention, the server includes means for uploading image data and video data from a user terminal, means for analyzing the image data and video data on the server and diagnosing the user's personality, means for generating a 3D model based on the personality diagnosis results, means for transferring the 3D model to a head-mounted display terminal and displaying it, means for generating a natural language dialogue between the user and the 3D model via the head-mounted display terminal, and means for recording the content of the dialogue and sending regular reports to registered family members. This allows elderly people to experience more realistic dialogue with their families and receive appropriate advice in specific life situations.

[0715] A "user terminal" is an electronic device that allows a user to take photos and videos and upload them to a server.

[0716] "Image data and video data" refers to photographs and video files taken or selected by a user using a user terminal.

[0717] A "server" is a remote computer system that receives image and video data sent from a user terminal and analyzes and processes it.

[0718] "Personality diagnosis" is the process of identifying the subject's personality and characteristics by analyzing image and video data.

[0719] A "3D model" is a three-dimensional avatar generated on a computer based on the results of analysis, and is a digital object that reflects a person's appearance and personality.

[0720] A "head-mounted display terminal" is a display device worn on the head, and is a device that provides VR (virtual reality) and AR (augmented reality) experiences.

[0721] "Natural language interaction" is communication between humans and computers in which the user speaks in spoken language and the system responds.

[0722] A "regular report" is a document that the system records and analyzes the content of conversations and periodically reports on the user's status and conversation trends.

[0723] The present invention provides a virtual family shop system that allows users to get family advice in specific situations. Specific embodiments are given below.

[0724] Hardware and Software Configuration

[0725] User device: An electronic device such as a smartphone or tablet that a user uses to take photos or videos.

[0726] Server: A computer system that runs on the cloud and analyzes image and video data, generates 3D models, and performs natural language processing.

[0727] Software used: Google Cloud Speech-to-Text, Google Dialogflow, Blender, GPT-4

[0728] Head-mounted display device: A display device worn by the user to interact with 3D models. An example of this is the Oculus Quest 2.

[0729] Data processing and calculation procedures

[0730] 1. Uploading photos and videos

[0731] The user wears a head-mounted display device and takes photos and videos of their family. This data is temporarily stored on the user's device and then sent to a server via the Internet. The server analyzes the received data and performs a personality diagnosis.

[0732] 2. Personality Test

[0733] The software used on the server (Google Cloud Speech-to-Text and Google Dialogflow) analyzes the appearance and personality of family members from the received photos and videos. For example, facial features are recognized from still images, and voice tone and speaking style are analyzed from videos. This allows for the identification of personalities and characteristics.

[0734] 3. 3D Avatar Generation

[0735] The server uses Blender to generate 3D avatars based on image data and personality assessments. The generated 3D avatars reflect the family members' appearance information and personality assessment results.

[0736] 4. Natural Language Interaction

[0737] When a user wearing a head-mounted display device starts a conversation with an avatar, their voice is converted into text data by Google Cloud Speech-to-Text, and then Google Dialogflow analyzes the text and generates an appropriate response. The response is then converted into natural-sounding speech using GPT-4 and played back on the head-mounted display device.

[0738] Specific examples

[0739] When choosing clothes, the user wears a head-mounted display device and asks, "Mom, which outfit looks good on me?" The generated 3D avatar responds, "I think the red dress looks great on you." This allows the user to experience a conversation with a family member and receive specific advice.

[0740] Prompt Sentence Examples

[0741] The user types the following into the system:

[0742] "Simulate a situation where a 3D avatar of a family member is giving product advice to a user. Code it to respond to the question 'Which dress do you like?' with 'A red dress would look good on you.'"

[0743] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0744] Step 1:

[0745] The user wears a head-mounted display device and takes photos and videos of their family. The captured data is temporarily stored on the user's device and then sent to a server via the Internet. The input is the photo or video data, and the output is the data sent to the server.

[0746] Step 2:

[0747] The server analyzes the received image and video data and uses image recognition and voice analysis technology to extract the family's physical features and personality traits. The input is image and video data, and the output is appearance information and personality assessment results. Specifically, Google Cloud Speech-to-Text and Google Dialogflow are used to recognize facial features from photos and analyze voice tone and speaking style from videos.

[0748] Step 3:

[0749] The server uses Blender to generate a 3D avatar based on image data and personality assessment. The input is appearance information and personality assessment results, and the output is a 3D avatar. Specifically, Blender is used to model the avatar based on appearance information and assessment results.

[0750] Step 4:

[0751] The generated 3D avatar is transferred from the server to the head-mounted display device and displayed. The input is the 3D avatar, and the output is the avatar displayed on the user's HMD. Specifically, the data is sent to the HMD using a data transfer protocol.

[0752] Step 5:

[0753] The user speaks to an avatar through a head-mounted display terminal. The user's voice input is captured by the HMD's microphone and sent to the user terminal. The input is the user's voice data, and the output is the text conversion result of the voice data.

[0754] Step 6:

[0755] The server receives the voice data and converts it to text data using Google Cloud Speech-to-Text. The input is voice data and the output is text data. Specifically, it calls an API that converts the voice data to text data.

[0756] Step 7:

[0757] The server analyzes the text data and generates an appropriate response using Google Dialogflow and GPT-4. The input is text data, and the output is response text data. Specifically, it uses Dialogflow to analyze the intent of the question and GPT-4 to generate natural dialogue.

[0758] Step 8:

[0759] The response text data generated by the server is converted into voice data and sent to the head-mounted display terminal. The input is the response text data, and the output is voice data. Specifically, the voice data is generated using a text-to-speech API and sent to the HMD.

[0760] Step 9:

[0761] The head-mounted display terminal plays back audio data and responds to the user as a dialogue. The input is audio data, and the output is an audio response that the user can hear. Specifically, the audio is played back through the HMD's speakers.

[0762] Step 10:

[0763] The server records the dialogue between the user and the 3D avatar, periodically analyzes it, and generates a report. The input is the dialogue log, and the output is the analysis results and a report. Specifically, the dialogue log is stored in a database, and an algorithm is executed to periodically analyze the content and generate a report.

[0764] Step 11:

[0765] The generated regular report is sent to the registered family member's email address. The input is the report, and the output is an email sent to the family member's email address. The specific operation is to use the email sending API to send the regular report.

[0766] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0767] This invention is a system that uses a user terminal, a server, and a generation AI to enable simulated conversations with family members, and further combines an emotion engine to recognize and respond to emotions in order to reduce feelings of isolation among the elderly. Specific embodiments for implementing this system are described below.

[0768] Uploading photos and videos

[0769] User:

[0770] A user selects and uploads family photos and videos using a user device such as a smartphone or tablet. As a specific example, a user selects five family photos and a 30-second video from the smartphone gallery.

[0771] Device:

[0772] The user's device temporarily stores the selected photos and videos in a buffer and then transmits them to the server via the Internet. Once the photos and videos have been transmitted, the device displays a confirmation message to the user.

[0773] Personality diagnosis and emotion recognition

[0774] server:

[0775] The server analyzes the received photos and videos. Using image recognition and voice analysis technology, it analyzes the family members' appearance and voice to extract personality traits and emotions. For example, it recognizes facial features from still images and analyzes the tone of voice and speaking style from videos. This results in a "gentle and kind personality" and "current emotional state."

[0776] 3D avatar generation and emotional reflection

[0777] server:

[0778] The server uses a 3D modeling tool based on the analysis results to generate 3D avatars of the family. These avatars combine appearance information, personality assessment results, and emotional states, and are designed to resemble the appearance and behavior of the actual family. The generated avatars are then transferred to the user's device.

[0779] Device:

[0780] The user's device decompresses the 3D avatar received from the server and displays it on the application screen. The user can move the avatar and change its orientation by touching it. The avatar's facial expression and attitude can change depending on the user's emotional state.

[0781] Natural language dialogue

[0782] User:

[0783] The user initiates a dialogue with the 3D avatar through the application, for example by asking, "How was your day today?" This input is recorded as text and / or audio.

[0784] Device:

[0785] The user terminal converts the voice input into text data and sends it to the server, or if the input is text, sends it as is to the server.

[0786] server:

[0787] The system analyzes the received text data and uses a natural language processing (NLP) engine to understand its meaning. It then uses an emotion engine to recognize the user's emotions and generates an appropriate response. For example, it could respond with a happy expression on the avatar, saying, "The weather was nice today, so I went for a walk."

[0788] Device:

[0789] The user device converts the response received from the server into the voice and facial expression output of the 3D avatar, and returns it to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[0790] Generate and send scheduled reports

[0791] server:

[0792] The conversation history between the user and the 3D avatar is periodically collected and stored in a database. The emotional engine is also used to analyze the user's emotional tendencies during the conversation.

[0793] server:

[0794] Based on the collected data, it extracts trends in specific topics and emotions and generates regular reports. For example, it can report from a week's worth of conversation logs that "health" and "weather" were frequently discussed, and that users were primarily "calm."

[0795] server:

[0796] The generated reports are sent to the registered family member's email address on a regular basis, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[0797] This system allows elderly people living alone to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. Furthermore, the introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[0798] The processing flow will be explained below.

[0799] MODE FOR CARRYING OUT THE INVENTION

[0800] Uploading photos and videos

[0801] Step 1:

[0802] User:

[0803] Open the app on your smartphone or tablet and select photos and videos of your family: five photos and a 30-second video from your gallery.

[0804] Step 2:

[0805] Device:

[0806] The selected photos and videos are temporarily stored in a buffer and then sent to a server over the Internet. Once the photos and videos have been sent, the device will display a confirmation message to the user saying "Upload complete."

[0807] Personality diagnosis and emotion recognition

[0808] Step 3:

[0809] server:

[0810] Receives uploaded photos and videos, stores them in the server's storage, records the storage path, and generates metadata for further processing.

[0811] Step 4:

[0812] server:

[0813] The system analyzes stored photos and videos. It uses image recognition and audio analysis technology to analyze the appearance and voice of family members to extract personality traits and emotions. For example, it can extract characteristics such as a "calm and gentle personality" and "current emotional state" from the audio in a video.

[0814] 3D avatar generation and emotional reflection

[0815] Step 5:

[0816] server:

[0817] Based on the analysis results, a 3D modeling tool is used to generate 3D avatars of the family members, combining appearance information with personality assessment results and emotional state to create the optimal avatar.

[0818] Step 6:

[0819] server:

[0820] The generated 3D avatar is stored on the server and transferred to the user's device. The avatar data is compressed to enable efficient transfer.

[0821] Step 7:

[0822] Device:

[0823] The app receives the 3D avatar transferred from the server, decompresses it, and displays it in the app. The user can move the avatar and change its orientation using touch controls. The avatar also has the ability to change its facial expression and attitude depending on the user's emotional state.

[0824] Natural language dialogue

[0825] Step 8:

[0826] User:

[0827] The user initiates a conversation with the 3D avatar through the app. For example, the user might say, "How was your day?"

[0828] Step 9:

[0829] Device:

[0830] The user's voice input is converted into text data and sent to the server. In addition, if the input is text, it is sent to the server as is.

[0831] Step 10:

[0832] server:

[0833] The system analyzes the received text data and uses a natural language processing (NLP) engine and an emotion engine to understand the meaning and emotion. It then generates an appropriate response and sends that text data back to the device. For example, it might respond with "The weather was nice today, so I went for a walk," along with a happy expression on the avatar.

[0834] Step 11:

[0835] Device:

[0836] The response received from the server is converted into the voice and facial expression output of the 3D avatar, and returned to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[0837] Generate and send scheduled reports

[0838] Step 12:

[0839] server:

[0840] The conversation history between the user and the 3D avatar is periodically collected and stored in a database. The emotional engine is also used to analyze the user's emotional tendencies during the conversation.

[0841] Step 13:

[0842] server:

[0843] Based on the collected data, it extracts trends in specific topics and emotions and generates regular reports. For example, it can report from a week's worth of conversation logs that "health" and "weather" were frequently discussed, and that users were primarily "calm."

[0844] Step 14:

[0845] server:

[0846] The generated reports are sent to the registered family member's email address on a regular basis, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[0847] This system allows elderly people living alone to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. Furthermore, the introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[0848] Example 2

[0849] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0850] There is a growing need for systems that can reduce the sense of isolation felt by the elderly and provide them with psychological care. However, existing systems lack natural dialogue and emotion recognition, making it difficult to provide appropriate responses and emotional care to users. Furthermore, it is difficult to communicate with family members and share the situation, and there is a lack of means to grasp the user's condition in a timely manner.

[0851] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0852] In this invention, the server includes means for uploading digital image data and video data from a user terminal, means for analyzing the digital image data and video data on the server and diagnosing the user's personality and emotions, means for generating a 3D character based on the personality and emotion diagnosis results, means for transferring the 3D character to the user terminal and displaying it, means for generating a natural language dialogue between the user and the 3D character via the user terminal, means for recording the dialogue and sending regular reports to registered family members, and means for generating responses according to the user's emotions using an emotion engine. This enables natural dialogue, allows appropriate care to be provided according to the user's emotional state, and enables family members to understand the user's condition in a timely manner.

[0853] A "user terminal" is an information processing device such as a smartphone, tablet, or personal computer operated by a user.

[0854] "Digital image data" refers to data of still images taken by a user, and is saved in a format such as JPEG or PNG.

[0855] "Video data" refers to video data shot by a user and saved in formats such as MP4 or AVI.

[0856] A "server" is a remote information processing device that provides functions such as data analysis, processing, and storage.

[0857] "Personality Assessment" is the process of identifying a user's personality traits through the analysis of image and audio data.

[0858] "Emotional diagnosis" is the process of identifying a user's current emotional state through the analysis of image and audio data.

[0859] A "3D character" is a three-dimensional virtual character generated based on the user's photo and video data.

[0860] "Natural language dialogue" refers to communication between a user and a system in natural language using voice or text.

[0861] "Regular reports" are reports generated based on the dialogue between the user and the 3D character and sent to family members.

[0862] An "emotion engine" is a software module that analyzes a user's emotions and generates an appropriate response.

[0863] A "natural language processing engine" is a software module that analyzes input text data and understands its meaning.

[0864] This invention is a system that uses a user terminal, a server, and a generation AI to enable simulated conversations with family members, and further combines an emotion engine to recognize and respond to emotions in order to reduce feelings of isolation among the elderly. Specific forms for realizing this system are shown below.

[0865] System Configuration

[0866] User terminal

[0867] The user terminals are smartphones, tablets, or personal computers. These devices provide the ability to take and store photos and videos of family members, and can transmit the data to a server via the Internet. They also provide an interface for displaying the generated 3D characters and interacting with them through applications.

[0868] server

[0869] The server receives and analyzes photo and video data uploaded from user devices and processes the data using generative AI models and emotion engines. The server has the following main functions:

[0870] 1. Data reception and storage function

[0871] 2. Image recognition and speech analysis functions (e.g., OpenCV, Google Cloud Speech-to-Text)

[0872] 3. Personality diagnosis and emotion recognition (e.g., NLP engines)

[0873] 4. 3D character generation function (e.g. Blender)

[0874] 5. Natural language processing engines (e.g., GPT-3, BERT)

[0875] 6. Emotion Engine

[0876] Explanation of specific processing contents

[0877] 1. Upload your data

[0878] A user selects and uploads family photos and videos using a user terminal such as a smartphone or tablet. For example, a user selects five family photos and a 30-second video from the smartphone gallery.

[0879] 2. Data Analysis

[0880] The server receives and analyzes photo and video data sent from the user's device. Image recognition is performed using OpenCV, and voice data is converted to text using the Google Cloud Speech-to-Text API for analysis. Personality diagnosis and emotion recognition are performed using an NLP engine, which extracts personality and emotional tendencies.

[0881] 3. 3D character generation

[0882] The server uses 3D modeling tools such as Blender to generate a 3D character based on the analysis results. This character reflects the appearance information obtained from the still images and the personality and emotional state extracted from the video. The generated character is then transferred to the user's device.

[0883] 4. Natural Language Interaction

[0884] The user initiates a natural language dialogue with the 3D character through the application. For example, they might say, "How was your day today?" This input is recorded as voice or text and sent from the user's device to the server. The server uses an NLP engine to analyze the input data and an emotion engine to generate an appropriate response. The generated response is converted into voice or facial expression and expressed by the 3D character on the user's device.

[0885] 5. Generate and send scheduled reports

[0886] The server periodically collects the user's interaction history with the 3D character and stores it in a database. It uses an emotion engine to analyze the user's emotional tendencies during the interaction, extracting specific topics and emotional patterns to generate periodic reports. These reports are sent to registered family members' email addresses, reporting the user's recent status and emotional tendencies.

[0887] Examples of concrete examples and prompts

[0888] Examples:

[0889] When a user asks, "How was your day today?", the server analyzes the question and generates a response such as, "The weather was nice today, so I went for a walk." The 3D character then displays a happy expression.

[0890] Example prompt sentence:

[0891] "The user asks, 'How was your day?' Generate a response that reflects a happy mood."

[0892] This system allows elderly people to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. The introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[0893] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0894] Step 1:

[0895] A user uses a smartphone or tablet to select and upload digital image data (photos) and video data (movies) of their family members. Specifically, the user opens the device's gallery app, selects five photos and a 30-second video, and taps the "Upload" button. This action saves the digital image data and video data in the device's buffer. The input is the selected image data and video data, and the output is the data saved in the buffer.

[0896] Step 2:

[0897] The device sends the image and video data stored in the buffer to the server. The device encrypts the data and uploads it to the server via the Internet. When the transmission is complete, the device displays a confirmation message to the user saying "Upload complete." The input is the data stored in the buffer, and the output is the data sent to the server.

[0898] Step 3:

[0899] The server receives image and video data from the user's device. The server checks the format of the data and prepares it for image recognition and voice analysis. The input is the data sent to the server, and the output is data in a format that can be analyzed.

[0900] Step 4:

[0901] The server uses OpenCV to analyze image data and extract the facial features of family members. It also uses the Google Cloud Speech-to-Text API to convert the video's audio data into text. An NLP engine is used to analyze speaking style and tone of voice from the text data to extract the family members' personalities and current emotional state. The input is data in an analyzable format, and the output is facial feature information, textual audio data, and personality and emotional diagnosis results.

[0902] Step 5:

[0903] The server uses a 3D modeling tool such as Blender to generate 3D characters of the family members based on the analysis results obtained in the previous step. These characters reflect facial feature information, personality diagnosis results, and emotional state. The input is facial feature information, text-converted voice data, and personality and emotional diagnosis results, and the output is the generated 3D characters.

[0904] Step 6:

[0905] The server transfers the data of the generated 3D character to the user terminal. The input is the data of the generated 3D character, and the output is the 3D character data sent to the user terminal.

[0906] Step 7:

[0907] The device decompresses the 3D character data received from the server and displays it on the application screen. The user can control the character by touching it. The avatar can change its facial expression and attitude depending on the user's emotional state. The input is the 3D character data sent from the server, and the output is the 3D character displayed in the application.

[0908] Step 8:

[0909] The user initiates a dialogue with the 3D character through the application, for example by asking, "How was your day today?" This input is recorded as voice or text. The input is what the user said, and the output is voice or text data.

[0910] Step 9:

[0911] The terminal converts the voice input into text data and sends it to the server. If the input is text, it sends it to the server as is. The input is voice or text data, and the output is the text data sent to the server.

[0912] Step 10:

[0913] The server analyzes the received text data and uses an NLP engine to understand the meaning of the input. It then uses an emotion engine to recognize the user's emotions and generates an appropriate response. For example, the response generated might be, "The weather was nice today, so I went for a walk." The input is text data, and the output is the generated response.

[0914] Step 11:

[0915] The device converts the response received from the server into voice and facial expressions, and returns it to the user in the form of a 3D character response. The avatar responds with natural facial expressions and movements. The input is the generated response, and the output is the response reflected in voice and facial expressions.

[0916] Step 12:

[0917] The server periodically collects the interaction history between the user and the 3D character and stores it in a database. The input is the interaction history, and the output is the stored data.

[0918] Step 13:

[0919] The server uses an emotion engine to analyze the user's emotional tendencies during the conversation. This analysis extracts specific topics and emotional patterns. The input is the saved conversation history data, and the output is the extracted emotional tendencies and topic patterns.

[0920] Step 14:

[0921] The server generates periodic reports based on the analysis results. For example, it reports that "health" and "weather" were frequently discussed based on a week's worth of conversation logs, and that the user was primarily "calm." The input is the analysis results, and the output is the generated periodic reports.

[0922] Step 15:

[0923] The server sends the generated periodic report to the registered family member's email address. The report includes information such as the frequency of conversations, topic trends, and the user's emotional state. The input is the generated periodic report, and the output is the sent report.

[0924] (Application example 2)

[0925] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0926] In addition to systems that provide psychological care through simulated conversations with family members for elderly people and users who feel isolated, there is a need for effective support in providing product explanations and customer service to customers in physical stores. Furthermore, it is necessary to improve the level of service in stores by using a system that uses avatars that can accurately convey product features during conversations with customers.

[0927] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data and video data from a user terminal, means for analyzing the image data and video data and diagnosing the user's personality, means for generating a 3D model based on the personality diagnosis results, means for uploading product photos and videos in the store and analyzing the product features, means for generating a 3D product avatar based on the features and displaying it on the user terminal, means for generating a natural language dialogue between the user and the 3D product avatar via the user terminal, and means for recording the content of the dialogue and generating a service report for the user. This enables psychological care for elderly users and users who feel isolated, and effective customer service support for customers in physical stores.

[0928] A "user terminal" is a device operated by a user, such as a computer, smartphone, or tablet.

[0929] "Image data" refers to digital image files such as photographs uploaded by users.

[0930] "Video data" refers to digital video files such as video clips uploaded by users.

[0931] A "server" is a computer system for receiving and processing data sent from a user terminal.

[0932] "Personality diagnosis" is a method of analyzing the personality traits and emotional state of a user or subject.

[0933] A "3D model" is a three-dimensional virtual object generated using computer graphics.

[0934] "Natural language dialogue" refers to communication using language that humans use on a daily basis.

[0935] "Recording the content of the dialogue" means saving the content of the dialogue between the user and the 3D model in a database or the like.

[0936] A "regular report" is a report that summarizes the content of conversations and the status of users over a certain period of time.

[0937] "Product features" refers to information about the physical properties and functions of a product.

[0938] A "3D product avatar" is a three-dimensional character that virtually recreates a product.

[0939] A "service report" is a report that summarizes the content of conversations with users and business negotiation history.

[0940] A "natural language processing engine" is software that analyzes input in natural language and generates appropriate responses.

[0941] An "emotion recognition engine" is software that analyzes and recognizes the emotional state of a user or target.

[0942] To implement this invention, the following system is used: The main components of the system are a user terminal, a server, a generative AI model, and an emotion recognition engine.

[0943] Users upload image and video data using user devices such as smartphones and tablets. This data is sent to a server via the Internet. The server analyzes the received image and video data to extract characteristics of the user and the product. For example, it analyzes the user's personality and the product's external characteristics. It also uses an emotion recognition engine to diagnose the user's emotions.

[0944] The server then uses a 3D modeling tool based on the analysis results to generate 3D models of users and products. For example, it generates avatars for "gentle and kind personalities" or "healthy products" based on facial features obtained from still images and voice tones obtained from videos. These 3D models are then transferred from the server to the user's device and displayed on the application screen. Users can then have natural conversations with the avatars.

[0945] User interaction input is recorded as voice or text and sent from the user's device to the server. The server then uses a natural language processing engine to analyze the input and generate an appropriate response. For example, if the user asks, "How was your day today?", the server might respond with, "The weather was nice today, so I went for a walk," and display a happy expression on the avatar.

[0946] Furthermore, the server records the content of conversations and generates periodic reports based on them. These reports are sent to registered family members and users and include information such as the frequency of conversations, topic trends, and emotional state. For example, a week's worth of conversation logs may be analyzed to determine that "health" and "weather" were frequently discussed, and the server may report to family members that the user was primarily "calm."

[0947] To illustrate, here is a prompt statement that generates the following program:

[0948] "Generate a Python program for a mobile application that uses uploaded product photos and videos to create a 3D avatar representing the products in a store. The avatar should be able to interact with users through natural language conversation. The program should use TensorFlow for emotion recognition, BERT for natural language processing, and SpeechRecognition for converting audio input to text."

[0949] This system not only enables psychological care for elderly people and users who feel isolated, but also enables effective customer service support for physical stores.

[0950] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0951] Step 1:

[0952] A user selects and uploads image data and video data using a user device such as a smartphone or tablet. The user device temporarily stores the selected data in buffer memory and sends it to the server. The input is image data and video data, and the output is the data sent to the server. Specifically, the user selects five product photos and a 30-second promotional video, and clicks the upload button.

[0953] Step 2:

[0954] The server receives image and video data sent from the user's device and prepares it for analysis. The server analyzes the data using image recognition and audio analysis technology to extract product features. The input is data from the user's device, and the output is the analysis results (product features). Specifically, TensorFlow is used to extract product features such as color, shape, and logo from images, and to analyze audio tone from videos.

[0955] Step 3:

[0956] The server uses a 3D modeling tool based on the analysis results to generate a 3D avatar of the product. This avatar combines information about the product's appearance and features. The input is the analysis results, and the output is the generated 3D avatar. Specifically, the operation uses a modeling tool such as Blender to generate a 3D model that faithfully reproduces the product's appearance.

[0957] Step 4:

[0958] The server transfers the generated 3D avatar to the user's device. The user's device receives the avatar and displays it on the application screen. The input is the 3D avatar, and the output is the avatar displayed on the user's device. Specifically, the user opens the app on their smartphone, and the 3D avatar of the product appears on the screen.

[0959] Step 5:

[0960] The user initiates a natural language dialogue with a 3D avatar via the user terminal. The user's voice input is converted into text on the user terminal and sent to the server. The input is the user's voice, and the output is text data. Specifically, the action is to ask, "What are the features of this product?"

[0961] Step 6:

[0962] The server analyzes the received text data and understands its meaning using a natural language processing (NLP) engine. It also uses an emotion recognition engine to recognize the user's emotions and generate an appropriate response. The input is text data, and the output is a response text. Specifically, it uses BERT to analyze the user's question and generates the response, "This product uses organic ingredients."

[0963] Step 7:

[0964] The server sends the generated response to the user's device and converts it into the voice and facial expression output of the 3D avatar. The user's device receives the response, and the avatar responds with natural facial expressions and movements. The input is the response text, and the output is voice and facial expressions. Specifically, the user listens to the avatar's explanation on their smartphone, and the avatar explains with a smile.

[0965] Step 8:

[0966] The server records the content of the conversation between the user and the 3D avatar and periodically analyzes it. Specific topics and emotional trends are extracted from the conversation and compiled into a service report. The input is the conversation content, and the output is the analysis results and report. Specifically, the server analyzes the conversation log for a week and finds that there were many questions about "health," and compiles this into a report.

[0967] Step 9:

[0968] The server periodically sends service reports to registered family members and users. These reports include information such as the frequency of interactions, topic trends, and the user's emotional state. The input is the service report, and the output is email or notification. Specifically, this operation sends a "weekly interaction history report" to the family member's email address.

[0969] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0970] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0971] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0972] [Third embodiment]

[0973] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0974] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0975] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0976] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0977] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0978] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0979] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0980] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0981] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0982] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0983] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0984] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0985] This invention is a system that enables simulated conversations with family members using a user terminal, a server, and a generation AI to reduce the sense of isolation felt by elderly people. Specific embodiments for implementing this system are described below.

[0986] Uploading photos and videos

[0987] User:

[0988] A user selects and uploads family photos and videos using a user device such as a smartphone or tablet. As a specific example, a user selects five family photos and a 30-second video from the smartphone gallery.

[0989] Device:

[0990] The user's device temporarily stores the selected photos and videos in a buffer and then transmits them to the server via the Internet. Once the photos and videos have been transmitted, the device displays a confirmation message to the user.

[0991] Personality Assessment

[0992] server:

[0993] The server analyzes the received photos and videos. Using image recognition and voice analysis technology, it extracts the family members' physical features and personality traits. For example, it recognizes facial features from still images and analyzes the tone of voice and speaking style from videos. This allows it to extract traits such as a "gentle and kind personality."

[0994] 3D avatar generation

[0995] server:

[0996] The server uses a 3D modeling tool based on the analysis results to generate 3D avatars of the family. These avatars combine appearance information and personality assessment results, and resemble the appearance and behavior of the actual family. The generated avatars are then transferred to the user's device.

[0997] Device:

[0998] The user device decompresses the 3D avatar received from the server and displays it on the application screen. The user can move and change the orientation of the avatar by touching it.

[0999] Natural language dialogue

[1000] User:

[1001] The user initiates a dialogue with the 3D avatar through the application, for example by asking, "How was your day today?" This input is recorded as text and / or audio.

[1002] Device:

[1003] The user terminal converts the voice input into text data and sends it to the server, or if the input is text, sends it as is to the server.

[1004] server:

[1005] The server analyzes the received text data and uses a natural language processing (NLP) engine to understand its meaning. It generates an appropriate response and sends the text data back to the device. For example, it generates a response such as, "The weather was nice today, so I went for a walk."

[1006] Device:

[1007] The user device converts the response received from the server into voice output for the 3D avatar, which responds to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[1008] Generate and send scheduled reports

[1009] server:

[1010] The server periodically analyzes the conversations between users and their 3D avatars to identify specific topics and emotional trends. For example, it might discover from a week's worth of conversation logs that "health" and "weather" are frequently discussed topics.

[1011] server:

[1012] Based on the extracted information, periodic reports are generated that include information on the frequency of interactions, topic trends, and the user's emotional state.

[1013] server:

[1014] The generated regular reports are sent to the registered email addresses of family members, allowing family members in remote locations to easily keep track of the user's status.

[1015] This system allows elderly people living alone to enjoy natural conversation with someone who resembles a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to feel at ease and keep track of the user's condition.

[1016] The processing flow will be explained below.

[1017] Uploading photos and videos

[1018] Step 1: User Selects Media File

[1019] User:

[1020] Open the app on your smartphone or tablet and select photos and videos of your family: five photos and a 30-second video from your gallery.

[1021] Step 2: Upload your media files

[1022] Device:

[1023] The selected media file is temporarily stored in a buffer and then sent over the Internet to a server. Once the file transfer is complete, the device displays a confirmation message to the user saying "Upload Complete."

[1024] Personality Assessment

[1025] Step 3: Receive and save media files

[1026] server:

[1027] Receives uploaded photos and videos, stores them in the server's storage, records the storage path, and generates metadata for further processing.

[1028] Step 4: Analysis and personality assessment

[1029] server:

[1030] The system analyzes stored photos and videos. It uses image recognition and voice analysis technology to extract personality traits from family members' appearances and voices. For example, a "gentle and kind personality" can be extracted from the voice of a video.

[1031] 3D avatar generation

[1032] Step 5: Preparing for 3D modeling

[1033] server:

[1034] Based on the personality test results and the characteristics of the media file, a 3D modeling tool is used to generate 3D avatars of your family members, combining appearance and personality traits to create the perfect avatar.

[1035] Step 6: Saving and transferring your avatar

[1036] server:

[1037] The generated 3D avatar is stored on the server and transferred to the user's device. The avatar data is compressed to enable efficient transfer.

[1038] Step 7: Receiving and displaying your avatar

[1039] Device:

[1040] The app receives the 3D avatar transferred from the server, decompresses it, and displays it in the app. The user can move and rotate the avatar using touch controls.

[1041] Natural language dialogue

[1042] Step 8: Initiating a conversation

[1043] User:

[1044] The user initiates a conversation with the 3D avatar through the app, specifically by asking, "How was your day?"

[1045] Step 9: Submitting input data

[1046] Device:

[1047] The user's voice input is converted into text data and sent to the server. In addition, if the input is text, it is sent to the server as is.

[1048] Step 10: Analyzing the interaction data

[1049] server:

[1050] It analyzes the received text data, understands its meaning using a natural language processing (NLP) engine, generates an appropriate response, and sends the text data back to the device.

[1051] Step 11: View the response

[1052] Device:

[1053] The response received from the server is converted into voice output by the 3D avatar and returned to the user as a dialogue. For example, the avatar might respond with a natural expression, "The weather was nice today, so I went for a walk."

[1054] Generate and send scheduled reports

[1055] Step 12: Recording interaction history

[1056] server:

[1057] The interaction history between the user and the 3D avatar is periodically collected and stored in a database, and the frequency and content of the interactions are organized and recorded.

[1058] Step 13: Generate reports

[1059] server:

[1060] The system analyzes conversation history to extract specific topics and sentiment trends, summarizing major topics such as "health" and "weather" on a weekly basis.

[1061] Step 14: Submitting the report

[1062] server:

[1063] The generated periodic reports are sent to registered family members via email or other means, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[1064] This process allows elderly people living alone to enjoy natural conversation with people who resemble family members, reducing feelings of loneliness and providing emotional care. In addition, regular reports to families allow them to understand the user's condition with peace of mind.

[1065] Example 1

[1066] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1067] Elderly people tend to feel isolated and require psychological care. Furthermore, families living far away have limited means of understanding the situation of their elderly relatives, making it difficult for them to provide support for their lives with peace of mind. Current technology does not allow for direct communication with family members, and there is a lack of mechanisms for providing detailed information about the elderly's living situation to family members.

[1068] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1069] In this invention, the server includes means for analyzing the received image data and video data, means for extracting the user's appearance features and personality traits based on the analysis, and means for generating a 3D model based on the personality diagnosis results. This reduces the elderly person's sense of isolation through natural conversation with a presence that resembles a family member, and by recording the content of the conversation and sending regular reports to the family, it becomes easier for the family to understand the elderly person's condition even remotely.

[1070] A "user terminal" is an electronic device such as a smartphone, tablet, or personal computer that is operated by a user.

[1071] "Image data" refers to still image data uploaded from a user terminal.

[1072] "Video data" refers to dynamic video data uploaded from a user terminal.

[1073] A "buffer memory" is a temporary storage area for data, and is used to increase the efficiency of data transfer.

[1074] The "Internet" is a huge computer network that connects computer networks all over the world.

[1075] A "server" is a computer or software that provides services to other computers over a network.

[1076] "Analysis" is the act of processing image data and video data to extract their contents and characteristics.

[1077] "Appearance features" refer to visual features such as a person's face, posture, and clothing that are extracted from images and videos.

[1078] "Personality traits" refer to a person's personality characteristics obtained through voice analysis and behavioral analysis.

[1079] "3D Model" means a three-dimensional digital character generated by the server and simulated for the user.

[1080] A "generative AI model" is an artificial intelligence algorithm that generates new data based on specific patterns or data.

[1081] A "prompt" is a sentence or question that is input into a generative AI model, and is the input data that the AI ​​uses to generate responses and data.

[1082] A "natural language processing engine" is an algorithm or software that understands and appropriately processes human language.

[1083] "Voice input" refers to the process of converting words spoken by a user into digital data through a microphone.

[1084] "Text data" refers to data obtained by converting voice input into text information.

[1085] "Speech conversion" refers to the process of converting text data back into speech.

[1086] A "regular report" is a document that is generated periodically to inform family members of the user's conversation content and emotional tendencies.

[1087] The present invention is a system that enables simulated conversations with family members using a user terminal, a server, and a generation AI to reduce the sense of isolation felt by elderly people. Specific embodiments for implementing this system are described below.

[1088] Uploading photos and videos

[1089] User:

[1090] Users select and upload photos and videos of their family using their smartphone, tablet, or other device. Specifically, users access the gallery through the app interface and select five photos and a 30-second video.

[1091] Device:

[1092] The user device temporarily stores the selected photos and videos in a buffer memory. Then, it sends this data to the server via an HTTP POST request. During this process, the user device displays a progress bar to indicate the progress, and once the transfer is complete, it displays the message "Upload complete" to the user.

[1093] Personality Assessment

[1094] server:

[1095] The server receives the photos and videos sent from the devices and stores them in cloud storage (e.g., AWS S3). It then analyzes the data using image recognition (e.g., OpenCV) and voice analysis (e.g., Google Cloud Speech-to-Text) technologies to extract the family's physical features and personality traits.

[1096] 3D avatar generation

[1097] server:

[1098] Based on the appearance and voice characteristics discovered from the analysis, the server applies a personality diagnosis algorithm (e.g., decision tree or neural network) to derive characteristics such as a "gentle and kind personality."

[1099] server:

[1100] Then, 3D avatars of the family are generated using 3D modeling software (e.g., Blender), recreating facial contours, hairstyles, and clothing styles, and adding facial expressions and movements based on their personalities. The resulting 3D avatars are then sent to the user's device in a compressed format, using the SSL / TLS protocol to ensure secure data transfer.

[1101] Device:

[1102] The user device decompresses the received 3D avatar data and displays it in the application's 3D view. The user can change the orientation of the avatar by touching or swiping the screen.

[1103] Natural language dialogue

[1104] User:

[1105] The user initiates a dialogue with the 3D avatar using voice or text input, for example, by asking, "How was your day today?"

[1106] Device:

[1107] The user device converts the voice input into text data via an API (e.g., Google Cloud Speech-to-Text). If the input is text, it is sent directly to the server.

[1108] server:

[1109] The server analyzes the received text data and generates an appropriate response using a generative AI model (e.g., GPT-3), such as "The weather was nice today, so I went for a walk."

[1110] Device:

[1111] The user device converts the response text received from the server into speech using a speech synthesis API (e.g., Amazon Polly) and synchronizes it with the mouth movements of the 3D avatar, allowing the user to enjoy natural conversations through the avatar.

[1112] Generate and send scheduled reports

[1113] server:

[1114] The server periodically analyzes the conversation between the user and the 3D avatar to extract trends in topics and emotions. Specifically, it uses an NLP engine to classify the conversation logs by topic and evaluates the user's emotional state using an emotion analysis algorithm.

[1115] server:

[1116] Based on the analysis results, regular reports are generated that include information such as the frequency of conversations, main topics, and emotional state, and are displayed in easy-to-read graphs and charts.

[1117] server:

[1118] The generated periodic reports are automatically sent to the family's email addresses. The server uses the SMTP protocol to realize the email sending function.

[1119] For example, some input prompts for a generative AI model might include:

[1120] "Generate a 3D avatar of a very kind man based on the following dialogue: 'Hi, Grandpa! What were you doing today?' 'Hi, Grandma! It's very cool today, so I was reading in the garden.'"

[1121] This system allows elderly people to enjoy natural conversation with someone who resembles a family member, reducing feelings of isolation. It also makes it easier for family members to keep track of the user's condition through regular reports.

[1122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1123] Step 1: User selection of photos and videos

[1124] User:

[1125] The user launches the app on a smartphone, tablet, or other device, and operates the app's interface to select five family photos and a 30-second video from the gallery.

[1126] Input: Five photos of your family and a 30-second video.

[1127] Output: Selected photos and videos.

[1128] Step 2: Upload photos and videos

[1129] Device:

[1130] The user's device temporarily stores the selected photos and videos in a buffer memory, then sends the data to the server via an HTTP POST request, displaying a progress bar during the transfer and a message indicating the upload is complete when the transfer is complete.

[1131] Input: Photos and videos stored in buffer memory.

[1132] Output: A completion message sent to the server.

[1133] Step 3: Receiving and analyzing image and video data

[1134] server:

[1135] The server receives image and video data sent from the device and stores it in cloud storage (e.g., AWS S3). It then analyzes the data using image recognition (e.g., OpenCV) and voice analysis (e.g., Google Cloud Speech-to-Text) technologies to extract the family's physical features and personality traits.

[1136] Input: Received image and video data.

[1137] Output: Analysis of family members' physical features and personality traits.

[1138] Step 4: Applying the personality assessment algorithm

[1139] server:

[1140] The server applies a personality diagnosis algorithm (e.g., decision tree or neural network) based on the appearance and voice characteristics obtained from the analysis results to derive characteristics such as a "gentle and kind personality."

[1141] Input: Physical and vocal characteristics of family members.

[1142] Output: Personality test results.

[1143] Step 5: Generate a 3D avatar

[1144] server:

[1145] The server uses 3D modeling software (e.g., Blender) to generate 3D avatars of the family based on the personality test results and appearance data, and the generated avatars are sent to the user's device in a compressed format.

[1146] Input: Personality test results and physical characteristics data.

[1147] Output: Generated 3D avatar data.

[1148] Step 6: Displaying the 3D Avatar

[1149] Device:

[1150] The user device decompresses the received 3D avatar data and displays it in the application's 3D view. The user can change the orientation of the avatar by touching or swiping the screen.

[1151] Input: Received 3D avatar data.

[1152] Output: 3D avatar displayed on the application screen.

[1153] Step 7: Initiating a natural language dialogue

[1154] User:

[1155] The user initiates a dialogue with the 3D avatar through voice or text input, for example, by asking, "How was your day today?"

[1156] Input: Voice or text input.

[1157] Output: The input audio or text data.

[1158] Step 8: Process and transmit voice / text data

[1159] Device:

[1160] The user device converts the voice input into text data via an API (e.g., Google Cloud Speech-to-Text). If the input is text, it is sent directly to the server.

[1161] Input: Audio or text data.

[1162] Output: The text data sent to the server.

[1163] Step 9: Generate a response

[1164] server:

[1165] The server analyzes the received text data and generates an appropriate response using a generative AI model (e.g., GPT-3), such as "The weather was nice today, so I went for a walk."

[1166] Input: Text data.

[1167] Output: The generated response text.

[1168] Step 10: Returning a response and converting it

[1169] Device:

[1170] The user device converts the response text received from the server into speech using a speech synthesis API (e.g., Amazon Polly) and synchronizes it with the mouth movements of the 3D avatar, allowing the user to enjoy natural conversations through the avatar.

[1171] Input: The response text received from the server.

[1172] Output: Voice response by avatar.

[1173] Step 11: Record and analyze the conversation

[1174] server:

[1175] The server periodically analyzes the conversation between the user and the 3D avatar to extract trends in topics and emotions. Specifically, it uses an NLP engine to classify the conversation logs by topic and evaluates the user's emotional state using an emotion analysis algorithm.

[1176] Input: Log data of the conversation.

[1177] Output: Trend analysis results of topics and sentiment.

[1178] Step 12: Generate and send scheduled reports

[1179] server:

[1180] The server generates periodic reports based on the analysis results and automatically sends them to the family's email addresses, including information on the frequency of conversations, main topics, and emotional state.

[1181] Input: Topic and sentiment trend analysis results.

[1182] Output: Regular reports sent to families.

[1183] This allows the elderly to enjoy natural conversation with people who resemble family members, reducing feelings of loneliness.Families can also keep track of the elderly's condition through regular reports, even when they are far away.

[1184] (Application example 1)

[1185] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1186] There are existing systems that provide simulated dialogue with family members to reduce the sense of isolation felt by elderly people. However, these systems make it difficult for users to enjoy a sophisticated dialogue experience, such as receiving advice from family members in more specific life situations. Furthermore, there is a lack of systems that can generate specific dialogue responses tailored to specific situations. There is a growing demand for solutions to this problem.

[1187] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1188] In this invention, the server includes means for uploading image data and video data from a user terminal, means for analyzing the image data and video data on the server and diagnosing the user's personality, means for generating a 3D model based on the personality diagnosis results, means for transferring the 3D model to a head-mounted display terminal and displaying it, means for generating a natural language dialogue between the user and the 3D model via the head-mounted display terminal, and means for recording the content of the dialogue and sending regular reports to registered family members. This allows elderly people to experience more realistic dialogue with their families and receive appropriate advice in specific life situations.

[1189] A "user terminal" is an electronic device that allows a user to take photos and videos and upload them to a server.

[1190] "Image data and video data" refers to photographs and video files taken or selected by a user using a user terminal.

[1191] A "server" is a remote computer system that receives image and video data sent from a user terminal and analyzes and processes it.

[1192] "Personality diagnosis" is the process of identifying the subject's personality and characteristics by analyzing image and video data.

[1193] A "3D model" is a three-dimensional avatar generated on a computer based on the results of analysis, and is a digital object that reflects a person's appearance and personality.

[1194] A "head-mounted display terminal" is a display device worn on the head, and is a device that provides VR (virtual reality) and AR (augmented reality) experiences.

[1195] "Natural language interaction" is communication between humans and computers in which the user speaks in spoken language and the system responds.

[1196] A "regular report" is a document that the system records and analyzes the content of conversations and periodically reports on the user's status and conversation trends.

[1197] The present invention provides a virtual family shop system that allows users to get family advice in specific situations. Specific embodiments are given below.

[1198] Hardware and Software Configuration

[1199] User device: An electronic device such as a smartphone or tablet that a user uses to take photos or videos.

[1200] Server: A computer system that runs on the cloud and analyzes image and video data, generates 3D models, and performs natural language processing.

[1201] Software used: Google Cloud Speech-to-Text, Google Dialogflow, Blender, GPT-4

[1202] Head-mounted display device: A display device worn by the user to interact with 3D models. An example of this is the Oculus Quest 2.

[1203] Data processing and calculation procedures

[1204] 1. Uploading photos and videos

[1205] The user wears a head-mounted display device and takes photos and videos of their family. This data is temporarily stored on the user's device and then sent to a server via the Internet. The server analyzes the received data and performs a personality diagnosis.

[1206] 2. Personality Test

[1207] The software used on the server (Google Cloud Speech-to-Text and Google Dialogflow) analyzes the appearance and personality of family members from the received photos and videos. For example, facial features are recognized from still images, and voice tone and speaking style are analyzed from videos. This allows for the identification of personalities and characteristics.

[1208] 3. 3D Avatar Generation

[1209] The server uses Blender to generate 3D avatars based on image data and personality assessments. The generated 3D avatars reflect the family members' appearance information and personality assessment results.

[1210] 4. Natural Language Interaction

[1211] When a user wearing a head-mounted display device starts a conversation with an avatar, their voice is converted into text data by Google Cloud Speech-to-Text, and then Google Dialogflow analyzes the text and generates an appropriate response. The response is then converted into natural-sounding speech using GPT-4 and played back on the head-mounted display device.

[1212] Specific examples

[1213] When choosing clothes, the user wears a head-mounted display device and asks, "Mom, which outfit looks good on me?" The generated 3D avatar responds, "I think the red dress looks great on you." This allows the user to experience a conversation with a family member and receive specific advice.

[1214] Prompt Sentence Examples

[1215] The user types the following into the system:

[1216] "Simulate a situation where a 3D avatar of a family member is giving product advice to a user. Code it to respond to the question 'Which dress do you like?' with 'A red dress would look good on you.'"

[1217] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1218] Step 1:

[1219] The user wears a head-mounted display device and takes photos and videos of their family. The captured data is temporarily stored on the user's device and then sent to a server via the Internet. The input is the photo or video data, and the output is the data sent to the server.

[1220] Step 2:

[1221] The server analyzes the received image and video data and uses image recognition and voice analysis technology to extract the family's physical features and personality traits. The input is image and video data, and the output is appearance information and personality assessment results. Specifically, Google Cloud Speech-to-Text and Google Dialogflow are used to recognize facial features from photos and analyze voice tone and speaking style from videos.

[1222] Step 3:

[1223] The server uses Blender to generate a 3D avatar based on image data and personality assessment. The input is appearance information and personality assessment results, and the output is a 3D avatar. Specifically, Blender is used to model the avatar based on appearance information and assessment results.

[1224] Step 4:

[1225] The generated 3D avatar is transferred from the server to the head-mounted display device and displayed. The input is the 3D avatar, and the output is the avatar displayed on the user's HMD. Specifically, the data is sent to the HMD using a data transfer protocol.

[1226] Step 5:

[1227] The user speaks to an avatar through a head-mounted display terminal. The user's voice input is captured by the HMD's microphone and sent to the user terminal. The input is the user's voice data, and the output is the text conversion result of the voice data.

[1228] Step 6:

[1229] The server receives the voice data and converts it to text data using Google Cloud Speech-to-Text. The input is voice data and the output is text data. Specifically, it calls an API that converts the voice data to text data.

[1230] Step 7:

[1231] The server analyzes the text data and generates an appropriate response using Google Dialogflow and GPT-4. The input is text data, and the output is response text data. Specifically, it uses Dialogflow to analyze the intent of the question and GPT-4 to generate natural dialogue.

[1232] Step 8:

[1233] The response text data generated by the server is converted into voice data and sent to the head-mounted display terminal. The input is the response text data, and the output is voice data. Specifically, the voice data is generated using a text-to-speech API and sent to the HMD.

[1234] Step 9:

[1235] The head-mounted display terminal plays back audio data and responds to the user as a dialogue. The input is audio data, and the output is an audio response that the user can hear. Specifically, the audio is played back through the HMD's speakers.

[1236] Step 10:

[1237] The server records the dialogue between the user and the 3D avatar, periodically analyzes it, and generates a report. The input is the dialogue log, and the output is the analysis results and a report. Specifically, the dialogue log is stored in a database, and an algorithm is executed to periodically analyze the content and generate a report.

[1238] Step 11:

[1239] The generated regular report is sent to the registered family member's email address. The input is the report, and the output is an email sent to the family member's email address. The specific operation is to use the email sending API to send the regular report.

[1240] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1241] This invention is a system that uses a user terminal, a server, and a generation AI to enable simulated conversations with family members, and further combines an emotion engine to recognize and respond to emotions in order to reduce feelings of isolation among the elderly. Specific embodiments for implementing this system are described below.

[1242] Uploading photos and videos

[1243] User:

[1244] A user selects and uploads family photos and videos using a user device such as a smartphone or tablet. As a specific example, a user selects five family photos and a 30-second video from the smartphone gallery.

[1245] Device:

[1246] The user's device temporarily stores the selected photos and videos in a buffer and then transmits them to the server via the Internet. Once the photos and videos have been transmitted, the device displays a confirmation message to the user.

[1247] Personality diagnosis and emotion recognition

[1248] server:

[1249] The server analyzes the received photos and videos. Using image recognition and voice analysis technology, it analyzes the family members' appearance and voice to extract personality traits and emotions. For example, it recognizes facial features from still images and analyzes the tone of voice and speaking style from videos. This results in a "gentle and kind personality" and "current emotional state."

[1250] 3D avatar generation and emotional reflection

[1251] server:

[1252] The server uses a 3D modeling tool based on the analysis results to generate 3D avatars of the family. These avatars combine appearance information, personality assessment results, and emotional states, and are designed to resemble the appearance and behavior of the actual family. The generated avatars are then transferred to the user's device.

[1253] Device:

[1254] The user's device decompresses the 3D avatar received from the server and displays it on the application screen. The user can move the avatar and change its orientation by touching it. The avatar's facial expression and attitude can change depending on the user's emotional state.

[1255] Natural language dialogue

[1256] User:

[1257] The user initiates a dialogue with the 3D avatar through the application, for example by asking, "How was your day today?" This input is recorded as text and / or audio.

[1258] Device:

[1259] The user terminal converts the voice input into text data and sends it to the server, or if the input is text, sends it as is to the server.

[1260] server:

[1261] The system analyzes the received text data and uses a natural language processing (NLP) engine to understand its meaning. It then uses an emotion engine to recognize the user's emotions and generates an appropriate response. For example, it could respond with a happy expression on the avatar, saying, "The weather was nice today, so I went for a walk."

[1262] Device:

[1263] The user device converts the response received from the server into the voice and facial expression output of the 3D avatar, and returns it to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[1264] Generate and send scheduled reports

[1265] server:

[1266] The conversation history between the user and the 3D avatar is periodically collected and stored in a database. The emotional engine is also used to analyze the user's emotional tendencies during the conversation.

[1267] server:

[1268] Based on the collected data, it extracts trends in specific topics and emotions and generates regular reports. For example, it can report from a week's worth of conversation logs that "health" and "weather" were frequently discussed, and that users were primarily "calm."

[1269] server:

[1270] The generated reports are sent to the registered family member's email address on a regular basis, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[1271] This system allows elderly people living alone to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. Furthermore, the introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[1272] The processing flow will be explained below.

[1273] MODE FOR CARRYING OUT THE INVENTION

[1274] Uploading photos and videos

[1275] Step 1:

[1276] User:

[1277] Open the app on your smartphone or tablet and select photos and videos of your family: five photos and a 30-second video from your gallery.

[1278] Step 2:

[1279] Device:

[1280] The selected photos and videos are temporarily stored in a buffer and then sent to a server over the Internet. Once the photos and videos have been sent, the device will display a confirmation message to the user saying "Upload complete."

[1281] Personality diagnosis and emotion recognition

[1282] Step 3:

[1283] server:

[1284] Receives uploaded photos and videos, stores them in the server's storage, records the storage path, and generates metadata for further processing.

[1285] Step 4:

[1286] server:

[1287] The system analyzes stored photos and videos. It uses image recognition and audio analysis technology to analyze the appearance and voice of family members to extract personality traits and emotions. For example, it can extract characteristics such as a "calm and gentle personality" and "current emotional state" from the audio in a video.

[1288] 3D avatar generation and emotional reflection

[1289] Step 5:

[1290] server:

[1291] Based on the analysis results, a 3D modeling tool is used to generate 3D avatars of the family members, combining appearance information with personality assessment results and emotional state to create the optimal avatar.

[1292] Step 6:

[1293] server:

[1294] The generated 3D avatar is stored on the server and transferred to the user's device. The avatar data is compressed to enable efficient transfer.

[1295] Step 7:

[1296] Device:

[1297] The app receives the 3D avatar transferred from the server, decompresses it, and displays it in the app. The user can move the avatar and change its orientation using touch controls. The avatar also has the ability to change its facial expression and attitude depending on the user's emotional state.

[1298] Natural language dialogue

[1299] Step 8:

[1300] User:

[1301] The user initiates a conversation with the 3D avatar through the app. For example, the user might say, "How was your day?"

[1302] Step 9:

[1303] Device:

[1304] The user's voice input is converted into text data and sent to the server. In addition, if the input is text, it is sent to the server as is.

[1305] Step 10:

[1306] server:

[1307] The system analyzes the received text data and uses a natural language processing (NLP) engine and an emotion engine to understand the meaning and emotion. It then generates an appropriate response and sends that text data back to the device. For example, it might respond with "The weather was nice today, so I went for a walk," along with a happy expression on the avatar.

[1308] Step 11:

[1309] Device:

[1310] The response received from the server is converted into the voice and facial expression output of the 3D avatar, and returned to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[1311] Generate and send scheduled reports

[1312] Step 12:

[1313] server:

[1314] The conversation history between the user and the 3D avatar is periodically collected and stored in a database. The emotional engine is also used to analyze the user's emotional tendencies during the conversation.

[1315] Step 13:

[1316] server:

[1317] Based on the collected data, it extracts trends in specific topics and emotions and generates regular reports. For example, it can report from a week's worth of conversation logs that "health" and "weather" were frequently discussed, and that users were primarily "calm."

[1318] Step 14:

[1319] server:

[1320] The generated reports are sent to the registered family member's email address on a regular basis, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[1321] This system allows elderly people living alone to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. Furthermore, the introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[1322] Example 2

[1323] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1324] There is a growing need for systems that can reduce the sense of isolation felt by the elderly and provide them with psychological care. However, existing systems lack natural dialogue and emotion recognition, making it difficult to provide appropriate responses and emotional care to users. Furthermore, it is difficult to communicate with family members and share the situation, and there is a lack of means to grasp the user's condition in a timely manner.

[1325] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1326] In this invention, the server includes means for uploading digital image data and video data from a user terminal, means for analyzing the digital image data and video data on the server and diagnosing the user's personality and emotions, means for generating a 3D character based on the personality and emotion diagnosis results, means for transferring the 3D character to the user terminal and displaying it, means for generating a natural language dialogue between the user and the 3D character via the user terminal, means for recording the dialogue and sending regular reports to registered family members, and means for generating responses according to the user's emotions using an emotion engine. This enables natural dialogue, allows appropriate care to be provided according to the user's emotional state, and enables family members to understand the user's condition in a timely manner.

[1327] A "user terminal" is an information processing device such as a smartphone, tablet, or personal computer operated by a user.

[1328] "Digital image data" refers to data of still images taken by a user, and is saved in a format such as JPEG or PNG.

[1329] "Video data" refers to video data shot by a user and saved in formats such as MP4 or AVI.

[1330] A "server" is a remote information processing device that provides functions such as data analysis, processing, and storage.

[1331] "Personality Assessment" is the process of identifying a user's personality traits through the analysis of image and audio data.

[1332] "Emotional diagnosis" is the process of identifying a user's current emotional state through the analysis of image and audio data.

[1333] A "3D character" is a three-dimensional virtual character generated based on the user's photo and video data.

[1334] "Natural language dialogue" refers to communication between a user and a system in natural language using voice or text.

[1335] "Regular reports" are reports generated based on the dialogue between the user and the 3D character and sent to family members.

[1336] An "emotion engine" is a software module that analyzes a user's emotions and generates an appropriate response.

[1337] A "natural language processing engine" is a software module that analyzes input text data and understands its meaning.

[1338] This invention is a system that uses a user terminal, a server, and a generation AI to enable simulated conversations with family members, and further combines an emotion engine to recognize and respond to emotions in order to reduce feelings of isolation among the elderly. Specific forms for realizing this system are shown below.

[1339] System Configuration

[1340] User terminal

[1341] The user terminals are smartphones, tablets, or personal computers. These devices provide the ability to take and store photos and videos of family members, and can transmit the data to a server via the Internet. They also provide an interface for displaying the generated 3D characters and interacting with them through applications.

[1342] server

[1343] The server receives and analyzes photo and video data uploaded from user devices and processes the data using generative AI models and emotion engines. The server has the following main functions:

[1344] 1. Data reception and storage function

[1345] 2. Image recognition and speech analysis functions (e.g., OpenCV, Google Cloud Speech-to-Text)

[1346] 3. Personality diagnosis and emotion recognition (e.g., NLP engines)

[1347] 4. 3D character generation function (e.g. Blender)

[1348] 5. Natural language processing engines (e.g., GPT-3, BERT)

[1349] 6. Emotion Engine

[1350] Explanation of specific processing contents

[1351] 1. Upload your data

[1352] A user selects and uploads family photos and videos using a user terminal such as a smartphone or tablet. For example, a user selects five family photos and a 30-second video from the smartphone gallery.

[1353] 2. Data Analysis

[1354] The server receives and analyzes photo and video data sent from the user's device. Image recognition is performed using OpenCV, and voice data is converted to text using the Google Cloud Speech-to-Text API for analysis. Personality diagnosis and emotion recognition are performed using an NLP engine, which extracts personality and emotional tendencies.

[1355] 3. 3D character generation

[1356] The server uses 3D modeling tools such as Blender to generate a 3D character based on the analysis results. This character reflects the appearance information obtained from the still images and the personality and emotional state extracted from the video. The generated character is then transferred to the user's device.

[1357] 4. Natural Language Interaction

[1358] The user initiates a natural language dialogue with the 3D character through the application. For example, they might say, "How was your day today?" This input is recorded as voice or text and sent from the user's device to the server. The server uses an NLP engine to analyze the input data and an emotion engine to generate an appropriate response. The generated response is converted into voice or facial expression and expressed by the 3D character on the user's device.

[1359] 5. Generate and send scheduled reports

[1360] The server periodically collects the user's interaction history with the 3D character and stores it in a database. It uses an emotion engine to analyze the user's emotional tendencies during the interaction, extracting specific topics and emotional patterns to generate periodic reports. These reports are sent to registered family members' email addresses, reporting the user's recent status and emotional tendencies.

[1361] Examples of concrete examples and prompts

[1362] Examples:

[1363] When a user asks, "How was your day today?", the server analyzes the question and generates a response such as, "The weather was nice today, so I went for a walk." The 3D character then displays a happy expression.

[1364] Example prompt sentence:

[1365] "The user asks, 'How was your day?' Generate a response that reflects a happy mood."

[1366] This system allows elderly people to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. The introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[1367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1368] Step 1:

[1369] A user uses a smartphone or tablet to select and upload digital image data (photos) and video data (movies) of their family members. Specifically, the user opens the device's gallery app, selects five photos and a 30-second video, and taps the "Upload" button. This action saves the digital image data and video data in the device's buffer. The input is the selected image data and video data, and the output is the data saved in the buffer.

[1370] Step 2:

[1371] The device sends the image and video data stored in the buffer to the server. The device encrypts the data and uploads it to the server via the Internet. When the transmission is complete, the device displays a confirmation message to the user saying "Upload complete." The input is the data stored in the buffer, and the output is the data sent to the server.

[1372] Step 3:

[1373] The server receives image and video data from the user's device. The server checks the format of the data and prepares it for image recognition and voice analysis. The input is the data sent to the server, and the output is data in a format that can be analyzed.

[1374] Step 4:

[1375] The server uses OpenCV to analyze image data and extract the facial features of family members. It also uses the Google Cloud Speech-to-Text API to convert the video's audio data into text. An NLP engine is used to analyze speaking style and tone of voice from the text data to extract the family members' personalities and current emotional state. The input is data in an analyzable format, and the output is facial feature information, textual audio data, and personality and emotional diagnosis results.

[1376] Step 5:

[1377] The server uses a 3D modeling tool such as Blender to generate 3D characters of the family members based on the analysis results obtained in the previous step. These characters reflect facial feature information, personality diagnosis results, and emotional state. The input is facial feature information, text-converted voice data, and personality and emotional diagnosis results, and the output is the generated 3D characters.

[1378] Step 6:

[1379] The server transfers the data of the generated 3D character to the user terminal. The input is the data of the generated 3D character, and the output is the 3D character data sent to the user terminal.

[1380] Step 7:

[1381] The device decompresses the 3D character data received from the server and displays it on the application screen. The user can control the character by touching it. The avatar can change its facial expression and attitude depending on the user's emotional state. The input is the 3D character data sent from the server, and the output is the 3D character displayed in the application.

[1382] Step 8:

[1383] The user initiates a dialogue with the 3D character through the application, for example by asking, "How was your day today?" This input is recorded as voice or text. The input is what the user said, and the output is voice or text data.

[1384] Step 9:

[1385] The terminal converts the voice input into text data and sends it to the server. If the input is text, it sends it to the server as is. The input is voice or text data, and the output is the text data sent to the server.

[1386] Step 10:

[1387] The server analyzes the received text data and uses an NLP engine to understand the meaning of the input. It then uses an emotion engine to recognize the user's emotions and generates an appropriate response. For example, the response generated might be, "The weather was nice today, so I went for a walk." The input is text data, and the output is the generated response.

[1388] Step 11:

[1389] The device converts the response received from the server into voice and facial expressions, and returns it to the user in the form of a 3D character response. The avatar responds with natural facial expressions and movements. The input is the generated response, and the output is the response reflected in voice and facial expressions.

[1390] Step 12:

[1391] The server periodically collects the interaction history between the user and the 3D character and stores it in a database. The input is the interaction history, and the output is the stored data.

[1392] Step 13:

[1393] The server uses an emotion engine to analyze the user's emotional tendencies during the conversation. This analysis extracts specific topics and emotional patterns. The input is the saved conversation history data, and the output is the extracted emotional tendencies and topic patterns.

[1394] Step 14:

[1395] The server generates periodic reports based on the analysis results. For example, it reports that "health" and "weather" were frequently discussed based on a week's worth of conversation logs, and that the user was primarily "calm." The input is the analysis results, and the output is the generated periodic reports.

[1396] Step 15:

[1397] The server sends the generated periodic report to the registered family member's email address. The report includes information such as the frequency of conversations, topic trends, and the user's emotional state. The input is the generated periodic report, and the output is the sent report.

[1398] (Application example 2)

[1399] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1400] In addition to systems that provide psychological care through simulated conversations with family members for elderly people and users who feel isolated, there is a need for effective support in providing product explanations and customer service to customers in physical stores. Furthermore, it is necessary to improve the level of service in stores by using a system that uses avatars that can accurately convey product features during conversations with customers.

[1401] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data and video data from a user terminal, means for analyzing the image data and video data and diagnosing the user's personality, means for generating a 3D model based on the personality diagnosis results, means for uploading product photos and videos in the store and analyzing the product features, means for generating a 3D product avatar based on the features and displaying it on the user terminal, means for generating a natural language dialogue between the user and the 3D product avatar via the user terminal, and means for recording the content of the dialogue and generating a service report for the user. This enables psychological care for elderly users and users who feel isolated, and effective customer service support for customers in physical stores.

[1402] A "user terminal" is a device operated by a user, such as a computer, smartphone, or tablet.

[1403] "Image data" refers to digital image files such as photographs uploaded by users.

[1404] "Video data" refers to digital video files such as video clips uploaded by users.

[1405] A "server" is a computer system for receiving and processing data sent from a user terminal.

[1406] "Personality diagnosis" is a method of analyzing the personality traits and emotional state of a user or subject.

[1407] A "3D model" is a three-dimensional virtual object generated using computer graphics.

[1408] "Natural language dialogue" refers to communication using language that humans use on a daily basis.

[1409] "Recording the content of the dialogue" means saving the content of the dialogue between the user and the 3D model in a database or the like.

[1410] A "regular report" is a report that summarizes the content of conversations and the status of users over a certain period of time.

[1411] "Product features" refers to information about the physical properties and functions of a product.

[1412] A "3D product avatar" is a three-dimensional character that virtually recreates a product.

[1413] A "service report" is a report that summarizes the content of conversations with users and business negotiation history.

[1414] A "natural language processing engine" is software that analyzes input in natural language and generates appropriate responses.

[1415] An "emotion recognition engine" is software that analyzes and recognizes the emotional state of a user or target.

[1416] To implement this invention, the following system is used: The main components of the system are a user terminal, a server, a generative AI model, and an emotion recognition engine.

[1417] Users upload image and video data using user devices such as smartphones and tablets. This data is sent to a server via the Internet. The server analyzes the received image and video data to extract characteristics of the user and the product. For example, it analyzes the user's personality and the product's external characteristics. It also uses an emotion recognition engine to diagnose the user's emotions.

[1418] The server then uses a 3D modeling tool based on the analysis results to generate 3D models of users and products. For example, it generates avatars for "gentle and kind personalities" or "healthy products" based on facial features obtained from still images and voice tones obtained from videos. These 3D models are then transferred from the server to the user's device and displayed on the application screen. Users can then have natural conversations with the avatars.

[1419] User interaction input is recorded as voice or text and sent from the user's device to the server. The server then uses a natural language processing engine to analyze the input and generate an appropriate response. For example, if the user asks, "How was your day today?", the server might respond with, "The weather was nice today, so I went for a walk," and display a happy expression on the avatar.

[1420] Furthermore, the server records the content of conversations and generates periodic reports based on them. These reports are sent to registered family members and users and include information such as the frequency of conversations, topic trends, and emotional state. For example, a week's worth of conversation logs may be analyzed to determine that "health" and "weather" were frequently discussed, and the server may report to family members that the user was primarily "calm."

[1421] To illustrate, here is a prompt statement that generates the following program:

[1422] "Generate a Python program for a mobile application that uses uploaded product photos and videos to create a 3D avatar representing the products in a store. The avatar should be able to interact with users through natural language conversation. The program should use TensorFlow for emotion recognition, BERT for natural language processing, and SpeechRecognition for converting audio input to text."

[1423] This system not only enables psychological care for elderly people and users who feel isolated, but also enables effective customer service support for physical stores.

[1424] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1425] Step 1:

[1426] A user selects and uploads image data and video data using a user device such as a smartphone or tablet. The user device temporarily stores the selected data in buffer memory and sends it to the server. The input is image data and video data, and the output is the data sent to the server. Specifically, the user selects five product photos and a 30-second promotional video, and clicks the upload button.

[1427] Step 2:

[1428] The server receives image and video data sent from the user's device and prepares it for analysis. The server analyzes the data using image recognition and audio analysis technology to extract product features. The input is data from the user's device, and the output is the analysis results (product features). Specifically, TensorFlow is used to extract product features such as color, shape, and logo from images, and to analyze audio tone from videos.

[1429] Step 3:

[1430] The server uses a 3D modeling tool based on the analysis results to generate a 3D avatar of the product. This avatar combines information about the product's appearance and features. The input is the analysis results, and the output is the generated 3D avatar. Specifically, the operation uses a modeling tool such as Blender to generate a 3D model that faithfully reproduces the product's appearance.

[1431] Step 4:

[1432] The server transfers the generated 3D avatar to the user's device. The user's device receives the avatar and displays it on the application screen. The input is the 3D avatar, and the output is the avatar displayed on the user's device. Specifically, the user opens the app on their smartphone, and the 3D avatar of the product appears on the screen.

[1433] Step 5:

[1434] The user initiates a natural language dialogue with a 3D avatar via the user terminal. The user's voice input is converted into text on the user terminal and sent to the server. The input is the user's voice, and the output is text data. Specifically, the action is to ask, "What are the features of this product?"

[1435] Step 6:

[1436] The server analyzes the received text data and understands its meaning using a natural language processing (NLP) engine. It also uses an emotion recognition engine to recognize the user's emotions and generate an appropriate response. The input is text data, and the output is a response text. Specifically, it uses BERT to analyze the user's question and generates the response, "This product uses organic ingredients."

[1437] Step 7:

[1438] The server sends the generated response to the user's device and converts it into the voice and facial expression output of the 3D avatar. The user's device receives the response, and the avatar responds with natural facial expressions and movements. The input is the response text, and the output is voice and facial expressions. Specifically, the user listens to the avatar's explanation on their smartphone, and the avatar explains with a smile.

[1439] Step 8:

[1440] The server records the content of the conversation between the user and the 3D avatar and periodically analyzes it. Specific topics and emotional trends are extracted from the conversation and compiled into a service report. The input is the conversation content, and the output is the analysis results and report. Specifically, the server analyzes the conversation log for a week and finds that there were many questions about "health," and compiles this into a report.

[1441] Step 9:

[1442] The server periodically sends service reports to registered family members and users. These reports include information such as the frequency of interactions, topic trends, and the user's emotional state. The input is the service report, and the output is email or notification. Specifically, this operation sends a "weekly interaction history report" to the family member's email address.

[1443] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1444] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1445] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1446] [Fourth embodiment]

[1447] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1448] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1449] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1450] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1451] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1452] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1453] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1454] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1455] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1456] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1457] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1458] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1459] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1460] This invention is a system that enables simulated conversations with family members using a user terminal, a server, and a generation AI to reduce the sense of isolation felt by elderly people. Specific embodiments for implementing this system are described below.

[1461] Uploading photos and videos

[1462] User:

[1463] A user selects and uploads family photos and videos using a user device such as a smartphone or tablet. As a specific example, a user selects five family photos and a 30-second video from the smartphone gallery.

[1464] Device:

[1465] The user's device temporarily stores the selected photos and videos in a buffer and then transmits them to the server via the Internet. Once the photos and videos have been transmitted, the device displays a confirmation message to the user.

[1466] Personality Assessment

[1467] server:

[1468] The server analyzes the received photos and videos. Using image recognition and voice analysis technology, it extracts the family members' physical features and personality traits. For example, it recognizes facial features from still images and analyzes the tone of voice and speaking style from videos. This allows it to extract traits such as a "gentle and kind personality."

[1469] 3D avatar generation

[1470] server:

[1471] The server uses a 3D modeling tool based on the analysis results to generate 3D avatars of the family. These avatars combine appearance information and personality assessment results, and resemble the appearance and behavior of the actual family. The generated avatars are then transferred to the user's device.

[1472] Device:

[1473] The user device decompresses the 3D avatar received from the server and displays it on the application screen. The user can move and change the orientation of the avatar by touching it.

[1474] Natural language dialogue

[1475] User:

[1476] The user initiates a dialogue with the 3D avatar through the application, for example by asking, "How was your day today?" This input is recorded as text and / or audio.

[1477] Device:

[1478] The user terminal converts the voice input into text data and sends it to the server, or if the input is text, sends it as is to the server.

[1479] server:

[1480] The server analyzes the received text data and uses a natural language processing (NLP) engine to understand its meaning. It generates an appropriate response and sends the text data back to the device. For example, it generates a response such as, "The weather was nice today, so I went for a walk."

[1481] Device:

[1482] The user device converts the response received from the server into voice output for the 3D avatar, which responds to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[1483] Generate and send scheduled reports

[1484] server:

[1485] The server periodically analyzes the conversations between users and their 3D avatars to identify specific topics and emotional trends. For example, it might discover from a week's worth of conversation logs that "health" and "weather" are frequently discussed topics.

[1486] server:

[1487] Based on the extracted information, periodic reports are generated that include information on the frequency of interactions, topic trends, and the user's emotional state.

[1488] server:

[1489] The generated regular reports are sent to the registered email addresses of family members, allowing family members in remote locations to easily keep track of the user's status.

[1490] This system allows elderly people living alone to enjoy natural conversation with someone who resembles a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to feel at ease and keep track of the user's condition.

[1491] The processing flow will be explained below.

[1492] Uploading photos and videos

[1493] Step 1: User Selects Media File

[1494] User:

[1495] Open the app on your smartphone or tablet and select photos and videos of your family: five photos and a 30-second video from your gallery.

[1496] Step 2: Upload your media files

[1497] Device:

[1498] The selected media file is temporarily stored in a buffer and then sent over the Internet to a server. Once the file transfer is complete, the device displays a confirmation message to the user saying "Upload Complete."

[1499] Personality Assessment

[1500] Step 3: Receive and save media files

[1501] server:

[1502] Receives uploaded photos and videos, stores them in the server's storage, records the storage path, and generates metadata for further processing.

[1503] Step 4: Analysis and personality assessment

[1504] server:

[1505] The system analyzes stored photos and videos. It uses image recognition and voice analysis technology to extract personality traits from family members' appearances and voices. For example, a "gentle and kind personality" can be extracted from the voice of a video.

[1506] 3D avatar generation

[1507] Step 5: Preparing for 3D modeling

[1508] server:

[1509] Based on the personality test results and the characteristics of the media file, a 3D modeling tool is used to generate 3D avatars of your family members, combining appearance and personality traits to create the perfect avatar.

[1510] Step 6: Saving and transferring your avatar

[1511] server:

[1512] The generated 3D avatar is stored on the server and transferred to the user's device. The avatar data is compressed to enable efficient transfer.

[1513] Step 7: Receiving and displaying your avatar

[1514] Device:

[1515] The app receives the 3D avatar transferred from the server, decompresses it, and displays it in the app. The user can move and rotate the avatar using touch controls.

[1516] Natural language dialogue

[1517] Step 8: Initiating a conversation

[1518] User:

[1519] The user initiates a conversation with the 3D avatar through the app, specifically by asking, "How was your day?"

[1520] Step 9: Submitting input data

[1521] Device:

[1522] The user's voice input is converted into text data and sent to the server. In addition, if the input is text, it is sent to the server as is.

[1523] Step 10: Analyzing the interaction data

[1524] server:

[1525] It analyzes the received text data, understands its meaning using a natural language processing (NLP) engine, generates an appropriate response, and sends the text data back to the device.

[1526] Step 11: View the response

[1527] Device:

[1528] The response received from the server is converted into voice output by the 3D avatar and returned to the user as a dialogue. For example, the avatar might respond with a natural expression, "The weather was nice today, so I went for a walk."

[1529] Generate and send scheduled reports

[1530] Step 12: Recording interaction history

[1531] server:

[1532] The interaction history between the user and the 3D avatar is periodically collected and stored in a database, and the frequency and content of the interactions are organized and recorded.

[1533] Step 13: Generate reports

[1534] server:

[1535] The system analyzes conversation history to extract specific topics and sentiment trends, summarizing major topics such as "health" and "weather" on a weekly basis.

[1536] Step 14: Submitting the report

[1537] server:

[1538] The generated periodic reports are sent to registered family members via email or other means, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[1539] This process allows elderly people living alone to enjoy natural conversation with people who resemble family members, reducing feelings of loneliness and providing emotional care. In addition, regular reports to families allow them to understand the user's condition with peace of mind.

[1540] Example 1

[1541] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1542] Elderly people tend to feel isolated and require psychological care. Furthermore, families living far away have limited means of understanding the situation of their elderly relatives, making it difficult for them to provide support for their lives with peace of mind. Current technology does not allow for direct communication with family members, and there is a lack of mechanisms for providing detailed information about the elderly's living situation to family members.

[1543] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1544] In this invention, the server includes means for analyzing the received image data and video data, means for extracting the user's appearance features and personality traits based on the analysis, and means for generating a 3D model based on the personality diagnosis results. This reduces the elderly person's sense of isolation through natural conversation with a presence that resembles a family member, and by recording the content of the conversation and sending regular reports to the family, it becomes easier for the family to understand the elderly person's condition even remotely.

[1545] A "user terminal" is an electronic device such as a smartphone, tablet, or personal computer that is operated by a user.

[1546] "Image data" refers to still image data uploaded from a user terminal.

[1547] "Video data" refers to dynamic video data uploaded from a user terminal.

[1548] A "buffer memory" is a temporary storage area for data, and is used to increase the efficiency of data transfer.

[1549] The "Internet" is a huge computer network that connects computer networks all over the world.

[1550] A "server" is a computer or software that provides services to other computers over a network.

[1551] "Analysis" is the act of processing image data and video data to extract their contents and characteristics.

[1552] "Appearance features" refer to visual features such as a person's face, posture, and clothing that are extracted from images and videos.

[1553] "Personality traits" refer to a person's personality characteristics obtained through voice analysis and behavioral analysis.

[1554] "3D Model" means a three-dimensional digital character generated by the server and simulated for the user.

[1555] A "generative AI model" is an artificial intelligence algorithm that generates new data based on specific patterns or data.

[1556] A "prompt" is a sentence or question that is input into a generative AI model, and is the input data that the AI ​​uses to generate responses and data.

[1557] A "natural language processing engine" is an algorithm or software that understands and appropriately processes human language.

[1558] "Voice input" refers to the process of converting words spoken by a user into digital data through a microphone.

[1559] "Text data" refers to data obtained by converting voice input into text information.

[1560] "Speech conversion" refers to the process of converting text data back into speech.

[1561] A "regular report" is a document that is generated periodically to inform family members of the user's conversation content and emotional tendencies.

[1562] The present invention is a system that enables simulated conversations with family members using a user terminal, a server, and a generation AI to reduce the sense of isolation felt by elderly people. Specific embodiments for implementing this system are described below.

[1563] Uploading photos and videos

[1564] User:

[1565] Users select and upload photos and videos of their family using their smartphone, tablet, or other device. Specifically, users access the gallery through the app interface and select five photos and a 30-second video.

[1566] Device:

[1567] The user device temporarily stores the selected photos and videos in a buffer memory. Then, it sends this data to the server via an HTTP POST request. During this process, the user device displays a progress bar to indicate the progress, and once the transfer is complete, it displays the message "Upload complete" to the user.

[1568] Personality Assessment

[1569] server:

[1570] The server receives the photos and videos sent from the devices and stores them in cloud storage (e.g., AWS S3). It then analyzes the data using image recognition (e.g., OpenCV) and voice analysis (e.g., Google Cloud Speech-to-Text) technologies to extract the family's physical features and personality traits.

[1571] 3D avatar generation

[1572] server:

[1573] Based on the appearance and voice characteristics discovered from the analysis, the server applies a personality diagnosis algorithm (e.g., decision tree or neural network) to derive characteristics such as a "gentle and kind personality."

[1574] server:

[1575] Then, 3D avatars of the family are generated using 3D modeling software (e.g., Blender), recreating facial contours, hairstyles, and clothing styles, and adding facial expressions and movements based on their personalities. The resulting 3D avatars are then sent to the user's device in a compressed format, using the SSL / TLS protocol to ensure secure data transfer.

[1576] Device:

[1577] The user device decompresses the received 3D avatar data and displays it in the application's 3D view. The user can change the orientation of the avatar by touching or swiping the screen.

[1578] Natural language dialogue

[1579] User:

[1580] The user initiates a dialogue with the 3D avatar using voice or text input, for example, by asking, "How was your day today?"

[1581] Device:

[1582] The user device converts the voice input into text data via an API (e.g., Google Cloud Speech-to-Text). If the input is text, it is sent directly to the server.

[1583] server:

[1584] The server analyzes the received text data and generates an appropriate response using a generative AI model (e.g., GPT-3), such as "The weather was nice today, so I went for a walk."

[1585] Device:

[1586] The user device converts the response text received from the server into speech using a speech synthesis API (e.g., Amazon Polly) and synchronizes it with the mouth movements of the 3D avatar, allowing the user to enjoy natural conversations through the avatar.

[1587] Generate and send scheduled reports

[1588] server:

[1589] The server periodically analyzes the conversation between the user and the 3D avatar to extract trends in topics and emotions. Specifically, it uses an NLP engine to classify the conversation logs by topic and evaluates the user's emotional state using an emotion analysis algorithm.

[1590] server:

[1591] Based on the analysis results, regular reports are generated that include information such as the frequency of conversations, main topics, and emotional state, and are displayed in easy-to-read graphs and charts.

[1592] server:

[1593] The generated periodic reports are automatically sent to the family's email addresses. The server uses the SMTP protocol to realize the email sending function.

[1594] For example, some input prompts for a generative AI model might include:

[1595] "Generate a 3D avatar of a very kind man based on the following dialogue: 'Hi, Grandpa! What were you doing today?' 'Hi, Grandma! It's very cool today, so I was reading in the garden.'"

[1596] This system allows elderly people to enjoy natural conversation with someone who resembles a family member, reducing feelings of isolation. It also makes it easier for family members to keep track of the user's condition through regular reports.

[1597] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1598] Step 1: User selection of photos and videos

[1599] User:

[1600] The user launches the app on a smartphone, tablet, or other device, and operates the app's interface to select five family photos and a 30-second video from the gallery.

[1601] Input: Five photos of your family and a 30-second video.

[1602] Output: Selected photos and videos.

[1603] Step 2: Upload photos and videos

[1604] Device:

[1605] The user's device temporarily stores the selected photos and videos in a buffer memory, then sends the data to the server via an HTTP POST request, displaying a progress bar during the transfer and a message indicating the upload is complete when the transfer is complete.

[1606] Input: Photos and videos stored in buffer memory.

[1607] Output: A completion message sent to the server.

[1608] Step 3: Receiving and analyzing image and video data

[1609] server:

[1610] The server receives image and video data sent from the device and stores it in cloud storage (e.g., AWS S3). It then analyzes the data using image recognition (e.g., OpenCV) and voice analysis (e.g., Google Cloud Speech-to-Text) technologies to extract the family's physical features and personality traits.

[1611] Input: Received image and video data.

[1612] Output: Analysis of family members' physical features and personality traits.

[1613] Step 4: Applying the personality assessment algorithm

[1614] server:

[1615] The server applies a personality diagnosis algorithm (e.g., decision tree or neural network) based on the appearance and voice characteristics obtained from the analysis results to derive characteristics such as a "gentle and kind personality."

[1616] Input: Physical and vocal characteristics of family members.

[1617] Output: Personality test results.

[1618] Step 5: Generate a 3D avatar

[1619] server:

[1620] The server uses 3D modeling software (e.g., Blender) to generate 3D avatars of the family based on the personality test results and appearance data, and the generated avatars are sent to the user's device in a compressed format.

[1621] Input: Personality test results and physical characteristics data.

[1622] Output: Generated 3D avatar data.

[1623] Step 6: Displaying the 3D Avatar

[1624] Device:

[1625] The user device decompresses the received 3D avatar data and displays it in the application's 3D view. The user can change the orientation of the avatar by touching or swiping the screen.

[1626] Input: Received 3D avatar data.

[1627] Output: 3D avatar displayed on the application screen.

[1628] Step 7: Initiating a natural language dialogue

[1629] User:

[1630] The user initiates a dialogue with the 3D avatar through voice or text input, for example, by asking, "How was your day today?"

[1631] Input: Voice or text input.

[1632] Output: The input audio or text data.

[1633] Step 8: Process and transmit voice / text data

[1634] Device:

[1635] The user device converts the voice input into text data via an API (e.g., Google Cloud Speech-to-Text). If the input is text, it is sent directly to the server.

[1636] Input: Audio or text data.

[1637] Output: The text data sent to the server.

[1638] Step 9: Generate a response

[1639] server:

[1640] The server analyzes the received text data and generates an appropriate response using a generative AI model (e.g., GPT-3), such as "The weather was nice today, so I went for a walk."

[1641] Input: Text data.

[1642] Output: The generated response text.

[1643] Step 10: Returning a response and converting it

[1644] Device:

[1645] The user device converts the response text received from the server into speech using a speech synthesis API (e.g., Amazon Polly) and synchronizes it with the mouth movements of the 3D avatar, allowing the user to enjoy natural conversations through the avatar.

[1646] Input: The response text received from the server.

[1647] Output: Voice response by avatar.

[1648] Step 11: Record and analyze the conversation

[1649] server:

[1650] The server periodically analyzes the conversation between the user and the 3D avatar to extract trends in topics and emotions. Specifically, it uses an NLP engine to classify the conversation logs by topic and evaluates the user's emotional state using an emotion analysis algorithm.

[1651] Input: Log data of the conversation.

[1652] Output: Trend analysis results of topics and sentiment.

[1653] Step 12: Generate and send scheduled reports

[1654] server:

[1655] The server generates periodic reports based on the analysis results and automatically sends them to the family's email addresses, including information on the frequency of conversations, main topics, and emotional state.

[1656] Input: Topic and sentiment trend analysis results.

[1657] Output: Regular reports sent to families.

[1658] This allows the elderly to enjoy natural conversation with people who resemble family members, reducing feelings of loneliness.Families can also keep track of the elderly's condition through regular reports, even when they are far away.

[1659] (Application example 1)

[1660] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1661] There are existing systems that provide simulated dialogue with family members to reduce the sense of isolation felt by elderly people. However, these systems make it difficult for users to enjoy a sophisticated dialogue experience, such as receiving advice from family members in more specific life situations. Furthermore, there is a lack of systems that can generate specific dialogue responses tailored to specific situations. There is a growing demand for solutions to this problem.

[1662] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1663] In this invention, the server includes means for uploading image data and video data from a user terminal, means for analyzing the image data and video data on the server and diagnosing the user's personality, means for generating a 3D model based on the personality diagnosis results, means for transferring the 3D model to a head-mounted display terminal and displaying it, means for generating a natural language dialogue between the user and the 3D model via the head-mounted display terminal, and means for recording the content of the dialogue and sending regular reports to registered family members. This allows elderly people to experience more realistic dialogue with their families and receive appropriate advice in specific life situations.

[1664] A "user terminal" is an electronic device that allows a user to take photos and videos and upload them to a server.

[1665] "Image data and video data" refers to photographs and video files taken or selected by a user using a user terminal.

[1666] A "server" is a remote computer system that receives image and video data sent from a user terminal and analyzes and processes it.

[1667] "Personality diagnosis" is the process of identifying the subject's personality and characteristics by analyzing image and video data.

[1668] A "3D model" is a three-dimensional avatar generated on a computer based on the results of analysis, and is a digital object that reflects a person's appearance and personality.

[1669] A "head-mounted display terminal" is a display device worn on the head, and is a device that provides VR (virtual reality) and AR (augmented reality) experiences.

[1670] "Natural language interaction" is communication between humans and computers in which the user speaks in spoken language and the system responds.

[1671] A "regular report" is a document that the system records and analyzes the content of conversations and periodically reports on the user's status and conversation trends.

[1672] The present invention provides a virtual family shop system that allows users to get family advice in specific situations. Specific embodiments are given below.

[1673] Hardware and Software Configuration

[1674] User device: An electronic device such as a smartphone or tablet that a user uses to take photos or videos.

[1675] Server: A computer system that runs on the cloud and analyzes image and video data, generates 3D models, and performs natural language processing.

[1676] Software used: Google Cloud Speech-to-Text, Google Dialogflow, Blender, GPT-4

[1677] Head-mounted display device: A display device worn by the user to interact with 3D models. An example of this is the Oculus Quest 2.

[1678] Data processing and calculation procedures

[1679] 1. Uploading photos and videos

[1680] The user wears a head-mounted display device and takes photos and videos of their family. This data is temporarily stored on the user's device and then sent to a server via the Internet. The server analyzes the received data and performs a personality diagnosis.

[1681] 2. Personality Test

[1682] The software used on the server (Google Cloud Speech-to-Text and Google Dialogflow) analyzes the appearance and personality of family members from the received photos and videos. For example, facial features are recognized from still images, and voice tone and speaking style are analyzed from videos. This allows for the identification of personalities and characteristics.

[1683] 3. 3D Avatar Generation

[1684] The server uses Blender to generate 3D avatars based on image data and personality assessments. The generated 3D avatars reflect the family members' appearance information and personality assessment results.

[1685] 4. Natural Language Interaction

[1686] When a user wearing a head-mounted display device starts a conversation with an avatar, their voice is converted into text data by Google Cloud Speech-to-Text, and then Google Dialogflow analyzes the text and generates an appropriate response. The response is then converted into natural-sounding speech using GPT-4 and played back on the head-mounted display device.

[1687] Specific examples

[1688] When choosing clothes, the user wears a head-mounted display device and asks, "Mom, which outfit looks good on me?" The generated 3D avatar responds, "I think the red dress looks great on you." This allows the user to experience a conversation with a family member and receive specific advice.

[1689] Prompt Sentence Examples

[1690] The user types the following into the system:

[1691] "Simulate a situation where a 3D avatar of a family member is giving product advice to a user. Code it to respond to the question 'Which dress do you like?' with 'A red dress would look good on you.'"

[1692] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1693] Step 1:

[1694] The user wears a head-mounted display device and takes photos and videos of their family. The captured data is temporarily stored on the user's device and then sent to a server via the Internet. The input is the photo or video data, and the output is the data sent to the server.

[1695] Step 2:

[1696] The server analyzes the received image and video data and uses image recognition and voice analysis technology to extract the family's physical features and personality traits. The input is image and video data, and the output is appearance information and personality assessment results. Specifically, Google Cloud Speech-to-Text and Google Dialogflow are used to recognize facial features from photos and analyze voice tone and speaking style from videos.

[1697] Step 3:

[1698] The server uses Blender to generate a 3D avatar based on image data and personality assessment. The input is appearance information and personality assessment results, and the output is a 3D avatar. Specifically, Blender is used to model the avatar based on appearance information and assessment results.

[1699] Step 4:

[1700] The generated 3D avatar is transferred from the server to the head-mounted display device and displayed. The input is the 3D avatar, and the output is the avatar displayed on the user's HMD. Specifically, the data is sent to the HMD using a data transfer protocol.

[1701] Step 5:

[1702] The user speaks to an avatar through a head-mounted display terminal. The user's voice input is captured by the HMD's microphone and sent to the user terminal. The input is the user's voice data, and the output is the text conversion result of the voice data.

[1703] Step 6:

[1704] The server receives the voice data and converts it to text data using Google Cloud Speech-to-Text. The input is voice data and the output is text data. Specifically, it calls an API that converts the voice data to text data.

[1705] Step 7:

[1706] The server analyzes the text data and generates an appropriate response using Google Dialogflow and GPT-4. The input is text data, and the output is response text data. Specifically, it uses Dialogflow to analyze the intent of the question and GPT-4 to generate natural dialogue.

[1707] Step 8:

[1708] The response text data generated by the server is converted into voice data and sent to the head-mounted display terminal. The input is the response text data, and the output is voice data. Specifically, the voice data is generated using a text-to-speech API and sent to the HMD.

[1709] Step 9:

[1710] The head-mounted display terminal plays back audio data and responds to the user as a dialogue. The input is audio data, and the output is an audio response that the user can hear. Specifically, the audio is played back through the HMD's speakers.

[1711] Step 10:

[1712] The server records the dialogue between the user and the 3D avatar, periodically analyzes it, and generates a report. The input is the dialogue log, and the output is the analysis results and a report. Specifically, the dialogue log is stored in a database, and an algorithm is executed to periodically analyze the content and generate a report.

[1713] Step 11:

[1714] The generated regular report is sent to the registered family member's email address. The input is the report, and the output is an email sent to the family member's email address. The specific operation is to use the email sending API to send the regular report.

[1715] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1716] This invention is a system that uses a user terminal, a server, and a generation AI to enable simulated conversations with family members, and further combines an emotion engine to recognize and respond to emotions in order to reduce feelings of isolation among the elderly. Specific embodiments for implementing this system are described below.

[1717] Uploading photos and videos

[1718] User:

[1719] A user selects and uploads family photos and videos using a user device such as a smartphone or tablet. As a specific example, a user selects five family photos and a 30-second video from the smartphone gallery.

[1720] Device:

[1721] The user's device temporarily stores the selected photos and videos in a buffer and then transmits them to the server via the Internet. Once the photos and videos have been transmitted, the device displays a confirmation message to the user.

[1722] Personality diagnosis and emotion recognition

[1723] server:

[1724] The server analyzes the received photos and videos. Using image recognition and voice analysis technology, it analyzes the family members' appearance and voice to extract personality traits and emotions. For example, it recognizes facial features from still images and analyzes the tone of voice and speaking style from videos. This results in a "gentle and kind personality" and "current emotional state."

[1725] 3D avatar generation and emotional reflection

[1726] server:

[1727] The server uses a 3D modeling tool based on the analysis results to generate 3D avatars of the family. These avatars combine appearance information, personality assessment results, and emotional states, and are designed to resemble the appearance and behavior of the actual family. The generated avatars are then transferred to the user's device.

[1728] Device:

[1729] The user's device decompresses the 3D avatar received from the server and displays it on the application screen. The user can move the avatar and change its orientation by touching it. The avatar's facial expression and attitude can change depending on the user's emotional state.

[1730] Natural language dialogue

[1731] User:

[1732] The user initiates a dialogue with the 3D avatar through the application, for example by asking, "How was your day today?" This input is recorded as text and / or audio.

[1733] Device:

[1734] The user terminal converts the voice input into text data and sends it to the server, or if the input is text, sends it as is to the server.

[1735] server:

[1736] The system analyzes the received text data and uses a natural language processing (NLP) engine to understand its meaning. It then uses an emotion engine to recognize the user's emotions and generates an appropriate response. For example, it could respond with a happy expression on the avatar, saying, "The weather was nice today, so I went for a walk."

[1737] Device:

[1738] The user device converts the response received from the server into the voice and facial expression output of the 3D avatar, and returns it to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[1739] Generate and send scheduled reports

[1740] server:

[1741] The conversation history between the user and the 3D avatar is periodically collected and stored in a database. The emotional engine is also used to analyze the user's emotional tendencies during the conversation.

[1742] server:

[1743] Based on the collected data, it extracts trends in specific topics and emotions and generates regular reports. For example, it can report from a week's worth of conversation logs that "health" and "weather" were frequently discussed, and that users were primarily "calm."

[1744] server:

[1745] The generated reports are sent to the registered family member's email address on a regular basis, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[1746] This system allows elderly people living alone to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. Furthermore, the introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[1747] The processing flow will be explained below.

[1748] MODE FOR CARRYING OUT THE INVENTION

[1749] Uploading photos and videos

[1750] Step 1:

[1751] User:

[1752] Open the app on your smartphone or tablet and select photos and videos of your family: five photos and a 30-second video from your gallery.

[1753] Step 2:

[1754] Device:

[1755] The selected photos and videos are temporarily stored in a buffer and then sent to a server over the Internet. Once the photos and videos have been sent, the device will display a confirmation message to the user saying "Upload complete."

[1756] Personality diagnosis and emotion recognition

[1757] Step 3:

[1758] server:

[1759] Receives uploaded photos and videos, stores them in the server's storage, records the storage path, and generates metadata for further processing.

[1760] Step 4:

[1761] server:

[1762] The system analyzes stored photos and videos. It uses image recognition and audio analysis technology to analyze the appearance and voice of family members to extract personality traits and emotions. For example, it can extract characteristics such as a "calm and gentle personality" and "current emotional state" from the audio in a video.

[1763] 3D avatar generation and emotional reflection

[1764] Step 5:

[1765] server:

[1766] Based on the analysis results, a 3D modeling tool is used to generate 3D avatars of the family members, combining appearance information with personality assessment results and emotional state to create the optimal avatar.

[1767] Step 6:

[1768] server:

[1769] The generated 3D avatar is stored on the server and transferred to the user's device. The avatar data is compressed to enable efficient transfer.

[1770] Step 7:

[1771] Device:

[1772] The app receives the 3D avatar transferred from the server, decompresses it, and displays it in the app. The user can move the avatar and change its orientation using touch controls. The avatar also has the ability to change its facial expression and attitude depending on the user's emotional state.

[1773] Natural language dialogue

[1774] Step 8:

[1775] User:

[1776] The user initiates a conversation with the 3D avatar through the app. For example, the user might say, "How was your day?"

[1777] Step 9:

[1778] Device:

[1779] The user's voice input is converted into text data and sent to the server. In addition, if the input is text, it is sent to the server as is.

[1780] Step 10:

[1781] server:

[1782] The system analyzes the received text data and uses a natural language processing (NLP) engine and an emotion engine to understand the meaning and emotion. It then generates an appropriate response and sends that text data back to the device. For example, it might respond with "The weather was nice today, so I went for a walk," along with a happy expression on the avatar.

[1783] Step 11:

[1784] Device:

[1785] The response received from the server is converted into the voice and facial expression output of the 3D avatar, and returned to the user as a dialogue. The avatar responds with natural facial expressions and movements.

[1786] Generate and send scheduled reports

[1787] Step 12:

[1788] server:

[1789] The conversation history between the user and the 3D avatar is periodically collected and stored in a database. The emotional engine is also used to analyze the user's emotional tendencies during the conversation.

[1790] Step 13:

[1791] server:

[1792] Based on the collected data, it extracts trends in specific topics and emotions and generates regular reports. For example, it can report from a week's worth of conversation logs that "health" and "weather" were frequently discussed, and that users were primarily "calm."

[1793] Step 14:

[1794] server:

[1795] The generated reports are sent to the registered family member's email address on a regular basis, and include information such as the frequency of conversations, topic trends, and the user's emotional state.

[1796] This system allows elderly people living alone to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. Furthermore, the introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[1797] Example 2

[1798] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1799] There is a growing need for systems that can reduce the sense of isolation felt by the elderly and provide them with psychological care. However, existing systems lack natural dialogue and emotion recognition, making it difficult to provide appropriate responses and emotional care to users. Furthermore, it is difficult to communicate with family members and share the situation, and there is a lack of means to grasp the user's condition in a timely manner.

[1800] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1801] In this invention, the server includes means for uploading digital image data and video data from a user terminal, means for analyzing the digital image data and video data on the server and diagnosing the user's personality and emotions, means for generating a 3D character based on the personality and emotion diagnosis results, means for transferring the 3D character to the user terminal and displaying it, means for generating a natural language dialogue between the user and the 3D character via the user terminal, means for recording the dialogue and sending regular reports to registered family members, and means for generating responses according to the user's emotions using an emotion engine. This enables natural dialogue, allows appropriate care to be provided according to the user's emotional state, and enables family members to understand the user's condition in a timely manner.

[1802] A "user terminal" is an information processing device such as a smartphone, tablet, or personal computer operated by a user.

[1803] "Digital image data" refers to data of still images taken by a user, and is saved in a format such as JPEG or PNG.

[1804] "Video data" refers to video data shot by a user and saved in formats such as MP4 or AVI.

[1805] A "server" is a remote information processing device that provides functions such as data analysis, processing, and storage.

[1806] "Personality Assessment" is the process of identifying a user's personality traits through the analysis of image and audio data.

[1807] "Emotional diagnosis" is the process of identifying a user's current emotional state through the analysis of image and audio data.

[1808] A "3D character" is a three-dimensional virtual character generated based on the user's photo and video data.

[1809] "Natural language dialogue" refers to communication between a user and a system in natural language using voice or text.

[1810] "Regular reports" are reports generated based on the dialogue between the user and the 3D character and sent to family members.

[1811] An "emotion engine" is a software module that analyzes a user's emotions and generates an appropriate response.

[1812] A "natural language processing engine" is a software module that analyzes input text data and understands its meaning.

[1813] This invention is a system that uses a user terminal, a server, and a generation AI to enable simulated conversations with family members, and further combines an emotion engine to recognize and respond to emotions in order to reduce feelings of isolation among the elderly. Specific forms for realizing this system are shown below.

[1814] System Configuration

[1815] User terminal

[1816] The user terminals are smartphones, tablets, or personal computers. These devices provide the ability to take and store photos and videos of family members, and can transmit the data to a server via the Internet. They also provide an interface for displaying the generated 3D characters and interacting with them through applications.

[1817] server

[1818] The server receives and analyzes photo and video data uploaded from user devices and processes the data using generative AI models and emotion engines. The server has the following main functions:

[1819] 1. Data reception and storage function

[1820] 2. Image recognition and speech analysis functions (e.g., OpenCV, Google Cloud Speech-to-Text)

[1821] 3. Personality diagnosis and emotion recognition (e.g., NLP engines)

[1822] 4. 3D character generation function (e.g. Blender)

[1823] 5. Natural language processing engines (e.g., GPT-3, BERT)

[1824] 6. Emotion Engine

[1825] Explanation of specific processing contents

[1826] 1. Upload your data

[1827] A user selects and uploads family photos and videos using a user terminal such as a smartphone or tablet. For example, a user selects five family photos and a 30-second video from the smartphone gallery.

[1828] 2. Data Analysis

[1829] The server receives and analyzes photo and video data sent from the user's device. Image recognition is performed using OpenCV, and voice data is converted to text using the Google Cloud Speech-to-Text API for analysis. Personality diagnosis and emotion recognition are performed using an NLP engine, which extracts personality and emotional tendencies.

[1830] 3. 3D character generation

[1831] The server uses 3D modeling tools such as Blender to generate a 3D character based on the analysis results. This character reflects the appearance information obtained from the still images and the personality and emotional state extracted from the video. The generated character is then transferred to the user's device.

[1832] 4. Natural Language Interaction

[1833] The user initiates a natural language dialogue with the 3D character through the application. For example, they might say, "How was your day today?" This input is recorded as voice or text and sent from the user's device to the server. The server uses an NLP engine to analyze the input data and an emotion engine to generate an appropriate response. The generated response is converted into voice or facial expression and expressed by the 3D character on the user's device.

[1834] 5. Generate and send scheduled reports

[1835] The server periodically collects the user's interaction history with the 3D character and stores it in a database. It uses an emotion engine to analyze the user's emotional tendencies during the interaction, extracting specific topics and emotional patterns to generate periodic reports. These reports are sent to registered family members' email addresses, reporting the user's recent status and emotional tendencies.

[1836] Examples of concrete examples and prompts

[1837] Examples:

[1838] When a user asks, "How was your day today?", the server analyzes the question and generates a response such as, "The weather was nice today, so I went for a walk." The 3D character then displays a happy expression.

[1839] Example prompt sentence:

[1840] "The user asks, 'How was your day?' Generate a response that reflects a happy mood."

[1841] This system allows elderly people to enjoy natural conversation with someone similar to a family member, reducing feelings of loneliness and providing emotional care. Regular reports are also sent to families, allowing them to understand the user's condition with peace of mind. The introduction of an emotion engine makes it possible to respond according to the user's emotional state, enabling more intimate and meaningful conversations.

[1842] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1843] Step 1:

[1844] A user uses a smartphone or tablet to select and upload digital image data (photos) and video data (movies) of their family members. Specifically, the user opens the device's gallery app, selects five photos and a 30-second video, and taps the "Upload" button. This action saves the digital image data and video data in the device's buffer. The input is the selected image data and video data, and the output is the data saved in the buffer.

[1845] Step 2:

[1846] The device sends the image and video data stored in the buffer to the server. The device encrypts the data and uploads it to the server via the Internet. When the transmission is complete, the device displays a confirmation message to the user saying "Upload complete." The input is the data stored in the buffer, and the output is the data sent to the server.

[1847] Step 3:

[1848] The server receives image and video data from the user's device. The server checks the format of the data and prepares it for image recognition and voice analysis. The input is the data sent to the server, and the output is data in a format that can be analyzed.

[1849] Step 4:

[1850] The server uses OpenCV to analyze image data and extract the facial features of family members. It also uses the Google Cloud Speech-to-Text API to convert the video's audio data into text. An NLP engine is used to analyze speaking style and tone of voice from the text data to extract the family members' personalities and current emotional state. The input is data in an analyzable format, and the output is facial feature information, textual audio data, and personality and emotional diagnosis results.

[1851] Step 5:

[1852] The server uses a 3D modeling tool such as Blender to generate 3D characters of the family members based on the analysis results obtained in the previous step. These characters reflect facial feature information, personality diagnosis results, and emotional state. The input is facial feature information, text-converted voice data, and personality and emotional diagnosis results, and the output is the generated 3D characters.

[1853] Step 6:

[1854] The server transfers the data of the generated 3D character to the user terminal. The input is the data of the generated 3D character, and the output is the 3D character data sent to the user terminal.

[1855] Step 7:

[1856] The device decompresses the 3D character data received from the server and displays it on the application screen. The user can control the character by touching it. The avatar can change its facial expression and attitude depending on the user's emotional state. The input is the 3D character data sent from the server, and the output is the 3D character displayed in the application.

[1857] Step 8:

[1858] The user initiates a dialogue with the 3D character through the application, for example by asking, "How was your day today?" This input is recorded as voice or text. The input is what the user said, and the output is voice or text data.

[1859] Step 9:

[1860] The terminal converts the voice input into text data and sends it to the server. If the input is text, it sends it to the server as is. The input is voice or text data, and the output is the text data sent to the server.

[1861] Step 10:

[1862] The server analyzes the received text data and uses an NLP engine to understand the meaning of the input. It then uses an emotion engine to recognize the user's emotions and generates an appropriate response. For example, the response generated might be, "The weather was nice today, so I went for a walk." The input is text data, and the output is the generated response.

[1863] Step 11:

[1864] The device converts the response received from the server into voice and facial expressions, and returns it to the user in the form of a 3D character response. The avatar responds with natural facial expressions and movements. The input is the generated response, and the output is the response reflected in voice and facial expressions.

[1865] Step 12:

[1866] The server periodically collects the interaction history between the user and the 3D character and stores it in a database. The input is the interaction history, and the output is the stored data.

[1867] Step 13:

[1868] The server uses an emotion engine to analyze the user's emotional tendencies during the conversation. This analysis extracts specific topics and emotional patterns. The input is the saved conversation history data, and the output is the extracted emotional tendencies and topic patterns.

[1869] Step 14:

[1870] The server generates periodic reports based on the analysis results. For example, it reports that "health" and "weather" were frequently discussed based on a week's worth of conversation logs, and that the user was primarily "calm." The input is the analysis results, and the output is the generated periodic reports.

[1871] Step 15:

[1872] The server sends the generated periodic report to the registered family member's email address. The report includes information such as the frequency of conversations, topic trends, and the user's emotional state. The input is the generated periodic report, and the output is the sent report.

[1873] (Application example 2)

[1874] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1875] In addition to systems that provide psychological care through simulated conversations with family members for elderly people and users who feel isolated, there is a need for effective support in providing product explanations and customer service to customers in physical stores. Furthermore, it is necessary to improve the level of service in stores by using a system that uses avatars that can accurately convey product features during conversations with customers.

[1876] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data and video data from a user terminal, means for analyzing the image data and video data and diagnosing the user's personality, means for generating a 3D model based on the personality diagnosis results, means for uploading product photos and videos in the store and analyzing the product features, means for generating a 3D product avatar based on the features and displaying it on the user terminal, means for generating a natural language dialogue between the user and the 3D product avatar via the user terminal, and means for recording the content of the dialogue and generating a service report for the user. This enables psychological care for elderly users and users who feel isolated, and effective customer service support for customers in physical stores.

[1877] A "user terminal" is a device operated by a user, such as a computer, smartphone, or tablet.

[1878] "Image data" refers to digital image files such as photographs uploaded by users.

[1879] "Video data" refers to digital video files such as video clips uploaded by users.

[1880] A "server" is a computer system for receiving and processing data sent from a user terminal.

[1881] "Personality diagnosis" is a method of analyzing the personality traits and emotional state of a user or subject.

[1882] A "3D model" is a three-dimensional virtual object generated using computer graphics.

[1883] "Natural language dialogue" refers to communication using language that humans use on a daily basis.

[1884] "Recording the content of the dialogue" means saving the content of the dialogue between the user and the 3D model in a database or the like.

[1885] A "regular report" is a report that summarizes the content of conversations and the status of users over a certain period of time.

[1886] "Product features" refers to information about the physical properties and functions of a product.

[1887] A "3D product avatar" is a three-dimensional character that virtually recreates a product.

[1888] A "service report" is a report that summarizes the content of conversations with users and business negotiation history.

[1889] A "natural language processing engine" is software that analyzes input in natural language and generates appropriate responses.

[1890] An "emotion recognition engine" is software that analyzes and recognizes the emotional state of a user or target.

[1891] To implement this invention, the following system is used: The main components of the system are a user terminal, a server, a generative AI model, and an emotion recognition engine.

[1892] Users upload image and video data using user devices such as smartphones and tablets. This data is sent to a server via the Internet. The server analyzes the received image and video data to extract characteristics of the user and the product. For example, it analyzes the user's personality and the product's external characteristics. It also uses an emotion recognition engine to diagnose the user's emotions.

[1893] The server then uses a 3D modeling tool based on the analysis results to generate 3D models of users and products. For example, it generates avatars for "gentle and kind personalities" or "healthy products" based on facial features obtained from still images and voice tones obtained from videos. These 3D models are then transferred from the server to the user's device and displayed on the application screen. Users can then have natural conversations with the avatars.

[1894] User interaction input is recorded as voice or text and sent from the user's device to the server. The server then uses a natural language processing engine to analyze the input and generate an appropriate response. For example, if the user asks, "How was your day today?", the server might respond with, "The weather was nice today, so I went for a walk," and display a happy expression on the avatar.

[1895] Furthermore, the server records the content of conversations and generates periodic reports based on them. These reports are sent to registered family members and users and include information such as the frequency of conversations, topic trends, and emotional state. For example, a week's worth of conversation logs may be analyzed to determine that "health" and "weather" were frequently discussed, and the server may report to family members that the user was primarily "calm."

[1896] To illustrate, here is a prompt statement that generates the following program:

[1897] "Generate a Python program for a mobile application that uses uploaded product photos and videos to create a 3D avatar representing the products in a store. The avatar should be able to interact with users through natural language conversation. The program should use TensorFlow for emotion recognition, BERT for natural language processing, and SpeechRecognition for converting audio input to text."

[1898] This system not only enables psychological care for elderly people and users who feel isolated, but also enables effective customer service support for physical stores.

[1899] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1900] Step 1:

[1901] A user selects and uploads image data and video data using a user device such as a smartphone or tablet. The user device temporarily stores the selected data in buffer memory and sends it to the server. The input is image data and video data, and the output is the data sent to the server. Specifically, the user selects five product photos and a 30-second promotional video, and clicks the upload button.

[1902] Step 2:

[1903] The server receives image and video data sent from the user's device and prepares it for analysis. The server analyzes the data using image recognition and audio analysis technology to extract product features. The input is data from the user's device, and the output is the analysis results (product features). Specifically, TensorFlow is used to extract product features such as color, shape, and logo from images, and to analyze audio tone from videos.

[1904] Step 3:

[1905] The server uses a 3D modeling tool based on the analysis results to generate a 3D avatar of the product. This avatar combines information about the product's appearance and features. The input is the analysis results, and the output is the generated 3D avatar. Specifically, the operation uses a modeling tool such as Blender to generate a 3D model that faithfully reproduces the product's appearance.

[1906] Step 4:

[1907] The server transfers the generated 3D avatar to the user's device. The user's device receives the avatar and displays it on the application screen. The input is the 3D avatar, and the output is the avatar displayed on the user's device. Specifically, the user opens the app on their smartphone, and the 3D avatar of the product appears on the screen.

[1908] Step 5:

[1909] The user initiates a natural language dialogue with a 3D avatar via the user terminal. The user's voice input is converted into text on the user terminal and sent to the server. The input is the user's voice, and the output is text data. Specifically, the action is to ask, "What are the features of this product?"

[1910] Step 6:

[1911] The server analyzes the received text data and understands its meaning using a natural language processing (NLP) engine. It also uses an emotion recognition engine to recognize the user's emotions and generate an appropriate response. The input is text data, and the output is a response text. Specifically, it uses BERT to analyze the user's question and generates the response, "This product uses organic ingredients."

[1912] Step 7:

[1913] The server sends the generated response to the user's device and converts it into the voice and facial expression output of the 3D avatar. The user's device receives the response, and the avatar responds with natural facial expressions and movements. The input is the response text, and the output is voice and facial expressions. Specifically, the user listens to the avatar's explanation on their smartphone, and the avatar explains with a smile.

[1914] Step 8:

[1915] The server records the content of the conversation between the user and the 3D avatar and periodically analyzes it. Specific topics and emotional trends are extracted from the conversation and compiled into a service report. The input is the conversation content, and the output is the analysis results and report. Specifically, the server analyzes the conversation log for a week and finds that there were many questions about "health," and compiles this into a report.

[1916] Step 9:

[1917] The server periodically sends service reports to registered family members and users. These reports include information such as the frequency of interactions, topic trends, and the user's emotional state. The input is the service report, and the output is email or notification. Specifically, this operation sends a "weekly interaction history report" to the family member's email address.

[1918] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1919] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1920] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1921] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1922] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1923] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1924] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1925] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1926] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1927] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1928] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1929] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1930] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1931] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1932] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1933] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1934] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs...

Claims

1. means for uploading image data and video data from a user terminal; A means for analyzing the image data and video data on a server and diagnosing the personality of the user; A means for generating a 3D model based on the personality diagnosis result; means for transferring and displaying the 3D model on a user terminal; means for generating a dialogue in natural language between a user and a 3D model via the user terminal; The system includes a means for recording the content of the above conversations and sending periodic reports to registered family members.

2. The system of claim 1 , wherein the 3D model is generated using a combination of appearance information and personality assessment results obtained from image data and video data.

3. 10. The system of claim 1, wherein the server includes a natural language processing engine for analyzing natural language data received from the user terminal and generating appropriate responses in real time.

Images