system

Abstract

To personify the home as a part of the family and make it a partner. [Solution] The information processing device 1 receives a message from a member (S51), requests an external generation AI to perform natural language analysis processing on the message (S54), generates a search formula for searching a family DB that stores family member data based on the results of the natural language analysis processing (S55), requests the family DB to search for data related to the message using the search formula (S56), provides the data related to the searched message and persona information previously set for the house, and requests the external generation AI to create a response to the message (S58), and outputs the response created by the generation AI as text, audio, or a photo or video via a user terminal 4, 5, a speaker 12 installed in the house, or a display 15 installed in the house (S60).

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device that personifies a house. [Background technology]

[0002] In recent years, voice assistant services such as Alexa (registered trademark) have become increasingly popular in ordinary homes, etc. Voice assistant services provide information such as news and weather forecasts, play music, set alarms, go shopping, and even have smart home functions that allow you to control smart devices such as lights, televisions, and air conditioners by voice by calling out to them.

[0003] Voice assistant services can help family members live convenient and efficient lives.

[0004] However, the service only serves to assist family members. Summary of the Invention [Problem to be solved by the invention]

[0005] The goal is to personify the home as a part of the family and make it a partner. [Means for solving the problem]

[0006] The system according to this embodiment comprises a plurality of cameras, a plurality of microphones, and a plurality of speakers installed in a home, and an information processing device. The information processing device has a memory unit for storing a program, a processor for executing the program, and a communication unit that is connected to the cameras, the microphones, and the speakers via a LAN, and is connected to an SNS server that provides an SNS service, a blog server that provides a blog service, a generation AI server, and a cloud storage server via an Internet line. The processor executes the program, a means for requesting the SNS server and the blog server to provide and receiving posted data posted by the family members living in the house to the SNS service and the blog service using accounts related to the family members; means for receiving photo data, video data, audio data, and sound data generated in the house from the camera and the microphone; a means for storing the posted data, the photo data, the video data, the audio data, and the acoustic data in the cloud storage server; The member can make a voice call via the microphone. question and means for receiving the The aforementioned The questions are converted into text data, A means for transmitting a request for natural language analysis processing to the generation AI server; Based on the result of the natural language analysis processing, The posting data, the photo data, the video data, the audio data, and the audio data stored in the cloud storage server are extracted from the posting data, the photo data, the video data, the audio data, and the audio data related to the question. a means for generating a search formula for The aforementioned question Related to the posting data, the photo data, the video data, the voice data, and the acoustic data means for transmitting a request to provide the search query to the cloud storage server together with the search query; The information provided by the cloud storage server question Related to the posting data, the photo data, the video data, the voice data, and the acoustic data and the above question and persona information preset for the house, Answering questions means for transmitting an instruction to the generation AI server to create the The AI ​​server generates the reply as a sound from the speaker. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a configuration diagram of an entire system including an information processing device according to this embodiment. [Figure 2] FIG. 2 is a diagram illustrating the configuration of the information processing apparatus of FIG. [Figure 3] FIG. 3 is a diagram showing an overall outline of a "house" realized by the information processing device according to this embodiment. [Figure 4]FIG. 4 is a flowchart showing the procedure for saving audio data in this embodiment. [Figure 5] FIG. 5 is a diagram showing an outline of the procedure for saving the audio data of FIG. [Figure 6] FIG. 6 is a flowchart showing the procedure for saving text data in this embodiment. [Figure 7] FIG. 7 is a diagram showing an outline of the procedure for saving the text data of FIG. [Figure 8] FIG. 8 is a flowchart showing the steps of the process for saving photo and video data in this embodiment. [Figure 9] FIG. 9 is a diagram showing an outline of the steps of the process of saving the photo and video data shown in FIG. [Figure 10] FIG. 10 is a flowchart showing the procedure for saving operation data of a home electric appliance in this embodiment, and a diagram showing an outline of the procedure. [Figure 11] FIG. 11 is a flowchart showing the processing procedure for an information processing device according to this embodiment to converse with family members. [Figure 12] FIG. 12 is a conceptual diagram showing a "house" conversing with family members using the information processing device according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. An information processing device according to this embodiment executes processing related to the personification of a house. This enables a house to function not only as a mere place to live but also as an assistant to the family, and to fulfill the role of a member of the family and a partner to the family. For convenience of explanation, a personified house will be referred to as a "house." Note that a family refers to a group of people related by blood or marriage who live together in a specific house, or who previously lived together but are now living apart for reasons that the other members deem legitimate, such as going on to college.

[0009] "House" stores photos, videos, family rules, tweets, diary entries, and other text provided by family members, as well as audio, sound, photos, and videos generated inside and outside the home, and the on / off actions of home appliances such as air conditioners and lights, in a family database (Family DB) for the current generation or multiple generations. It then uses the Family DB to communicate with family members. "House" converses with family members, sharing stories about the home and helping to reaffirm family bonds. For example, it can share with family members what flowers were used to decorate the home on the first day of the move, or the memories the children have made, using audio, text, photos, or videos. Furthermore, "House" receives data on the operation of home appliances installed throughout the house, understanding each family member's daily rhythm and health status and providing advice to improve their quality of life. For example, if the eldest son, Taro, watches too much TV late at night, it can advise him on the importance of getting enough sleep and support healthy lifestyle habits. House also analyzes data on the operation of home appliances installed in the home and provides advice on efficient housework and security management.

[0010] "HOUSE" is a new type of home that combines the warmth of being a part of the family with the convenience of the latest technology. "HOUSE" is a partner that makes the time you spend with your family richer and warmer.

[0011] FIG. 1 shows the overall configuration of the "house," and FIG. 2 shows the physical configuration of an information processing device 1 according to this embodiment, which is the core of the "house." Information processing device 1 is connected to an SNS server 6 that provides SNS services, a blog server 7 that provides blog services, and mainly portable user terminals 4 and 5, such as smartphones and tablets, owned by family members, via a gateway 9 and a communication network, typically an Internet line 16. Each family member creates an account (email address, phone number, password, username) for the "house" service provided by information processing device 1 in advance, and is able to log in to the service using the account. The login process will be omitted in the following explanation.

[0012] In addition, the information processing device 1 is connected via an internet line 16 to a generation AI server 2 for operating and providing generation-type artificial intelligence (generation AI), and a cloud storage server 3 for storing various data related to family members.

[0013] A plurality of cameras 10, a plurality of microphones 11, a plurality of speakers 12 installed inside and outside the house, as well as a plurality of air conditioners 13, a plurality of lights 14, a television display 15, and other home appliances installed inside and outside the house are connected to the information processing device 1 via an in-house communication network, typically a LAN 17. Of course, the cameras 10, microphones 11, speakers 12, air conditioners 13, lights 14, display 15, and other home appliances may be connected to the information processing device 1 via an internet line 16 as IoT (Internet of Things).

[0014] As shown in FIG. 2 , the information processing device 1 includes a processor 21 connected to a RAM 22, a ROM 23, a storage unit 24, an input device 25, a display 26, and a communication unit 27 via a system bus 20. The processor 21 is configured, for example, with a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 21 executes programs loaded from the storage unit 24 and ROM 23 to the RAM 22 to perform functional processing of the "house." The RAM 22 functions as the processor 21's main memory, work area, etc. The ROM 23 or the storage 214 stores a BIOS (Basic Input Output System), an operating system program (OS), a "house" processing program according to this embodiment, other programs for implementing various functions, and various data required for these processes. The input device 25 includes a keyboard (KB), a pointing device such as a mouse or a touch panel, etc. The display 26 is typically implemented as an LCD (Liquid Crystal Display). In addition to the "house" processing program, the storage unit 24 also stores data relating to fictional personas that are preset for the "house."

[0015] Examples of "house" personas are: Name; Mother Appearance: 160cm tall, large face, big eyes, slightly chubby Gender and age: 500-year-old woman Personality: He has a very kind and gentle personality. His greatest wish is for his family to live in harmony. He takes pride in providing a comfortable space for his family. Hobbies: Gardening Emotions: I find myself humming on calm, warm days. I'm afraid of nights when no one is around. I feel lonely when my family is away for a long time. I get as excited as a puppy when my family returns. Experience: She has watched over generations of families. She has watched her children grow up and shared their moments of joy and sorrow. The persona of the "house" may be initially set as described above and revised periodically or as needed.

[0016] Figure 3 shows an overview of the "house." The processor 21 executes the "house" processing program. A means for receiving data relating to photos, videos, and text such as diaries and family rules provided by family members from user terminals 4 and 5; A means for requesting and receiving data relating to photos, videos, and text such as tweets posted to an SNS service or a blog service using an account provided by a family member from an SNS server (6) or a blog server (7); A means for receiving data relating to photos, videos, voices and sounds generated inside and outside the house from a camera 10 and a microphone 11; A means for receiving data relating to the operation of home appliances installed inside and outside the house, such as on / off, from an air conditioner 13, a light 14, a display 15, and other home appliances installed inside and outside the house; A means for generating supplementary information from data relating to photographs, videos, and text such as diaries and family rules provided by the family members, data relating to photographs, videos, and text such as tweets posted on SNS services or blog services, data relating to photographs, videos, voice and sound generated inside and outside the home, and data relating to the on / off and other operations of home appliances installed inside and outside the home, including information (ID) identifying the members appearing in the photographs and videos, the date of shooting, the location of shooting, the provider, and words contained in the text; a means for storing various data such as photographs provided by the family members together with accompanying information in a cloud storage server; It functions as a step.

[0017] The processor 21 also executes the "house" processing program. means for receiving messages such as text or voice questions from user terminals 4, 5 or via a microphone 11 installed in the home; means for executing natural language analysis processing on the message or requesting an external generation AI server 2 to perform natural language analysis processing on the message; A means for generating a search expression for searching the family database to extract data such as photos and text related to the message based on the result of the natural language analysis processing; A means for requesting a cloud storage server 3 to search the family DB using a search expression; means for receiving data related to the message extracted by searching the family DB from the cloud storage server; A means for generating a prompt (instruction sentence) that instructs the generation AI to create a response for the "house" to respond to the message, using data related to the message and persona information previously set for the "house" as reference information; means for transmitting the prompt together with the reference information to the generation AI server 2; A means for receiving a response in response to a message from an external generation AI server 2; It functions as a means for outputting the response created by the generation AI as text, audio, or photos or videos via the user terminals 4, 5, the speaker 12, and the display 15 installed in the home.

[0018] Next, inputting and saving various data will be specifically described. FIG. 4 shows the procedure for saving voice data by the information processing device 1. FIG. 5 shows an overview of the procedure. When a family member, for example, the eldest daughter, utters, for example, "I played well in the basketball game yesterday and got praised by the coach," in the living room of the house, the voice is detected by the microphone 10. The detected voice data is transmitted from a communication unit equipped in the microphone 10 to the information processing device 1 via the LAN 17 (S11). The information processing device 1 creates a voice profile from the voice data and compares it with the voice profiles of pre-registered family members (S12). If the current voice profile matches any of the voice profiles of the family members, the speaker is identified as the eldest daughter, one of the family members. If the current voice profile does not match any of the voice profiles of the family members, the speaker is recognized as not being a family member. If the speaker is not a family member (S13, No), the process returns to step S11 and waits for the next voice to be received. If the speaker is identified as a family member (S13, Yes), the process proceeds to step S14.

[0019] In step S14, the voice data is converted into a text file by internal processing of the information processing device 1 and subjected to natural language processing. Natural language processing may be performed internally by the information processing device 1, but is typically requested from an external generation AI server 2. Furthermore, additional data (metadata) to be attached to the voice data is generated from the location (detection location) of the voice data, the date (detection date), the natural language processing results, member IDs identifying family members, and the like (S15). For example, the following morphemes are analyzed by natural language processing: yesterday (noun), club activities (noun), particle of no, basketball (noun), particle of no, game (game), noun, particle of de, active (verb), auxiliary verb of shite (shite), coach (noun), particle of ni, praise (verb), auxiliary verb of rare (rare), auxiliary verb of ta (ta), particle of no, particle of yo (yo), particle of ne (ne), and final particle of ne (ne), and nouns and verbs are extracted from them. Furthermore, time expressions are converted into the Gregorian calendar and the date.

[0020] In step S16, the voice data is accompanied by additional data and sent together with a storage request to the cloud storage server 3. The voice data and additional data are stored in the family DB.

[0021] FIG. 6 shows the procedure for saving text data by the information processing device 1. FIG. 7 shows an overview of the procedure. Here, we will explain an example in which, for example, a family member, the eldest daughter, posts (e.g., tweets) to an SNS, saying, "I did well in the basketball game yesterday, and the coach praised me." The information processing device 1 is provided with and stores SNS accounts registered by the family members. The information processing device 1 requests the eldest daughter's posting data from the SNS server 6 using the eldest daughter's SNS account (S21). For example, text data, such as "I did well in the basketball game yesterday, and the coach praised me," is received from the SNS server 6 as the eldest daughter's posting data (S22). Of course, the text data may be transmitted directly from the user terminals 4 and 5 to the information processing device 1. In step S23, the information processing device 1 requests the external generation AI server 2 to perform natural language processing on the text data. Of course, the natural language processing may be an internal process of the information processing device 1. The data to be attached to the posted data is generated from the posting destination (ABC), posting date, natural language processing results, member IDs identifying family members, etc. (S24) As described above, nouns and verbs are extracted from the morphemes decomposed by natural language processing.

[0022] In step S25, the posted data is accompanied by additional data and sent together with a storage request to the cloud storage server 3. The posted data and additional data are stored in the family DB.

[0023] FIG. 8 shows the procedure for storing photo and video data by the information processing device 1. FIG. 9 shows an overview of the procedure. Photo and video data are captured by a camera 10 installed inside or outside the home and transmitted to the information processing device 1, or uploaded to the information processing device 1 from a user terminal 4 of a family member. Here, we will explain an example in which a family member, for example, a father, uploads photos of his trip to Kamikochi from his user terminal 4 to the information processing device 1. The father transmits photo data from his user terminal 4 to the information processing device 1. The photo data is received by the information processing device 1 (S31). In the information processing device 1, face recognition processing is first performed on the photo data (S32). First, a facial region is detected in the photo data, and facial features such as the position and shape of the eyes, nose, and mouth are extracted from the detected face. The extracted features are compared with the facial features of each family member registered in advance. As a result of the comparison, it is determined that the person appearing in the photo data is either a family member or not a family member. In the latter case (S33, No), the process returns to step S31 and waits for the next photo or video to be provided. In the former case (S33, Yes), the process proceeds to step S34.

[0024] In step S34, the photo data is subjected to image analysis processing. The image analysis processing may be an internal process of the information processing device 1, but is typically requested from the external generation AI server 2. The image analysis processing identifies the location and situation in which the photo was taken, as well as the objects and people included in the photo. From the member ID of the provider of the photo data, the date the photo or video was taken (shooting date) obtained from the photo metadata, and the results of the image analysis processing, additional data to be attached to the photo data is generated (S35). For example, the image analysis processing identifies the situation: travel, the location: Kamikochi, and the people included in the photo: parents and the eldest daughter.

[0025] In step S36, the photo data is accompanied by additional data and sent together with a storage request to the cloud storage server 3. The photo data and additional data are stored in the family DB. Note that the same processing is performed for video data as for photo data.

[0026] FIG. 10(a) shows a procedure for storing operation data of home appliances by the information processing device 1. FIG. 10(b) shows an overview thereof. The information processing device 1 waits to receive operation data, such as on / off, from home appliances such as the air conditioner 13, the light 14, and the television 15 (S41). When operation data is received (S41, Yes), the information processing device 1 generates operation data related to the home appliances (home appliance operation data) from the date and time of generation of the operation data, the installation location of the home appliance, the name of the home appliance, and the operation (S42). In step S43, the home appliance operation data is sent to the cloud storage server 3 together with a storage request. The home appliance operation data is stored in the family DB. Note that the operation data from home appliances is not limited to on / off, and may also include the set temperature and cooling / heating status of the air conditioner 13, the brightness of the light 14, the volume and channel of the television 15, etc.

[0027] FIG. 11 shows a procedure in which the information processing device 1 as the "House" converses with family members. FIG. 12 shows an overview of the procedure. A message, such as a question, from a family member is input to the information processing device 1 as text data or voice data via the user terminal 4, 5 or the microphone 11 (S51). When a family member speaks to the "House," the "House" typically utters a name, "Mother," pre-set for the "House" as a wake word for responding to the family member's speech. Next, the speaker is recognized (S52) by a login username or voice profile processing similar to that of step S12. If the speaker is a family member via step S53, the process proceeds to the next step S54. If the speaker is not a family member, the process returns to step S51 and waits for the next message to be input.

[0028] In step S54, the text data received from the user terminals 4 and 5 is directly processed, and the voice data is converted into a text file and subjected to natural language processing. The natural language processing may be an internal process of the information processing device 1, but is typically requested from the external generation AI server 2.

[0029] For the question "What was your favorite toy when I was a child?", the natural language processing results in the following morphemes: "I" (noun), "ga" (particle), "kodomo" (child) (noun), "no" (particle), "koro" (around) (noun), "favourite" (noun), "datta" (auxiliary verb), "toy" (noun), "wa" (particle), and "?" (symbol). Similarly, for the question "What flowers did you decorate on the first day of your move? Can you also show me a photo?", the following morphemes are obtained: "moving" (noun), "first day" (noun), "ni" (particle), "decorated" (verb), "hana" (noun), "wa" (particle), "nani" (noun), "datta" (auxiliary verb), "?" (symbol), "photo" (noun), "mo" (particle), "mise" (verb), and "te" (particle).

[0030] In step 55, the results of the natural language processing are used to generate a search expression for searching the family database to extract data related to the message. The search expression typically consists of multiple words contained in the message combined with a Boolean operator, typically "and." Time expressions contained in the message are converted to years and months and days and incorporated into the search expression with "and."

[0031] The search formula is formed by combining multiple words, such as nouns and verbs, contained in the message resulting from natural language analysis processing using Boolean operators AND (logical conjunction), OR (logical disjunction), and NOT (logical negation). For example, when the message "What was your favorite toy when you were a child?" is subjected to natural language processing, the morphological analysis results are broken down into "I" (pronoun), "ga" (particle), "child" (noun), "no" (particle), "koro" (around) (noun), "favorite" (noun), "watashi" (was) (auxiliary verb), "toy" (noun), "wa" (particle), and "?" (symbol). Further syntactic analysis results in the subject: "I," modifier: "when I was a child," predicate: "was my favorite," and object: "toy." Based on these results of natural language analysis processing, the search formula converts "when I was a child" to "1995-2005," and then combines "1995-2005" with "member ID," "favorite," and "toy" using AND. Of course, Boolean operators are selected appropriately for each word.

[0032] The search formula is sent to the cloud storage server 3 together with the search request (S56). Data such as text and photos extracted by the search formula are sent to the information processing device 1 as data related to the message (S57). By extracting data related to the message in advance from a huge amount of data, it is possible to improve the accuracy of the response generation process (described below) and reduce the processing time, which are important in the progress of the conversation.

[0033] A prompt (instruction sentence) is generated to instruct the generation AI to create a response (reply) from the "house" to the message. The prompt is sent to the generation AI server 2 together with the text data of the message, "data related to the message" such as text and photos as reference information, and persona information previously set for the "house" (S58). The response data generated by the generation AI is received from the generation AI server 2 (S59). The response data is text data related to a reply to the message, and may also include photos and videos.

[0034] In step S60, the text data of the response data is transmitted to the speaker's user terminal 4, 5 and displayed on the screen. The text data of the response data is also converted into an analog voice signal and output as voice from the speaker 12. When the response data includes a photo or video, the photo or video is displayed on the display 15.

[0035] The information processing device 1 periodically receives data on the operation of the home appliances 13, 14, and 15 from the cloud storage server 3, for example, for the most recent month, and grasps the lifestyle rhythms and health status of each family member. The data on the operation of the home appliances 13, 14, and 15 is compared with the current status of waking up, for example, in relation to the lifestyle rhythms and health status, and advice is provided to improve the quality of life. For example, if the eldest son, Taro, watches too much TV late at night, advice on the importance of sleep is provided to support healthy lifestyle habits. The information processing device 1 also analyzes data on the operation of the home appliances 13, 14, and 15 installed in the home, and provides advice on efficient housework and security management.

[0036] As described above, according to this embodiment, the "House" can store photos, videos, and text such as family rules, tweets, and diaries provided by family members, as well as voice, sound, photos, and videos generated inside and outside the home, and the on / off operations of home appliances such as air conditioners and lights installed in the home, as shared family memories and recollections for the current generation or multiple generations in a family database (Family DB). Using the Family DB, the "House" can converse with family members, share stories related to the home, and help reaffirm family ties. Furthermore, the "House" can understand the daily rhythms and health status of each family member based on data related to the operation of home appliances and provide advice to improve quality of life. The "House" is a new type of home that combines the warmth of being a part of the family with the convenience of the latest technology.

[0037] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0038] 1...information processing device, 2...generative AI server, 3...cloud storage server, 4, 5...user terminal, 6...SNS server, 7...blog service, 9...gateway, 10...camera, 11...microphone, 12...speaker, 13...air conditioner, 14...light, 15...display, 16...internet line, 17...LAN.

Claims

1. The system comprises a plurality of cameras, a plurality of microphones, and a plurality of speakers installed in a house, and an information processing device; The information processing device has a storage unit that stores a program, a processor that executes the program, and a communication unit that is connected to the camera, the microphone, and the speaker via a LAN, and is connected to an SNS server that provides an SNS service, a blog server that provides a blog service, a generation AI server, and a cloud storage server via an Internet line, The processor executes the program, a means for requesting the SNS server and the blog server to provide and receiving posted data posted by the family members residing in the house to the SNS service and the blog service using accounts related to the family members; means for receiving photo data, video data, audio data, and sound data generated in the house from the camera and the microphone; a means for storing the posted data, the photo data, the video data, the audio data, and the acoustic data in the cloud storage server; means for receiving a voice question from the member via the microphone; means for converting the question into text data and transmitting a request for natural language analysis processing to the generation AI server; a means for generating a search expression for extracting the posted data, the photo data, the video data, the audio data, and the audio data related to the question from the posted data, the photo data, the video data, the audio data, and the audio data stored in the cloud storage server based on the result of the natural language analysis processing; means for transmitting a request to provide the posted data, the photo data, the video data, the voice data, and the acoustic data related to the question together with the search query to the cloud storage server; means for transmitting to the generation AI server an instruction to create a response to the question, together with the posted data, the photo data, the video data, the voice data, and the acoustic data related to the question provided by the cloud storage server, the question, and persona information preset for the house; A system that functions as a means for outputting the reply created by the generation AI server as voice from the speaker.

2. The system according to claim 1 , wherein the photo data, the video data, the audio data, and the sound data are accompanied by the date and location at which they were detected.

3. The processor further functions as a means for transmitting, to the generating AI server, a request to identify the members appearing in the photo data and the video data, together with the photo data and the video data; The system according to claim 1 , wherein the photo data and the video data are stored in the cloud storage server together with information identifying the members shown in the respective images.

4. The system according to claim 1 , wherein the search formula is formed by combining a plurality of words included in the text data of the question with Boolean operators.

5. the processor further functions as a means for receiving data relating to the operation of a plurality of home appliances installed in the home from the home appliances; The system of claim 1 , wherein data relating to the operation of the home appliance is stored on the cloud storage server.

6. The system of claim 5 , wherein the processor further functions as a means for sending a request to the generating AI server to create advice for maintaining the member's lifestyle habits, together with data regarding the operation of the home appliance and the persona information.

7. The system of claim 5 , wherein the processor further functions as a means for sending to the generating AI server a request for the creation of advice regarding household chores to be performed by the member and security in the home, together with data regarding the operation of the home appliance and the persona information.

Images