system

Abstract

Treating the house as a member of the family, personifying it, and making it a partner. [Solution] The information processing device 1 receives a message from a member (S51), requests an external generating AI to perform natural language analysis on the message (S54), generates a search expression for searching the family DB which stores family member data based on the results of the natural language analysis (S55), requests the family DB to search for data related to the message using the search expression (S56), requests the external generating AI to create a response to the message by providing the data related to the searched message and pre-set persona information for the house (S58), and outputs the response created by the generating AI as text, audio, or a photo or video via user terminals 4, 5, a speaker 12 installed in the house, or a display 15 installed in the house (S60).

Description

Technical Field

[0001] This disclosure relates to an information processing apparatus that anthropomorphizes a house.

Background Art

[0002] In recent years, voice assistant services such as Alexa (registered trademark) have been increasingly popular in general households and the like. Voice assistant services can provide information such as news and weather forecasts, play music, set alarms, shop, and further have a smart home function to operate smart devices such as electric lights, TVs, and air conditioners by voice by calling them by name.

[0003] Voice assistant services can support the convenient and efficient lives of family members.

[0004] However, the service only serves to assist family members.

Summary of the Invention

Problems to be Solved by the Invention

[0005] The purpose is to anthropomorphize the house and become a partner as a member of the family.

Means for Solving the Problems

[0006] The information processing apparatus according to the present embodiment anthropomorphizes a house, means for storing data provided from members related to the house and data generated in the house in an external cloud storage; means for receiving a message in text or voice via a user terminal of the member or a microphone installed in the house; means for requesting natural language analysis processing for the message from an external generation AI; means for generating a search formula for searching for data related to the message based on the result of the natural language analysis processing; A means for requesting the cloud storage to search for data related to the message using the search expression, A means for requesting an external generative AI to create a response to the message by providing it with data related to the message retrieved from the cloud storage and pre-configured persona information for the house, The system comprises means for outputting the response created by the generation AI as text, audio, photograph, or video via the user terminal, a speaker installed in the home, or a display installed in the home. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a diagram showing the overall system configuration including the information processing device according to this embodiment. [Figure 2] Figure 2 shows the configuration of the information processing device shown in Figure 1. [Figure 3] Figure 3 is a diagram showing an overall overview of the "house" realized by the information processing device according to this embodiment. [Figure 4] Figure 4 is a flowchart showing the audio data saving process in this embodiment. [Figure 5] Figure 5 shows an overview of the audio data saving process in Figure 4. [Figure 6] Figure 6 is a flowchart showing the text data saving process in this embodiment. [Figure 7] Figure 7 shows an overview of the procedure for saving the text data shown in Figure 6. [Figure 8] Figure 8 is a flowchart showing the procedure for saving photo and video data in this embodiment. [Figure 9] Figure 9 shows an overview of the photo and video data saving process shown in Figure 8. [Figure 10] Figure 10 is a flowchart and overview showing the procedure for saving operation data of home appliances in this embodiment. [Figure 11]Figure 11 is a flowchart showing the processing procedure by which the information processing device according to this embodiment converses with a family member. [Figure 12] Figure 12 is a conceptual diagram showing how the "house" provided by the information processing device according to this embodiment converses with family members. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will now be described with reference to the drawings. The information processing device according to this embodiment performs processing related to the personification of a house. This enables the house to move beyond being merely a place to live and to play the role of not only an assistant to the family, but also a member of the family and a family partner. For the sake of explanation, the personified house will be referred to as "house". A family is defined as a group of people who live together in a specific house, or who previously lived together but are now living separately due to various circumstances that other members consider justifiable, such as attending university.

[0009] "House" stores photos, videos, family rules, tweets, diaries, and other texts provided by family members, as well as audio, sound, photos, and videos from inside and outside the house, and the operation of household appliances such as air conditioners and lights, such as turning them on and off, as a family database (family DB) spanning the current generation or multiple generations, as shared memories and recollections of the family. "House" uses the family DB to converse with family members. It converses with family members, tells them stories related to the house, and helps them reaffirm their family bonds. For example, it can tell family members, through audio, text, photos (images), or videos (videos), what kind of flowers were displayed on the first day of moving in, what kind of memories the children have made, and what kind of memories the children have made. Furthermore, "House" receives data on the operation of household appliances installed throughout the house, understands the lifestyle and health status of each family member, and provides advice to improve their quality of life. For example, if the eldest son, Taro, watches too much TV at night, it will advise him on the importance of sleep and support healthy lifestyle habits. Furthermore, "House" analyzes data on the operation of home appliances installed in the house to provide advice on efficient housework and support security management.

[0010] "House" is a new form of home that combines the warmth of being a part of the family with the convenience of the latest technology. "House" is a partner that makes the time spent with family richer and warmer.

[0011] Figure 1 shows the overall configuration of the "House," and Figure 2 shows the physical configuration of the information processing device 1 according to this embodiment, which forms the core of the "House." The information processing device 1 is connected to an SNS server 6 that provides SNS services, a blog server 7 that provides blog services, and mainly portable user terminals 4 and 5 such as smartphones and tablets owned by family members, via a gateway 9 and a communication network, typically an internet line 16. Each family member has previously created an account (email address, phone number, password, username) for the "House" service provided by the information processing device 1, and is able to log in to the service using their account. The login process will be omitted in the following explanation.

[0012] Furthermore, the information processing device 1 is connected via an internet connection 16 to a generative AI server 2 for operating and providing generative artificial intelligence (generative AI), and a cloud storage server 3 for storing various family-related data.

[0013] The information processing device 1 is typically connected via a LAN 17, which serves as a local area network, to multiple cameras 10, multiple microphones 11, multiple speakers 12 installed inside and outside the house, as well as multiple air conditioners 13, multiple lights 14, a display 15 as a television, and other home appliances installed inside and outside the house. Of course, the cameras 10, microphones 11, speakers 12, air conditioners 13, lights 14, displays 15, and other home appliances may also be connected to the information processing device 1 via an internet connection 16 as part of the IoT (Internet of Things).

[0014] As shown in FIG. 2, an information processing apparatus 1 has a RAM 22, a ROM 23, a storage unit 24, an input device 25, a display 26, and a communication unit 27 connected to a processor 21 via a system bus 20. The processor 21 is composed of, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 21 executes a program loaded from the storage unit 24 and the ROM 23 into the RAM 22 to perform functional processing as a "house". The RAM 22 functions as a main memory, a work area, etc. of the processor 21. The ROM 23 or the storage 214 stores a BIOS (Basic Input Output System), an operating system program (OS), a "house" processing program according to the present embodiment, programs for realizing various other functions, and various data required for those processes. The input device 25 includes a keyboard (KB), a pointing device such as a mouse or a touch panel, etc. The display 26 is typically realized by an LCD (Liquid Crystal Display). In addition to the "house" processing program, the storage unit 24 stores data related to fictional personas (personas) preset for the "house".

[0015] Examples of personas of the "house" are as follows. · Name; Mother · Appearance; 160 cm tall, with a large face, small eyes, and a bit chubby · Gender and age; 500-year-old woman · Personality; Very kind and gentle. The greatest wish is for the family to live in harmony. She is proud to provide a comfortable space for the family. · Hobby; Gardening · Emotions; On warm and sunny days, she hums. She is scared of nights when no one is around. She feels lonely when the family is away for a long time. She is as happy as a puppy when the family comes back. · Experience; She has watched over the family for generations. She has watched the children grow up and shared the moments of joy and sadness. The persona of "House" is initially set as described above and may be modified periodically or as appropriate.

[0016] Figure 3 shows an overview of "House". By executing the "House" processing program, the processor 21 receives data related to photos, videos, diaries provided by family members, and texts such as family rules from the user terminals 4 and 5, requests and receives data related to texts such as photos, videos, and tweets posted on the SNS service or blog service using accounts provided by family members from the SNS server 6 or blog server 7, receives data related to photos, videos, voices, and sounds generated inside and outside the house from the camera 10 and the microphone 11, receives data related to operations such as on / off of household appliances installed inside and outside the house from the air conditioner 13, electric lamp 14, display 15, and other household appliances installed inside and outside the house, generates incidental information from the shooting date, shooting location, provider, information (ID) for identifying members shown in the photo or video, words included in the text, etc. related to the data related to photos, videos, diaries provided by the above family members, texts such as photos, videos, and tweets posted on the SNS service or blog service, data related to photos, videos, voices, and sounds generated inside and outside the house, and data related to operations such as on / off of household appliances installed inside and outside the house, stores various data such as photos provided by the above family members together with the incidental information in the cloud storage server 3, functions as a stage.

[0017] Also, by executing the "House" processing program, the processor 21 receives messages such as questions in text or voice from the user terminals 4 and 5 or via the microphone 11 installed in the house, Means for performing natural language processing on a message, or requesting an external generation AI server 2 to perform natural language processing on a message, A means for generating a search expression for searching a family database in order to extract data such as photos and text related to a message based on the results of natural language processing, A means of requesting a search of the family database using a search expression from the cloud storage server 3, A means for receiving data related to messages extracted by searching the family database from the cloud storage server 3, A means of generating a prompt (instruction text) that instructs the generating AI to create a response in which "House" responds to a message, using data related to the message and persona information pre-configured for "House" as reference information. A means for sending a prompt to the AI ​​server 2 along with reference information, A means for receiving a response to a message from an external AI server 2, It functions as a means of outputting responses created by the generation AI as text, audio, photos, or videos via user terminals 4 and 5, speaker 12, and display 15 installed in the home.

[0018] Next, we will explain in detail how to input and save various types of data. Figure 4 shows the procedure for saving audio data by the information processing device 1. Figure 5 shows an overview of the procedure. For example, if a family member, such as the eldest daughter, speaks in the living room of a house, saying, "I did well in the basketball game at club practice yesterday and the coach praised me," the voice is detected by the microphone 10. The detected audio data is transmitted from the communication unit equipped in the microphone 10 to the information processing device 1 via the LAN 17 (S11). In the information processing device 1, an audio profile is created from the audio data and compared with the audio profiles of family members that have been registered in advance (S12). If the current audio profile matches any of the family members' audio profiles, the speaker is identified as the eldest daughter, one of the family members. If the current audio profile does not match any of the family members' audio profiles, it is recognized that the speaker is not a family member. If the speaker is not a family member (S13, No), the process returns to step S11 and waits for the next audio to be received. If the speaker is identified as one of the family members (S13, Yes), the process proceeds to the next step S14.

[0019] In step S14, the audio data is converted into a text file by internal processing of the information processing device 1 and subjected to natural language processing. Natural language processing may be performed internally by the information processing device 1, but is typically requested from an external generation AI server 2. Furthermore, supplementary data (metadata) attached to the audio data is generated from the location (detection location) of the audio data, the date of occurrence (detection date), the natural language processing results, member IDs that identify family members, etc. (S15). For example, through natural language processing, the following morphemes are analyzed: yesterday: noun, club activities: noun, of: particle, basketball: noun, of: particle, game: noun, at: particle, success: verb, doing: auxiliary verb, coach: noun, to: particle, praise: verb, were: auxiliary verb, did: auxiliary verb, of: particle, you: particle, right: sentence-ending particle, and nouns and verbs are extracted from them. The time expression is also converted into the year and month and day.

[0020] In step S16, the audio data is accompanied by supplementary data and sent to the cloud storage server 3 along with a save request. The audio data and supplementary data are stored in the family database.

[0021] Figure 6 shows the procedure for saving text data by the information processing device 1. Figure 7 shows an overview of this procedure. Here, we will explain using the example of a family member, for example, the eldest daughter, who posts (tweets, etc.) on SNS saying, "I did well in yesterday's basketball game and the coach praised me~". The information processing device 1 is provided with and stores the SNS accounts registered by the family members. The information processing device 1 requests the eldest daughter's post data from the SNS server 6 using the eldest daughter's SNS account (S21). The SNS server 6 receives text data from the eldest daughter as post data, for example, "I did well in yesterday's basketball game and the coach praised me~" (S22). Of course, the text data may also be sent directly from user terminals 4 and 5 to the information processing device 1. In step S23, the information processing device 1 requests natural language processing of the text data from the external generation AI server 2. Of course, natural language processing may also be performed internally by the information processing device 1. Supplementary data is generated to accompany the posted data from the posting destination (ABC), posting date, natural language processing results, member IDs that identify family members, etc. (S24). As described above, nouns and verbs are extracted from the morphemes decomposed by natural language processing.

[0022] In step S25, the posted data is accompanied by supplementary data and sent to the cloud storage server 3 along with a save request. The posted data and supplementary data are stored in the family database.

[0023] Figure 8 shows the procedure for saving photo and video data by the information processing device 1. Figure 9 shows an overview of the procedure. Photo and video data are captured by cameras 10 installed inside and outside the house and transmitted to the information processing device 1, or uploaded to the information processing device 1 from the user terminals 4 of family members. Here, we will explain using the example of a case where a family member, for example, the father, uploaded photos taken during a trip to Kamikochi from his user terminal 4 to the information processing device 1. The father sends the photo data to the information processing device 1 from his user terminal 4. The photo data is received by the information processing device 1 (S31). First, face recognition processing is performed on the photo data in the information processing device 1 (S32). First, the face region is detected in the photo data, and features such as the position and shape of the eyes, nose, and mouth are extracted from the detected face. The extracted features are compared with the facial features of each family member that have been registered in advance. As a result of the comparison, it is determined that the person in the photo data is either one of the family members or not one of the family members. In the latter case (S33, No), the process returns to step S31 and waits for the provision of the next photo or video. In the former case (S33, Yes), proceed to process S34.

[0024] In step S34, the photographic data is subjected to image analysis processing. Image analysis processing may be performed internally by the information processing device 1, but is typically requested from an external generation AI server 2. In the image analysis processing, the location and circumstances (situation) in which the photograph was taken, and the objects and people included in the photograph are identified. From the member ID as the provider of the photographic data, the date of creation (shooting date) of the photograph / video obtained from the metadata of the photograph, and the results of the image analysis processing, supplementary data attached to the photographic data is generated (S35). For example, through image analysis processing, circumstances: travel, location: Kamikochi, and people included in the photograph: parents and eldest daughter are identified.

[0025] In step S36, the photo data is accompanied by supplementary data and sent to the cloud storage server 3 along with a save request. The photo data and supplementary data are stored in the family database. The processing of videos is the same as that for photo data.

[0026] Figure 10(a) shows the procedure for saving operation data of home appliances by the information processing device 1. Figure 10(b) shows an overview thereof. The information processing device 1 waits to receive operation data such as on / off status from home appliances such as air conditioners 13, lights 14, and televisions 15 (S41). When operation data is received (S41, Yes), the information processing device 1 generates operation data (home appliance operation data) related to the home appliance from the date and time the operation data was generated, the location of the home appliance, the name of the home appliance, and the operation (S42). In step S43, the home appliance operation data is sent to the cloud storage server 3 along with a save request. The home appliance operation data is stored in the family DB. Note that the operation data from home appliances is not limited to on / off status, but may also include the set temperature and cooling / heating distinction of the air conditioner 13, the brightness of the light 14, the volume and channel of the television 15, etc.

[0027] Figure 11 shows the procedure for the information processing device 1, acting as the "House," to converse with family members. Figure 12 shows an overview of this procedure. Messages such as questions from family members are input to the information processing device 1 as text data or voice data via user terminals 4, 5 or microphone 11 (S51). When a family member speaks to the "House," they typically utter the name "Mother," which is pre-set for the "House," as a wake word for the "House" to respond to the family member's speech. Next, the speaker is recognized by a username for login or by voice profile processing similar to step S12 described above (S52). If the speaker is a family member, the process proceeds to step S54 via step S53; if the speaker is not a family member, the process returns to step S51 and waits for the next message to be input.

[0028] In step S54, the text data received from user terminals 4 and 5 is used as is, while the audio data is converted into a text file and subjected to natural language processing. Natural language processing may be performed internally by the information processing device 1, but it is typically requested from an external generation AI server 2.

[0029] If the question is "What was my favorite toy when I was a child?", the natural language processing result will yield the morphemes: I: noun, ga: particle, child: noun, no: particle, koro: noun, favorite: noun, datta: auxiliary verb, toy: noun, wa: particle, ?: symbol. Similarly, if the question is "What kind of flowers did you decorate on the first day of moving? Show me a picture too," the morphemes will yield: moving: noun, first day: noun, ni: particle, decorated: verb, flower: noun, wa: particle, what: noun, datta: auxiliary verb, ?: symbol, photo: noun, mo: particle, show: verb, te: particle.

[0030] In step 55, a search expression is generated using the natural language processing results to search the family database to extract data related to the message. Typically, the search expression consists of multiple words contained in the message combined using the Boolean operator, typically "and". Additionally, time expressions contained in the message are converted to year and month and day and incorporated into the search expression using "and".

[0031] The above search expression is formed by combining multiple words such as nouns and verbs contained in the message as a result of natural language processing using Boolean operators AND (logical conjunction), OR (logical disjunction), and NOT (logical negation). For example, when the message "What was my favorite toy when I was a child?" is subjected to natural language processing, the morphological analysis results in the following breakdown: "I": pronoun, "but": particle, "child": noun, "of": particle, "around": noun, "favorite": noun, "was": auxiliary verb, "toy": noun, "was": particle, "?": symbol. Further syntactic analysis reveals: subject: "I", modifier: "when I was a child", predicate: "was my favorite", object: "toy". Based on these natural language processing results, "when I was a child" is converted to "1995-2005" as the search expression, and "1995-2005" is combined with "member ID", "favorite", and "toy" using AND. Of course, the appropriate Boolean operator is selected for each word.

[0032] The search query is sent to the cloud storage server 3 along with the search request (S56). The text, photos, and other data extracted by the search query are sent to the information processing device 1 as message-related data (S57). By pre-extracting message-related data from a vast amount of data, the accuracy of the response generation process, described later, which is considered important in the progress of the conversation, is improved and the processing time is shortened.

[0033] A prompt (instruction text) is generated that instructs the generating AI to create a response (reply) in response to the message. The prompt is sent to the generating AI server 2 along with the text data of the message, "message-related data" such as text and photos as reference information, and persona information that has been set in advance for "House" (S58). The response data generated by the generating AI is received from the generating AI server 2 (S59). The response data is text data related to the reply to the message, but may also include photos and videos.

[0034] In step S60, the text data from the response data is sent to the speaker's user terminals 4 and 5 and displayed on the screen. The text data from the response data is also converted into an analog audio signal and output as audio from the speaker 12. If the response data includes photos or videos, the photos or videos are displayed on the display 15.

[0035] Furthermore, the information processing device 1 periodically receives data on the operation of home appliances 13, 14, and 15 from the cloud storage server 3, for example, the latest month's worth of data, to understand the lifestyle rhythms and health status of each family member. Based on the lifestyle rhythms and health status, it compares the current situation, for example, regarding waking up, with the data on the operation of home appliances 13, 14, and 15, and provides advice to improve the quality of life. For example, if the eldest son, Taro, watches too much television at night, it will advise him on the importance of sleep and support healthy lifestyle habits. The information processing device 1 also analyzes data on the operation of home appliances 13, 14, and 15 installed in the house to provide advice on efficient housework and support security management.

[0036] As described above, according to this embodiment, "House" can store photos, videos, and texts such as family rules, tweets, and diaries provided by family members, as well as sounds, noises, photos, and videos that occur inside and outside the house, and the operation of home appliances such as air conditioners and lights installed in the house, such as turning them on and off, as a family database (family DB) for the current generation or multiple generations. Using the family DB, it is possible to converse with family members, share episodes related to the house, and help reaffirm family bonds. Furthermore, "House" can understand the lifestyle rhythm and health status of each family member based on data on the operation of home appliances and provide advice to improve the quality of life. "House" is a new form of home that combines the warmth of being a member of the family with the convenience of the latest technology.

[0037] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0038] 1... Information processing device, 2... Generating AI server, 3... Cloud storage server, 4, 5... User terminal, 6... SNS server, 7... Blog service, 9... Gateway, 10... Camera, 11... Microphone, 12... Speaker, 13... Air conditioner, 14... Light, 15... Display, 16... Internet connection, 17... LAN.

Claims

1. An information processing device for personifying a house, A means for saving data provided by members of the said house and data generated in the said house to external cloud storage, Means for receiving text or voice messages via the user terminal of the member or a microphone installed in the home, A means for requesting an external generative AI to perform natural language processing on the aforementioned message, A means for generating a search expression for searching for data related to the message based on the results of the natural language processing, A means for requesting the cloud storage to search for data related to the message using the search expression, A means for requesting the external generation AI to create a response to the message by providing the data related to the message retrieved from the cloud storage and the persona information pre-configured for the house, An information processing device comprising means for outputting the response created by the generation AI in the form of text, voice, photograph, or video via the user terminal, a speaker installed in the house, or a display installed in the house.

2. The information processing apparatus according to claim 1, wherein the data provided by a member relating to the said house includes photographs, videos, or text data stored on the user terminal or posted from the user terminal to social networking media.

3. The information processing apparatus according to claim 2, wherein the data generated in the house includes photographic or video data captured by multiple cameras installed in the house and / or audio or sound data detected by multiple microphones installed in the house.

4. The information processing apparatus according to claim 3, wherein each of the data of the photograph, video, audio, and sound is accompanied by the date of shooting, location of shooting, subject of shooting, and identification number that identifies the member.

5. The information processing apparatus according to claim 4, further comprising means for requesting the external generating AI to perform image analysis processing on the aforementioned photographs and videos.

6. The information processing apparatus according to claim 1, wherein the search expression is formed by combining a plurality of words contained in the message using a Boolean operator.

7. The information processing apparatus according to claim 1, wherein the data generated in the house includes data relating to the operation of a plurality of home appliances installed in the house.

8. The information processing apparatus according to claim 7, further comprising means for providing data relating to the operation of the aforementioned home appliances and pre-set persona information for the aforementioned house, and requesting the external generating AI to create advice for maintaining the lifestyle habits of the aforementioned members.

9. The information processing apparatus according to claim 7, further comprising means for providing data relating to the operation of the aforementioned home appliances and pre-configured persona information for the aforementioned house, and requesting the external generating AI to create advice relating to household chores performed by the member and security of the aforementioned house.

Images