Information processing device and information processing method

Abstract

To provide an information processing device and an information processing method capable of promoting conversation among care-receivers.SOLUTION: An information processing device includes a storage part for storing a plurality of images at respective different periods including a caregiver taking care of a care-receiver, a period information acquisition part for acquiring period information showing past periods received by a first terminal of the care-receiver from the first terminal, and a service provision part for displaying an image of the caregiver corresponding to the periods shown by the period information on the screen of the first terminal to promote conversation between the care-receiver and the caregiver.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing method.

Background Art

[0002] A technique for monitoring in real time the movements of a person under care, such as a care recipient, by means of a humanoid avatar three-dimensional animation image and voice has been proposed in Patent Document 1.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, when the care recipient being assisted has a conversation, if the care recipient cannot recognize who the conversation partner is, it becomes difficult for a conversation to be established between the care recipient and the conversation partner.

[0005] According to some aspects of the present disclosure, there are provided an information processing apparatus and an information processing method capable of promoting the conversation of a care recipient.

Means for Solving the Problems

[0006] An information processing apparatus according to an aspect of the present disclosure includes a storage unit that stores a plurality of images at different times including a caregiver who assists a care recipient, a time information acquisition unit that acquires from the first terminal time information indicating a past time received by the first terminal of the care recipient, and a service providing unit that causes an image of the caregiver corresponding to the time indicated by the time information to be displayed on the screen of the first terminal and provides a service for promoting a conversation between the care recipient and the caregiver.

[0007] An information processing method according to another aspect of the present disclosure is an information processing method having a storage unit that stores a plurality of images at different times, including an assistant who assists a person with disabilities. The method acquires time information indicating a past time received by a first terminal of the person with disabilities from the first terminal, and causes an image of the assistant corresponding to the time indicated by the time information to be displayed on a screen of the first terminal, thereby providing a service that promotes conversation between the person with disabilities and the assistant.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14A

Figure 14B

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 20

Modes for Carrying Out the Invention

[0009] <First Embodiment> Hereinafter, the first embodiment will be described. FIG. 1 is a diagram showing a configuration example of the system 100 of the first embodiment. The system 100 is used, for example, as a virtual communication system (virtual system).

[0010] The system 100 of the first embodiment is used when the assisted person and the assistant have a conversation while looking at each other's images. Hereinafter, it will be described on the assumption that the assisted person is an elderly person and the assistant is a family member of the assisted person (for example, an adult child). However, the assisted person does not have to be an elderly person. Also, the assistant does not have to be a family member of the elderly person.

[0011] In the real world, in order for the assisted person and the assistant to have a smooth conversation, both parties having the conversation need to recognize who the other party of the conversation is. In this regard, when the cognitive ability of the assisted person declines, the assisted person may not be able to recognize who the other party of the conversation is. In this case, it becomes difficult to establish a smooth conversation between the assisted person and the assistant.

[0012] Here, as one of the cognitive decline phenomena, there is a phenomenon called retrograde amnesia of memory. Retrograde amnesia of memory means that the memory of the care recipient is lost retrogressively from the present to a certain extent into the past. For example, when the person the care recipient is talking to has a face of a caregiver relatively close to the present, the care recipient cannot recognize the person. On the other hand, if the face of the person is that of a past age (e.g., a student or a minor) beyond a certain extent, the care recipient may be able to recognize the face of the person.

[0013] Therefore, the system 100 of the first embodiment causes a plurality of temporally sequential past images of the person (caregiver) with whom the care recipient is having a conversation to be displayed on the screen of the care recipient's terminal. The terminal sequentially displays each image and also displays a message indicating whether the person in the displayed image can be recognized. When the care recipient performs an operation of designating an image among the displayed images, the system 100 generates an avatar image using the designated image. The system 100 controls so that the conversation between the care recipient and the caregiver proceeds smoothly while displaying the avatar image (the avatar image of the caregiver recognized by the care recipient) on the care recipient's terminal. This will be described in detail below.

[0014] The system 100 includes a care recipient terminal 110, a caregiver terminal 120, and a server 130. The care recipient terminal 110 is the first terminal held by the care recipient 110H. The care recipient terminal 110 includes a touch panel 110A. The caregiver terminal 120 is the second terminal held by the caregiver 120H. The caregiver terminal 120 includes a touch panel 120A.

[0015] The server 130 is a server that performs the control of each embodiment. The care recipient terminal 110 and the caregiver terminal 120 are communicably connected to the server 130. The server 130 performs the control of each embodiment. The server 130 manages users for each account. The user is, for example, the caregiver 120H. In this embodiment, the same account is assigned to the caregiver 120H and the care recipient 110H.

[0016] Server 130 requests the assistant terminal 120 to upload images of a plurality of photos of the assistant 120H at different times from the present to the past. The assistant terminal 120 stores the images of the plurality of photos. Date information is associated with each of the plurality of photos. Details will be described later.

[0017] In response to the above request, the assistant terminal 120 uploads images (photo data) of a plurality of photos of the assistant 120H to the server 130. The server 130 stores the uploaded images of the plurality of photos. Hereinafter, the images of the photos are simply referred to as images.

[0018] Server 130 sequentially displays the above plurality of images on the touch panel 110A of the assisted terminal 110. If the images include people other than the assistant 120, the server 130 sequentially displays on the touch panel 110A of the assisted terminal 110 an image obtained by extracting the portion of the assistant 120 from the above plurality of images. At the same time, the server 130 transmits to the assisted terminal 110 question data for asking whether each of the images can be recognized as to who it is. The assisted terminal 110 displays the question data on the touch panel 110A. Note that the server 130 may display on the touch panel 110A of the assisted terminal 110 one image of the assistant 120H at one past time point.

[0019] Based on the above question data, the assisted person 110H accepts, as answer data, the designation of the images that can be recognized among the respective images. The assisted terminal 110 transmits the answer data to the server 130.

[0020] Server 130 generates an avatar image of the assistant 120H using the image of the photo indicated by the answer data. The generated avatar image is an avatar image based on the image at the time when the assisted person 110H could recognize the other party, the assistant 120H. Server 130 performs conversation control using the generated avatar image. At this time, the server 130 displays the generated avatar image on the touch panel 110A of the assisted terminal 110.

[0021] For example, based on the speech data spoken by the caregiver 120H using the caregiver terminal 120, the server 130 controls the operation of the avatar image displayed on the touch panel 110A of the care recipient terminal 110 so that the avatar image appears to be speaking.

[0022] The server 130 transmits an image of the care recipient 110H being photographed to the caregiver terminal 120. The caregiver terminal 120 displays the image of the care recipient 110H on the touch panel 120A.

[0023] The server 130 obtains the speech content spoken by the caregiver 120H from the caregiver terminal 120 and transmits the obtained speech content to the care recipient terminal 110. The server 130 obtains the speech content spoken by the care recipient 110H from the care recipient terminal 110 and transmits the obtained speech content to the caregiver terminal 120. Thereby, a voice conversation is carried out between the care recipient 110H and the caregiver 120H. The voice conversation may be carried out using a separate communication line in addition to the Internet line.

[0024] As described above, the server 130 causes the avatar image at the time when the care recipient 110H can recognize among a plurality of images of the caregiver 120H in time series to be displayed on the touch panel 110A of the care recipient terminal 110, and controls the conversation between the care recipient 110H and the caregiver 120H. Thereby, since the care recipient 110H can recognize who the conversation partner is, a smooth conversation between the care recipient 110H and the caregiver 120H can be promoted.

[0025] Hereinafter, the details of each device and each process will be described. FIG. 2 is a diagram showing a configuration example of the server 130 according to the first embodiment. The server 130 includes a control unit 210, a storage unit 211, and a communication unit 212. The control unit 210 performs various controls of the server 130. The control unit 210 has, for example, a processor and a memory. By the processor executing the program stored in the memory, various processes executed by the server 130 are realized.

[0026] The processor is a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), etc. The memory is a semiconductor memory such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. Also, the processing unit 220 may be an FPGA (field-programmable gate array), etc. The above program may be stored in a non-transitory recording medium and provided to the control unit 210.

[0027] The storage unit 211 stores various data. The storage unit 211 stores a plurality of images of the caregiver 120H described above and various tables described later. In the present embodiment, the storage unit 211 stores a plurality of images of the care recipient 120H from the present to the past for each account of the caregiver 120H. The caregiver 120H uses the caregiver terminal 120 to perform an operation of uploading a plurality of images of the care recipient 120H from the present to the past to the server 130 in response to a request for uploading an image described later. As a result, the plurality of images of the care recipient 120H from the present to the past are stored in the storage unit 211. The communication unit 212 performs wireless communication with the care recipient terminal 110 and the caregiver terminal 120.

[0028] The control unit 210 includes a time information acquisition unit 221, a service provision unit 222, a display control unit 223, a biometric information acquisition unit 224, an image acquisition unit 225, and an emotion analysis unit 226.

[0029] The time information acquisition unit 221 acquires time information indicating the past received by the care recipient terminal 110 associated with the care recipient 110H. For example, as described above, the server 130 displays a plurality of images of the care recipient 120H (e.g., the screen of FIG. 6 described later) from the present to the past in chronological order on the touch panel 110A of the care recipient terminal 110.

[0030] The assisted person 110H performs an operation of designating any one of a plurality of images on the touch panel 110A. The assisted person terminal 110 receives the operation. Then, the assisted person terminal 110 transmits information indicating the designated image to the server 130 as the above-described response data. As will be described later, date information is associated with each image in the server 130. The timing information acquisition unit 221 acquires the date information associated with the image indicated by the response data as timing information. Details will be described later.

[0031] The service providing unit 222 acquires an image corresponding to the timing information from among a plurality of images of the assistant 120H stored in the storage unit 211, and generates an avatar image of the acquired image. Date information is associated with each image of the assistant 120H in the image table T1 described later. The service providing unit 222 may, for example, refer to the image table T1, acquire an image of the assistant 120H corresponding to the timing information, and generate an avatar image of the acquired image.

[0032] The service providing unit 222 causes the generated avatar image to be displayed on the touch panel 110A of the assisted person terminal 110. The service providing unit 222 may control to change the movement of the mouth of the generated avatar image according to the speech content spoken by the assistant 120H.

[0033] As described above, the display control unit 223 performs control to display the plurality of images of the above-described assistant 120H on the touch panel 110A of the assisted person terminal 110 in chronological order.

[0034] The biometric information acquisition unit 224 acquires the biometric information of the assisted person 110H from a measuring device that measures the biometric information of the assisted person 110H. The measuring device may be, for example, a bed sensor described later. The bed sensor is, for example, a pneumatic sensor arranged at the bottom of the bed when the assisted person 110H sleeps. Based on the detected body vibration, the bed sensor acquires sleep information regarding the respiration rate, heart rate, activity level, posture, wakefulness / sleep, and getting out of bed / staying in bed as biometric information. The biometric information acquisition unit 224 may acquire the above sleep information from the bed sensor as the biometric information of the assisted person 110H. The above measuring device may be a sphygmomanometer, a thermometer, or the like.

[0035] The image acquisition unit 225 acquires an image of the face of the assisted person 110H captured by the camera of the assisted person terminal 110. The emotion analysis unit 226 performs emotion analysis of the assisted person 110H based on the acquired image of the face of the assisted person 110H.

[0036] The emotion analysis unit 226 may perform emotion analysis of the assisted person 110H using, for example, AI (Artificial Intelligence) technology. For example, using a learned model obtained by performing machine learning with data in which information indicating emotions such as joy, anger, sorrow, and happiness is associated with a plurality of face images as training data, the emotion analysis unit 226 may perform emotion analysis of the assisted person 110H from the image of the face of the assisted person 110H. The emotion analysis unit 226 may perform emotion analysis of the assisted person 110H by a method other than AI technology.

[0037] Next, the assisted person terminal 110 and the caregiver terminal 120 will be described. FIG. 3 is a diagram showing an example of the configuration of the assisted person terminal and the caregiver terminal. In FIG. 3, the terminal 300 corresponds to the assisted person terminal 110 and the caregiver terminal 120. The configuration of the caregiver terminal 120 may be different from the configuration of the assisted person terminal 110. The terminal 300 is, for example, a device such as a smartphone or a tablet terminal.

[0038] The terminal 300 includes a control unit 310, a storage unit 320, a communication unit 330, a camera 340, a microphone 350, a speaker 360, and a touch panel 370. The control unit 310 performs various controls of the terminal 300. The control unit 310 has, for example, a processor and a memory. By the processor executing a program stored in the memory, various processes executed by the control unit 310 are realized.

[0039] The storage unit 320 stores various information. The communication unit 330 communicates with the server 130. The camera 340 is an imaging device that images a predetermined field of view. When the terminal 300 is the assisted person terminal 110, the camera 340 images an image including the face of the assisted person 110H. When the terminal 300 is the assistant terminal 120, the camera 340 images an image including the face of the assistant 120H.

[0040] The microphone 350 is a sound collection device that collects the speech content of the user (such as the assisted person 110H or the assistant 120H) using the terminal 300. The speech content collected by the microphone 350 is transmitted to the server 130 as speech data. The speaker 360 is a voice output device that outputs voice to the user using the terminal 300.

[0041] The touch panel 370 has a display function and an input function. The touch panel 370 corresponds to the touch panel 110A of the assisted person terminal 110 and the touch panel 120A of the assistant terminal 120. The touch panel 370 (the touch panel 110A and the touch panel 120A) may be a display unit having the above display function.

[0042] Next, an example of the image table will be described. FIG. 4 is a diagram showing an example of the image table. The image table T1 includes items of accounts and image data.

[0043] An account is identification information assigned to a user who uses the services provided by server 130. Both the care recipient 110H and the caregiver 120H are assigned to one account. For example, when the caregiver 120H first uses the services provided by server 130, the control unit 210 assigns one account to the caregiver 120H. Hereinafter, the caregiver 120H and the care recipient 110H corresponding to the account "001" will be described.

[0044] The item of image data includes information for identifying an image (image identification information) and information on the date indicating the image (date information). The image identification information and the date information are associated with each other. As described above, the caregiver 120H uses the caregiver terminal 120 to perform an operation of uploading images of a plurality of past photos of the caregiver 120H at different times to the server 130.

[0045] Each of the plurality of images is associated with a date. For example, it is assumed that the caregiver 120H uses the camera 240 of the caregiver terminal 120 to take a picture of the caregiver 120H. The control unit 310 of the caregiver terminal 120 may associate the shooting date of the shooting operation with the image of the caregiver 120H.

[0046] Also, it is assumed that the caregiver 120H uses the camera 240 of the caregiver terminal 120 to perform an operation of taking a picture of an old photo of the caregiver 120H. The date when the photo of the caregiver 120H was taken does not match the actual date when the photo was taken.

[0047] For example, the control unit 310 of the caregiver terminal 120 may display a menu screen for taking an old photo on the touch panel 370 (touch panel 120A). In response to receiving an operation of selecting the menu screen, the control unit 310 of the caregiver terminal 120 may display a message prompting the input of the date of the taken photo on the touch panel 370. When a date is input on the touch panel 370 in response to the message, the control unit 310 may associate the image of the taken photo (old photo of the caregiver 120H) with the input date.

[0048] Also, when the caregiver terminal 120 can communicate with a scanner (not shown), the control unit 310 may associate the old photo of the caregiver 120H read by the scanner with the date input according to the above message.

[0049] As described above, the server 130 acquires a plurality of images respectively associated with dates from the caregiver terminal 120. The server 130 may acquire a plurality of images respectively associated with dates from the caregiver terminal 120 all at once, or may acquire them at different timings.

[0050] "Date: d" in FIG. 4 represents the date associated with each of the above-described images. Among "Date d-n" (n is a natural number), as the value of "n" increases, it indicates that the associated image is an old image. For example, Image 1 is the latest image, and Image 5 is the oldest image. The control unit 210 of the server 130 manages the image table in FIG. 4 for each account.

[0051] FIG. 5 is a diagram showing an example of chronological images of the caregiver. Images 1 to 5 in FIG. 5 are chronological images of the past caregiver 120H corresponding to the account "001". Image 1 is the latest image among the plurality of images of the caregiver 120H stored in the storage unit 211 of the server 130. As going from Image 1 to Image 5, the image of the caregiver 120H becomes an older image. Image 5 is the oldest image among the plurality of images of the caregiver 120H stored in the storage unit 211 of the server 130.

[0052] Next, the specification of an image recognizable by the care recipient 110H among a plurality of past images of the caregiver 120H will be described. As described above, the care recipient 110H designates one image among the images of the caregiver 120H displayed in chronological order using the care recipient terminal 110.

[0053] FIG. 6 is a diagram showing an example of image specification. The control unit 210 of the server 130 acquires images 1 to 5 of the past caregiver 120H corresponding to the account "001" from the storage unit 211.

[0054] The display control unit 223 causes the acquired Images 1 to 5 to be sequentially displayed on the touch panel 120A of the assisted person terminal 120 in chronological order. As shown in the example of FIG. 6, the display control unit 223 first causes Image 1 (the latest image of the assistant 120H) to be displayed on the touch panel 120A of the assisted person terminal 120. At this time, the display control unit 223 causes the YES button 110B and the NO button 110C to be displayed on the touch panel 120A of the assisted person terminal 120.

[0055] When the YES button 110B is pressed, the assisted person terminal 110 transmits information indicating that the YES button 110B has been pressed to the server 130. When receiving the information, the control unit 210 of the server 130 determines that the assisted person 110H can recognize the time (the time of date d) of Image 1 of the assistant 120H.

[0056] When the NO button 110C is pressed, the assisted person terminal 110 transmits information indicating that the NO button 110C has been pressed to the server 130. When receiving the information, the control unit 210 of the server 130 determines that the assisted person 110H cannot recognize the time of Image 1 of the assistant 120H. In the example of FIG. 6, it is assumed that the NO button 110C has been pressed for Image 1. In this case, the control unit 210 determines that the assisted person 110H cannot recognize the time of date d associated with Image 1.

[0057] When the control unit 210 of the server 130 determines that the time of Image 1 cannot be recognized, it causes the second newest Image 2 to be displayed on the touch panel 120A of the assisted person terminal 120. Based on which of the YES button 110B and the NO button 110C has been pressed, the control unit 210 of the server 130 determines that the assisted person 110H cannot recognize the time of Image 2 of the assistant 120H. In the example of FIG. 6, it is assumed that the NO button 110C has been pressed. In this case, the control unit 210 determines that the assisted person 110H cannot recognize the time of date (d - 1) associated with Image 1.

[0058] When the control unit 210 of the server 130 determines that it cannot recognize the timing of Image 2, it causes the third newest Image 3 to be displayed on the touch panel 120A of the assisted person terminal 120. Based on which of the YES button 110B and the NO button 110C is pressed, the control unit 210 of the server 130 determines that the assisted person 110H cannot recognize the timing of Image 3 of the assistant 120H.

[0059] In the example of FIG. 6, it is assumed that the YES button 110B is pressed. In this case, the control unit 210 determines that the assisted person 110H can recognize the timing of the date (d−2) associated with Image 3.

[0060] The service providing unit 222 generates an avatar image using Image 3 that is determined to be recognizable by the assisted person 110H. In the example of FIG. 6, an avatar image 3 using the same image as Image 3 is generated. The avatar image 3 is an avatar image that virtualizes the past face of the assistant 120H at the time of the date (d−2).

[0061] The service providing unit 222 displays the avatar image 3 on the touch panel 110A of the assisted person terminal 110. Also, the service providing unit 222 displays a real-time image of the assisted person 110H on the touch panel 120A of the assistant terminal 120. The real-time image of the assisted person 110H may be an image captured by the camera 340 of the assisted person terminal 110.

[0062] The service providing unit 222 controls the conversation between the assisted person terminal 110 and the assistant terminal 120. The service providing unit 222 may perform control to change the movement of the mouth of the avatar image 3 displayed on the touch panel 110A of the assisted person terminal 110 according to the speech content of the assistant 120H spoken to the speaker 360 of the assistant terminal 120.

[0063] As described above, when a conversation is carried out using the assisted person terminal 110 and the assistant terminal 120, an avatar image of the assisted person 110H at a past time recognized by the assisted person 110H is displayed on the touch panel 110A of the assisted person terminal 110. Thereby, the assisted person 110H can recognize who the conversation partner is, so that a smooth conversation can be promoted.

[0064] Next, an example of the processing flow of the first embodiment will be described. FIG. 7 is a sequence chart showing an example of the processing flow of the first embodiment. For example, it is assumed that when the assistant 120H uses the assistant terminal 120 to start using the service of the server 130, a login operation is required.

[0065] The assistant 120H performs a login operation on the assistant terminal 120 (step S100). Based on the login operation, the control unit 210 of the server 130 recognizes the account of the assistant 120H who uses the service of the server 130. Thereafter, the control unit 210 transmits a request to upload the current and past images of the assistant 120H to the assistant terminal 120 corresponding to the account via the communication unit 212.

[0066] As will be described later, the current image of the assistant 120H is used to identify the assistant 120H by performing face authentication. Also, the current image of the assistant 120H may be the latest image of the assistant 120H.

[0067] For example, when the assistant 120H first uses the service provided by the server 130, the server 130 makes a request to transmit a current image including the face of the assistant 120H. In response to the request, when the assistant 120H uses the camera 340 of the assistant terminal 120 to take an image including his / her own face, the assistant terminal 120 may upload the taken image (image including the face of the assistant 120H) to the server 130 as the current image.

[0068] When the assistant terminal 120 receives an operation to upload an image, it uploads the current image of the current assistant 120H stored in the storage unit 320 and a plurality of past images of the assistant 120H at different times to the server 130 (step S101). As described above, a date is associated with each of the plurality of images. For example, date information may be included in the metadata associated with each image, or date information may be included in the file name. Thus, a date may be associated with each of the plurality of images.

[0069] The current image of the current assistant 120H to be uploaded and the plurality of past images of the assistant 120H at different times may be selected by the assistant 120H. For example, the control unit 310 of the assistant terminal 120 performs face recognition on each image stored in the storage unit 320, and acquires a plurality of images including faces from the storage unit 320.

[0070] For example, the control unit 310 may display a list of the acquired plurality of images (images including faces) on the touch panel 370, and allow the assistant 120H to select the current image to be uploaded and the plurality of past images of the assistant 120H at different times. The assistant terminal 120 may upload the selected current image and the plurality of past images of the assistant 120H at different times to the server 130.

[0071] The communication unit 212 of the server 130 acquires the uploaded plurality of images. The control unit 210 stores each of the acquired images in the storage unit 211 in association with an account (step S102). Further, the control unit 210 associates image identification information and date information with each of the acquired images in association with the account, and records them in the image table T1 of FIG. 4.

[0072] The control unit 210 performs face authentication on the face of the current caregiver 120H by analyzing the image of the caregiver 120H (step S103). For each of the plurality of past images of the caregiver 120H that have been uploaded, the control unit 210 extracts, as an image of the caregiver 120H, the image (image including the face) of the person with the highest matching rate with the face of the caregiver 120H authenticated in step S103 (step S104).

[0073] For example, past images may include persons other than the caregiver 120. In this case, it is necessary to exclude persons other than the caregiver 120 from the past images. The process of step S104 is performed to exclude persons other than the caregiver 120. Each uploaded past image is associated with a date. Therefore, the control unit 210 associates the date associated with the image (past image) from which the image of the caregiver 120H was extracted with the image of the caregiver 120H extracted in step S104. As a result, date information is associated with the extracted image of the caregiver 120H. Each of the following past images of the caregiver 120H will be described as being the image extracted in step S104.

[0074] The control unit 210 arranges each of the extracted past images of the caregiver 120H in chronological order (step S105). At this time, the control unit 210 arranges each of the extracted past images of the caregiver 120H in chronological order based on the date information associated with each of the plurality of past images of the caregiver 120H that have been uploaded.

[0075] The control unit 210 transmits, via the communication unit 212, each of the arranged past images of the caregiver 120H to the cared-for person terminal 110 (step S107). The cared-for person terminal 110 displays each received image in order along the time series on the touch panel 110A (step S107).

[0076] As shown in the example of FIG. 6, the cared-for person terminal 110 may also display, on the touch panel 110A, a message for confirming whether the cared-for person terminal 110 can recognize who the caregiver 120H is, together with the image of the caregiver 120H displayed on the touch panel 110A.

[0077] The above message does not have to be a visual message, for example. For example, the assisted person terminal 110 may cause the speaker 360 to utter a voice of "Do you know this person?"

[0078] Assume that the microphone 350 picks up an affirmative answer (for example, "Yes") or a negative answer (for example, "No") from the assisted person 110H using the assisted person terminal 110. If the collected answer is an affirmative answer, the assisted person terminal 110 may determine that the YES button 110B has been pressed, and if the collected answer is a negative answer, the assisted person terminal 110 may determine that the NO button 110C has been pressed.

[0079] The assisted person terminal 110 determines which of a plurality of past images of the assistant 120H sequentially displayed in time series on the touch panel 110A has been recognized. The assisted person terminal 110 transmits information indicating the recognized image among the plurality of past images of the assistant 120H to the server 130 (step S108). The information indicating the recognized image is time information indicating the period when the assisted person 110H can recognize.

[0080] The service providing unit 222 of the server 130 generates an avatar image of the image of the assistant 120H corresponding to the above time information (step S109). The avatar image is an avatar image based on the image recognized by the assisted person 110H.

[0081] The service providing unit 222 starts conversation control using the generated avatar image (step S110). The service providing unit 222 displays the generated avatar image on the touch panel 110A of the assisted person terminal 110. As described above, the service providing unit 222 may operate the mouth part of the avatar image according to the utterance content uttered by the assistant 120H to the assistant terminal 120.

[0082] As described above, in this embodiment, the server 130 causes the touch panel 110A of the assisted person terminal 110 to display a plurality of past images of the assistant 120H. The assisted person terminal 110 receives a designation of an image that the assisted person 110H has recognized among the plurality of displayed images. The server 130 causes the avatar image of the designated image to be displayed on the touch panel 110A of the assisted person terminal 110 and performs conversation control between the assistant 120H and the assisted person 110H. Thereby, for example, even if the cognitive ability of the assisted person 110H, who is an elderly person, has declined, conversation control using the avatar image of an image that the assisted person 110H can recognize can be performed. Therefore, since the assisted person 110H can recognize who the conversation partner is, the conversation between the assistant 120H and the assisted person 110H can be promoted.

[0083] The timing information indicating the past time point that the assisted person 110H has recognized may be acquired by a method other than the above-described method using a plurality of past images of the assistant 120H. For example, the server 130 causes the touch panel 110A of the assisted person terminal 110 to display the timing information for each era. Based on the operation by which the assisted person 110H designates the timing information for each era, the server 130 may acquire the timing information indicating the timing that the assisted person 110H can recognize.

[0084] For example, it is assumed that the storage unit 211 of the server 130 stores the respective representative timing information for each year. Instead of images 1 to 5 in FIG. 6, the server 130 causes the touch panel 110A of the assisted person terminal 110 to display the respective representative timing information for each year. The assisted person terminal 110 may transmit, as the timing information, the information of the year corresponding to the timing information when the YES button 110B is pressed to the server 130.

[0085] The service providing unit 222 acquires an image corresponding to the date information closest to the time information (information on the year corresponding to the time situation information) transmitted by the care recipient terminal 110 among a plurality of past images of the caregiver 120H stored in the storage unit 211. Then, the service providing unit 222 may generate an avatar image of the acquired image and display the generated avatar image on the touch panel 110A of the care recipient terminal 110. Also in this case, the same effects as those of the present embodiment can be obtained.

[0086] In step S104 described above, for each of the plurality of past images of the caregiver 120H uploaded, the control unit 210 extracts, as the image of the caregiver 120H, the image of the person with the highest matching rate with the face of the caregiver 120H that was face-authenticated in step S103. The control unit 210 may group the plurality of past images of the caregiver 120H uploaded into a plurality of groups.

[0087] For example, the control unit 210 groups the uploaded plurality of past images for each photo recognized as having the same face. Also, the control unit 210 recognizes and extracts the face rectangles from the plurality of images, extracts feature amounts from the images of the face rectangles of each face, and performs grouping by a known method such as K-means. The control unit 210 may display one sample of the plurality of past images belonging to the group on the touch panel 110A of the care recipient terminal 110.

[0088] Here, when the timing of the captured photo is close, it is easy to be recognized as having the same face. As a result, by arranging each image belonging to the grouped group in chronological order, in effect, all the uploaded photos can be arranged in chronological order collectively, making it easier to organize.

[0089] Assume that there are many images of the past of the caregiver 120H that have been uploaded and the same face (or a similar face) appears frequently. In this case, if we simply arrange each image with the same face (or a similar face) in chronological order and display it on the touch panel 110A of the care recipient terminal 110, the care recipient 110H will have to confirm and select the same face (or a similar face) many times, which is not user-friendly.

[0090] Therefore, the control unit 210 displays each image in FIG. 6 on the touch panel 110A for each of the above grouped groups. For example, assume that the NO button 110C is pressed in image 1 of FIG. 6. When the NO button 110C is pressed, the control unit 210 may display an image (photo) belonging to a group different from the group to which image 1 belongs on the touch panel 110A. As a result, since photos of groups with faces clearly different from the face of the caregiver 120H will not be displayed on the touch panel 110A, the number of times the care recipient 110H selects an image can be reduced. This is the same for the increment in step S204 and the decrement in step S211 of FIG. 10 described in the second embodiment to be described later. <Second Embodiment>

[0091] (First Example) Next, the first example of the second embodiment will be described. FIG. 8 is a diagram showing a configuration example of the system 400 of the second embodiment. The system 400 of the second embodiment has a bed sensor 151 and a treadmill 152 added to the system 100 of FIG. 1.

[0092] In addition, the camera 340 of the care recipient terminal 110 captures the face of the care recipient 110H for a predetermined period of time. The care recipient terminal 110 transmits the captured face image to the server 130 as a captured image. The face of the care recipient 110H for a predetermined period of time may be captured, for example, by a camera installed in the room where the care recipient 110H lives.

[0093] As described above, the bed sensor 151 acquires sleep information regarding the respiration rate, heart rate, activity level during sleep (body movements other than respiration and heartbeat (e.g., body activity level such as turning over in bed)), sleeping posture, wakefulness / sleep, and getting out of bed / staying in bed. The bed sensor 151 transmits the acquired sleep information to the server 130. The biological information acquisition unit 224 of the server 130 acquires the sleep information transmitted by the bed sensor 151 as biological information. The control unit 210 recognizes the sleep time of the care recipient 110H based on the acquired sleep information.

[0094] For example, when the care recipient 110H gets into bed, the bed sensor 151 detects the body vibration (body movement, vibration) of the user U through the mattress. Based on the body vibration detected by the bed sensor 151, information regarding the respiration rate, heart rate, activity level, posture, wakefulness / sleep, and getting out of bed / staying in bed can be obtained.

[0095] The treadmill 152 is a sports equipment. When the care recipient 110H uses the treadmill 152 to perform exercises such as walking or jogging, the server 130 is transmitted with exercise information regarding the exercise performed by the care recipient 110H. The sports equipment may be, for example, an ergometer, an air bike (registered trademark), or the like.

[0096] FIG. 9 is a diagram showing an example of a management table. The management table T2 is a table for managing various situations of the care recipient 110H. The management table T2 includes items such as an account, a set avatar image, an average sleep time, a previous day's sleep time, an exercise history, the emotion index of the penultimate time, and the emotion index of the previous time.

[0097] The account is the same as the account in the image table T1 in FIG. 4. The set avatar image is information indicating the avatar image generated by the service providing unit 222 in the first embodiment. When generating the avatar image, the service providing unit 222 records, in the item of the set avatar image in the management table T2, the past image of the caregiver 120H corresponding to the avatar image (an image recognizable by the care recipient 110H) and the information of the date associated with the image.

[0098] The average sleep time represents the average sleep time of the care recipient 110H. As described above, the biological information acquisition unit 224 acquires sleep information from the bed sensor 151. The control unit 210 analyzes the information on the daily sleep time based on the acquired sleep information. The control unit 210 calculates the average sleep time for a certain period in the past (for example, the past one week) based on the analyzed daily sleep time. The control unit 210 records the calculated average sleep time information in the average sleep time item of the management table T2.

[0099] The previous day's sleep time represents the sleep time of the care recipient 110H on the previous day. As described above, the control unit 210 analyzes the information on the daily sleep time based on the acquired sleep information. The previous day's sleep time is obtained from the analyzed sleep time one day before. The control unit 210 records the information indicating the previous day's sleep information in the previous day's sleep time item of the management table T2.

[0100] The exercise history is information representing the history of the care recipient 110H's exercise in a certain period in the past (for example, the past one week). As described above, the treadmill 152 transmits exercise information regarding the exercise of the care recipient 110H to the server 130. For example, the control unit 210 aggregates information regarding the exercise of the care recipient 110H at regular intervals and records the aggregated exercise-related information in the exercise history item.

[0101] The emotional indicators of the penultimate and the previous emotional indicators are values obtained by indexing the emotions of the care recipient 110H. In this embodiment, it is assumed that the emotional indicator is the happiness level. The happiness level is obtained by performing emotional analysis on the facial expression of the face included in the image of the care recipient 110H.

[0102] As described above, the camera 340 of the care recipient terminal 110 captures the face of the care recipient 110H for a predetermined time. The predetermined time may be, for example, the time when the care recipient 110H and the caregiver 120H have a conversation using the service provided by the server 130. Also, the predetermined time may be set as a predetermined time within a day (for example, the time when the care recipient 110H is awake, such as from 7:00 to 20:00, etc.).

[0103] While the assisted person 110H and the caregiver 120H are having a conversation using the service provided by the server 130, the camera 340 of the assisted person terminal 110 captures the face of the assisted person 110H. The assisted person terminal 110 always transmits the image (captured image) captured by the camera 340 to the server 130.

[0104] The emotion analysis unit 226 of the server 130 performs emotion analysis of the assisted person 110H based on the facial expression of the face of the assisted person 110H included in the captured image transmitted by the assisted person terminal 110. The emotion analysis unit 226 performs emotion analysis of the assisted person 110H using, for example, AI technology.

[0105] As an example, the emotion analysis unit 226 may perform emotion analysis of the assisted person 110H by inputting the above captured image into a learned model that has been machine-learned using a plurality of training data in which a plurality of training images including human faces are associated with information indicating human emotions. The human emotions in the present embodiment are, for example, happiness, sadness, anger, fear, surprise, disgust, neutrality, etc.

[0106] In the present embodiment, the emotion analysis unit 226 analyzes the emotions shown by the assisted person 110H while the assisted person 110H and the caregiver 120H are having a conversation in each of the penultimate and previous times. The emotion analysis unit 226 analyzes, for example, the facial expression of the face of the assisted person 110H included in the captured image to analyze the time showing each emotion among the emotions of happiness, sadness, anger, fear, surprise, disgust, and neutrality.

[0107] As a result of analyzing the facial expression of the face of the assisted person 110H included in the captured image, the emotion analysis unit 226 analyzes the time showing happiness as positive time. On the other hand, the emotion analysis unit 226 analyzes the time showing any one of the emotions of anger, fear, surprise, and disgust as negative time. The emotion analysis unit 226 analyzes the time showing a neutral emotion as a time that does not belong to either positive time or negative time.

[0108] The emotion analysis unit 226 divides the positive time by the negative time for the above-mentioned predetermined time. Thereby, the emotion analysis unit 226 may calculate the happiness level as an emotion index of the care recipient 110H. If the value of the happiness level is higher than "1.0", it indicates that the care recipient 110H was happy during the above-mentioned predetermined time. If the value of the happiness level is lower than "1.0", it indicates that the care recipient 110H was not happy during the above-mentioned predetermined time. When the value of the happiness level is "1.0", it may be determined that it is impossible to determine whether the care recipient 110H was happy during the above-mentioned predetermined time.

[0109] Each time the care recipient 110H and the caregiver 120H have a conversation using the service provided by the server 130, the emotion analysis unit 226 updates the items of the emotion index of the time before last and the emotion index of the last time.

[0110] Also, the emotion analysis unit 226 may calculate the happiness level as an emotion index by another method while the care recipient 110H and the caregiver 120H are having a conversation using the service provided by the server 130. For example, the emotion analysis unit 226 may calculate the happiness level as an emotion index by dividing the number of times indicating happiness (positive times) by the number of times indicating any one of the emotions of anger, fear, surprise, and disgust (negative times).

[0111] Next, the processing of the first example of the second embodiment will be described. The processing of the first example is an example of generating an avatar image based on sleep information. FIG. 10 is a flowchart showing an example of the flow of the processing of the first example of the second embodiment.

[0112] The control unit 210 refers to the sleep time information in the management table T2 of FIG. 9 (step S201). At this time, the control unit 210 refers to the items of the average sleep time and the previous day's sleep time corresponding to the account "001". The account "001" represents both the caregiver 120H and the care recipient 110H.

[0113] The control unit 210 refers to the set avatar image in the management table T2 of FIG. 9 (step S202). At this time, the control unit 210 refers to the item of the set avatar image corresponding to the account "001".

[0114] Based on the average sleep time and the previous day's sleep time referred to in step S201, the control unit 210 determines whether the sleep of the care recipient 110H has improved (step S203). For example, when the previous day's sleep time is longer than the average sleep time, the control unit 210 determines Yes in step S203. Also, when the previous day's sleep time is shorter than the average sleep time, or when the previous day's sleep time is the same as the average sleep time, the control unit 210 determines No in step S203.

[0115] When the control unit 210 determines Yes in step S203, it performs the processes of steps S204 to S209. Triggered by the determination that the sleep of the care recipient 110H has improved, the control unit 210 automatically performs the processes of steps S204 to S209. The following processes of steps S204 to S209 are also referred to as recovery confirmation processing or recovery confirmation tests.

[0116] The recovery confirmation process will be described. The control unit 210 increments the image identification variable (step S204). The image identification variable represents the date when the care recipient 110H could be recognized. The image identification variable corresponds to the date among the set avatar images in the management table T2 of FIG. 9. In the management table T2 of FIG. 9, the image identification variable is "d-2". Therefore, when the image identification variable is incremented in step S204, the image identification variable becomes "d-1".

[0117] The display control unit 223 transmits, via the communication unit 212, the image of the caregiver 120H corresponding to the incremented image identification variable to the care recipient terminal 110 (step S205). At this time, the display control unit 223 causes the screen shown in the example of FIG. 6 to be displayed on the touch panel 110A of the care recipient terminal 110.

[0118] For example, when the image identification variable is "d-1", the display control unit 223 refers to the image table T1 in FIG. 4 and acquires the image 2 corresponding to the date "d-1". Then, together with the message "Do you know this person?", the YES button 110B and the NO button 110C are displayed on the touch panel 110A of the assisted person terminal 110.

[0119] As described above, the item of the set avatar image recorded in the management table T2 indicates the date "d-2". On the other hand, the display control unit 223 displays the image 2 corresponding to the date "d-1" on the touch panel 110A of the assisted person terminal 110. That is, the display control unit 223 displays an image of the assistant 120H that is newer than the image of the assistant 120H corresponding to the set avatar image on the touch panel 110A of the assisted person terminal 110.

[0120] The control unit 210 determines whether the image displayed on the touch panel 110A of the assisted person terminal 110 has been recognized (step S206). When the YES button 110B displayed on the touch panel 110A of the assisted person terminal 110 is pressed, the control unit 210 determines Yes in step S206. On the other hand, when the NO button 110C displayed on the touch panel 110A of the assisted person terminal 110 is pressed, the control unit 210 may determine No in step S206.

[0121] When the control unit 210 determines No in step S206, the process returns to step S204. On the other hand, when the control unit 210 determines Yes in step S206, the process proceeds to step S207. When the control unit 210 determines Yes in step S206, it can be estimated that the cognitive function of the assisted person 110H has recovered due to the improvement of the sleep of the assisted person 110H. This is because there is a correlation between sleep and cognitive function.

[0122] The service providing unit 222 generates an avatar image based on the image of the caregiver 120H corresponding to the incremented image identification variable (step S207). At this time, the service providing unit 222 updates the image and date among the items of the set avatar image in the management table T2 to the image and date corresponding to the incremented image identification variable.

[0123] The service providing unit 222 causes a compliment message to be displayed on the touch panel 110A of the care recipient terminal 110 (step S208). The compliment message is a message that praises the care recipient 110H. The compliment message is stored in the storage unit 211, for example, as a message to be displayed when performing the recovery confirmation process.

[0124] For example, the compliment message is a message such as "There is no problem with your current sleep and lifestyle. Let's keep working hard together!" The service providing unit 222 automatically causes the compliment message to be displayed on the touch panel 110A of the care recipient terminal 110.

[0125] The service providing unit 222 starts conversation control using the avatar image generated in step S207 (step S209). As a result, the care recipient 110H can have a conversation with the caregiver 120H while looking at the avatar image displayed on the touch panel 110A of the care recipient terminal 110. The process of step S208 may be performed after the process of step S209.

[0126] As described above, when the control unit 210 determines No in step S203, the process proceeds to step S210. The control unit 210 determines whether the sleep of the care recipient 110H has deteriorated based on the average sleep time and the previous day's sleep time referred to in step S201 (step S210).

[0127] For example, if the sleep time the previous day is shorter than the average sleep time, the control unit 210 determines Yes in step S210. On the other hand, if the sleep time the previous day is the same as the average sleep time, the control unit 210 determines No in step S210 and ends the flowchart of FIG. 10.

[0128] When the control unit 210 determines Yes in step S210, it performs the processes of steps S211 to S216. The control unit 210 automatically performs the processes of steps S211 to S216 triggered by determining that the sleep of the assisted person 110H has deteriorated. The following processes of steps S211 to S215 are also referred to as a backward confirmation process or a backward confirmation test.

[0129] The backward confirmation process will be described. The control unit 210 decrements the image identification variable (step S211). In the management table T2 of FIG. 9, the image identification variable of the set avatar image is "d - 2". Therefore, when the image identification variable is decremented in step S211, the image identification variable becomes "d - 3".

[0130] The display control unit 223 transmits, via the communication unit 212, the image of the assistant 120H corresponding to the decremented image identification variable to the assisted person terminal 110 (step S212). At this time, the display control unit 223 causes the screen shown in the example of FIG. 6 to be displayed on the touch panel 110A of the assisted person terminal 110.

[0131] For example, when the image identification variable is "d - 3", the display control unit 223 refers to the image table T1 of FIG. 4 and acquires the image 4 corresponding to the date "d - 3". Then, together with the message "Do you know this person?", the YES button 110B and the NO button 110C are displayed on the touch panel 110A of the assisted person terminal 110.

[0132] That is, the display control unit 223 causes an image of the assistant 120H in the past rather than the image of the assistant 120H corresponding to the set avatar image to be displayed on the touch panel 110A of the assisted person terminal 110.

[0133] The control unit 210 determines whether the image displayed on the touch panel 110A of the assisted person terminal 110 has been recognized (step S213). The determination in step S213 is the same as the determination in step S206.

[0134] If the control unit 210 determines No in step S213, it returns the process to step S211. On the other hand, if the control unit 210 determines Yes in step S213, it advances the process to step S214. When the control unit 210 determines Yes in step S213, it can be presumed that the cognitive function of the assisted person 110H has regressed due to the deterioration of the sleep of the assisted person 110H.

[0135] The service providing unit 222 generates an avatar image based on the image of the assistant 120H corresponding to the decremented image identification variable (step S214). At this time, the service providing unit 222 updates the image and date among the items of the set avatar image in the management table T2 to the image and date corresponding to the decremented image identification variable.

[0136] The service providing unit 222 causes the improvement advice to be displayed on the touch panel 110A of the assisted person terminal 110 (step S216). The improvement advice is a message that encourages the assisted person 110H to improve sleep, lifestyle, etc. The improvement advice is stored in the storage unit 211, for example, as a message to be displayed when performing the regression confirmation process.

[0137] For example, the improvement advice is a message such as "It seems that you are not able to sleep at night because you are taking naps during the day. Get out of bed and move your body during the day!" The service providing unit 222 automatically causes the improvement advice to be displayed on the touch panel 110A of the assisted person terminal 110.

[0138] The service providing unit 222 starts conversation control using the avatar image generated in step S214 (step S216). As a result, the assisted person 110H can have a conversation with the assistant 120H while looking at the avatar image displayed on the touch panel 110A of the assisted person terminal 110. The process of step S215 may be performed after the process of step S216.

[0139] As described above, the control unit 210 performs the recovery confirmation process of steps S204 to S209 triggered by determining that the sleep of the assisted person 110H has improved. If the control unit 210 determines Yes in step S206, it may evaluate that the cognitive function of the assisted person 110H has recovered. In this case, the control unit 210 may notify the assistant terminal 120 of information indicating that the cognitive function of the assisted person 110H has recovered.

[0140] The control unit 210 performs the regression confirmation process of steps S211 to S216 triggered by determining that the sleep of the assisted person 110H has deteriorated. If the control unit 210 determines Yes in step S213, it may evaluate that the cognitive function of the assisted person 110H has regressed. In this case, the control unit 210 may notify the assistant terminal 120 of information indicating that the cognitive function of the assisted person 110H has regressed.

[0141] Also, in step S206, the control unit 210 causes an image of the assistant 120H that is one newer than the image at the set time to be displayed on the touch panel 110A of the assisted person terminal 110. Thereby, a more appropriate avatar image of the assistant 120H can be presented to the assisted person 110H for whom it can be estimated that the cognitive function has recovered.

[0142] In step S213, the control unit 210 causes an image of the assistant 120H that is one older than the image at the set time to be displayed on the touch panel 110A of the assisted person terminal 110. Thereby, a more appropriate avatar image of the assistant 120H can be presented to the assisted person 110H for whom it can be estimated that the cognitive function has regressed.

[0143] In addition, the service providing unit 222 can motivate the assisted person 110H to further improve sleep, lifestyle, etc. by displaying a compliment message or improvement advice on the touch panel 110A of the assisted person terminal 110.

[0144] In the above-described example, in steps S203 and S210, an example of determining whether sleep has improved or deteriorated based on the previous day's sleep time and the average sleep time has been described. However, other indicators related to sleep may be used to determine whether sleep has improved or deteriorated.

[0145] For example, based on the measurement result of stress (stress level), it may be determined whether sleep has improved or deteriorated. There is a correlation between human stress and the interval between heartbeats (inter-beat interval). As the inter-beat interval decreases, stress increases, and as the inter-beat interval increases, stress decreases. The inter-beat interval can be obtained from, for example, the above-described bed sensor. The control unit 210 may determine that sleep has deteriorated when the stress level based on the inter-beat interval obtained from the bed sensor is equal to or higher than a predetermined level. Further, the control unit 210 may determine that sleep has improved when the stress level based on the inter-beat interval obtained from the bed sensor is less than a predetermined level.

[0146] Further, instead of the above-described objective sleep-related indicators such as sleep time and stress level, subjective sleep-related indicators of the care recipient 110H may be used, or both objective sleep-related indicators and subjective sleep-related indicators may be used. For example, the control unit 210 may display a questionnaire related to sleep on the touch panel 110A. The questionnaire may be, for example, a questionnaire that asks whether the sleep was good. When the care recipient 110H inputs an answer to the questionnaire displayed on the touch panel 110A, the control unit 210 may determine whether the sleep has improved or deteriorated based on the input answer. For example, when the input answer indicates that the sleep was good, the control unit 210 may determine that the sleep has improved. Also, when the input answer indicates that the sleep was not good, the control unit 210 may determine that the sleep has deteriorated.

[0147] (Second example) Next, the process of the second example of the second embodiment will be described. The process of the second example is an example of the change of the avatar image based on the exercise history of the care recipient 110H. FIG. 11 is a flowchart showing an example of the flow of the process of the second example of the second embodiment.

[0148] The control unit 210 refers to the exercise history information in the management table T2 of FIG. 9 (step S301). At this time, the control unit 210 refers to the exercise history items corresponding to the account "001".

[0149] The control unit 210 determines whether the exercise history information of the care recipient 110H satisfies a predetermined exercise standard (step S302). For example, the predetermined exercise standard is that aerobic exercise is performed two or more times a week and the exercise time per session is 10 minutes. The exercise standard may be set to any standard. For example, the caregiver 120H may operate the caregiver terminal 120 to arbitrarily set the exercise standard.

[0150] For example, among the management tables T2 in FIG. 9, the exercise history information corresponding to the account "001" indicates that aerobic exercise is performed twice a week, and the exercise time per session is 15 minutes. In this case, the control unit 210 determines that the exercise history of the assisted person 110H meets the predetermined exercise criteria.

[0151] When the control unit 210 determines Yes in step S302, the process proceeds to step S303. The service providing unit 222 changes the face of the generated avatar image to a positive expression (step S303).

[0152] When the exercise history of the assisted person 110H does not meet the predetermined exercise criteria, the control unit 210 determines No in step S302, and the process proceeds to step S304. The service providing unit 222 changes the face of the generated avatar image to a negative expression (step S304).

[0153] For example, the service providing unit 222 may change the face of the generated avatar image to a positive expression by changing the images of the eyes and mouth parts of the generated avatar image to the expressions of the eyes and mouth when a person smiles. Also, the service providing unit 222 may change the face of the generated avatar image to a negative expression by changing the images of the eyes and mouth parts of the generated avatar image to the expressions of the eyes and mouth when a person is angry.

[0154] As described above, the service providing unit 222 changes the expression of the avatar image to a positive expression or a negative expression according to the exercise history information of the assisted person 110H. Thereby, it is possible to promote exercise for the assisted person 110H.

[0155] The service providing unit 222 may change the face of the avatar image to a positive expression or a negative expression based on other indicators other than the exercise history. For example, the control unit 210 may change the face of the avatar image to a positive expression or a negative expression based on the lifestyle of the assisted person 110H.

[0156] For example, assume that the biometric information obtained from the above-described bed sensor indicates that the subject has been in bed for a first period of time (e.g., 3 hours) or more during non-nighttime hours (e.g., from 8:00 am to 8:00 pm). In this case, the service providing unit 222 may determine that the living habits of the care recipient 110H are poor and change the face of the avatar image to a negative expression. On the other hand, if the time the subject has been in bed during non-nighttime hours is less than the first period of time, the service providing unit 222 may determine that the living habits of the care recipient 110H are good and change the face of the avatar image to a positive expression.

[0157] Further, assume that the biometric information obtained from the above-described bed sensor indicates that the time the subject has been in bed is within a second period of time (e.g., 30 minutes) based on a first time (e.g., 8:00 pm) that is a desirable bedtime, and the time the subject has gotten out of bed is within the second period of time based on a second time (e.g., 8:00 am) that is a desirable wake-up time. In this case, the service providing unit 222 may determine that the living habits of the care recipient 110H are good and change the face of the avatar image to a positive expression.

[0158] On the other hand, if the biometric information obtained from the above-described bed sensor indicates that the time the subject has been in bed is not within the second period of time based on the first time, or the time the subject has gotten out of bed is not within the second period of time based on the second time, the service providing unit 222 may determine that the living habits of the care recipient 110H are poor and change the face of the avatar image to a negative expression.

[0159] Also, for example, assume that the control unit 210 records the daily wake-up time and bedtime of the care recipient 110H in the management table T2 of FIG. 9. The wake-up time and bedtime can be obtained based on, for example, the time the subject has been in bed and the time the subject has gotten out of bed included in the biometric information obtained from the above-described bed sensor.

[0160] For example, when the service providing unit 222 refers to the management table T2 and determines that the standard deviation of the variation in the daily wake-up time or bedtime is equal to or greater than a predetermined value, it may determine that the lifestyle of the care recipient 110H is poor and change the face of the avatar image to a negative expression. On the other hand, when the service providing unit 222 determines that the above standard deviation is less than the predetermined value, it may determine that the lifestyle of the care recipient 110H is good and change the face of the avatar image to a positive expression.

[0161] (Third example) Next, the processing of the third example of the second embodiment will be described. The processing of the third example is an example of a recommendation for an interview with the caregiver 120H. FIG. 12 is a flowchart showing an example of the processing flow of the third example of the second embodiment.

[0162] The control unit 210 recognizes the current date (step S401). The control unit 210 refers to the date corresponding to the image identification variable (step S402). At this time, the control unit 210 refers to the date information among the items of the set avatar image corresponding to the account "001" in the management table T2 of FIG. 9.

[0163] The control unit 210 determines whether the difference (number of days difference) between the current date recognized in step S401 and the date corresponding to the image identification variable referred to in step S402 is within a certain range (step S403). The said certain range can be set arbitrarily.

[0164] When the difference in the number of days is within the certain range, the control unit 210 determines Yes in step S403 and advances the process to step S404. The control unit 210 transmits a message recommending an in-person interview to the caregiver terminal 120 (step S404). The said message is displayed on the touch panel 120A of the caregiver terminal 120.

[0165] When the difference in the above-mentioned number of days is within a certain range, the date recognizable by the assisted person 110H is close to the current date. Therefore, when the assisted person 110H and the caregiver 120H have a face-to-face interview, there is a high possibility that the assisted person 110H can recognize who the caregiver 120H is. In this case, a smooth conversation between the assisted person 110H and the caregiver 120H can be realized.

[0166] When the difference in the above-mentioned number of days exceeds a certain range, the control unit 210 determines No in step S403, and ends the flowchart of FIG. 12 without executing step S404.

[0167] When the control unit 210 determines Yes in step S403, the service providing unit 222 may perform conversation control using the actual image of the caregiver 120H. In this case, the server 130 acquires the image of the caregiver 120H captured by the camera 340. The service providing unit 222 may display the image of the caregiver 120H on the touch panel 110A of the assisted person terminal 110 instead of the avatar image. Even in this case, a smooth conversation using the actual images of the assisted person 110H and the caregiver 120H can be realized. (Fourth Example)

[0168] Next, a fourth example of the second embodiment will be described. The fourth example is an example of generating an avatar image based on sentiment analysis. FIG. 13 is a flowchart showing an example of the flow of the fourth process of the second embodiment.

[0169] The control unit 210 refers to the information on the sentiment index of the penultimate time in the management table T2 of FIG. 9 (step S501). At this time, the control unit 210 refers to the item of the sentiment index of the penultimate time corresponding to the account "001".

[0170] The control unit 210 refers to the information on the sentiment index of the previous time in the management table T2 of FIG. 9 (step S502). At this time, the control unit 210 refers to the item of the sentiment index of the previous time corresponding to the account.

[0171] The control unit 210 determines whether the happiness level of the care recipient 110H has increased based on the previous-previous emotional index referred to in step S501 and the previous emotional index referred to in step S502 (step S503). If the value of the previous-previous emotional index is higher than the value of the previous emotional index, the control unit 210 determines that the happiness level of the care recipient 110H has increased. In this case, the control unit 210 determines Yes in step S503 and advances the process to step S504.

[0172] On the other hand, if the value of the previous-previous emotional index is less than or equal to the value of the previous emotional index, the control unit 210 determines that the happiness level of the care recipient 110H has not increased. In this case, the control unit 210 determines No in step S503 and advances the process to step S509.

[0173] When the control unit 210 determines Yes in step S503, it performs the processes of steps S504 to S508. Triggered by the determination that the happiness level of the care recipient 110H has increased, the control unit 210 automatically performs the processes of steps S504 to S509. The processes of steps S504 to S508 are the same as the above-described recovery confirmation process.

[0174] The recovery confirmation process will be described. The control unit 210 increments the image identification variable (step S504). The display control unit 223 transmits, via the communication unit 212, an image of the caregiver 120H corresponding to the incremented image identification variable to the care recipient terminal 110 (step S505). The image of the caregiver 120H is an image of the caregiver 120H that is one new from the image indicated by the set avatar image.

[0175] The display control unit 223 causes the screen shown in the example of FIG. 6 to be displayed on the touch panel 110A of the care recipient terminal 110. The control unit 210 determines whether the image displayed on the touch panel 110A of the care recipient terminal 110 has been recognized (step S506).

[0176] When the YES button 110B displayed on the touch panel 110A of the assisted person terminal 110 is pressed, the control unit 210 determines Yes in step S506. On the other hand, when the NO button 110C displayed on the touch panel 110A of the assisted person terminal 110 is pressed, the control unit 210 determines No in step S506.

[0177] When the control unit 210 determines No in step S506, it returns the process to step S504. On the other hand, when the control unit 210 determines Yes in step S506, it advances the process to step S507. When the control unit 210 determines Yes in step S506, it can be presumed that the cognitive function of the assisted person 110H has recovered due to an increase in the happiness level of the assisted person 110H. This is because it can be presumed that there is a correlation between the happiness level and the cognitive function.

[0178] The service providing unit 222 generates an avatar image based on the image of the caregiver 120H corresponding to the incremented image identification variable (step S507). At this time, the service providing unit 222 updates the image and date among the items of the set avatar image in the management table T2 to the image and date corresponding to the incremented image identification variable.

[0179] The service providing unit 222 starts conversation control using the avatar image generated in step S507 (step S508). As a result, the assisted person 110H can have a conversation with the caregiver 120H while looking at the avatar image displayed on the touch panel 110A of the assisted person terminal 110. The process of step S508 may be performed after the process of step S509.

[0180] As described above, when the control unit 210 determines No in step S503, it advances the process to step S509. The control unit 210 determines whether the happiness level of the assisted person 110H has decreased based on the emotion index of the penultimate time referred to in step S501 and the emotion index of the previous time referred to in step S502 (step S509).

[0181] If the emotion index of the penultimate time is the same as that of the previous time, the control unit 210 determines No in step S509 and ends the flowchart of FIG. 13. On the other hand, if the emotion index of the previous time is lower than that of the penultimate time, the control unit 210 determines Yes in step S509 and advances the process to step S510.

[0182] When the control unit 210 determines Yes in step S509, it performs the processes of steps S211 to S215. The control unit 210 automatically performs the processes of steps S510 to S514 triggered by the fact that the happiness level of the assisted person 110H has decreased. The following processes of steps S510 to S514 are the same as the above-described backward confirmation process.

[0183] The backward confirmation process will be described. The control unit 210 decrements the image identification variable (step S510). The display control unit 223 transmits, via the communication unit 212, an image of the assistant 120H corresponding to the decremented image identification variable to the assisted person terminal 110 (step S511). The image of the assistant 120H is an image of the assistant 120H one time older than the image indicated by the set avatar image.

[0184] The display control unit 223 causes the screen shown in the example of FIG. 6 to be displayed on the touch panel 110A of the assisted person terminal 110. The control unit 210 determines whether the image displayed on the touch panel 110A of the assisted person terminal 110 has been recognized (step S512). The determination in step S512 is the same as the determination in step S506.

[0185] When the control unit 210 determines No in step S512, it returns the process to step S510. On the other hand, when the control unit 210 determines Yes in step S512, it advances the process to step S513. When the control unit 210 determines Yes in step S512, it can be estimated that the cognitive function of the assisted person 110H has regressed due to the decrease in the happiness level of the assisted person 110H.

[0186] The service providing unit 222 generates an avatar image based on the image of the caregiver 120H corresponding to the decremented image identification variable (step S513). At this time, the service providing unit 222 updates the image and date among the items of the set avatar image in the management table T2 to the image and date corresponding to the decremented image identification variable.

[0187] The service providing unit 222 starts conversation control using the avatar image generated in step S513 (step S514). As a result, the care recipient 110H can have a conversation with the caregiver 120H while looking at the avatar image displayed on the touch panel 110A of the care recipient terminal 110. The process of step S513 may be performed after the process of step S514.

[0188] It is assumed that the control unit 210 acquires the communication information between the care recipient 110H and the caregiver 120H while performing conversation control using the avatar image in step S508 or step S514. The communication information is the image of the care recipient 110H and the voice of the conversation, and the image of the caregiver 120H and the voice of the conversation.

[0189] The content of the conversation included in the communication information can be obtained based on the voice data collected by the speaker 360 of the care recipient terminal 110 and the speaker 360 of the caregiver terminal 120. Also, the above communication includes, for example, the expression of the care recipient 110H during the conversation. The expression can be obtained based on the image captured by the camera 340 of the care recipient terminal 110.

[0190] The control unit 210 stores the content of the above communication in the storage unit 21 in association with the account (step S515). The avatar image used in step S508 or step S514 has changed from the image shown in the set avatar image of FIG. 9. By storing the content of the above communication in the storage unit 211 by the control unit 210, the situation of the conversation between the assisted person 110H and the assistant 120H using the changed avatar image can be confirmed later. The situation of the conversation is, for example, whether the conversation is established, or whether the assisted person 110H is enjoying the communication using the changed avatar image, etc.

[0191] As described above, the control unit 210 may notify the assistant terminal 120 of information indicating that the happiness level of the assisted person 110H has increased, or information indicating that the happiness level of the assisted person 110H has decreased.

[0192] In addition, the control unit 210 can present a more appropriate avatar image of the assistant 120H to the assisted person 110H according to whether the happiness level of the assisted person 110H has increased or decreased.

[0193] (Fourth example) Next, a fourth example of the second embodiment will be described. The control unit 210 increments the image identification variable in step S204 of FIG. 10. When the control unit 210 determines No in step S206, it executes step S204 again. As a result, the image identification variable is further incremented.

[0194] The control unit 210 recognizes the value (increment value) by which the image identification variable has been increased until it determines Yes in step S206. The increment value represents the degree of recovery of the cognitive function due to the improvement of the sleep of the assisted person 110H.

[0195] The control unit 210 decrements the image identification variable in step S211 of FIG. 10. When the control unit 210 determines No in step S213, it executes step S211 again. As a result, the image identification variable is further incremented.

[0196] The control unit 210 recognizes the decreased value (decrease value) of the image identification variable until it determines Yes in step S213. The decrease value represents the degree of decline in cognitive function due to the deterioration of the sleep of the assisted person 110H.

[0197] The control unit 210 may cause a screen indicating whether the cognitive function of the assisted person 110H has recovered or declined to be displayed on the touch panel 120A of the caregiver terminal 120. FIG. 14A is a diagram showing an example of a presentation screen regarding changes in cognitive function.

[0198] The presentation screen displayed on the touch panel 120A of the caregiver terminal 120 includes information on the date recognized by the assisted person 110H. The date information can be obtained, for example, from the date information of the set avatar image in the management table T2 of FIG. 9.

[0199] The presentation screen displayed on the touch panel 120A of the caregiver terminal 120 includes information indicating whether the cognitive function of the assisted person 110H has recovered or declined. The presentation screen in the example of FIG. 14A includes information indicating that the cognitive function of the assisted person 110H has declined.

[0200] The presentation screen displayed on the touch panel 120A of the caregiver terminal 120 includes change degree information 120V. The change degree information 120V is visual information representing the degree of recovery or decline of cognitive function. In the example of FIG. 14A, the angle of the arrow with respect to a predetermined reference line (dashed line) represents the degree of recovery or decline of cognitive function.

[0201] As the above-described increase value increases, the control unit 210 increases the angle of the change degree information 120V upward from a predetermined reference line. As the above-described decrease value increases, the control unit 210 increases the angle of the change degree information 120V downward from a predetermined reference line. Thereby, the caregiver 120H can visually recognize the degree of recovery or decline of the cognitive function of the assisted person 110H.

[0202] (Fifth example) Next, a fifth example of the second embodiment will be described. As described above, the assisted person terminal 110 is used when using the services provided by the server 130. After starting the conversation control using the avatar image in step S209 or S216 in FIG. 10, the control unit 210 ends the conversation control. For example, when an operation to disconnect the conversation is performed on the assisted person terminal 110 or the assistant terminal 120, the conversation control by the server 130 (the service provided by the server 130) ends.

[0203] After the conversation control by the server 130 ends, the control unit 210 may display a past photo of the assisted person 110H on the touch panel 110A of the assisted person terminal 110. FIG. 14B is a diagram showing an example of the screen after the conversation control ends.

[0204] The screen after the conversation control ends includes an image of at least one of the assisted person 110H and the assistant 120H. The control unit 210 refers to the date information of the set avatar image corresponding to the account "001" in the management table T2 of FIG. 9. The date information represents the date corresponding to the time when it is estimated that the assisted person 110H can recognize.

[0205] A plurality of images corresponding to the account of the assistant 120H are stored in the storage unit 211. The control unit 210 acquires a plurality of images among the respective images that are earlier than the date of the set avatar image. Then, the control unit 210 performs a face authentication process on the persons included in the acquired plurality of images, and extracts one or more images whose matching rate with the face of the assisted person 110H or the assistant 120H is higher than a predetermined rate.

[0206] When the control unit 210 extracts a plurality of images with a high matching rate as described above, the emotion analysis unit 226 performs an emotion analysis of the assisted person 110H on the extracted plurality of images. The control unit 210 identifies the image in which the assisted person 110H is analyzed to be the happiest among the plurality of images on which the emotion analysis has been performed. The control unit 210 may include the identified image in the screen after the conversation control ends.

[0207] In this case, a photo that is presumed to have made the care recipient 110H the happiest is displayed on the touch panel 110A of the care recipient terminal 110. This can increase the happiness level of the care recipient 110H.

[0208] (Sixth example) Next, a sixth example of the second embodiment will be described. After starting the conversation control using the avatar image described above, the control unit 210 may cause the touch panel 110A of the care recipient terminal 110 to display news information of the past on the date that the care recipient 110H can recognize.

[0209] The server 130 may obtain the news information, for example, from a predetermined server (a server that stores past news). The date that the care recipient 110H can recognize can be obtained based on the date information of the set avatar image in the management table T2 of FIG. 9.

[0210] (Seventh example) In the above-described first and second embodiments, an example in which the care recipient 110H and the caregiver 120H have a conversation has been described. However, the server 130 may perform automatic conversation control using the generated avatar image.

[0211] In step S207 or S214 of FIG. 10, the service providing unit 222 generates an avatar image. The service providing unit 222 may perform conversation control using the conversation database without the intervention of the caregiver 120H.

[0212] FIG. 15 is a diagram showing an example of a conversation database. The conversation database T11 is stored, for example, in the storage unit 211. The conversation database T11 may be stored in a predetermined database server. In this case, the server 130 obtains the conversation database from the database server. The conversation database T11 is a database that stores conversation patterns.

[0213] The conversation database T11 includes items of utterance ID, response, and utterance content. The utterance ID is an ID that identifies the utterance content. The response indicates whether the response from the assisted person 110H is an affirmative response or a negative response. The utterance content is the content spoken by the service providing unit 222.

[0214] The service providing unit 222 transmits the utterance content with the utterance ID "001" as voice data to the assisted person terminal 110. The speaker 360 of the assisted person terminal 110 outputs the utterance content with the utterance ID "001".

[0215] Next, the service providing unit 222 transmits the utterance content with the utterance ID "002" as voice data to the assisted person terminal 110. The speaker 360 of the assisted person terminal 110 outputs the utterance content with the utterance ID "002". The utterance content with the utterance ID "002" is the utterance content of a question to the assisted person 110H.

[0216] Suppose the assisted person 110H gives an affirmative response. The affirmative response is, for example, "Yes". The microphone 350 of the assisted person terminal 110 collects the voice of the affirmative response. The assisted person terminal 110 transmits the data of the collected voice to the server 130. When the service providing unit 222 determines that the data of the collected voice indicates an affirmative response, based on the conversation database, it transmits the utterance content with the utterance ID "003" as voice data to the assisted person terminal 110.

[0217] Suppose the assisted person 110H gives a negative response. The negative response is, for example, "No". The microphone 350 of the assisted person terminal 110 collects the voice of the negative response. The assisted person terminal 110 transmits the data of the collected voice to the server 130. When the service providing unit 222 determines that the data of the collected voice indicates a negative response, based on the conversation database, it transmits the utterance content with the utterance ID "004" as voice data to the assisted person terminal 110.

[0218] As described above, communication through conversation can be carried out between the server 130 and the care recipient 110H without the intervention of the caregiver 120H. As a result, since the care recipient 110H has more opportunities to engage in conversation, it is possible to suppress the decline of the cognitive function of the care recipient 110H.

[0219] (Eighth example) In the above-described first and second embodiments, an example was described in which the control unit 210 uses past time-series images of the caregiver 120H to identify at what times the care recipient 110H can recognize.

[0220] The control unit 210 may use past time-series images of the care recipient 110H instead of past time-series images of the caregiver 120H to identify at what times the care recipient 110H can recognize. Even in this case, it is possible to identify at what times the care recipient 110H can recognize.

[0221] (Ninth example) Next, a ninth example of the second embodiment will be described. As shown in the example of FIG. 10, the server 130 is communicably connected to the treadmill 152. Assume that the treadmill 152 has a monitor and a speaker.

[0222] The control unit 210 may display on the monitor of the treadmill 152 various screens for performing the above-described conversation control to be displayed on the touch panel 110A of the care recipient terminal 110. Further, the control unit 210 may output the sound output from the speaker 360 of the care recipient terminal 110 from the speaker of the treadmill 152.

[0223] The care recipient 110H engages in conversation with the caregiver 120H based on the above-described conversation control while exercising using the treadmill 152. That is, the care recipient 110H performs two tasks, namely, exercise and conversation, simultaneously. Training to perform two tasks simultaneously (dual-task training) contributes to the recovery of the cognitive function of the care recipient 110H. Therefore, the control unit 210 may perform the above-described conversation control using the monitor and speaker of the treadmill 152 or the like. <Third Embodiment>

[0224] (First Example) Next, a first example of the third embodiment will be described. It is assumed that the assistant 110H in the third embodiment is a caregiver rather than a family member of the care recipient 110H. For example, in a scene such as home care, communication by conversation is carried out between the care recipient 110H and the caregiver. In the third embodiment, it is assumed that a plurality of caregivers are in charge of caring for the care recipient 110H.

[0225] FIG. 16 is a diagram showing a configuration example of the system 500 according to the third embodiment. In the system 500 of the third embodiment, a staff terminal 170 is communicably connected to the server 130 instead of the assistant terminal 120. The staff terminal 170 is a terminal corresponding to the caregiver 170H. The configuration of the staff terminal 170 is the same as the configuration of the terminal 300 in FIG. 3.

[0226] Further, the server 130 is connected to a care record database 160. The care record database 160 is a database that stores care records recording information on the daily care of the care recipient 110H. The care record database 160 may be, for example, software operating on a personal computer.

[0227] When using the service provided by the server 130, the caregiver 170H performs an authentication operation using the staff terminal 170. The authentication operation is, for example, the input of an ID and a password. By performing the authentication operation on the staff terminal 170, the server 130 identifies the caregiver 170H.

[0228] FIG. 17 is a diagram showing an example of a staff table. The staff table T3 is stored in the storage unit 211 of the server 130. The staff table T3 includes items of accounts and caregiver data. The account is an account corresponding to the assistant 120 and the care recipient 110H.

[0229] The items of caregiver data include an ID and emotion information. The ID is an ID that identifies caregiver 170H. The emotion information is information indicating whether the emotion towards caregiver 170H identified by the ID is positive or negative.

[0230] Care recipient 110H has different emotions (positive or negative) towards each of the multiple caregivers 170H. For example, if care recipient 110H is an elderly person, the elderly person may have positive or negative emotions towards caregiver 170H due to compatibility with the caregiver.

[0231] The control unit 210 of the server 130 acquires an image of care recipient 110H when each of the multiple caregivers 170H has a conversation with care recipient 110H. The image of care recipient 110H is an image captured by the camera 340 of care recipient terminal 110.

[0232] The emotion analysis unit 226 performs emotion analysis based on the acquired image of care recipient 110H using the same method as in the second embodiment. Through this emotion analysis, the emotion analysis unit 226 can analyze the emotion of care recipient 110H towards caregiver 170H, who is the other party. The result of the emotion analysis is, for example, one of happiness, sadness, anger, fear, surprise, disgust, and neutral as described above.

[0233] For example, assuming that the result of the emotion analysis indicates happiness. In this case, the control unit 210 records "positive" in the emotion item corresponding to the ID that identifies caregiver 170H, who is the other party in the conversation with care recipient 110H, in the staff table T3.

[0234] For example, assuming that the result of the emotion analysis indicates anger. In this case, the control unit 210 records "negative" in the emotion item corresponding to the ID that identifies caregiver 170H, who is the other party in the conversation with care recipient 110H, in the staff table T3.

[0235] For example, assume that the result of sentiment analysis indicates neutrality. In this case, the control unit 210 does not record information in the sentiment item corresponding to the ID that identifies the caregiver 170H, who is the person the care recipient 110H conversed with, in the staff table T3.

[0236] In the third embodiment, the control unit 210 performs conversation control using the staff table T3. FIG. 18 is a flowchart showing an example of the processing flow of the first example of the third embodiment. Assume that the caregiver 170H performs a login operation to log in to the server 130 using the staff terminal 170.

[0237] The staff terminal 170 transmits information on the login operation of the caregiver 170H (for example, ID and password). The server 130 receives the information on the login operation of the caregiver 170H (step S601).

[0238] The control unit 210 of the server 130 authenticates the caregiver 170H based on the received login operation information (step S602). As a result, the caregiver 170H is identified. The control unit 210 refers to the staff table T3 stored in the storage unit 211 (step S603).

[0239] The service providing unit 222 generates an avatar image of the caregiver 170H whose sentiment is positive among the items of the caregiver data in the staff table T3 (step S604). The image of each caregiver 170H is stored in the storage unit 211 in advance. The control unit 210 extracts the caregiver 170H whose sentiment is positive from the images of each caregiver 170H stored in the storage unit 211.

[0240] In the case of the staff table T3 in FIG. 17, the care staff 170H whose emotion corresponds positively is the care staff 170H with IDs "001" and "002". In this case, the control unit 210 extracts, for example, the care staff 170H corresponding to the ID with the smallest numerical value among the IDs of a plurality of care staff 170H whose emotion corresponds positively. The service providing unit 222 generates an avatar image of the care staff 170H based on the image of the care staff 170H corresponding to the ID "001" among the images of each care staff 170H stored in the storage unit 211.

[0241] The service providing unit 222 starts conversation control using the avatar image generated in step S604 (step S605). While the conversation control is being performed, the camera 340 of the care recipient terminal 110 captures the expression of the care recipient 110H. The care recipient terminal 110 transmits the captured image to the server 130.

[0242] The emotion analysis unit 226 performs emotion analysis on the image of the expression of the care recipient 110H received by the server 130 (step S606). The emotion analysis unit 226 determines whether a negative emotion has been detected as a result of the emotion analysis (step S607).

[0243] If a negative emotion is not detected, the control unit 210 determines No in step S607 and ends the flowchart in FIG. 18. If a negative emotion is detected, the control unit 210 determines Yes in step S607 and advances the process to step S608.

[0244] The control unit 210 refers to the staff table T3 stored in the storage unit 211 (step S608). The service providing unit 222 generates avatar images of other care staff 170H whose emotion is positive among the items of the care staff data in the staff table T3 (step S609).

[0245] In the case of the staff table T3 in FIG. 17, another caregiver 170H whose emotion corresponds positively is the caregiver 170H with the ID "002". Based on the image of the caregiver 170H corresponding to the ID "002" among the images of each caregiver 170H stored in the storage unit 211, the service providing unit 222 generates an avatar image of the caregiver 170H.

[0246] The service providing unit 222 starts conversation control using the avatar image generated in step S609 (step S610). As a result, the avatar image displayed on the touch panel 110A of the care recipient terminal 110 can be switched from the avatar image of the caregiver 170H showing negative emotion to the avatar image of the caregiver 170H corresponding to positive emotion.

[0247] The emotion analysis unit 226 determines whether negative emotion is detected for the avatar image of another caregiver 170H (step S611). The determination in step S611 is the same as the determination in step S607.

[0248] If negative emotion is not detected for the avatar image of another caregiver 170H, the control unit 210 ends the flowchart in FIG. 18. If negative emotion is detected for the avatar image of another caregiver 170H, the control unit 210 returns the process to step S607. In this case, the processes from step S608 to S610 are performed again.

[0249] As described above, among a plurality of caregivers 170H, conversation control using the avatar image of the caregiver 170H with whom the care recipient 110H has positive emotion can be realized. Thereby, the conversation of the care recipient 110H can be smoothly promoted.

[0250] (Second example) Next, a second example of the third embodiment will be described. In the second example of the third embodiment, the server 130 cooperates with the care record database 160 to generate an avatar image and perform conversation control. The server 130 performs conversation control with the care recipient 110H who operates the care recipient terminal 110 using the generated avatar image without going through the care staff 170H and the caregiver 120H.

[0251] The care record database 160 stores care records corresponding to the care recipient 110H. The care record is information recording the daily care situation regarding the care recipient 110H and includes information on conversations between the care recipient 110H and the care staff 170H. The care record is recorded by the care staff 170H.

[0252] For example, the care staff 170H uses the staff terminal 170 to input care records. The staff terminal 170 transmits information identifying the care recipient 110H and the information of the input care record to the care record database 160. The care record database 160 stores the information of the care record in association with the care recipient 110H. As a result, information regarding the care of the care recipient 110H accumulates in the care record database 160.

[0253] The care record is, for example, information such as "Greeted in the room at 7:00 am on August 1, 2023. Asked, 'You seem to be sleeping well lately. How about taking a walk?' and since the answer was 'Yes' with a smiling expression, recommended a walk", and is associated with the information of the care recipient 110H.

[0254] The above care record is information indicating a conversation between the care recipient 110H and the care staff 170H and includes information indicating the content of the question and information indicating whether the answer of the care recipient 110H is an affirmative answer or a negative answer.

[0255] The control unit 210 may generate a conversation database based on the information of a plurality of care records recorded in the care record database 160. For example, the control unit 210 may extract conversation information and information indicating whether it is an affirmative answer or a negative answer from the information of a plurality of care records, and generate a conversation database.

[0256] Alternatively, the control unit 210 may generate a conversation database using a learned model obtained by machine learning with the information of a plurality of care records recorded in the care record database 160 as training data.

[0257] FIG. 19A is a diagram showing an example of a conversation database in the third embodiment. Utterance IDs "101" to "104" are the contents uttered when the care recipient 110H is in a positive state. Utterance IDs "201" to "204" are the contents uttered when the care recipient 110H is in a negative state.

[0258] FIG. 19B is a diagram showing an example of a news database in the third embodiment. The news database T42 in FIG. 19B is a database that stores information about daily news. For example, information about daily news may be obtained from a predetermined server that distributes information about news, and the obtained information may be recorded in the news database T42.

[0259] FIG. 20 is a flowchart showing an example of the processing flow of a second example in the third embodiment. The control unit 210 acquires information on the care record corresponding to the care recipient 110H from the care record database 160 via the communication unit 212 (step S701).

[0260] The control unit 210 refers to the staff table T3 and generates an avatar image based on the images of the care staff 170H whose emotions correspond positively among the plurality of care staff 170H corresponding to the care recipient 110H (step S702).

[0261] When a plurality of pieces of information of the caregiver 170H whose emotion corresponds positively are recorded in the staff table T3, the control unit 210 may generate an avatar image based on the image of the caregiver 170H corresponding to the ID with the smallest numerical value, as in the first example of the present embodiment.

[0262] The control unit 210 determines whether the care recipient 110H is included in the captured image captured by the camera 340 of the care recipient terminal 110 (step S703). For example, the control unit 210 may determine Yes in step S703 when a person's face is included in the captured image and as a result of face recognition of the person's face, the face of the person and the face of the care recipient 110H are above a predetermined rate.

[0263] When the control unit 210 determines No in step S703, it is estimated that the care recipient 110H who is the target of conversation is not near the care recipient terminal 110, and thus the flowchart of FIG. 20 is terminated.

[0264] When the control unit 210 determines Yes in step S703, the process proceeds to step S704. The control unit 210 determines whether there is negative information about the care recipient 110H (step S704). For example, the control unit 210 acquires information on the most recent care record among the plurality of care record information about the care recipient 110H recorded in the care record database 160.

[0265] For example, assuming that the care record of "7:00 am on August 1, 2023" described above is the most recent care record. The information of the care record includes information of "smiling expression" but does not include negative information such as "angry expression". In this case, since there is no negative information about the care recipient 110H, the control unit 210 determines No in step S704 and proceeds with the process to step S706.

[0266] On the one hand, when the most recent care record contains negative information such as an "angry expression", the control unit 210 determines Yes in step S704. When the control unit 210 determines Yes in step S704, the process proceeds to step S705.

[0267] The service providing unit 222 starts conversation control using the negative utterance information (step S705). The negative utterance information is, for example, the information of the utterance content corresponding to the utterance IDs "201" to "204" in the conversation database T41 of FIG. 19.

[0268] When the control unit 210 determines that there is no negative information about the care recipient 110H, the process proceeds to step S706. The control unit 210 determines whether there is positive information about the care recipient 110H (step S706).

[0269] For example, when the information of the most recent care record among the information of a plurality of care records about the care recipient 110H recorded in the care record database 160 contains positive information about the care recipient 110H, the control unit 210 determines Yes in step S706.

[0270] For example, the most recent care record of "7:00 am on August 1, 2023" described above contains the information of a "smiling expression". Based on the positive information of the "smiling expression", the service providing unit 222 determines Yes in step S706 (step S707).

[0271] The service providing unit 222 starts conversation control using the positive utterance information (step S705). The positive utterance information is, for example, the information of the utterance content corresponding to the utterance IDs "101" to "104" in the conversation database T41 of FIG. 19A.

[0272] If neither positive nor negative emotion information is included in the most recent care record, the control unit 210 determines No in step S706. The service providing unit 222 starts conversation control using other utterance information (step S708).

[0273] The other utterance information is information about daily news recorded in the news database T42 in FIG. 19B. The service providing unit 222 controls the assisted person terminal 110 to utter the information of the utterance content in order from the utterance ID "301" in the news database T42 in FIG. 19B.

[0274] Steps S705, S707, and S708 are automatically uttered by the service providing unit 222. Therefore, communication by conversation can be performed between the server 130 and the assisted person 110H without the intervention of the caregiver 120H. As a result, since the assisted person 110H has more opportunities to have conversations, it is possible to suppress the decline of the cognitive function of the assisted person 110H. <Fourth Embodiment>

[0275] Next, with reference to FIG. 7, the fourth embodiment will be described. As described above, in step S100, the caregiver 120H performs a login operation on the caregiver terminal 120. At this time, the control unit 310 of the terminal 300 may display a message for asking for the date of birth and gender on the touch panel 120A.

[0276] When the caregiver 120H inputs the date of birth and gender to the touch panel 120A, the control unit 310 of the terminal 300 transmits the input date of birth and gender information to the server 130 via the communication unit 330.

[0277] In step S108, the assisted person terminal 110 transmits the time information to the server 130. As described above, the time information is information indicating the time that the assisted person 110H can recognize.

[0278] The control unit 210 of the server 130 recognizes the gender of the caregiver 120H based on the gender information received from the caregiver terminal 120. Further, the control unit 210 calculates the age of the caregiver 120H that can be recognized by the care recipient 110H based on the time (e.g., year) indicated by the time information received from the care recipient terminal 120 and the date of birth information received from the caregiver terminal 120. The age of the caregiver 120H that can be recognized by the care recipient 110H is also the age of the avatar described above.

[0279] The service providing unit 222 may convert the frequency of the voice acquired from the caregiver terminal 120 into a frequency corresponding to the recognized gender of the caregiver 120H and the calculated age of the avatar, transmit it to the care recipient terminal 110, and perform conversation control using the avatar image in step S110. Thereby, conversation control using a voice that is presumed to be recognizable by the care recipient 110H can be realized.

[0280] There is a correlation between the voice emitted by a human and their age and gender. The service providing unit 222 may perform the above frequency conversion based on the correlation.

[0281] Further, the control unit 210 of the server 130 may estimate the age and gender of the caregiver 120H based on an image of the caregiver 120H's face. For example, assume that the camera 340 of the caregiver terminal 120 has transmitted an image of the caregiver 120H's face to the server 130.

[0282] The control unit 210 may estimate the age and gender of the caregiver 120H by inputting the above image of the caregiver 120H's face into a learned model obtained by performing machine learning using data in which an image of a human face is associated with age and gender as training data. In this case, it is not necessary to input the date of birth and gender of the caregiver 120H.

[0283] The control unit 210 may calculate the age (age of the avatar) of the caregiver 120H that can be recognized by the care recipient 110H based on the time (e.g., year) indicated by the time information received from the care recipient terminal 120 and the estimated age.

[0284] Then, the service providing unit 222 may convert the frequency of the voice acquired from the caregiver terminal 120 into a frequency corresponding to the estimated gender of the caregiver 120H and the calculated age of the avatar, transmit it to the care recipient terminal 110, and perform conversation control using the avatar image in step S110. As described above, since conversation control using a voice estimated to be recognizable by the care recipient 110H can be realized, the conversation between the caregiver 120H and the care recipient 110H can be promoted.

[0285] Those skilled in the art will easily understand that many modifications can be made without substantially departing from the novel matters and effects of each of the above embodiments. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, in the specification or drawings, a term described at least once together with a broader or synonymous different term can be replaced with that different term at any location in the specification or drawings. Also, all combinations of each embodiment and modification are included within the scope of the present disclosure. Further, the configurations and operations of each device, etc. are not limited to those described in each embodiment, and various modified implementations are possible.

[0286] Regarding the various examples described above, the following appendices are disclosed. (Appendix 1) A storage unit that stores a plurality of images at different times including an image of a caregiver who assists a care recipient; A time information acquisition unit that acquires from the first terminal time information indicating a past time received by the first terminal of the care recipient; A service providing unit that displays an image of the caregiver corresponding to the time indicated by the time information on the screen of the first terminal and provides a service that promotes conversation between the care recipient and the caregiver; An information processing apparatus comprising: (Appendix 2) The service providing unit generates an avatar image of the image of the caregiver corresponding to the time indicated by the time information, and provides a service that promotes conversation between the care recipient and the caregiver using the generated avatar image, the information processing apparatus according to Appendix 1. (Appendix 3) The service providing unit is the information processing apparatus according to Appendix 2, which acquires the speech content of the caregiver from the second terminal of the caregiver and operates the mouth of the avatar image according to the acquired speech content. (Appendix 4) The information processing apparatus according to Appendix 1 further includes a display control unit that sequentially displays the plurality of images on the screen of the first terminal. The time information acquisition unit acquires, as the time information, an image for which a designation has been received from among the plurality of images including the caregiver that are sequentially displayed. The service providing unit is the information processing apparatus according to Appendix 1, which provides a service of displaying the image for which the designation has been received on the screen of the first terminal to promote conversation between the care recipient and the caregiver. (Appendix 5) The information processing apparatus according to Appendix 1 further includes a display control unit that displays the time information on the screen of the second terminal of the caregiver. (Appendix 6) The information processing apparatus according to Appendix 2 further includes a biological information acquisition unit that acquires the biological information from a measuring device that measures the biological information of the care recipient. The service providing unit is the information processing apparatus according to Appendix 2, which changes the avatar image of the caregiver to be displayed on the screen of the first terminal according to the biological information. (Appendix 7) The biological information is sleep information regarding the sleep of the care recipient. The information processing apparatus according to Appendix 6 is one in which it is evaluated whether the cognitive function of the care recipient has recovered or regressed based on the sleep information. (Appendix 8) The information processing apparatus according to Appendix 6 further includes a display control unit that displays, on the screen of the second terminal of the caregiver, information indicating the result of the evaluation indicating whether the cognitive function of the care recipient has recovered or regressed. (Appendix 9) The information processing apparatus further includes a communication unit. The communication unit acquires exercise information regarding the exercise performed by the care recipient from an exercise device. The service providing unit is the information processing apparatus according to Appendix 2 that changes the expression of the avatar image according to the acquired motion information. (Appendix 10) An image acquisition unit that acquires an image of the face of the care recipient from an imaging device that captures the face of the care recipient, An emotion analysis unit that performs emotion analysis on whether the care recipient shows a positive emotion or a negative emotion based on the acquired image of the face of the care recipient, and further includes: The service providing unit is the information processing apparatus according to Appendix 2 that changes the avatar image of the caregiver to be displayed on the screen of the first terminal according to the result of the emotion analysis. (Appendix 11) The information processing apparatus according to Appendix 10, wherein it is evaluated whether the cognitive function of the care recipient has recovered or regressed based on the result of the emotion analysis. (Appendix 12) The selected image of the caregiver is the avatar image of the caregiver at the time indicated by the time information, The service providing unit is the information processing apparatus according to Appendix 2 that operates the avatar image with reference to a conversation database that stores conversation patterns. (Appendix 13) An image acquisition unit that acquires an image of the face of the care recipient from an imaging device that captures the face of the care recipient, An emotion analysis unit that performs emotion analysis on whether the care recipient shows a positive emotion or a negative emotion based on the acquired image of the face of the care recipient, and further includes: The service providing unit generates an avatar image of the image of the caregiver who shows the most positive emotion in the result of the emotion analysis among a plurality of caregivers who care for the care recipient, and uses the generated avatar image to provide a service that promotes conversation between the care recipient and the caregiver. The information processing apparatus according to Appendix 1. (Appendix 14) An information processing method having a storage unit that stores a plurality of images at different times including a caregiver who assists a care recipient, Obtain period information indicating a past period received by the first terminal of the assisted person from the first terminal, Display an image of the caregiver corresponding to the period indicated by the period information on the screen of the first terminal, and provide a service that promotes conversation between the assisted person and the caregiver. Information processing method.

Explanation of Signs

[0287] 100…System, 110…Assisted person terminal, 120…Caregiver terminal, 130…Server, 210…Control unit, 211…Storage unit, 212…Communication unit, 221…Period information acquisition unit, 222…Service provision unit, 223…Display control unit, 224…Biological information acquisition unit, 225…Image acquisition unit, 226…Emotion analysis unit, 300…Terminal, 310…Control unit, 320…Storage unit, 330…Communication unit, 340…Camera, 350…Microphone, 360…Speaker, 270…Touch panel

Claims

1. A storage unit that stores a plurality of images taken at different times, including an assistant who assists a person in need of assistance; A time information acquisition unit that acquires, from the first terminal of the person in need of assistance, time information indicating a past time received by the first terminal; A service providing unit that provides a service of displaying, on the screen of the first terminal, an image of the assistant corresponding to the time indicated by the time information to prompt a conversation between the person in need of assistance and the assistant; An information processing apparatus comprising the above.

2. The service providing unit generates an avatar image of the assistant corresponding to the time indicated by the time information, and uses the generated avatar image to provide a service of prompting a conversation between the person in need of assistance and the assistant. The information processing apparatus according to Claim 1.

3. Further comprising a display control unit that displays the plurality of images on the screen of the first terminal in chronological order, The time information acquisition unit acquires, as the time information, an image for which a designation has been received among the plurality of images including the assistant that are displayed in chronological order, The service providing unit provides a service of displaying the designated image on the screen of the first terminal to prompt a conversation between the person in need of assistance and the assistant. The information processing apparatus according to Claim 1.

4. Further comprising a biometric information acquisition unit that acquires biometric information from a measuring device that measures the biometric information of the person in need of assistance, The service providing unit changes the avatar image of the assistant to be displayed on the screen of the first terminal according to the biometric information. The information processing apparatus according to Claim 2.

5. An image acquisition unit that acquires an image of the face of the person in need of assistance from an imaging device that captures the face of the person in need of assistance, Further comprising an emotion analysis unit that performs emotion analysis as to whether the person in need of assistance is showing a positive emotion or a negative emotion based on the acquired image of the face of the person in need of assistance, The service providing unit changes the avatar image of the assistant to be displayed on the screen of the first terminal according to the result of the emotion analysis. The information processing apparatus according to Claim 2.

6. An information processing method having a storage unit that stores a plurality of images taken at different times, including an assistant who assists a person in need of assistance, comprising: acquiring, from the first terminal of the person in need of assistance, time information indicating a past time received by the first terminal; A service is provided that displays an image of the caregiver corresponding to the time indicated by the time information on the screen of the first terminal to prompt a conversation between the care recipient and the caregiver. Information processing method.

Images