Estimation system, estimation device, and program

Abstract

This disclosure provides an estimation system that can estimate user attributes for each refrigerator user. [Solution] The estimation system in this disclosure comprises a first detection unit that detects a user who has placed food into or taken out of a refrigerator, which is equipped with a camera that photographs the refrigerator from the front and above, and a first estimation unit that estimates the attributes of the user detected by the first detection unit based on information obtained from the photographic results of the camera.

Description

Technical Field

[0001] The present disclosure relates to an estimation system, an estimation device, and a program.

Background Art

[0002] Patent Document 1 discloses a technique for estimating the attributes of a user of a refrigerator using a learned model generated by machine learning.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides an estimation system, an estimation device, and a program that can estimate the attributes of a user for each user of a refrigerator.

Means for Solving the Problems

[0005] The estimation system in the present disclosure includes a first estimation unit that estimates a user who has stocked or removed food from the refrigerator on which a photographing device for photographing the refrigerator from the front upper side of the refrigerator is installed, and a second estimation unit that estimates the attributes of the user estimated by the first estimation unit based on the information obtained from the photographing result of the photographing device.

[0006] Further, the estimation device in the present disclosure includes a first estimation unit that estimates a user who has stocked or removed food from the refrigerator on which a photographing device for photographing the refrigerator from the front upper side of the refrigerator is installed, and a second estimation unit that estimates the attributes of the user estimated by the first estimation unit based on the information obtained from the photographing result of the photographing device.

[0007] Furthermore, the program in this disclosure functions as a processor comprising: a first estimation unit that estimates a user who has placed food into or taken out of a refrigerator, which is equipped with a camera that photographs the refrigerator from the front and above; and a second estimation unit that estimates the attributes of the user estimated by the first estimation unit based on information obtained from the photographic results of the camera. [Effects of the Invention]

[0008] The estimation system, estimation device, and program described herein can estimate user attributes for each refrigerator user. [Brief explanation of the drawing]

[0009] [Figure 1] Diagram showing the configuration of the estimation system in Embodiment 1. [Figure 2] Diagram showing the configuration of the imaging device in Embodiment 1. [Figure 3] Diagram showing the configuration of the terminal device in Embodiment 1 [Figure 4] Diagram showing the configuration of the server device in Embodiment 1 [Figure 5] A diagram showing an example of records stored in the management DB in Embodiment 1. [Figure 6] Flowchart showing the operation of the server device in Embodiment 1 [Figure 7] Flowchart showing the operation of the terminal device in Embodiment 1 [Figure 8] A diagram showing an example of the first notification screen in Embodiment 1. [Figure 9] A diagram showing an example of the second notification screen in Embodiment 1. [Figure 10] Flowchart showing the operation of the server device in Embodiment 1 [Figure 11] A diagram showing the configuration of the server device in another embodiment. [Figure 12] A diagram showing an example of records stored in the management database in another embodiment. [Modes for carrying out the invention]

[0010] (Knowledge, etc., underlying the present disclosure) When the inventors arrived at the idea of the present disclosure, there was a technology for estimating the attributes of users of a refrigerator as disclosed in Patent Document 1. However, in the technology disclosed in Patent Document 1, although the attributes of the users who own the refrigerator can be estimated, the inventors discovered a problem that the attributes of users other than the users who own the refrigerator, such as the family members of the users who own the refrigerator, cannot be estimated. In order to solve this problem, the inventors have come to constitute the subject matter of the present disclosure. Therefore, the present disclosure provides an estimation system, an estimation device, and a program that can estimate the attributes of users for each user of a refrigerator.

[0011] Hereinafter, embodiments will be described in detail with reference to the drawings. However, there may be cases where a more detailed description than necessary is omitted. For example, there may be cases where a detailed description of already well-known matters or a redundant description of substantially the same configuration is omitted. It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0012] (Embodiment 1) [1-1. Configuration] [1-1-1. Configuration of the estimation system] FIG. 1 is a diagram showing the configuration of an estimation system 1 in Embodiment 1. The estimation system 1 is a system for estimating the attributes of a user P of the refrigerator 2. The attributes of the user P refer to the characteristics and properties of the user P. In other words, the attributes of the user P indicate what kind of person the user P is. Examples of the attributes of the user P include a health-oriented tendency, a night type, a morning type, a person who eats out, a person who eats at home, a person who stocks up on weekends, a person who purchases food by home delivery, a person who goes to a physical store to purchase food, and a frail state.

[0013] The estimation system 1 includes a refrigerator 2. The refrigerator 2 includes a main cabinet 20 with an open front. In the main cabinet 20, a refrigerating chamber 21, an ice-making chamber 22, a fresh freezing chamber 23, a freezing chamber 24, and a vegetable chamber 25 are formed. These storage chambers are cooled by cold air. A rotary door 21A is installed at the opening in the front of the refrigerating chamber 21. The door 21A includes a rotary right door 21B and a rotary left door 21C. Drawers 22A, 23A, 24A, 25A capable of accommodating food are respectively installed in each of the ice-making chamber 22, the fresh freezing chamber 23, the freezing chamber 24, and the vegetable chamber 25. Each of the drawers 22A to 25A has an open top surface in the installed state. The door 21A and the drawers 22A to 25A are examples of "opening and closing bodies".

[0014] Hereinafter, when the ice-making chamber 22, the fresh freezing chamber 23, the freezing chamber 24, and the vegetable chamber 25 are not distinguished, they are denoted by the symbol "26" and expressed as "drawer storage chamber 26". Also, hereinafter, when the drawers 22A, 23A, 24A, 25A are not distinguished, they are denoted by the symbol "26A" and expressed as "drawer 26A".

[0015] The estimation system 1 includes a photographing device 3. The photographing device 3 is detachably provided on the top surface 20A of the main cabinet 20 of the refrigerator 2. The photographing device 3 photographs the refrigerator 2 from the front upper part of the refrigerator 2 in the state of being installed in the refrigerator 2 (hereinafter, appropriately expressed as "installed state").

[0016] The estimation system 1 includes a terminal device 4. The terminal device 4 is a portable computer such as a smartphone. An application program for performing various notifications to the user P is installed in the terminal device 4, and it communicates with the server device 5 by the function of this application program. In the following description, this application program is referred to as "notification app" and denoted by the symbol "411". The server device 5 is an example of an "estimation device".

[0017] In Figure 1, a solid line represents a user P who is at home, and a dotted line represents a user P who has left home H. When terminal device 4 is used by user P who is at home, it communicates with server device 5 via communication device 6, or without communication device 6. Also, when terminal device 4 is used by user P who has left home H and cannot establish a communication connection with communication device 6, it communicates with server device 5 without communication device 6.

[0018] The communication device 6 connects to a network NW, which is composed of a public telephone network, and communicates with the server device 5 via the network NW. The communication device 6 functions as an interface device for connecting the refrigerator 2, the camera 3, and the terminal device 4 to the network NW. The communication device 6 establishes a local network at home H.

[0019] The estimation system 1 includes a server device 5. Server device 5 is a device that processes information using the imaging device 3 and terminal device 4 as clients. Server device 5 is connected to the network NW and communicates with each device connected to the network NW. In each diagram, server device 5 is represented by a single block, but this does not necessarily mean that server device 5 is composed of a single device.

[0020] [1-1-2. Configuration of the imaging device] Next, we will explain the configuration of the imaging device 3. Figure 2 shows the configuration of the imaging device 3.

[0021] The imaging device 3 comprises an imaging control device 30, a wide-angle camera 31, a narrow-angle camera 32, a first distance measuring sensor 33, a second distance measuring sensor 34, a thermopile 35, and a wireless communication module 36.

[0022] Before describing the imaging control device 30, we will first describe each device connected to the imaging control device 30. The wide-angle camera 31 is a wider-angle camera than the narrow-angle camera 32, and in its installed state, it photographs the refrigerator compartment 21 from the front and above of the refrigerator 2. The shooting range of the wide-angle camera 31, when viewed from the front of the refrigerator 2 in its installed state, includes the door pockets of the right door 21B and left door 21C when they are fully open, and the opening of the refrigerator compartment 21. Also, when viewed from the side of the refrigerator 2 in its installed state, the shooting range of the wide-angle camera 31 includes the open right door 21B and left door 21C, the front end of the shelf 21D inside the refrigerator compartment 21, and the opening of the refrigerator compartment 21.

[0023] The narrow-angle camera 32 has a narrower field of view than the wide-angle camera 31, and in its installed state, it photographs the drawer 26A from the front and above of the refrigerator 2. The shooting range of the narrow-angle camera 32, when viewed from the front of the refrigerator 2 in its installed state, includes the area including the drawer 26A when viewed from the side of the refrigerator 2 in its fully extended state.

[0024] The first distance measuring sensor 33 is a distance measuring sensor for detecting the opening and closing of the right door 21B. The first distance measuring sensor 33 is, for example, an optical sensor that emits infrared light. The first distance measuring sensor 33 detects the distance between itself and the right door 21B and outputs a detected value indicating the detected distance. The sensing target of the first distance measuring sensor 33 is the part of the right door 21B above the upper door gasket.

[0025] The second distance measuring sensor 34 is a sensor for detecting the opening and closing of the left door 21C. The second distance measuring sensor 34 is an optical sensor that emits infrared light, for example. The second distance measuring sensor 34 detects the distance between itself and the left door 21C and outputs a detected value indicating the detected distance. The sensing target of the second distance measuring sensor 34 is the part of the left door 21C above the upper door gasket.

[0026] The thermopile 35 is a sensor for detecting the opening and closing of drawer 26A. In its installed state, the thermopile 35 is positioned so that it can detect the opening and closing of drawer 26A from the front upper part of refrigerator 2. The thermopile 35 outputs a detected value indicating the detected temperature.

[0027] The wireless communication module 36 is equipped with hardware such as communication circuits and connectors that conform to a predetermined wired communication standard, and communicates with the server device 5 according to the control of the shooting control device 30.

[0028] The image capture control device 30 includes an image capture processor 300 such as a CPU (Central Processing Unit), an image capture memory 310, and interface circuits to which other devices and sensors are connected. In this embodiment, an example is shown in which a wide-angle camera 31, a narrow-angle camera 32, a first distance measuring sensor 33, a second distance measuring sensor 34, a thermopile 35, and a wireless communication module 36 are connected to the image capture control device 30. Note that the devices connected to the image capture control device 30 are not limited to these devices, and other devices such as LEDs may be connected. Note that the image capture control device 30 may be composed of multiple circuit boards.

[0029] The imaging memory 310 is a storage device that stores programs and data executed by the imaging processor 300. The imaging memory 310 is composed of non-volatile storage devices such as ROM (Read Only Memory). The imaging memory 310 also includes volatile storage devices that constitute the work area of ​​the imaging processor 300, such as RAM (Random Access Memory). The imaging memory 310 stores data processed by the imaging processor 300, control programs 311 executed by the imaging processor 300, imaging device ID (Identification) 312, and the like. The imaging device ID 312 is identification information that uniquely identifies imaging device 3.

[0030] The shooting processor 300 functions as a shooting communication control unit 301, an open / close determination unit 302, a shooting control unit 303, a recording control unit 304, and a generation unit 305 by reading and executing the control program 311.

[0031] The camera communication control unit 301 communicates with the server device 5 via the wireless communication module 36.

[0032] The open / closed determination unit 302 determines whether the right door 21B is open or closed based on the detection value output by the first distance measuring sensor 33. If the open / closed determination unit 302 receives a detection value different from the detection value corresponding to the closed state, it determines that the right door 21B is open. On the other hand, if the open / closed determination unit 302 receives a detection value corresponding to the closed state, it determines that the right door 21B is closed.

[0033] The opening / closing determination unit 302 determines whether the left door 21C is open or closed based on the detection value output by the second distance measuring sensor 34, similar to the determination for the right door 21B.

[0034] The open / closed determination unit 302 determines whether the drawer 26A is open or closed based on the detected value output by the thermopile 35. If the open / closed determination unit 302 receives a detected value from the thermopile 35 indicating a temperature below a predetermined temperature, it determines that the drawer 26A is open. On the other hand, if the open / closed determination unit 302 receives a detected value from the thermopile 35 indicating a temperature above a predetermined temperature, it determines that the drawer 26A is closed.

[0035] If the opening / closing determination unit 302 determines that the door 21A or drawer 26A is open, the shooting control unit 303 instructs the wide-angle camera 31 and the narrow-angle camera 32 to start shooting. Conversely, if the opening / closing determination unit 302 determines that the door 21A or drawer 26A is closed, the shooting control unit 303 instructs the wide-angle camera 31 and the narrow-angle camera 32 to stop shooting.

[0036] When the open / close determination unit 302 determines that the door 21A or drawer 26A is open, the recording control unit 304 starts recording the shooting results of the wide-angle camera 31 and the narrow-angle camera 32. That is, the recording control unit 304 starts recording recording data 313 to the shooting memory 310. The recording data 313 includes first recording data showing the shooting results of the wide-angle camera 31 and second recording data showing the shooting results of the narrow-angle camera 32. The recording data 313 also includes shooting time information indicating the shooting time. The shooting time indicated by this shooting time information indicates at least the shooting start date and time when each camera started shooting and the shooting end date and time when each camera finished shooting.

[0037] When the open / close determination unit 302 determines that the door 21A and the drawer 26A are closed, the recording control unit 304 terminates recording of the shooting results from the wide-angle camera 31 and the narrow-angle camera 32.

[0038] The generation unit 305 generates the first upload data D1. The first upload data D1 is data to be uploaded to the server device 5 and includes the shooting device ID 312 and the recorded data 313. The generation unit 305 generates the first upload data D1 each time the recording control unit 304 finishes recording. When the generation unit 305 generates the first upload data D1, the shooting communication control unit 301 transmits the generated first upload data D1 to the server device 5.

[0039] [1-1-3. Terminal Device Configuration] Next, we will describe the configuration of terminal device 4. Figure 3 shows the configuration of terminal device 4.

[0040] The terminal device 4 comprises a terminal control device 40, a terminal communication module 41, a touch panel 42, and a GNSS (Global Navigation Satellite System) receiver 43.

[0041] Before describing the terminal control device 40, we will first describe the terminal communication module 41 and the touch panel 42. The terminal communication module 41 is equipped with communication circuits and connectors that conform to a predetermined communication standard, and communicates with the server device 5 according to the control of the terminal control device 40. The touch panel 42 comprises a display panel and a touch sensor that is superimposed on or integrated with the display panel. The GNSS receiver 43 is a device that detects the current location of the terminal device 4 and receives positioning signals transmitted from multiple positioning satellites. The GNSS receiver 43 outputs current location information indicating the current location of the terminal device 4 to the terminal control device 40.

[0042] The terminal control device 40 includes a terminal processor 400 such as a CPU, a terminal memory 410, and interface circuits to which other devices and sensors are connected. In this embodiment, an example is shown in which a terminal communication module 41, a touch panel 42, and a GNSS receiver 43 are connected to the terminal control device 40. Note that the devices connected to the terminal control device 40 are not limited to these devices, and other devices such as LEDs may also be connected.

[0043] The terminal memory 410 is a storage device that stores programs and data executed by the terminal processor 400. The terminal memory 410 is composed of non-volatile storage devices such as ROM. The terminal memory 410 also includes volatile storage devices that constitute the work area of ​​the terminal processor 400, such as RAM. The terminal memory 410 stores data processed by the terminal processor 400, as well as notification applications 411 executed by the terminal processor 400, the camera ID 312, and family relationship information 412. The family relationship information 412 will be described later.

[0044] The terminal processor 400 functions as an application execution unit 401 by reading and executing the notification application 411. The application execution unit 401 communicates with the server device 5 via the terminal communication module 41. The application execution unit 401 generates second upload data D2 at predetermined intervals and transmits the generated second upload data D2 to the server device 5. The second upload data D2 is data to be uploaded to the server device 5 and includes the camera ID 312, family relationship information 412, and current location information output by the GNSS receiver 43. The application execution unit 401 displays a notification screen TG (see Figure 8, etc.) on the touch panel 42 to display notifications. The application execution unit 401 receives various inputs from user P via the touch panel 42. The application execution unit 401 is an example of a "notification unit".

[0045] [1-1-4. Server Device Configuration] Next, we will describe the configuration of server device 5. Figure 4 shows the configuration of server device 5.

[0046] The server device 5 comprises a server control device 50 and a server communication module 51.

[0047] Before describing the server control device 50, we will describe the server communication module 51. The server communication module 51 is equipped with communication circuits and connectors that conform to a predetermined communication standard, and communicates with the imaging device 3 and the terminal device 4 in accordance with the control of the server control device 50.

[0048] The server control device 50 includes a server processor 500 such as a CPU, server memory 520, and interface circuits to which other devices and sensors are connected. In this embodiment, an example is shown in which a server communication module 51 is connected to the server control device 50. The Server Processor 500 is an example of a "processor".

[0049] The server memory 520 is a storage device that stores programs and data executed by the server processor 500. The server memory 520 is composed of non-volatile storage devices such as ROM. The server memory 520 also includes volatile storage devices that constitute the work area of ​​the server processor 500, such as RAM. The server memory 520 stores data processed by the server processor 500, control programs 521 executed by the server processor 500, management database 522, etc. Control program 521 is an example of a "program".

[0050] The management DB 522 has one record R for each imaging device 3 that communicates with the server device 5.

[0051] Figure 5 shows an example of a record R stored in the management DB522. One record R has imaging device ID 312.

[0052] Furthermore, each record R includes, for each user P who makes up the family using the refrigerator 2 on which the camera 3 is installed, gender information, age information, family relationship information 412, height information, time spent at home information, terminal location information, terminal communication information, preference information, attribute information, food received information, food taken out information, movement information, and storage method information.

[0053] Gender information is information that indicates gender. Age information is information that indicates age. Family relationship information 412 is information that indicates family relationships. Family relationships refer to the interpersonal relationships between family members. Height information is information that indicates height. Home time information indicates the amount of time spent at home during a day. Terminal location information indicates the current location of the terminal device 4 used by the corresponding user P. While the current location indicated by the terminal location information may be shown as latitude and longitude, this embodiment exemplifies a case where it is shown converted to a pre-registered location name. Terminal communication information is information used to communicate with the terminal device 4 used by the corresponding user P. For example, terminal communication information is address information. Preference information indicates the types of food that the corresponding user P prefers. Attribute information is information that indicates the attributes of user P.

[0054] In this embodiment, gender information, age information, family relationship information 412, height information, time spent at home information, and preference information are information related to user P that has been registered in advance by user P.

[0055] The stored food information pertains to the food stored in refrigerator 2, and for each food item stored in refrigerator 2, it includes a set of information such as food type, storage area, and opening / closing time. Food type information is information that indicates the type of food. Storage area information indicates the storage area where the food is stored. The opening and closing time information indicates the time it takes for door 21A or drawer 26A to be opened and then closed when food is being stored.

[0056] The retrieval food information is information relating to food retrieved from refrigerator 2, and for each food retrieved from refrigerator 2, it includes a set of food type information and opening / closing time information. Note that the sets included in the retrieval food information are deleted from the retrieval food information after a predetermined period has elapsed since they were included. Furthermore, the opening / closing time information included in the retrieval food information indicates the opening / closing time from when the door 21A or drawer 26A is opened until it is closed when the food is retrieved.

[0057] The motion information refers to information indicating the movements of user P when receiving or retrieving food. The movements indicated by the motion information may be characteristic movements obtained by analyzing the video data 313. Alternatively, the movements indicated by the motion information may be movements that user P has registered in advance.

[0058] The storage method information indicates how the food is stored. The storage method indicated by the storage method information may be a characteristic storage method obtained by analyzing the video data 313. Alternatively, the storage method indicated by the storage method information may be a storage method that user P has registered in advance.

[0059] The server processor 500 functions as a server communication control unit 501, a food type detection unit 502, an in / out detection unit 503, a storage area detection unit 504, a user estimation unit 505, an attribute estimation unit 506, a lifestyle change estimation unit 507, a lifestyle estimation unit 508, and an update unit 509 by reading and executing a control program 521 stored in the server memory 520. The server communication control unit 501 is an example of a "notification instruction unit". The user estimation unit 505 is an example of a "first estimation unit". The attribute estimation unit 506 is an example of a "second estimation unit". The lifestyle change estimation unit 507 is an example of a "third estimation unit". The lifestyle estimation unit 508 is an example of a "fourth estimation unit".

[0060] [1-1-4-1. Server Communication Control Unit] The server communication control unit 501 communicates with the imaging device 3 and the terminal device 4 via the server communication module 51.

[0061] [1-1-4-2. Food type detection unit] The food type detection unit 502 detects food items to be placed in or taken out of the refrigerator 2 from the recorded data 313 included in the first uploaded data D1 received by the server communication control unit 501. The food type detection unit 502 detects food items to be placed in or taken out of the refrigerator 2 based on features such as shape and color. Next, the food type detection unit 502 detects the type of food item that was detected. The food type detection unit 502 detects the type of food item that was detected based on features such as shape and color. The food type detection unit 502 may also detect the type of food item detected by AI (Artificial Intelligence). Alternatively, the food type detection unit 502 may detect the type of food item by recognizing the characters on the label printed on the food item's packaging or by recognizing the barcode.

[0062] [1-1-4-3. Inbound / Outbound Detection Unit] The in / out detection unit 503 detects whether the food detected by the food type detection unit 502 is food that has been put into the refrigerator 2 or food that has been taken out of the refrigerator 2, based on the recorded data 313 that the food type detection unit 502 has processed. This detection is performed based on the direction of movement of the food in the multiple captured images shown in the recorded data 313. That is, if the direction of movement of the detected food is toward the refrigerator 2, the in / out detection unit 503 detects that the detected food is food that has been put into the refrigerator 2. Also, if the direction of movement of the detected food is toward the refrigerator 2, the in / out detection unit 503 detects that the detected food is food that has been taken out of the refrigerator 2.

[0063] [1-1-4-4. Accommodation Area Detection Unit] The storage area detection unit 504 detects, based on the recorded data 313 processed by the food type detection unit 502, the storage area where the food detected by the food type detection unit 502 was stored, or the storage area from which the food was taken out. When the food type detection unit 502 detects food from the first recorded data, the storage area detection unit 504 detects whether the food detected by the food type detection unit 502 is stored in or out of the right door 21B, the left door 21C, or the shelf 21D. The storage area detection unit 504 detects whether the food is stored in or out of the right door 21B, the left door 21C, or the shelf 21D based on the direction of movement of the food in the multiple captured images shown in the first recorded data. Next, if the storage area detection unit 504 detects that the food is stored in or out of the shelf 21D, it detects whether it is on the left or right side of which shelf 21D it is on, counting from the top or bottom. The detection of whether it is on the left or right side of which shelf 21D it is on is performed based on the direction of movement of the food in the multiple captured images shown in the first recorded data and the detection of the shelf 21D on which the food was placed or removed.

[0064] When the food type detection unit 502 detects food from the second recorded data, the storage area detection unit 504 detects that the destination for the food detected by the food type detection unit 502 is a drawer storage compartment 26. Next, the storage area detection unit 504 detects the type of drawer storage compartment 26 to which the detected food is stored or shipped. In this embodiment, each of the drawer storage compartments 26 is marked to identify the type of drawer storage compartment 26. The storage area detection unit 504 detects the type of drawer storage compartment 26 based on the markers captured in the images included in the second recorded data.

[0065] [1-1-4-5. User Estimation Section] When the server communication control unit 501 receives the first upload data D1, the user estimation unit 505 estimates the user P who placed or took food out of the refrigerator 2. In estimating user P, the user estimation unit 505 identifies the record R of the camera device ID 312 included in the first upload data D1 from the management DB 522.

[0066] The following are several examples of estimation methods used by the user estimation unit 505. <Example 1> In Example 1, the user estimation unit 505 estimates the user P based on the shooting time when the shooting device 3 took the picture. The user estimation unit 505 estimates user P based on the recording time information contained in the recording data 313 of the first upload data D1 received by the server communication control unit 501 and the identified record R. For example, suppose the identified record R is the record R in Figure 5, and the recording time information indicates that it is in the 8 o'clock hour. According to the time spent at home information of record R in Figure 5, the only user P who is at home in the 8 o'clock hour is user P whose family relationship is "mother". Therefore, in this example, the user estimation unit 505 estimates that user P who used refrigerator 2 is user P whose family relationship is "mother".

[0067] <Example 2> In Example 2, the user estimation unit 505 estimates the user P based on the storage area within the refrigerator 2 where the food was stored or taken out. The user estimation unit 505 estimates the user P based on the accommodation area detected by the accommodation area detection unit 504. For example, suppose the identified record R is the record R in Figure 5, and the storage area detection unit 504 detects that the storage area is "the left side of the first shelf 21D from the top." According to the height information of record R in Figure 5, the tallest user P is user P whose family relationship is "father." Therefore, in this example, the user estimation unit 505 estimates that user P who used the refrigerator 2 is user P whose family relationship is "father." For example, suppose the identified record R is the record R in Figure 5, and the storage area detection unit 504 detects that the storage area is "the left side of the fourth shelf 21D from the top." According to the stored food information of record R in Figure 5, the shortest user P is user P whose family relationship is "child." Therefore, in this example, the user estimation unit 505 estimates that user P who used refrigerator 2 is user P whose family relationship is "child."

[0068] <Example 3> In Example 3, the user estimation unit 505 estimates the user P based on the type of food detected by the food type detection unit 502. For example, suppose the identified record R is the record R in Figure 5, and the food type detection unit 502 detects that the food type is "beer". According to the preference information of record R in Figure 5, user P who prefers "beer" is user P whose family relationship is "father". Therefore, in this example, the user estimation unit 505 estimates that user P who used refrigerator 2 is user P whose family relationship is "father". For example, suppose the food type detection unit 502 detects that the food type is "yogurt," and in the stored food information of the identified record R, user P who stores the most "yogurt" is user P whose family relationship is "mother." In this example, the user estimation unit 505 estimates that user P who used refrigerator 2 is user P whose family relationship is "mother."

[0069] <Example 4> In Example 4, the user estimation unit 505 estimates the user P based on the current location of the terminal device 4. For example, let's assume that the identified record R is the record R in Figure 5. According to the terminal location information of record R in Figure 5, the only user P who is at home is user P whose family relationship is "mother". Therefore, in this example, the user estimation unit 505 estimates that user P who used refrigerator 2 is user P whose family relationship is "mother".

[0070] <Example 5> In Example 5, the user estimation unit 505 estimates user P based on user P's movements when receiving or retrieving food. In estimating user P, the user estimation unit 505 analyzes the recorded data 313 of the first uploaded data D1 received by the server communication control unit 501 to detect user P's movements when receiving or retrieving food. For example, suppose the identified record R is the record R shown in Figure 5, and the detected movement of user P is "extending the left hand toward the right." According to the movement information of record R in Figure 5, user P who performs the movement "extending the left hand toward the right" is user P whose family relationship is "father." Therefore, in this example, the user estimation unit 505 estimates that user P who used the refrigerator 2 is user P whose family relationship is "father." For example, suppose the detected user P's movements were to open the refrigerator compartment 21, then the freezer compartment 24, then the vegetable compartment 25, in that order. According to the movement information of record R in Figure 5, user P, who performs the movement of "refrigerator compartment, then freezer compartment, then vegetable compartment," is user P whose family relationship is "mother." Therefore, in this example, the user estimation unit 505 estimates that user P, who used the refrigerator 2, is user P whose family relationship is "mother." Alternatively, the user estimation unit 505 may estimate user P based on the movements indicating the order in which the refrigerator compartments 21 are opened and the opening and closing time information of the refrigerator compartment 21.

[0071] <Example 6> In Example 6, the user estimation unit 505 estimates user P based on how food is stored in the refrigerator 2. In estimating user P, the user estimation unit 505 performs image analysis on the recorded data 313 of the first uploaded data D1 received by the server communication control unit 501 to detect how food is stored. For example, suppose the identified record R is the record R in Figure 5, and the detected food is stored horizontally. According to the storage method information of record R in Figure 5, the user P who stores the food horizontally is the user P whose family relationship is "father". Therefore, in this example, the user estimation unit 505 estimates that the user P who used refrigerator 2 is the user P whose family relationship is "father".

[0072] As described above, the estimation methods of the user estimation unit 505 have been illustrated. The user estimation unit 505 may estimate user P using any of Examples 1 to 6, or it may estimate user P by combining some or all of Examples 1 to 6. Furthermore, if the user estimation unit 505 estimates user P using one or more of Examples 1 to 6 and is unable to narrow it down to a single user, it may estimate user P using one or more of the unexecuted Examples 1 to 6 and continue processing the unexecuted Examples 1 to 6 until it can narrow it down to a single user.

[0073] [1-1-4-6.Attribute estimation section] The attribute estimation unit 506 estimates the attributes of user P as estimated by the user estimation unit 505. When estimating attributes, the attribute estimation unit 506 identifies the record R of the imaging device ID 312 included in the first upload data D1 received by the server communication control unit 501 from the management DB 522. Then, from the identified record R, the attribute estimation unit 506 identifies the incoming food information or outgoing food information of user P as estimated by the user estimation unit 505.

[0074] For example, if the attribute estimation unit 506 determines that the identified incoming or outgoing food information indicates the incoming or outgoing of a predetermined number or more fermented foods, it estimates that the user P estimated by the user estimation unit 505 is health-conscious.

[0075] Furthermore, for example, if the attribute estimation unit 506 determines that the identified incoming or outgoing food information indicates the incoming or outgoing of a predetermined number or more of specified health foods, it estimates that the user P estimated by the user estimation unit 505 is health-conscious.

[0076] Furthermore, for example, if the attribute estimation unit 506 determines that the attribute of user P estimated by the user estimation unit 505 is that of a night owl if the identified incoming food information or outgoing food information includes a predetermined number or more opening / closing time information indicating nighttime hours (for example, 22:00 to 03:00).

[0077] Furthermore, for example, if the attribute estimation unit 506 determines that the attribute of user P estimated by the user estimation unit 505 is that of a morning person, if the identified incoming food information or outgoing food information includes a predetermined number of opening / closing time pieces indicating the morning time period (for example, 05:00 to 07:00).

[0078] Furthermore, for example, if the attribute estimation unit 506 determines that the identified incoming or outgoing food information indicates the inflow or outflow of fresh food items exceeding a predetermined number, it estimates that the user P estimated by the user estimation unit 505 is someone who prefers to eat at home.

[0079] Furthermore, for example, if the attribute estimation unit 506 determines that the identified incoming or outgoing food information indicates that the incoming or outgoing fresh food is below a predetermined number, it estimates that the user P estimated by the user estimation unit 505 is a frequent eater.

[0080] Furthermore, for example, if the identified incoming food information includes a predetermined number of opening / closing time pieces of information indicating opening / closing times on weekends (e.g., Saturday and Sunday), the attribute estimation unit 506 estimates that the user P estimated by the user estimation unit 505 is a person who stockpiles on weekends. Also, for example, if the identified incoming food information includes a predetermined number of food items stockpiled on weekends, or a predetermined number compared to weekdays, the attribute estimation unit 506 estimates that the user P estimated by the user estimation unit 505 is a person who stockpiles on weekends.

[0081] Furthermore, for example, if the attribute estimation unit 506 determines that the identified incoming food information tends to increase in quantity by more than a predetermined number during the same time period, it estimates that the user P estimated by the user estimation unit 505 is a person who purchases food via home delivery.

[0082] Furthermore, for example, if the identified incoming food information does not show a tendency for the storage volume to increase by more than a predetermined number during the same time period, the attribute estimation unit 506 estimates that the user P estimated by the user estimation unit 505 is a person who goes to a physical store and purchases food.

[0083] Furthermore, for example, if the identified incoming food information indicates that multiple food items have been received, the attribute estimation unit 506 estimates that the user P estimated by the user estimation unit 505 is in a frail state.

[0084] Furthermore, for example, if the attribute estimation unit 506 finds that the number of types of food indicated by the specified outgoing food information is greater than the number of types of food indicated by the specified incoming food information, it estimates that the number of types of food being received has decreased, and therefore estimates that the attributes of user P estimated by the user estimation unit 505 are in a frail state. Also, for example, if the attribute estimation unit 506 finds that the interval between receiving or shipping food is longer than a certain amount of time, or that the interval between receiving or shipping food is slower than a certain amount of time compared to other users, or that the interval between receiving or shipping food tends to be slower compared to other users, it estimates that the attributes of user P estimated by the user estimation unit 505 are in a frail state. Furthermore, if the attribute estimation unit 506 indicates that the identified incoming or outgoing food information shows that the interval between incoming or outgoing food is longer than a certain period of time, or that the interval between incoming or outgoing food is slower than a certain period of time compared to other users, or that the interval between incoming or outgoing food tends to be slower than other users, the user estimation unit 505 may estimate that the user P's attributes are such that they are under excessive stress, are fatigued, or have some kind of physical or mental abnormality.

[0085] The attribute estimation unit 506 may estimate attributes by referring to a rule base that associates the content indicated by the incoming food information or outgoing food information with the corresponding attributes. Alternatively, the attribute estimation unit 506 may estimate attributes using AI. When estimating attributes using AI, the attribute estimation unit 506 inputs the identified incoming food information or outgoing food information into a trained model and outputs the attributes from the trained model. This trained model is a model that has been machine-learned to assign attributes to the content indicated by the incoming food information or outgoing food information.

[0086] [1-1-4-7. Lifestyle Change Estimation Section] The lifestyle change estimation unit 507 estimates the lifestyle changes of user P detected by the user estimation unit 505. Lifestyle refers to the way of life and the way of living. The lifestyle change estimation unit 507 identifies the record R of the imaging device ID included in the first upload data D1 received by the server communication control unit 501 from the management DB 522. Then, from the identified record R, the lifestyle change estimation unit 507 identifies either the incoming food information or the outgoing food information of user P estimated by the user estimation unit 505.

[0087] For example, the lifestyle change estimation unit 507 estimates that the lifestyle of user P, as estimated by the user estimation unit 505, has changed if there is an increase or decrease in the types of food consumed by user P, as estimated by the user estimation unit 505. In this example, the lifestyle change estimation unit 507 refers to the identified outbound food information and estimates that the lifestyle has changed if the types of outbound food are increasing or decreasing over time.

[0088] Furthermore, for example, the lifestyle change estimation unit 507 estimates that the lifestyle of user P, as estimated by the user estimation unit 505, has changed if there is an increase or decrease in the time that user P opens and closes the refrigerator 2, as estimated by the user estimation unit 505. In this example, the lifestyle change estimation unit 507 refers to the opening and closing time information of the identified incoming food information or outgoing food information, and estimates that the lifestyle has changed if the opening and closing time is on an increasing or decreasing trend.

[0089] For example, the lifestyle change estimation unit 507 estimates that the lifestyle of user P has changed if there is a change in the type of specific food (e.g., Foods for Specified Health Uses) consumed by user P, as estimated by the user estimation unit 505. In this example, the lifestyle change estimation unit 507 refers to the identified outbound food information and estimates that the lifestyle has changed if there is a change in the type of specific food that has been outbound.

[0090] [1-1-4-8. Lifestyle Estimation Section] The lifestyle estimation unit 508 estimates the lifestyle after the change if the lifestyle change estimation unit 507 has estimated a change in lifestyle. The lifestyle estimation unit 508 estimates the lifestyle after the change based on the incoming food information or outgoing food information that the lifestyle change estimation unit 507 has processed.

[0091] For example, the lifestyle estimation unit 508 refers to the stockpiled food information and estimates that the lifestyle after the change is one of working from home or being a maximalist if the types of stockpiled food are increasing over time.

[0092] Furthermore, for example, the lifestyle estimation unit 508 refers to the outbound food information and estimates that the lifestyle after the change is one of commuting to the office or being a minimalist if the variety of outbound food items is decreasing over time.

[0093] Furthermore, for example, the lifestyle estimation unit 508 refers to the opening and closing time information of the incoming or outgoing food information, and if the opening and closing time is on an increasing trend, it estimates that the lifestyle after the change is a slow life.

[0094] Furthermore, for example, the lifestyle estimation unit 508 refers to the opening and closing time information of the incoming food information or the outgoing food information, and if the opening and closing time is on a decreasing trend, it estimates that the changed lifestyle is a first lifestyle.

[0095] The lifestyle estimation unit 508 may estimate a lifestyle by referring to a rule base that associates the content indicated by the incoming food information or outgoing food information with a lifestyle. Alternatively, the lifestyle estimation unit 508 may estimate a lifestyle using AI. When estimating a lifestyle using AI, the lifestyle estimation unit 508 inputs the identified incoming food information or outgoing food information into a trained model and outputs lifestyle information from the trained model. This trained model is a model that has been machine-learned to associate lifestyles with the content indicated by the incoming food information or outgoing food information.

[0096] [1-1-4-9. Update section] The update unit 509 updates the management DB 522. When the server communication control unit 501 receives the second upload data D2, the update unit 509 identifies the record R of the imaging device ID 312 included in the second upload data D2 from the management DB 522. Next, the update unit 509 identifies the terminal location information corresponding to the family relationship information 412 included in the second upload data D2 from the identified record R. Then, the update unit 509 updates the current location of the terminal device 4 indicated by the identified terminal location information to the current location information indicated in the second upload data D2.

[0097] When the server communication control unit 501 receives the first upload data D1, the update unit 509 identifies the record R of the imaging device ID 312 included in the first upload data D1 from the management DB 522. Next, the update unit 509 detects the shooting time indicated by the shooting time information of the recording data 313 included in the first uploaded data D1 as the opening / closing time. Next, the update unit 509 generates a set of food type information indicating the type of food detected by the food type detection unit 502, storage area information indicating the storage area detected by the storage area detection unit 504, and detected opening and closing time information. Next, when the inventory detection unit 503 detects an inventory inflow, the update unit 509 adds the generated set to the inventory food information of user P, estimated by the user estimation unit 505, from among the identified record R. On the other hand, when the inventory detection unit 503 detects an outbound shipment, the update unit 509 deletes any sets of contents that are the same as the generated set from the sets of contents included in the inventory food information of user P estimated by the user estimation unit 505, among the inventory food information of the identified record R. Next, when the inbound / outbound detection unit 503 detects an outbound shipment, the update unit 509 generates a pair of food type information indicating the type of food detected by the food type detection unit 502, and the detected opening / closing time information. Then, the update unit 509 includes the generated pair in the outbound food information of user P, which is estimated by the user estimation unit 505 from the outbound food information of the identified record R.

[0098] When the server communication control unit 501 receives the first upload data D1, the update unit 509 identifies the record R of the imaging device ID 312 included in the first upload data D1 from the management DB 522. Next, the update unit 509 updates the attribute of the identified record R, which is indicated by the attribute information of user P estimated by the user estimation unit 505, to the attribute estimated by the attribute estimation unit 506.

[0099] [1-2. Operation] Next, the operation of each part of the estimation system 1 in this embodiment will be described. First, we will explain the operation of the server device 5 when it receives the first upload data D1.

[0100] Figure 6 is a flowchart showing the operation of the server device 5. The flowchart in Figure 6 begins when the server device 5 receives the first upload data D1.

[0101] The food type detection unit 502 detects the type of food that has been put into or taken out of the refrigerator 2 based on the first upload data D1 received by the server communication control unit 501 (step SA1).

[0102] Next, the inbound / outbound detection unit 503 detects whether the food detected by the food type detection unit 502 is an inbound or outbound food, based on the first upload data D1 received by the server communication control unit 501 (step SA2).

[0103] Next, the storage area detection unit 504 detects the storage area of ​​the food detected by the food type detection unit 502 based on the first upload data D1 received by the server communication control unit 501 (step SA3). Note that steps SA2 and SA3 can be performed in reverse order, or simultaneously.

[0104] Next, the user estimation unit 505 estimates the user P who put food into or took out of the refrigerator 2 based on the first upload data D1 received by the server communication control unit 501 (step SA4).

[0105] Next, the attribute estimation unit 506 estimates the attributes of user P that were estimated in step SA4 (step SA5).

[0106] Next, the attribute estimation unit 506 determines whether the attribute estimated in step SA5 is different from the attribute estimated previously (step SA6). Step SA6 is described in detail below. The attribute estimation unit 506 identifies the record R of the imaging device ID 312 contained in the first upload data D1 received by the server communication control unit 501 from the management DB 522. Next, the attribute estimation unit 506 identifies the attribute information of user P detected in step SA4 from the identified record R. Next, if the attribute indicated by the identified attribute information is the same as the attribute estimated in step SA5, the attribute estimation unit 506 makes a negative determination in step SA6. On the other hand, if the attribute indicated by the identified attribute information is different from the attribute estimated in step SA5, the attribute estimation unit 506 makes an affirmative determination in step SA6.

[0107] If the attribute estimation unit 506 determines that the attribute is different from the one estimated previously (step SA6: YES), the server communication control unit 501 determines whether or not to issue a notification instruction for information corresponding to the attribute estimated by the attribute estimation unit 506, based on the current weather or the current time (step SA7). Hereinafter, this notification instruction will be referred to as the "first notification instruction".

[0108] Here, we will describe step SA7 in detail. For example, suppose the attribute estimated by the attribute estimation unit 506 is "health-conscious," and the information corresponding to the attribute notified by the first notification instruction includes a suggestion for outdoor exercise. In this example, even if the notification is sent on a rainy day, user P may not be able to exercise outdoors on that day. Therefore, in this example, the server communication control unit 501 makes an affirmative determination in step SA7 if the current weather is not rainy, and a negative determination in step SA7 if the current weather is rainy.

[0109] Furthermore, for example, suppose the attribute estimated by the attribute estimation unit 506 is "night owl," and the information corresponding to the attribute notified by the first notification instruction includes a suggestion to switch to an early-rising owl. In this example, even if the notification is made at night, user P may not be able to immediately switch to an early-rising owl. Therefore, in this example, the server communication control unit 501 makes a positive determination in step SA7 if the current time is, for example, between 08:00 and 20:59, and a negative determination in step SA7 if the current time is, for example, not between 08:00 and 20:59.

[0110] As these examples show, depending on the timing of notification of information corresponding to an attribute, the notification may not be convenient for user P due to the current weather or time. Therefore, as in the example above, step SA7 determines whether or not to issue the first notification instruction based on the current weather or the current time.

[0111] In step SA7, the server communication control unit 501 obtains the current weather by communicating with an external weather server. Also in step SA7, the server communication control unit 501 determines, based on the current weather or the current time, whether the current time is a transmission timing corresponding to the attribute estimated in step SA5. If the server communication control unit 501 determines that the current time is a transmission timing, it decides to issue the first notification instruction; if it determines that the current time is not a transmission timing, it decides not to issue the first notification instruction.

[0112] If the server communication control unit 501 determines to issue a first notification instruction (step SA7: YES), it sends the first notification instruction to the terminal device 4 (step SA8). The first notification instruction sent in step SA8 includes information corresponding to the attribute estimated in step SA5.

[0113] In step SA8, the server communication control unit 501 identifies record R of the imaging device ID 312 contained in the first uploaded data D1, which triggered the start of the flowchart in Figure 6, from the management DB 522. Next, the server communication control unit 501 identifies the terminal communication information of user P, which was detected in step SA4, from the identified record R. Then, based on the identified terminal communication information, the server communication control unit 501 sends a first notification instruction to the terminal device 4.

[0114] Returning to the explanation of step SA7, if the server communication control unit 501 determines that it will not issue the first notification instruction (step SA7: NO), it proceeds to step SA9.

[0115] Returning to the explanation of step SA6, if the attribute estimation unit 506 determines that the attribute is the same as the one estimated previously (step SA6: NO), the update unit 509 updates the management DB 522 based on the first upload data D1 received by the server communication control unit 501 and the processing results of steps SA1 to SA5 (step SA9).

[0116] Next, the lifestyle change estimation unit 507 estimates the lifestyle change based on the first upload data D1 received by the server communication control unit 501 and the user P estimated in step SA4 (step SA10).

[0117] Next, the lifestyle estimation unit 508 determines whether or not it was estimated that a lifestyle had changed in step SA10 (step SA11).

[0118] If the lifestyle estimation unit 508 determines that no change in lifestyle has been estimated (step SA11: NO), the server processor 500 terminates processing.

[0119] On the other hand, if the lifestyle estimation unit 508 determines that a change in lifestyle has been estimated (step SA11: YES), the lifestyle estimation unit 508 estimates the lifestyle (step SA12).

[0120] Next, the server communication control unit 501 sends a notification instruction to the terminal device 4 containing information corresponding to the lifestyle estimated by the lifestyle estimation unit 508 (step SA13). Hereinafter, this notification instruction will be referred to as the "second notification instruction." The second notification instruction sent in step SA13 includes information corresponding to the lifestyle estimated in step SA12. The transmission in step SA13 is performed in the same manner as the transmission in step SA8.

[0121] Next, with reference to Figure 7, the operation of terminal device 4 related to receiving notification instructions will be explained. Figure 7 is a flowchart showing the operation of terminal device 4.

[0122] The application execution unit 401 determines whether or not it has received the first notification instruction from the server device 5 (step SB1).

[0123] If the application execution unit 401 determines that it has received the first notification instruction (step SB1: YES), it displays the notification screen TG on the touch panel 42 based on the received first notification instruction (step SB2). Hereafter, the notification screen TG based on the first notification instruction will be referred to as "First Notification Screen TG1," with the designation "TG1."

[0124] Figure 8 shows an example of the first notification screen TG1. The first notification screen TG1 is a notification screen TG that notifies information corresponding to the attributes estimated by the attribute estimation unit 506. Figure 8 shows an example of the first notification screen TG1 that provides information corresponding to health consciousness.

[0125] The information corresponding to the attributes notified by the first notification screen TG1 includes status information J1 showing the current status of user P, suggestion information J2 offering suggestions to user P, and access information J3 for accessing the site. In Figure 8, status information J1 is the string "I have a healthy diet." Also in Figure 8, suggestion information J2 is the string "Why not exercise outdoors to aim for a healthier body?" Also in Figure 8, access information J3 is the string "Click here for nearby parks" and an icon for searching for nearby parks.

[0126] The current status information J1 may vary depending on the attribute estimated by the attribute estimation unit 506. For example, if the attribute estimated by the attribute estimation unit 506 is "Fresnel state," the first notification screen TG1 will notify the user with the current status information J1, "There is a lot of duplication in the food storage."

[0127] Furthermore, the suggested information J2 may vary depending on the attribute estimated by the attribute estimation unit 506. For example, if the attribute estimated by the attribute estimation unit 506 is "Fresnel state," the first notification screen TG1 will notify the user of suggested information J2 such as "Let's review your diet" or "Let's incorporate exercise into our lives."

[0128] Furthermore, the access information J3 may vary depending on the attribute estimated by the attribute estimation unit 506. For example, if the attribute estimated by the attribute estimation unit 506 is "Fresnel state," the first notification screen TG1 will notify the user of access information J3 which includes the string "Let's exercise nearby" and an icon for searching for a nearby gym.

[0129] Returning to the explanation of Figure 7, if the application execution unit 401 determines that it has not received the first notification instruction (step SB1: NO), it determines whether or not it has received the second notification instruction from the server device 5 (step SB3).

[0130] If the application execution unit 401 determines that it has not received the second notification instruction (step SB3: NO), it terminates processing.

[0131] If the application execution unit 401 determines that it has received the second notification instruction (step SB3: YES), it displays the notification screen TG on the touch panel 42 based on the received second notification instruction (step SB4). Hereafter, the notification screen TG based on the second notification instruction will be referred to as "Second Notification Screen TG2," with the designation "TG2."

[0132] Figure 9 shows an example of the second notification screen TG2. The second notification screen TG2 is a notification screen TG that notifies information corresponding to the lifestyle estimated by the lifestyle estimation unit 508. Figure 9 shows an example of the second notification screen TG2 that provides information corresponding to working from home.

[0133] The lifestyle-related information notified by the second notification screen TG2 includes status information J4 showing user P's current situation, suggestion information J5 offering suggestions to user P, and access information J6 for accessing a site. In Figure 9, status information J4 is the text "Your lifestyle is working from home." Also in Figure 9, suggestion information J5 is the text "Why not exercise outdoors to combat lack of exercise?" Also in Figure 9, access information J6 is the text "Click here for nearby parks" and an icon for searching for nearby parks.

[0134] The current status information J4 may vary depending on the lifestyle estimated by the lifestyle estimation unit 508. For example, if the lifestyle estimated by the lifestyle estimation unit 508 is "slow life," the second notification screen TG2 will notify the user with the current status information J4, "Your lifestyle is slow life."

[0135] Furthermore, the suggestion information J5 may vary depending on the lifestyle estimated by the lifestyle estimation unit 508. For example, if the lifestyle estimated by the lifestyle estimation unit 508 is "slow life," the second notification screen TG2 will notify the user with the suggestion information J5, "How about reading a book at a coffee shop?"

[0136] Furthermore, the access information J6 may vary depending on the lifestyle estimated by the lifestyle estimation unit 508. For example, if the lifestyle estimated by the lifestyle estimation unit 508 is "slow life," the second notification screen TG2 will notify the user of access information J6, which includes the text "Let's read a book at a nearby coffee shop" and an icon for searching for a nearby coffee shop.

[0137] Figure 6 shows the operation when the first notification instruction is sent to the terminal device 4 upon receipt of the first uploaded data D1. In this embodiment, the server device 5 sends the first notification instruction to the terminal device 4 periodically (for example, every week, every month).

[0138] Figure 10 is a flowchart showing the operation of the server device 5. The operation shown in Figure 10 is performed by processing the attribute information and terminal communication information of one user P from record R.

[0139] The server communication control unit 501 determines whether or not the cycle for sending the first notification instruction has arrived (step SC1).

[0140] If the server communication control unit 501 determines that the cycle for sending the first notification instruction has not yet arrived (step SC1: NO), it determines step SC1 again.

[0141] On the other hand, if the server communication control unit 501 determines that the cycle for sending the first notification instruction has arrived (step SC1: YES), it sends the first notification instruction to the terminal device 4 (step SC2). The first notification instruction sent in step SC2 includes information corresponding to the attribute indicated by the attribute information of the item to be processed. In step SC2, the server communication control unit 501 sends the first notification instruction based on the terminal communication information of the item to be processed.

[0142] [1-3. Effects, etc.] As described above, the estimation system 1 includes a user estimation unit 505 that estimates the user P who put food into or took out of the refrigerator 2, which is equipped with a camera 3 that photographs the refrigerator 2 from the front and above. The estimation system 1 also includes an attribute estimation unit 506 that estimates the attributes of the user P estimated by the user estimation unit 505 based on the information obtained from the photographic results of the camera 3.

[0143] According to this, the user P who received or took out food can be estimated, and attributes can be estimated for the estimated user P, so that the attributes of user P can be estimated for each user P of refrigerator 2.

[0144] The user estimation unit 505 estimates user P from among the family members who use refrigerator 2.

[0145] According to this, the attributes of each user P who makes up the family using refrigerator 2 can be estimated.

[0146] The imaging device 3 starts imaging when the door 21A or drawer 26A of the refrigerator 2 is open. The user estimation unit 505 estimates user P based on at least one of the following: the imaging time when the imaging device 3 took images, the storage area inside the refrigerator 2 where the food is placed in or taken out, the type of food detected, the current location of the terminal device 4 used by user P, the movements of user P when placing in or taking out food, and the way the food is stored in the refrigerator 2.

[0147] Depending on the user P, the timing of opening the door 21A or drawer 26A of refrigerator 2 may differ. Also, depending on the user P, the storage area in which food is placed or removed may differ. Also, depending on the user P, the type of food placed or removed from refrigerator 2 may differ. Also, depending on the user P, the current location of the terminal device 4 being used may differ. Also, depending on the user P, the movements when placing or removing food may differ. Also, depending on the user P, the way food is stored in refrigerator 2 may differ. Therefore, by estimating user P based on elements that may differ depending on the user P, user P can be estimated appropriately.

[0148] The user estimation unit 505 estimates user P based on information about user P that has been registered in advance by user P.

[0149] According to this, user P can be estimated by taking into account information that could be characteristic of user P based on the information registered by user P. Therefore, user P can be estimated appropriately.

[0150] The estimation system 1 includes an application execution unit 401 that notifies the user P estimated by the user estimation unit 505 of information corresponding to the attributes estimated by the attribute estimation unit 506.

[0151] According to this, user P can obtain information corresponding to their own attributes, thereby improving user P's convenience.

[0152] The estimation system 1 includes a lifestyle change estimation unit 507 that estimates changes in the lifestyle of user P detected by the user estimation unit 505. The estimation system 1 also includes an application execution unit 401 that, when the lifestyle change estimation unit 507 estimates changes in user P's lifestyle, notifies user P, as estimated by the user estimation unit 505, of information corresponding to the changes in user P's lifestyle.

[0153] According to this, if there is a change in lifestyle, user P can obtain information that corresponds to their lifestyle, thereby improving user P's convenience.

[0154] The lifestyle change estimation unit 507 estimates at least one of the following as changes in user P's lifestyle: an increase or decrease in the types of food that user P eats, an increase or decrease in the amount of time user P opens and closes the refrigerator 2, and a change in the types of specific foods that user P eats, as estimated by the user estimation unit 505.

[0155] As lifestyle changes, the types of food that user P consumes may increase or decrease. Furthermore, as lifestyle changes, the time user P spends opening and closing refrigerator 2 may increase or decrease. Also, as lifestyle changes, the types of specific foods that user P consumes may change. Therefore, changes in user P's lifestyle can be accurately estimated.

[0156] The estimation system 1 includes a lifestyle estimation unit 508 that estimates the user P's lifestyle after the lifestyle change estimation unit 507 has estimated a change in the user P's lifestyle.

[0157] According to this method, the user P who receives or takes out food is estimated, and if the lifestyle of the estimated user P changes, their lifestyle is estimated. Therefore, the lifestyle of each user P of refrigerator 2 can be estimated.

[0158] The estimation system 1 includes a server communication control unit 501 that, if the attribute estimation unit 506 estimates an attribute different from the previous one, issues a notification instruction to the application execution unit 401 for information corresponding to the attribute, based on the current weather or the current time.

[0159] Depending on the current weather or time, even if information corresponding to attributes is provided, it may not be a highly convenient notification. However, by taking the current weather or time into account when providing information corresponding to attributes, highly convenient notifications can be achieved.

[0160] The server device 5 includes a user estimation unit 505 and an attribute estimation unit 506. Furthermore, the control program 521 causes the server processor 500 to function as a user estimation unit 505 and an attribute estimation unit 506.

[0161] According to this, it produces the same effect as the estimation system 1 described above.

[0162] (Other embodiments) As described above, Embodiment 1 has been presented as an example disclosed in this application. However, the technology in this disclosure is not limited to this and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to combine the components described in Embodiment 1 to create new embodiments. Therefore, other embodiments are described below as examples.

[0163] In the first embodiment described above, the system estimates user P based on pre-registered information such as gender, age, and height, and then estimates the attributes of the estimated user P. In other embodiments, the attributes of user P are estimated even if no information has been registered in advance.

[0164] Other embodiments will be described below. Figure 11 shows the configuration of the server device 5 in another embodiment. Comparing Figure 4 and Figure 11, the server device 5 stores the management DB 523 in the server memory 520.

[0165] Figure 12 shows an example of record R1 stored in the management DB523. Management DB523 has one record R1 for each food item that is received or issued. Record R1 has imaging device ID312. Record R1 also contains one or more pieces of terminal communication information. Record R1 also contains stored data.

[0166] The stored data contains one or more pieces of information about incoming goods. This information pertains to the process of receiving goods. It includes information on food type, storage area, opening / closing times, movement, storage method, and estimated height. The estimated height information indicates the estimated height. Furthermore, the stored data contains one or more pieces of information about items being stored. These items are related to the storage process. The storage information includes information about the type of food, opening and closing times, movement information, storage method information, and estimated height information.

[0167] When the server communication control unit 501 receives the first upload data D1, the update unit 509 identifies record R1 of imaging device ID 312 included in the first upload data D1 from the management DB 523. This identification is performed if record R of imaging device ID 312 included in the first upload data D1 cannot be identified from the management DB 522. This allows the management DB 523 to be targeted for update if information has not been registered in advance.

[0168] Next, the update unit 509 generates storage information when the storage detection unit 503 detects storage entry. In generating the storage information, food type information, storage area information, and opening / closing time information are obtained using the same method as in Embodiment 1. In generating the storage information, movement information and storage method information are obtained by analyzing the recorded data 313. In generating the storage information, estimated height information is obtained by estimating the height of user P as seen in the recorded data 313 from user P's size, etc., through analysis of the recorded data 313. Then, the update unit 509 stores the generated storage information in the storage data of the identified record R1. Next, the update unit 509 generates outbound information when the inbound / outbound detection unit 503 detects an outbound shipment. In generating the outbound information, food type information and opening / closing time information are obtained using the same method as in Embodiment 1. In generating the outbound information, movement information and storage method information are obtained by analyzing the recorded data 313. In generating the outbound information, estimated height information is obtained by estimating the height of user P as seen in the recorded data 313 based on user P's size, etc. Then, the update unit 509 stores the generated outbound information in the stored data of the identified record R1.

[0169] When a predetermined trigger occurs, the attribute estimation unit 506 identifies a record R1 from the management DB 523, inputs at least one of the incoming or outgoing information of the identified record R1 into the trained model, and outputs attributes from the trained model. As a result, the attribute estimation unit 506 can estimate the attributes of user P even if the information has not been registered in advance. This trained model is a model that has been machine-learned to acquire attributes based on the content indicated by the incoming or outgoing information.

[0170] In this other embodiment, information corresponding to the attribute estimated based on record R1 is transmitted to terminal device 4 based on terminal communication information of record R1.

[0171] In the above-described embodiment 1, examples of attributes estimated by the attribute estimation unit 506 included health consciousness, night owls, early risers, those who eat out, those who eat at home, those who stock up on groceries on weekends, those who purchase food through home delivery, those who go to physical stores to purchase food, and the state of frailty. However, the attributes estimated by the attribute estimation unit 506 are not limited to these examples and may include other attributes such as meticulousness or carelessness.

[0172] In the first embodiment described above, examples of lifestyles estimated by the lifestyle estimation unit 508 were given as working from home, working in the office, maximalist, minimalist, slow life, and fast life. However, the lifestyles estimated by the lifestyle estimation unit 508 are not limited to these examples and may include other lifestyles such as living alone on a business trip, newlywed life, and pregnancy.

[0173] In the above-described embodiment 1, gender information, age information, family relationship information 412, height information, time spent at home information, and preference information were given as examples of information that is registered in advance by user P. In other embodiments, information that is not used for estimation of user P among gender information, age information, family relationship information 412, height information, time spent at home information, and preference information may not be registered in advance, that is, may not be included in record R.

[0174] In the first embodiment described above, the server processor 500 functions as a "first estimation unit" and a "second estimation unit". In other embodiments, either the imaging processor 300 or the terminal processor 400 may function as the "first estimation unit" and the "second estimation unit". In these other embodiments, the imaging device 3 or the terminal device 4 stores information necessary for estimating user P and estimating attributes. In these other embodiments, the imaging device 3 or the terminal device 4 is an example of an "estimation device". Also in these other embodiments, the imaging processor 300 or the terminal processor 400 is an example of a "processor". Also in these other embodiments, the control program 311 or the notification application 411 is an example of a "program".

[0175] Furthermore, if the imaging device 3 is an "estimation device," the imaging processor 300 may function as a food type detection unit 502, an in / out detection unit 503, and a storage area detection unit 504. Also, if the imaging device 3 is an "estimation device," the imaging device 3 may upload attribute information indicating the estimated attributes, information on the detected user P, images of the detected food, etc., to the server device 5.

[0176] In other embodiments, the device that notifies attributes and lifestyle may not be the terminal device 4, but for example, an interactive device equipped with voice and a microphone, or the camera device 3 may notify attributes and lifestyle by voice or the like.

[0177] Furthermore, in other embodiments, at least one of the functions of the food type detection unit 502, the in / out detection unit 503, the storage area detection unit 504, the user estimation unit 505, the attribute estimation unit 506, the lifestyle change estimation unit 507, the lifestyle estimation unit 508, and the update unit 509 may be implemented by the shooting processor 300 or the terminal processor 400.

[0178] In other embodiments, the server device 5 may cooperate with other devices such as a weighing scale, and from the information obtained from the cooperated devices, it may perform estimation of user P, estimation of attributes, estimation of changes in lifestyle, and estimation of lifestyle.

[0179] In other embodiments, the information may not only be notified to user P, but also to user P's family. In other embodiments, information for all family members (such as information corresponding to attributes) may be notified to a user P who is part of a family. In this case, the application execution unit 401 may display information for all family members on the same screen and notify them of the information.

[0180] In other embodiments, the camera may be configured to stop recording or taking pictures after a predetermined time has elapsed since the door 21A or drawer 26A was opened. For example, the predetermined time is 1 minute. However, if it is detected that the door 21A or drawer 26A is open, that a person is present around the refrigerator 2, or that user P is putting food in or taking it out, the predetermined time may be extended. Furthermore, if it is detected that the door 21A or drawer 26A is open, that a person is present around the refrigerator 2, or that user P is putting food in or taking it out, the camera may stop recording or taking pictures after a predetermined time has elapsed since the detection.

[0181] In another embodiment, instead of the recorded video data 313, still images captured by each camera may be stored in the shooting memory 310. In this other embodiment, there may be multiple captured images for each camera stored in the shooting memory 310.

[0182] In the above-described embodiment 1, the refrigerator 2 and the imaging device 3 are separate components. In other embodiments, the refrigerator 2 may include the imaging device 3. That is, in other embodiments, the refrigerator 2 and the imaging device 3 may be integrated into a single unit.

[0183] In the embodiment described above, a refrigerator 2 was illustrated in which the widths of the right door 21B and the left door 21C are different. In other embodiments, the widths of the right door 21B and the left door 21C may be the same, or the left door 21C may be wider than the right door 21B.

[0184] In other embodiments, the main casing 20 of the refrigerator 2 may have more or fewer storage compartments than the five types described above. Also, in other embodiments, the number of doors provided at the opening of the refrigerator compartment 21 may be just one.

[0185] The imaging processor 300, the terminal processor 400, and the server processor 500 may consist of a single processor or multiple processors. These processors may also be hardware programmed to implement the corresponding functional units. That is, these processors may consist of, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0186] The configurations of the parts of the estimation system 1 shown in Figures 2, 3, 4, and 11 are examples, and the specific implementation is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each part be implemented individually, and it is also possible to configure the system so that a single processor executes a program to realize the functions of each part. Furthermore, some of the functions realized by software in the above-described embodiment may be implemented by hardware, or some of the functions realized by hardware may be implemented by software.

[0187] The operational step units shown in Figures 6, 7, and 10 are divided according to the main processing content to facilitate understanding of the operation, and the operation is not limited by the way the processing units are divided or the names of the processing units. Depending on the processing content, it may be further divided into more step units. Alternatively, it may be divided so that one step unit includes even more processing. Furthermore, the order of the steps may be changed as appropriate, as long as it does not impede the intent of this disclosure.

[0188] Since the embodiments described above are for illustrative purposes of the technology described herein, various modifications, substitutions, additions, omissions, etc., can be made within the claims or their equivalents.

[0189] (Note) Based on the above description of embodiments, the following technologies are disclosed.

[0190] (Technology 1) An estimation system comprising: a first estimation unit that estimates the user who put food into or took out of the refrigerator, which is equipped with a camera that photographs the refrigerator from the front and above; and a second estimation unit that estimates the attributes of the user estimated by the first estimation unit based on information obtained from the photographic results of the camera. According to this method, it is possible to estimate the user who brought food in or out of storage, and then estimate the attributes of the estimated user, thereby estimating the user's attributes for each refrigerator user.

[0191] (Technology 2) The estimation system according to Technology 1, wherein the first estimation unit estimates the user from among the family members who use the refrigerator. According to this, it is possible to estimate the attributes of each user who makes up a family that uses the refrigerator.

[0192] (Technology 3) The estimation system according to Technology 1 or Technology 2, wherein the imaging device starts imaging when the opening / closing body of the refrigerator is in the open position, and the first estimation unit estimates the user based on at least one of the following: the imaging time during which the imaging device took images, the storage area in the refrigerator where the food is placed in or taken out, the type of food detected, the current location of the terminal device used by the user, the user's movements when placing in or taking out the food, and the way the food is stored in the refrigerator. Depending on the user, the timing of opening the refrigerator door may differ. Also, depending on the user, the storage area where food is placed in or taken out may differ. Also, depending on the user, the type of food placed in or taken out of the refrigerator may differ. Also, depending on the user, the current location of the terminal device being used may differ. Also, depending on the user, the movements when placing in or taking out food may differ. Also, depending on the user, the way food is stored in the refrigerator may differ. Therefore, by estimating the user based on elements that may differ from user to user, the user can be estimated appropriately.

[0193] (Technology 4) The estimation system according to any one of the technologies 1 to 3, wherein the first estimation unit estimates the user based on information relating to the user that has been registered in advance by the user. According to this method, it is possible to estimate a user by taking into account information that could be considered a characteristic of the user, based on the information registered by the user. Therefore, it is possible to estimate the user appropriately.

[0194] (Technology 5) An estimation system according to any one of technologies 1 to 4, comprising a notification unit that notifies the user or the user's family, as estimated by the first estimation unit, of information corresponding to the attribute estimated by the second estimation unit. According to this, users or their families can obtain information corresponding to their attributes, thereby improving convenience for users or their families.

[0195] (Technology 6) An estimation system according to any one of Technology 1 to Technology 5, comprising: a third estimation unit that estimates changes in the user's lifestyle estimated by the first estimation unit; and a notification unit that, when the third estimation unit estimates changes in the user's lifestyle, notifies the user estimated by the first estimation unit of information corresponding to the changes in the user's lifestyle. According to this, if there is a change in lifestyle, users can obtain information that corresponds to their lifestyle, thereby improving user convenience.

[0196] (Technology 7) The estimation system according to Technology 6, wherein the third estimation unit estimates, as a change in the user's lifestyle, at least one of the following: an increase or decrease in the types of food consumed by the user as estimated by the first estimation unit; an increase or decrease in the amount of time the user opens and closes the refrigerator as estimated by the first estimation unit; and a change in the types of specific food consumed by the user as estimated by the first estimation unit. As lifestyles change, the types of food a user consumes may increase or decrease. Furthermore, changes in lifestyle may lead to changes in the amount of time a user spends opening and closing their refrigerator. Additionally, changes in lifestyle may result in changes in the types of specific foods a user consumes. Therefore, changes in a user's lifestyle can be accurately estimated.

[0197] (Technology 8) The estimation system according to Technology 6 or Technology 7, further comprising a fourth estimation unit that estimates the user's lifestyle after a change, if the third estimation unit estimates a change in the user's lifestyle. According to this method, the system estimates the users who receive or take out food items, and if the lifestyles of these estimated users change, it estimates their lifestyles. Therefore, it is possible to estimate the lifestyle of each user of a refrigerator.

[0198] (Technology 9) The estimation system according to Technical Reference 5, further comprising a notification instruction unit that, if the second estimation unit estimates an attribute different from the previous one, issues a notification instruction to the notification unit for information corresponding to the attribute based on the current weather or the current time. Depending on the current weather or time, even if information corresponding to attributes is provided, it may not be a highly convenient notification. However, by taking the current weather or time into account when providing information corresponding to attributes, highly convenient notifications can be achieved.

[0199] (Technology 10) An estimation device comprising: a first estimation unit that estimates the user who put food into or took out of the refrigerator, which is equipped with a camera that photographs the refrigerator from the front and above; and a second estimation unit that estimates the attributes of the user estimated by the first estimation unit based on information obtained from the photographic results of the camera. According to this, it produces the same effect as the estimation system described in Technology 1.

[0200] (Technology 11) A program that functions as a processor comprising: a first estimation unit that estimates the user who put food into or took out of a refrigerator, which is equipped with a camera that photographs the refrigerator from the front and above; and a second estimation unit that estimates the attributes of the user estimated by the first estimation unit based on information obtained from the photographic results of the camera. According to this, it produces the same effect as the estimation system described in Technology 1. [Industrial applicability]

[0201] As described above, the estimation system, estimation device, and program according to the present invention can be used for the purpose of estimating the attributes of refrigerator users. [Explanation of Symbols]

[0202] 1 Estimation System 2 Refrigerator 3. Imaging device (estimation device) 4. Terminal device (estimation device) 5. Server device (estimated device) 6. Communication equipment 20 box body 20A Top 21 Refrigerator 21A Door (opening / closing mechanism) 21B Right door (opening / closing mechanism) 21C Left door (opening / closing mechanism) 21D shelf 22 Ice maker 23 Fresh freezing room 24 Freezer 25 Vegetable compartment 26 Drawer storage compartment 22A~26A Drawer (opening / closing body) 30. Shooting control device 31 Wide-angle camera 32 Narrow-angle camera 33. First distance measuring sensor 34. Second distance measuring sensor 35 Thermopile 36 Wireless communication modules 40 Terminal control device 41 Terminal communication module 42 Touch Panel 43 GNSS receiver 50 Server Control Units 51 Server communication module 300 Shooting Processor (Processor) 301 Image Capture and Communication Control Unit 302 Open / Close Determination Unit 303 Imaging Control Unit 304 Recording Control Unit 305 Generation part 310 Shooting Memory 311 Control program (program) 312 Imaging device ID 313 Recorded data 400 Terminal Processors (Processors) 401 Application execution unit (notification unit) 410 Terminal memory 411 Notification apps (programs) 412 Family Relationship Information 500 Server Processors (Processors) 501 Server Communication Control Unit (Notification Instruction Unit) 502 Food type detection unit 503 Inbound / Outbound Detection Unit 503 Food type detection unit 504 Accommodation Area Detection Unit 505 User Estimation Unit (First Estimation Unit) 506 Attribute estimation section (second estimation section) 507 Lifestyle Change Estimation Unit (Third Estimation Unit) 508 Lifestyle Estimation Department (4th Estimation Department) 509 Update Department 520 Server Memory 521 Control program (program) 522 Management DB D1 First Upload Data D2 Second Upload Data H Home J1, J4 Current Status Information J2, J5 Proposal Information J3, J6 Access Information NW Network TG notification screen TG1 1st notification screen TG2 2nd notification screen

Claims

1. A first estimation unit is installed to estimate the user who put food into or took out the refrigerator, and a camera is installed to photograph the refrigerator from the front and above the refrigerator. The system includes a second estimation unit that estimates the user's attributes based on information obtained from the imaging results of the imaging device, which were estimated by the first estimation unit. Estimation system.

2. The first estimation unit estimates the user from among the family members who use the refrigerator. The estimation system according to claim 1.

3. The aforementioned photographic device starts taking pictures when the opening / closing mechanism of the refrigerator is in the open position. The first estimation unit is, The user is estimated based on at least one of the following: the time the camera took a photograph; the storage area in the refrigerator where the food is placed in or taken out; the type of food detected; the current location of the terminal device used by the user; the user's movements when placing in or taking out the food; and the way the food is stored in the refrigerator. The estimation system according to claim 1 or 2.

4. The first estimation unit is, Based on the information relating to the user that has been registered in advance by the user, the user is estimated. The estimation system according to claim 1 or 2.

5. The system includes a notification unit that notifies the user or the user's family, as estimated by the first estimation unit, of information corresponding to the attribute estimated by the second estimation unit. The estimation system according to claim 1 or 2.

6. A third estimation unit estimates the changes in the user's lifestyle that the first estimation unit has estimated, The system includes a notification unit that, when the third estimation unit estimates a change in the user's lifestyle, notifies the user estimated by the first estimation unit of information corresponding to the change in the user's lifestyle. The estimation system according to claim 1 or 2.

7. The third estimation unit described above is: As a change in the user's lifestyle, The system estimates at least one of the following: an increase or decrease in the types of food consumed by the user, an increase or decrease in the time the user opens and closes the refrigerator, and a change in the types of specific foods consumed by the user, as estimated by the first estimation unit. The estimation system according to claim 6.

8. If the third estimation unit estimates a change in the user's lifestyle, the system includes a fourth estimation unit that estimates the user's lifestyle after the change. The estimation system according to claim 6.

9. If the second estimation unit estimates an attribute different from the previous one, the notification instruction unit provides a notification instruction to the notification unit for information corresponding to the attribute, based on the current weather or the current time. The estimation system according to claim 5.

10. A first estimation unit is installed to estimate the user who put food into or took out the refrigerator, and a camera is installed to photograph the refrigerator from the front and above the refrigerator. The system includes a second estimation unit that estimates the user's attributes based on information obtained from the imaging results of the imaging device, which were estimated by the first estimation unit. Estimation device.

11. The processor, A first estimation unit is installed to estimate the user who put food into or took out the refrigerator, and a camera is installed to photograph the refrigerator from the front and above the refrigerator. Based on the information obtained from the imaging results of the imaging device, a second estimation unit functions to estimate the user's attributes that the first estimation unit has estimated. program.

Images