Storage support device and method

Abstract

To provide a storage support device and method for supporting storage of articles.SOLUTION: A storage support device 1 comprises a control unit 10, a sensor unit 20, a recording unit 30, and an interface unit 40. The control unit 10 includes a registration processing unit 11, an object detection unit 12, a tracking processing unit 13, a voice dealing unit 14, and a notification unit 15, and integrally controls the entire operation of the storage support device 1. The sensor unit 20 outputs, to the control unit 10, sensor information SI20 obtained by using its own constituents. The recording unit 30 includes information on prescribed storage destinations in which registered articles should be stored.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a storage assistance device and method. [Background technology]

[0002] In tidying up, items that have not been put away are stored in storage locations. A tidying up support system that supports tidying up has been proposed (see, for example, Patent Document 1 below). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 087854 Summary of the Invention [Problem to be solved by the invention]

[0004] If assistance is provided for storing items for tidying up, etc., items can be stored properly, which is highly convenient.

[0005] An object of the present invention is to provide a storage assistance device and method that can assist in storing items. [Means for solving the problem]

[0006] The storage assistance device of the present invention is a storage assistance device that assists in the storage of items and is equipped with a control unit that identifies which of a plurality of registered items each item is, and a database that contains information on specified storage destinations where each registered item should be stored, and the control unit notifies a user of storage assistance information that assists in the storage of each item based on the results of the identification and the database. Effect of the Invention

[0007] According to the present invention, it is possible to provide a storage assistance device and method that can assist in storing items. [Brief description of the drawings]

[0008] [Figure 1] 1 is an overall configuration diagram of a storage assistance device according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a diagram showing a state in which a storage assistance device according to an embodiment of the present invention is applied to a residence. [Diagram 3] FIG. 2 is a schematic configuration diagram of a database according to the embodiment of the present invention. [Figure 4] FIG. 2 is a schematic configuration diagram of a registrant database, a storage destination database, an item database, and a comprehensive management database according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram showing an internal configuration of a registrant database in the embodiment of the present invention. [Figure 6] FIG. 11 is a diagram showing an internal configuration of a storage destination database according to the embodiment of the present invention. [Figure 7] FIG. 4 is a diagram showing an internal configuration of an article database according to the embodiment of the present invention. [Figure 8] FIG. 2 is a diagram showing an internal configuration of a comprehensive management database according to an embodiment of the present invention. [Figure 9] 5 is a flowchart of an operation using the storage assistance device according to the embodiment of the present invention. [Figure 10] FIG. 2 is a diagram showing a correspondence between a plurality of living areas and a plurality of rooms in a house according to an embodiment of the present invention. [Figure 11] FIG. 1 is a diagram showing an example of movement of an article according to an embodiment of the present invention. [Figure 12] FIG. 4 is a diagram showing an example of display content on a display unit according to an embodiment of the present invention. [Figure 13] FIG. 11 is a diagram showing an assumed state according to Example EX1_A belonging to an embodiment of the present invention. [Figure 14] FIG. 11 is a diagram showing an example EX1_A according to an embodiment of the present invention, illustrating information exchange between a user and a storage assistance device. [Figure 15] FIG. 11 is a diagram showing an example of the display contents of the display unit according to Example EX1_B of the embodiment of the present invention. [Figure 16]FIG. 11 is a diagram showing an assumed state according to Example EX2 belonging to an embodiment of the present invention. [Figure 17] FIG. 11 is a diagram showing an example EX3 according to an embodiment of the present invention, illustrating an exchange of information between a user and a storage assistance device. [Figure 18] FIG. 11 is a diagram showing an assumed state according to Example EX6 belonging to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, examples of embodiments of the present invention will be described in detail with reference to the drawings. In each of the drawings, the same parts are given the same reference numerals, and duplicated descriptions of the same parts are generally omitted. In this specification, for the sake of simplicity, a symbol or code referring to information, a signal, a physical quantity, or a member may be written, and the name of the information, signal, physical quantity, or member corresponding to the symbol or code may be omitted or abbreviated.

[0010] FIG. 1 shows a configuration block diagram of a storage support device 1 according to an embodiment of the present invention. The storage support device 1 can also be referred to as a storage support system 1. The storage support device 1 can be used in a residence in which one or more people reside. The residence may be in any form. Therefore, for example, the residence to which the storage support device 1 is applied may be a detached residence or an apartment building. Here, it is assumed that the storage support device 1 is applied to a residence HH, as shown in FIG. 2.

[0011] The storage support device 1 includes a control unit 10, a sensor unit 20, a recording unit 30, and an interface unit 40. The entire storage support device 1 may be installed inside the residence HH, but some of the components of the storage support device 1 may be installed outside the residence HH. For example, the recording unit 30 may be a recording device installed outside the residence HH. Both the control unit 10 and the recording unit 30 may be installed outside the residence HH.

[0012] The control unit 10 is configured to include a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc., and comprehensively controls the overall operation of the storage assistance device 1. The control unit 10 has a program memory (not shown) made of a magnetic disk, a semiconductor memory, etc., and various functions of the control unit 10 are realized by executing programs stored in the program memory with the CPU. As functional blocks realized by executing the above programs with the CPU, the control unit 10 includes a registration processing unit 11, an object detection unit 12, a tracking processing unit 13, a voice response unit 14, and a notification unit 15, the functions of which will be described later.

[0013] The sensor unit 20 includes a camera unit 21, a light detection unit 22, and an environment measurement unit 23. The sensor unit 20 outputs sensor information SI20 obtained using its own components to the control unit 10. A sensing space is set within the living space of the residence HH. The sensing space may be the entire living space of the residence HH, or may be a part of the living space of the residence HH. In the following, the sensing space is considered to coincide with the entire living space of the residence HH. Each of the components of the camera unit 21, the light detection unit 22, and the environment measurement unit 23 is connected to the control unit 10 by wire or wirelessly, and the sensor information SI20 is sent from the sensor unit 20 to the control unit 10 by wire or wirelessly.

[0014] The camera unit 21 includes one or more fixed cameras 21a. The total number of fixed cameras 21a constituting the camera unit 21 can be any number as long as it is one or more, but in the following, it is assumed that the camera unit 21 includes multiple fixed cameras 21a. Each fixed camera 21a is fixedly installed at an appropriate position in the residence HH (for example, on the ceiling of each room in the residence HH). Each fixed camera 21a captures an object within its own field of view, and generates and outputs image information (hereinafter also referred to as captured image information) representing the capture result. The field of view of any camera can be expressed as the camera's shooting range or shooting area. The field of view of each fixed camera 21a is set within the living space of the residence HH. The field of view of the camera unit 21 corresponds to a combination of the fields of view of the multiple fixed cameras 21a included in the camera unit 21. By making the fields of view of the multiple fixed cameras 21a different from each other (however, they may overlap partially), the entire living space in the residence HH or a large part of it is included in the field of view of the camera unit 21. The sensing space is included in the field of view of the camera unit 21, and the field of view of the camera unit 21 may be considered to correspond to the sensing space. The captured image information SI21 output from the camera unit 21 includes captured image information generated by each fixed camera 21a. The sensor information SI20 includes the captured image information SI21.

[0015] The light detection unit 22 includes one or more LIDARs (Light Detection and Ranging) 22a, and detects any object (including a person) using light. The total number of LIDARs 22a constituting the light detection unit 22 is arbitrary as long as it is one or more, but in the following, it is considered that the light detection unit 22 includes multiple LIDARs 22a. Each LIDAR 22a is fixedly installed at an appropriate position in the residence HH (for example, the ceiling of each room in the residence HH). Each LIDAR 22a detects the distance between itself and the object by receiving reflected light from the object of the light emitted by itself, and generates and outputs light detection information indicating the detection result. In each LIDAR 22a, the range of the object whose distance can be detected is called the detection range. The detection range of each LIDAR 22a is set within the living space of the residence HH. The detection range of the light detection unit 22 corresponds to the combination of the detection ranges of the multiple LIDARs 22a included in the light detection unit 22. By making the detection ranges of the multiple LIDARs 22a different from one another (however, they may overlap partially), the entire or a large part of the living space in the residence HH is included in the detection range of the light detection unit 22, and the sensing space is included in the detection range of the light detection unit 22. The light detection information SI22 output from the light detection unit 22 includes the light detection information generated by each LIDAR 22a. The sensor information SI20 includes the light detection information SI22.

[0016] The environment measurement unit 23 includes one or more environment measurement sensors 23a. Each environment measurement sensor 23a measures the environment at a measurement target position, and generates and outputs environment measurement information indicating the measurement result. The measured environment includes the temperature and humidity at the measurement target position. The environment measurement information SI23 output from the environment measurement unit 23 includes the environment measurement information generated by each environment measurement sensor 23a. The sensor information SI20 includes the environment measurement information SI23.

[0017] The recording unit 30 is a recording device consisting of any recording medium such as a magnetic disk or a semiconductor memory. The control unit 10 can record any information in the recording unit 30 and can read any information recorded in the recording unit 30. The control unit 10 and the recording unit 30 are connected to each other in a wired or wireless manner so as to be capable of two-way communication. Under the control of the control unit 10, a database DB is formed and held in the recording unit 30. Details of the database DB will be described later.

[0018] The interface unit 40 is a man-machine interface between the storage assistance device 1 and the user, and includes a display unit 41, an operation unit 42, a microphone 43, and a speaker 44. In this embodiment, a user is a user of the storage assistance device 1, and any person located within the residence HH can be a user. Each registered person described below can be a user. The control unit 10 and the interface unit 40 are connected to each other via wire or wirelessly in a manner that allows two-way communication with each other.

[0019] The display unit 41 is a display device composed of a liquid crystal display panel or the like, and displays an arbitrary image under the control of the control unit 10. The operation unit 42 accepts an input of an arbitrary operation from the user. The display unit 41 and the operation unit 42 may form a touch panel, and the operation on the operation unit 42 may be an operation on the touch panel. The microphone 43 picks up sounds around the microphone 43 and converts the picked up sounds into an electric signal. The target of the sound picked up by the microphone 43 is mainly the voice uttered by the user. The speaker 44 outputs an arbitrary sound under the control of the control unit 10. The control unit 10 may be provided with an artificial intelligence having a voice recognition function and capable of having a voice conversation with a person, in which case the microphone 43 and the speaker 44 may be used to realize communication between the user and the storage support device 1. In the following description, a user operation refers to an operation input by the user to the storage support device 1 via the interface unit 40. The user operation may be an operation on the touch panel, an operation on an operating member in the operation unit 42, or a voice operation by the user's voice. Any instruction from the user refers to an instruction given by the user to the storage assistance device 1 via the interface unit 40. Since an instruction from the user is realized by a user operation, it may be understood that they refer to substantially the same thing.

[0020] In addition, a terminal device provided separately from the storage assistance device 1 may be used as the interface unit 40. The terminal device is, for example, a smartphone, a tablet terminal, a personal computer, or a television receiver. The microphone 43 and the speaker 44 may be provided in a so-called smart speaker.

[0021] As shown in Fig. 3, the database DB includes a registrant database DB_a, a storage destination database DB_b, an item database DB_c, and a general management database DB_d. Figs. 4(a) to (d) show simplified structures of the databases DB_a to DB_d.

[0022] In the initial registration process described below, one or more people are registered as registrants in the storage assistance device 1. Registration in the storage assistance device 1 may be understood to be synonymous with registration in the database DB. The registrant database DB_a stores a registrant data set 610 for each registrant. Note that registrants can also be added after the initial registration process.

[0023] One or more storage destinations are registered in the storage assistance device 1 in the initial registration process described below. A storage destination refers to a storage location (furniture, space, etc.) in which some kind of item can be stored, and is provided in the residence HH. For example, a chest of drawers, a closet, a toy box, a storage case, a bookshelf, a library, a storeroom, a locker, etc. correspond to storage destinations. A storage destination data set 630 is stored in the storage destination database DB_b for each storage destination. Note that storage destinations can also be added after the initial registration process.

[0024] In the initial registration process described below, one or more items are registered in the storage assistance device 1. The registered items may be referred to as registered items below. In this embodiment, an item refers to an item that can be stored in any storage destination. An item data set 650 is stored in the item database DB_c for each registered item. Note that registered items may be added even after the initial registration process.

[0025] The comprehensive management database DB_d is a database for managing various information regarding registered articles, and a management data set 670 is stored in the comprehensive management database DB_d for each registered article. The comprehensive management database DB_d also includes data other than the management data set 670, which will be explained later. Note that, although it is assumed here that the databases DB_c and DB_d are provided separately, a database in which the databases DB_c and DB_d are integrated may be configured instead of the databases DB_c and DB_d. The database in which the databases DB_c and DB_d are integrated will include an article data set 650 and a management data set 670 for each registered article.

[0026] FIG. 5 shows an example of the structure of the registrant database DB_a and the data in the registrant database DB_a. In this embodiment, the terms "information" and "data" may be considered to be synonymous with each other. Here, it is assumed that there are multiple registrants, and the i-th registrant is referred to as the i-th registrant. i represents any natural number. The registrant data set 610 for each registrant includes information 611-616. That is, the registrant data set 610 for the first registrant includes information 611-616 for the first registrant, and the registrant data set 610 for the second registrant includes information 611-616 for the second registrant (similarly for other registrants). In FIG. 5, in order to prevent the drawing from becoming too complicated or to simplify the illustration, the contents of only some of the information 611-616 are illustrated, and the contents of some of the information are omitted.

[0027] The i-th registrant's information 611 indicates the name of the i-th registrant (real name or nickname, etc.). Information 612 indicates a registrant ID unique to each registrant. Hereinafter, the registrant ID for the i-th registrant will be referred to by the symbol "aa[i]".

[0028] Information 613 indicates the designated storage area of ​​each registrant. In storage assistance device 1, the living space of residence HH is divided into multiple residential areas for management, and one or more of the multiple residential areas are set as the designated storage area for each registrant. The designated storage area for the i-th registrant indicates the residential area where the item owned by the i-th registrant should be stored.

[0029] The control unit 10 can execute a person identification process that identifies which registered person each person in the sensing space is by image recognition. The information 614 is image information for image recognition that is referred to for the identification. The information 614 of the i-th registered person includes image information obtained by photographing the i-th registered person. In order to improve the accuracy of the identification, it is preferable to include image information based on a plurality of photographed images obtained by photographing the i-th registered person from a plurality of directions in the information 614 of the i-th registered person. The information 614 of the i-th registered person includes at least image information of the face image of the i-th registered person. After the information 614 of each registered person is stored in the registered person database DB_a, the control unit 10 can execute a person identification process based on the photographed image information SI21 (see FIG. 1) and the information 614 of each registered person. In the person identification process, the light detection information SI22 may be further referred to.

[0030] Information 615 is preference information that indicates the preferences of each registrant (the contents are not shown in FIG. 5). Information 616 is other information that is not classified into information 611 to 615 (the contents are not shown in FIG. 5). Information 616 of the i-th registrant can include any information related to the i-th registrant.

[0031] FIG. 6 shows an example of the structure of the storage destination database DB_b and data in the storage destination database DB_b. Here, it is assumed that there are multiple storage destinations, and the i-th storage destination registered in the storage assistance device 1 is referred to as the i-th storage destination. The storage destination described in this embodiment refers to the storage destination registered in the storage assistance device 1 unless otherwise specified. The storage destination data set 630 for each storage destination includes information 631-638. That is, the storage destination data set 630 for the first storage destination includes information 631-638 for the first storage destination, and the storage destination data set 630 for the second storage destination includes information 631-638 for the second storage destination (similar to other storage destinations). In FIG. 6, in order to prevent the drawing from becoming too complicated or to simplify the illustration, the contents of only some of the information 631-638 are illustrated, and the illustration of the contents of some information is omitted.

[0032] The information 631 of the i-th storage destination indicates the name (nickname) of the i-th storage destination. Information 632 indicates a storage destination ID unique to each storage destination. Hereinafter, the storage destination ID for the i-th storage destination will be referred to by the symbol "bb[i]". Each storage destination has a storage space capable of storing items. The information 633 of the i-th storage destination indicates the total volume of the storage space possessed by the i-th storage destination (hereinafter referred to as the total storage volume). The information 634 of the i-th storage destination indicates the area in which the i-th storage destination is located. The area in which the i-th storage destination is located will be any one of the multiple residential areas described above.

[0033] The information 635 of the i-th storage destination represents a three-dimensional map (hereinafter referred to as a 3D map) of the i-th storage destination. The three-dimensional map of the i-th storage destination represents at least the size and shape of the storage space of the i-th storage destination. The three-dimensional map of the i-th storage destination is data set in a virtual three-dimensional space, and the information 635 of the i-th storage destination defines the storage space of the i-th storage destination in the virtual three-dimensional space. The virtual three-dimensional space is a virtual space formed by the control unit 10.

[0034] The information 636 of the i-th storage destination indicates the estimated remaining capacity of the i-th storage destination (not shown in FIG. 6). The estimated remaining capacity of the i-th storage destination indicates the difference between the total storage volume of the i-th storage destination and the total volume of items currently stored in the storage space of the i-th storage destination, and the estimated remaining capacity of the i-th storage destination is obtained by subtracting the latter from the former. In other words, the estimated remaining capacity of the i-th storage destination indicates the maximum volume of items that can be additionally stored in the i-th storage destination.

[0035] The information 637 of the i-th storage destination is environmental information that indicates the environment of the i-th storage destination (the contents are not shown in FIG. 6). The information 637 indicates the temperature and humidity of the storage space of the i-th storage destination as the environment of the i-th storage destination. The information 637 of the i-th storage destination is obtained from the environment measurement sensor 23a installed in the storage space of the i-th storage destination. The information 638 of the i-th storage destination is other information that is not classified into the information 631 to 637 (the contents are not shown in FIG. 6). The information 638 of the i-th storage destination can include any information related to the i-th storage destination.

[0036] FIG. 7 shows an example of the structure of the item database DB_c and the data in the item database DB_c. Here, it is assumed that there are multiple registered items, and the i-th registered item is referred to as the i-th registered item. The item data set 650 for each registered item includes information 651-661. That is, the item data set 650 for the first registered item includes information 651-661 for the first registered item, and the item data set 650 for the second registered item includes information 651-661 for the second registered item (similarly for other registered items). In FIG. 7, in order to prevent the drawing from becoming too complicated or to simplify the illustration, the contents of only some of the information 651-661 are illustrated, and the illustration of the contents of some of the information is omitted.

[0037] The information 651 of the i-th registered article indicates the name (nickname) of the i-th registered article. The name (nickname) of the registered article may be a general product name or a nickname given by the owner or the like. Information 652 indicates an article ID unique to each registered article. Hereinafter, the article ID for the i-th registered article will be referred to by the symbol "cc[i]". The information 653 of the i-th registered article indicates one or more article images (image information of one or more article images) which are photographed images of the i-th registered article. The information 654 of the i-th registered article is shape information indicating the shape (e.g., three-dimensional shape) of the i-th registered article. The information 655 of the i-th registered article indicates the volume (volume) of the i-th registered article.

[0038] In the control unit 10, any object is classified into one of a plurality of types. The process for realizing this classification is called object classification process. For example, the above-mentioned plurality of types include toys, stationery, books, electronic devices, kitchen utensils, and sports equipment. The information 656 of the i-th registered object is classification information indicating which type of object the i-th registered object belongs to.

[0039] The information 657 of the i-th registered item indicates the owner of the i-th registered item. One of the registrants will be the owner of the i-th registered item. The information 658 of the i-th registered item is registration date information indicating the date (item registration date) when the i-th registered item was registered in the storage assistance device 1 and the database DB. The information 659 of the i-th registered item indicates the prescribed storage destination where the i-th registered item should be stored. The prescribed storage destination is one of the storage destinations to which a storage destination ID (632) is assigned in the storage destination database DB_b. The storage destination ID of the prescribed storage destination of the i-th registered item is set as the information 659 of the i-th registered item.

[0040] Information 660 of the i-th registered item represents the usage history of the i-th registered item (the contents are not shown in FIG. 7). Information 661 is other information that is not classified into information 651 to 660 (the contents are not shown in FIG. 7). Information 661 of the i-th registered item can include any information related to the i-th registered item.

[0041] FIG. 8 shows an example of the structure of the comprehensive management database DB_d and the data in the comprehensive management database DB_d. As described above, the comprehensive management database DB_d has a management data set 670 for each registered item. The management data set 670 for each registered item includes information 671-680. That is, the management data set 670 for the first registered item includes information 671-680 for the first registered item, and the management data set 670 for the second registered item includes information 671-680 for the second registered item (similarly for other registered items). In FIG. 7, in order to prevent the drawing from becoming too complicated or to simplify the illustration, the contents of only some of the information 671-680 are illustrated, and the contents of some of the information are omitted.

[0042] The information 671 of the i-th registered item indicates the item ID of the i-th registered item. That is, the information 671 of the i-th registered item is the same as the information 652 of the i-th registered item (FIG. 7). Based on the information 671 and 652, the management data set 670 of the i-th registered item and the item data set 650 of the i-th registered item are associated with each other.

[0043] The information 672 of the i-th registered item indicates the storage location where the i-th registered item is currently stored (hereinafter, may be referred to as the current storage location). The storage location ID of the current storage location of the i-th registered item is set as the information 672 of the i-th registered item. The information 673 of the i-th registered item is information indicating the user of the i-th registered item. The user of the i-th registered item is one or more registrants. The registrant ID of the user of the i-th registered item is set as the information 673 of the i-th registered item. The information 674 of the i-th registered item is information indicating the final user of the i-th registered item. The registrant ID of the final user of the i-th registered item is set as the information 674 of the i-th registered item.

[0044] Information 675 of the i-th registered item indicates the movement history of the i-th registered item (contents not shown in FIG. 8). Information 676, 677, 678 of the i-th registered item indicate the disposal suggestion flag, the disposal estimated flag, and the consumables flag for the i-th registered item, respectively (contents not shown in FIG. 8). Information 679 of the i-th registered item indicates the status (item status) of the i-th registered item (contents not shown in FIG. 8). Information 680 is other information not classified into information 671-679 (contents not shown in FIG. 8). Any information related to the i-th registered item can be included in information 680 of the i-th registered item.

[0045] The comprehensive management database DB_d further includes information 685. The information 685 indicates the date and time when the contents of the comprehensive management database DB_d were last updated (update date and time).

[0046] 9, in order to operate the storage assistance device 1, an installation process in step S1 and a calibration process in step S2 are first carried out, followed by an initial registration process in step S3. After that, the process proceeds to an actual operation process in step S4.

[0047] In the installation process of step S1, any person installs each of the fixed cameras 21a, each of the LIDARs 22a, and each of the environmental measurement sensors 23a at appropriate positions within the residence HH.

[0048] The control unit 10 performs a calibration process in step S2. In the calibration process, a total space 3D map representing the overall three-dimensional shape of the living space of the residence HH is defined in the above-mentioned virtual three-dimensional space. A pre-prepared total space 3D map may be provided to the storage assistance device 1, or the total space 3D map may be created using each fixed camera 21a and each LIDAR 22a.

[0049] In the calibration process, each position in the entire space 3D map is associated with each position in the sensing space, which is an actual three-dimensional space. In addition, in the calibration process, the position and shooting direction of each fixed camera 21a and the position and light emission direction of each LIDAR 22a are specified in the entire space 3D map. As a result, each position on the entire space 3D map is associated with the position and field of view (shooting range) of each fixed camera 21a and the position and detection range of each LIDAR 22a. After the calibration process, the control unit 10 can grasp which part in the sensing space each fixed camera 21a is shooting and in which direction, and from which position in the sensing space each LIDAR 22a is emitting light and in which direction. The control unit 10 can recognize the position and movement of each person or each item in the sensing space based on the grasped content, the shooting image information SI21, and the light detection information SI22. The installation process and the calibration process may be performed under the management of a service staff other than the user.

[0050] As described above, the living space of the residence HH is composed of multiple living areas. The positional relationship of the multiple living areas and the shape and size of each living area are specified in the entire space 3D map. Here, the living space of the residence HH includes a Western-style room on the first floor, a Japanese-style room on the first floor, a Western-style room on the second floor, a living room, and a kitchen (see FIG. 10). The Western-style room on the first floor, the Japanese-style room on the first floor, the Western-style room on the second floor, the living room, and the kitchen are set as the first to fifth living areas, respectively. There may be further living areas different from the first to fifth living areas, but in this embodiment, for the sake of concrete explanation, attention is focused on only these five living areas, and the existence of other living areas is ignored. The first to fifth living areas are included in the above-mentioned sensing space.

[0051] In the installation process, one or more fixed cameras 21a are installed in each living area. Furthermore, one or more LIDARs 22a may be installed in each living area. One or more storage locations are installed in each living area. An environmental measurement sensor 23a is installed in each storage location (however, the environmental measurement sensor 23a may not be installed in one or more storage locations). The position of each storage location may be defined in a full-space 3D map.

[0052] In the initial registration process of step S3, the registration processing unit 11 (see FIG. 1) executes the initial registration process. The initial registration process includes a person registration process, a storage destination registration process, and an item registration process. Each process constituting the initial registration process may be executed based on a user operation.

[0053] The person registration process will be described (see FIG. 5 as appropriate). In the person registration process, the registration processing unit 11 registers one or more persons as one or more registrants in the database DB based on the user operation. The registration processing unit 11 generates a registrant data set 610 in the registrant database DB_a for the number of people to be registered based on the content of the user operation in the person registration process, and acquires the name of each registrant to set information 611 for each registrant. The registration processing unit 11 sets information 612 by assigning a registrant ID to each registrant. In this embodiment, it is assumed that there are three or more registrants including a first to a third registrant, the third registrant is a child, and the first and second registrants are the parents of the third registrant.

[0054] In the person registration process, the registration processing unit 11 sets information 613 of each registered person based on user operation. As described above, the information 613 of the i-th registered person represents the designated storage area of ​​the i-th registered person. Here, it is assumed that the first registered person frequently uses the Western-style room and the Japanese-style room on the first floor, and the user desires to store the belongings of the first registered person in the Western-style room or the Japanese-style room on the first floor. Then, by inputting this information by user operation in association with the first registered person, the Western-style room and the Japanese-style room on the first floor are set as the designated storage area of ​​the first registered person. Similarly, it is assumed here that the designated storage area of ​​the second registered person is set to the living room and kitchen, and the designated storage area of ​​the third registered person is set to the Western-style room and the living room on the second floor.

[0055] In addition, in the person registration process, the registration processing unit 11 acquires information 614 for each registrant and stores it in the registrant database DB_a. The photographing for obtaining the information 614 may be performed by the fixed camera 21a in the camera unit 21, or may be performed by another camera. In the person registration process, if information indicating the preference of the i-th registrant is input by a user operation, the registration processing unit 11 sets preference information 615 of the i-th registrant based on the input contents. In addition, in the person registration process, the registration processing unit 11 can store other information 616 in the database DB_a based on a user operation.

[0056] In this way, information 611-614 of each registrant shown in Fig. 5 is set in the person registration process. Information 615 of one or more registrants may be set in the person registration process. Information 615 of each registrant may be set in the actual operation process. Information 616 may include information set in the person registration process, and may include information set and updated in the actual operation process.

[0057] The storage destination registration process will be described (see FIG. 6 as appropriate). In the storage destination registration process, the registration processing unit 11 acquires the name of each storage destination based on a user operation in the storage destination registration process, and sets information 631 for each storage destination. The registration processing unit 11 sets information 632 by assigning a storage destination ID to each storage destination.

[0058] In the storage destination registration process, the storage space of the i-th storage destination is photographed by the fixed camera 21a capable of photographing the storage space of the i-th storage destination or by any other camera. The photographed result is provided to the registration processing unit 11, which estimates the total storage volume of the i-th storage destination based on the photographed result and generates a 3D map of the i-th storage destination. The estimated result of the total storage volume of the i-th storage destination is saved as information 633. However, the information 633 may be set based on a user operation. The generated 3D map of the i-th storage destination is stored in the database DB_b as information 635 of the i-th storage destination.

[0059] In the deposit registration process, the residential area (which may be referred to as the location area hereinafter) in which each deposit is located is set based on a user operation. When the location area of ​​the jth deposit is included in the designated deposit area (corresponding to information 613 in FIG. 5) of the ith registrant, the jth deposit is associated with the ith registrant as one of the deposits for depositing the ith registrant's belongings (j is an integer). As a result, each deposit is associated with a certain registrant. One deposit may be associated with only one registrant, or one deposit may be associated with multiple registrants. Here, it is assumed that the first to fourth deposits are installed in Western-style rooms on the second floor, and the fifth and sixth deposits are installed in Western-style rooms on the first floor. When this information is input into the storage assistance device 1 by the user's operation, information 634 indicates that the location of the first to fourth storage locations is the second Western-style room (therefore the third living area; see FIG. 10), and that the location of the fifth and sixth storage locations is the first Western-style room (therefore the first living area; see FIG. 10).

[0060] 6 is set in the receiving destination registration process. Information 636 and 637 are set and updated in the actual operation process. Information 638 may include information set in the receiving destination registration process, or may include information set and updated in the actual operation process.

[0061] The item registration process will be described (see FIG. 7 as appropriate). In the item registration process, the registration processing unit 11 acquires the name of each item to be registered as a registered item based on a user operation in the item registration process, and sets information 651 of each registered item. The registration processing unit 11 sets information 652 by assigning an item ID to each registered item.

[0062] In the product registration process, the registration processing unit 11 acquires information 653 to 656 for each registered product and stores it in the product database DB_c.

[0063] In the item registration process, the i-th registered item is photographed by the fixed camera 21a capable of photographing the i-th registered item or by any other camera, and the registration processing unit 11 acquires information 653 to 656 based on the photographing results. The i-th registered item is photographed multiple times from different directions (viewpoints). A plurality of photographed images obtained by photographing the i-th registered item multiple times are included in the information 653 as an item image of the i-th registered item. The registration processing unit 11 generates information indicating the shape (e.g., three-dimensional shape) of the i-th registered item based on the multiple photographed images in the information 653 of the i-th registered item as shape information 654 of the i-th registered item, estimates the size (volume) of the i-th registered item, and includes the estimated result in the information 655 of the i-th registered item. Furthermore, the registration processing unit 11 determines the type of the i-th registered item based on the multiple photographed images in the information 653 of the i-th registered item, and sets the determination result of the type of the i-th registered item to the information 656 of the i-th registered item. A trained deep neural network (DNN) for determining the type of each registered article may be provided in the registration processing unit 11. The process of determining the type of the i-th registered article corresponds to the above-mentioned object classification process. However, at least one of the information 655 and 656 of the i-th registered article may be set based on a user operation.

[0064] In the product registration process, information 657 and 659 of each registered product is set based on a user operation. In the product registration process, the date on which a certain product was registered as the i-th registered product is set in information 658 of the i-th registered product.

[0065] In this way, information 651-659 of each registered item shown in Fig. 7 is set in the item registration process. Information 660 in Fig. 7 is set and updated in the actual operation process. Information 672-677 and 679 in Fig. 8 are also set and updated in the actual operation process (information 678 will be described later). Information 661 in Fig. 7 and information 680 in Fig. 8 may include information set in the item registration process, or may include information set and updated in the actual operation process.

[0066] 1 again, the functions of each functional block of the control unit 10 other than the registration processing unit 11 will be described. The object detection unit 12, tracking processing unit 13, voice response unit 14, and notification unit 15 operate beneficially in the actual operation process. Unless otherwise specified, the various operations described below are assumed to be operations after the initial registration process in step S3 is completed.

[0067] The object detection unit 12 performs object detection processing based on the captured image information SI21 of the camera unit 21, the information of each registered person (image information for image recognition) 614, and the information of each registered item (item image) 653. The object detection processing may be performed by further referring to the light detection information SI22. In the object detection processing, the position of each object in the sensing space is detected and each object in the sensing space is identified. The object detection processing may also include detection of the color of each object in the sensing space. In this embodiment, a person and an object belong to an object, and an object is an object that is not a person. In the object detection processing, it is identified whether an arbitrary target object in the sensing space is a person or an object, which registered person it is if it is a person, and which registered item it is if it is an item. When it is assumed that the target object is a person, the object detection processing corresponds to the above-mentioned person identification processing. Therefore, the object detection processing includes a person identification processing, and the object detection unit 12 is understood to include a person identification unit that performs a person identification processing. Assuming that the target object is an article, the object detection process corresponds to an article identification process that identifies whether the target object (target article) is any one of a plurality of registered articles. Therefore, the object detection process includes an article identification process, and the object detection unit 12 is understood to include an article identification unit that performs the article identification process.

[0068] The tracking processing unit 13 performs a tracking process for tracking the position of each object (including each registered person and each registered article) in the sensing space based on the results of the object detection processes that are executed sequentially.

[0069] The voice response unit 14 performs voice recognition processing to recognize the contents of the user's utterance collected by the microphone 43, and also executes response processing to the contents of the user's utterance. In the response processing, an image showing the contents of the response may be displayed on the display unit 41, or instead of or in addition to this, a voice showing the contents of the response may be output from the speaker 44. Although not shown in particular, the control unit 10 may be provided with a character input function, in which case the characters input by the user on the touch panel of the display unit 41 may be recognized by the control unit 10, and the control unit 10 may perform response processing based on the recognized characters.

[0070] The notification unit 15 provides various notifications to the user. Any notification to the user may be a notification by displaying an image showing the notification content on the display unit 41 (hereinafter referred to as image notification), or a notification by outputting a sound showing the notification content from the speaker 44 (hereinafter referred to as sound notification), or may include both image notification and sound notification. In the following description, a notification to the user by the control unit 10 may be understood as a notification by the notification unit 15.

[0071] Some information in the database DB will now be explained.

[0072] Please refer to Fig. 8 (also see Fig. 1). Through object detection processing by the object detection unit 12 and tracking processing by the tracking processing unit 13, each object in the sensing space is identified as a registered person or a registered article, and the position of each object is tracked. Based on the tracking result, the control unit 10 generates information 675 indicating the movement history of each registered article. The information 675 of the i-th registered article indicates the history of changes in the position (location) of the i-th registered article.

[0073] For example, consider the movement (position change) of an item 711 as shown in FIG. 11. The item 711 is an arbitrary registered item (i-th registered item). Consider a case in which the item 711 is taken out of the storage location 712 by the registrant 713 at time t1 on May 1, 2020, and moved from the storage location 712 together with the registrant 713, and then the item 711 is placed at a certain position 714 in the sensing space, and then, at time t2 on the same day, the item 711 is moved from the position 714 together with the registrant 713 and stored in the storage location 715. The fact F1 is recognized by the control unit 10 through object detection processing and tracking processing based on the captured image information SI21 and the light detection information SI22 between times t1 and t2, and the control unit 10 includes the contents of the recognized fact F1 in the information 675 of the i-th registered item as the item 711.

[0074] Furthermore, consider a case in which, after time t2, fact F2 exists in which item 711 is taken out of storage location 715 by registrant 716 at time t3 on May 1, 2020, moved from storage location 715 together with registrant 716, and then item 711 is placed at a certain position 717 in the sensing space, and then, at time t4 on the same day, item 711 is moved from position 717 together with registrant 716 and stored at storage location 718. Through object detection processing and tracking processing based on captured image information SI21 and light detection information SI22 between times t3 and t4, fact F2 is recognized by control unit 10, and control unit 10 includes the contents of recognized fact F2 in information 675 of the i-th registered item as item 711.

[0075] After that, every time the item 711 is moved, the details of the movement are included in the information 675 of the i-th registered item as the item 711. However, any movement history that is more than a predetermined period (e.g., one year) from the present time may be deleted from the information 675. The time indicated in the movement history of each registered item may be the time acquired by a clock unit (not shown) in the storage assistance device 1, or may be the time acquired by a clock unit (not shown) of any other device (the same applies to the usage history described below).

[0076] The control unit 10 can generate information 672 (see FIG. 8) indicating the current storage destination of the i-th registered item using the results of the object detection process and the tracking process. In the example shown in FIG. 11, the current storage destination of the i-th registered item as the item 711 is storage destination 712 before time t1, storage destination 715 from after time t2 to just before time t3, and storage destination 718 after time t4. Note that, during a period when the i-th registered item is not stored in any storage destination (for example, a period when the item 711 as the i-th registered item is placed at position 714 or 717), flag information of "in use" is stored in the information 672 of the i-th registered item instead of the current storage destination. Note that, during a period when the tracking process of the item 711 is being performed, specifically, during a period when the item 711 is placed at position 714 or 717, captured image information of the item 711 and captured image information around the item 711 may be acquired and stored in the recording unit 30.

[0077] The information 673 of the i-th registered article indicates the user of the i-th registered article as described above. The user of the i-th registered article is a person who has used the i-th registered article, and a person who has been in continuous contact with the i-th registered article for a certain period of time or more may be regarded as the user of the i-th registered article. In the control unit 10, it is determined whether each registered person is the user of the registered article for each registered article based on the captured image information SI21 and the light detection information SI22. Among the users of the i-th registered article, the last user of the i-th registered article (the person who last used the i-th registered article) is the final user of the i-th registered article, and the final user of the i-th registered article is indicated by the information 674 of the i-th registered article. In the example of FIG. 11, the final user of the i-th registered article as the article 711 is the registered person 713 immediately before time t3, and is the registered person 716 after time t4. However, the registered persons 713 and 716 may be the same registered person or different registered persons.

[0078] As described above, the information 660 of the i-th registered article is usage history information (contents not shown in FIG. 7) that indicates the usage history of the i-th registered article. The usage history of the i-th registered article includes the usage rate, usage frequency, and usage timing of the i-th registered article. The control unit 10 considers the i-th registered article to be in a non-used state when the i-th registered article is stored in any storage location, and considers the i-th registered article to be in a used state when the i-th registered article is removed from any storage location. A period during which any registered article is in a used state is called a usage period, and a period during which any registered article is in a non-used state is called a non-used period. The usage rate of the i-th registered article is the ratio of the usage period of the i-th registered article to the sum of the usage period and non-used period of the i-th registered article. The usage frequency of the i-th registered article represents the number of times the i-th registered article has been switched from a non-used state to a used state during a period of interest having a certain length of time. The usage timing of the i-th registered article represents the date and time of the usage period of the i-th registered article. The period from a predetermined period (eg, one year) before the present time to the present time is set as the usage history calculation period, and the usage history of the i-th registered item during the usage history calculation period is included in information 660 of the i-th registered item.

[0079] Referring again to Fig. 8, the discard suggested flag based on the information 676 of the i-th registered item is a flag indicating whether or not the i-th registered item is a suggested discarded item. The discarded estimated flag based on the information 677 of the i-th registered item is a flag indicating whether or not the i-th registered item is an estimated discarded item. The consumables flag based on the information 678 of the i-th registered item is a flag indicating whether or not the i-th registered item is a consumables item. The method of using these flags will be described later.

[0080] The information 679 of the i-th registered item indicates the state (item state) of the i-th registered item. Regarding the i-th registered item, the information 679 may indicate whether the i-th registered item is normal, broken, or damaged, or may indicate the degree of consumption of the i-th registered item if the i-th registered item is a consumable item. The control unit 10 can generate and update the information 679 of the i-th registered item based on the captured image information SI21 including the result of capturing an image of the i-th registered item, etc.

[0081] 6, the control unit 10 can derive the estimated remaining volume of the ith storage destination as information 636 of the ith storage destination based on the total storage volume (633) of the ith storage destination and the sum of the volumes of the registered items currently stored in the ith storage destination. When deriving this, information 655 and 672 are referenced (see FIGS. 7 and 8).

[0082] 12 shows the display contents of the display unit 41 in an actual operation process. Here, it is assumed that a touch panel is formed on the display screen of the display unit 41. Icons B1 to B6 are displayed on the display screen of the display unit 41. When the user presses any of the icons B1 to B6, the control unit 10 executes a process corresponding to the pressed icon.

[0083] This embodiment includes the following Examples EX1_A, EX1_B, and EX2 to EX11. In these Examples, the specific operation of the storage assistance device 1 will be described. In each of the Examples described later, if there is any matter that contradicts the matters described above, the description in each Example may take precedence. Furthermore, unless there is a contradiction, matters described in any of the following Examples can also be applied to any of the other Examples (i.e., any two or more of the multiple Examples can also be combined).

[0084] [Example EX1_A] Example EX1_A will be described. When the user presses icon B1 (see FIG. 12) (when a first instruction is received from the user), the control unit 10 executes the tidying up support process. Even if icon B1 is not pressed, the tidying up support process may be executed by the user speaking a keyword related to tidying up. The task (action) to be performed by the user, which is to store each item that is not stored in its storage destination in its storage destination, is called the tidying up task. The tidying up support process supports the user in tidying up the task.

[0085] Now, consider a state ST1 in which an object OBJ_11 is not stored in any storage location, as shown in Fig. 13. In state ST1, an object OBJ_11, which is a registered object, is placed on any floor in the living space. Any registered person can be selected to perform the tidying up task, but here, it is assumed that a third registered person, who is a child, is the person to perform the tidying up task.

[0086] During the execution period of the tidying support process, the third registrant picks up the item OBJ_11 and makes an inquiry 810 to the storage support device 1 as to where the item OBJ_11 should be stored (see FIG. 14). The inquiry 810 may be a spoken inquiry (for example, a spoken inquiry such as "Where should I put this toy?") or an inquiry made by operating a touch panel or an operating member other than a touch panel (the same applies to any inquiry described below). The inquiry may also be made by another person (for example, the first or second registrant). When making a spoken inquiry, it is assumed that the microphone 43 is located in a position where it can pick up the contents of the speaker's speech (the same applies to any inquiry described below).

[0087] Upon receiving the inquiry 810, the control unit 10 recognizes that the item that is the subject of the inquiry is item OBJ_11 based on the captured image information SI21, and identifies which registered item item OBJ_11 is based on the result of object detection processing on item OBJ_11. The control unit 10 notifies the user of storage assistance information that supports the storage of item OBJ_11 based on the identification result and database DB (see FIG. 14). Users to whom the notification is to be sent include the sender of the inquiry (here, the third registered user), and may also include other registered users.

[0088] The notification of storage assistance information assists in proper storage of each item (e.g., efficient storage is supported).

[0089] The notification of the storage assistance information preferably includes a visual notification, which promotes proper storage of each item. The visual notification includes displaying the storage assistance information on the display unit 41. As described above, the display unit 41 may be a display unit of a smartphone, a tablet terminal, or the like.

[0090] The notification of the storage assistance information may be a notification that clearly indicates the prescribed storage destination of the item OBJ_11. For example, if the item OBJ_11 is the first registered item and the prescribed storage destination of the first registered item is the first storage destination indicated by the storage destination ID "bb[1]" (see Figs. 6 and 7), a notification clearly indicating that the prescribed storage destination of the item OBJ_11 is the first storage destination may be sent as the notification of the storage assistance information (for example, a voice or image notification saying "Put it away in the green toy box" is sent). The same applies when the item OBJ_11 is a registered item other than the first registered item.

[0091] The notification of the storage assistance information may include route guidance that guides the movement route from the location of the user (here, the third registrant) who made the inquiry to the specified storage destination of the item OBJ_11. In the route guidance, an image showing the movement route is displayed on the display unit 41.

[0092] Furthermore, during the execution period of the tidying up support process, when it is detected based on the captured image information SI21 that the third registered person has picked up the item OBJ_11, even if the above inquiry is not made, the control unit 10 may make a notification similar to that when the above inquiry is made.

[0093] When notifying the user of storage assistance information to assist in storing item OBJ_11, the control unit 10 may notify (e.g., explicitly indicate) the specified storage destination for item OBJ_11, and may also notify (present) the user of a recommended storage method for storing item OBJ_11 in the specified storage destination.

[0094] Notification of the recommended storage method will be described assuming that the item OBJ_11 is the first registered item and the specified storage destination of the item OBJ_11 is the first storage destination.

[0095] In the notification of the recommended storage method, for example, a judgment index Q1 including a 3D map (635; see FIG. 6) of the first storage destination, which is the specified storage destination of the item OBJ_11, a volume of the item OBJ_11 (655; see FIG. 7), and shape information of the item OBJ_11 (654; see FIG. 7) is used. EX1_A The control unit 10 determines the judgment index Q1 EX1_A Based on this, the control unit 10 determines which position in the specified storage location of the item OBJ_11 is best for storing the item OBJ_11, and notifies the user (third registrant) of the result of the determination as a notification of a recommended storage method. It is preferable that the control unit 10 is equipped with a DNN that has undergone machine learning to realize such a determination. In the notification of the recommended storage method, the control unit 10 uses the above-mentioned determination index Q1 EX1_A A moving image based on the above is displayed on the display unit 41, and the moving image provides guidance as to where and how the article OBJ_11 should be stored in the first storage destination.

[0096] Also, for example, within state ST1, assume a state ST1a in which the control unit 10 recognizes that one or more other registered items are already stored in the first storage location (the control unit 10 can recognize this from information 672 of each registered item).

[0097] In state ST1a, a judgment index Q2 including a 3D map (635; see FIG. 6) of the first storage destination which is the specified storage destination of the item OBJ_11, a volume (655; see FIG. 7) of the item OBJ_11, shape information (654; see FIG. 7) of the item OBJ_11, a volume (655; see FIG. 7) of the one or more other registered items, and shape information (654; see FIG. 7) of the one or more other registered items. EX1_A is referred to. The control unit 10 determines the judgment index Q2 EX1_A Based on this, the control unit 10 determines which position in the specified storage location of the item OBJ_11 is best for storing the item OBJ_11, and notifies the user (third registrant) of the result of the determination as a notification of a recommended storage method. It is preferable that the control unit 10 is equipped with a DNN that has undergone machine learning to realize such a determination. In notifying the user of the recommended storage method, the control unit 10 uses the above determination index Q2 EX1_AA moving image based on the above is displayed on the display unit 41, and the moving image provides guidance as to where and how to store the item OBJ_11 in the first storage destination. In this case, if it is recommended to store the item OBJ_11 in the back of the specified storage destination, a moving image showing the state in which the other registered items already stored in the specified storage destination are temporarily removed from the specified storage destination, the item OBJ_11 is stored in the recommended position within the specified storage destination, and then the other registered items that have been removed are stored again in the specified storage destination is generated and displayed on the display unit 41.

[0098] Each of the above moving images for notifying the user of the recommended storage method may be a moving image that utilizes Augmented Reality (AR).

[0099] Notification (presentation) of the recommended storage method helps to properly store each item. In particular, when the person who should be tidying up is a small child, it may be difficult for them to properly tidy up by themselves. Notification of the recommended storage method is expected to have the effect of teaching children the proper way to tidy up.

[0100] Although the focus has been on one item OBJ_11, when a tidying up operation is to be performed on a plurality of items, the same processing and notification as for the item OBJ_11 may be performed for each item (registered item).

[0101] [Example EX1_B] Example EX1_B will be described. Example EX1_B shows another specific example of the tidying up support process. As in Example EX1_A, consider state ST1 in FIG. 13, and assume that the third registrant, who is a child, is the person who should perform the tidying up work. As shown in FIG. 14, during the execution period of the tidying up support process, the third registrant picks up the item OBJ_11 and makes an inquiry 810 to the storage support device 1 about where to store the item OBJ_11 (for example, an utterance of "Where should I put this toy?").

[0102] Upon receiving the inquiry 810, the control unit 10 recognizes that the item being the subject of the inquiry is item OBJ_11 based on the captured image information SI21. Furthermore, the control unit 10 identifies which registered item the item OBJ_11 is based on the result of the object detection process for the item OBJ_11, and notifies the user of storage support information that supports the storage of the item OBJ_11 based on the identification result and the database DB. However, when notifying the user of the storage support information, the control unit 10 according to the embodiment EX1_B notifies the user of the prescribed storage destination of the item OBJ_11 by presenting a game to the user (here, the third registrant).

[0103] For example, in a game, some task is presented to the user (here, the third registered person) by image display on the display unit 41 or audio output from the speaker 44, and when the task is accomplished, the specified storage destination of the item OBJ_11 is clearly indicated to the user (here, the third registered person) to notify him / her. However, even if the task is not accomplished, the specified storage destination of the item OBJ_11 may be clearly indicated to the user (here, the third registered person).

[0104] A quiz game can be used as a specific example of the game. FIG. 15 shows an example of a quiz game corresponding to state ST1. In the quiz game, the control unit 10 displays an image 820 on the display unit 41, in which a plurality of options are shown as candidates for storage destinations for storing the item OBJ_11. The control unit 10 then prompts the user (here, the third registrant) to perform a selection operation to select a storage destination suitable for storing the item OBJ_11 (i.e., the default storage destination of the item OBJ_11). The image 820 includes text characters indicating each option and an image showing the appearance of the storage destination of each option. Only one of the multiple options corresponds to the default storage destination of the item OBJ_11. In other words, a question in which the default storage destination of the item OBJ_11 is the correct answer is presented to the user (here, the third registrant). The user (here, the third registrant) answers the question by selecting any one of the multiple options through a selection operation. The control unit 10 causes the display unit 41 to display (display an image) in accordance with this response, thereby allowing the user (here, the third registrant) to recognize and be notified of the specified storage destination of the item OBJ_11.

[0105] When an option indicating the specified storage destination of the item OBJ_11 is selected in the user's selection operation on the interface unit 40, the control unit 10 gives a correct response, and when an option indicating the specified storage destination of the item OBJ_11 is not selected, the control unit 10 gives an incorrect response. The correct response includes a correct response process that displays on the display unit 41 an image (video) indicating that the user's selection is correct, and through the correct response process, the specified storage destination of the item OBJ_11 is clearly indicated to the user. The incorrect response includes an incorrect response process that displays on the display unit 41 an image (video) indicating that the user's selection is incorrect. In the incorrect response, the correct option, i.e., the specified storage destination of the item OBJ_11, is clearly indicated to the user together with or via the incorrect response process.

[0106] Although a quiz game has been given as a specific example of a game, the content of the game is arbitrary. For example, a so-called shooting game may be adopted, and the user may be notified of the specified storage destination of the object OBJ_11 when a certain condition is achieved in the shooting game.

[0107] In particular, for children, it is necessary to let them experience tidying up many times without getting bored until they acquire the habit of tidying up. By incorporating a game-like element into the notification of storage support information, it is expected that children will enjoy tidying up. In other words, it is possible to teach children how to tidy up properly while having fun. The same applies to adults, who can also enjoy tidying up.

[0108] In the case of a correct answer or an incorrect answer, before the item OBJ_11 is actually stored in any of the storage destinations, a moving image showing the state in which the item OBJ_11 is stored in the storage destination selected by the selection operation may be displayed on the display unit 41 using augmented reality as appropriate. This allows the user (here, the third registrant) to reconsider whether or not it is OK to store the item OBJ_11 in the storage destination selected by the selection operation (whether or not it is correct to store the item OBJ_11 in the storage destination selected by the selection operation). This is useful for mastering the correct tidying up.

[0109] Incidentally, the other information 616 of the registrant may include game points. When a correct response is made, the control unit 10 adds a first score to the game points of the user who played the game, and when an incorrect response is made, the control unit 10 adds a second score to the game points of the user who played the game. The user who played the game is the third registrant in this case. The first score is greater than the second score. The second score may be zero. A mechanism for giving some kind of privilege to the third registrant when the game points reach a certain value or more may be incorporated in the storage assistance device 1, or a mechanism for giving some kind of privilege to the third registrant may be agreed upon between the residents of the residence HH.

[0110] When the user is notified of the specified storage destination of the item OBJ_11, a recommended storage method may also be notified as in the embodiment EX1_A. Although the focus is on one item OBJ_11, when multiple items need to be put away, the same processing and notification as for the item OBJ_11 may be performed for each item (registered item).

[0111] Furthermore, in the tidying up support process, after the specified storage destination of the item OBJ_11 is notified or recognized by the user (here, the third registered person), the control unit 10 may also play a game until the item OBJ_11 is actually stored in the specified storage destination. For example, the following tidying up game may be played. In the tidying up game, the control unit 10 notifies the user (here, the third registered person) of the specified storage destination of the item OBJ_11, then notifies the user (here, the third registered person) of the start of tidying up, and thereafter measures the time it takes for the item OBJ_11 to be stored in the specified storage destination (hereinafter referred to as tidying up time). The control unit 10 can detect whether the item OBJ_11 has been stored in the specified storage destination based on image information captured by one or more fixed cameras 21a that capture images of the item OBJ_11 and the specified storage destination.

[0112] Furthermore, in the tidying game, the control unit 10 sets an ideal storage state for the state (storage position, storage direction, etc.) of storing the item OBJ_11 in the specified storage destination. In the tidying game, the control unit 10 specifies the actual storage state of the item OBJ_11 in the specified storage destination based on the captured image information of one or more fixed cameras 21a that capture the item OBJ_11 and the specified storage destination, and compares the specified state with the ideal storage state. In the tidying game, the control unit 10 derives a tidying result score that indicates the quality of the actual tidying work of the item OBJ_11 by scoring the comparison result and the above-mentioned tidying time, and notifies the user (here, the third registered user) of the tidying result score. If the tidying result score satisfies a predetermined condition, the game is cleared, and the control unit 10 may give the user (here, the third registered user) some kind of reward (such as displaying a moving image that is likely to interest the user on the display unit 41). The tidying result score may be added to the above-mentioned game points. The same applies to items other than item OBJ_11.

[0113] [Example EX2] Example EX2 will be described. When the user presses icon B2 (see FIG. 12) (when a second instruction is received from the user), the control unit 10 executes the item addition process. Even if icon B2 is not pressed, the item addition process may be executed by the user speaking a keyword related to item addition. After multiple registered items are set by the item registration process in the initial registration step, there are cases where it is desired to newly register another item as a registered item in the storage assistance device 1. This other item to be newly registered is referred to as a new item. Through the item addition process, the new item is incorporated into the multiple registered items in the storage assistance device 1 (is added to the multiple registered items).

[0114] The item addition process is executed by the registration processing unit 11. Assume that a total of n registered items are currently registered in the storage assistance device 1 and a new item is to be registered (n is an integer equal to or greater than 2). In this case, the new item becomes the (n+1)th registered item through the item addition process.

[0115] In the item addition process, the registration processing unit 11 acquires the name of the new item based on a user operation in the item addition process, and sets the acquired name to the (n+1)th registered item information 651. The registration processing unit 11 assigns a unique item ID to the new item, thereby setting the (n+1)th registered item information 652.

[0116] In the item addition process, the registration processing unit 11 acquires information 653 to 656 on the (n+1)th registered item as a new item, and stores it in the item database DB_c.

[0117] In the item addition process, the new item is photographed by the fixed camera 21a capable of photographing the new item or by any other camera, and the registration processing unit 11 acquires information 653-656 of the (n+1)th registered item based on the photographing results. The new item is photographed multiple times from different directions (viewpoints). The multiple photographed images obtained by photographing the new item multiple times are included in the information 653 as item images of the (n+1)th registered item. The registration processing unit 11 generates information indicating the shape (e.g., three-dimensional shape) of the (n+1)th registered item as shape information 654 of the (n+1)th registered item based on the multiple photographed images in the information 653 of the (n+1)th registered item, and estimates the size (volume) of the (n+1)th registered item and includes the estimation result in the information 655 of the (n+1)th registered item. Furthermore, the registration processing unit 11 determines the type of the (n+1)th registered article based on the multiple captured images in the (n+1)th registered article information 653, and sets the determination result of the type of the (n+1)th registered article in the (n+1)th registered article information 656. At least one of the (n+1)th registered article information 655 and 656 may be set based on a user operation.

[0118] In the item addition process, the owner of the new item is set based on a user operation, and the set owner of the new item is set in the (n+1)th registered item information 657. In the item addition process, the date on which the new item was registered as the (n+1)th registered item is set in the (n+1)th registered item information 658.

[0119] In the item addition process, after information 651 to 658 of the (n+1)th registered item as a new item is set based on a user operation, the registration processing unit 11 can execute the following prescribed storage destination suggestion process.

[0120] In the default storage destination suggestion process, a storage destination (hereinafter referred to as a suggested storage destination) where the new item should be stored is selected (set) from among the multiple storage destinations based on the photographed image of the new item included in the (n+1)th registered item information 653 and information on the multiple storage destinations registered in the database DB. The number of registered persons may be one, but when there are multiple registered persons as in this embodiment, the suggested storage destination is selected from one or more storage destinations associated with the owner of the new item. The suggested storage destination is notified to the user (for example, the user who pressed icon B2) by the control unit 10.

[0121] For example, referring to FIG. 16, if the owner of the new item is the i B A registrant who is the i B jth to the registrant B ~th(j B +3) Consider state ST2 where only the storage destination is associated (i B and j B In the state ST2, the jth B ~th(j B +3) one of the storage destinations is selected as the proposed storage destination. Upon receiving the notification of this selection, the user (basically, the i B When the (n+1)th registered item is registered by the (n+2)th registered item (registered user) and the (n+3)th registered item is registered, the (n+1)th registered item is registered in the (n+2)th registered item. When the (n+1)th registered item is registered, the (n+3)th registered item is registered in the (n+1)th registered item.

[0122] However, the jth B ~th(j B Of the (n+3) storage destinations, two or more storage destinations may be selected as suggested storage destinations. In this case, the two or more suggested storage destinations are notified to the user, and the user specifies one of the two or more suggested storage destinations by user operation. Then, the specified suggested storage destination is set in information 659 (see FIG. 7) as the default storage destination for the (n+1)th registered item.

[0123] By referring to the photographed image of the new item, it becomes possible to select a preferable storage location for the new item, which makes it easier to organize and tidy up the items.

[0124] In the prescribed storage destination suggestion process, for example, the registration processing unit 11 can select a suggested storage destination based on the volume (655) of the new item based on the photographed image of the new item and the estimated remaining volume (636) of each storage destination associated with the owner of the new item. In state ST2, if the volume of the new item is the jth storage destination, B ~th(j B +2) is larger than the estimated remaining capacity of each storage destination and is the (jth B +3) If it is smaller than the estimated remaining capacity of the storage destination, B +3) Select the storage location as the suggested storage location.

[0125] When selecting the proposed storage destination, shape information (654) of the new item based on the photographed image of the new item may be taken into consideration. In state ST2, the registration processing unit 11 registers the three-dimensional shape of the new item based on the photographed image of the new item and the jth B ~th(j B +3) By referring to each 3D map (635) of the storage destination, B ~th(j B +3) From among the storage locations, a storage location suitable for storing new items can be selected as a suggested storage location.

[0126] When selecting a proposed storage destination, the environmental information (637) of each storage destination may be referred to. In state ST2, the registration processing unit 11 judges the temperature or humidity recommended for storing the new item from the result of the object classification process for the new item. The registration processing unit 11 compares the result of the judgment with the jth storage destination. B ~th(j B +3) Based on the storage environment information (637), the jth B ~th(j B +3) From among the storage locations, a storage location suitable for storing new items can be selected as a suggested storage location.

[0127] In the default storage destination suggestion process, the suggested storage destination may be selected and notified taking into consideration the preference information of the owner of the new item. The control unit 10 learns what types of items each registered person tends to store in which storage destination, and generates preference information (615) for each registered person based on the learning results. For the sake of concrete explanation, state ST2 in FIG. 16 will be considered again. The preference information considered in state ST2 is the preference information of the i-th owner of the new item. B This is the registrant's preference information (615; see FIG. 5).

[0128] Before the proposed storage location process for the new item is performed, B Of the registered goods owned by the registrant, all or almost all of them are classified as stationery. B The storage destination is set to the specified storage destination and the jth B All or almost all of the items stored in the storage facility and classified as books are B +1) The destination is set to the default destination and the (j B All or almost all of the items stored in the +1 storage location and classified as electronic equipment are classified as j B +2) The storage destination is set to the specified storage destination and the (j B +2) All or almost all of the items classified as sports equipment are stored in the storage area. B +3) The storage destination is set to the default storage destination and the (j B +3) It is assumed that the item is stored in the storage location. Based on this storage situation, the control unit 10 B The registrants may purchase stationery, books, electronic devices, and sports equipment under the jth B ,th(j B +1), th (j B +2), th (j B +3) Preference information that indicates a tendency to store items in the i B In this case, for example, if the new item is classified as a sports item in the object classification process, the registration processing unit 11 acquires the i-th B Based on the registrant's preference information 615 (i.e., according to the above tendency), B +3) It is possible to select the storage location as a suggested storage location for new items.

[0129] By selecting a suggested storage destination taking into consideration the preference information of the owner of the new item, it is possible to suggest a suggested storage destination that matches the preference of the owner of the new item.

[0130] In the item addition process, a default storage destination for a new item may be set based on a user instruction, regardless of the default storage destination suggestion process.

[0131] User (here, the i B When a user (a registered user) designates a specific storage destination (a storage destination that the user considers desirable) as the default storage destination for a new item, the control unit 10 basically sets the default storage destination for the new item according to the designation. However, depending on the storage status of the specific storage destination or the size of the new item, it may not be possible to store the new item in the specific storage destination. In such a case, the control unit 10 may perform the following replacement proposal process. The replacement proposal process will be described assuming state ST2 in FIG. 16.

[0132] Now, the above specific storage location is jth B The jth B If the volume of the new item (655) is larger than the estimated remaining volume of the storage destination, the control unit 10 determines that the new item cannot be stored. B Based on the 3D map of the storage location and the shape information of the new item (654), B The control unit 10 also determines that the new item cannot be stored when it is determined that the new item cannot be contained in the storage space of the storage destination. For example, if the new item is a long item and the total length of the new item is too large compared to the size of the storage space of the i-th storage destination, it is determined that the new item cannot be stored.

[0133] When the control unit 10 determines that the bag cannot be stored, B +1)~th(j B +3) Estimated remaining capacity of storage location and (j B +1)~th(j B +3) Based on the 3D map of the storage location, the storage location that can store the new item is selected as a replacement candidate. B +1)~th(j B+3) Select from the storage destination. Two or more replacement candidates may be selected.

[0134] After extracting the replacement candidates, the control unit 10 B The user (in this case, the i-th item) is informed that the new item cannot be stored in the storage location. B The control unit 10 notifies the user (a registered user) of the replacement candidate and notifies the user of the replacement candidate. If there is one replacement candidate, the notification of the replacement candidate suggests to the user that the replacement candidate be set as the default storage destination for new items. If a user operation indicates agreement with this suggestion, the control unit 10 sets the replacement candidate as the default storage destination for new items and includes the setting result in the item database DB_c. If there are multiple replacement candidates, the control unit 10 notifies the user that any of the multiple replacement candidates can be set as the default storage destination for new items. Upon receiving this notification, the user selects any one of the multiple replacement candidates through a user operation. Then, the control unit 10 sets the selected replacement candidate as the default storage destination for new items and includes the setting result in the item database DB_c.

[0135] When the control unit 10 sets the replacement candidate as the default storage destination for the new item, the control unit 10 notifies the user (the i B The user (the i-th user in this example) may be informed of how to store the new item in a specified storage location. B The notification may include route guidance for guiding the movement route from the current location of the registered user (registered user) to the specified storage location of the new item, or may include notification of the recommended storage method as described above. The above guidance may also be provided by any specified storage location suggestion process.

[0136] Here, an explanation will be given on the volume of an object. For convenience of explanation, any object that has been focused on will be referred to as an object of attention, and the volume of the object of attention will be explained. The object of attention is a registered object. However, the object of attention may be an object other than a registered object. First, the reference interpretation method will be explained. In the reference interpretation method, the volume of the object of attention is literally the actual volume of the object of attention, and the actual volume of the object of attention (for example, an estimated result of the actual volume of the object of attention) is included in the object database DB_c as information 655 of the object of attention. For example, if the object of attention is an object having a hollow part such as a bucket, the volume of the object of attention does not include the volume of the hollow part. The control unit 10 according to this embodiment, including Example EX2, employs the reference interpretation method.

[0137] However, the control unit 10 may adopt a method of assuming the smallest rectangular parallelepiped that entirely encloses the target item, and regarding the volume of the assumed rectangular parallelepiped as the volume of the target item (hereinafter referred to as the deformation interpretation method). When the deformation interpretation method is adopted, if the target item is the bucket, the volume of the target item includes the volume of the hollow portion. When the deformation interpretation method is adopted, the volume of the target item under the deformation interpretation method is included in the item database DB_c as the target item information 655. When the deformation interpretation method is adopted, an image is created in the control unit 10 of filling a 3D map of a certain target storage destination with registered items assumed to be rectangular parallelepipeds. Then, the control unit 10 determines whether a new item can be further stored in the target storage destination.

[0138] [Example EX3] Example EX3 will be described. When the user presses icon B3 (see FIG. 12) (when a third instruction is received from the user), the control unit 10 executes an item search process. Even if the icon B3 is not pressed, the item search process may be executed by the user speaking a keyword related to the item search.

[0139] Referring to FIG. 17, in the embodiment EX3, it is assumed that a user 850 is searching for an arbitrary registered item (hereinafter referred to as a search target item). The user 850 may be any registered person. During the execution period of the item search process, the user 850 makes an inquiry 860 about the location of the search target item (i.e., an inquiry about where the search target item is currently located) to the storage assistance device 1. It is assumed that first to nth registered items are registered in the database DB as registered items (n is an integer of 2 or more). The inquiry 860 includes information for estimating or recognizing which of the first to nth registered items the search target item is, and based on that information, the control unit 10 identifies which of the first to nth registered items the search target item is.

[0140] For example, the name or item ID of the search target item may be included in the inquiry 860, and in this case, the above-mentioned identification may be possible immediately. Each registered item may be displayed in a list on the display unit 41, and the search target item may be selected from among them based on a user operation. Also, for example, a conversation may be held between the user 850 and the control unit 10 by voice or the like, and the user 850 may input (transmit) the characteristics of the search target item to the storage assistance device 1 during the conversation. Based on the input characteristics, the control unit 10 may narrow down candidates for the search target item from among the first to nth registered items, and finally identify which of the first to nth registered items the search target item is. When identifying which of the first to nth registered items the search target item is, the control unit 10 may display images (653; see FIG. 7) of one or more registered items as candidates for the search target item on the display unit 41, and ask the user 850 a confirmation inquiry such as "Is this the one you are looking for?".

[0141] The object detection process and tracking process based on the captured image information SI21 of the camera unit 21 capturing the image of each registered article tracks each registered article in the sensing space (specifically, the position and movement trajectory of each registered article are tracked), and information based on the tracking result is included in the database DB. As described above, the object detection process and tracking process may also utilize the light detection information SI22 using the LIDAR 22a. The information based on the tracking result includes the information 672 and 675 shown in FIG. 8. The control unit 10 responds 862 to the inquiry 860 based on the information (672, 675) based on the tracking result of the search target item (see FIG. 17). The response 862 to the inquiry 860 includes a response by display on the display unit 41, and may further include a response by audio output from the speaker 44 (the same applies to any other response described later).

[0142] This makes it easier for users to find what they are looking for.

[0143] The item being searched for is C Explain the example of response 862 assuming that the item is a registered item (i C (n is a natural number equal to or less than n). The control unit 10 C Based on the information of registered goods 672 or 675, C The current location of the registered item is included in the response 862. Basically, the control unit 10 C The i-th item indicated in the registered item information 672 (see FIG. 8) C The current storage location of the registered item is included in the contents of the response 862. C When the registered item information 672 stores flag information of "in use", the i-th C Based on information 675 indicating the movement history of the registered goods, CThe current location of the registered item is included in response 862. For example, if the search target item is item 711 shown in Fig. 11 and inquiry 860 is made when item 711 is stored in storage location 715, response 862 will include the fact that the search target item (711) is in storage location 715. Also, for example, if the search target item is item 711 shown in Fig. 11 and inquiry 860 is made when item 711 is at location 717, response 862 will include the fact that the search target item (711) is at location 717.

[0144] Incidentally, by including the movement history information 675 in the database DB, even if the true location of the searched item becomes unknown to the storage assistance device 1 due to some circumstances (such as a tracking failure), the searched item may be found by tracing the movement history in the information 675. The control unit 10 can notify the user of the movement history information (675) of each registered item in response to a user operation at any timing.

[0145] [Example EX4] Example EX4 will be described. When the user presses icon B4 (see FIG. 12) (when a fourth instruction is received from the user), the control unit 10 executes a disposal suggestion process (item organization process). Even if icon B4 is not pressed, the disposal suggestion process may be executed by the user speaking a keyword related to item disposal or item organization.

[0146] In the disposal suggestion process, the control unit 10 notifies the user of a suggestion to discard one or more registered items (hereinafter, referred to as a disposal suggestion notice). When making the disposal suggestion, the disposal suggestion flag according to the information 676 is referred to (see FIG. 8). As described above, the disposal suggestion flag according to the information 676 of the i-th registered item is a flag indicating whether the i-th registered item is a disposal suggested item. The control unit 10 assigns a value of "1" or "0" to the disposal suggestion flag (676) of the i-th registered item based on the usage history information 660 of the i-th registered item. The initial value of the disposal suggestion flag of each registered item is "0". A disposal suggestion flag of "1" for the i-th registered item indicates that the i-th registered item is a disposal suggested item. A disposal suggested item is an item that is suggested to be discarded, and therefore the disposal of the disposal suggested item is suggested in the disposal suggestion notice. In other words, in the disposal suggestion notice, the disposal of a registered item whose corresponding disposal suggestion flag is "1" is suggested to be discarded to the user.

[0147] In this way, by identifying items suggested for disposal based on the usage history information (660) and making disposal suggestions, for example, disposal of items that are hardly used is encouraged, leading to improved organization and tidying of the residence HH.

[0148] For example, when the usage rate or frequency of the i-th registered article based on the usage history information 660 is equal to or lower than a predetermined value, the control unit 10 assigns "1" to the disposal suggestion flag for the i-th registered article. Alternatively, for example, when it is determined based on the usage history information 660 that the i-th registered article has been kept in an unused state for a certain period of time or more, the control unit 10 may assign "1" to the disposal suggestion flag for the i-th registered article.

[0149] In addition to the usage history information 660, the control unit 10 may also consider the registration date information 658 (see FIG. 7) when determining the value of the disposal suggestion flag and notifying the disposal suggestion. That is, for example, when the registration date of the i-th registered item is a predetermined number of days (e.g., 365 days x 2) or more from the present, and the usage rate or frequency of the i-th registered item based on the usage history information 660 is equal to or less than a predetermined value, the control unit 10 may assign "1" to the disposal suggestion flag for the i-th registered item. Alternatively, for example, when the registration date of the i-th registered item is a predetermined number of days (e.g., 365 days x 2) or more from the present, and it is determined based on the usage history information 660 that the i-th registered item has been kept in a non-used state for a certain period of time or more, the control unit 10 may assign "1" to the disposal suggestion flag for the i-th registered item.

[0150] This, for example, encourages the disposal of old and rarely used items, leading to tidying up the home HH.

[0151] In the disposal proposal processing, the control unit 10 may make the disposal proposal notification based on the usage history information 660, or based on the usage history information 660 and the registration date information 658, while also taking into consideration the preference information 615 of each registrant (see FIG. 5).

[0152] This allows, for example, suggestions for disposal of items that do not match the owner's preferences (realizing more reasonable disposal suggestions).

[0153] For example, if the first to third toys are classified into a first type and the fourth to sixth toys are classified into a second type by the object classification process, the first to sixth toys are each classified into the i D1 ~I D6 Assume that the object was registered in the database on April 1, 2020 (i D1 ~i D6are different natural numbers). The first type and the second type are different from each other. The owner of the first to sixth toys is the third registrant. It is assumed that the third registrant played with all of the first to sixth toys from April 1, 2020 to April 30, 2020, but used only the first to third toys (played only with the first to third toys) and did not use the fourth to sixth toys (did not play with the fourth to sixth toys) from May 1, 2020 to December 1, 2020. During the period from April 1, 2020 to December 1, 2020, the control unit 10 recognizes the above situation based on the captured image information SI21, or based on the captured image information SI21 and the light detection information SI22, and generates the preference information 615 of the third registrant based on the recognition result. As a result, the preference information 615 of the third registrant as of December 1, 2020 includes information that the third registrant prefers the first type of toy and does not prefer the second type of toy.

[0154] In this case, for example, on December 2, 2020, the control unit 10 performs the i-th D1 ~I D3 The value of each discard proposal flag of the registered object is maintained at “0”, while the i-th D4 ~I D6 It is possible to assign "1" to each disposal proposal flag of a registered object.

[0155] However, if a registrant other than the third registrant D4 ~I D6 Since there are cases where registered objects are used, it is advisable to control the value of the disposal suggestion flag for each registered object by referring to not only the preference information 615 but also the usage history information 660 (and further to the registration date information 658). In summary, for example, a method can be adopted in which the control unit 10 derives a disposal suggestion evaluation value for the i-th registered object based on the usage history information 660 for the i-th registered object, the registration date information 658 for the i-th registered object, and the preference information 615 of the owner of the i-th registered object, and when the disposal suggestion evaluation value reaches a predetermined value or more, "1" is assigned to the disposal suggestion flag for the i-th registered object.

[0156] [Example EX5] Example EX5 will be described. When the user presses icon B5 (see FIG. 12) (when a fifth instruction is received from the user), the control unit 10 executes the storage change suggestion process. Even if icon B5 is not pressed, the storage change suggestion process may be executed by the user uttering a predetermined keyword.

[0157] The object detection process and tracking process based on the captured image information SI21 of the camera unit 21 capturing the images of the registered articles and the registered persons tracks the registered articles and the registered persons in the sensing space (specifically, the positions and movement trajectories of the registered articles and the registered persons are tracked), and information based on the tracking results is included in the database DB. As described above, the object detection process and tracking process may also utilize the light detection information SI22 using the LIDAR 22a. The information based on the tracking results includes information 672 and 675 shown in FIG. 8.

[0158] The information 675 of the i-th registered article indicates whether the i-th registered article was used by any registered person during a given period, and if so, by which registered person. When the i-th registered article is taken out of any storage location, the i-th registered article is considered to be in use by a registered person. The control unit 10 can identify the user of the i-th registered article based on the captured image information SI21 (and further based on the light detection information SI22) during the period in which the i-th registered article is in use (usage period). By accumulating, for each registered article, whether the registered article was used by any registered person at each time, and if so, by which registered person, the control unit 10 recognizes the usage status of each registered article by each registered person.

[0159] The control unit 10 involved in the storage change proposal process proposes a new default storage destination for one or more registered items based on the recognition result of the usage status of each registered item by each registrant. This proposal is made using the display unit 41 or speaker 44, and when an instruction to agree to the proposal is received from the user, the control unit 10 updates the default storage destination for each registered item according to the content proposed by itself.

[0160] The designated depository before the update is specifically referred to as the old designated depository, and the designated depository after the update that corresponds to the new designated depository is specifically referred to as the new designated depository. For example, E1 The owner of the registered article is the first registrant and E1 The i-th item whose old designated storage location is associated with the first registrant E2 Assume that the item is set as the storage destination. Starting from this state, E1 The registered article is not used at all or very little by the first registrant and the second registrant is in a position i E1 When a registered item is used for a long time or frequently, the control unit 10 related to the storage change proposal processing performs the i-th E1 The i-th item associated with the second registrant as the new designated storage location for the registered item E3 If the user agrees with this proposal, E3 Storage location is i E1 It will become the new storage location for registered goods (i E1 ~i E3 is an integer; where i E2 ≠i E3 ).

[0161] According to this embodiment, a better default storage destination is set according to the usage status of each registered article.

[0162] [Example EX6] Example EX6 will be described. When the user presses icon B6 (see FIG. 12) (when a sixth instruction is received from the user), the control unit 10 executes the layout change process. Even if icon B6 is not pressed, the layout change process may be executed by the user speaking a predetermined keyword.

[0163] The layout change process is executed when the storage locations of the registered items are rearranged in various ways due to rearrangements in the residence HH, etc. Here, it is assumed that the storage locations in which the registered items should be stored are switched from a first storage location group to a second storage location group due to rearrangements in the residence HH. For convenience, this switching is referred to as storage location switching. Each of the first storage location group and the second storage location group is composed of multiple storage locations. Before the storage location switching, the multiple storage locations constituting the first storage location group are installed in the residence HH, and after the storage location switching, the multiple storage locations constituting the second storage location group are installed in the residence HH. The multiple storage locations constituting the first storage location group and the multiple storage locations constituting the second storage location group may include the same storage locations, but here, it is assumed that the multiple storage locations constituting the first storage location group and the multiple storage locations constituting the second storage location group are completely different.

[0164] Basically, there is no change in the information in the registrant database DB_a before and after the storage destination is switched, but the designated storage area information 613 may change (changes in information 613 will be ignored here). There is no change in the information in the item database DB_c before and after the storage destination is switched, except for the specified storage destination information 659. The specified storage destination information 659 is newly set during the layout change process. There is no change in the information in the general management database DB_d before and after the storage destination is switched, except for the current storage destination information 672. The current storage destination information 672 may be discarded when the layout change process is executed.

[0165] In the layout change process, the storage destination database DB_b before the layout change process is discarded, and a new storage destination database DB_b is constructed by executing the storage destination registration process in the initial registration process for each storage destination constituting the second storage destination group. In the newly constructed storage destination database DB_b, information 631-635 is set for each storage destination constituting the second storage destination group (see FIG. 6). Based on information 634 on the location area of ​​each storage destination constituting the second storage destination group and information 613 on the designated storage area in the registrant database DB_a (see FIG. 5), each storage destination constituting the second storage destination group is associated with one of the registrants in the database DB (in DB_a and DB_b). Based on the contents of this association, the control unit 10 proposes to the user one of the storage destinations in the second storage destination group as a new specified storage destination for each registered item. This suggestion is made using the display unit 41 or the speaker 44, and when the control unit 10 receives an instruction from the user indicating that the suggestion is agreed to, the control unit 10 sets the specified storage destination for each registered item in accordance with the content of the suggestion.

[0166] For example, as shown in FIG. 18, the jth storage location, which is one of four storage locations in the second storage group, F ~th(j F +3) The storage location is the i F In the case where the registered item 910 is associated with a registrant, the owner of the registered item 910 is the i F Consider state ST6, which is a registered user (i F and j F In state ST6, the control unit 10 for the layout change process F ~th(j F +3) Propose one of the storage locations to the user as a new default storage location for the registered item 910. At this time, the volume (655) and shape information (654) of the registered item 910, and the jth F ~th(j F +3) Based on the 3D map (635) of storage locations, the control unit 10 may suggest to the user a storage location that is favorable for the registered item 910 as a new default storage location for the registered item 910.

[0167] According to this embodiment, when the storage destination is switched due to a change in the layout or the like, a suitable default storage destination is quickly proposed using registered information, and the organization and tidying of items is quickly completed.

[0168] For example, the jth item may be used as a new designated storage location for the registered item 910. F When proposing a storage location, the control unit 10 inputs the image (653), shape (654), and volume (655) of the registered item 910 and the jth F Based on the 3D map (635) of the storage location, the registered item 910 is the jth F A moving image showing the state in which the registered article 910 is stored in the storage destination may be generated and displayed on the display unit 41. F It is possible to present to the user an image of how the item will be stored in the storage location in an easily understandable manner.

[0169] [Example EX7] Example EX7 will be described. As described above, each registered article is tracked in the sensing space by the object detection process and the tracking process (specifically, the position and movement trajectory of each registered article are tracked), and the current storage location (672) of each registered object is detected based on the tracking result. The control unit 10 may perform the following storage location difference response process.

[0170] In the storage destination discrepancy handling process, the control unit 10 compares the current storage destination of the i-th registered item with the specified storage destination of the i-th registered item, and if there is a discrepancy between them, a predetermined discrepancy handling notice is given to the user. The storage destination discrepancy handling process can be executed for each registered item. The control unit 10 may execute the storage destination discrepancy handling process, for example, periodically, or based on a user instruction.

[0171] The discrepancy handling notification may include a notification indicating that the current storage location of the i-th registered item is different from the specified storage location of the i-th registered item. The discrepancy handling notification may include a notification prompting the user to return the i-th registered item to the specified storage location of the i-th registered item. This promotes correct storage of the i-th registered item.

[0172] In the discrepancy handling notification, the control unit 10 may inquire of the user as to whether to change the prescribed storage destination of the i-th registered item to the current storage destination of the i-th registered item. When the user who has received this inquiry inputs an instruction to affirm the change to the storage assistance device 1, the control unit 10 may change the prescribed storage destination of the i-th registered item to the current storage destination of the i-th registered item (i.e., the information 659 of the i-th registered item may be updated to information indicating the current storage destination of the i-th registered item). This allows the prescribed storage destination to be updated to match the actual storage destination of the i-th registered item to be in line with reality.

[0173] [Example EX8] Example EX8 will be described. The expendables flag according to the information 678 of the i-th registered article (see FIG. 8) indicates whether the i-th registered article is expendable. The expendables flag of "1" for the i-th registered article indicates that the i-th registered article is expendable. The determination of whether the i-th registered article is expendable is realized in the above-mentioned object classification process.

[0174] In particular, when any ith registered item is a consumable item, it is advisable to manage and register the number of ith registered items present in the residence HH in a database DB. Here, the number of the ith registered item is set in other information 680 of the ith registered item (see FIG. 8). The control unit 10 detects the number of the ith registered item present in the residence HH based on the captured image information SI21, or based on the captured image information SI21 and the light detection information SI22, and calculates the number of detected items (hereinafter, the detected number NUM of the ith registered items). DET (hereinafter referred to as "other information") can be set in the other information 680 of the i-th registered item.

[0175] The number of detected registered items NUM DET It is preferable that the number of registered articles i is notified to the user by being displayed on the display unit 41 based on a user operation. This allows the user to easily know the number of registered articles i.

[0176] The required number of specific types of items (mainly consumables) is set by the control unit 10. A specific type of item is an item that is consumed in multiples during an event, such as paper cups, disposable toothbrushes, and juice in a plastic bottle. The required number is determined in advance by a user operation. For example, if the i-th registered item is a paper cup, the i-th registered item belongs to a specific type of item, and the detected number NUM of the i-th registered item DET Together with the required number NUM of the i-th registered item NEED is also set in the other information 680 of the i-th registered article. DET and the required number of registered items NUM NEED When a certain condition is met between, for example, the difference (NUM DET -NUM NEED ) is equal to or smaller than zero or a positive predetermined value, the control unit 10 may send a replenishment notification to the user to prompt the user to replenish (increase the number of) the i-th registered item.

[0177] [Example EX9] An embodiment EX9 will be described. The discarded presumption flag according to the information 677 of the i-th registered item (see FIG. 8) is a flag indicating whether the i-th registered item is an estimated discarded item. The discarded presumption flag of "1" for the i-th registered item indicates that the i-th registered item is an estimated discarded item. An estimated discarded item refers to an item estimated by the control unit 10 to be a discarded item. The initial value of the discarded presumption flag of each registered item is "0". For example, the control unit 10 estimates whether the i-th registered item has been discarded based on the result of the voice recognition process for the user's utterance or the contents of the input operation to the interface unit 40. Then, when it is estimated that the i-th registered item has been discarded, the discarded presumption flag (677) of the i-th registered item is substituted with "1". For example, if the name of a specific registered item is "BBB", when a conversation between registered people includes an utterance such as "I threw away BBB a while ago", the control unit 10 assigns "1" to the discarded inference flag (677) for the specific registered item based on the result of speech recognition processing of the content of the utterance. Note that discarding the i-th registered item means that the i-th registered item will no longer exist in the living space (including the storage location) of the residence HH.

[0178] Information based on the discarded presumed flag of each registered item may be notified to the user, for example by being displayed on the display unit 41 based on a user operation. This makes it possible to inform the user of whether or not a registered item has been discarded. A registered person may forget that he or she has discarded an item, even though he or she has discarded it. Also, another registered person may discard an item without the registered person's knowledge. In these cases, the above notification may be useful.

[0179] [Example EX10] Example EX10 will now be described.

[0180] When the specified storage destination of an arbitrary i-th registered item is registered, the user can input arbitrary additional information into the storage assistance device 1, and the additional information can be stored in the database DB in association with the i-th registered item. For example, when the first storage destination is registered as the specified storage destination of the i-th registered item, the user can include in the additional information the reason why the first storage destination is set as the specified storage destination of the i-th registered item. Thereafter, the stored additional information can be notified to the user at any time by being displayed on the display unit 41 based on a user operation. This makes it possible to share the reason among multiple registrants.

[0181] In the embodiment EX6, an example was given of switching the storage location of each registered item within the same residence HH, but the method shown in the embodiment EX6 can also be applied to a move that involves a change of residence itself.

[0182] The light detection unit 22 may be omitted from the sensor unit 20 in Fig. 1. In this case, any process or operation described above as being executed based on the captured image information SI21 and the light detection information SI22 will be executed based on the captured image information SI21 without using the light detection information SI22. Also, the environment measurement unit 23 may be omitted from the sensor unit 20.

[0183] Although the above embodiments have been described on the assumption that there are multiple registrants, the total number of registrants may be one.

[0184] [Example EX11] Example EX11 will now be described.

[0185] Although the above describes an example in which the storage support device 1 is applied to a residence HH, the storage support device 1 may be applied to any factory, any facility, any vehicle, etc., instead of the residence HH.

[0186] Vehicles to which the storage support device 1 is applied include vehicles. Vehicles include any type of automobile, and may be general passenger cars or public vehicles such as buses or ambulances. Vehicles to which the storage support device 1 is applied have multiple storage locations. When the storage support device 1 is applied to a vehicle, the storage support device 1 is installed in an appropriate location in the vehicle, and the above-mentioned living space is interpreted as the space inside the vehicle's cabin. When the storage support device 1 is applied to a vehicle, each registered item is an item that should be loaded into the vehicle, and it is preferable that a specified storage location is selected and set for each registered item from multiple storage locations provided in the vehicle.

[0187] In many cases, storage locations for tools and the like are designated in vehicles. For example, ambulances are equipped with many medical instruments, and many of the medical instruments have designated storage locations (storage destinations), so the storage assistance device 1 is expected to function beneficially. In addition, the operation of the storage assistance device 1 when moving house can also be applied to assisting in loading and unloading luggage in vehicles.

[0188] Any device according to the present invention (e.g., storage assistance device 1) is referred to as a target device for convenience. The target device can be configured by hardware such as an integrated circuit, or a combination of hardware and software. Any specific function that is all or part of the functions realized by the target device may be written as a program, and the program may be stored in a memory that can be mounted on the target device. The specific function may be realized by executing the program on a program execution device (e.g., a microcomputer that can be mounted on the target device). The program may be recorded on any recording medium. The recording medium on which the program is recorded may be mounted on or connected to a device (such as a server device) different from the target device.

[0189] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above embodiments are merely examples of the present invention, and the meanings of the terms of the present invention and each component are not limited to those described in the above embodiments. The specific numerical values ​​shown in the above description are merely examples, and can be changed to various numerical values ​​as a matter of course. [Explanation of symbols]

[0190] 1 Storage support device 10 Control section 11 Registration processing section 12 Object detection unit 13 Tracking processing section 14 Voice response section 15 Notification Department 20 Sensor section 21 Camera Division 22 Light detection unit 23 Environmental Measurement Department 30 Recording Section 40 Interface section 41 Display section 42 Operation section 43 Microphone 44 Speaker DB Database DB_a Registration database DB_b Storage database DB_c Item database DB_d Integrated management database

Claims

1. A storage assistance device that assists in the storage of items, A control unit that identifies which of a plurality of registered articles each of the articles is a registered article; A database including information on the designated storage location for storing each registered item; The control unit notifies a user of storage assistance information for assisting in storage of each item based on a result of the identification and the database, When notifying a user of the storage assistance information for a focused item to be stored, the control unit notifies the user of the specified storage destination for the focused item by presenting a game to the user, presents the user with a question in the game in which the specified storage destination for the focused item is a correct answer, and notifies the user of the specified storage destination for the focused item by performing a display according to the user's answer to the question. , storage assistance device.

2. the database includes information regarding a plurality of storage locations; When a new item is added to the plurality of registered items, the control unit selects a storage destination where the new item should be stored from among the plurality of storage destinations based on the photographed result of the new item by the camera and information on the plurality of storage destinations, and notifies the user of the selection. The storage assistance device according to claim 1.

3. A storage assistance device that assists in the storage of items, A control unit that identifies which of a plurality of registered articles each of the articles is a registered article; A database including information on the designated storage location for storing each registered item; The control unit notifies a user of storage assistance information for assisting in storage of each item based on a result of the identification and the database, the database includes information regarding a plurality of storage locations; When adding a new item to the plurality of registered items, the control unit selects a storage destination where the new item should be stored from among the plurality of storage destinations based on the photographed result of the new item by the camera and information on the plurality of storage destinations, and notifies the user of the selection; In the database, a plurality of registrants and a plurality of depositories are registered, and each depository is associated with one of the registrants; When the control unit sets an owner of the new item to a specific registrant and adds the new item to the plurality of registered items, the control unit selects a storage destination in which the new item should be stored from one or more storage destinations associated with the specific registrant and notifies the user of the selection. , storage assistance device.

4. The database includes preference information for each registrant, When the control unit sets the owner of the new item to the specific registrant and adds the new item to the plurality of registered items, the control unit makes the selection while taking into consideration preference information of the specific registrant. The storage assistance device according to claim 3.

5. A storage assistance device that assists in the storage of items, A control unit that identifies which of a plurality of registered articles each of the articles is a registered article; A database including information on the designated storage location for storing each registered item; The control unit notifies a user of storage assistance information for assisting in storage of each item based on a result of the identification and the database, In the database, a plurality of registrants and a plurality of depositories are registered, and each depository is associated with one of the registrants; The control unit recognizes the usage status of each registered item by each registered person using a camera unit that photographs each registered item and each registered person, and when the specified storage destination is set for each registered item, proposes a new specified storage destination for any registered item based on the recognition result of the usage status. , storage assistance device.

6. A storage assistance device that assists in the storage of items, A control unit that identifies which of a plurality of registered articles each of the articles is a registered article; A database including information on the designated storage location for storing each registered item; The control unit notifies a user of storage assistance information for assisting in storage of each item based on a result of the identification and the database, In the database, a plurality of registrants and the plurality of registered items are registered, and an owner of each registered item is registered; When the plurality of storage locations in which the plurality of registered items are to be stored are switched from a first storage location group to a second storage location group, each storage location constituting the second storage location group is associated with any one of the registrants in the database, and the control unit proposes to the user, based on the contents of the association, any one of the storage locations in the second storage location group as a new prescribed storage location for each registered item. , storage assistance device.

7. The storage assistance device is installed in a vehicle, Each registered item is an item to be loaded onto the vehicle, and the specified storage location is provided on the vehicle. A storage assistance device according to any one of claims 1 to 6.

8. The control unit detects a current storage location of each registered item by tracking each registered item using a camera unit that photographs each registered item, and when the current storage location of any registered item differs from the specified storage location of the registered item, performs a predetermined notification to the user. The storage assistance device according to claim 1.

9. The control unit, in the predetermined notification in the case where the current storage destination of any registered item is different from the designated storage destination of the registered item, asks the user whether to change the designated storage destination of the registered item to the current storage destination of the registered item. The storage assistance device according to claim 8.

10. A storage assistance method implemented by a computer for assisting in the storage of items, comprising: an identification step of identifying each item as one of a plurality of registered items; and a notification step of notifying a user of storage support information for supporting storage of each item based on the result of the identification and a database including information on a prescribed storage destination in which each registered item should be stored, When notifying a user of the storage assistance information for a target item to be stored, the specified storage destination for the target item is notified to the user by presenting a game to the user, a question is presented to the user in the game, the correct answer of which is the specified storage destination for the target item, and the specified storage destination for the target item is notified to the user by performing a display according to the user's answer to the question. , storage assistance methods.

11. A storage assistance method implemented by a computer for assisting in the storage of items, comprising: an identification step of identifying each item as one of a plurality of registered items; and a notification step of notifying a user of storage support information for supporting storage of each item based on the result of the identification and a database including information on a prescribed storage destination in which each registered item should be stored, the database includes information regarding a plurality of storage locations; In the storage support method, when a new item is added to the plurality of registered items, a storage destination in which the new item should be stored is selected from the plurality of storage destinations based on the photographed result of the new item by the camera and information on the plurality of storage destinations, and notifies the user of the selection; In the database, a plurality of registrants and a plurality of depositories are registered, and each depository is associated with one of the registrants; In the storage support method, when the owner of the new item is set to a specific registrant and the new item is added to the plurality of registered items, a storage destination where the new item should be stored is selected from one or more storage destinations associated with the specific registrant and notified to the user. , storage assistance methods.

12. A storage assistance method implemented by a computer for assisting in the storage of items, comprising: an identification step of identifying each item as one of a plurality of registered items; and a notification step of notifying a user of storage support information for supporting storage of each item based on the result of the identification and a database including information on a prescribed storage destination in which each registered item should be stored, In the database, a plurality of registrants and a plurality of depositories are registered, and each depository is associated with one of the registrants; In the storage support method, a camera unit that photographs each registered item and each registered person is used to recognize the usage status of each registered item by each registered person, and when the specified storage destination is set for each registered item, a new specified storage destination for any registered item is proposed based on the recognition result of the usage status. , storage assistance methods.

13. A storage assistance method implemented by a computer for assisting in the storage of items, comprising: an identification step of identifying each item as one of a plurality of registered items; and a notification step of notifying a user of storage support information for supporting storage of each item based on the result of the identification and a database including information on a prescribed storage destination in which each registered item should be stored, In the database, a plurality of registrants and the plurality of registered items are registered, and an owner of each registered item is registered; When the plurality of storage destinations for storing the plurality of registered items are switched from a first storage destination group to a second storage destination group, each storage destination constituting the second storage destination group is associated with any of the registered users in the database, and the storage support method proposes to the user one of the storage destinations in the second storage destination group as a new prescribed storage destination for each of the registered items based on the contents of the association. , storage assistance methods.

14. A storage support program for causing a computer to execute the storage support method according to any one of claims 10 to 13.

Images