Information processing device

Abstract

To provide an information processing apparatus suitable for managing articles of family members.SOLUTION: In an information processing apparatus 10, a CPU 111 acquires first article information and location information from message information. The message information is transmitted by an SNS server 30 to the Internet 40. The first article information indicates an article. The location information indicates a location. The CPU 111 updates a prediction model with the acquired first article information and the location information. The CPU 111 acquires second article information indicating an article from a captured image obtained through a camera 12, and acquires result information indicating a result of estimating a location of the article indicated by the image information, on the basis of the second article information and the prediction model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0005]

[0001] The present invention relates to an information processing apparatus.

Background Art

[0002] Patent Documents 1 and 2 describe a current position system (hereinafter simply referred to as "system") for estimating the location of an article that can be taken out from a storage location.

[0003] In the system of Patent Document 1, books in a library are attached with RFID tags on which serial numbers are recorded. On desks around the library, an information processing apparatus and a reader device are installed. An apparatus ID, which is information for identifying the apparatus itself, is assigned to the information processing apparatus. When a reader places a book on the desktop, the reader device reads the serial number from the RFID tag of this book. The information processing apparatus transmits the apparatus ID and the serial number to a server device. The server device manages on which desk the book is currently being read based on the apparatus ID and the serial number.

[0004] In the system of Patent Document 2, RFID tags are attached to articles that are taken in and out of a warehouse or the like. The RFID tag receives position information indicating its current position from a distribution device installed on the ceiling of the warehouse or the like. The RFID tag transmits the received position information and the MAC address pre-assigned to itself to an information management server. The information management server manages the in-out status of the article to which the RFID tag is attached based on the position information and the MAC address transmitted by the RFID tag.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006]

Patent Document 2

[0007] When people move out of their parents' home and start living in a different residence, they sometimes leave their belongings that they used while living together in a storage room at their parents' house. Conversely, people may also be allowed to store their children's belongings in their parents' storage room. In these situations, because parents have free access to both the child's and the parent's belongings, the owner (the child or parent) may lose track of where these items are located.

[0008] To address the above problems and share the location of items among family members, it is conceivable to apply the systems described in Patent Documents 1 and 2. These systems are based on the use of RFID tags and are suitable for managing the location of a large number of items in a limited space. However, the items owned by family members are relatively few in number and move to various places along with people. For managing the location of family items with these characteristics, the systems in Patent Documents 1 and 2 are time-consuming and costly because they require attaching RFID tags to each item.

[0009] This invention has been made in view of the circumstances described above, and its purpose is to provide an information processing device suitable for managing the location of family members' belongings. [Means for solving the problem]

[0010] (1) The information processing device according to the present invention includes a communication interface, an image input device, and a controller. The controller performs a first acquisition process to acquire first item information and location information from message information received through the communication interface. The message information is transmitted to the internet by an external server providing an SNS. The first item information is information indicating an item. The location information is information indicating a location. The controller performs a learning process to update a prediction model with the first item information and location information acquired in the first acquisition process, a second acquisition process to acquire image information through the image input device and acquire second item information indicating an item from the acquired image information, and a third acquisition process to acquire result information indicating the estimated location of the item indicated by the image information based on the second item information acquired in the second acquisition process and the prediction model.

[0011] According to the above process, the first item information and location information used to update the prediction model are obtained from message information distributed via social media. The second item information used to obtain result information is obtained from image information from an image input device. Therefore, RFID tags are not required for managing the location of items.

[0012] (2) In the first acquisition process, the controller further acquires at least one selected from the message information, including person information indicating a person, date and time information indicating a date and time, and weather information indicating the weather. In the learning process, the controller updates the prediction model with the person information, date and time information, and weather information acquired in the first acquisition process. In the second acquisition process, the controller further acquires the same type of information as that acquired in the first acquisition process from among the person information, date and time information, and weather information. In the third acquisition process, the controller provides the prediction model with the person information, date and time information, and weather information acquired in the second acquisition process.

[0013] The above process increases the amount of information provided to the prediction model, thereby improving the accuracy of the estimation results.

[0014] (3) The controller executes the third acquisition process in response to request information transmitted from an external mobile terminal device via the communication interface. The controller further executes a transmission process to send the result information obtained in the third acquisition process to the mobile terminal device.

[0015] According to the above process, users of mobile devices can obtain result information even remotely from the information processing device.

[0016] (4) The image input device described above is installed in a storage room in the house where the above-mentioned items are stored.

[0017] With the above configuration, since the image input device is located in the storage room, it is easy for family members to operate the image input device when taking items out, etc.

[0018] (5) The controller updates the prediction model stored on the cloud server connected to the Internet during the learning process described above.

[0019] According to the above process, there is no need to store the predictive model in the information processing device, so the information processing device can be realized at a low cost. [Effects of the Invention]

[0020] According to the present invention, it is possible to provide an information processing device suitable for managing the location of family members' belongings. [Brief explanation of the drawing]

[0021] [Figure 1] (A) is a schematic diagram showing the configuration of system 100 in the embodiment, and (B) is a block diagram of system 100 shown in figure (A). [Figure 2] (A) is a schematic diagram showing various information 110, (B) is a schematic diagram showing the structure of the person table 71, and (C) is a schematic diagram showing the structure of the house table 72. [Figure 3](A) is a schematic diagram showing a part of the structure of the item table 73, (B) is a schematic diagram showing the structure of the remaining part of the item table 73, and (C) is a schematic diagram showing the structure of the keyword table 74. [Figure 4] (A) is a schematic diagram showing the structure of the usage record information 75, and (B) is a schematic diagram showing the structure of the account DB 322. [Figure 5] A flowchart showing the processing of the third program 19 in the embodiment. [Figure 6] (A) is a flowchart showing the processing of the first program 17, and (B) is a flowchart showing the processing of the second program 18. [Figure 7] (A) is a schematic diagram showing the message information 80, and (B) is a schematic diagram showing the input form information 90. [Figure 8] (A) is a block diagram showing the configuration of the information processing apparatus 10 in the first modification example, and (B) is a block diagram showing the configuration of the system 100 in the first modification example. [Figure 9] A flowchart showing the processing of the third program 19 in the second modification example. [Embodiment for Carrying Out the Invention]

[0022] Hereinafter, a system 100 including an information processing apparatus 10 according to an embodiment of the present invention will be described with reference to the drawings. Note that the embodiments described below are merely examples of the present invention, and it is needless to say that the embodiments of the present invention can be appropriately changed without departing from the gist of the present invention.

[0023] [System Configuration] As shown in FIG. 1(A), the system 100 includes an information processing apparatus 10, a plurality of mobile terminal devices 20, and an SNS server 30 (an example of an external server). The information processing apparatus 10, the plurality of mobile terminal devices 20, and the SNS server 30 can communicate with each other through the Internet 40.

[0024] The information processing device 10 is installed inside the house 50 and comprises a main unit 11, a camera 12 (an example of an image input device), and a touch panel 13.

[0025] Husband A and wife B reside in house 50. Husband A and wife B have two children, eldest son C and eldest daughter F. Eldest son C and eldest daughter F each have their own separate households from husband A and wife B. Eldest son C lives in his own home (hereinafter also referred to as "eldest son's house") with wife D and child E. Eldest daughter F lives in her own home (hereinafter also referred to as "eldest daughter's house") with husband G and child H. Eldest son C and eldest daughter F each store items 60 that they used when they lived with husband A and wife B in the storage room 51 of house 50, even after becoming independent. Eldest son C and eldest daughter F each store items 60 belonging to child E and child H in the storage room 51. Examples of items 60 include a child's bicycle 61, a digital video camera 62, or a digital still camera 63. The items 60 stored in storage shed 51 are shared by the household of husband A and wife B, the household of eldest son C, and the household of eldest daughter F.

[0026] As shown in Figure 1(B), the main unit 11 includes a CPU 111, memory 112, and communication IF 113. The combination of CPU 111 and memory 112 is an example of a controller.

[0027] Memory 112 consists of RAM, ROM, and EEPROM. Memory 112 stores the SNS application 16, the first program 17, the second program 18, the third program 19, and various information 110. "SNS" is an acronym for Social Networking Service.

[0028] The SNS application 16 is generally an application program executed by the CPU 111 when a registered user uses the SNS. The SNS allows users to build a social network with other registered users on the web. In this embodiment, all family members are registered users of the SNS. The information processing device 10 executes the SNS application 16 not for the purpose of building a social network, but to acquire message information sent to the Internet 40 by the SNS server 30 and use it in the learning process described later. The message information includes text information representing the content of user posts in written form and image information representing it in images. The message information may also consist of only text information or only image information.

[0029] The first program 17, the second program 18, and the third program 19 describe the procedures for a learning process (see Figure 6(A)), an estimation process (see Figure 6(B)), and a registration process (see Figure 5). In the estimation process, the CPU estimates the location of item 60 and transmits result information indicating the estimation result. In the learning process, the CPU updates the prediction model used in the estimation process. In the registration process, the CPU creates a prediction model. Note that each step constituting the first program 17, the second program 18, and the third program 19 may be described in a single program. The first program 17, the second program 18, and the third program 19 may be executed by any of the portable terminal devices 20 instead of the information processing device 10.

[0030] As shown in Figure 2(A), the various types of information 110 include a person table 71, a housing table 72, an item table 73, a keyword table 74, and usage record information 75. The combination of the person table 71, the housing table 72, and the item table 73 is an example of a predictive model (see Figure 2(A)).

[0031] As shown in Figure 2(B), the person table 71 includes person records 710, which are records created for each family member. In the case of an eight-person family as in this embodiment, eight person records 711 to 718 are created in the person table 71. Each person record 710 includes a user ID, a face image, and a first index assigned to each candidate location estimated in the estimation process. The user ID is information that identifies the person in question as a string, and may be a name, relationship, nickname, or handle name, but in this embodiment, it identifies a registered user of the SNS. The face image is information that shows an image of the face of the person in question. Note that in Figure 2(B), only the face image of husband A is shown as an example, and face images of other family members are not shown. The first index is information that shows an index used to estimate the location of item 60 in the estimation process (see Figure 6(B)). In this embodiment, this index increases with the number of times the corresponding person has been to the corresponding location. In this embodiment, the locations used are those recorded in the keyword table 74 (see Figure 3(C)), such as "storage room," "living room," "Japanese-style room," "utility room," "outdoors," "inside the car," and "children's room." Note that the person table 71 may register image information showing the fingerprints or iris of the person in question instead of a facial image.

[0032] As shown in Figure 2(C), the housing table 72 includes housing records 720, which are records created for each housing. In this embodiment, three housing records 721 to 723 are created in the housing table 72. Housing record 721 is for housing 50, housing record 722 is for the eldest son's house, and housing record 723 is for the eldest daughter's house. Each housing record 720 includes housing information and a second index assigned to each location. Housing information is information that identifies the target housing. The second index is information that indicates an index for location estimation. This index indicates whether or not there is a location corresponding to the target housing. As locations, those recorded in the keyword table 74 (see Figure 3(C)) are used. Specifically, if there is no corresponding location, the index is set to "0". For example, since the eldest daughter's house does not have a Japanese-style room, the corresponding second index is set to "0". If there is a corresponding location, the second index is set to "1". For example, since the eldest son's house has a storage shed, a corresponding second indicator of "1" is set. However, although house 50 has a storage shed 51, an indicator of "0" is exceptionally set for storage shed 51.

[0033] As shown in Figure 3(A), the item table 73 contains item records 730, which are records created for each item 60. When there are three items 60 (see Figure 1(A)), the item table 73 contains three item records 731-733. Each item record 730 contains the item name, item image, and a third index assigned to each location. The locations used are those recorded in the keyword table 74 (see Figure 3(C)). The item name is information that identifies the item 60 in question as a string. The item image is information that shows the item 60 in question as an image. The third index is information that indicates an index for location estimation, and this index increases with the number of times the corresponding item 60 has been taken to the corresponding location.

[0034] As shown in Figure 3(B), each item record 730 further includes an item flag. The item flag is information indicating whether the item 60 in question has been taken out of the storage room 51. In this embodiment, several flags are used. Each flag is set to either "0" or "1". A "1" as an item flag indicates "taken out".

[0035] As shown in Figure 3(C), the keyword table 74 registers multiple keywords. These multiple keywords are string-like information and include a first keyword, a second keyword, a third keyword, and a fourth keyword. The first keyword is a keyword related to housing information. The second keyword is a keyword related to location. The third keyword is a keyword for searching for item 60. The fourth keyword is a keyword related to actions on item 60. Examples of the first to fourth keywords are shown in Figure 3(C).

[0036] The first keyword is related to housing information and may differ among multiple families. Therefore, it is preferable that the first keyword be appropriate for the target family before the information processing device 10 is installed in the house 50. The second keyword is a keyword related to location, and the fourth keyword is a keyword related to the action on the item 60. That is, the second and fourth keywords hardly differ among multiple families, so it is preferable that they be registered before the information processing device 10 is installed in the house 50. The third keyword is related to the item 60 itself and therefore differs among multiple families, so it is preferable that it be registered when a new item record 730 is created, as will be described later with reference to Figure 5.

[0037] As shown in Figure 4(A), the usage record information 75 is a collection of log records 750. A log record 750 is created each time an event occurs in the system 100. In this embodiment, there are two types of events. The first event is taking item 60 out of the storage room 51. The second event is returning item 60 to the storage room 51.

[0038] Each of the log records 750 includes the creation date and time, item name, user ID, and operation type. The reservation flag, scheduled date and time, and matching flag are described in the first modification described below, not in this embodiment. The creation date and time indicates the date and time when the target log record 750 was created. This date and time is also the date and time when the target event occurred. The item name and user ID are as described above. The operation type is information indicating whether the operation type is "takeout" or "return".

[0039] As shown in Figure 1, the storage shed 51 is fitted with a door 52 that opens and closes the entrance to the storage shed 51. The camera 12 is mounted on the ceiling 53 of the storage shed 51 and its shooting range is around the entrance. The camera 12 generates image information (hereinafter also referred to as "captured image") that shows the situation within the shooting range and transmits it to the main unit 11.

[0040] The touch panel 13 is mounted on the wall 54 of the storage room 51 near the entrance. The touch panel 13 has a display and sensors and displays various screens under the control of the main unit 11. The touch panel 13 outputs operation signals to the CPU 21 in response to instructions given on the display by a finger or pen.

[0041] Communication IF14 sends data and information to wired LANs, wireless LANs, etc., that make up the Internet 40, and receives data and information from wired LANs, wireless LANs, etc.

[0042] The multiple mobile terminal devices 20 are smartphones or tablet devices used by family members. In this embodiment, the multiple mobile terminal devices 20 are mobile terminal devices 20A to 20H for eight family members. Each mobile terminal device 20 is equipped with a CPU 21, memory 22, camera 23, touch panel 24, and wireless communication IF 25.

[0043] Memory 22 consists of RAM, ROM, and EEPROM. Memory 22 stores an SNS application 221 similar to SNS application 16, and a program 222 that describes the processes to be executed by the CPU 21.

[0044] The camera 23 and touch panel 24 are mounted on the casing of the mobile terminal device 20. The camera 23 generates image information (hereinafter also referred to as "captured image") of the area within the shooting range according to human operation and transfers it to the RAM of the mobile terminal device 20. The touch panel 24 has a display and sensors and displays various screens on the display under the control of the CPU 21. The touch panel 24 outputs operation signals to the CPU 21 in response to instructions given on the display by a finger or pen.

[0045] Wireless communication IF25 transmits data and information to wireless LANs, mobile phone networks, etc., which make up the Internet 40, and receives data and information from wireless LANs, mobile phone networks, etc.

[0046] The SNS server 30 is managed by the SNS provider and is equipped with a CPU 31, memory 32, and a communication interface 33.

[0047] Memory 32 consists of RAM, ROM, and EEPROM, and stores the SNS server program 321 and the account DB 322.

[0048] The SNS server program 321 is a program that describes various processes necessary for providing the SNS. Specifically, the SNS server program 321 sends the aforementioned message information from the communication interface 33 onto the internet 40 in response to a request from a mobile terminal device 20 carried by an SNS user.

[0049] As shown in Figure 4(B), the account DB 322 includes account records 330, which are records created for each SNS account. In this embodiment, account records 331 to 338 for users of mobile terminal devices 20A to 20H are created in the account DB 322 as account records 330. Note that account records 333 to 337 are not shown in Figure 4(B). Each account record 330 includes at least a user ID and group information. The user ID is information that identifies the user of the target account. The group information is the user IDs of people in the same family.

[0050] [Processing of Program 3, Section 19] The CPU 111 of the information processing device 10 starts executing the third program 19 when one of the family members (hereinafter also referred to as "entrant") opens the door 52. The opening of the door 52 is detected, for example, by a sensor (not shown) installed on the door 52. Based on the detection result from this sensor, the CPU 111 starts executing the third program 19.

[0051] As shown in Figure 5, in S101, the CPU 111 activates the camera 12 and the touch panel 13. In S102, the CPU 111 receives captured images from the camera 12 at a predetermined frame rate for a predetermined time (for example, about 10 seconds).

[0052] In S103, the CPU 111 performs face extraction processing using each of the received captured images. In the face extraction processing, image information indicating a person's face (hereinafter also referred to as the "extracted face image") is identified from each captured image.

[0053] In S104, the CPU 111 determines whether or not there is a face image in the person table 71 that is similar to the extracted face image. In Figure 5, S104 is labeled "Similar image present?". If it is determined that there is no similar image (No in S104), the process proceeds to S105. If it is determined that there is a similar image (Yes in S104), the process proceeds to S108.

[0054] In S105, the CPU 111 determines whether or not the person entering the room has ever registered their face image in the person table 71. In Figure 5, S105 is labeled "Registered in the past?". More specifically, in S105, the CPU 111 first displays a first query screen on the touch panel 13. The first query screen includes the message, "Have you ever registered a face image?". The first query screen further includes a dialog box to prompt a response to this message. If the CPU 111 determines "Yes" based on the operation signal from the touch panel 13 (Yes in S105), the process proceeds to S106. On the other hand, if the CPU 111 determines "No" (No in S105), the process proceeds to S107.

[0055] In S106, the CPU 111 changes the display screen on the touch panel 13 from the first inquiry screen to the first message screen. The first message screen contains the message, "Please face the camera 12." After displaying the first message screen, the CPU 111 waits for a certain period of time and then executes S102 to perform face extraction processing on another captured image.

[0056] In S107, the CPU 111 changes the display screen on the touch panel 13 from the first inquiry screen to the person registration screen. The person registration screen includes a message that reads, "Please enter a name." The person registration screen also includes a software keyboard. Based on the operation signal from the touch panel 13, the CPU 111 obtains a user ID that identifies the name of the person entering the room as a string. The CPU 111 creates a person record 710 consisting of the obtained user ID, a face image showing the extracted face image, and a first indicator assigned to each location, and adds it to the person table 71. Each of the first indicators is set to "0" as an initial value. After S107, the process transitions to S108.

[0057] In S108, CPU 111 executes the entry determination process. In the entry determination process executed after determining Yes in S104, CPU 111 determines that the user ID corresponding to the face image searched in S104 belongs to the entry person. In the entry determination process executed after S107, CPU 111 determines that the user ID obtained in S107 belongs to the entry person.

[0058] In S109, the CPU 111 executes image recognition processing (an example of the second acquisition process). In the image recognition processing, an attempt is made to identify the image portion representing an item (hereinafter also referred to as the "extracted item image") from the captured image received in S102 or S113 described later. More specifically, if an image portion similar to an item image registered in the item table 73 is recognized in the captured image, the identification of the extracted item image is considered successful. If the identification is successful ("success" in S109), the process proceeds to S110. If the identification fails ("failure" in S109), the process proceeds to S116.

[0059] In S110, the CPU 111 determines whether or not there is an item image similar to the extracted item image in the item table 73. In Figure 5, S110 is labeled "Are there similar item images?". If it is determined that there are no similar items (No in S110), the process proceeds to S111. If it is determined that there are similar items (Yes in S110), the process proceeds to S117.

[0060] In S111, it is determined whether the item shown in the extracted item image has been stored in the storage room 51 for some time. In Figure 5, S111 is labeled "Previously stored?". More specifically, the CPU 111 displays a second inquiry screen on the touch panel 13. The second inquiry screen includes the message, "Has this item been in the storage room 51 for some time?". The second inquiry screen further includes a dialog box to prompt a response to this message. Based on the operation signal from the touch panel 13, if the CPU 111 determines that the item has been stored in the storage room 51 for some time (Yes in S111), the process proceeds to S112. If the CPU 111 determines that the item has not been stored in the storage room 51 for some time (No in S111), the process proceeds to S114.

[0061] In S112, the CPU 111 changes the display screen on the touch panel 13 from the second inquiry screen to the second message screen. The second message screen contains the message, "Please point the item towards the camera 12." The CPU 111 waits for a certain period of time after displaying the second message screen. After that, the process transitions to S113, which is the same as S102, and then to S109.

[0062] In S114, the CPU 111 executes the process of creating a new item record 730. Specifically, in the new registration process, the CPU 111 changes the display screen on the touch panel 13 from the second inquiry screen to the item registration screen. The item registration screen includes a message that reads, "Please enter the item name," and a software keyboard. Based on the operation signal from the touch panel 13, the CPU 111 obtains the item name, which identifies the item name in characters. The CPU 111 creates an item record 730 consisting of the item name, an item image showing the extracted item image, and a third indicator assigned to each location, and adds it to the item table 73. Each of the third indicators is set to "0" as an initial value. After the execution of S114, the process transitions to S115.

[0063] In S115, the CPU 111 changes the display screen on the touch panel 13 from the item registration screen to the keyword registration screen. The keyword registration screen includes a message that reads, "Please enter keywords to search for items," and a software keyboard. Based on the operation signal from the touch panel 13, the CPU 111 obtains a third keyword and registers it in the keyword table 74.

[0064] In S116, the first event occurs, so the CPU 111 executes the extraction process. In the extraction process, the CPU 111 resumes receiving captured images at a predetermined frame rate. The CPU 111 performs the same image recognition process as in S109 for each of the received captured images. In this image recognition process, the CPU 111 creates a new log record 750 depending on whether it has successfully identified the extracted product image from the captured image. In the new log record 750, the creation date and time is the current date and time information obtained by an API prepared in advance by the OS, etc. The item name is the item name in the item table 73 that corresponds to the extracted product image obtained in S116. The user ID is the user ID determined in S108. The operation type is registered as "extraction". In the extraction process, the CPU 111 updates the item flag in the item table 73 that corresponds to the item image identified in S109 to "1".

[0065] In S117, a second event occurs, so CPU 111 executes the return process. In the return process, a new log record 750 is created, just like in the take-out process. However, in the log record 750 created in the return process, "Return" is registered as the operation type. In the return process, CPU 111 further updates the item flag in the item table 73 to "0", which corresponds to the item image identified in S109.

[0066] After any of steps S115 through S117 are executed, the process shown in Figure 5 ends.

[0067] [Processing of Program 17] Wife D is the user of the mobile terminal device 20A. Wife D takes an item 60 that she has taken from the storage room 51 to various locations. Here, let's assume that the item 60 that Wife D is taking is a child's bicycle 61. At the location where she has taken the child's bicycle 61, Wife D operates the mobile terminal device 20A to launch the SNS application 221. After that, Wife D uses the touch panel 24 to input the content to post as text or to specify an image that represents the content to post. As shown in Figure 6(A), in S201, the CPU 21 of the mobile terminal device 20A generates message information that represents the content to post. In the example shown in Figure 7(A), the message information 80 includes text information 81 and image information 82. The text information 81 expresses the content "I'm playing with my child on a bicycle outdoors." The image information 82 is an image representing the child and the bicycle. In S201, the CPU 21 sends the generated message information to the internet 40 via wireless communication IF 25. Although not shown in Figure 6(A), the other mobile terminal devices 20B to 20H also generate message information and send it to the Internet 40 in the same manner as mobile terminal device 20A.

[0068] In the SNS server 30, the CPU 31 executes the SNS server program 321. In S202, the CPU 31 receives message information sent to the internet 40 by the mobile terminal device 20A. The CPU 31 sequentially stores the received message information in the memory 32 so that the SNS application 221 can display it on the touch panel 24 in each of the mobile terminal devices 20A to 20H. The CPU 31 processes message information from the other mobile terminal devices 20B to 20H in the same way as message information from mobile terminal device 20A.

[0069] In the information processing device 10, the CPU 111 is executing the first program 17. In S203, the CPU 111 periodically accesses the SNS server 30 via the internet 40 and downloads message information stored in memory 32 via the communication IF 113. The communication IF 113 transfers the received message information to memory 112 under the control of the CPU 111.

[0070] In S204, the CPU 111 identifies each of the message information newly received in S203 as a target for processing. The CPU 111 then performs text mining and image recognition processing on the identified targets. In text mining, the CPU 111 extracts each keyword recorded in keyword table 74 and the user ID from the message information of the target. In image recognition, the CPU 111 attempts to identify the image portion that represents an item from the image information included in the target, using the same process as in S109 (see Figure 5). Note that S203 and S204 are examples of the first acquisition process.

[0071] In S205, CPU111 updates the first index in the person table 71 that corresponds to the user ID and second keyword (i.e., a keyword related to location) obtained in S204. Specifically, if the user ID is that of wife D and the second keyword "outdoors" is obtained in S205, the first index corresponding to wife D and "outdoors" is incremented by a predetermined number (for example, "1").

[0072] In S206, the CPU 111 updates the third index in the item table 73 that corresponds to the item and the second keyword (i.e., the keyword related to location) identified in S204. Specifically, if "bicycle" is identified as an item and "outdoor" is obtained as the second keyword in S204, the third index corresponding to "bicycle" and "outdoor" is incremented by a predetermined number (for example, "1").

[0073] S205 and S206 are executed repeatedly for the number of processing targets identified in S204. Additionally, S205 and S207 update the first and third indicators in the person table 71 and item table 73, respectively, as part of the prediction model. Therefore, S205 and S206 are an example of the learning process.

[0074] Once S206 finishes, CPU111 waits for the next execution timing of S203.

[0075] [Processing of Program 2, Item 18] Let's assume that husband A is the user of the mobile terminal device 20B. If husband A wants to check the location of an item 60 stored in the storage room 51, he operates the mobile terminal device 20B to start program 222. After that, husband A operates the touch panel 24 according to a predetermined procedure. In response to this operation, as shown in S301 of Figure 6(B), the CPU 21 of the mobile terminal device 20B sends first request information from the wireless communication IF 25 to the internet 40 to request the transmission of input form information necessary for location confirmation.

[0076] The CPU 111 of the information processing device 10 periodically executes S302 of the second program 18. In S302, triggered by the CPU 111 receiving the first request information via the communication IF 113, the processing of the second program 18 transitions to S303.

[0077] In S303, the CPU 111 sends input form information for querying the item 60 to be located to the Internet 40 via the communication IF 113. Specifically, the CPU 111 retrieves the item name corresponding to the item flag indicating "1" from the item table 73. The CPU 111 lays out the retrieved item name in the input form information. The CPU 111 further lays out an object (e.g., a radio button) that can specify the retrieved item name. As shown in Figure 3, if the item flag for the child's bicycle 61 is "1", then, as illustrated in Figure 7(B), the input form information 90 will have a string 91 indicating the child's bicycle 61 and a radio button 92 corresponding to the child's bicycle 61. For convenience, Figure 7(B) shows the input form information 90 displayed on the touch panel 24 of the mobile terminal device 20B.

[0078] In S304, the CPU 21 of the mobile terminal device 20B receives form information via wireless communication IF 25. The CPU 21 displays the item name and object included in the received form information on the touch panel 24. Husband A can confirm the item name displayed on the touch panel 24 and find out what type of item 60 is currently out of storage 51. If Husband A needs to confirm the location of item 60 out of storage 51, he specifies the corresponding object on the touch panel 24. Here, let's assume that the radio button 92 corresponding to the child's bicycle 61 is selected and the send button is tapped. In S305, the CPU 21 generates second request information to request the transmission of the result of the location confirmation in response to the object being manipulated. The second request information includes the item name corresponding to the object specified on the touch panel 24. The CPU 21 sends the generated second request information to the internet 40 via wireless communication IF 25.

[0079] In S306, the CPU 111 of the information processing device 10 receives the second request information via the communication IF 113 and processes all item names included in the second request information.

[0080] In S307, CPU 111 identifies the user ID from log record 750 in the usage record information 75, which records the name of the item to be processed and the operation type as "removal". In this case, the user ID of wife D is identified.

[0081] In S308, CPU 111 identifies a first indicator that satisfies the first condition from the person table 71 (see Figure 2(B)), a second indicator that satisfies the second condition from the house table 72 (see Figure 2(C)), and a third indicator that satisfies the third condition from the item table 73 (see Figure 3(A)). The first and second conditions may correspond to the user ID identified in S307. Specifically, if the user ID of wife D is identified in S307, the first indicator that satisfies the first condition is the first indicator that corresponds to all locations registered in the person record 714 (see Figure 2(B)). The second indicator that satisfies the second condition is the second indicator that corresponds to all locations registered in the house record 722 (see Figure 2(C)). The third condition corresponds to the name of the item to be processed identified in S306. Specifically, if the item to be processed is a child's bicycle 61, the third indicator that satisfies the third condition is the third indicator corresponding to all locations registered in item record 731 (see Figure 3(A)).

[0082] In S309, CPU 111 calculates an estimated result that numerically indicates the probability that item 60, the item to be processed, is located at a given location by multiplying the first, second, and third indicators identified in S308 by the indicators assigned to the same location. S309 is executed for all locations; that is, an estimated result is obtained for each of the locations. S309 is an example of the third acquisition process.

[0083] The following is a specific example of the processing in S309. When the location for which the estimation result should be obtained is "storage room", the CPU 111 obtains "30", the first index corresponding to "storage room", in person record 714 (see Figure 2(B)), "1", the second index corresponding to "storage room", in house record 722 (see Figure 2(C)), and the third index corresponding to "storage room", in item record 731 (see Figure 3(A)). 3Multiplying by the index "50", we obtain the estimated result "1500". If the location for which we want to find the estimated result is "living room", then in person record 714, the first index corresponding to "living room" is "60", in house record 722, the second index corresponding to "living room" is "1", and in item record 731, the index corresponding to "living room" is... 3 Multiplying by the index "0" yields an estimated result of "0". Estimated results for other locations can be obtained using the same process as above. The estimated results for "Japanese-style room", "utility room", and "children's room" are all "0". The estimated result for "outdoors" is "3200". The estimated result for "inside the car" is "400".

[0084] In S310, the CPU 111 sends result information indicating the location (in the above example, "outdoors") corresponding to at least the highest-level estimation result obtained in S309 to the Internet 40 via the communication IF 113. S310 is an example of a transmission process.

[0085] In S311, the CPU 21 of the mobile terminal device 20B receives result information via wireless communication IF 25. The CPU 21 displays the estimated result indicated by the received result information on the touch panel 24. As a result, husband A can obtain the estimated location of item 60.

[0086] [Effects of the Embodiment] According to the above process, the items and candidate locations used to update the prediction model (i.e., the person table 71 and the item table 73) are identified by S203 and S204 from message information distributed via SNS. The extracted item images used to obtain the result information are obtained from images captured by the camera 12. Therefore, RFID tags are not required for managing the location of items. This provides an information processing device suitable for managing the location of family members' belongings.

[0087] Furthermore, if it is confirmed that the person who took the item 60 from the storage room 51 will operate their mobile terminal device 20 to send message information via S201 (see Figure 6(A)), then it is sufficient to contact the person who took the item via smartphone. However, it is not guaranteed that the person who took the item 60 from the storage room 51 will operate their mobile terminal device 20 to send message information via S201 (see Figure 6(A)). Therefore, the information processing device 10 of this embodiment is suitable for managing the location of family members' belongings.

[0088] According to the above process, the result information is transmitted to the mobile terminal device 20 in S310, so the result information can be obtained even from a location far from the information processing device 10.

[0089] In this embodiment, the camera 12 and touch panel 13 are installed in the storage shed 51, making it easy for family members to operate them when taking items 60 out of or returning them to the storage shed 51.

[0090] [First variation] In this embodiment, the memory 112 stored a person table 71, a house table 72, and an item table 73 as examples of a prediction model. In a modified example, instead of the person table 71, house table 72, and item table 73, the memory 112 stores a prediction model 76 (see Figure 8(A)) that has been machine-trained using keywords and item images obtained in S204 as training data during the learning process (see Figure 6(A)). The CPU 111 may use the machine-trained prediction model 76 to obtain an estimated result predicted from the item name obtained in S306 and the user ID obtained in S307.

[0091] In the first modified example, in S204 of the learning process (see Figure 6(A)), further information such as person information indicating family members, date and time information indicating the weather, and weather information indicating the weather may be obtained as training data by text mining or image recognition processing of the message information to be processed. In this case, the prediction model 76 is trained using the person information, date and time information, weather information, keywords, and item images obtained in S204 as training data. Note that in S204, it is sufficient to obtain at least one of the person information, date and time information, and weather information that is predetermined. In this case, the CPU 111 may use the trained prediction model 76 to obtain an estimated result predicted from the item name obtained in S306, the user ID obtained in S307, and the date and time information and weather information obtained by APIs provided by the OS, etc.

[0092] According to the first modification, the accuracy of the estimation results is improved because a machine learning-prepared predictive model 76 is used. Furthermore, the accuracy of the estimation results is also improved because at least one of the following is further used in machine learning and location estimation: person information, date and time information, and weather information.

[0093] In the first modified example, the prediction model 76 was stored in memory 112. However, this is not the only option; as shown in Figure 8(B), the prediction model 76 may also be stored in a cloud server 35 accessible by the information processing device 10 via the internet 40. This eliminates the need to store the prediction model 76 in the information processing device 10, thus allowing the information processing device 10 to be provided at a lower cost.

[0094] [Second variation] In the second variation, before entering the storage room 51 of the house 50, one of the family members operates the information processing device 10 to start program 222 and then performs operations according to a predetermined procedure. This operation triggers the CPU 21 of the mobile terminal device 20 to send reservation form information for reserving the pickup or return of item 60 from the wireless communication IF 25 to the internet 40. The reservation form information includes the item name, user ID, reservation date and time, and operation type. The reservation date and time is information indicating the scheduled date and time for pickup or return.

[0095] The CPU 111 of the information processing device 10 receives reservation form information via the communication IF 113. The CPU 111 creates a new log record 750 in the usage record information 75. In the new log record 750, the creation date and time is registered as information indicating the current date and time obtained from an API such as the OS. The item name, user ID, operation type, and scheduled date and time are registered as those included in the reservation form information. If the log record 750 was created based on the reservation form information, "1" is registered as the reservation flag. The matching flag is information indicating whether or not a matching process (see Figure 9) has been executed for the target log record 750. "0" is registered as the matching flag to indicate that the matching process has not been executed.

[0096] In the second modified example, the third program 19 differs from the third program 19 in the embodiment in that, as shown in Figure 9, it executes a matching process consisting of S501 and S502 immediately after the execution of S108. In S501, the CPU 111 obtains the current date and time information from an API such as the OS, and then determines whether or not there is a log record 750 in the usage record information 75 that satisfies the fourth condition. The fourth condition is that the scheduled date and time is included within the range of the reference time from the current date and time. If it is determined that there is "no" (No in S501), the process transitions to S109. If it is determined that there is "yes" (Yes in S501), the process transitions to S502. In S502, the CPU 111 changes the matching flag in the log record 750 to "1". In S502, the CPU 111 identifies the item name registered in the log record 750. In S109, the CPU 111 prioritizes image recognition processing on the item image corresponding to the item name identified in S502. This may shorten the time from the start of S109 to the determination of "Yes" in S110.

[0097] [Other variations] In this embodiment, if the item 60 could not be identified from the captured image obtained within a predetermined time after the door 52 opened (indicated as "failure" in S109 of Figure 5), the first event occurred, and the removal process in S116 was executed. Conversely, if the item 60 could be identified from the captured image obtained within a predetermined time after the door 52 opened (indicated as "Yes" in S110 of Figure 5), the second event occurred, and the return process in S117 was executed.

[0098] However, the CPU 111 may continue to receive captured images from the time the door 52 is opened until one of the following processes is completed: removal (S116 in Figure 5), return (S117), or update (S115). In this case, the CPU 111 attempts to identify the extracted product image by performing the same image recognition process as in S109 (an example of the second acquisition process) on each of the captured images received every first hour.

[0099] Upon identifying the item image, the CPU 111 displays a third message and a dialog box prompting a response to the third message on the touch panel 13. The third message prompts the entrant to specify one of the following: take out item 60, return item 60, or register item 60 as a new item. Depending on whether the entrant specifies return of item 60 or registration of item 60 as a new item on the touch panel 13, the CPU 111 decides, based on the operation signal from the touch panel 13, whether to perform a take-out process (S116 in Figure 5), a return process (S117), or an update process (S115).

[0100] Meanwhile, if the CPU 111 is unable to identify an item image for a second period of time longer than the first period, it displays a fourth message and a dialog box prompting the user to respond to the fourth message on the touch panel 13. The fourth message prompts the user to point the item 60 towards the camera 12. After displaying the fourth message, the CPU 111 identifies the extracted item image from the image captured by the camera 12 and, based on the operation signal from the touch panel 13, decides whether to perform a take-out process (S116 in Figure 5), a return process (S117), or an update process (S115). [Explanation of Symbols]

[0101] 100 System 10. Information Processing Devices 12...Camera 13. Touch panel 111···CPU 112...memory 20, 20A~20H... Mobile terminal device 21..CPU 22...memory 23...Camera 30..SNS server 35. Cloud Server 40. Internet 50... Housing 51... Storage shed 60...goods

Claims

1. An information processing device having a communication interface, an image input device, and a controller, The controller executes a first acquisition process to obtain first item information and location information from message information received via the communication interface. This message information is transmitted to the internet by an external server providing the SNS. The first item information is information indicating an item, and the location information is information indicating a location. The above controller performs an update process to update estimation data using the first item information and location information acquired in the first acquisition process, A second acquisition process involves acquiring image information through the above-mentioned image input device and acquiring second item information indicating an item from the acquired image information. A third acquisition process is performed to acquire result information indicating the estimated location of the item shown in the image information, based on the second item information acquired in the second acquisition process described above and the estimation data described above. The above estimation data includes indices corresponding to items and locations, including an index indicating the number of times an item was taken to a location. The above controller updates the above indicators included in the estimation data based on the first item information and location information obtained in the first acquisition process in the above update process. The controller described above, in the third acquisition process described above, obtains an estimation result that numerically indicates the probability that the item shown in the image information is located at each of the locations included in the estimation data, based on the second item information described above and the estimation data described above, and obtains result information that indicates the location corresponding to at least the highest-ranking estimation result among the obtained estimation results.

2. The above controller is In the above first acquisition process, in addition to the above first item information and the above location information, at least one of the following is acquired as first additional information from the above message information: person information indicating a person, date and time information indicating a date and time, and weather information indicating the weather. In the update process described above, the estimation data is updated with the first item information, location information, and first additional information obtained in the first acquisition process described above. In the second acquisition process described above, in addition to the second item information, information of the same type as the first additional information is further acquired as second additional information from among person information, date and time information, and weather information. The information processing apparatus according to claim 1, wherein in the third acquisition process described above, the information processing apparatus acquires the result information based on the second item information and second additional information acquired in the second acquisition process described above, and the estimation data described above.

3. The above controller is In response to request information transmitted from an external mobile terminal device via the above communication interface, the above third acquisition process is executed. The information processing apparatus according to claim 1 or 2, further performing a transmission process to transmit the result information obtained in the third acquisition process described above to the mobile terminal device.

4. The above image input device is installed in a storage room in a house where the above articles are stored, as described in any one of claims 1 to 3.

5. The information processing apparatus according to any one of claims 1 to 4, wherein the controller updates the estimation data stored in the cloud server connected to the Internet during the update process.

Images