Room tidying recommendation system

Abstract

Get appropriate recommendations regarding tidying up. [Solution] The tidying recommendation system 1 includes a camera 21 that photographs the inside of a room and outputs the captured image as a room image 31, a storage unit 13 that stores a reference image 32 of the room taken in advance, and a control unit 11. The control unit 11 performs the following processes (S22-S26): dividing the room image 31 and the reference image 32 into multiple regions and comparing each region to obtain an overall clutter level P indicating the overall degree of clutter in the room and a partial clutter level Q indicating the degree of clutter in a part of the room; and obtaining recommendations regarding tidying up the room based on the obtained overall clutter level P and partial clutter level Q (S32-S34).

Description

Technical Field

[0001] The present invention relates to a tidying recommendation system that recommends to users to tidy up their living rooms.

Background Art

[0002] Regarding systems that recommend tidying up and cleaning to users, the following technologies have been conventionally known. Patent Document 1 describes a tidying support system that recognizes the mess situation in a house and the behavior of users in the house based on an image of the situation in the house, and presents tidying support information based on the mess situation and the behavior of the users to the users.

[0003] Patent Document 2 describes a garbage disposal site monitoring system including a sensor device that photographs an image of a garbage disposal site and transmits it to a monitoring server, and a monitoring server that calculates the degree of mess of the garbage disposal site based on the image received from the sensor device and issues a notification to prompt cleaning when the degree of mess is greater than or equal to a reference value.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The system described in Patent Document 1 recognizes the overall clutter level of a house based on captured images, but does not recognize the clutter level of specific areas within the house. The system described in Patent Document 2 calculates the overall clutter level of a garbage area, but does not calculate the clutter level of specific areas within that area. Furthermore, these systems do not adequately reflect the user's circumstances, such as their level of busyness or fatigue, in the tidying support information and notifications output by the system. As a result, it is not possible to obtain appropriate tidying recommendations that are tailored to the degree of clutter in specific areas or the user's circumstances.

[0006] This invention has been made in view of the above circumstances, and its purpose is to provide a means for obtaining suitable recommendations regarding tidying up. [Means for solving the problem]

[0007] (1) The tidying recommendation system of the present invention comprises a camera that photographs the inside of a room and outputs the captured image as a room image, a storage unit that stores a reference image of the room taken in advance, and a control unit. The control unit performs a clutter level acquisition process that divides the room image and the reference image into a plurality of regions and compares each region to obtain an overall clutter level indicating the overall degree of clutter in the room and a partial clutter level indicating the degree of clutter in a part of the room, and a recommendation acquisition process that obtains recommendations regarding tidying up the room based on the acquired overall clutter level and partial clutter level.

[0008] According to the above tidying recommendation system, the system compares the room image with a reference image for each area to obtain the overall and partial clutter levels of the room. Based on these two types of clutter levels, it obtains recommendations for tidying up the room, thereby providing appropriate recommendations that are tailored to the overall and partial clutter levels of the room.

[0009] (2) Preferably, the control unit has a first threshold for classifying the overall level of clutter and a second threshold for classifying the partial level of clutter, and in the recommendation acquisition process, the control unit may acquire the class to which the acquired overall level of clutter belongs based on the first threshold, acquire the class to which the acquired partial level of clutter belongs based on the second threshold, and acquire the recommendation based on the two acquired classes.

[0010] With the above configuration, recommendations can be easily obtained based on a comparison of the overall level of clutter with a first threshold, and a comparison of the partial level of clutter with a second threshold, thereby providing suitable recommendations tailored to the overall and partial levels of clutter in a living space.

[0011] (3) Preferably, the control unit may further perform the following: a process to obtain a subjective scale regarding the implementation of cleaning the living room based on input from the user; and a process to obtain an objective scale regarding the cleaning of the living room without input from the user; a process to obtain a tolerance score regarding the implementation of cleaning the living room based on at least one of the obtained subjective scale and objective scale; and a process to obtain the first threshold and the second threshold based on the obtained tolerance score.

[0012] According to the above configuration, a tolerance score is obtained based on at least one of a subjective scale and an objective scale, and a first threshold and a second threshold are obtained based on the obtained tolerance score, thereby enabling the acquisition of suitable recommendations tailored to the user's situation.

[0013] (4) Preferably, the control unit may have a plurality of first thresholds and a plurality of second thresholds.

[0014] According to the above configuration, the overall level of clutter can be classified into at least three classes based on multiple first thresholds, and the partial level of clutter can be classified into at least three classes based on multiple second thresholds, thereby obtaining suitable recommendations according to the overall and partial levels of clutter in a living space.

[0015] (5) Preferably, the storage unit stores a table containing the recommendations corresponding to combinations of the multiple classes of overall clutter and the multiple classes of partial clutter, and the control unit may, in the recommendation acquisition process, acquire the recommendations from the table corresponding to combinations of the acquired overall clutter and the acquired partial clutter.

[0016] With the above configuration, recommendations for tidying up can be easily obtained by referring to a table that stores recommendations corresponding to combinations of multiple classes for overall clutter and multiple classes for partial clutter.

[0017] (6) Preferably, in the clutter acquisition process, the control unit may calculate an evaluation value for each region of the room image and the reference image based on the sum of the differences between the pixel values ​​of the room image and the pixel values ​​of the reference image, and acquire a value based on the average of the evaluation values ​​as the overall clutter level.

[0018] According to the above configuration, an evaluation value is calculated based on the sum of the differences in pixel values ​​for each region of the living room image and the reference image, and an overall clutter level indicating the overall degree of clutter in the living room can be obtained based on the average value of the calculated evaluation values.

[0019] (7) Preferably, the control unit may, in the clutter acquisition process, acquire a value based on the standard deviation of the evaluation value as the partial clutter level.

[0020] According to the above configuration, based on the standard deviation of the calculated evaluation value, a partial disorder degree indicating the partial disorder level in the living room can be obtained.

[0021] (8) Preferably, in the disorder degree acquisition process, the control unit may obtain, as the partial disorder degree, a value obtained by dividing the standard deviation of the evaluation value by the maximum value that the standard deviation of the evaluation value can take when the average value of the evaluation value is given.

[0022] According to the above configuration, based on the calculated standard deviation of the evaluation value and the maximum value that the standard deviation can take, an overall disorder degree indicating the partial disorder level in the living room can be obtained.

[0023] (9) Preferably, in the disorder degree acquisition process, the control unit may obtain, as the partial disorder degree, a value based on the maximum value or the minimum value of the evaluation value.

[0024] According to the above configuration, based on the maximum value or the minimum value of the calculated evaluation value, a partial disorder degree indicating the partial disorder level in the living room can be easily obtained.

[0025] (10) Preferably, in the disorder degree acquisition process, the control unit may perform rounding processing on the evaluation value and perform correction processing on the partial disorder degree based on the evaluation value after the rounding processing.

[0026] According to the above configuration, by correcting the partial disorder degree based on the evaluation value after the rounding processing, a suitable partial disorder degree can be obtained.

[0027] (11) Preferably, the storage unit stores a plurality of the reference images with different shooting conditions, and before executing the disorder degree acquisition process, the control unit selects, as the reference image for comparison with the living room image, the reference image taken under the shooting conditions corresponding to the shooting conditions of the living room image from the plurality of reference images stored in the storage unit.

[0028] According to the above configuration, by selecting a reference image to be compared with the room image according to the shooting conditions, an image with a small difference from the room image can be selected as the reference image, and the overall and partial clutter levels can be obtained.

[0029] (12) Preferably, the control unit may perform a process to convert the pixel values ​​of at least one of the room image and the reference image so as to reduce the difference between the pixel values ​​of specific pixels in the room image and the pixel values ​​of specific pixels in the reference image, before performing the clutter acquisition process.

[0030] According to the above configuration, by transforming the pixel values ​​of at least one of the room image and the reference image so that the difference in pixel values ​​of specific pixels between the room image and the reference image decreases, the difference in pixel values ​​between the room image and the reference image can be reduced, thereby obtaining the overall clutter level and the partial clutter level.

[0031] (13) Preferably, the tidying recommendation system may further include a communication unit that transmits the recommendation acquired by the control unit to a user terminal.

[0032] With the above configuration, the acquired recommendations can be sent to the user's terminal, allowing the user to recognize the acquired recommendations.

[0033] (14) Preferably, the tidying recommendation system may further include a notification unit that notifies the user of the recommendation acquired by the control unit.

[0034] With the above configuration, the user can be made aware of the acquired recommendations by notifying them of the recommendations. [Effects of the Invention]

[0035] According to the present invention, it is possible to obtain appropriate recommendations regarding tidying up. [Brief explanation of the drawing]

[0036] [Figure 1] Figure 1 is a block diagram showing the configuration of a tidying-up recommendation system 1 according to an embodiment of the present invention. [Figure 2] Figure 2(A) shows a room image 31 captured by camera 21 of the tidying recommendation system 1, and Figure 2(B) shows a reference image 32 captured in advance by camera 21. [Figure 3] Figure 3(A) shows the sleep duration database 33 stored in the memory unit 13 of the tidying recommendation system 1, and Figure 3(B) shows the stress level database 34 stored in the memory unit 13. [Figure 4] Figure 4 shows the recommendation table 35 stored in the memory unit 13. [Figure 5] Figure 5(A) is a flowchart showing the operation of the control unit 11 of the tidying recommendation system 1, and Figure 5(B) is a flowchart showing the pre-processing of the control unit 11. [Figure 6] Figure 6 is a flowchart of the tidying-up recommendation process of the control unit 11. [Figure 7] Figure 7 is a flowchart of the partial clutter acquisition process of the control unit 11. [Figure 8] Figure 8 is a flowchart of the post-processing steps of the control unit 11. [Figure 9] Figure 9(A) shows the confirmation screen 61 displayed on the display unit 46 of the user terminal 4, and Figure 9(B) shows the question screen 63 displayed on the display unit 46. [Figure 10] Figure 10(A) shows the room image 31 shown in Figure 2(A) divided into multiple regions, and Figure 10(B) shows the degree of clutter obtained for each region shown in Figure 10(A). [Figure 11] Figure 11(A) shows the rules for obtaining a subjective scale based on user input, Figure 11(B) shows the rules for obtaining an objective scale based on data, and Figure 11(C) shows the rules for obtaining a threshold based on tolerance scores. [Figure 12] Figure 12 is a flowchart showing the operation of the control unit 41 of the user terminal 4. [Modes for carrying out the invention]

[0037] The following description will refer to the drawings and explain a tidying-up recommendation system according to an embodiment of the present invention. It should be noted that the embodiments described below are merely examples of the present invention, and the embodiments of the present invention can be appropriately modified without altering the essence of the invention.

[0038] [Configuration of the tidying-up recommendation system 1] As shown in Figure 1, the tidying recommendation system 1 according to this embodiment includes a home server 10 and a camera 21. The tidying recommendation system 1 may further include a heart rate sensor 22 and a sleep sensor 23. The tidying recommendation system 1 is a system that obtains recommendations regarding tidying up a living room. Recommendation means "suggestion" or "recommendation". The person who will tidy up the living room based on the recommendation is determined to be a specific individual. Hereinafter, this person will be referred to as the "user".

[0039] The home server 10 is a computer installed in a living room. The home server 10 is, for example, a personal computer. The home server 10 comprises a CPU 12, a storage unit 13, a microphone 14, an input unit 15, a display unit 16, a communication unit 17, and an interface unit 18. The storage unit 13 stores programs (not shown) that the CPU 12 executes, and data that the CPU 12 references. The CPU 12 performs various processes by executing the programs stored in the storage unit 13. The CPU 12 and the storage unit 13 function as a control unit 11.

[0040] Microphone 14 is an input device for inputting the user's voice. Input unit 15 is an input device other than microphone 14 for the user to input. Input unit 15 is, for example, a mouse, keyboard, or touchpad. Display unit 16 is a display that shows the screen. Display unit 16 is, for example, a liquid crystal display. Communication unit 17 is a circuit that performs wireless communication. Communication unit 17 communicates according to communication standards such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). Interface unit 18 is an interface circuit for connecting camera 21, heart rate sensor 22, and sleep sensor 23. CPU 12, memory unit 13, microphone 14, input unit 15, display unit 16, communication unit 17, and interface unit 18 are connected to bus 19.

[0041] [User terminal 4 and wearable terminal 5] The tidying recommendation system 1 communicates wirelessly with a user terminal 4 and a wearable terminal 5. The user terminal 4 is a terminal device operated by the user. The user terminal 4 is, for example, a smartphone. The user terminal 4 comprises a control unit 41, an input unit 45, a display unit 46, and a communication unit 47. The control unit 41 includes a CPU and a memory unit (not shown). The control unit 41, input unit 45, display unit 46, and communication unit 47 are connected to a bus 49. The functions of the control unit 41, input unit 45, display unit 46, and communication unit 47 are the same as those of the control unit 11, input unit 15, display unit 16, and communication unit 17, respectively.

[0042] The wearable terminal 5 is a small terminal device worn on the user's body. The wearable terminal 5 comprises a control unit (not shown), a sensor unit 55, a display unit (not shown), and a communication unit 57. The functions of each component of the wearable terminal 5 are the same as those of each component of the user terminal 4. The sensor unit 55 measures the user's heart rate and outputs heart rate data. The communication unit 57 transmits the heart rate data output from the sensor unit 55 to the home server 10.

[0043] [Camera 21, room image 31, and reference image 32] Camera 21 is mounted, for example, on the ceiling or wall of the room, so that it can film the inside of the room. The user tidies up the room before activating the tidying recommendation system 1. In the tidied room, furniture that is not normally moved (for example, chests of drawers or TV stands) is laid out in its designated place, and movable items (for example, clothes, dishes, magazines, etc.) are put away in places where they cannot be seen.

[0044] After tidying up their room, the user operates the input unit 15 of the home server 10 to cause the camera 21 to photograph the room. The camera 21 photographs the room and outputs the captured image. The image captured by the camera 21 at this time is called the "reference image". In the drawing, the reference image is denoted by reference numeral 32. The reference image 32 is an image of the room that was photographed in advance. The reference image 32 is stored in the storage unit 13 via the interface unit 18.

[0045] The camera 21 continues to photograph the interior of the room and outputs the captured images. The images captured by the camera 21 at this time are called "room images." In the drawings, the room images are denoted by reference numeral 31. Room image 31 is the most recently captured image of the interior of the room. Room image 31 is also stored in the storage unit 13 via the interface unit 18.

[0046] Figures 2(A) and (B) show examples of room image 31 and reference image 32, respectively. The room shown in room image 31 is in a messier state than the room shown in reference image 32. The tidying recommendation system 1 compares room image 31 and reference image 32 to obtain recommendations regarding tidying up the room. Note that reference image 32 is an image of a room that the user judged to be tidy, and the criteria for judging a room to be tidy differ from user to user. Therefore, the degree to which the room shown in reference image 32 is tidy also differs from user to user. Hereafter, room image 31 and reference image 32 are assumed to be 256-level grayscale images.

[0047] [Heart rate sensor 22 and sleep sensor 23] The heart rate sensor 22 is a non-contact sensor placed in the living space. The heart rate sensor 22 measures the heart rate of the user in the living space and outputs heart rate data. The heart rate sensor 22 measures the heart rate non-contact by, for example, detecting minute skin displacements caused by the heartbeat using millimeter-wave radar. The heart rate data acquired by the heart rate sensor 22 is supplied to the control unit 11 via the interface unit 18.

[0048] The sleep sensor 23 is a sensor placed around the bed in the living room, or under the bedding. The sleep sensor 23 detects the heart rate, respiration, and body movements of the user lying in bed, and determines whether the user is asleep or awake. The results detected by the sleep sensor 23 are supplied to the control unit 11 via the interface unit 18. Note that the tidying recommendation system 1 receives heart rate data from the wearable terminal 5, so it is not necessarily required to have a heart rate sensor 22 or a sleep sensor 23.

[0049] [Sleep Time Database 33] In addition to the room image 31 and reference image 32, the memory unit 13 stores a sleep duration database 33 (Figure 3(A)), a stress level database 34 (Figure 3(B)), and a recommendation table 35 (Figure 4). The sleep duration database 33 and the stress level database 34 are updated based on heart rate data received from the wearable terminal 5.

[0050] As described above, the communication unit 57 of the wearable terminal 5 transmits heart rate data output from the sensor unit 55 to the home server 10. The control unit 11 of the home server 10 receives the heart rate data from the wearable terminal 5 via the communication unit 17. The control unit 11 performs signal processing on the received heart rate data to determine whether the user is awake or asleep. Based on the determination result, the control unit 11 obtains the user's sleep time.

[0051] As shown in Figure 3(A), the sleep time database 33 stores 14 days' worth of the user's sleep time acquired by the control unit 11. More specifically, the sleep time database 33 stores the current day's sleep time, the sleep time from 1 day to 14 days ago, and the average value. The average value is the average of the sleep time from 1 day to 14 days ago. When the control unit 11 acquires new sleep time, it overwrites the current day's sleep time and the sleep time from 1 day to 13 days ago with the sleep time from 1 day to 14 days ago, and overwrites the current day's sleep time with the newly acquired sleep time. At this time, the previous sleep time from 14 days ago is discarded. Subsequently, the control unit 11 calculates the average value of the sleep time from 1 day to 14 days ago and writes the calculated average value to the sleep time database 33. If the tidying recommendation system 1 is equipped with a sleep sensor 23, the control unit 11 may update the sleep time database 33 based on the results acquired by the sleep sensor 23.

[0052] [Stress Level Database 34] The control unit 11 obtains the user's stress level based on heart rate data received from the wearable device 5. The control unit 11 extracts 3 minutes of heart rate data per hour from the heart rate data received from the wearable device 5. For example, the control unit 11 obtains CSI (Cardiac Sympathetic Index) and CVI (Cardiac Vagal Index) values ​​by performing a Lorenz plot analysis on the 3 minutes of heart rate data from 55 to 58 minutes past the hour.

[0053] As shown in Figure 3(B), the stress level database 34 stores 48 hours' worth of CSI and CVI values ​​acquired by the control unit 11. More specifically, the stress level database 34 stores the most recent CSI and CVI values, the CSI and CVI values ​​from 1 hour to 48 hours prior, the average CSI value, and the average CVI value. The average CSI value is the average of the CSI values ​​from 1 hour to 48 hours prior. The average CVI value is the average of the CVI values ​​from 1 hour to 48 hours prior.

[0054] When the control unit 11 acquires new CSI and CVI values, it overwrites the most recent CSI and CVI values ​​and the CSI and CVI values ​​from 1 hour to 47 hours ago with the CSI and CVI values ​​from 1 hour to 48 hours ago, respectively, and then overwrites the most recent CSI and CVI values ​​with the newly acquired CSI and CVI values, respectively. At this time, the previous CSI and CVI values ​​from 48 hours ago are discarded. Subsequently, the control unit 11 calculates the average value of the CVI values ​​from 1 hour to 48 hours ago and the average value of the CVI values ​​from 1 hour to 48 hours ago, and writes the two calculated average values ​​to the stress level database 34. If the tidying recommendation system 1 is equipped with a heart rate sensor 22, the control unit 11 may update the stress level database 34 based on the heart rate data output from the heart rate sensor 22.

[0055] [Operation of the control unit 11] The control unit 11 performs the operations shown in Figures 5 to 8 to obtain recommendations regarding tidying up the living room. Before the control unit 11 starts operating, the memory unit 13 stores the reference image 32 and the recommendation table 35. While the control unit 11 is operating, the living room image 31 stored in the memory unit 13 is updated using images captured by the camera 21. The sleep time database 33 and the stress level database 34 are updated based on heart rate data received from the wearable terminal 5.

[0056] The control unit 11 executes the loop process shown in Figure 5(A) once a day. First, in S1, the control unit 11 performs pre-processing (Figure 5(B)), then in S2, it performs tidying recommendation processing (Figure 6), and then in S3, it performs post-processing (Figure 8). After that, the control unit 11 proceeds to S1. In pre-processing, the control unit 11 waits until it is time to execute tidying recommendation processing. In tidying recommendation processing, the control unit 11 obtains recommendations regarding tidying up the living room. In post-processing, the control unit 11 makes the user aware of the obtained recommendations and evaluates the tidying up of the living room performed by the user.

[0057] [Pre-processing] At the beginning of the preprocessing (Figure 5(B)), the control unit 11 waits until 9:00 AM (S11), and then proceeds to S12. Next, the control unit 11 determines whether it is a weekday or not (S12). In S12, the control unit 11 refers to a calendar (not shown) stored in the memory unit 13 to determine whether it is a weekday or not. If the control unit 11 determines that it is a weekday (S12: Yes), it proceeds to S13. In this case, the control unit 11 waits until 7:00 PM (S13), and then proceeds to S14. If the control unit 11 determines in S12 that it is not a weekday (S12: No), it proceeds to S14 without executing S13.

[0058] Next, the control unit 11 sends a start notification to the user terminal 4 indicating that the tidying-up recommendation process will begin (S14). The control unit 11 controls the communication unit 17 to send the start notification to the user terminal 4.

[0059] When the control unit 41 of the user terminal 4 receives a start notification from the home server 10 via the communication unit 47, it displays the confirmation screen 61 shown in Figure 9(A) on the display unit 46. The confirmation screen 61 displays instructions to start the tidying-up recommendation process, a confirmation prompt, and a button 62. When the user views the confirmation screen 61, they operate the input unit 45 to select the button 62. When the button 62 is selected, the control unit 41 controls the communication unit 47 to send a confirmation notification to the home server 10 indicating that the user has confirmed the information.

[0060] Next, the control unit 11 of the home server 10 determines whether or not it has received a confirmation notification (S15). In S15, the control unit 11 receives a confirmation notification from the user terminal 4 via the communication unit 17. The control unit 11 determines that it has received a confirmation notification if it receives the notification within, for example, 3 minutes after sending the start notification. If the control unit 11 determines that it has received a confirmation notification (S15: Yes), it terminates the preprocessing and returns. If the control unit 11 determines in S15 that it did not receive a confirmation notification (S15: No), it proceeds to S16. In this case, the control unit 11 waits for 1 hour (S16) and proceeds to S14 after 1 hour has elapsed.

[0061] [Tidying-up recommendation processing] Following the preprocessing, the control unit 11 performs a tidying recommendation process (Figure 6). At the beginning of the tidying recommendation process, the control unit 11 acquires a room image 31 (S21). In S21, the control unit 11 controls the camera 21 to take a picture of the room. The camera 21 takes a picture of the room and outputs the captured room image 31. The room image 31 is stored in the storage unit 13 via the interface unit 18. The storage unit 13 has a reference image 32 stored in it beforehand.

[0062] Next, the control unit 11 divides the room image 31 and the reference image 32 into multiple regions (S22). In S22, the control unit 11 divides the room image 31 and the reference image 32 stored in the storage unit 13 in the vertical and horizontal directions. For example, if the room image 31 shown in Figure 2(A) is divided into 16 regions of the same size, the result shown in Figure 10(A) is obtained. Hereinafter, the number of regions will be N (where N is a natural number of 2 or more), and the divided regions will be denoted as Rk (where k is a natural number of 1 or more and less than or equal to N).

[0063] Next, the control unit 11 obtains the degree of clutter Ek for each region Rk (S23). In S23, the control unit 11 obtains the degree of clutter Ek for each region Rk according to the following equation (1).

number

[0064] The clutter level Ek is the value obtained by dividing the average of the absolute values ​​of the difference between the pixel value Xk(i,j) of the room image 31 and the pixel value Yk(i,j) of the reference image 32 in region Rk by the maximum grayscale value of 255 (expressed as a percentage). The clutter level Ek is an evaluation value based on the sum of the differences between the pixel values ​​of the room image 31 and the pixel values ​​of the reference image 32 for each region Rk of the room image 31 and the reference image 32. For example, the control unit 11 acquires 16 clutter levels shown in Figure 10(B) based on the room image 31 shown in Figure 2(A) and the reference image 32 shown in Figure 2(B).

[0065] Next, the control unit 11 obtains an overall clutter level P based on N clutter levels Ek (S24). The overall clutter level P indicates the overall degree of clutter in the room. In S24, the control unit 11 obtains the average value of the N clutter levels Ek as the overall clutter level P. The control unit 11 obtains a value based on the average value of the evaluation values ​​(in this case, the average value) as the overall clutter level P. For example, the control unit 11 obtains an overall clutter level P of 23% based on the 16 clutter levels shown in Figure 10(B).

[0066] Next, the control unit 11 obtains a partial clutter level Q based on N clutter levels Ek (S25, S26). In S25, the control unit 11 executes the partial clutter level acquisition process shown in Figure 7, and in S26, it sets the value Qr obtained in the partial clutter level acquisition process as the partial clutter level Q. The partial clutter level Q indicates the degree of partial clutter in the living space.

[0067] At the beginning of the partial clutter acquisition process (Figure 7), the control unit 11 obtains the average value H of N clutter values ​​Ek according to equation (2a) (S41). Next, the control unit 11 obtains the standard deviation V of N clutter values ​​Ek according to equation (2b) (S42). Next, the control unit 11 obtains the maximum value Vmax that the standard deviation of N clutter values ​​Ek can take, given the average value H obtained in S41, according to equation (2c) (S43). Next, the control unit 11 takes the value Qr obtained by dividing the standard deviation V obtained in S42 by the maximum value Vmax obtained in S43 (expressed as a percentage), according to equation (2d) (S44). Next, the control unit 11 terminates the partial clutter acquisition process and returns.

number

[0068] Here, we will explain the maximum value Vmax obtained in S43. Given the mean H of N scatter values ​​Ek, the minimum standard deviation of the N scatter values ​​Ek is 0. On the other hand, the standard deviation of the N scatter values ​​Ek is maximized when all N scatter values ​​Ek are either 0% or 100%. Let a be the number of regions Rk where the scatter value Ek is 100%, and b be the number of regions Rk where the scatter value Ek is 0%. Then equations (3a) and (3b) hold. a + b = N …(3a) (100 × a + 0 × b) / N = H …(3b)

[0069] From equations (3a) and (3b), a and b are calculated as follows. a = H·N / 100 b = N(1 - H / 100) Therefore, given the mean H of N scatter values ​​Ek, the maximum possible standard deviation Vmax of the N scatter values ​​Ek can be calculated as follows. Vmax = √[{a(100-H) 2 +bH 2} / N] =√{H(100-H)}

[0070] The value Qr obtained in S44 (partial clutter level Q obtained in S26) is a value within the range of 0% to 100%. The larger the partial clutter level Q, the greater the difference between the cluttered and clean parts of the room. The control unit 11 acquires the value Qr obtained in S44 as the partial clutter level Q, which indicates the degree of partial clutter in the room.

[0071] For example, based on the 16 levels of clutter shown in Figure 10(B), the control unit 11 obtains a mean level H of 23% and a standard deviation V of 18.6%. Given a mean level of 23%, it obtains a maximum value Vmax of 38.9% for the standard deviation of clutter and a partial level Q of 47.7%.

[0072] Next, the control unit 11 obtains a subjective scale regarding the implementation of tidying up the living room (S27). The subjective scale is obtained based on input from the user. The subjective scale will be a relatively large value if it is acceptable not to tidy up the living room. For example, if the user answers "tired", the subjective scale will be a relatively larger value than if the user answers "energetic".

[0073] In S27, the control unit 11 controls the communication unit 17 to send a response request to the user terminal 4 indicating that it is requesting answers to four questions. When the control unit 41 of the user terminal 4 receives the response request from the home server 10 via the communication unit 47, it displays the question screen 63 shown in Figure 9(B) on the display unit 46. The question screen 63 displays four questions, eight buttons 64a to 64h, and a send button 65. When the user views the question screen 63, they operate the input unit 45 of the user terminal 4 to select the button 64 corresponding to the answer to each question.

[0074] The first question concerns "busyness." The user operates the input unit 45 to select either button 64a, which corresponds to "able to tidy up," or button 64b, which corresponds to "unable to tidy up." The second question concerns "fatigue." The user operates the input unit 45 to select either button 64c, which corresponds to "energetic," or button 64d, which corresponds to "tired." The third question concerns "the state of roommates" regarding tidying up the room. The user operates the input unit 45 to select either button 64e, which corresponds to "angry," or button 64f, which corresponds to "nothing." The fourth question concerns "receiving guests," that is, whether the user thinks they can invite guests to their room. The user operates the input unit 45 to select either button 64g, which corresponds to "cannot invite guests," or button 64h, which corresponds to "can invite guests." Finally, the user operates the input unit 45 to select the send button 65.

[0075] When the transmit button 65 is selected, the control unit 41 controls the communication unit 47 to send the four responses to the home server 10. The control unit 11 of the home server 10 receives the four responses from the user terminal 4 via the communication unit 17. Subsequently, the control unit 11 obtains subjective scales based on the four responses, according to the rules shown in Figure 11(A). For example, if the answer to the question "busyness" is "able to tidy up," the control unit 11 assigns a subjective scale score of 0, and if the answer is "unable to tidy up," it assigns a subjective scale score of 1. The control unit 11 obtains four subjective scales based on the four responses.

[0076] Next, the control unit 11 acquires an objective scale regarding the implementation of room tidying (S28). The objective scale is acquired based on data, without user input. Similar to the subjective scale, the objective scale will be a relatively large value when not tidying the room is acceptable. For example, if the amount of sleep on a given day is shorter than usual, the objective scale will be a relatively larger value than if the amount of sleep on that day was the same as usual.

[0077] In S28, the control unit 11 reads the sleep duration for the day and the average sleep duration (hereinafter referred to as average sleep duration) from the sleep duration database 33 stored in the memory unit 13. Based on the two values ​​read, the control unit 11 obtains an objective scale related to sleep duration according to the rules shown in Figure 11(B). For example, if the sleep duration for the day is 0.9 times or more the average sleep duration, the control unit 11 sets the objective scale to 0 points, and if the sleep duration for the day is 0.6 times or more but less than 0.8 times the average sleep duration, the objective scale is set to 2 points.

[0078] The control unit 11 obtains the user's energy level by performing voice emotion analysis on the user's voice data input via the microphone 14. Voice emotion analysis can be performed using commercially available programs. The energy level will be relatively high when the user is energetic. Based on the latest energy level, the control unit 11 obtains an objective scale related to the energy level according to the rules shown in Figure 11(B). For example, if the energy level is 0.9 times or more the standard value, the control unit 11 sets the objective scale to 0 points, and if the energy level is 0.6 times or more but less than 0.8 times the standard value, the objective scale is set to 2 points.

[0079] The control unit 11 reads the most recent CVI value and the average value of the CVI values ​​(hereinafter referred to as the average CVI value) from the stress level database 34 stored in the memory unit 13. The CVI value will be relatively large when the user is experiencing a high level of stress. Based on the two values ​​read, the control unit 11 obtains an objective scale related to the stress level according to the rules shown in Figure 11(B). For example, if the most recent CVI value is less than 1.1 of the average CVI value, the control unit 11 sets the objective scale to 0 points, and if the most recent CVI value is 1.2 times or more but less than 1.4 times the average CVI value, the objective scale is set to 2 points.

[0080] The control unit 11 extracts the user's scheduled visits from the schedule data (not shown) stored in the memory unit 13. Based on the extracted visitor schedules, the control unit 11 determines an objective scale for the visitor schedule according to the rules shown in Figure 11(B). For example, the control unit 11 sets the objective scale to 0 points if there is a visitor scheduled within one week, and sets the objective scale to 2 points if there is a visitor scheduled more than two weeks away but within four weeks.

[0081] Next, the control unit 11 obtains a tolerance score based on subjective and objective scales (S29). The tolerance score will be relatively large if it is permissible not to tidy up the living space. In S29, the control unit 11 calculates the sum of the four subjective scales obtained in S27 and the sum of the four objective scales obtained in S28, and calculates the tolerance score according to the following equation (4). Tolerance score = (Total score on subjective scale) + (Total score on objective scale) × 2 …(4)

[0082] Next, the control unit 11 obtains two thresholds TP1 and TP2 for the overall clutter level P, and two thresholds TQ1 and TQ2 for the partial clutter level Q, based on the tolerance score calculated in S29 (S30). Thresholds TP1 and TQ1 are thresholds that serve as a standard for determining whether or not it is permissible not to tidy up the living space, and are called permissible thresholds. Thresholds TP2 and TQ2 are thresholds that serve as a standard for determining whether or not the living space should be tidied up, and are called absolute thresholds. Note that the permissible threshold TP1 is smaller than the absolute threshold TP2, and the permissible threshold TQ1 is smaller than the absolute threshold TQ2.

[0083] In S30, based on the tolerance score calculated in S29, the control unit 11 obtains the allowable threshold TP1 for the overall clutter P, the absolute threshold TP2 for the overall clutter P, the allowable threshold TQ1 for the partial clutter Q, and the absolute threshold TQ2 for the partial clutter Q, according to the rules shown in Figure 11(C). For example, if the tolerance score is 5 points or less, the control unit 11 sets the allowable thresholds TP1 and TQ1 to 30% and the absolute thresholds TP2 and TQ2 to 60%. If the tolerance score is 11 points or more and 15 points or less, the control unit 11 sets the allowable thresholds TP1 and TQ1 to 50% and the absolute thresholds TP2 and TQ2 to 60%.

[0084] Next, the control unit 11 corrects the two thresholds TP1 and TP2 related to the overall clutter level P (S31). In S31, if both the objective scale for vitality and the objective scale for stress level obtained in S28 are 3 points, the control unit 11 subtracts 10% from the acceptable threshold TP1 and the absolute threshold TP2 related to the overall clutter level P. When both the objective scale for vitality and the objective scale for stress level are 3 points, it means that the user is not energetic and is feeling stressed. In this case, the user finds it difficult to move their body, but is able to tidy up a small area. The correction in S31 takes this point into consideration.

[0085] [Recommendation Table 35] Here, Figure 4 is referenced to explain the recommendation table 35 stored in the memory unit 13. The overall clutter level P is classified into three classes CP1 to CP3 based on the tolerance threshold TP1 and the absolute threshold TP2. The partial clutter level Q is classified into three classes CQ1 to CQ3 based on the tolerance threshold TQ1 and the absolute threshold TQ2. More specifically, the overall clutter level P is classified into class CP1 if it is less than the tolerance threshold TP1, class CP2 if it is greater than or equal to the tolerance threshold TP1 and less than the absolute threshold TP2, and class CP3 if it is greater than or equal to the absolute threshold TP2. The partial clutter level Q is classified into class CQ1 if it is less than the tolerance threshold TQ1, class CQ2 if it is greater than or equal to the tolerance threshold TQ1 and less than the absolute threshold TQ2, and class CQ3 if it is greater than or equal to the absolute threshold TQ2.

[0086] Recommendation table 35 stores nine recommendations corresponding to combinations of overall clutter level P (classes CP1-CP3) and partial clutter level Q (classes CQ1-CQ3). Figure 4 shows the room condition, whether tidying is needed, and the recommendation for each of the nine class combinations. For example, for the combination of class CP1 and class CQ1, recommendation table 35 stores "clean" as the room condition, "no tidying needed" as the tidying need, and "very tidy" as the recommendation. Also, for the combination of class CP2 and class CQ3, recommendation table 35 stores "acceptable" as the room condition, "partial tidying needed" as the tidying need, and "tidy up at least some of the items you've placed."

[0087] The nine recommendations stored in recommendation table 35 are predetermined based on the overall clutter level P and the partial clutter level Q. Recommendation table 35 stores recommendations indicating that room cleaning is unnecessary for classes with low overall clutter levels P and Q, and recommendations recommending room cleaning for classes with high overall clutter levels P and Q. Furthermore, the nine recommendations are all different from each other. By referring to recommendation table 35, if the overall clutter level P is high and the partial clutter level Q is low, a recommendation recommending cleaning the entire room can be obtained. Conversely, if the overall clutter level P is low and the partial clutter level Q is high, a recommendation recommending partial room cleaning can be obtained.

[0088] [Continuing the tidying-up recommendation process] Figure 6 is referred to again to explain the continuation of the tidying recommendation process. When the control unit 11 reaches S32, the overall clutter level P, the partial clutter level Q, two thresholds TP1 and TP2 related to the overall clutter level P, and two thresholds TQ1 and TQ2 related to the partial clutter level have been acquired.

[0089] Next, the control unit 11 obtains the class to which the overall clutter level P belongs by comparing it with thresholds TP1 and TP2 (S32). In S32, the control unit 11 obtains the class to which the overall clutter level P belongs by comparing the overall clutter level P obtained in S24 with the two thresholds TP1 and TP2 related to the overall clutter level P obtained in S30. The control unit 11 obtains class CP1 as the class to which the obtained overall clutter level P belongs if it is less than the allowable threshold TP1, class CP2 if it is greater than or equal to the allowable threshold TP1 and less than the absolute threshold TP2, and class CP3 if it is greater than or equal to the absolute threshold TP2.

[0090] Next, the control unit 11 obtains the class to which the partial clutter Q belongs by comparing it with thresholds TQ1 and TQ2 (S33). In S33, the control unit 11 obtains the class to which the partial clutter Q belongs by comparing the partial clutter Q obtained in S25 and S26 with the two thresholds TQ1 and TQ2 related to the partial clutter Q obtained in S30. The control unit 11 obtains class CQ1 as the class to which the obtained partial clutter Q belongs if it is less than the allowable threshold TQ1, class CQ2 if it is greater than or equal to the allowable threshold TQ1 and less than the absolute threshold TQ2, and class CQ3 if it is greater than or equal to the absolute threshold TQ2.

[0091] Next, the control unit 11 obtains recommendations from the recommendation table 35 that correspond to the two classes (S34). In S34, the control unit 11 obtains recommendations from the recommendation table 35 according to the combination of the class to which the overall clutter level P obtained in S32 belongs and the class to which the partial clutter level Q obtained in S33 belongs. For example, if the class to which the overall clutter level P belongs is class CP1 and the class to which the partial clutter level Q belongs is class CQ1, the control unit 11 obtains the recommendation "Very well tidy" from the recommendation table 35. Also, if the class to which the overall clutter level P belongs is class CP2 and the class to which the partial clutter level Q belongs is class CQ3, the control unit 11 obtains the recommendation "Let's tidy up at least some of the things that are lying around" from the recommendation table 35.

[0092] Next, the control unit 11 displays the recommendation acquired in S34 on the display unit 16 (S35). The display unit 16 notifies the user of the recommendation acquired by the control unit 11. Next, the control unit 11 controls the communication unit 17 to transmit the recommendation acquired in S34 to the user terminal 4 (S36). In S36, along with the acquired recommendation, the control unit 11 transmits to the user terminal 4 the class to which the overall clutter level P acquired in S32 belongs, and the class to which the partial clutter level acquired in S33 belongs. Next, the control unit 11 finishes the tidying recommendation process and returns.

[0093] [Operation of the control unit 41 of user terminal 4] The control unit 41 of the user terminal 4 performs the operations shown in Figure 12 in order to process the recommendation sent from the home server 10. At the beginning of the process shown in Figure 12, the control unit 41 receives the recommendation from the home server 10 via the communication unit 47 (S71). In S71, the control unit 41 receives two classes that were sent from the home server 10 along with the recommendation.

[0094] Next, the control unit 41 displays the recommendation received in S71 on the display unit 46 (S72). In S72, the control unit 41 displays a recommendation on the display unit 46 such as, for example, "It's very well tidy" or "Let's tidy up at least some of the things that are there."

[0095] Next, the control unit 41 displays a screen on the display unit 46 asking the user if they are able to tidy up their room (S73). The screen displayed in S73 shows, for example, the question "Can you tidy up?", a button corresponding to "Yes", and a button corresponding to "No". The user looks at the displayed screen and operates the input unit 45 of the user terminal 4 to select either the button corresponding to "Yes" or the button corresponding to "No".

[0096] Next, the control unit 41 receives a response from the user via the input unit 45 (S74). Next, the control unit 41 determines whether the user is able to tidy up the room (S75). In S75, the control unit 41 determines that the user is able to tidy up the room if the response received in S74 is "yes".

[0097] If the control unit 41 determines that the user is able to tidy up the room (S75: Yes), the process proceeds to S76. In this case, the control unit 41 displays a message encouraging the user to tidy up on the display unit 46 (S76). In S76, the control unit 41 displays a message such as "Please do your best to tidy up" on the display unit 46.

[0098] If the control unit 41 determines in S75 that the user is unable to tidy up the room (S75: No), it proceeds to S77. In this case, the control unit 41 determines whether the level of clutter in the room is within an acceptable range (S77). In S77, the control unit 41 determines whether the level of clutter in the room is within an acceptable range based on the two classes received in S71. An acceptable range means that the overall level of clutter P is less than the absolute threshold TP2, and the partial level of clutter Q is less than the absolute threshold TQ2.

[0099] If the control unit 41 determines that the level of clutter in the room is within an acceptable range (S77: Yes), it proceeds to S78. In this case, the control unit 41 displays a message encouraging rest on the display unit 46 (S78). In S78, the control unit 41 displays a message such as "Please rest well" on the display unit 46.

[0100] If the control unit 41 determines in S77 that the level of clutter in the room is not within an acceptable range (S77: No), it proceeds to S79. In this case, the control unit 41 displays a tidying-up warning on the display unit 46 (S79). In S79, the control unit 41 displays a message on the display unit 46, such as "Please tidy up your room." The user sees the displayed message and tidies up the room. However, the user may not tidy up the room.

[0101] The control unit 41 executes one of S76, S78, or S79, and then proceeds to S80. Next, the control unit 41 displays a screen on the display unit 46 asking whether the room has been tidied up (S80). The screen displayed in S80 shows the question "Have you tidied up your room?", a button corresponding to "Yes", and a button corresponding to "No". The user looks at the displayed screen and operates the input unit 45 to select either the button corresponding to "Yes" or the button corresponding to "No".

[0102] Next, the control unit 41 receives a response from the user via the input unit 45 (S81). Next, the control unit 41 controls the communication unit 47 to transmit the response received in S81 to the home server 10 (S82). Next, the control unit 41 proceeds to S71.

[0103] [Post-processing] The control unit 11 of the home server 10 performs post-processing (Figure 8) following the tidying-up recommendation process. At the beginning of the post-processing, the control unit 41 receives a response regarding the tidying-up from the user terminal 4 via the communication unit 47 (S51). This response was transmitted by the control unit 41 of the user terminal 4 in S82.

[0104] Next, the control unit 41 determines whether or not the user has cleaned the room (S52). In S52, the control unit 11 determines whether or not the user has cleaned the room based on the response received in S51.

[0105] If the control unit 11 determines that the user has tidied up the room (S52: Yes), it proceeds to S53. In this case, the control unit 11 performs the same processing as in S21 to S26 from S53 to S58. That is, the control unit 11 controls the camera 21 to acquire a room image 31 (S53). Next, the control unit 11 divides the room image 31 and the reference image 32 into multiple regions (S54). Next, the control unit 11 acquires the degree of clutter Ek for each region Rk (S55). Next, the control unit 11 acquires the overall degree of clutter P2 and the partial degree of clutter Q2 based on the degree of clutter Ek for each region (S56-S58). The overall degree of clutter P2 and the partial degree of clutter Q2 acquired in S56-S58 are acquired for the room after the user has tidied up.

[0106] Next, the control unit 11 obtains the degree of improvement based on the degree of clutter P and Q before cleaning the room, and the degree of clutter P2 and Q2 after cleaning the room (S59). The degree of improvement indicates the extent to which the cleanliness of the room has improved as a result of cleaning the room. In S59, the control unit 11 obtains the degree of improvement R (expressed as a percentage) according to the following equation (5), based on the overall degree of clutter P and partial degree of clutter Q before cleaning the room, and the overall degree of clutter P2 and partial degree of clutter Q2 after cleaning the room. R = (100 - P²)(100 - Q²) / {(100-P)(100-Q)}×100 …(5) For example, if P=Q=60% and P2=Q2=20%, then according to equation (5), the improvement R obtained will be 400%.

[0107] Next, the control unit 11 controls the communication unit 17 to transmit the improvement level acquired in S59 to the user terminal 4 (S60). When the control unit 41 of the user terminal 4 receives the improvement level from the home server 10 via the communication unit 47, it displays the received improvement level on the display unit 46. At this time, the display unit 46 displays a message such as, for example, "The clutter level of the living room has improved by 400%."

[0108] After executing S60, the control unit 11 finishes the post-processing and returns. If the control unit 11 determines in S52 that the user did not clean up (S52: No), it finishes the post-processing and returns without executing S53 through S60.

[0109] In the above explanation, the display unit 16 is an example of a notification unit. The recommendation table 35 is an example of a table. The clutter level Ek is an example of an evaluation value. The thresholds TP1 and TP2 are examples of first thresholds. The thresholds TQ1 and TQ2 are examples of second thresholds. S22 to S26 is an example of the clutter level acquisition process. S32 to S34 is an example of the recommendation acquisition process.

[0110] [Effects of the Embodiment] As described above, the tidying recommendation system 1 according to this embodiment includes a camera 21, a storage unit 13, and a control unit 11. The control unit 11 performs a clutter level acquisition process (S22-S26) which acquires an overall clutter level P and a partial clutter level Q by dividing the living room image 31 and the reference image 32 into N (multiple) regions Rk and comparing each region, and a recommendation acquisition process (S32-S34) which acquires recommendations regarding tidying up the living room based on the acquired overall clutter level P and partial clutter level Q.

[0111] Therefore, according to the tidying recommendation system 1 of this embodiment, the overall clutter level P and partial clutter level Q of the room are obtained by comparing the room image 31 and the reference image 32 for each area, and recommendations for tidying up the room are obtained based on the two types of clutter levels obtained, thereby obtaining suitable recommendations according to the overall and partial clutter levels of the room.

[0112] Furthermore, the control unit 11 has thresholds TP1 and TP2 (first thresholds) for classifying the overall clutter level P, and thresholds TQ1 and TQ2 (second thresholds) for classifying the partial clutter level Q. In the recommendation acquisition process, the control unit 11 acquires the class to which the acquired overall clutter level P belongs based on thresholds TP1 and TP2 (S32), acquires the class to which the acquired partial clutter level Q belongs based on thresholds TQ1 and TQ2 (S33), and acquires a recommendation based on the two acquired classes (S34). Therefore, by acquiring a recommendation based on the results of comparing the overall clutter level P with thresholds TP1 and TP2, and the results of comparing the partial clutter level Q with thresholds TQ1 and TQ2, a suitable recommendation according to the overall and partial clutter levels of the room can be easily obtained.

[0113] Furthermore, the control unit 11 further executes the following processes based on user input: acquiring a subjective scale regarding the implementation of room tidying (S27); acquiring an objective scale regarding room tidying without user input (S28); acquiring a tolerance score regarding the implementation of room tidying (S29) based on the acquired subjective and objective scales; and acquiring thresholds TP1, TP2, TQ1, and TQ2 based on the acquired tolerance score (S30). Therefore, by acquiring a tolerance score based on the subjective and objective scales, and acquiring thresholds TP1, TP2, TQ1, and TQ2 based on the acquired tolerance score, appropriate recommendations tailored to the user's situation can be obtained. Depending on the user's situation, it may be preferable not to actively recommend room tidying. In such cases, recommendations with tolerance can be obtained according to the user's situation.

[0114] Furthermore, the control unit 11 has two (or more) thresholds TP1 and TP2 for classifying the overall clutter level P, and two (or more) thresholds TQ1 and TQ2 for classifying the partial clutter level Q. Therefore, the overall clutter level P is classified into three classes based on the two thresholds TP1 and TP2, and the partial clutter level Q is classified into three classes based on the two thresholds TQ1 and TQ2, thereby obtaining nine types of suitable recommendations according to the overall and partial clutter levels of the room.

[0115] Furthermore, the memory unit 13 stores a recommendation table 35 (table) that stores recommendations corresponding to combinations of three (or more) classes CP1 to CP3 for overall clutter level P and three (or more) classes CQ1 to CQ3 for partial clutter level Q. In the recommendation acquisition process, the control unit 11 acquires recommendations from the recommendation table 35 that correspond to combinations of the acquired overall clutter level P classes and the acquired partial clutter level Q classes. Therefore, by referring to the recommendation table 35 that stores recommendations corresponding to combinations of the three classes for overall clutter level P and the three classes for partial clutter level Q, recommendations related to tidying up can be easily obtained.

[0116] Furthermore, in S23, the control unit 11 calculates a degree of clutter Ek (evaluation value) for each region Rk of the room image 31 and the reference image 32 based on the sum of the differences between the pixel values ​​Xk(i,j) of the room image 31 and the pixel values ​​Yk(i,j) of the reference image 32. In S24, it obtains the average value of the degree of clutter Ek as the overall degree of clutter P. In S25 and S26, it obtains a value V / Vmax based on the standard deviation V of the degree of clutter Ek as the partial degree of clutter Q. The value V / Vmax is obtained by dividing the standard deviation V of the degree of clutter Ek by the maximum value Vmax that the standard deviation of the degree of clutter Ek can take when the mean value H of the degree of clutter Ek is given. Therefore, for each region Rk of the living room image 31 and the reference image 32, the degree of clutter Ek is calculated based on the sum of the differences in pixel values. Based on the average value of the calculated degree of clutter Ek, ​​the overall clutter P is obtained. Based on the standard deviation V of the calculated degree of clutter Ek and the maximum value Vmax that the standard deviation can take, the partial degree of clutter Q can be obtained.

[0117] Furthermore, the tidying recommendation system 1 further includes a communication unit 17 that transmits the recommendations acquired by the control unit 11 to the user terminal 4. Therefore, by transmitting the acquired recommendations to the user terminal 4, the acquired recommendations can be made known to the user. In addition, the tidying recommendation system 1 further includes a display unit 16 (notification unit) that notifies the user of the recommendations acquired by the control unit 11. Therefore, by displaying the acquired recommendations on the display unit 16, the acquired recommendations can be made known to the user.

[0118] [Differentiation] Various modifications can be made to the tidying recommendation system 1 according to the above embodiment. In the tidying recommendation system 1, if the shooting conditions of the room image 31 and the shooting conditions of the reference image 32 are different, the difference between the pixel values ​​of the room image 31 and the pixel values ​​of the reference image 32 becomes large. As a result, the overall clutter level P and the partial clutter level Q may become larger than expected, and it may not be possible to obtain suitable recommendations regarding tidying up the room.

[0119] The storage unit 13 of the tidying recommendation system according to the first modified example stores a plurality of reference images 32 with different shooting conditions. The storage unit 13 stores a plurality of reference images 32 with different shooting conditions, such as the shooting date, shooting time, and weather conditions at the time of shooting. Before executing S22 (before executing the clutter level acquisition process), the control unit 11 executes a process to select a reference image 32 from the plurality of reference images 32 stored in the storage unit 13 that was taken under shooting conditions similar to the shooting conditions of the living room image 31 (shooting conditions corresponding to the shooting conditions of the living room image 31) as the reference image 32 to be compared with the living room image 31.

[0120] For example, if the room image 31 was taken on September 1st, at 9:00 AM, and under sunny weather conditions, the control unit 11 selects a reference image taken under similar shooting conditions from among the multiple reference images 32 stored in the memory unit 13 as the reference image 32 to compare with the room image 31. According to the first modified example of the tidying recommendation system, by selecting the reference image 32 to be compared with the room image 31 according to the shooting conditions, the system can select an image with a small difference from the room image 31 as the reference image 32, thereby obtaining the overall clutter level P and the partial clutter level Q.

[0121] The room image 31 and the reference image 32 may include images of the ceiling, walls, etc. The pixel values ​​of pixels corresponding to the ceiling and walls do not change easily depending on the tidiness of the room. Therefore, pixels whose pixel values ​​do not change easily even if the tidiness of the room changes are selected as "specific pixels" from the room image 31 and the reference image 32. The number of specific pixels may be one or two or more. The difference between the pixel values ​​of the specific pixels in the room image 31 and the specific pixels in the reference image 32 is thought to be due to differences in the shooting conditions of the room image 31 and the shooting conditions of the reference image 32, etc.

[0122] In the second modified tidying recommendation system, the control unit 11 performs a process to convert the pixel values ​​of at least one of the room image 31 and the reference image 32 so as to reduce the difference between the pixel value of a specific pixel in the room image 31 and the pixel value of a specific pixel in the reference image 32, before executing S22. For example, if there is one specific pixel, the control unit 11 adds or subtracts a predetermined value (the difference between the pixel value of a specific pixel in the room image 31 and the pixel value of a specific pixel in the reference image 32) to the pixel values ​​of all pixels in the room image 31 so that the pixel value of a specific pixel in the room image 31 matches the pixel value of a specific pixel in the reference image 32. Alternatively, the control unit 11 may multiply the pixel values ​​of all pixels in the room image 31 by a predetermined value (the ratio of the pixel value of a specific pixel in the room image 31 to the pixel value of a specific pixel in the reference image 32). The control unit 11 may perform the same calculation on the pixel values ​​of all pixels in the reference image 32, or it may perform the same calculation on the pixel values ​​of all pixels in the room image 31 and the reference image 32.

[0123] If there are two or more specific pixels, the control unit 11 may perform a similar calculation on the pixel values ​​of all pixels in at least one of the room image 31 and the reference image 32 in order to match the average value of the pixel values ​​of the specific pixels in the room image 31 with the average value of the pixel values ​​of the specific pixels in the reference image 32. Alternatively, the control unit 11 may perform a similar calculation on the pixel values ​​of all pixels in at least one of the room image 31 and the reference image 32 in order to reduce the difference between the average value of the pixel values ​​of the specific pixels in the room image 31 and the average value of the pixel values ​​of the specific pixels in the reference image 32 to below a predetermined level.

[0124] According to the second modified example of the tidying recommendation system, by converting the pixel values ​​of at least one of the room image 31 and the reference image 32 so that the difference in pixel values ​​of specific pixels between the room image 31 and the reference image 32 is reduced, the difference between the room image 31 and the reference image 32 can be reduced, thereby obtaining the overall clutter level P and the partial clutter level Q.

[0125] In the third modified tidying recommendation system, the control unit 11 rounds the clutter level Ek between S23 and S24 (during the clutter level acquisition process). For example, in the rounding process, the control unit 11 rounds N clutter levels Ek to an integer multiple of 10%. The clutter level Ek after rounding will be one of 11 values: 0%, 10%, ..., 90%, 100%. In this case, the two thresholds TQ1 and TQ2 for the partial clutter level Q are determined to be values ​​between the rounded values ​​(for example, 5%, 15%, ..., 95%, etc.). The tidying recommendation system according to the third modified version also provides the same effects as the tidying recommendation system 1 according to the embodiment.

[0126] The degree of clutter Ek after rounding will be one of 11 values, so they are often the same value. Taking this into consideration, in the fourth modified tidying recommendation system, the control unit 11 performs a correction process on the partial clutter Q after obtaining it in S25 and S26. In the correction process, the control unit 11 obtains the minimum value Emin and the maximum value Emax of the N degrees of clutter Ek, ​​and obtains the number M of the N degrees of clutter Ek that match the minimum value Emin. Subsequently, if the number M is less than or equal to a predetermined number (for example, 2 if N=16) and the difference between the maximum value Emax and the minimum value Emin is greater than or equal to a predetermined value (for example, 20%), the control unit 11 uses the partial clutter Q obtained in S25 and S26 as is. If the number M is greater than the predetermined number, or if the difference between the maximum value Emax and the minimum value Emin is less than the predetermined value, the control unit 11 sets the partial clutter Q to 0%. This allows us to minimize the partial scattering Q by ignoring the differences in scattering Ek values ​​when the N scattering values ​​Ek are close to each other.

[0127] Thus, in the clutter level acquisition process, the control unit 11 performs rounding on the clutter level Ek and then performs correction processing on the partial clutter level Q based on the rounded clutter level Ek. Therefore, according to the fourth modified example of the tidying recommendation system, a suitable partial clutter level Q can be obtained by correcting the partial clutter level Q based on the rounded clutter level Ek.

[0128] In the fifth modified example of the tidying recommendation system, the control unit 11 performs a partial clutter level acquisition process in S25 that is different from that shown in Figure 7. For example, in the partial clutter level acquisition process, the control unit 11 sets the value Qr to the maximum value of N clutter levels Ek, or to a value obtained by multiplying the maximum value of N clutter levels Ek by a predetermined coefficient (for example, twice the maximum value of N clutter levels Ek). The control unit 11 acquires such a value Qr as a partial clutter level Q that indicates the degree of clutter in a part of the room.

[0129] In the clutter level acquisition process, the control unit 11 acquires a value based on the maximum value of the clutter level Ek as the partial clutter level Q. Therefore, according to the tidying recommendation system according to the fifth modified example, the partial clutter level Q can be easily acquired.

[0130] In the sixth modified example of the tidying recommendation system, the control unit 11 does not perform S28 to acquire an objective scale in the tidying recommendation process (Figure 7), but instead in S29 acquires a tolerance score based on the subjective scale acquired in S27. Alternatively, the control unit 11 may not perform S27 to acquire a subjective scale in the tidying recommendation process (Figure 7), but instead in S29 acquires a tolerance score based on the objective scale acquired in S28.

[0131] Thus, the control unit 11 may perform at least one of the following: a process to acquire a subjective scale regarding the implementation of tidying up the living room based on input from the user, and a process to acquire an objective scale regarding the implementation of tidying up the living room without input from the user.

[0132] In the modified tidying recommendation system, the control unit 11 may divide the living room image 31 and the reference image 32 into any number of regions, two or more. Also, in the modified tidying recommendation system, the control unit 11 may obtain the degree of clutter in region Rk according to a calculation formula other than formula (1). For example, the degree of clutter may be the value obtained by dividing the average of the squared differences between the pixel values ​​of the living room image 31 and the pixel values ​​of the reference image 32 by the maximum grayscale value of 255.

[0133] Alternatively, for example, the control unit 11 may obtain the degree of clutter Fk of region Rk according to the following equation (6).

number

[0134] Furthermore, in the modified tidying recommendation system, the control unit 11 may have three or more thresholds for classifying the overall clutter level P, and may also have three or more thresholds for classifying the partial clutter level Q. In this case, the recommendation table 35 stores 16 or more recommendations corresponding to four or more combinations of overall clutter level P classes and four or more partial clutter level Qs. Furthermore, in the modified tidying recommendation system, the number of thresholds for classifying the overall clutter level P and the number of thresholds for classifying the partial clutter level Q may be different. Furthermore, in the modified tidying recommendation system, the same recommendations may be stored in the recommendation table 35.

[0135] Furthermore, in the modified tidying recommendation system, the control unit 11 may acquire subjective scales other than busyness, fatigue, the state of cohabitants, and whether or not guests are expected. The control unit 11 may also acquire objective scales other than sleep duration, energy level, stress level, and planned visits. The control unit 11 may also acquire subjective scales related to stress level based on CSI and CVI values ​​stored in the stress level database 34 stored in the memory unit 13. The control unit 11 may also acquire tolerance scores according to calculation formulas other than formula (4). The control unit 11 may also acquire subjective scales according to rules other than those shown in Figure 11(A), acquire objective scales according to rules other than those shown in Figure 11(B), and acquire thresholds TP1, TP2, TQ1, and TQ2 according to rules other than those shown in Figure 11(C). The control unit 11 may acquire the tolerable threshold TQ1 according to different rules than those for the tolerable threshold TP1, and the absolute threshold TQ2 according to different rules than those for the absolute threshold TP2.

[0136] Furthermore, in the modified tidying recommendation system, the user terminal 4 may be a smart speaker. In this case, input from the user and output to the user are performed by voice. In the modified tidying recommendation system, the camera 21, heart rate sensor 22, and sleep sensor 23 may be connected to the communication unit 17.

[0137] [Note 1] A camera that takes pictures of the room and outputs the captured images as room images, A storage unit that stores reference images of the above-mentioned room taken in advance, It comprises a control unit and, The above control unit, A clutter level acquisition process that divides the above room image and the above reference image into multiple regions and compares each region to obtain an overall clutter level indicating the overall degree of clutter in the room and a partial clutter level indicating the degree of clutter in a part of the room. A tidying recommendation system that performs a recommendation acquisition process to acquire recommendations for tidying up the room based on the acquired overall level of clutter and the acquired partial level of clutter.

[0138] [Note 2] The control unit has a first threshold for classifying the overall degree of clutter and a second threshold for classifying the partial degree of clutter. The above control unit, in the recommendation acquisition process, acquires the class to which the acquired overall clutter belongs based on the first threshold, acquires the class to which the acquired partial clutter belongs based on the second threshold, and acquires the recommendation based on the two acquired classes, as described in Appendix 1 of the tidying recommendation system.

[0139] [Note 3] The above control unit, A process to obtain a subjective scale regarding the implementation of tidying up the above-mentioned room based on input from the user, and at least one of the processes to obtain an objective scale regarding the tidying up of the above-mentioned room without relying on input from the user, A process to obtain a tolerance score regarding the implementation of the cleaning of the living room, based on at least one of the subjective scale and the objective scale obtained above, The tidying recommendation system described in Appendix 2 further performs the process of obtaining the first threshold and the second threshold based on the tolerance score obtained above.

[0140] [Note 4] The control unit is a tidying-up recommendation system according to Appendix 2 or 3, having a plurality of first thresholds and a plurality of second thresholds.

[0141] [Note 5] The above-mentioned storage unit stores a table containing the above-mentioned recommendations, corresponding to the combination of the multiple classes of overall clutter and the multiple classes of partial clutter. The above control unit, in the recommendation acquisition process, acquires the above recommendation from the table corresponding to the combination of the acquired overall clutter level class and the acquired partial clutter level class, according to any of the appendices 2 to 4 of the tidying recommendation system.

[0142] [Note 6] The above control unit, in the clutter level acquisition process, calculates an evaluation value for each region of the room image and the reference image based on the sum of the differences between the pixel values ​​of the room image and the pixel values ​​of the reference image, and acquires a value based on the average of the above evaluation values ​​as the overall clutter level, according to any one of the appendices 1 to 5.

[0143] [Note 7] The above control unit is a tidying recommendation system according to Appendix 6, which in the above clutter level acquisition process acquires a value based on the standard deviation of the above evaluation value as the above partial clutter level.

[0144] [Note 8] The tidying recommendation system described in Appendix 7, wherein the control unit obtains the value obtained by dividing the standard deviation of the evaluation value by the maximum value that the standard deviation of the evaluation value can take when the average value of the evaluation value is given, as the partial tidying level in the tidying level acquisition process.

[0145] [Note 9] The above control unit is a tidying recommendation system according to Appendix 6, which in the above clutter level acquisition process acquires a value based on the maximum or minimum value of the above evaluation value as the above partial clutter level.

[0146] [Note 10] The tidying recommendation system according to any one of the appendices 6 to 9, wherein the control unit performs rounding on the evaluation value in the clutter level acquisition process, and performs correction processing on the partial clutter level based on the evaluation value after rounding.

[0147] [Note 11] The above-mentioned memory unit stores multiple reference images with different shooting conditions, The tidying recommendation system according to any one of the appendices 1 to 10, wherein the control unit, before executing the clutter level acquisition process, selects from a plurality of reference images stored in the storage unit a reference image taken under shooting conditions corresponding to the shooting conditions of the living room image, as the reference image to be compared with the living room image.

[0148] [Note 12] The tidying recommendation system according to any one of the appendices 1 to 11, wherein the control unit performs a process to convert the pixel values ​​of at least one of the room image and the reference image so as to reduce the difference between the pixel values ​​of a specific pixel in the room image and the pixel values ​​of the specific pixel in the reference image, before performing the clutter level acquisition process.

[0149] [Note 13] A tidying-up recommendation system according to any one of the appendices 1 to 12, further comprising a communication unit that transmits the recommendations acquired by the control unit to a user terminal.

[0150] [Note 14] A tidying-up recommendation system according to any one of the appendices 1 to 13, further comprising a notification unit that notifies the user of the recommendation acquired by the control unit. [Explanation of symbols]

[0151] 1. Decluttering Recommendation System 4. User terminal 5. Wearable devices 10. Home Server 11. Control Unit 13...Storage section 16... Display section (notification section) 17. Communications Department 21...Camera 31. Room images 32...Reference image 35. Recommendation Table (Table)

Claims

1. A camera that takes pictures of the room and outputs the captured images as room images, A storage unit that stores reference images of the above-mentioned room taken in advance, It comprises a control unit and, The above control unit, A clutter level acquisition process that divides the above room image and the above reference image into multiple regions and compares each region to obtain an overall clutter level indicating the overall degree of clutter in the room and a partial clutter level indicating the degree of clutter in a part of the room. Based on the acquired overall and partial clutter levels, a recommendation acquisition process is performed to obtain recommendations regarding tidying up the room. The control unit has a first threshold for classifying the overall degree of clutter and a second threshold for classifying the partial degree of clutter. The control unit, in the recommendation acquisition process, acquires the class to which the acquired overall clutter belongs based on the first threshold, acquires the class to which the acquired partial clutter belongs based on the second threshold, and acquires the recommendation based on the two acquired classes. The above control unit, A process to obtain a subjective scale regarding the implementation of tidying up the above-mentioned room based on input from the user, and at least one of the processes to obtain an objective scale regarding the tidying up of the above-mentioned room without relying on input from the user, A process to obtain a tolerance score regarding the implementation of the cleaning of the living room, based on at least one of the subjective scale and the objective scale obtained above, A tidying-up recommendation system that further performs the process of obtaining the first threshold and the second threshold based on the tolerance score obtained above.

2. A camera that takes pictures of the room and outputs the captured images as room images, A storage unit that stores reference images of the above-mentioned room taken in advance, It comprises a control unit and, The above control unit, A clutter level acquisition process that divides the above room image and the above reference image into multiple regions and compares each region to obtain an overall clutter level indicating the overall degree of clutter in the room and a partial clutter level indicating the degree of clutter in a part of the room. Based on the acquired overall and partial clutter levels, a recommendation acquisition process is performed to obtain recommendations regarding tidying up the room. The above-mentioned control unit, in the clutter level acquisition process, calculates an evaluation value for each region of the room image and the reference image based on the sum of the differences between the pixel values ​​of the room image and the reference image, acquires a value based on the average of the above-mentioned evaluation values ​​as the overall clutter level, and acquires a value based on the standard deviation of the above-mentioned evaluation values ​​as the partial clutter level, thereby providing a tidying recommendation system.

3. The tidying recommendation system according to claim 2, wherein the control unit, in the tidying-to-tidy

4. A camera that takes pictures of the room and outputs the captured images as room images, A storage unit that stores reference images of the above-mentioned room taken in advance, It comprises a control unit and, The above control unit, A clutter level acquisition process that divides the above room image and the above reference image into multiple regions and compares each region to obtain an overall clutter level indicating the overall degree of clutter in the room and a partial clutter level indicating the degree of clutter in a part of the room. Based on the acquired overall and partial clutter levels, a recommendation acquisition process is performed to obtain recommendations regarding tidying up the room. The control unit, in the clutter level acquisition process, calculates an evaluation value for each region of the room image and the reference image based on the sum of the differences between the pixel values ​​of the room image and the reference image, acquires a value based on the average of the evaluation values ​​as the overall clutter level, and acquires a value based on the maximum or minimum of the evaluation values ​​as the partial clutter level.

5. A camera that takes pictures of the room and outputs the captured images as room images, A storage unit that stores reference images of the above-mentioned room taken in advance, It comprises a control unit and, The above control unit, A clutter level acquisition process that divides the above room image and the above reference image into multiple regions and compares each region to obtain an overall clutter level indicating the overall degree of clutter in the room and a partial clutter level indicating the degree of clutter in a part of the room. Based on the acquired overall and partial clutter levels, a recommendation acquisition process is performed to obtain recommendations regarding tidying up the room. The above-mentioned memory unit stores multiple reference images with different shooting conditions, The above-mentioned control unit is a tidying-up recommendation system that, before executing the clutter level acquisition process, selects from a plurality of reference images stored in the storage unit, a reference image taken under shooting conditions corresponding to the shooting conditions of the living room image, as the reference image to be compared with the living room image.

6. A camera that takes pictures of the room and outputs the captured images as room images, A storage unit that stores reference images of the above-mentioned room taken in advance, It comprises a control unit and, The above control unit, A clutter level acquisition process that divides the above room image and the above reference image into multiple regions and compares each region to obtain an overall clutter level indicating the overall degree of clutter in the room and a partial clutter level indicating the degree of clutter in a part of the room. Based on the acquired overall and partial clutter levels, a recommendation acquisition process is performed to obtain recommendations regarding tidying up the room. The above-mentioned control unit is a tidying-up recommendation system that, before executing the clutter level acquisition process, performs a process to convert the pixel values ​​of at least one of the room image and the reference image so that the difference between the pixel values ​​of specific pixels in the room image and the pixel values ​​of specific pixels in the reference image decreases.

Images