System, refrigerator
The refrigerator system addresses the lack of voice-based recipe search and inefficient imaging by using strategically placed voice recognition and cameras, providing efficient and energy-saving food monitoring and recipe access.
Patent Information
- Application Number
- JP2023186326
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-10-31
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2033-03-29
AI Technical Summary
Existing refrigerators lack the ability to search for recipes using voice recognition, and existing systems that use cameras for food storage monitoring are inefficient and energy-intensive.
A refrigerator system equipped with voice recognition means on the front and inside, integrated with a display terminal, that detects user presence to activate voice recognition, allowing recipe searches without constant operation, and uses cameras and interior lighting strategically for clear imaging.
Enables voice-based recipe search and efficient, energy-saving food monitoring with clear imaging, reducing erroneous recognition and energy consumption.
Smart Images

Figure 0007588307000001 
Figure 0007588307000002 
Figure 0007588307000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a system and a refrigerator that allows users to search for recipes related to food stored in a storage unit using only voice. [Background technology]
[0002] In general, many refrigerators and refrigeration systems of this type have been proposed. For example, the refrigerator and refrigeration system described in Patent Document 1 is configured such that a small camera is installed inside the storage compartment, and the food storage status inside the storage compartment is photographed by the small camera and can be confirmed on a display terminal such as a mobile phone via an Internet line.
[0003] Patent Document 2 also discloses a refrigerator and refrigeration system similar to those in Patent Document 1, and the system described in Patent Document 1 is configured to use a small camera to photograph the state of food inside the storage compartment when the refrigerator door is closed.
[0004] Furthermore, the refrigerator and refrigeration system described in Patent Document 3, in addition to the configuration described in Patent Document 1, further comprises a display terminal on the surface of the refrigerator, and this display terminal comprises a fixed terminal unit built into the surface of the refrigerator and a display terminal such as a mobile phone that can be removed from the surface of the refrigerator and carried around, and both of the above terminals are capable of displaying on their display the food storage status inside the storage unit photographed with a small camera via an Internet line, so that the food storage status inside the storage unit can be known without opening the refrigerator door, even when at home as well as when away from home. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2002-236798 A [Patent Document 2] JP 2007-46833 A [Patent Document 3] JP 2002-81818 A Summary of the Invention [Problem to be solved by the invention]
[0006] An object of the present invention is to provide a system and a refrigerator capable of searching for recipes. [Means for solving the problem]
[0007] The system of the present invention is a system including a refrigerator and a display terminal, the refrigerator including first voice recognition means for recognizing voice, which are provided on a front surface of the refrigerator, human detection means for detecting the presence of a person in front of the refrigerator, and control means, and when the human detection means detects the presence of a person in front of the refrigerator, the control means activates the first voice recognition means, and when voice is recognized by the first voice recognition means, the control means judges voice data recognized by the first voice recognition means, and after the voice data is judged by the control means, the voice data is displayed on the display terminal, and a confirmation means is provided for determining whether or not to perform a recipe search using the voice data, and when it is judged by the confirmation means to perform a recipe search using the voice data, a result of the recipe search is displayed on the display terminal. The refrigerator further includes a refrigerator compartment and a second voice recognition means for recognizing voice, the second voice recognition means being provided on a side surface inside the refrigerator compartment, and when the presence of a person in front of the refrigerator is detected by the human detection means, the control means activates the first voice recognition means and the second voice recognition means, and determines which voice data to adopt from the first voice data recognized by the first voice recognition means and the second voice data recognized by the second voice recognition means based on a voice input level. It is characterized by:
[0008] Further, a refrigerator of the present invention is a refrigerator comprising first voice recognition means for recognizing voice, which are provided on a front surface of the refrigerator, human detection means for detecting the presence of a person in front of the refrigerator, control means, and display means, wherein when the human detection means detects the presence of a person in front of the refrigerator, the control means activates the first voice recognition means, and when a voice is recognized by the first voice recognition means, the control means judges voice data recognized by the first voice recognition means, and after the voice data is judged by the control means, the voice data is displayed on the display means, and a confirmation means is provided for determining whether or not to perform a recipe search using the voice data, and when it is judged by the confirmation means to perform a recipe search using the voice data, a result of the recipe search is displayed on the display means. The refrigerator further includes a refrigerator compartment and a second voice recognition means for recognizing voice, the second voice recognition means being provided on a side surface inside the refrigerator compartment, and when the presence of a person in front of the refrigerator is detected by the human detection means, the control means activates the first voice recognition means and the second voice recognition means, and determines which voice data to adopt from the first voice data recognized by the first voice recognition means and the second voice data recognized by the second voice recognition means based on a voice input level.It is characterized by: Effect of the Invention
[0009] According to the present invention, it is possible to provide a system and a refrigerator capable of searching for recipes. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a perspective view of a refrigerator according to an embodiment of the present invention; [Diagram 2] Schematic cross-sectional view of the refrigerator [Diagram 3] FIG. 2 is a perspective view showing the inside of the refrigerator when the storage door is opened. [Figure 4] (a) is a perspective view of the refrigerator with a display terminal integrally mounted thereon, and (b) is a perspective view of the refrigerator with a detachable display terminal mounted thereon. [Diagram 5] 1A and 1B show examples of a mounting configuration of a display terminal used in the refrigerator, in which (a) is a cross-sectional view in the case where the display terminal is integrally mounted, and (b) is a cross-sectional view in the case where the display terminal is detachably mounted. [Figure 6] 1 shows an example of the display viewing configuration of a display terminal used in the refrigerator, where (a) and (b) are schematic cross-sectional views when the outer panel of the door is made of glass, and (c) is a schematic cross-sectional view when the outer panel of the door is made of a one-way mirror. [Figure 7] A vertical cross-sectional view showing an example of camera installation in the same refrigerator. [Figure 8] An example of camera installation in the refrigerator is shown. (a) is a cross-sectional view of the refrigerator door when the door is closed, and (b) is a front view of the inside of the door when the door is open. [Figure 9] (a)(b)(c) are longitudinal cross-sectional views showing other examples of camera installation in the same refrigerator. [Figure 10] FIG. 1 is an explanatory diagram showing an example of a camera attached to the refrigerator. [Figure 11] FIG. 1 is a system configuration diagram of a refrigerator system according to the embodiment. [Figure 12] A block diagram showing the refrigerator configuration of the refrigerator system. [Figure 13] Block diagram showing the server configuration of the refrigerator system [Figure 14] A flowchart showing the photographing and data storage operations of the refrigerator system when the door is closed. [Figure 15] A flowchart showing the photographing and data storage operations during cooling operation of the refrigerator compartment of the refrigerator system. [Figure 16] A flowchart showing the photographing and data storage operations of the refrigerator system when the door is opened. [Figure 17] A flowchart showing another example of the photographing operation in FIGS. 14 to 16. [Figure 18] A flowchart showing a case where data storage and synthesis operations in the refrigerator system are performed by a server device. [Figure 19] A flowchart showing a case where data storage and synthesis operations are performed on the refrigerator side in the refrigerator system. [Figure 20] A flowchart showing the case where data storage and synthesis operations in the refrigerator system are performed on the display terminal side. [Figure 21] A flowchart showing normal image data transmission and reception in the refrigerator system. [Figure 22] A flowchart showing transmission and reception of image data when the image data is stored in a display terminal in the refrigerator system. [Figure 23] A flowchart showing a case where an individual image and a composite image are switched and transmitted in the refrigerator system. [Figure 24] A flowchart showing image data transmission and reception when a pre-transmission request is made in the refrigerator system. [Diagram 25] FIG. 11 is a front view of a display terminal for explaining switching of a transmission / reception path in the refrigerator system. [Figure 26] FIG. 2 is a diagram for explaining image synthesis processing in the refrigerator system; [Figure 27] FIG. 1 is a perspective view of a refrigerator according to an embodiment of the present invention; [Figure 28] FIG. 2 is a perspective view showing the inside of the refrigerator when the storage door is opened. [Figure 29] A block diagram showing the refrigerator configuration related to the voice recognition of the refrigerator system. [Diagram 30] A flow chart showing the voice recognition and data storage operations of the refrigerator system when the door is opened. [Diagram 31] A flowchart showing the voice recognition and data storage operations of the refrigerator system when a human presence sensor is present. [Diagram 32] A flowchart showing the operation of stopping voice recognition when the door is closed in the voice recognition system of the refrigerator. [Diagram 33] A flowchart showing the operation of stopping voice recognition when there is no human presence sensor for voice recognition in the refrigerator system. [Diagram 34] A flowchart showing a recipe search process in the voice recognition of the refrigerator system performed by the server device. [Diagram 35] A flowchart showing a case where a recipe search process in the voice recognition of the refrigerator system is performed on the display terminal side. [Diagram 36] A flowchart showing a case where a recipe search result in the voice recognition of the refrigerator system is displayed on a display terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Specific and detailed embodiments of the present invention will be described below, but first, the essential features of the embodiments of the present invention will be described.
[0012] An embodiment of a refrigerator in the present invention includes a storage compartment partitioned by an insulated wall and an insulated door for storing items, an insulated door covering the storage compartment, voice recognition means for recognizing voices from outside, communication means for transmitting voice information by the voice recognition means to an external device, display means for displaying information received from the external device, search means for performing a search via the communication means based on the content displayed on the display means or the voice information by the voice recognition means, and a human presence sensor for detecting the presence or absence of a person in front of the storage compartment, the display means having a function of operating the voice recognition means by issuing a voice recognition start instruction, and operating the voice recognition means when the human presence is detected by the human presence sensor, and the voice recognition means are arranged in front of the insulated door and on a side surface inside the storage compartment, and are configured to switch the voice recognition data to be adopted depending on the voice input level of each voice recognition means.
[0013] This allows the voice recognition means to be activated when the human presence sensor detects the presence of a person, making it possible to input the name of a food by voice, and easily searching for recipes from a database on the cloud using only the input voice. Also, by linking it to the human presence sensor, the voice recognition means does not need to be constantly operated, which prevents erroneous recognition and saves energy.
[0014] Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the following embodiment.
[0015] [Embodiment 1] <1. Configuration> 1-1. Refrigerator configuration 1-1-1. Configuration of French door type refrigerator First, a configuration example of a French door type refrigerator will be described with reference to Figs. 1 to 9.
[0016] In Figs. 1 to 3, the refrigerator body 1 is composed of an outer box 2 made of metal (for example, iron plate) that opens to the front, an inner box 3 made of hard resin (for example, ABS), and a heat insulating material 4 such as hard foamed urethane that is foamed and filled between the outer box 2 and the inner box 3. The refrigerator body 1 has a plurality of storage compartments formed therein by partitions. These storage compartments are, from the top of the refrigerator body 1, a refrigerator compartment 5, a switchable compartment 6 located below the refrigerator compartment 5, an ice-making compartment 7 arranged in parallel with the switchable compartment 6, a freezer compartment 8 located below the switchable compartment 6 and the ice-making compartment 7, and a vegetable compartment 9 located below the freezer compartment 8. The front of the refrigerator compartment 5 is freely closed by double-door doors 10-1 and 10-2 that can be opened and closed, and the front parts of the switchable compartment 6, the ice-making compartment 7, the freezer compartment 8, and the vegetable compartment 9 are freely closed by drawer doors 12a, 12b, 13, and 14 (hereinafter referred to as drawer doors).
[0017] A cooling chamber 16 is provided at the rear of the refrigerator body 1, and is provided with a cooler 17 for generating cold air, a blower fan 18 for supplying cold air to each chamber, and a damper (air volume adjustment means) 22 for adjusting the volume of air to the refrigerator chamber. A compressor 19 is provided at the back of the top surface of the refrigerator body 1, and a refrigerant is sealed in a refrigeration cycle that is formed by connecting a condenser (not shown), a heat dissipation pipe 20 for dissipating heat, a capillary tube 21, and the cooler 17 in sequence in a ring shape, to perform a cooling operation.
[0018] Here, the doors 10-1, 10-2, 12a, 12b, 13, and 14 are filled with hard urethane foam to have thermal insulation properties, similar to the refrigerator body 1, and among the doors 10-1 and 10-2 of the storage chamber 5 that serves as the refrigerator compartment, one door 10-1 is narrower in width than the other door 10-2, and a center pillar 11 is attached to its open end to prevent cold air from leaking. In this refrigerator, a display terminal 15 is provided on the front of the other door 10-2, which is wider. As shown in FIG. 4(a), this display terminal 15 is composed of a display unit such as a liquid crystal panel and a control unit that controls it, and is attached integrally to the door 10, or as shown in FIG. 4(b), it is formed of a smartphone or tablet PC and is detachable from the door 10. Either method may be used to provide the display terminal 15.
[0019] Figures 5(a) and (b) show examples of the mounting configuration of the display terminal 15 to the door 10, with Figure 5(a) showing the case where the display terminal 15 is integrally attached and Figure 5(b) showing the case where the display terminal 15 is attached so that it can be freely attached and detached.
[0020] 5(a) and (b), the door 10 is mainly composed of an exterior panel 10a facing the front side of the refrigerator, an inner panel 10b facing the inside of the refrigerator, and a heat insulating material 4 filled between the exterior panel 10a and the inner panel 10b. The exterior panel 10a is composed of a metal member such as a color steel plate or a transparent member such as a glass plate, and the inner panel 10b is a synthetic resin member formed by vacuum molding. A raised wall portion 10b-1 that is square-shaped when viewed from the inside of the refrigerator and protrudes toward the inside of the refrigerator is provided around the periphery of the inner panel 10b. As already mentioned, the heat insulating material 4 is made of, for example, hard foamed urethane.
[0021] In the door 10 of FIG. 5(a) in which the display terminal 15 is integrated, a resin-made storage frame body 23 is provided on the front side of the door 10, which is continuous with the outer cover panel 10a and covers the heat insulating material 4 (if the outer cover panel 10a is a glass plate, it is provided by being bonded integrally to the glass plate). The display terminal 15 is fixed to the storage frame body 23 via an end holder 24. A touch panel 25 is provided on the front side. A gap 26 is provided between the display terminal 15 and the touch panel 25 to prevent the touch panel 25 from malfunctioning due to noise from the display terminal 15. The display terminal 15 and the touch panel 25 are fixed by attaching a frame 27 covering their outer periphery to the surface of the door 10. By providing the storage frame 23 in this way, the heat insulating material 4 does not come into direct contact with the display terminal 15, so that it is possible to prevent problems such as the display terminal 15 being exposed to high temperatures or being deformed when the heat insulating material 4 is foamed. On the other hand, in the door 10 of FIG. 5(b) in which the display terminal 15 is detachable, a recess 28 is formed by providing the storage frame 23 on the front part of the door 10, and the display terminal 15 is configured to be detachable in the recess 28.
[0022] Figures 6(a), (b), and (c) show an example of a display see-through configuration of the door 10 and the display terminal 15. Figures 6(a) and (b) show the case where the outer cover panel 10a of the door 10 is a glass plate 10a-1, while Figure 3(c) shows the case where the outer cover panel 10a of the door 10 is configured as a one-way mirror 10a-2.
[0023] In FIG. 6(a), a smoke sheet 29 is provided between the display terminal 15 embedded in the door 10 and the glass plate 10a-1, with only the portion facing the display terminal 15 being transparent. Even when the display terminal 15 is off and looks black, the portion of the glass plate 10a-1 other than the display terminal 15 also looks black due to the smoke sheet 29, so that the boundary with the display terminal 15 is not noticeable, and the appearance can be maintained good. In FIG. 6(b), a transparent liquid crystal display is used as the display unit of the display terminal 15, and a smoke sheet 29a that is entirely black is laid along the back surface of the transparent liquid crystal display, and this form also makes it possible to maintain a good appearance. Furthermore, in FIG. 6(c), the glass plate 10a-1 of FIG. 6(a) and (b) is replaced by a magic mirror 10a-2, making the smoke sheet 29 unnecessary, and in this case too, the appearance can be maintained good. In both cases, an example is shown in which the display terminal 15 is attached integrally to the door 10. In place of the smoked sheet 29, the rear surface of the glass plate 10a-1 may be painted to give a dark appearance.
[0024] Figures 7, 8(a) and (b) show examples of installation of the camera and interior lighting which are the main parts of the present invention. Figure 4 is a vertical cross-sectional view of the refrigerator with the door closed, Figure 8(a) is a horizontal cross-sectional view showing the inner door part of the refrigerator with the door closed, and Figure 8(b) is a front view of the inner door part when the door is open.
[0025] 7, 8(a) and 8(b), shelf boards 30-1 to 30-4 are provided in a plurality of vertical stages in storage compartment 5 of refrigerator. These shelf boards 30-1 to 30-4 are made of a transparent material such as a transparent synthetic resin material or glass, except for the surrounding frames.
[0026] Reference numeral 31 denotes a storage container provided below the lowest shelf 30-4 so as to be freely removable. The lowest shelf 30-4 is fixed since it constitutes the upper surface of storage container 31, but shelves 30-1 to 30-3 are placed on side wall protrusions 32-1 to 32-4 provided on both side walls of storehouse 5, respectively, and are therefore freely removable. In particular, among shelves 30-1 to 30-3 excluding the lowest shelf 30-4, central shelf 30-2 is provided with a plurality of corresponding side wall protrusions 32-2 at different heights, allowing the height of the shelf to be adjusted.
[0027] Numerals 33-1 to 33-3 denote door pockets, which are provided in a plurality of upper and lower stages, on the inner surface of the door 10 that opens and closes the opening of the storage facility 5. These door pockets 33-1 to 33-3 are also made of a material having transparency.
[0028] Reference numerals 34-1 and 34-2 denote lighting devices (hereinafter referred to as interior lighting devices) that serve as interior lighting devices, and are disposed between both side walls of the front opening of the storage room 5 and the front opening of the storage room 5, which are located forward of the front ends of the shelves 30-1 to 30-4. The interior lighting 34-1, 34-2 are provided on a board on which a plurality of LEDs are arranged in the vertical direction, for example, for the purpose of saving energy, and are covered with a light-transmitting cover.
[0029] Reference numerals 35-1 and 35-2 denote photographing devices such as cameras (hereinafter referred to as cameras) that are arranged to be positioned on the front side of the storage facility 5, similar to the interior lighting fixtures 34-1 and 34-2. For example, the cameras are composed of CCD cameras or CMOS cameras, and although there is no particular limitation on the type, it is preferable that the cameras are resistant to condensation, etc.
[0030] The cameras 35-1, 35-2 are provided on the inner surface of the wider door 10-2 that does not have a center pillar 11, of the doors 10-1, 10-2 for the storage cabinet 5, near the open end of the door 10-2 and set so as to be located approximately in the center of the refrigerator 51.
[0031] In this embodiment, one camera 35-1 is attached to the underside of the upper door pocket 33-1 at approximately the center in the front-to-rear direction, and the other camera 35-2 is attached near the top of the bottom door pocket 33-3. Here, the front-to-rear direction of the door pocket 33 is the direction perpendicular to the plane formed by the inner panel 10b of the door 10, and is the same as the front-to-rear direction of the storage cabinet 5 when the door 10 is closed.
[0032] The upper camera 35-1 is provided so as to be located approximately in front of the top shelf board 30-1, and the lower camera 35-2 is provided so as to be located in front of the third shelf board 30-3 in the figure. In other words, both cameras are provided so as to be located approximately in front of the shelf boards 30-1 and 30-3 which are removable but not designed for height adjustment, rather than the shelf board 30-2 which is placed on the multiple side wall protrusions 32-2 which are designed for height adjustment.
[0033] As described above, both of the cameras 35-1, 35-2 are provided on the side of door 10-2 where center pillar 11 (see FIG. 3) is not provided, and are set to be located approximately in the center of refrigerator 51, and are located forward of the front ends of shelves 30-1 to 30-4 as is clear from FIG. 7, and are located near end Y of the open end opposite pivot portion X of door 10, that is, approximately in the center in the left-right direction of refrigerator 51, as shown in FIG. 8(b). Thus, these two cameras 35-1, 35-2 can capture images of almost the entire interior of the refrigerator.
[0034] The cameras 35-1 and 35-2 may be configured arbitrarily according to the type of refrigerator to be installed. For example, a camera 35-3 dedicated to the inside of a storage container may be additionally installed as shown in FIG. 9(a), or the lower camera 35-2 may be configured so as to be movable up and down as shown in FIG. 9(b), or the lower camera 35-2 may be configured so as to be rotatable in the vertical direction as shown in FIG. 9(c).
[0035] As is apparent from FIG. 7, the interior lighting lamps 34-1, 34-2 are provided at a position on the inner side of the storage facility relative to the cameras 35-1, 35-2 so that the light from the interior lighting lamps 34-1, 34-2 does not directly enter the photographing portions of the cameras 35-1, 35-2.
[0036] More specifically, as shown in FIG. 8(a), the photographing sections of the cameras 35-1 and 35-2 are provided outside the range where the amount of irradiation is 1 / 2 or more of the amount of irradiation (maximum amount of irradiation) in the irradiation direction of the interior lighting 34-1 and 34-2, that is, outside the range of an irradiation angle of 60 degrees in this example.
[0037] Although not shown, a light-shielding section may be provided on the camera side of the interior lighting 34-1, 34-2, or the light sources such as LEDs constituting the interior lighting 34-1, 34-2 may be arranged to be inclined diagonally backward so that the light from the interior lighting 34-1, 34-2 does not directly enter the shooting sections of the cameras 35-1, 35-2. This may be selected appropriately depending on the camera installation conditions, etc.
[0038] 1-1-3.Camera configuration Figure 10 shows an example of a camera used to photograph the interior of a warehouse. As explained in the above examples, it is most preferable to install the cameras at separate positions above and below when photographing the interior of a warehouse, but verification has confirmed that the entire interior can be photographed even if the upper and lower cameras 35-1 and 35-2 are installed close to each other as shown in Figure 10.
[0039] In such a case, it is possible to integrate the cameras 35-1 and 35-2 into a camera module 41. Modularization in this way can facilitate installation and reduce costs. Modularization is also possible by using a wider-angle lens such as a fisheye lens.
[0040] 1-1-4. Microphone configuration Figures 27 and 28 show examples of microphones used for voice recognition. Since voice recognition may be performed with the refrigerator door open or without the door being opened, the microphones are attached to the front surface 100-1 of the refrigerator and to the side surface 100-2 inside the refrigerator. This makes it possible to increase the voice input level during voice recognition.
[0041] 1-1-5. Human sensor configuration Figure 27 shows an example of a human presence sensor used for voice recognition. Since voice recognition may be performed without opening the refrigerator door, a human presence sensor 101 is attached to the front of the refrigerator. This eliminates the need to start / stop the voice recognition device.
[0042] 1-2. Overall configuration of the refrigerator system Next, an entire refrigerator system using the refrigerator described above will be described with reference to Figs. 11 to 26.
[0043] Fig. 11 shows the overall configuration of a refrigerator system 50 using the above-mentioned refrigerator. As shown in Fig. 11, this refrigerator system 50 is a system for checking the food status in a refrigerator 51 using a display terminal 15 or the like.
[0044] This refrigerator system 50 includes a refrigerator 51 and a display terminal 15, as well as a wireless adapter (communication device) 53, a gateway device (relay device) 54, a router device 55, the Internet 56, and a server device 57.
[0045] The refrigerator 51 is placed in a residence A of a user, for example, in a kitchen room in the residence A.
[0046] Wireless adapter (communication device) 53 is electrically connected to a control unit (to be described later) of refrigerator 51, and is configured to communicate with router device 55 via gateway device (relay device) 54. Wireless adapter 53 receives a signal transmitted from router device 55, and outputs the received signal to the control unit of refrigerator 51. Based on this signal, refrigerator 51 is configured to perform a corresponding operation.
[0047] Wireless adapter 53 is also configured to obtain identification information of refrigerator 51 (for example, serial number, model number, etc.) and interior image data (described later) from a storage unit in refrigerator 51, and transmit the information to router device 55 via gateway device (relay device) 54. Wireless adapter 53 may be provided integrally with refrigerator 51, or may be configured to be detachable.
[0048] The wireless adapter 53 includes a "connect" button 53a. This button is for newly connecting to relay device (relay device) 54. When the user operates “connect” button 53a, wireless adapter 53 is configured to obtain identification information of refrigerator 51 (such as a serial number and a model number) from the memory unit of refrigerator 51.
[0049] As described above, the display terminal 15 is a general-purpose portable terminal such as a smartphone or a tablet PC (personal computer), and in this embodiment, it can be detachably mounted on the door 10 of the refrigerator 51 described above, and has a function (means) of connecting to the Internet 56 and a function (means) of communicating with the gateway device 54 described later.
[0050] The display terminal 15 (e.g., a smartphone) is configured to connect to the Internet 56 via a telephone network (e.g., a 3G network). In addition to or as an alternative thereto, for example, the display terminal 15 is configured to connect to the Internet 56 via a public wireless LAN using a Wi-Fi communication function.
[0051] By connecting to the Internet 56, for example, the display terminal 15 can obtain a program for obtaining image data of food in the refrigerator that is the subject of the present invention and data for performing energy-saving operation (such as data on the number of times the door is opened and closed) from the homepage of the manufacturer of the refrigerator 51 via the Internet 56. By installing the obtained program in the display terminal 15 and starting the installed program, the display terminal 15 can obtain image data of food in the refrigerator, etc. In other words, the display terminal 15 can generate and transmit an operation signal for obtaining image data of the refrigerator 51 by operating the operation screen of the display 15a. Details will be described later.
[0052] The display terminal 15 is also configured to connect to a router device 55 by general-purpose communication not via the Internet 56, such as Wi-Fi communication, Bluetooth (registered trademark) communication, or infrared communication, and to communicatively connect to a gateway device 54 via the router device 55. A device for this purpose (such as a Wi-Fi antenna) is built into the display terminal 15.
[0053] Gateway device (relay device) 54 is a device that relays communication between wireless adapter 53 and display terminal 15, and is configured to communicate with wireless adapter 53 using, for example, a signal in a specific low-power wireless special small frequency band (924.0 to 928.0 MHz). The frequency band for communication between gateway device 54 and wireless adapter 53 is preferably a low frequency band that can reach long distances, and is installed together with router device 55 in user's residence A where refrigerator 51 is installed.
[0054] The gateway device 54 is also configured to be communicatively connected to the display terminal 15 via the router device 55. That is, the gateway device 54 is configured so that communication (first communication) can be performed between the display terminal 15 and the router device 55 only, not via the Internet 56.
[0055] The gateway device 54 is further configured to be connectable to the Internet 56 via the router device 55. As a result, the gateway device 54 can communicate with the display terminal 15 (second communication) and also with the server device 57 via the Internet 56 and the router device 55.
[0056] Therefore, the gateway device 54 provides the display terminal 15 with a communication that does not go through the Internet 56 (i.e., a first communication that goes only through the router device 55) and a communication that goes through the Internet 56 (i.e., a second communication that goes through both the Internet 56 and the router device 55). The reason for this configuration will be described later.
[0057] Server device 57 is prepared, for example, by the manufacturer of refrigerator 51. Server device 57 manages information necessary for access and authentication regarding display terminal 15, refrigerator 51, and wireless adapter 53. For example, server device 57 authenticates (determines) whether or not the identification information of display terminal 15 included in a communication request signal matches the identification information of display terminal 15 stored (registered) in association with wireless adapter 53. If they match, server device 57 authenticates the accessing display terminal 15 as a legitimate display terminal and permits communication with refrigerator 51 in residence A.
[0058] Server device 57 also stores the image data of the foods in the refrigerator transmitted from refrigerator 51, data for energy-saving operation, and the like. These data are transmitted to display terminal 15 based on a data call signal from display terminal 15. That is, the user can operate display terminal 15 to display and check the image data of the foods in the refrigerator.
[0059] 1-3. Refrigerator system configuration FIG. 12 shows the system configuration of a refrigerator used in the above-mentioned refrigeration system.
[0060] In FIG. 12, refrigerator 51 includes an interface (I / F) 58, a control unit 59, a memory unit 60, a door open / close detection unit 61, interior illumination lamps 34-1, 34-2, cameras 35-1, 35-2, a fan drive circuit 62, a compressor drive circuit 63, a blower fan 18, and a compressor 19, and receives power from an AC power source 64 such as a commercial power source in residence A, for example.
[0061] The interface (I / F) 58 exchanges data between the wireless adapter 53 and the control unit 59 of the refrigerator 51 .
[0062] The control unit 59 controls the operation of the refrigerator. The control unit 59 also controls the cameras 35-1, 35-2 and the interior lights 34-1, 34-2 to photograph the food in the refrigerator, temporarily stores the image data in the storage unit 60, and transmits the image data to the server device 57 on the Internet 56. Furthermore, if a request signal for advance transmission of image data is received from the display terminal 15 via the server device 57, the control unit 59 instructs the display terminal 15 to transmit the image data via the server device 57 immediately after photographing the food. The details will be described later.
[0063] In addition to the identification symbols and control programs, memory unit 60 rewrites and stores image data of food inside the refrigerator captured by cameras 35-1 and 35-2 and data for energy saving operation.
[0064] The interior lighting 34-1, 34-2 are turned on and off by the control unit 59 based on a signal from the door open / close detection unit 61. The lighting is turned on when the door 10 is opened and is turned off after a delay of several seconds when the door 10 is closed.
[0065] The cameras 35-1 and 35-2 are controlled by the control unit 59 based on a signal from the door open / close detection unit 61, and capture images of the food inside the refrigerator when the door 10 is closed. The details will be described later.
[0066] The fan drive circuit 62 receives a control signal from the control unit 59 and controls the drive of the rotation of the blower fan 18 .
[0067] The compressor driving circuit 63 receives a control signal from the control unit 59 and controls the compressor 19. Controls the rotation speed and other drive functions.
[0068] The blower fan 18 operates under the control of a fan drive circuit 62 to generate an air current for circulating cool air.
[0069] The compressor 19 compresses the refrigerant (not shown) circulating inside the refrigerator 51 under the control of the compressor drive circuit 63.
[0070] FIG. 29 shows the system configuration of a refrigerator related to voice recognition used in the above refrigeration system.
[0071] In FIG. 29, refrigerator 51 includes an interface (I / F) 58, a control unit 59, a memory unit 60, a door open / close detection unit 61, interior illumination lamps 34-1, 34-2, microphones 100-1, 100-2, a fan drive circuit 62, a compressor drive circuit 63, a blower fan 18, and a compressor 19, and receives power from an AC power source 64 such as a commercial power source in residence A, for example.
[0072] The interface (I / F) 58 exchanges data between the wireless adapter 53 and the control unit 59 of the refrigerator 51 .
[0073] The control unit 59 controls the operation of the refrigerator. The control unit 59 also controls the microphones 100-1, 100-2 to recognize external voices, temporarily stores the voice recognition data in the storage unit 60, and transmits the data to the server device 57 on the Internet 56. Furthermore, if there is a request signal for advance transmission of image data from the display terminal 15 via the server device 57, the control unit 59 instructs the display terminal 15 to transmit the image data via the server device 57 immediately after photographing the food. The details will be described later.
[0074] The storage unit 60 rewrites and stores, in addition to the identification symbols and control programs, data recognized by voice through the microphones 100-1 and 100-2 and data for energy-saving operation.
[0075] The interior lighting 34-1, 34-2 are turned on and off by the control unit 59 based on a signal from the door open / close detection unit 61. The lighting is turned on when the door 10 is opened and is turned off after a delay of several seconds when the door 10 is closed.
[0076] The microphones 100-1 and 100-2 are controlled by the control unit 59 based on a signal from the door open / close detection unit 61, and start the voice recognition device when the door 10 is opened, and stop the voice recognition device after a certain time has passed when the door 10 is closed. The details will be described later.
[0077] The fan drive circuit 62 receives a control signal from the control unit 59 and controls the drive of the rotation of the blower fan 18 .
[0078] The compressor drive circuit 63 receives a control signal from the control unit 59 and controls the drive of the compressor 19, such as the number of revolutions.
[0079] The blower fan 18 operates under the control of a fan drive circuit 62 to generate an air current for circulating cool air.
[0080] The compressor 19 compresses the refrigerant (not shown) circulating inside the refrigerator 51 under the control of the compressor drive circuit 63.
[0081] 1-4. Server device configuration Next, a configuration example of the server device of the refrigerator system in the first embodiment will be described with reference to Fig. 13. Here, the configuration of server device 57 in Fig. 11 will be taken as an example.
[0082] As shown in FIG. 13, the server device 57 includes a server interface 65, a server control unit 66, and a data storage unit 67.
[0083] The server interface 65 is a communication unit that exchanges data and the like between the display terminal 15 and the router device 55 and the server device 57 via the Internet 56 under the control of the server control unit 66 .
[0084] The server control unit 66 controls the overall operation of this server device 57. The server control unit 66 also processes and combines image data of food sent from the refrigerator 51 and stores it in the data storage unit 67, and, upon receiving a signal from the display terminal 15 requesting advance transmission of image data, sends the latest image data of the image data stored in the data storage unit 67 to the display terminal 15 each time that image data is sent from the refrigerator 51, causing the display terminal 15 to rewrite and update the data. Of course, if a data call signal is sent from the display terminal 15 without a signal requesting advance transmission of image data, the latest image data stored in the data storage unit 67 is sent to the display terminal 15.
[0085] The data storage unit 67 stores necessary data such as a management program and application programs for executing the refrigerator system. Furthermore, the data storage unit 67 rewrites and stores the latest data of food image data transmitted from the refrigerator 51 under the control of the server control unit 66. The data storage unit 67 is configured by, for example, a hard disc drive (HDD) or a solid state drive (SSD).
[0086] The server device 57 can be constructed as either a server configured with a single large computer or a cloud-type server configured with a group of multiple computers.
[0087] <2. Operation> Next, the operation of the refrigerator and refrigeration system having the above configuration will be described. Note that in the following descriptions, the same operations will be described with the same step numbers.
[0088] 2-1. Food photography 2-1-1. Shooting with the door closed FIG. 14 is a flow chart showing the operation of photographing food items stored in the refrigerator 51 when the door 10 of the refrigerator 51 is closed.
[0089] When the door 10 is opened to put food in or take it out (Y in step 1), the interior lights 34-1 and 34-2 are turned on (step 2), and when the door 10 is closed (Y in step 3), the illuminance of the interior lights 34-1 and 34-2 is reduced (step 4), and after a period of time, for example several seconds, the cameras 35-1 and 35-2 are activated to photograph the food inside the refrigerator (step 5). After that, the photographed image data is stored in the memory unit 60 of the refrigerator 51 (step 6), and the interior lights 34-1 and 34-2 are turned off (step 7).
[0090] Here, if the illuminance of the interior lighting 34-1, 34-2 is reduced during the above photographing, the food can be photographed clearly, which is preferable. In other words, if the interior lighting 34-1, 34-2 is turned on and food is photographed, the light reflected from the interior of the storage cabinet is strongly reflected on the food from all sides except the opening of the storage cabinet, since the interior of the storage cabinet is whitish (the light bounces off the food at high illuminance without reducing the illuminance), making the food look whitish and making it difficult to photograph it clearly. However, As in the above example, by reducing the illuminance of the interior lighting 34-1, 34-2, the effect of reflected light from the interior surface of the storage facility can be significantly reduced, making it possible to take clear images.
[0091] 2-1-2. Shooting during cooling operation of the refrigerator compartment FIG. 15 is a flow chart showing the operation when, after food has been taken in and out of the refrigerator, food in the storage compartment is photographed while the refrigerator compartment is being cooled. When the door 10 is opened to put food in or take it out (Y in step 1), the interior lights 34-1 and 34-2 are turned on (step 2), the door 10 is closed (Y in step 3), and it is determined whether the refrigerator compartment is in cooling operation (step 8). If the blower fan 18 is ON, the damper 22 is open, and the refrigerator compartment is in cooling operation, the illuminance of the interior lights 34-1 and 34-2 is reduced (step 4), and the cameras 35-1 and 35-2 are activated to photograph the food inside the refrigerator (step 5), just as when the door is closed. Then, the photographed image data is stored in the memory unit 60 of the refrigerator 51 (step 6), and the interior lights 34-1 and 34-2 are turned off (step 7).
[0092] In this way, by taking pictures after the refrigerator compartment is cooled, even if condensation occurs on the surface of food newly placed in the storage compartment due to the temperature difference between the inside and outside of the storage compartment, the condensation is removed by the cooling operation. Therefore, the pictures taken by cameras 35-1 and 35-2 are clear enough to read the characters on the food packaging containers, which is preferable.
[0093] During the above operation, when the door 10 is closed (Y in step 1), the interior lights 34-1, 34-2 are turned off once, and then turned on again before photographing to reduce the illuminance, thereby making it possible to reduce wasteful power consumption caused by the interior lights 34-1, 34-2 continuing to be turned on.
[0094] 2-1-3. Shooting with the door open FIG. 16 is a flow chart showing the operation in the case of a single door, in which photographing of food inside the storage cabinet is performed when the door 10 is opened to a predetermined angle.
[0095] When the door 10 is opened to take food in or out (Y in step 1), the interior lights 34-1 and 34-2 are turned on (step 2), and when the door is opened to a predetermined angle, for example, 30 degrees as shown in Fig. 13 (Y in step 10), the illuminance of the interior lights 34-1 and 34-2 is reduced (step 4), and the cameras 35-1 and 35-2 are activated to photograph the food inside the refrigerator (step 5). After that, the photographed image data is stored in the memory unit 60 of the refrigerator 51 (step 6), and when the door is closed (Y in step 11), the interior lights 34-1 and 34-2 are turned off (step 7).
[0096] If, as in this example, a photograph is taken when the door 10 is opened to a specified angle, a wide range of the interior of the storehouse can be photographed as already described, and the image produced will be natural and unobtrusive, which is preferable.
[0097] When photographing when the door is opened at a predetermined angle, photographing is performed at least twice, once when the door is opened and once when the door is closed. In this case, by storing in memory unit 60 the image data photographed last during one operation from opening to closing the door (during the door closing operation), it is possible to photograph the latest state of the food in the storage unit after food has been taken in and out.
[0098] Needless to say, in step 10 of detecting whether the opening angle of door 10 reaches a predetermined angle, if door 10 is closed before reaching the predetermined angle, interior lighting 34-1, 34-2 are turned off and the system waits for door 10 to be opened again.
[0099] Although the typical shooting operations have been described above, examples other than the above shooting operations may be used. For example, a photographing operation may be performed at a fixed time or when a predetermined time has elapsed since the previous photographing, or when a photographing instruction is received from the display terminal 15.
[0100] 2-1-4. Lighting during shooting As explained in Figures 14, 15, and 16, food lighting during photography is done by turning on the interior lighting 34-1 and 34-2 and then reducing the illuminance. In some cases, it may be possible to turn on only the interior lighting 34-1 on both sides of the storage compartment or the interior lighting 34-2 on the top of the storage compartment and reduce the illuminance for photography. The reason for taking photos with reduced illuminance will be described later.
[0101] As another example of irradiating food while photographing, the following can be considered: Fig. 17 is a flow chart showing the on / off states of the interior lights 34-1, 34-2 while photographing with the cameras 35-1, 35-2, and corresponds to the operations shown by the dashed lines in Figs. 14, 15, and 16.
[0102] 14, 15, and 16 are examples in which the interior lighting 34-1, 34-2 are turned on and then their illuminance is reduced, but in this example, one of the interior lighting 34-1 on each side of the storage compartment is turned off, in other words, only one of the interior lighting 34-1 on each side of the storage compartment is turned on to illuminate the food and take the picture. In this case, the interior lighting 34-2 on the top of the storage compartment may be turned on or off, but one of them may be appropriately selected so that the illuminance inside the storage compartment is optimal for photography.
[0103] In Fig. 17, before starting to photograph using cameras 35-1 and 35-2, one of the interior lights 34-1 on both sides of the storage compartment that is already on, for example the right interior light, is turned off (step 12), and the food is photographed (step 13) as illuminated by the left interior light and stored (step 14). After that, the right interior light is turned on (step 15), and then the left interior light is turned off (step 16), and the food is photographed as illuminated by the right interior light (step 17). The image data is stored (step 18), and the right interior light is turned off (step 19). The image data thus stored while illuminating the food from one side is combined into one image and stored.
[0104] The reason why the interior lighting 34-1, 34-2 are turned on one by one to photograph the food and then combined into one is as follows. That is, as already mentioned, the illuminance of the interior lighting 34-1, 34-2 is reduced when photographing, but if the illuminance is reduced too much, the image becomes dark and lacks clarity, so there is a limit to the reduction in illuminance. Therefore, a certain level of illuminance is maintained, but even with that illuminance, the effect of reflected light may remain if the storage volume is small. However, as explained in this example, if the left and right sides of the interior lighting 34-1 are turned on one by one to photograph the food, the effect of reflected light can be reduced, and the side on which the interior lighting 34-1 is turned on can be photographed with appropriate brightness. Therefore, if the left and right images of the food photographed with this appropriate brightness are combined to perform a composition process, the whole image becomes clear, which is preferable.
[0105] 2-2. Saving and compositing images Next, the storage and synthesis of image data captured by each of the above-mentioned operations will be described.
[0106] 2-2-1-Saving and compositing on the server FIG. 18 shows a case where image data is stored and combined in server device 57. When the interior of the refrigerator is photographed by the operations shown in FIGS. 14 to 16 including FIG. 17, refrigerator 51 connects to Internet 56 via wireless adapter 53, gateway device 54 and router device 55 (step 20), and transmits the individual image data photographed and stored by each camera to Internet 56. The image data transmitted via the Internet 56 is then sent to the server device 57 (step 21). The image data transmitted via the Internet 56 is subjected to a synthesis process (step 22) in which the individual image data are combined by the server control unit 66 of the server device 57. The synthesized image data is stored in the data storage unit 67 of the server device 57 (step 23).
[0107] If the images are synthesized and stored in server device 57 as in this example, the load on control unit 59 and memory unit 60 on the refrigerator side can be reduced, and an internet-compatible refrigerator can be provided at a low cost.
[0108] 2-2-2-Storage and synthesis on the refrigerator side Fig. 19 shows a case where image data is stored and combined on the refrigerator side, and when the interior of the refrigerator is photographed by the operations shown in Figs. 14 to 16 including Fig. 17, control unit 59 of refrigerator 51 extracts the individual image data from each camera stored in storage unit 60, combines and combines them (step 24). The combined image data is stored (step 25), and refrigerator 51 connects to Internet 56 via wireless adapter 53, gateway device 54 and router device 55 (step 20), and transmits the stored image data to server device 57 connected to Internet 56 (step 21). The combined image data transmitted via Internet 56 is stored in data storage unit 67 of server device 57 (step 23).
[0109] In this example, the load on control unit 59 and memory unit 60 on the refrigerator side increases, but image data can be transmitted and received to and from display terminal 15 without going through data memory unit 67 of server device 57, making it possible to reduce the system operating costs.
[0110] 2-2-3. Saving and compositing on the display device Fig. 20 shows a case where image data is stored and synthesized on the display terminal 15 side, and when an image of the interior is taken by the operations shown in Figs. 14 to 16 including Fig. 17 described above, refrigerator 51 connects to Internet 56 via wireless adapter 53, gateway device 54 and router device 55 (step 20), and transmits the image data to display terminal 15 instead of the server (step 26). The image data transmitted via Internet 56 is subjected to synthesis processing (step 27) by display terminal 15 to combine the image data. The synthesized image data is stored in the data storage unit of display terminal 15 (step 28).
[0111] In this example, although the load on display terminal 15 increases, the load on control unit 59 and memory unit 60 on the refrigerator side can be reduced, and an Internet-enabled refrigerator can be provided at a low cost.
[0112] Although typical examples of image data transmission and synthesis operations have been described above, the storage and synthesis do not necessarily have to be performed in the same location, such as server device 57. For example, the image data may be synthesized in refrigerator 51 and stored in server device 57, or any other location may be set according to the system.
[0113] 2-3.Normal transmission and reception of image data 2-3-1. Sending and receiving image data stored on a server or refrigerator FIG. 21 is a flow chart showing transmission and reception between server device 57 or refrigerator 51 and display terminal 15, and shows transmission and reception of image data stored in server device 57 or refrigerator 51.
[0114] First, the user presses a server connection button (not shown) displayed on the display of the display terminal 15, for example, at a shopping destination, to request reading of image data stored in the data storage unit 67 of the server device 57 (step 31). It is detected whether image data has been received from the server device 57 (step 32), and if no image data has been transferred, a connection is made from the display terminal 15 to the Internet 56 (step 33). The image data stored in the data storage unit 67 of the server device 57 or the storage unit 60 of the refrigerator is retrieved (step 34), and the image data is transferred via the Internet 56 to the display terminal 15 from which the data was retrieved, such as a smartphone (step 35). It is detected whether image data has been received from the server device 57 (step 32), and if the image data has been received, the image data is output to the display of the display terminal 15 (step 36). Thereafter, it is detected whether an enlargement button (not shown) displayed on the display of display terminal 15 has been pressed (step 37), and if it is detected that the enlargement button has been pressed, image data is created by digitally enlarging the image data in the storage output to the display by 1.1 times or more, and the enlarged image data is output to the display of display terminal 15 (step 36).
[0115] Then, it is detected whether the numeric keypad buttons (not shown) displayed on the display have been pressed (step 38), and if it is detected that the numeric keypad buttons have been pressed, the enlarged image data in the storage that has been output to the display is made the image data of the location specified by the numeric keypad buttons, and is output to the display (step 36). It is further detected whether the numeric keypad buttons have been pressed (step 38). If it is not detected, the process ends.
[0116] The enlargement button and the numeric keypad button for displaying image data at a predetermined location may be operated by sliding a finger over the surface of the display, and this configuration improves ease of use.
[0117] As described above, this refrigerator system allows the user to check the contents of refrigerator 51 at a glance when shopping, preventing forgetting to buy food and eliminating the waste of buying extra food. Also, in households where both partners work, the user can check the contents of storage compartment 5 of refrigerator 51 just before leaving home and purchase food on the way home, saving time for the user.
[0118] In addition, by enlarging the display or displaying a specific portion, even a display terminal 15 as small as the display terminal 15 can display an enlarged image, allowing ingredients that are difficult to see on a small display terminal 15 to be seen with certainty.
[0119] 2-3-2. Sending and receiving image data stored on a display terminal Fig. 22 shows a case where image data has already been stored in display terminal 15. In this case, image data has been sent to and stored in display terminal 15 as described in Fig. 24, and when a request to read this image data is made (step 31), the image data is retrieved from the storage section of display terminal 15 and output to the display of display terminal 15 (step 36). Subsequent enlarged display operations and local display are as described in Fig. 25.
[0120] 2-3-3. Switching and receiving image data The images captured by cameras 35-1, 35-2 include individual images captured by each camera 35-1, 35-2 (the upper or lower half of the storehouse), and a composite image created by combining these individual captured images (a single composite image showing the inside of the storehouse from top to bottom).This shows an example in which these images can be switched between, transmitted, received, and displayed on display terminal 15.
[0121] FIG. 23 is a flow chart showing the operation in such a case.
[0122] First, the user presses an individual image / composite image call button (not shown) displayed on the display of the display terminal 15 (step 41) to call up either an individual image or a composite image (step 42). Either image data of a separate image or a composite image is captured and displayed on the display of the display terminal 15.
[0123] As in this example, by making it possible to switch the image displayed on display terminal 15 to either an individual image or a composite image, the composite image can usually be called up and displayed to check the food situation in the storehouse at a glance, and when the food is difficult to identify because it is in a corner of the storehouse, for example, the image can be switched to the individual image for reliable identification, improving usability.
[0124] 2-4. Transmission and reception of image data, etc. when requested in advance FIG. 24 shows a flow chart when a pre-transmission request for an image is made from the display terminal 15. In FIG.
[0125] First, the user operates display terminal 15 to request a pre-request signal (step 51). Server device 57 or refrigerator 51 receives this request signal (step 52), reads and transmits image data (individual image data or composite image data, the same applies below) stored in data storage unit 67 or storage unit 60 (step 53). Display terminal 15 detects whether image data has been received from server device 57 or refrigerator 51 (step 54), and if no image data has been received, connects to Internet 56 from display terminal 15 (step 55). Server device 57 or similar retrieves image data stored in refrigerator 51 (step 56), and transmits the image data to display terminal 15 via Internet 56 (step 57). It detects whether image data has been received from server device 57 (step 54), and if image data is detected, waits until the image data is called (N in step 58). If image data is called (Y in step 58), the image data is output to the display of display terminal 15 (step 59).
[0126] As a result, when image data is requested by display terminal 15, even if access to server device 57 is concentrated and transmission of the image data from server device 57 is delayed, the image data can be displayed on the display in real time at the same time as the image data is requested, allowing the image to be confirmed. This makes it possible to avoid a situation where the customer has to wait for the image data to be displayed at the shopping destination, thereby shortening shopping time.
[0127] Furthermore, if server device 57 is set to transmit image data to display terminal 15 each time door 10 of refrigerator 51 is closed when a request for advance image transmission is received from display terminal 15, and if the user of display terminal 15 sets the device to ignore the image data when it is transmitted, the user can know each time the door is opened or closed. Therefore, if the user thinks that the number of times the door is opened and closed is increasing and power consumption is likely to increase, the user can call home and tell their children to refrain from using refrigerator 51, which also improves convenience for energy conservation.
[0128] 2-5. Switching the transmission and reception path of image data, etc. 25 shows the display when a communication route is selected on the display terminal 15. The flow of sending and receiving image data has already been described, but changing the communication route speeds up the acquisition of image data.
[0129] Specifically, the user can select whether the display terminal 15 is to directly communicate with the gateway device 54 via the router device 55 or to communicate with the server device 57 via the Internet 56 .
[0130] In FIG. 25, 70 and 71 are communication path selection buttons displayed on the display of the display terminal 15. The 70 is a button for selecting a communication path between the display terminal 15 and the router device 55 and the gateway device 54. The "Direct" selection button is for direct connection to the refrigerator 51, and 71 is a "Server" selection button for further connection to the server device 57 or the refrigerator 51 via the Internet 56. 72 and 73 are data selection buttons for data to be called up, 72 is a button for image data of food, and 73 is a button for energy saving related data such as the number of times the door is opened and closed. 74 is a send button.
[0131] When "Direct" selection button 70 is selected, "Direct" communication is set for display terminal 15. When "Direct" communication is set, and the user touches send button 74, an image data request signal for display terminal 15 is transmitted directly from display terminal 15 to gateway device (relay device) 54 via router device 55. Gateway device 54 transmits the received data request signal to wireless adapter 53 of refrigerator 51. Upon receiving the image data request signal, wireless adapter 53 transmits the signal to control unit 59 of refrigerator 51. Upon receiving the image data request signal, control unit 59 of refrigerator 51 reads the image data stored in storage unit 60, and transmits the image data to display terminal 15 via wireless adapter 53 and router device 55.
[0132] Therefore, when the user is located within residence A as shown by the dashed line in Figure 11, in a broad sense, when display terminal 15 is located in a position where it can communicate directly with router device 55, display terminal 15 can communicate with gateway device 54 via router device 55 and directly obtain image data from memory unit 60 of refrigerator 51.
[0133] As a result, in the event that a delay occurs in the response of image data transmission to a user's operation on display terminal 15 due to line congestion caused by concentrated access to Internet 56, the user can switch display terminal 15 to "direct" communication, thereby enabling image data to be obtained without being affected by line congestion on Internet 56. Therefore, the user can check the food status in storehouse 5 by displaying images on the display of display terminal 15 in real time without waiting time.
[0134] On the other hand, when the user is outside residence A as shown by the solid line in Figure 11, or in a broader sense, when display terminal 15 is in a location where it cannot directly communicate with router device 55, the user can touch "server" selection button 71, which causes display terminal 15 to communicate with server device 57 via Internet 56 and read image data of refrigerator 51. In other words, when the user is outside residence A, if the user touches "server" selection button 71 to set up "server" communication, the user can obtain image data of the refrigerator using display terminal 15 and know the food storage status even if he is in a remote location.
[0135] In this way, the refrigerator system in this embodiment allows the user to select either "direct" communication or "server" communication. When the user is within residence A, the user can select "direct" communication and comfortably view images without being affected by congestion on the Internet line.
[0136] Although the refrigerator and the refrigerator system of the present invention have been described above by using specific configurations as examples, the present invention is not limited to the above configurations.
[0137] For example, in the above embodiment, gateway device 54 is provided in the communication path between display terminal 15, wireless adapter 53 in refrigerator 51, and Internet 56. However, gateway device 54 is not necessarily required. Display terminal 15, wireless adapter 53 in refrigerator 51, and Internet 56 may be configured to communicate directly via router device 55, thereby simplifying the system configuration.
[0138] In the above embodiment, when the user is outside the residence A, the image data is transferred from the server device 57. However, it may be configured so that image data stored in storage unit 60 of refrigerator 51 is directly acquired from refrigerator 51 by communicating with Internet 56, router device 55 and gateway device 54 from display terminal 15 without going through server device 57. In this case and when "direct" communication is performed, it is preferable that image combination processing is performed in storage unit 60 of refrigerator 51 and the image data after this combination processing is transmitted.
[0139] Furthermore, in this embodiment, a "Direct" selection button 70 and a "Server" selection button 71 are displayed on the display of the display terminal 15, and the user can select between "Direct" communication and "Server" communication. However, the display terminal 15 may be provided with a function to automatically switch to "Direct" communication when it catches radio waves from the router device 55, so that "Direct" communication is automatically selected, thereby improving usability.
[0140] 2-6. Voice recognition of food names 2-6-1.Voice recognition operation when opening the door FIG. 30 is a flow chart showing the operation of the voice recognition device when the door 10 of the refrigerator 51 is opened.
[0141] When the door 10 is opened to check the food (Y in step 101), the voice recognition devices 100-1 and 100-2 are activated (step 102), and the interior lights 34-1 and 34-2 are turned on (step 103). When the voice recognition device recognizes the voice (Y in step 104), the voice input levels of the voice recognition devices 100-1 and 100-2 are checked by a microphone discrimination process to determine which voice recognition data to use (step 105), the adopted voice data is analyzed by a voice analysis process to confirm the input food name (step 106), and the confirmed voice data is stored in the storage unit 60 of the refrigerator 51 (step 107). When the voice recognition device does not recognize the voice (N in step 104), the state of the door 10 is checked, and if the door 10 is not closed, the voice of the voice recognition device is checked again (N in step 108). If the door 10 is closed (Y in step 108), the interior lights 34-1, 34-2 are turned off (step 109), and after a period of time, for example several seconds, the voice recognition devices 100-1, 100-2 are stopped (step 110).
[0142] Here, by linking the voice recognition device to the door 10, it is possible to start the voice recognition device when the door is opened and stop the voice recognition device when the door is closed, which is preferable. In other words, since it is not necessary to operate the voice recognition device all the time, it is possible to prevent erroneous voice recognition and to achieve energy saving.
[0143] 2-6-2. Voice recognition operation by presence of human sensor FIG. 31 is a flow chart showing the operation of the voice recognition device when it is detected that a person is in front of the refrigerator 51.
[0144] When the human presence sensor 101 detects the presence of a person in front of the refrigerator (Y in step 111), the voice recognition devices 100-1 and 100-2 are started (step 102). When the voice recognition device recognizes a voice (Y in step 104), the voice input level of the voice recognition devices 100-1 and 100-2 is checked by a microphone discrimination process to determine which voice recognition data to adopt (step 105), the adopted voice data is analyzed by a voice analysis process to confirm the input food name (step 106), and the confirmed voice data is stored in the storage unit 60 of the refrigerator 51 (step 107). When the voice recognition device does not recognize a voice (N in step 104), the state of the human presence sensor 101 is checked and if a person is detected, the voice of the voice recognition device is checked again (N in step 112). When a person is not detected (Y in step 112), the voice recognition devices 100-1 and 100-2 are stopped after a while, for example, several seconds, has elapsed (step 110).
[0145] Here, by linking the voice recognition device to the human presence sensor 101, the voice recognition device can be started when a person is detected and stopped when the person is no longer detected, which is preferable. In other words, since it is not necessary to operate the voice recognition device all the time, it is possible to prevent erroneous voice recognition and to achieve energy saving.
[0146] Although typical voice recognition operations have been described above, other voice recognition operations may be considered, such as voice recognition operations that are started when a voice recognition start instruction is received from the display terminal 15.
[0147] 2-6-3.Voice recognition operation when closing the door FIG. 32 is a flow chart showing the operation of the voice recognition device when the door 10 of the refrigerator 51 is closed. If the door 10 is closed (Y in step 108), the interior lighting 34-1, 34-2 are turned off (step 109), and after a while, for example, a few seconds, the voice recognition devices 100-1, 100-2 are stopped (step 110). If the door 10 is not closed, the door state is checked again (N in step 108).
[0148] Here, by linking the voice recognition device to the door 10, it is possible to preferably stop the voice recognition device when the door is closed. In other words, there is no need to take the trouble to manually stop the voice recognition device, and there is no risk of forgetting to stop the voice recognition device, which makes it possible to prevent erroneous voice recognition and achieve energy saving.
[0149] 2-6-4. Voice recognition operation with or without a human sensor FIG. 33 is a flow chart showing the operation of the voice recognition device when it is detected that no one is in front of the refrigerator 51. If the state of the human sensor 101 is checked and no human is detected (Y in step 112), the voice recognition devices 100-1 and 100-2 are stopped after a while, for example, a few seconds, has elapsed (step 110). If a human is detected, the state of the human sensor is checked again (N in step 112).
[0150] Here, by linking the voice recognition device to the human presence sensor 101, it is possible to preferably stop the voice recognition device when no human is detected. In other words, there is no need to take the trouble to manually stop the voice recognition device, and there is no risk of forgetting to stop the voice recognition device, so that erroneous voice recognition can be prevented and energy can be saved.
[0151] Although typical voice recognition operations have been described above, other voice recognition operations may be considered, such as voice recognition operations that are stopped when a voice recognition stop command is received from the display terminal 15.
[0152] 2-7. Search for food recipes using voice recognition Next, recipe search using data obtained by voice recognition through the above-mentioned operations will be described.
[0153] 2-7-1. Search for recipes on the server Fig. 34 shows a case where a recipe is searched for in server device 57 according to voice recognition data. When voice recognition is performed by the operations shown in Figs. 30 and 31, refrigerator 51 connects to Internet 56 via wireless adapter 53, gateway device 54, and router device 55 (step 113), and transmits the voice-recognized data to server device 57 connected to Internet 56 (step 114). The voice recognition data transmitted via Internet 56 A recipe search process (step 115) is performed using the voice recognition data by the server control unit 66 of the server device 57. The recipe search results are stored in the data storage unit 67 of the server device 57 (step 116).
[0154] As in this example, if recipe search is performed and stored in server device 57, the load on control unit 59 and memory unit 60 on the refrigerator side can be reduced, and an internet-compatible refrigerator can be provided at a low cost.
[0155] 2-7-2. Search for recipes on the display device Fig. 35 shows a case where a recipe is searched for on the display terminal 15 side according to voice recognition data, and when voice recognition is performed by the operations shown in Figs. 30 and 31, refrigerator 51 connects to Internet 56 via wireless adapter 53, gateway device 54, and router device 55 (step 113), and transmits the voice recognized data not to the server but to display terminal 15 (step 117). The voice recognition data transmitted via Internet 56 is used by display terminal 15 to perform recipe search processing (step 118). The recipe search results are saved in the data storage unit of display terminal 15 (step 119).
[0156] In this example, although the load on display terminal 15 increases, the load on control unit 59 and memory unit 60 on the refrigerator side can be reduced, and an Internet-enabled refrigerator can be provided at a low cost.
[0157] The above describes typical examples of the transmission of voice-recognized data and recipe search operations, but the storage and recipe search do not necessarily have to be performed in the same place, such as server device 57. For example, the recipe search process can be performed on the display terminal 15 side and the recipe search results can be stored in server device 57, or any other setting can be used depending on the system.
[0158] 2-8. Displaying recipes on a display device 2-8-1. Displaying recipe search results on a display device FIG. 36 is a flow chart showing transmission and reception between server device 57 or refrigerator 51 and display terminal 15, and shows transmission and reception of data stored in server device 57 or refrigerator 51.
[0159] If voice recognition is available, display terminal 15 receives the voice recognition data identified by the voice analysis process of refrigerator 51 and displays it on display terminal 15 (step 120). After the voice recognition data is displayed, a recipe search is performed on server device 57 or display terminal 15 by pressing a search start button on display terminal 15 or by speaking a specific keyword, and the recipe search status is continuously checked until the recipe search is completed (N in step 121). When the recipe search is completed (Y in step 121), the recipe search results are displayed on display terminal 15 (step 122). The recipe search results may be automatically adjusted to an optimal display position depending on the user's height.
[0160] As described above, according to this refrigerator system, the results of a recipe search can be displayed on display terminal 15. By removing display terminal 15, there is no restriction on where the display terminal can be viewed, and it can be carried close to where cooking is being done to check the results, improving the efficiency of cooking work.
[0161] <3. Effects> As described above with reference to specific configurations, the refrigerator and refrigerator system according to the embodiment of the present invention achieve various functions and effects. However, the refrigerator according to the gist of the present invention, that is, By providing a voice recognition device on the front of the refrigerator door and on the inside side of the refrigerator, voice recognition is possible regardless of whether the refrigerator door is open or closed, making it possible to provide a refrigerator that allows recipe searches using only voice. Also, by linking it to door opening / closing and a human presence sensor, it is not necessary to operate the voice recognition means all the time, making it possible to prevent erroneous recognition and to provide a refrigerator that contributes to energy conservation.
[0162] In the refrigerator of this embodiment, cameras 35-1, 35-2 photograph the food in storage cabinet 5 from the front side of storage cabinet 5, but interior lighting 34-1, 34-2 are also provided on the front side of storage cabinet 5 and illuminate the food in storage cabinet 5 from the same front side as the direction of photography by cameras 35-1, 35-2. Therefore, the photographed image is clear and not dark or difficult to see, as in the case where interior lighting 34-1, 34-2 are provided on the back wall of storage cabinet 5. In addition, there is no need to request server device 57 to perform image processing to brighten and clarify the photographed image.
[0163] Moreover, the cameras 35-1, 35-2 and interior lighting 34-1, 34-2 are provided so as to be positioned forward of the front ends of the shelves 30-1 to 30-4, so that the images of the inside of the storage cabinet 5 taken by the cameras 35-1, 35-2 are the same as the state that the user would actually see when he or she opens the door 10 of the refrigerator 51. Therefore, when the user displays the image data on the display terminal 15, he or she can check the state of the food in the storage cabinet 5 without feeling any discomfort. Moreover, the interior lighting 34-1, 34-2 illuminate the food on the shelves 30-1 to 30-4 from in front of the front ends thereof, so that the food placed on the front ends of the shelves 30-1 to 30-4 can also be photographed brightly and clearly.
[0164] Furthermore, since the cameras 35-1, 35-2 are provided on the inner surface of the door 10 of the refrigerator 51 and the interior lights 34-1, 34-2 are provided on both side walls of the opening edge of the storage cabinet 5, the cameras 35-1, 35-2 are positioned in the approximate center of the left and right of the storage cabinet 5. Therefore, the images captured by the cameras 35-1, 35-2 can be closer to the state that the user actually sees when he opens the door 10 of the refrigerator 51. In addition, by increasing the distance between the cameras 35-1, 35-2 and the food on the shelf, a wider range within the storage cabinet 5 can be captured, and the storage status of a wider range of food can be checked on the display terminal 15.
[0165] In addition, the cameras 35-1 and 35-2 are provided so as to be located approximately in the center of the door pockets 33-1 and 33-2 on the inside of the door in the front-to-rear direction, so they can photograph not only the food in the storage cabinet 5 but also parts of the food stored in the door pockets 33-1 and 33-2, such as milk cartons. Since the food is something that the user stored himself, he can recognize what it is even if only a part of it is shown in the picture. Therefore, it is possible to photograph almost all the food in the storage cabinet.
[0166] Also, by providing a camera 35-3 dedicated to the storage container as shown in Fig. 9(a), it is possible to clearly photograph the food inside the storage container 31. Furthermore, by making the camera 35-2 movable up and down as shown in Fig. 9(b), or by making it freely rotatable up and down as shown in Fig. 9(c), it is possible to efficiently photograph the food inside the storage container 31 with only two cameras, i.e., with a small number of cameras.
[0167] On the other hand, the interior lighting lamps 34-1, 34-2 are provided at a position behind the cameras 35-1, 35-2 as is clear from Fig. 7, and are provided so that the light from the interior lighting lamps 34-1, 34-2 does not directly enter the shooting section of the cameras 35-1, 35-2. This prevents the light from the interior lighting lamps 34-1, 34-2 from entering the cameras 35-1, 35-2 and causing the cameras 35-1, 35-2 to take poor pictures, and good images can always be obtained. This is achieved by providing a light-shielding section on the camera side of the interior lighting lamps 35-1, 35-2, or by configuring the interior lighting lamps 34-1, 34-2. The same effect can be obtained by arranging the light source such as an LED diagonally backward.
[0168] The interior lighting 34-1, 34-2 provides an illuminance of 150 lux to 200 lux inside the storage chamber. If the food is photographed at this illuminance, as explained in the photographing operation section of FIG. 14, the food will appear whitish and difficult to distinguish. Therefore, in this embodiment, the illuminance is reduced for photographing. This allows clear photographing even with a CCD camera or CMOS camera. The illuminance of the interior lighting 34-1, 34-2 may be controlled by the server control unit 66 so that the illuminance is lower than normal only when photographing with the door 10 closed or the door 10 open. By controlling in this way, the illuminance can be maintained as before during normal photographing, making it easier to see the interior of the storage chamber and allowing clear photographing. The reduction in illuminance can be achieved in various ways, such as by turning off one or both of the interior lights 34-1, 34-2 or by reducing the illuminance of both of the interior lights 34-1, 34-2, and the like, and may be selected as appropriate.
[0169] In addition, the images of the inside of storehouse 5 taken by cameras 35-1, 35-2 are transmitted to server device 57 and combined and synthesized in server device 57, refrigerator 51, or display terminal 15. In this embodiment, the image combining process is facilitated.
[0170] In other words, the cameras 35-1, 35-2 are provided on the inner door surface corresponding to the front ends of the shelves 30-1 to 30-4, i.e., in this example, the front ends of the upper shelf 30-1 and the third shelf 30-3, so that the cameras 35-1, 35-2 take pictures centered on the shelves 30-1, 30-3, as shown by line Z in Figure 26.
[0171] Therefore, the image data of the inside of the storehouse 5 taken by the two cameras 35-1 and 35-2 can be combined at the second shelf 30-2. That is, as shown in the "photographed raw image" in FIG. 27, the shelf 30-2 is commonly shown in both the upper and lower images, and the curvature of this "photographed raw image" is corrected to create a "curve-corrected image", and the common shelf 30-2 that appears in the upper and lower images is aligned to create a photographed image of the entire storehouse. As a result, it is no longer necessary to perform extremely difficult correction processing for aligning and combining each food item placed on the shelf when combining images, and the image combining process becomes easier. Therefore, the processing program for combining images can be simplified, and the control part of the server device 57, the refrigerator 51, or the display terminal 15 does not need to have a large capacity for the image combining process, which contributes to speeding up the processing speed.
[0172] Furthermore, since the two cameras 35-1, 35-2 are provided approximately in front of the shelves 30-1 and 30-3, which cannot be adjusted in height, even if the height of the height-adjustable shelf 30-2 is changed, the image combining process can be easily performed by adjusting the range to be combined according to the height of the shelf 30-2 during the image combining process. [Industrial Applicability]
[0173] The present invention can provide a refrigerator and a refrigerator system that can search for recipes related to foods stored in the storage compartment using only voice, and can be applied to home refrigerators. [Explanation of symbols]
[0174] 1 Refrigerator body 2 Outer box 3 Inner box 4. Insulation 5 Storage room (refrigerated room) 6. Switch room (storage room) 7 Ice room (storage room) 8 Freezer (storage room) 9 Vegetable room (storage room) 10 Doors 10a Outer panel 10a-1 Glass plate 10a-2 Magic Mirror 10b Inner plate 10b-1,10b-2 Raised wall part 11 Center pillar 12a, 12b, 13, 14 Drawer doors 15 Display terminal 15a Display 16 Cooling room 17 Cooler 18 Blower fan 19 Compressor 20 Heat dissipation pipe 21 Capillary tube 22 Damper 23 Storage frame 24 Terminal holder 25 Touch Panel 26 Gap 27 Frame 28 Recess 29, 29a Smoke Sheet 30-1~30-4 Shelf 31 Storage Container 32-1~32-4 Side wall convex part 33-1~33-8 Door pocket 34-1, 34-2 Interior lighting (lighting equipment) 35-1, 35-2, 35-3 Camera (photography device) 41 Camera module 50 Refrigeration System 51 Refrigerator 53 Wireless adapter (communication device) 53a "Connect" button 54 Gateway device (relay device) 55 Router device 56 Internet 57 Server equipment 58 Interface (I / F) 59 Controller (control unit) 60 Storage section 61 Door opening / closing detector 62 Fan drive circuit 63 Compressor drive circuit 64 AC power supply 65 Server Interface 66 Server control section 67 Data Storage Unit 70 "Direct" selection button 71 "Server" selection button 72 Image data selection button 73 Energy saving related data selection button 74 Send button 100-1, 100-2 Microphone (voice recognition device) 101 Human Sensor A Residence
Claims
1. In a system including a refrigerator and a display terminal, The refrigerator includes: a first voice recognition means for recognizing a voice, the first voice recognition means being provided on a front surface of the refrigerator; A human detection means for detecting the presence of a human in front of the refrigerator; and a control means. When the presence of a person in front of the refrigerator is detected by the person detection means, the control means activates the first voice recognition means, When the voice is recognized by the first voice recognition means, the control means judges the voice data recognized by the first voice recognition means, After the control means determines the voice data, the voice data is displayed on the display terminal; a confirmation means for determining whether or not to execute a recipe search using the voice data; When the confirmation means determines that a recipe search using the voice data is to be performed, a result of the recipe search is displayed on the display terminal, the refrigerator further comprises a refrigerator compartment; and a second voice recognition means for recognizing voice, the second voice recognition means being provided on a side surface inside the refrigerator compartment; When the presence of a person in front of the refrigerator is detected by the person detection means, the control means activates the first voice recognition means and the second voice recognition means, A system characterized in that it determines which voice data to adopt from among the first voice data recognized by the first voice recognition means and the second voice data recognized by the second voice recognition means, based on a voice input level.
2. The refrigerator further includes a door provided in the refrigeration chamber and a door opening / closing detection means for detecting an opening / closing state of the door, 2. The system according to claim 1, wherein when the door open / close detection means detects that the door is open, the control means activates the first voice recognition means and the second voice recognition means.
3. The display terminal displays a selection screen for selecting whether or not to execute a recipe search using the voice data. Having the means, 3. The system according to claim 1, wherein when the selection means selects to execute a recipe search using the voice data, a result of the recipe search is displayed on the display terminal.
4. The system described in claim 1 or 2, characterized in that when the first voice recognition means or the second voice recognition means recognizes a keyword for performing the recipe search, the results of the recipe search are displayed on the display terminal.
5. The refrigerator further includes a lighting means provided in the refrigeration chamber for illuminating the inside of the refrigeration chamber, When the door open / close detection means detects an open state of the door, the illumination means is turned on; 3. The system according to claim 2, wherein when the door open / close detection means detects that the door is closed, the lighting means is turned off, and the first voice recognition means and the second voice recognition means are stopped a predetermined time after the lighting means is turned off.
6. 2. The system according to claim 1, further comprising a transmitting means for transmitting the first voice data or the second voice data to an outside.
7. 7. The system according to claim 1, wherein the voice data is a food name.
8. 3. The system of claim 2, wherein the door is a double door.
9. A refrigerator, a first voice recognition means for recognizing a voice, the first voice recognition means being provided on a front surface of the refrigerator; A human detection means for detecting the presence of a human in front of the refrigerator; A control means; A display means, When the presence of a person in front of the refrigerator is detected by the person detection means, the control means activates the first voice recognition means, When the voice is recognized by the first voice recognition means, the control means judges the voice data recognized by the first voice recognition means, After the voice data is determined by the control means, the voice data is displayed on the display means; a confirmation means for determining whether or not to execute a recipe search using the voice data; When the confirmation means determines that a recipe search using the voice data is to be performed, a result of the recipe search is displayed on the display means; The refrigerator further includes a refrigerator compartment and a second voice recognition means for recognizing voice, the second voice recognition means being provided on a side surface inside the refrigerator compartment, When the presence of a person in front of the refrigerator is detected by the person detection means, the control means activates the first voice recognition means and the second voice recognition means, a voice input level determining unit that determines which voice data to adopt from among the first voice data recognized by the first voice recognition means and the second voice data recognized by the second voice recognition means.
Citation Information
Patent Citations
Refrigerator
JP2002081818A
Device and method for managing food in refrigerator
JP2002236798A
Operating device in food material management system
JP2003222460A
Electric appliance, refrigerator, and operating method for refrigerator
JP2005241215A
Article storage, article storage monitoring system, and refrigerator monitoring system
JP2007046833A
