refrigerator
By introducing door opening and closing detection and sound recognition control in the refrigerator, combined with alarm actions, the problem of continuous recognition of the refrigerator when the door is open is solved, and sound recognition is realized that timely starts and automatically stops, improving the user experience.
Patent Information
- Application Number
- CN201911059240.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-11-08
- Filing Date
- 2019-11-01
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-11-01
AI Technical Summary
The existing refrigerators continue to recognize sounds while the door is open, and the user needs to stop manually, but it is easy to forget, resulting in unnecessary recognition, affecting the user experience.
The door opening and closing detection unit detects the status of the refrigerator door, and only starts the sound recognition when the specific sound recognition conditions are met, otherwise the identification will not be performed, and it will automatically stop when the identification is completed, and notify the user by combining different alarm actions of the alarm buzzer.
It realizes sound recognition at the right time, reduces unnecessary recognition, improves user experience, and completes identification and stop without manual operation of the user.
Smart Images

Figure CN111161736B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a refrigerator with a sound acquisition function. Background Art
[0002] In recent years, refrigerators with sound recognition functions have been developed that acquire user voices through sound acquisition devices such as microphones (hereinafter referred to as microphones (mic)) and based on the acquired voices.
[0003] Patent document 1 discloses a configuration in which an operation association function section is provided at the upper front of the refrigerator body, and the operation association function section contains the sound association function of the microphone, speaker, etc.
[0004] Patent Document 2 discloses a household appliance, such as a refrigerator, in which the sound acquisition unit is activated and acquires sound when a predefined phenomenon that triggers the activation of the sound acquisition unit is detected, and the user is notified that the sound acquisition unit is in an operational state. The aforementioned phenomenon includes the refrigerator door being open and a predetermined button on the refrigerator being operated. Furthermore, the notification of the sound acquisition unit's operational status is achieved through sound output and the illumination of a notification light. The notification light is located on a control panel located on the front of the refrigerator's front door. With the above configuration, the refrigerator described in Patent Document 2 can alleviate user anxieties regarding the sound acquisition function, such as whether the user's voice is accurately acquired and whether the sound acquisition function is deactivated when sound acquisition is not desired.
[0005] Existing technical documents
[0006] Patent Literature
[0007] Patent Document 1: Japanese Patent No. 3904390.
[0008] Patent document 2: Japanese Patent Application Publication No. 2017-4231. Summary of the Invention
[0009] Technical problem to be solved by the invention
[0010] However, in the configuration described in Patent Document 2, sound recognition continues even when the refrigerator door is open. In this case, although the user is notified to perform sound recognition, stopping the sound recognition requires user intervention.
[0011] In other words, the voice recognition process is necessary each time the user does not want to stop the voice recognition process, but it becomes unnecessary if the user forgets to stop the voice recognition process.
[0012] Therefore, one aspect of the present invention is to provide a refrigerator capable of recognizing a user's voice at an appropriate time.
[0013] Solution for solving the problem
[0014] To address the aforementioned issues, the present invention provides a refrigerator comprising: a sound acquisition unit; a door opening / closing detection unit for detecting the opening and closing of the refrigerator door; and a sound recognition control unit that, when the door opening / closing detection unit detects that the door is open, determines whether the sound data acquired by the sound acquisition unit meets predetermined sound recognition start conditions. If the sound data meets the sound recognition start conditions, the sound recognition unit initiates sound recognition of the sound data. Conversely, if the sound data does not meet the sound recognition start conditions, the sound recognition unit does not initiate sound recognition of the sound data.
[0015] Invention Effects
[0016] According to one aspect of the present invention, it is possible to recognize the user's speech at an appropriate time. Attached Figure Description
[0017] Figure 1 This is an explanatory diagram showing the configuration of a voice dialogue system for a refrigerator that includes an embodiment of the present invention.
[0018] Figure 2 Yes Figure 1 The diagram shows a block diagram of the refrigerator's control system.
[0019] Figure 3 Yes Figure 1 The diagram shown is a block diagram of the control system of the server.
[0020] Figure 4 Yes Figure 1 The image shown is a three-dimensional view of the refrigerator's exterior.
[0021] Figure 5 It means to remove. Figure 4 The diagram shows a perspective view of the upper part of the refrigerator with the cover component in its position.
[0022] Figure 6 It means and Figure 4 The circuit diagram shown illustrates how the microphone is turned on and off by the connection between the door switch and the open / closed switch of the refrigerator.
[0023] Figure 7 Yes Figure 1 The flowchart shown is for the operation of the refrigerator in the voice dialogue system.
[0024] Figure 8 It means focusing on Figure 7 The flowchart illustrates the relationship between the opening and closing of the door and the necessity of sound recognition in the operation of the refrigerator.
[0025] Figure 9 This refers to a refrigerator with regard to other embodiments of the present invention. Figure 7 The flowchart illustrates the relationship between the opening and closing of the door and the necessity of sound recognition in the actions shown.
[0026] Figure 10 This is a flowchart illustrating the operation of the alarm control unit of a refrigerator in another embodiment of the present invention.
[0027] Figure 11 This is a flowchart illustrating the operation of the alarm control unit of a refrigerator in another embodiment of the present invention.
[0028] Figure 12 This is a perspective view showing the appearance of a refrigerator according to another embodiment of the present invention.
[0029] Figure 13 It means to remove. Figure 12 The diagram shows a perspective view of the upper part of the refrigerator with the cover component in its position.
[0030] Figure 14 (a) means Figure 5 The diagram shows a longitudinal cross-section of the microphone mounting section and its position relative to the first door 61. Figure 14 (b) is viewed from below. Figure 14 The lower cover of (a) is shown in the diagram below.
[0031] Figure 15 (a) means Figure 14 (a) is a longitudinal sectional view showing the construction of another example of a microphone mounting part and the positional relationship of the microphone mounting part with the first door. Figure 15 (b) is viewed from below. Figure 15 The lower cover of (a) is shown in the diagram below. Detailed Implementation
[0032] [Implementation Method 1]
[0033] Hereinafter, a detailed description will be given of one embodiment of the present invention.
[0034] (The composition of voice dialogue system 1)
[0035] Figure 1 This is an explanatory diagram showing the configuration of the voice dialogue system 1 of the refrigerator 13 including this embodiment. (See diagram below.) Figure 1As shown, in the voice dialogue system 1, the server 11 is connected to the refrigerator 13 installed in the user's residence 12 via the communication network 15.
[0036] Refrigerator 13 is a network appliance with wireless communication capabilities for connecting to communication network 15 and communicating with server 11. The wireless communication capability can be built into the refrigerator 13 itself or it can be provided in a communication adapter that is externally attached to the refrigerator 13.
[0037] The user's residence 12 is equipped with a wireless LAN (Wireless Local Area Network) that is part of the communication network 15. The wireless LAN's wireless access point 14 is connected to the communication network 15, which includes the Internet. The wireless access point 14 is, for example, a communication device such as a WiFi router or WiFi access point. In this embodiment, although the communication network 15 is exemplified as including the Internet, telephone line networks, mobile communication networks, CATV (Cable TeleVision) communication networks, satellite communication networks, etc., can also be used.
[0038] Refrigerator 13 has a voice dialogue function that acquires the user's spoken voice and responds to the acquired voice. In this case, refrigerator 13 sends the acquired voice data to server 11. Server 11 performs voice recognition on the voice data received from refrigerator 13 to determine the content of the spoken voice, and sends the response voice data based on the determined content back to refrigerator 13. The response voice data is the voice data acquired from refrigerator 13, such as voice data answering a question from the user, or voice data indicating the establishment of a dialogue based on the user's spoken content.
[0039] Alternatively, the refrigerator 13 can function as a voice recognition device, using voice recognition to determine the content of the speech, sending the determined content to the server 11, and receiving the response voice data from the server 11. Alternatively, the refrigerator 13 can be used instead of the server 11 to perform voice recognition and prepare the response voice data.
[0040] (Overall structure of refrigerator 13)
[0041] Figure 2 This is a block diagram showing the configuration of the control system of refrigerator 13. (Example) Figure 2As shown, the refrigerator 13 includes a control unit 21, a storage unit 22, a communication unit 23, a sound output unit 24, a sound acquisition unit 25, a door switch (door opening / closing detection unit) 26, an interior light 27, an alarm buzzer (warning unit) 28, and an operation panel 29. The control unit 21 includes a sound output control unit 31, a sound recognition control unit 33, a light control unit 34, an alarm control unit (warning control unit) 35, and an operation panel control unit 36.
[0042] The control unit 21 is composed of a microcomputer, such as a CPU (Central Processing Unit) or a dedicated processor, and controls the operation of each part of the refrigerator 13.
[0043] The storage unit 22 stores various information about the use of the refrigerator 13, as well as audio data of responses to voice dialogues downloaded from the server 11. The communication unit 23, under the control of the control unit 21, communicates with the server 11 via the communication network 15.
[0044] The sound output unit 24 has a speaker and outputs sound based on sound data under the control of the sound output control unit 31. The sound acquisition unit 25 has a microphone 81 (see reference). Figure 5 The sound is acquired and then converted into an electrical signal.
[0045] Door switch 26 detects the open / closed state of the refrigerator door 13, for example, it becomes conductive when the door is open and disconnected when the door is closed. Interior light 27 illuminates when the refrigerator door 13 is open. Alarm buzzer 28 is a buzzer that warns the user when the refrigerator door 13 is open.
[0046] The control panel 29 is a user interface with a display unit, used for controlling the control unit 36, allowing the user to input instructions to the refrigerator 13 and notifying the user of the refrigerator 13's status (internal temperature, set temperature, etc.) via the display unit. In this embodiment, the control panel 29 is located inside the refrigerator 13.
[0047] The voice output control unit 31 controls the voice output unit 24. Specifically, when voice data to be spoken is available, the voice output control unit 31 causes the voice data to be spoken through the voice output unit 24. The voice data to be spoken includes voice data received from the server 11, voice data stored in the storage unit 22, etc.
[0048] The voice recognition control unit 33 initiates voice recognition by sending the user's voice data to the server 11 when the user's speech, as indicated by the voice data acquired by the voice acquisition unit 25, meets the voice recognition start conditions of the server 11. After the voice recognition on the server 11 begins, the voice recognition control unit 33 sequentially sends the user's voice data acquired by the voice acquisition unit 25 to the server 11.
[0049] On the other hand, if the user's speech, as indicated by the voice data acquired by the voice acquisition unit 25, does not meet the conditions for starting voice recognition on the server 11, the voice recognition control unit 33 will not send the user's voice data to the server 11. Therefore, voice recognition on the server 11 will not begin.
[0050] Furthermore, if the voice recognition control unit 33 receives voice data from the voice acquisition unit 25 indicating that the user's speech meets the voice recognition termination condition on the server 11, it stops sending voice data to the server 11. Alternatively, if the voice recognition control unit 33 receives a termination instruction from the user (e.g., through user operation input on the operation panel 29) on the server 11, it stops sending voice data to the server 11. In this way, voice recognition on the server 11 terminates.
[0051] The conditions for initiating voice recognition include, for example, an upward inflection at the end of the user's speech, meaning the user's speech is in the form of a question, or the user's speech contains a predetermined starting keyword. An example of an interrogative sentence is, "I bought ○○ beforehand, where did I put it?" A starting keyword is, for example, "Huh?" or "Where?" Furthermore, both these interrogative sentences and starting keywords indicate the start of a conversation between the user and refrigerator 13.
[0052] The voice recognition termination condition is when the user's speech contains a predetermined termination keyword. Termination keywords include, for example, statements like "stop" or "no." Such termination keywords, for instance, indicate the end of the user's conversation with refrigerator 13. Furthermore, the voice recognition termination indication refers to the operation of a key set for the termination of voice recognition on the control panel 29.
[0053] To perform the aforementioned actions, the voice recognition control unit 33 has a frequency analysis function that detects the rising intonation at the end of the user's voice data. Furthermore, it has a voice recognition function that detects whether the user's voice data packet contains a start keyword and an end keyword. Additionally, the end keyword can also be detected through the voice recognition processing on the server 11.
[0054] While the sound recognition control unit 33 is sending the sound data acquired by the sound acquisition unit 25 to the server 11, that is, while the server 11 is performing sound recognition on the sound data acquired by the sound acquisition unit 25, the lighting control unit 34 controls the library light 27 to enter a different lighting state than the normal lighting state. In this way, the lighting control unit 34 informs the user that sound recognition is in progress regarding the user's voice.
[0055] Furthermore, lighting states that differ from normal lighting states include, for example, a state in which the lamp 27 periodically alternates between being bright (bright state) and dark (dark state), or a state in which the lamp 27 flashes. Also, when the lighting control unit 34 can change the lighting color of the lamp 27, it can sequentially change the lighting color while the sound recognition control unit 33 is sending sound data to the server 11.
[0056] The alarm control unit 35 detects the open / closed state of the refrigerator 13 door by switching the door switch 26 on and off. If the door remains open for a predetermined time after the detection point, the alarm buzzer 28 will be activated to warn the user.
[0057] Furthermore, the alarm control unit 35 activates the alarm buzzer 28 to inform the user of the situation when the transmission of user audio data to the server 11 begins or ends.
[0058] The control panel 36 accepts input from the user on the control panel 29, and displays the status of the refrigerator 13 (internal temperature, set temperature, etc.) on the display panel of the control panel 29.
[0059] (The composition of server 11)
[0060] Figure 3 This is a block diagram showing the structure of the control system of server 11. Server 11 manages refrigerator 13, as follows: Figure 3 As shown, it includes a control unit 41, a storage unit 42, a communication unit 43, and a voice recognition unit 44. The control unit 41 includes a voice data providing unit 51.
[0061] The control unit 21, for example, includes a microcomputer consisting of an arithmetic processing unit such as a CPU (Central Processing Unit) and a dedicated processor, which controls the operation of each part of the server 11.
[0062] The storage unit 42 stores various information used by the server 11, and also stores various article examples, such as the article examples referred to by the sound data providing unit 51 when creating the sound data provided to the refrigerator 13. The communication unit 43 communicates with the refrigerator 13 via the communication network 15 under the control of the control unit 41.
[0063] The voice recognition unit 44 performs voice recognition on the voice data sent from the refrigerator 13 and received by the communication unit 43, and converts it into text. The voice recognition unit 44 not only has the function of converting voice data into text, but also includes the function of converting it into meaningful text.
[0064] The voice data providing unit 51 provides the answering voice data after voice recognition to the refrigerator 13. In this case, the voice data providing unit 51 generates the answering voice data with reference to the storage unit 42, or obtains the answering voice data via the communication network 15.
[0065] (A summary of the structure of refrigerator 13)
[0066] Figure 4 This is a perspective view showing the appearance of refrigerator 13. Figure 5 It means to remove. Figure 4 A perspective view of the upper part of the refrigerator 13 with the cover component (part) 71 shown in the state of the cover. Figure 6 This is a circuit diagram showing how the microphone 81 is turned on and off in conjunction with the opening and closing of the door switch 26 (the opening and closing of the first door 61).
[0067] like Figure 4 As shown, the refrigerator 13 has doors 61-65 from top to bottom, for example, from the first to the fifth door. The first door 61 is the refrigerator compartment door, the second door 62 is the freezer compartment door, the third door 63 is the ice maker compartment door, the fourth door 64 is the freezer compartment door, and the fifth door 65 is the vegetable compartment door. The first door 61 is a left-right opening type. The control panel 29 is, for example, located on the open end side of the larger of the two doors of the first door 61.
[0068] The refrigerator 13 has a lid component 71 on top. Inside the lid component 71, there is a hinge that can be opened and closed to support the first door 61, a door switch 26 that detects the opening and closing of the first door 61, and a microphone 81 provided by the sound acquisition unit 25.
[0069] like Figure 5 As shown, the microphone 81 is mounted near the microphone mounting portion 72 at the front end of the lower cover 71a of the cover member 71, in the width direction of the refrigerator 13, with the sound input portion facing downwards. The microphone mounting portion 72 has a generally cylindrical microphone holding portion 72a, and a through hole is formed in the portion of the lower cover 71a below the microphone 81.
[0070] like Figure 6As shown, the door switch 26 becomes conductive when the first door 61 is open and disconnected when the first door 61 is closed. The microphone 81 is connected to the power supply 82 in conjunction with the conductive action of the door switch 26, thus becoming conductive. In this way, the microphone 81 can be easily turned on and off as needed.
[0071] (Refrigerator 13's actions)
[0072] In the above configuration, the operation of the refrigerator 13 of the voice dialogue system 1 will be explained below. Figure 7 This is a flowchart illustrating the operation of the refrigerator 13 of the voice dialogue system 1. Here, the operation of detecting the opening and closing of the first door 61 by switching the door switch 26 on and off will be explained. Furthermore, the first door 61 is simply represented as a door.
[0073] like Figure 7 As shown, if the voice dialogue system 1 starts operating, the control unit 21 monitors the on / off state of the door switch 26, that is, the open / closed state of the door. The voice recognition control unit 33 of the control unit 21, if the door is in the open state (door switch 26 is on) (S11), acquires the user's voice through the microphone 81 of the voice acquisition unit 25 and determines whether to start voice recognition (S12). In this determination, the voice recognition control unit 33 determines whether the user's speech meets the aforementioned voice recognition start conditions.
[0074] If the result of the judgment in S12 is yes, the voice recognition control unit 33 will send the user's voice data obtained by the microphone 81 to the server 11 (S13).
[0075] Then, the control unit 21 notifies the user that the user's voice is being identified (S14). Therefore, during the period when the lighting control unit 34 sends the user's voice data to the server 11, it controls the light in the storage room 27 to be in a different lighting state than usual. Specifically, it controls the light in the storage room 27 to be in a state of periodically alternating between a bright and dark state, or to a state where the light in the storage room 27 flashes. Alternatively, if the lighting control unit 34 can change the lighting color of the light in the storage room 27, it sequentially changes the lighting color during the period when it sends the user's voice data to the server 11. Furthermore, the alarm control unit 35 activates the alarm buzzer 28 when it begins sending the user's voice data to the server 11. In addition, it is sufficient to perform at least one of the above-mentioned notification actions of the lighting control unit 34 and the alarm control unit 35.
[0076] Subsequently, the voice recognition control unit 33 determines whether to terminate voice recognition (S15). This determination involves whether the user's voice meets the aforementioned voice recognition termination conditions or whether there is a termination instruction from the user's voice recognition. If the determination result of S15 is yes, proceed to S20 to terminate the transmission of the user's voice data to the server 11. In S20, the alarm control unit 35 activates the alarm buzzer 28 to notify the user that the transmission of the user's voice data to the server 11 has ended. At this time, the alarm buzzer 28 is set to have a different timbre and pitch than when it announced the start of the transmission of the user's voice data to the server 11 (S14). This allows the user to recognize the start and end of the transmission of voice data to the server 11, alleviating any anxiety regarding the operational status of the voice recognition process.
[0077] Next, if the determination result of S15 is negative and the sound output control unit 31 receives the answer sound data from the server 11 (S16), it outputs the sound data as sound by playing it through the sound output unit 24 (S17).
[0078] Subsequently, if the door becomes closed (door switch 26 is turned off) (S18), and the closed state of the door lasts for a predetermined time (e.g., 10-20 seconds) or more (after a predetermined time has elapsed from the detection point of the closed state of the door) (S19), the voice recognition control unit 33 terminates the transmission of the user's voice data to the server 11 (S20).
[0079] Furthermore, if the result of the determination in S12 is negative, and the door remains open for a predetermined time (e.g., 10 seconds) or more (S21), then the sound output control unit 31 will initiate a conversation with the user via the sound output unit 24 (S22). The conversation may be something like "Is there anything you want to say?" or "How are you today?" and the sound data will be stored, for example, by the storage unit 22.
[0080] Next, if the user responds to the message from refrigerator 13 in S22 (S23), proceed to S13 and send the user's voice data obtained through microphone 81 to server 11. Then, the actions from S14 onwards are performed in the same manner.
[0081] On the other hand, if there is no response from the user in S23 and the door remains open for an extended period, the alarm control unit 35 determines that an alarm is needed for the door to remain open (S25) and activates the alarm buzzer 28 (S26). Subsequently, if the operation returns to S24 and the door remains open (door switch 26 is disconnected), the voice communication system 1, i.e., the refrigerator 13, will cease operation.
[0082] Furthermore, if in S18 the door is in the open state (door switch 26 is on), and the open state of the door is sufficient for an alarm to be triggered for an extended period, then the alarm control unit 35 determines that an alarm is required for the open door state (S27) and activates the alarm buzzer 28 (S28). Afterward, the operation returns to that in S13.
[0083] Furthermore, in S19, if the door is closed for less than a predetermined time, the voice recognition control unit 33 sends the user's voice data to the server 11 (S29), and the control unit 21 (light control unit 34, alarm control unit 35) informs the user about the voice recognition process (S30), returning to the operation in S19. Additionally, after the operations in S29 and S30, similar to S16 and S17, if the server 11 receives a response voice data, the voice data is played back through the voice output unit 24 and output as sound.
[0084] (The relationship between the opening and closing of the refrigerator door and the necessity of sound recognition)
[0085] Next, Yu Figure 8 Indicates focusing on Figure 7 The operation of the refrigerator 13 shown illustrates the relationship between the opening and closing of the door and the necessity of sound recognition. Figure 8 This is a flowchart illustrating the operation of refrigerator 13, which focuses on the relationship between the opening and closing of the door and the necessity of sound recognition.
[0086] Figure 8 The actions indicated by the numbers of each step S are due to... Figure 7 The actions indicated by the numbers S in each step are the same, therefore they are omitted. Figure 8 Description of the action.
[0087] (Advantages of Refrigerator 13)
[0088] When the refrigerator 13 door is open, sound data is acquired from the microphone 81. The sound data represents the user's speech. Sound recognition begins if the sound recognition start conditions are met, and does not begin if the sound recognition start conditions are not met. Furthermore, the sound recognition start conditions are that the user's speech is in the form of an interrogative question, or that the user's speech contains a predetermined start keyword, which indicates that the user's speech has begun a dialogue with the refrigerator 13.
[0089] Therefore, when a user wishes to perform voice recognition, or when it is possible to perform voice recognition, they can speak as if the voice recognition start conditions are met, thereby eliminating the need for operations such as manipulating the control panel 29 to indicate the start of voice recognition. Similarly, when a user does not wish to perform voice recognition, they do not speak as if the voice recognition start conditions are met, thereby eliminating the need for operations such as manipulating the control panel 29 to indicate the stop of voice recognition. In this way, the refrigerator 13 can improve convenience and perform voice recognition on the user's voice at the appropriate time.
[0090] Furthermore, in refrigerator 13, during voice recognition via server 11, the voice recognition ends when the user's speech, as shown in the voice data obtained from microphone 81, meets the voice recognition termination condition. Moreover, the voice recognition termination condition here is that the user's speech contains a predetermined termination keyword. This termination keyword is a speech indicating the end of the user's dialogue with refrigerator 13.
[0091] Therefore, when the sound recognition ends, the user does not need to perform any operations such as operating the control panel 29 to indicate the end of the sound recognition. This improves the convenience of the refrigerator 13.
[0092] Furthermore, the refrigerator 13 continues to perform voice recognition based on the user's voice data from the moment the door is closed until a predetermined time has elapsed. In this way, the refrigerator 13 can recognize the user's voice even when the user is speaking while closing the door, or when the user speaks immediately after closing the door, thereby improving convenience.
[0093] [Implementation Method Two]
[0094] Other embodiments of the present invention will be described below. Furthermore, for ease of explanation, components having the same function as those described in the above embodiments will be labeled with the same reference numerals, and their descriptions will not be repeated.
[0095] (The configuration of the sound recognition control unit 33 of the refrigerator 91)
[0096] In the refrigerator 13 of the aforementioned embodiment, the voice recognition control unit 33 initiates voice recognition by sending the user's voice data to the server 11 when the user's speech, as shown in the voice data acquired by the voice acquisition unit 25, meets the voice recognition start conditions of the server 11. Furthermore, the voice recognition start conditions in this case are either that the user's speech is in the interrogative form (with a rising intonation at the end) or that the user's speech contains a predetermined start keyword.
[0097] In this embodiment, the sound recognition start condition for the refrigerator 91 is set to the case where, with the door open, the volume of the sound data acquired by the microphone 81 of the sound acquisition unit 25 is above a predetermined threshold, and the sound data is identified as the user's voice. Therefore, the sound recognition control unit 33, when the volume of the sound data acquired by the microphone 81 is above the predetermined threshold and the sound data is identified as the user's voice, sends the user's voice data acquired by the microphone 81 to the server 11 to start sound recognition.
[0098] (Refrigerator 91's actions)
[0099] In the above configuration, the operation of the refrigerator 91 of the voice dialogue system 1 will be explained below.
[0100] Voice recognition control unit 33, in Figure 7 In the operation, if the door is in the open state (door switch 26 is turned on) (S11), it is determined whether to start sound recognition (S12). In this determination, the sound recognition control unit 33 determines whether the sound data acquired by the microphone 81 of the sound acquisition unit 25 meets the sound recognition start conditions. Specifically, in S12, if the volume of the sound data acquired by the microphone 81 is above a predetermined threshold, and the sound data is identified as the user's voice, the sound recognition control unit 33 determines that sound recognition needs to be started. If the result of the determination in S12 is yes, the sound recognition control unit 33 sends the sound data acquired by the microphone 81 to the server 11 (S13). Other operations of the refrigerator 91 are the same as in... Figure 7 The operation is the same as that of the refrigerator 13 already described.
[0101] (The relationship between the opening and closing of the refrigerator door and the necessity of sound recognition)
[0102] then, Figure 9 Indicates focusing on Figure 7 The operation of refrigerator 91 is explained, including the relationship between the opening and closing of the door and the necessity of sound recognition. Figure 9 This is a flowchart illustrating the operation of refrigerator 91, focusing on the relationship between door opening and closing and the necessity of sound recognition. Furthermore, Figure 9 In the actions indicated by the numbers of each step S, and with Figure 7 The actions of each step S have the same number, because they are related to the actions in... Figure 7 The actions already described are the same, therefore the description is omitted. Furthermore, the actions in S51 and S52 are the same as those described earlier. Figure 7 The action corresponding to the action of S12.
[0103] like Figure 9As shown, if the door is in the open state (door switch 26 is turned on) (S11), the voice recognition control unit 33 acquires sound data through the microphone 81 of the sound acquisition unit 25, and determines the necessity of starting voice recognition based on the acquired sound data (S12). In this case, if the volume of the sound data acquired by the microphone 81 is above a predetermined threshold (S51), and the sound data is identified as the user's voice (S52), the voice recognition control unit 33 sends the user's voice data to the server 11 to start voice recognition (S13).
[0104] Furthermore, in S51, if the volume shown by the sound data obtained by the microphone 81 is less than the predetermined threshold, and in S52, if the sound data cannot be identified as the user's voice, the sound recognition control unit 33 does nothing.
[0105] Furthermore, in S13, if the user's voice is sent to the server 11 to start voice recognition, the user's voice data is sent to the refrigerator 91 to provide an appropriate response to the user from the server 11, and the refrigerator 91 can also play and output the voice data.
[0106] (Advantages of Refrigerator 91)
[0107] In refrigerator 91, when the door of refrigerator 91 is open, similar to refrigerator 13, sound data is acquired from microphone 81. The sound data represents the user's speech, and sound recognition begins if the sound recognition start condition is met; otherwise, sound recognition does not begin. Furthermore, the sound recognition start condition here is that the volume of the sound data acquired from microphone 81 is above a predetermined threshold, and the sound data is recognized as the user's voice.
[0108] Therefore, when a user wishes to perform voice recognition, they can speak at a volume that meets the start conditions for voice recognition, i.e., at a volume above a predetermined threshold as indicated by the voice data. This eliminates the need for operations such as those performed on the control panel 29 to indicate the start of voice recognition. Similarly, when a user does not wish to perform voice recognition, they do not speak at the aforementioned level to meet the start conditions. This eliminates the need for operations such as those performed on the control panel 29 to indicate the stop of voice recognition. In this way, the refrigerator 91 improves convenience and can perform voice recognition on the user's voice at appropriate times.
[0109] [Implementation Method 3]
[0110] Other embodiments of the present invention will be described below. Furthermore, for ease of explanation, components having the same function as those described in the above embodiments will be labeled with the same reference numerals, and their descriptions will not be repeated.
[0111] Figure 10 This is a flowchart illustrating the operation of the alarm control unit 35 of the refrigerator 92 in the voice dialogue system 1 of this embodiment. If the door is open and the alarm is required for a certain period of time, it is determined that an alarm for the door being open is needed (S25, S27). Figure 7 The alarm control unit 35 of the refrigerator 13 shown only activates the alarm buzzer 28 (S26, S28).
[0112] In contrast, the alarm control unit 35 of the refrigerator 92 in this embodiment, when determining that an alarm is needed indicating that the door is open, will activate... Figure 10 The action. That is, the alarm control unit 35 of the refrigerator 92, such as Figure 10 As shown, the door becomes open (S61). For example, if the door remains open for more than 2 minutes (above a predetermined time), an alarm is triggered indicating that the door needs to be open (S62).
[0113] In this situation, if the volume indicated by the sound data obtained by the microphone 81 is below the threshold (S63), the alarm control unit 35 activates the alarm buzzer 28 (S64) to trigger an alarm indicating that the door is open. Then, it proceeds to step S66. This alarm activation by the alarm control unit 35 is related to... Figure 7 The actions of S26 and S28 shown are the same, which are normal (normal) alarm actions, such as causing the alarm buzzer 28 to sound for a long time at short intervals.
[0114] On the other hand, in S63, if the volume indicated by the sound data obtained by the microphone 81 is above a threshold, the alarm control unit 35 will issue an alarm with a short warning tone (S65). Afterwards, it proceeds to the action in S66. This alarm action of the alarm control unit 35 differs from the normal alarm action in S64; for example, it causes the alarm buzzer 28 to sound for short periods at long intervals.
[0115] In this case, if the volume indicated by the sound data acquired by the microphone 81 is above a threshold, that is, if it is determined that the user is speaking with the refrigerator 92 door open, the alarm control unit 35 will, for example, cause the alarm buzzer 28 to sound briefly with a long interval. Thus, the control of the alarm buzzer 28 by the alarm control unit 35 in S65 is different from the normal control of the alarm buzzer 28 by the alarm control unit 35 in S64, and the alarm will not interfere with the user's speech or the acquisition of the user's sound data by the microphone 81.
[0116] Furthermore, in addition to the controls described above, the control of the alarm control unit 35 on the alarm buzzer 28 in S65 can also include controls that only lengthen the interval between alarm buzzer 28's actions (lengthen the silence time), controls that only decrease the volume of the alarm buzzer 28, or controls that lengthen the interval between alarm buzzer 28's actions to sound for a short time and at a low volume. Alternatively, the alarm control unit 35 can replace the alarm action of the alarm buzzer 28 in S65 by issuing a warning to the user that the door is open through a statement from the sound output unit 24. Furthermore, the refrigerator 93, as described later, could also be configured not to perform an alarm action.
[0117] Such control, for example, in the case where the alarm control unit 35 performs alarm control in multiple stages from weak alarm intensity to strong alarm intensity, is equivalent to control of weak alarm intensity.
[0118] (Advantages of refrigerator #92)
[0119] When refrigerator 92 determines that an alarm is needed because the door has been open for a predetermined time, and the volume indicated by the sound data obtained from microphone 81 is below a predetermined threshold, it determines that the user has not spoken and performs the normal alarm action. On the other hand, similarly, when refrigerator 92 determines that an alarm is needed because the door is open, that is, when the volume indicated by the sound data obtained from microphone 81 is above a predetermined threshold, it determines that the user has spoken, and performs an alarm action different from the normal alarm action in a way that will not be considered as the user speaking or interfere with the acquisition of the user's sound data from microphone 81.
[0120] Furthermore, in this embodiment, the refrigerator 92 (sound recognition control unit 33) controls different alarm actions depending on whether the volume of the sound data obtained from the microphone 81 is below a predetermined threshold or above the predetermined threshold. However, instead of the case where the volume of the sound data obtained from the microphone 81 is below the predetermined threshold, the refrigerator 92 (sound recognition control unit 33) can also be configured to control different alarm actions depending on whether the volume of the sound data obtained from the microphone 81 is above the predetermined threshold or whether the volume of the sound data obtained from the microphone 81 is above the predetermined threshold.
[0121] Therefore, even if the refrigerator 92 activates the door open alarm when the door remains open for a predetermined time, the alarm will not interfere with the user's speech or the acquisition of the user's voice data by the microphone 81.
[0122] [Implementation Method Four]
[0123] Other embodiments of the present invention will be described below. Furthermore, for ease of explanation, components having the same function as those described in the above embodiments will be labeled with the same reference numerals, and their descriptions will not be repeated.
[0124] Figure 11 This is a flowchart illustrating the operation of the alarm control unit 35 of the refrigerator 93 in the voice dialogue system 1 of this embodiment. The alarm control unit 35 of the refrigerator 93 in this embodiment replaces... Figure 7 The actions of S25 (S27) and S26 (S28) become the basis for performing Figure 11 The actions of S72 and S73.
[0125] In other words, the alarm control unit 35 of refrigerator 93, such as Figure 11 As shown, if the door is in the open state (S71) and there is no sound input to the microphone 81 for a certain period of time (S72), it is determined that an alarm for the door being open is required. In this case, the alarm control unit 35 activates the alarm buzzer 28 (S73), thereby triggering an alarm for the door being open. Then, the process proceeds to S74. This alarm activation by the alarm control unit 35, along with... Figure 7 The actions of S26 and S28 shown are the same, which are normal alarm actions.
[0126] On the other hand, in S27, if the microphone 81 continues to receive sound input for a certain period of time, the alarm control unit 35 will not trigger the alarm for the door being open (so that the alarm buzzer 28 does not activate), and will proceed to S74.
[0127] In this way, if the alarm control unit 35 receives sound input from the microphone 81 for a certain period of time, that is, if the user continues to speak with the refrigerator 92 door open for a certain period of time, then the alarm will not activate (the alarm buzzer 28 will not sound). With this control of the alarm control unit 35, the alarm will not interfere with the user's speech or the acquisition of the user's voice data from the microphone 81.
[0128] Furthermore, in S72, if the alarm control unit 35 receives sound input from the microphone 81 for a certain period of time, it replaces the configuration of not triggering an alarm for the door open state, and performs alarm control unit 35 of the refrigerator 92 (e.g., Figure 10 The same action as the action in S65 can also be constructed.
[0129] (Advantages of the Refrigerator 93)
[0130] If the refrigerator 93 determines that an alarm is needed when the door is open for a predetermined time (i.e., if there is no sound input from the microphone 81 for a certain period of time), it will perform the normal alarm action since the user has not spoken. On the other hand, if the refrigerator 93 determines that an alarm is needed when the door is open (i.e., if there is sound input from the microphone 81 for a certain period of time), it will determine that the user has spoken and will not perform an alarm action.
[0131] Furthermore, in this embodiment, the refrigerator 93 (sound recognition control unit 33) represents a control method that switches between the presence and absence of alarm action, depending on whether there is no sound input from the microphone 81 for a certain period of time or whether there is sound input. However, the refrigerator 93 (sound recognition control unit 33) can also be configured to switch between the presence and absence of alarm action, as described above, instead of the case where there is no sound input from the microphone 81 for a certain period of time, where the sound acquisition unit 25 acquires sound and the server 11 does not perform sound recognition, and instead of the case where there is sound input from the microphone 81 for a certain period of time, where the sound acquisition unit 25 acquires sound and the server 11 performs sound recognition.
[0132] Therefore, when the refrigerator 93 remains in the open position for a predetermined time, even if an alarm is triggered for the open position, it will not interfere with the user's speech or the acquisition of the user's voice data by the microphone 81.
[0133] [Implementation Method Five]
[0134] In the above implementation, the case of opening and closing the first door 61 is used as an example to describe the control of sound recognition action, alarm action, etc. However, the configuration of the above implementation is not limited to the case of opening and closing the first door 61, and can also be applied to the case of opening and closing other doors 62 to 65.
[0135] [Implementation Method Six]
[0136] Other embodiments of the present invention will be described below. Furthermore, for ease of explanation, components having the same function as those described in the above embodiments will be labeled with the same reference numerals, and their descriptions will not be repeated.
[0137] Figure 12 This is a perspective view showing the appearance of the refrigerators 91-93 according to this embodiment. Figure 13 It means to remove. Figure 12 A perspective view of the upper part of the refrigerator 13 with the cover component 71 in the state shown.
[0138] In the above embodiments, the refrigerators 91-93 are described as having a first door 61 with two doors that open to the left and right. However, refrigerators 91-93, as Figure 12 As shown, it can also be in the form of a door 61 that can be opened to the right and left.
[0139] In this situation, such as Figure 13 As shown, with Figure 5 In the same case, the microphone 81 is mounted facing downwards in the microphone mounting part 72 of the lower cover 71a of the cover component 71.
[0140] [Implementation Method Seven]
[0141] Figure 14 (a) means Figure 5 The longitudinal cross-sectional view showing the structure of the microphone mounting part 72 and its positional relationship with the first door 61 is shown. Figure 14 (b) is viewed from below. Figure 14 The lower part of the cover 71a is shown in the diagram below, which illustrates the construction of (a).
[0142] Figure 15 (a) means Figure 14 (a) is a longitudinal sectional view showing the construction of another example of the microphone mounting portion 72 and its positional relationship with the first door 61. Figure 15 (b) is viewed from below. Figure 15 The lower part of the cover 71a is shown in the diagram below, which illustrates the construction of (a).
[0143] like Figure 14 As shown in (a) and (b), the microphone mounting portion 72 has a microphone holding portion 72a for holding the microphone 81, a through hole 72b through the lower cover 71a, and a cone-shaped portion 72c in which the area around the through hole 72b on the lower side of the lower cover 71a is machined into a downwardly expanding cone shape. The microphone mounting portion 72 has a cone-shaped portion 72c, thereby improving sound collection performance.
[0144] Furthermore, the positional relationship between the microphone mounting section 72 and the first door 61 is as follows: Figure 14As shown in (a), the cone-shaped portion 72c may be completely blocked from the top of the first door 61 (first door 61a) when the first door 61 is closed. Alternatively, when the first door 61 is closed, the cone-shaped portion 72c may not be completely blocked from the top of the first door 61 (first door 61b), but rather a portion of the cone-shaped portion 72c may protrude outward from the first door 61 (first door 61b). In the latter case, even when the first door 61 (first door 61b) is closed, the microphone 81 can easily pick up external sounds. Therefore, the latter configuration is suitable for controlling sound recognition operations, alarm operations, etc., in conjunction with the opening and closing of the second to fifth doors 65 other than the first door 61.
[0145] Also, such as Figure 14 The microphone mounting part 72 shown in (a) and (b) can also be constructed as follows: Figure 15 The constructions shown in (a) and (b) are as follows. Figure 15 The microphone mounting part 72 shown in (a) and (b) and Figure 14 The microphone mounting part 72 and the cone-shaped part 72c shown in (a) and (b) have different shapes.
[0146] 〔Summarize〕
[0147] The refrigerator of the present invention includes: a sound acquisition unit; a door opening / closing detection unit for detecting the opening and closing of the refrigerator door; and a sound recognition control unit that, when the door opening / closing detection unit detects that the door is open, determines whether the sound data acquired by the sound acquisition unit meets a predetermined sound recognition start condition. If the sound data meets the sound recognition start condition, the sound recognition unit performs sound recognition of the sound data. On the other hand, if the sound data does not meet the sound recognition start condition, the sound recognition unit does not perform sound recognition of the sound data.
[0148] The refrigerator of the second embodiment of the present invention may also be configured as follows, in the first embodiment described above: the sound recognition control unit, when performing sound recognition in the sound recognition unit, determines whether the sound data acquired by the sound acquisition meets a predetermined sound recognition termination condition, and if the sound data meets the sound recognition termination condition, causes the sound recognition unit to terminate the sound recognition of the sound data.
[0149] The refrigerator of the third embodiment of the present invention may also be configured as follows, in either embodiment one or two: when the door opening / closing detection unit detects that the door is closed, the sound recognition control unit continues to perform sound recognition of the sound data of the sound recognition unit for a predetermined time starting from the detection time of the closed state of the door, and after a predetermined time has elapsed from the detection time of the closed state of the door, the sound recognition of the sound data of the sound recognition unit ends.
[0150] The refrigerator of the fourth embodiment of the present invention, in any of the above embodiments one to three, may also be configured as follows: a warning unit for issuing a warning to the user; a warning control unit for activating the warning unit when the door is in the open state and the warning is required for an extended period; the warning control unit controls the warning unit to perform a normal warning action when the sound acquisition unit acquires a sound and the sound recognition unit does not perform sound recognition, and on the other hand, controls the warning unit to perform an action different from the normal action when the sound acquisition unit acquires a sound and the sound recognition unit performs sound recognition.
[0151] The refrigerator of embodiment five of the present invention, in any of embodiments one to three above, may also be configured as follows: a warning unit for issuing a warning to the user; a warning control unit for activating the warning unit when the door is in the open state and the warning is required for an extended period; the warning control unit controls the warning unit to perform a normal warning action when the volume indicated by the sound data obtained by the sound acquisition unit is less than a predetermined threshold, and controls the warning unit to perform an action different from the normal action when the volume indicated by the sound data obtained by the sound acquisition unit is greater than or equal to the predetermined threshold.
[0152] The refrigerator of the sixth embodiment of the present invention, in any of the embodiments one to three above, may also be configured as follows: a warning unit for issuing a warning to the user; a warning control unit for activating the warning unit when the door is in the open state and the warning is required for a continuous period of time; the warning control unit controls the warning unit to perform a normal warning action when there is no sound input from the sound acquisition unit for a certain period of time, and on the other hand, controls the warning unit to perform an action different from the normal action when there is sound input from the sound acquisition unit for a certain period of time.
[0153] The refrigerator of embodiment seven of the present invention may be configured in any of embodiments one to six as follows: the sound acquisition unit has a microphone; the microphone is mounted with the sound input unit facing downward in the through hole of a component having a vertical through hole on the top of the refrigerator; and the area around the through hole on the bottom of the component is formed into a downwardly extending cone shape.
[0154] The present invention provides a voice dialogue system of type eight, comprising: a refrigerator of any one of types one to seven above; and a server, including the voice recognition unit, wherein the voice recognition unit performs voice recognition on the voice data received from the refrigerator to determine the content of the speech, and replies the voice data of the response based on the determined content to the refrigerator.
[0155] This invention is not limited to the embodiments described above, and various modifications can be made within the scope indicated in the claims. Embodiments obtained by appropriately combining the technical methods disclosed in different embodiments are also included within the scope of this invention. Furthermore, combining the technical methods disclosed in each embodiment can form new technical features.
[0156] Explanation of reference numerals in the attached figures
[0157] 1. Voice Dialogue System
[0158] 11 servers
[0159] 12 User Residences
[0160] 13. Refrigerators 91-93
[0161] 15. Communication Networks
[0162] 21 Control Department
[0163] 22 Storage Department
[0164] 24. Sound Output Section
[0165] 25 Sound Acquisition Department
[0166] 26. Door switch (door opening / closing detection unit)
[0167] 27 Warehouse Lights
[0168] 28. Alarm buzzer (warning unit)
[0169] 29 Control Panel
[0170] 31 Sound Output Control Unit
[0171] 33 Voice Recognition and Control Unit
[0172] 34 Lighting Control Unit
[0173] 35. Alarm Control Department (Warning Control Department)
[0174] 36. Control Panel
[0175] 41 Control Department
[0176] 42 Storage Section
[0177] 44. Voice Recognition Department
[0178] 51. Sound Data Provision Department
[0179] 61-65 First Gate - Fifth Gate
[0180] 61a, 61b First Gate
[0181] 71. Cover component (part)
[0182] 71a lower cover
[0183] 72 Microphone mounting section
[0184] 72a Microphone Holder
[0185] 72b Through Hole
[0186] 72c Conical part
[0187] 81 microphones
Claims
1. A refrigerator, characterized in that: include: Sound Acquisition Department; A door opening and closing detection unit, detecting the opening and closing of the refrigerator door; and The sound recognition control unit determines whether the sound data acquired by the sound acquisition unit satisfies a predetermined sound recognition start condition when the door opening and closing detection unit detects the open state of the door. If the sound data satisfies the sound recognition start condition, the sound recognition unit performs sound recognition of the sound data. If the sound data does not satisfy the sound recognition start condition, the sound recognition unit does not perform sound recognition of the sound data. The predetermined voice recognition start condition includes at least one of the following (i) to (iii), (i) The user's utterance is in the form of a question; (ii) the user's utterance contains a predetermined start keyword indicating that the user intends to start a conversation with the refrigerator; (iii) The volume of the sound represented by the sound data acquired by the sound acquisition unit is greater than or equal to a predetermined threshold, and the sound data is sound data of the user's voice.
2. The refrigerator according to claim 1, characterized in that The voice recognition control unit determines whether the voice data acquired by the voice acquisition unit satisfies a predetermined voice recognition end condition when the voice recognition unit performs voice recognition, and causes the voice recognition unit to end voice recognition of the voice data when the voice data satisfies the voice recognition end condition. The predetermined voice recognition termination condition is that the user's voice contains a predetermined termination keyword indicating that the user intends to end the conversation with the refrigerator.
3. The refrigerator according to claim 1, characterized in that: When the door opening and closing detection unit detects the closed state of the door, the sound recognition control unit continues the sound recognition of the sound data by the sound recognition unit for a predetermined time from the detection point of the closed state of the door, and ends the sound recognition of the sound data by the sound recognition unit after the predetermined time has passed from the detection point of the closed state of the door.
4. The refrigerator according to any one of claims 1 to 3, characterized in that: include: The warning department is responsible for issuing warnings to users; and a warning control unit that activates the warning unit when the door is in an open state for a period of time requiring a warning; in The warning control unit controls the warning unit in a manner that performs a normal warning action when the sound acquisition unit acquires sound and the sound recognition unit does not perform sound recognition. On the other hand, when the sound acquisition unit acquires sound and the sound recognition unit performs sound recognition, the warning control unit controls the warning unit in a manner that performs an action different from the normal action.
5. The refrigerator according to any one of claims 1 to 3, characterized in that: include: The warning department is responsible for issuing warnings to users; and a warning control unit that activates the warning unit when the door is in an open state for a period of time requiring a warning; in The warning control unit controls the warning unit in a manner that performs a normal warning action when the volume indicated by the sound data obtained by the sound acquisition unit is less than a predetermined threshold value, and on the other hand, controls the warning unit in a manner that performs an action different from the normal action when the volume indicated by the sound data obtained by the sound acquisition unit is greater than a predetermined threshold value.
6. The refrigerator according to any one of claims 1 to 3, characterized in that: include: The warning department is responsible for issuing warnings to users; and The warning control unit activates the warning unit when the door is in the open state for a period of time that requires a warning; The warning control unit controls the warning unit in a manner that performs a normal warning action when there is no sound input to the sound acquisition unit for more than a certain period of time. On the other hand, when there is sound input to the sound acquisition unit for more than a certain period of time, the warning unit controls the warning unit in a manner that performs an action different from the normal action.
7. The refrigerator according to any one of claims 1 to 3, characterized in that: The sound acquisition unit has a microphone; The microphone is installed in the through hole of a member having a through hole in the up-down direction provided on the upper surface of the refrigerator, with the sound input portion facing downward; The periphery of the through hole on the lower surface of the member is formed in a tapered shape that spreads downward.
8. A voice dialogue system for a refrigerator, characterized in that: include: The refrigerator according to any one of claims 1 to 7; and The server includes the voice recognition unit, and performs voice recognition of the voice data received from the refrigerator by the voice recognition unit to identify the content of the speech, and replies voice data of a response based on the identified content to the refrigerator.
