Refrigerator
Patent Information
- Application Number
- JP2023148504
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-09-13
- Publication Date
- 2025-03-26
Smart Images

Figure 2025041305000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a refrigerator that recognizes a user's gesture and automatically opens the refrigerator door. [Background technology]
[0002] 2. Description of the Related Art The refrigerator described in Patent Document 1 is known as a refrigerator that recognizes a user's gesture and automatically opens the door.
[0003] For example, paragraph 0072 of the document states, "As shown in FIG. 6, a gesture sensor 50 is disposed on the top of refrigerator 200. Gesture sensor 50 is composed of an image recognition sensor such as a CCD camera and a distance sensor such as an infrared sensor, and can identify characteristic parts of the human body of user 31, such as the head, shoulders, elbows, wrists, and fingertips, and can measure the distance to those parts." Paragraph 0074 states, "For example, assume that user 31 points at vegetable compartment 16 around refrigerator 200. At this time, gesture sensor 50 detects elbow joint 52 of user 31 and distance Db to elbow joint 52, as well as fingertip 51 and distance Dc to fingertip 51. From these numerical values, angle θ of user 31's pointing direction Lc and distance De to refrigerator 200 can be obtained (see FIG. 6)." (See FIG. 1). Furthermore, since the installation height Hc and the installation angle β of the gesture sensor 50 are known, the height Hd of the refrigerator 200 pointed to by the user 31 can be calculated, and it can be determined that the vegetable compartment 16 is being pointed to.' Paragraph 0076 states, 'When the vegetable compartment 16 is pointed to for a predetermined period of time, the display LED 19 and LED 20f on the operation unit 17 light up or blink, and the user 31 is notified that the vegetable compartment 16 has been selected.' Paragraph 0077 states, 'When the user 31 makes a beckoning motion from a state in which the vegetable compartment 16 has been selected, the movement distance per unit time of the fingertip 51 and the elbow joint 52, etc. is calculated in the same process as when the distance Db and the height Hd were calculated. When the calculated numerical value is a value corresponding to the beckoning motion, the door of the vegetable compartment 16 is automatically opened.'
[0004] Thus, Patent Document 1 discloses a refrigerator in which a door selected by a user's pointing gesture is automatically opened by the user's beckoning gesture. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] Patent Publication No. 2022-43232 Summary of the Invention [Problem to be solved by the invention]
[0006] However, hand heights vary greatly depending on the height of users, and even users of the same height have individual characteristics (habits) in the height of their hands and the angle of their fingers when pointing at a door. Therefore, in Patent Document 1, which uses the pointing direction determined by a gesture sensor installed on the top of the refrigerator as a door selection gesture, it is difficult to accurately identify which door among multiple doors lined up vertically is being pointed at.
[0007] Therefore, an object of the present invention is to provide a refrigerator that recognizes gestures to automatically open the refrigerator doors, and allows a user to more accurately specify the door he or she wants to select. [Means for solving the problem]
[0008] In order to solve the above problems, the refrigerator of the present invention is a refrigerator that recognizes a gesture and automatically opens a door, and includes a plurality of doors provided on the refrigerator, an electric assist unit that supplies opening power to each door, a camera unit that images a user standing in front of the refrigerator, and a control unit that outputs an automatic door-opening command to the electric assist unit based on an output of the camera unit, wherein the camera unit includes an image sensor that captures image data, and a gesture recognition unit that recognizes a first gesture to select a target door for automatic opening based on the shape of a hand of one arm imaged by the image sensor. Effect of the Invention
[0009] According to the refrigerator of the present invention, when a gesture is recognized to automatically open the refrigerator doors, the door that the user wants to select can be more accurately specified. [Brief description of the drawings]
[0010] [Figure 1] FIG. 2 is a front view of the refrigerator of the first embodiment. [Diagram 2] FIG. 2 is a perspective view of the camera unit according to the first embodiment. [Diagram 3] 4 is an example of image data captured by the camera unit of the first embodiment. [Figure 4] FIG. 2 is a functional block diagram of the refrigerator according to the first embodiment. [Diagram 5] 4 is a flowchart of an automatic door opening process of the refrigerator according to the first embodiment. [Figure 6] 13 is an example of a first gesture according to the first embodiment. [Figure 7] 13 is an example of a second gesture according to the first embodiment. [Figure 8] 13A to 13C are diagrams showing an editing operation method for a first gesture detection area according to the second embodiment. [Figure 9] FIG. 11 is a functional block diagram of a refrigerator according to a second embodiment. [Figure 10] 13A is a diagram for explaining a forbidden open region according to a second embodiment, and FIG. 13B is a diagram for explaining editing of the forbidden open region. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of a refrigerator according to the present invention will be described with reference to the drawings. EXAMPLES
[0012] 1 is a front view of a refrigerator 100 of the embodiment 1. As shown in the figure, the front of the refrigerator 100 of this embodiment is provided with six doors 1 (1a to 1f). In addition, the top of the refrigerator 100 is provided with a camera unit 2 that captures an image of a gesture of a user standing in front of the refrigerator 100.
[0013] <Door 1> Vertically long doors 1a and 1b provided on the left and right sides of the upper part of refrigerator 100 are doors to the refrigerator compartment and are double-door sliding doors. Horizontal door 1c provided below door 1a is the door to the ice-making compartment, which has a pull-out function. Horizontal door 1d provided below door 1b is the door to the small freezer compartment, which has a pull-out function. Horizontal door 1e provided below doors 1c and 1d is the door to the large freezer compartment, which has a pull-out function. Horizontal door 1f provided below door 1e is the door to the vegetable compartment, which has a pull-out function.
[0014] In the refrigerator 100 of this embodiment, the four relatively large doors 1a, 1b, 1e, and 1f among the six doors 1a to 1f are provided with a function of automatically opening the doors in response to a gesture by the user.
[0015] <Camera unit 2> Fig. 2 is a perspective view of camera unit 2 in Fig. 1, seen from below. As shown here, camera unit 2 of this embodiment is disposed on the ceiling outside refrigerator 100, has a shape that protrudes forward from the front surface of door 1 in a closed state, and has a lens 21 on the underside of its tip that faces a user standing in front of refrigerator 100. This lens 21 is a wide-angle lens such as a fisheye lens, and an image sensor 22 such as a CCD (Charge Coupled Device) is provided above it.
[0016] Therefore, when a user is standing in front of the refrigerator 100, the image sensor 22 can capture image data as shown in Fig. 3. Note that, although details will be described later, a first gesture detection area in Fig. 3 is an area set to limit the target range of gesture detection in order to reduce the possibility of mistaking an unconscious action of the user for a command (gesture).
[0017] Furthermore, lighting 23 (light source) such as an LED is provided in the center of the bottom surface of camera unit 2. When the surrounding environment of refrigerator 100 is dark, lighting 23 is used to illuminate a user standing in front of refrigerator 100, allowing image sensor 22 to capture a clear image of the user.
[0018] Furthermore, an indicator 24 simulating the door arrangement of refrigerator 100 is provided on the underside of the base of camera unit 2. This indicator 24 is a device that notifies the user of the door selected by gesture recognition by lighting or blinking a portion corresponding to door 1 selected by the user's gesture. Since gesture recognition is performed using camera unit 2, providing camera unit 2 with indicator 24 that reflects the gesture recognition result can reduce the need to manufacture different refrigerators 100. When a gesture recognition function is to be installed, it is sufficient to install camera unit 2 in refrigerator 100, and it is possible to reduce the need to change whether or not to install an indicator in refrigerator 100 depending on whether or not the gesture recognition function is installed, that is, to manufacture different models.
[0019] <Function blocks of refrigerator 100> FIG. 4 is a functional block diagram of the refrigerator 100. As shown in FIG. 4, each of the doors 1 that automatically open is provided with an electric assist unit 11 that supplies opening power to each door, and a door sensor 12 that detects the open / closed state of each door. The camera unit 2 is provided with a buzzer 25 and a gesture recognition unit 26 in addition to the lens 21, image sensor 22, lighting 23, and indicator 24 described above. The refrigerator 100 is further provided inside with a control unit 3 that controls the electric assist unit 11 and the like in response to input from the camera unit 2, and a wireless unit 4 that communicates with a mobile terminal 200 such as a smartphone. The gesture recognition unit 26 and the control unit 3 are specifically arithmetic devices such as a CPU that execute a predetermined program.
[0020] <Automatic door opening process flowchart> FIG. 5 is a flowchart of an automatic door opening process by the refrigerator 100 of FIG.
[0021] First, in step S1, the image sensor 22 of the camera unit 2 captures image data of the area in front of the refrigerator 100. If the environment around the refrigerator 100 is dark, the light 23 may be turned on in this step. "Capturing an image" here refers to the control unit 3 performing calculations such as recognizing information in the image, and does not necessarily involve storing the image data.
[0022] Next, in step S2, gesture recognition unit 26 of camera unit 2 determines whether a first gesture for identifying automatic door opening is captured in the image data acquired in step S1. If the first gesture is captured, the process proceeds to step S3, and if the first gesture is not captured, the process returns to step S1. A start condition for step S1 may be set to be that camera unit 2 or a human presence sensor such as an infrared sensor provided in refrigerator 100 has detected a person in front of refrigerator 100. Alternatively, a start condition for step S1 may be set to be that all doors 1 of refrigerator 100 are closed.
[0023] Fig. 6 is an example of the first gesture. In this example, a unique left hand shape is assigned to each door to be automatically opened, so that the gesture recognition unit 26 can identify with high accuracy the door that the user wants to open based on the left hand shape (particularly the number of fingers) in the first gesture detection area shown in Fig. 3. When recognizing the first gesture from the left hand shape, a well-known technique such as template matching can be used to determine which gesture the gesture corresponds to based on the similarity between a prepared template image of the left hand shape and image data of the captured left hand shape.
[0024] In step S3, indicator 24 of camera unit 2 lights or blinks a portion corresponding to the door selected by the first gesture detected in step S2. This allows the user to understand the recognition result of the first gesture. In this step, buzzer 25 may be sounded to notify the user of the timing of recognition of the first gesture.
[0025] In step S4, the image sensor 22 of the camera unit 2 captures image data of the area in front of the refrigerator 100. If the environment around the refrigerator 100 is dark, the light 23 may be turned on in this step.
[0026] In step S5, the gesture recognition unit 26 of the camera unit 2 determines whether a second gesture is captured in the image data acquired in step S4. The second gesture is a gesture for starting automatic door opening or a gesture for canceling the door selected by the first gesture. If the captured second gesture is an automatic door opening gesture, the process proceeds to step S6. If the captured second gesture is a cancel gesture, the process returns to step S1. If the captured second gesture is not captured, the process returns to step S4. In this embodiment, the second gesture detection area is the entire range of the captured image.
[0027] FIG. 7 is an example of the second gesture. In this example, the user's right arm movement (e.g., a movement of waving the right arm, a movement of pulling the right arm) or a facial expression (e.g., a smile) is set as an automatic door opening gesture, and a movement of shaking the head is set as a cancel gesture. Therefore, the gesture recognition unit 26 can specify the timing of automatic door opening based on the gesture of the user's right arm, etc. Note that when recognizing the second gesture from the right hand movement, etc., a well-known technique such as template matching can be used to determine which gesture corresponds based on the similarity between a template image of the right arm movement, etc. prepared in advance and image data of the captured right arm movement, etc.
[0028] In step S6, the gesture recognition unit 26 of the camera unit 2 notifies the control unit 3 of information identifying the door 1 that is to be automatically opened. Then, upon receiving the notification, the control unit 3 transmits an automatic door-open command to the electric assist unit 11 of the selected door 1. As a result, the door 1 selected by the user's gesture is automatically opened. Note that in this step, the buzzer 25 may be sounded to notify the user of the timing of recognition of the second gesture.
[0029] In this step, the control unit 3 may operate as follows. (1) If the user is approaching the refrigerator 100 within a predetermined distance and it is predictable that the door 1 will come into contact with the user if the door 1 is automatically opened, the control unit 3 may withhold output of the automatic door-open command to the electric assist unit 11 and output the automatic door-open command after the user has sufficiently moved away from the refrigerator 100. Withholding the output of the automatic door-open command will also be described in the second embodiment. (2) If the door sensor 12 of the selected door 1 is outputting information indicating an open state, there is no need to automatically open the door 1, so the control unit 3 may cancel the output of an automatic door-open command to the electric assist unit 11 of the selected door 1. (3) When the door sensor 12 of a door 1 other than the selected door 1 outputs information indicating an open state, the control unit 3 may cancel the output of the automatic door-open command to the electric assist unit 11 of the selected door 1. This is to prevent an accident in which the user's hand or the like is caught between the inner surface of an already opened door 1 (e.g., door 1f in FIG. 1) and the outer surface of a door 1 that is about to be automatically opened (e.g., door 1e in FIG. 1).
[0030] In FIG. 6, the left hand shape is the first gesture, and in FIG. 7, the right arm movement is the second gesture, but the right hand shape may be the first gesture and the left arm movement may be the second gesture. Also, the same arm or hand may be used for the first gesture and the second gesture. In the former case, it is easy to distinguish between the first gesture and the second gesture, so that the recognition accuracy of the gesture recognition unit 26 is easily ensured. In the latter case, it is convenient for the user who performs the gesture movement. In addition, the first gesture and the second gesture may be recognized regardless of which arm or hand is used.
[0031] <How to set the first and second gestures> In the gesture recognition unit 26 of this embodiment, the gestures exemplified in Fig. 6 and Fig. 7 are pre-registered, but the pre-registered gestures may not be to the user's liking or may make the user feel uncomfortable. Therefore, in the refrigerator 100 of this embodiment, the correspondence between the door 1 to be selected and the first gesture, as exemplified in Fig. 6, or the correspondence between the command and the second gesture, as exemplified in Fig. 7, can be edited using gesture setting application software downloaded to the mobile terminal 200, such as a smartphone.
[0032] For this reason, in this embodiment, a large number of candidates for the first gesture and the second gesture are prepared, and the content of the first gesture corresponding to each door can be set for each user, and the content of the second gesture corresponding to each command can be set for each user, using an application on the mobile terminal 200.
[0033] Therefore, if the facial image of each user is registered in advance in the gesture recognition unit 26, each user can automatically open the desired door 1 by using the first gesture and the second gesture that he or she has set for himself or herself. However, it is preferable to prohibit the registration of the same gesture for two or more commands. <Effects of this embodiment> As described above, according to the refrigerator of the present invention, the hand shape of one arm is used as a first gesture to select the door to be automatically opened, and the movement of the other arm is used as a second gesture to start the automatic door opening. Therefore, when the gestures are recognized and the refrigerator door is automatically opened, the door that the user wants to select can be more accurately identified and the automatic door opening can be executed.
[0034] <Other> In this embodiment, automatic door opening is enabled by a gesture action, but the refrigerator 100 or the camera unit 2 may be made to perform an action other than opening the door. EXAMPLES
[0035] The configuration of this embodiment differs from that of the first embodiment in the following respects, but the remaining points can be the same as those of the first embodiment. <Gesture detection area settings> FIG. 8 is a diagram showing a method of editing the first gesture detection area. Using application software downloaded to the mobile terminal 200 such as a smartphone, the user can edit a part or all of the size, shape, position, and number of the first gesture detection area. In this example, the position of one first gesture detection area A that was set in advance is changed (moved) to the position of a new first gesture detection area B. This allows the user to change the first gesture detection area to a state in which the user can easily perform a gesture, and perform an automatic door opening operation without stress. Here, when changing the shape of the first gesture detection area, it is possible to rotate, enlarge, reduce, etc. Also, the shape of the gesture detection area is illustrated as a rectangle in FIG. 8, but it may be any shape such as a polygon or a circle, and the shape may be changed by the user. The size can be changed, for example, by a pinch operation on the mobile terminal 200. The shape can be changed, for example, by moving the user's finger while touching the edge of the gesture detection area displayed on the mobile terminal 200. The position can be changed, for example, by moving the user's finger while touching the vicinity of the center of the gesture detection area displayed on the mobile terminal 200. To change the number, buttons for increasing or decreasing the number can be provided on the display screen of the mobile terminal 200.
[0036] In this embodiment, a second gesture detection area, which is an area in which a second gesture is detected, in the image data captured by the image sensor 22 is also editable in the same manner as the first gesture detection area.
[0037] Also, while the application software is editing the first and / or second gesture detection areas, the video captured by the image sensor 22 may be checked in real time. In this case, when the application software of the mobile terminal 200 is launched and starts the editing function of the first and / or second gesture detection areas, the application software transmits a video request signal to the camera unit 2. Upon receiving this signal, the camera unit 2 continues to continuously transmit image data captured by the image sensor 22. This allows the user to edit each gesture detection area while checking real-time or recent past images or video images.
[0038] The user may perform a test demonstration of the first gesture or the second gesture, and the gesture detection area may be automatically set according to the range of the gesture. In this case, it is also preferable that the user can appropriately change or adjust the gesture detection area automatically set by the gesture recognition unit 26. <Door opening determination by face recognition> 9 is a functional block diagram of the refrigerator 100 of this embodiment. In this embodiment, the refrigerator 100 can recognize the presence or absence of authority to operate the refrigerator 100 by a gesture operation for each user. For example, when the refrigerator 100 recognizes the first and second gestures made by one or more valid users of the refrigerator 100, the refrigerator 100 executes the operation as usual. On the other hand, even if the refrigerator 100 recognizes the first and second gestures made by an invalid user, such as a child, a visitor, or another person who is not the original owner of the refrigerator 100, the refrigerator 100 does not execute the corresponding operation.
[0039] Therefore, in the refrigerator 100 of this embodiment, the face of a person reflected in the image data captured by the image sensor 22 of the camera unit 2 is recognized, and the door is automatically opened only by a gesture of a proper user. The proper user (proper facial information) can be set through application software operated on the mobile terminal 200.
[0040] The refrigerator 100 in this embodiment includes an acquisition unit 27 that stores and acquires face information of a user from an image acquired by the image sensor 22, and a determination unit 28 that determines a degree of match between the face information of a user standing near the front position of the refrigerator 100 and captured by the image sensor 22 and the face information stored in the acquisition unit 27. The acquisition unit 27 does not necessarily have to be capable of storing face information acquired from an image acquired by the image sensor 22, but may be capable of storing image data captured by the terminal camera of the mobile terminal 200, for example. The storage function of the acquisition unit 27 does not necessarily have to be provided in the refrigerator 100, and may be provided in the mobile terminal 200 or a server with which the refrigerator 100 can communicate. In this case, the camera unit 2 may function to access an external storage function (memory, storage, etc.) via the wireless unit 4 to acquire face information. However, from the viewpoint of avoiding storing face information externally for reasons of privacy, security, etc., it is preferable to provide the refrigerator 100 or the camera unit 2 with the storage function. The timing for the acquisition unit 27 to acquire face information may be when the device is first started up, or may be any timing determined by the user.
[0041] When the camera unit or the human presence sensor described in the first embodiment recognizes the presence of a person, the light 23 of the camera unit 2 may be turned on to enable the face of the user to be captured clearly. Since the light 23 is in the same direction as the lens 21 for the user standing in front of the refrigerator 100, turning the light on can also guide the user to face the camera unit 2. Also, the indicator 24 may be appropriately turned on or flashed to guide the direction of the user's face toward the lens 21 and image sensor 22 of the camera unit 2.
[0042] The refrigerator 100 of this embodiment notifies the result of the determination by the determination unit 28 in different forms. For example, when the determination unit 28 determines that the user has the authority to perform an action by gesture, the light 23 of the camera unit 2 is caused to blink. On the other hand, when the determination unit 28 determines that there is a mismatch, the buzzer 25 is caused to sound twice. Here, the blinking of the light, the number of times and intervals of the buzzer sounding, and other predetermined operation parameters are not limited to the above forms, and any operation may be used as long as the user can identify the meaning of the notification.
[0043] Only when the determination unit 28 determines that the authority exists, the gesture recognition unit 26 executes a corresponding operation upon recognizing the first gesture or the second gesture.
[0044] <Open prohibited area (automatic door open command output pending)> FIG. 10 is a diagram for explaining (a) the open-prohibited area of this embodiment, and (b) the editing of the open-prohibited area. When the automatic door opening based on gesture detection is performed in a situation where the user is closer to the refrigerator 100 than a predetermined distance, the door may collide with the user. Therefore, the refrigerator 100 in this embodiment is configured not to automatically open the door if a person is recognized in the open-prohibited area of the image data captured by the camera unit 2. The open-prohibited area is within a range set along the edges of the doors 1a and 1b of the refrigerator 100, for example, as shown in FIG. 10(a). The output of the automatic door opening may be suspended by not proceeding to step S2 when a part of a person's body is recognized in the open-prohibited area in the image data acquired in step S1 of FIG. 5, or by suspending the execution of step S6 until a part or all of a person is no longer recognized in the open-prohibited area.
[0045] The open-prohibited area may be arbitrarily set by the user. The user changes the area by moving any point, for example, among vertices 201a to 201e, of the area forming the open-prohibited area on the mobile terminal 200. Here, the number of vertices does not have to be five as shown in FIG. 10. When changing the range of the open-prohibited area, only a change that includes the entire open-prohibited area in the default setting (at the time of shipment from the factory) may be permitted. In other words, reduction from the open-prohibited area in the default setting may be prohibited. When changing the open-prohibited area, the open-prohibited area may be prohibited from entering the first gesture detection area, or a warning may be issued if the open-prohibited area enters the first gesture detection area. [Explanation of symbols]
[0046] 100 Refrigerator 1, 1a~1f Doors 11 Electric assist unit 12 Door sensor 2 Camera unit 21 Lens 22 Image Sensor 23 Lighting 24 Indicators 25 Buzzer 26 Gesture recognition unit 27 Acquisition Department 28 Judgment section 3. Control Unit 4 Wireless Unit 200 Mobile Devices
Claims
1. A refrigerator that recognizes gestures and automatically opens the door, A plurality of doors provided on the refrigerator; An electric assist unit that supplies opening power to each door, a camera unit that captures an image of a user standing in front of the refrigerator; A control unit that outputs an automatic door opening command to the electric assist unit based on an output of the camera unit; Equipped with The camera unit includes: an image sensor for capturing image data; a gesture recognition unit that recognizes a first gesture for selecting a target door for automatic door opening based on a shape of a hand of one arm captured by the image sensor; A refrigerator comprising:
2. In the refrigerator according to claim 1, The gesture recognition unit, after recognizing the first gesture, notifies the control unit of a target door for automatic door opening when it recognizes a second gesture for starting automatic door opening based on the movement of the other arm captured by the image sensor. A refrigerator characterized by:
3. In the refrigerator according to claim 1, The camera unit is provided with an indicator that notifies the user of the door selected based on the first gesture.
4. In the refrigerator according to claim 2, The device is provided with a wireless unit for communicating with a mobile terminal. The refrigerator is characterized in that the type of the first gesture or the second gesture can be changed using the mobile terminal.
5. In the refrigerator according to claim 1, The gesture recognition unit recognizes a gesture for canceling a door selection based on the first gesture.
6. In the refrigerator according to claim 1, The control unit prevents the electric assist unit from automatically opening the door when a user is in an opening prohibited area.
7. In the refrigerator according to claim 1, Each door is equipped with a door sensor that detects whether it is open or closed. The control unit does not cause the electric assist unit to automatically open the door when any of the door sensors outputs information indicating an open state.
8. In the refrigerator according to claim 1, A part of the image data captured by the image sensor is defined as a gesture detection area, and a first gesture performed within the gesture detection area is recognized.
9. The refrigerator according to claim 8, The device is provided with a wireless unit for communicating with a mobile terminal. The gesture detection area is editable on the mobile terminal.
10. In the refrigerator according to claim 1, The camera unit includes: It is installed on the exterior ceiling of the refrigerator. a lens disposed below the image sensor; and an indicator and / or a light source disposed near the lens to notify the user of a door selected based on the first gesture; The refrigerator, wherein the indicator and / or the light source can be lit up or blinking when the image sensor captures an image of a person.
11. The refrigerator according to claim 6, The device is provided with a wireless unit for communicating with a mobile terminal. The refrigerator is characterized in that the range of the opening prohibited area can be expanded beyond a default setting by using the mobile terminal, but cannot be reduced beyond the default setting.
Citation Information
Patent Citations
refrigerator
JP2022043232A