Tableware equipment, operation method of tableware equipment, learning device for tableware equipment, and learning method for tableware equipment

Abstract

To provide a technique capable of correctly acquiring a placement state of tableware placed on a placement member.SOLUTION: A facility for tableware comprises: a storage chamber which includes a placement member on which tableware is placed and stores the tableware; an imaging device 200 which is configured to be able to capture a tableware placement image being an image indicating the state where a user places the tableware on the placement member; and a control device 60. The control device 60 comprises: a tableware placement image acquisition unit which acquires the tableware placement image from the imaging device 200; and a tableware placement state estimation unit which inputs the tableware placement image to a learned learning machine that has performed machine learning for estimating the tableware placement state being the placement state of the tableware on the basis of the tableware placement image and executes calculation processing of the learned learning machine to estimate the tableware placement state.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present specification relates to a tableware installation, a method for operating a tableware installation, a learning device for a tableware installation and a learning method for a tableware installation. [Background technology]

[0002] Patent Document 1 discloses a tableware equipment including a mounting member on which tableware is placed, a storage compartment for storing the tableware, an imaging device configured to capture images of the interior of the storage compartment, and a control device. In this tableware equipment, the placement position of the tableware on the mounting member is identified by image analysis based on the image of the interior of the storage compartment captured by the imaging device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-235781 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when multiple dishes are placed on a placement member, some dishes may be hidden behind other dishes in an image of the storage chamber. In such a case, it is difficult to accurately determine the placement position and placement angle of the dishes hidden behind other dishes based on image analysis of the image of the storage chamber. This specification provides a technology that can more accurately obtain the placement state of the dishes placed on the placement member. [Means for solving the problem]

[0005] This specification discloses a tableware equipment. The tableware equipment includes a placement member on which tableware is placed, a storage unit for storing the tableware, an imaging device configured to capture a tableware placement image that shows a state in which a user is placing the tableware on the placement member, and a control device. The control device includes a tableware placement image acquisition unit that acquires the tableware placement image from the imaging device, a tableware related information acquisition unit that acquires tableware related information, which is information related to the tableware, by image analysis based on the tableware placement image; The tableware placement state is represented by the tableware placement image. and the tableware-related information The tableware placement image is input to a trained learning device that has performed machine learning to estimate the tableware placement image based on the tableware placement image. and the tableware-related information The present invention is also provided with a tableware placement state estimation unit that estimates the tableware placement state by inputting the above and executing the calculation processing of the trained learning device.

[0006] According to the tableware equipment described above, the placement state of a piece of tableware hidden behind another piece of tableware can be estimated with high accuracy. The placement state of the tableware placed on the placement member can be obtained more accurately. The tableware placement state is estimated based on tableware-related information (e.g., information about the size, shape, material, etc. of the tableware) obtained by image analysis of the tableware placement image, so that the tableware placement state can be estimated more accurately.

[0007] This specification also discloses another tableware equipment. The tableware equipment includes a placement member on which tableware is placed, a storage unit for storing the tableware, an imaging device configured to capture a tableware placement image that shows a state in which a user places the tableware on the placement member, and a control device. The control device includes a tableware placement image acquisition unit that acquires the tableware placement image from the imaging device, a resolution conversion unit that reduces the resolution of the tableware placement image to generate a low-resolution captured image, and a tableware placement state estimation unit that inputs the low-resolution captured image to a trained learning device that has performed machine learning to estimate a tableware placement state, which is a placement state of the tableware, based on the low-resolution captured image, and executes arithmetic processing of the trained learning device to estimate the tableware placement state.

[0008] According to the above tableware equipment, the placement state of tableware hidden behind other tableware can be estimated with high accuracy. The placement state of tableware placed on the placement member can be obtained more accurately. In addition, since a low-resolution captured image is input to the learning device instead of a tableware placement image, the amount of calculation required for the learning device's arithmetic processing can be reduced, and the calculation load can be reduced.

[0009] The tableware equipment may further include a washing device for washing the tableware stored in the storage compartment. The control device may further include a washing device control unit for setting washing conditions based on the tableware placement state and controlling the washing device based on the washing conditions.

[0010] According to the above configuration, the washing conditions when the washing device washes the tableware can be appropriately set based on the placement state of the tableware.

[0011] The control device may further include a dish stain information acquisition unit that acquires dish stain information, which is information regarding stains on the dishes, by image analysis based on the dish placement image. The washing device control unit may set the washing conditions based on the dish stain information and control the washing device based on the washing conditions.

[0012] According to the above configuration, the washing conditions for the washing device to wash dishes are set based on dish stain information (e.g., information about the type of stain adhering to the dishes, the degree of staining, etc.), so that more appropriate washing conditions can be set.

[0013] The tableware equipment may further include a drying device for drying the tableware stored in the storage compartment. The control device may further include a drying device control unit for setting drying conditions based on the tableware placement state and controlling the drying device based on the drying conditions.

[0014] According to the above configuration, the drying conditions when the drying device dries the tableware can be appropriately set based on the placement state of the tableware.

[0015] This specification also discloses an operation method of a tableware equipment. The tableware equipment includes a placement member on which tableware is placed, a storage unit for storing the tableware, an imaging device configured to capture a tableware placement image that shows a state in which a user is placing the tableware on the placement member, and a control device. The operation method includes a step in which the control device acquires the tableware placement image from the imaging device, The control device acquires tableware-related information, which is information related to the tableware, by image analysis based on the tableware placement image. The control device displays a tableware placement state, which is a tableware placement state, on the tableware placement image. and the tableware-related information The tableware placement image is input to a trained learning device that has performed machine learning to estimate the tableware placement image based on the tableware placement image. and the tableware-related information and executing arithmetic processing of the trained learning device to estimate the tableware placement state.

[0016] This specification also discloses an operation method of another tableware equipment. The tableware equipment includes a placement member on which tableware is placed, a storage unit for storing the tableware, an imaging device configured to capture a tableware placement image that shows a state in which a user places the tableware on the placement member, and a control device. The operation method includes a step in which the control device acquires the tableware placement image from the imaging device, a step in which the control device reduces the resolution of the tableware placement image to generate a low-resolution image, and a step in which the control device inputs the low-resolution image into a trained learning device that has performed machine learning to estimate a tableware placement state, which is a tableware placement state, based on the low-resolution image, and executes a calculation process of the trained learning device to estimate the tableware placement state.

[0017] The tableware equipment may further include a washing device for washing the tableware stored in the storage compartment. The operating method may further include a step in which the control device sets washing conditions based on the tableware placement state and controls the washing device based on the washing conditions.

[0018] The method may further include a step of the control device acquiring dish stain information, which is information regarding stains on the dishes, by image analysis based on the dish placement image. In the step of controlling the operation of the washing device, the control device may set the washing conditions based on the dish stain information and control the washing device based on the washing conditions.

[0019] The tableware equipment may further include a drying device for drying the tableware stored in the storage compartment. The operating method may further include a step in which the control device sets drying conditions based on the tableware placement state and controls the drying device based on the drying conditions.

[0020] This specification also discloses a learning device for tableware equipment. The tableware equipment includes a placement member on which tableware is placed, a storage unit for storing the tableware, an imaging device configured to capture a tableware placement image that shows a state in which a user is placing the tableware on the placement member, and a control device. The learning device includes a tableware placement image acquired from the imaging device, tableware-related information, which is information related to the tableware and is associated with the tableware-placed image and obtained by image analysis based on the tableware-placed image; a learning data acquisition unit that is associated with the tableware placement image and acquires learning data including a tableware placement state that is a placement state of the tableware; and the tableware-related information The system is equipped with a learning processing unit that performs machine learning of a learning device so as to obtain an output including the tableware placement state in response to an input including the tableware placement state.

[0021] According to the above configuration, the image of the tableware being placed and tableware-related information It is possible to construct a trained learning machine for estimating the dish placement state based on the above.

[0022] This specification also discloses a learning device for another tableware equipment. The tableware equipment includes a placement member on which tableware is placed, a storage unit for storing the tableware, an imaging device configured to capture a tableware placement image that is an image showing a state in which a user places the tableware on the placement member, and a control device. The learning device includes a low-resolution image obtained by reducing the resolution of the tableware placement image acquired from the imaging device, a learning data acquisition unit that acquires learning data including a tableware placement state that is associated with the low-resolution image and is the placement state of the tableware, and a learning processing unit that performs machine learning of a learning device to obtain an output including the tableware placement state in response to an input including the low-resolution image.

[0023] According to the above configuration, For low resolution images Based on this, a trained learning machine can be constructed to estimate the dish placement state.

[0024] This specification also discloses a learning method for tableware equipment. The tableware equipment includes a placement member on which tableware is placed, a storage unit for storing the tableware, an imaging device configured to capture a tableware placement image that shows a state in which a user is placing the tableware on the placement member, and a control device. The learning method includes a learning device that captures the tableware placement image acquired from the imaging device, tableware-related information, which is information related to the tableware and is associated with the tableware-placed image and obtained by image analysis based on the tableware-placed image; acquiring learning data corresponding to the tableware placement image and including a tableware placement state that is a placement state of the tableware; and the tableware-related information The method includes a step of performing machine learning of a learning device so as to obtain an output including the tableware placement state for an input including the tableware placement state.

[0025] This specification also discloses a learning method for another tableware equipment. The tableware equipment includes a placement member on which tableware is placed, a storage unit for storing the tableware, an imaging device configured to capture a tableware placement image that is an image showing a state in which a user places the tableware on the placement member, and a control device. The learning method includes a step in which the learning device acquires learning data including a low-resolution image obtained by reducing the resolution of the tableware placement image acquired from the imaging device and a tableware placement state that is associated with the low-resolution image and is the placement state of the tableware, and a step in which the learning device performs machine learning of a learner so as to obtain an output including the tableware placement state in response to an input including the low-resolution image. [Brief description of the drawings]

[0026] [Figure 1] FIG. 2 is a diagram illustrating a schematic configuration of a system 2 according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a side view of the dishwasher / dryer 10 and the imaging device 200 of the embodiment. [Diagram 3] FIG. 1 is a vertical cross-sectional view of a dishwasher / dryer 10 according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing a schematic configuration of a control device 60 of the dishwasher-dryer 10 of the embodiment. [Diagram 5] FIG. 2 is a diagram illustrating a schematic configuration of a control device 308 of a server device 300 according to the embodiment. [Figure 6] 4 is a flowchart showing a process performed by a control device 60 of the dishwasher-dryer 10 of the embodiment. [Figure 7] 10 is a flowchart showing a process performed by a control device 308 of a server device 300 according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] (Example) Hereinafter, system 2 according to an embodiment will be described with reference to the drawings. As shown in Fig. 1, system 2 includes a dishwasher 10, a relay device 100, an imaging device 200, and a server device 300. Dishwasher 10 is an example of tableware equipment, and server device 300 is an example of a learning device.

[0028] 2, dishwasher / dryer 10 is installed in a system kitchen 3 and used. Dishwasher / dryer 10 includes a main body 12, a storage compartment 14, and a door 15.

[0029] As shown in FIG. 3, the door 15 is provided with an operation panel 16 and multiple exhaust paths 18 (only one is shown in FIG. 3). The operation panel 16 is provided with various buttons such as a start button, lamps, etc. One end of the multiple exhaust paths 18 communicates with the outside through a single exhaust port 18b provided on the front surface of the door 15 (left side in FIG. 3). The other end of the multiple exhaust paths 18 communicates with the inside of the storage 14 through multiple openings 18a provided in the front part of the storage 14. Each exhaust path 18 is provided with an exhaust damper 17 that adjusts the opening degree of the exhaust path 18. By switching between opening and closing of each exhaust damper 17, the opening 18a through which the dry air flows out from the storage 14 can be switched, and thereby the position from which the dry air flows out from the storage 14 can be switched.

[0030] The storage cabinet 14 is stored in a space formed by the main body 12 and the door 15. The storage cabinet 14 is slidably supported by the main body 12. The storage cabinet 14 is slidable in the front-to-rear direction (left-to-right direction in Figure 3) relative to the main body 12. The storage cabinet 14 is connected to the door 15. When the door 15 is pulled out forward (leftward in Figure 3), the storage cabinet 14 is pulled out together with the door 15. The storage cabinet 14 has an opening 14a formed at its upper end. The storage cabinet 14 stores a washing nozzle 20, a dish basket 61, etc. Various types of tableware 19 can be placed in the dish basket 61. The dish basket 61 is an example of a placement member.

[0031] A lid 56 is disposed above the storage facility 14. The lid 56 is connected to the main body 12 by a lifting mechanism (not shown). When the door 15 is closed (when the storage facility 14 is housed in the main body 12), the lid 56 descends to fit closely against the opening 14a of the storage facility 14 and cover it. A seal member 70 is attached to the underside of the lid 56 along the periphery. When the lid 56 descends, the opening 14a of the storage facility 14 is sealed by the seal member 70. When the storage facility 14 is pulled out of the main body 12, the lid 56 ascends to open the opening 14a of the storage facility 14.

[0032] A suction recess 31 is formed on the bottom surface 39 of the storage 14. A cleaning pump 27 is provided below the bottom surface 39. The cleaning pump 27 rotates the impeller 28 by an electric motor built in the storage 14. The cleaning pump 27 can rotate the impeller 28 in the forward direction or in the reverse direction. When the cleaning pump 27 is rotated in the reverse direction, water can be drained from the storage 14. The space in which the impeller 28 is disposed and the suction recess 31 are connected by a suction passage 32. When the cleaning pump 27 rotates the impeller 28 in the forward direction, water is sent from the suction recess 31 to the cleaning nozzle 20 through the suction passage 32. The cleaning nozzle 20 sprays the sent water inside the storage 14. The cleaning nozzle 20 is equipped with a plurality of jet ports 20a, 20b, 20c, flow paths (not shown) communicating with each of the jet ports 20a, 20b, 20b, and a switching valve 21 that switches between communication and non-communication between each of the paths and the suction flow path 32. By operating the switching valve 21, it is possible to change which of the jet ports 20a, 20b, 20c of the cleaning nozzle 20 the water is sprayed from, and thus it is possible to change the water spray range within the storage 14 and the direction in which the cleaning nozzle 20 sprays water.

[0033] An electric heater 30 is attached above the bottom surface 39 on which the washing pump 27 is disposed. The heater 30 heats water when water is contained in the storage compartment 14, and heats the air inside the storage compartment 14 when no water is contained. A thermistor 55 is disposed below the heater 30. The thermistor 55 detects the temperature of the water when water is contained in the storage compartment 14, and detects the temperature of the air inside the storage compartment 14 when no water is contained.

[0034] A water level detector 45 is provided at the bottom of the front outside of the storage 14. The water level detector 45 has a water level chamber 46, a float 47, a bar 48, and a switch 49. The water level chamber 46 is connected to the suction recess 31 by a water level path 44, and is connected to the bottom of the storage 14 by a water level path 38. The float 47 is disposed in the water level chamber 46. The bar 48 is fixed to the float 47 and protrudes upward from the water level chamber 46. The switch 49 is disposed above the bar 48. When water is supplied to the storage 14, the water is introduced into the water level chamber 46 through the water level path 44. The water level 58 of the water level chamber 46 rises as the water level of the storage 14 rises. The float 47 rises as the water level 58 of the water level chamber 46 rises. When the float 47 rises and the upper end of the bar 48 comes into contact with the switch 49, a signal is sent from the switch 49 to a control device 60, which will be described later. When the switch 49 of the water level detector 45 is turned ON, the water level 58 is located above each of the openings 18a, and each of the exhaust paths 18 is water-sealed.

[0035] A detergent dispensing mechanism 71 is provided on the upper part of the outside of the front part of the storage compartment 14. The detergent dispensing mechanism 71 includes a detergent tank 72 and a detergent dispensing valve 73. A detergent supply path 74 that communicates with the outside at the top surface of the door 15 and a detergent dispensing path 75 that communicates with the inside of the storage compartment 14 are connected to the detergent tank 72. The detergent dispensing valve 73 is provided in the detergent dispensing path 75. The detergent tank 72 is replenished with detergent via the detergent supply path 74. The detergent dispensing valve 73 is normally closed, and is opened when detergent is dispensed into the storage compartment 14. The amount of detergent dispensed from the detergent tank 72 into the storage compartment 14 can be adjusted by adjusting the time that the detergent dispensing valve 73 is open.

[0036] A drain hose 34 is connected to the rear wall 33 (the wall on the right side in FIG. 3) of the main body 12. The drain hose 34 communicates with the bottom of the storage facility 14 via a drain passage 36. The washing pump 27 sends water from within the storage facility 14 to the drain hose 34 by rotating the impeller 28 in the reverse direction.

[0037] Moreover, one end of a water supply hose 40 is connected to the rear wall 33 of the main body 12. The other end of the water supply hose 40 is connected to a water supply pipe 88. One end of the water supply hose 40 is connected to a water supply valve 41 inside the main body 12 via a first water supply flow path 42. The water supply valve 41 is driven to open and close by a built-in solenoid. When the water supply valve 41 is opened, water is supplied into the storage compartment 14, and when the water supply valve 41 is closed, the supply of water into the storage compartment 14 is stopped. The water supply valve 41 and the storage compartment 14 are connected by a second water supply flow path 43.

[0038] A plurality of air supply paths 63 are formed between the rear wall 33 of the main body 12 and the rear wall 51 of the storage 14. One end of the plurality of air supply paths 63 communicates with the interior of the storage 14 through a plurality of openings 63a provided at the rear of the storage 14. The other end of the plurality of air supply paths 63 communicates with a single drying fan 52 arranged outside the storage 14. The drying fan 52 drives the fan 53 to rotate with a built-in motor to send air into each air supply path 63. The drying fan 52 is provided with a rotation speed sensor 50 that detects the rotation speed of the fan 53. Each air supply path 63 is provided with an air supply damper 65 that adjusts the opening degree of the air supply path 63. By switching the opening and closing of each air supply damper 65, the opening 63a through which the dry air flows into the storage 14 can be switched, and the position from which the dry air flows into the storage 14 can be switched. An opening 63a of each air supply path 63 is provided with an air supply louver 67 for adjusting the direction of the dry air flowing from that air supply path 63 into the inside of the storage facility 14.

[0039] A control device 60 and a Bluetooth (registered trademark) (hereinafter also referred to as BT) interface (hereinafter also referred to as I / F) 62 are mounted on the front of the dishwasher / dryer 10. The BTI / F 62 is a short-range wireless I / F for performing BT communication with an external device.

[0040] The control device 60 includes a CPU, a ROM, a RAM, an EEPROM, etc. The control device 60 controls the operation of the dishwasher / dryer 10 in accordance with a program 66 (see FIG. 1) stored in a memory 64 (see FIG. 1). The control device 60 will be described in detail later.

[0041] In the following description, the pre-wash operation, main wash operation, and rinsing operation of the dishes 19 in the dishwasher 10 are collectively referred to as the washing operation, and the switching valve 21, washing pump 27, heater 30, water supply valve 41, switch 49, thermistor 55, detergent injection valve 73, etc. that operate during the washing operation are collectively referred to as the washing device 76. Also, in the dishwasher 10, the exhaust damper 17, heater 30, drying fan 52, rotation speed sensor 50, thermistor 55, air supply damper 65, air supply louver 67, etc. that operate during the drying operation of the dishes 19 are collectively referred to as the drying device 77.

[0042] As shown in FIG. 1, the dishwasher / dryer 10 can communicate with the relay device 100 and the imaging device 200 via the BTI / F 62. In this embodiment, the relay device 100 is a kitchen remote control installed in a kitchen to operate a water heater (not shown). The relay device 100 is attached to a wall surface near the dishwasher / dryer 10. The relay device 100 includes a BTI / F 102, a Wi-Fi (registered trademark) I / F 104, and a control unit 106. The BTI / F 102 is a short-range wireless I / F for performing BT communication with an external device. The relay device 100 can perform BT communication with the dishwasher / dryer 10 and the imaging device 200 via the BTI / F 102. The Wi-Fi I / F 104 is a wireless I / F for performing Wi-Fi communication with an external device. The relay device 100 can connect to the Internet 150 via the Wi-Fi I / F 104. The control unit 106 includes a CPU, a ROM, a RAM, an EEPROM, etc. The control unit 106 controls the operations of the BTI / F 102 and the Wi-Fi I / F 104.

[0043] The imaging device 200 includes a camera 202, a BTI / F 204, and a control unit 206. As shown in FIG. 2, the imaging device 200 is attached to a position above the dishwasher / dryer 10, such as on the range hood 4 or the ceiling 6. The camera 202 can capture an image of the inside of the storage compartment 14 from above when the storage compartment 14 of the dishwasher / dryer 10 is pulled out from the main body 12. The BTI / F 204 shown in FIG. 1 is a short-range wireless I / F for performing BT communication with an external device. The imaging device 200 can communicate with the dishwasher / dryer 10 and the relay device 100 via the BTI / F 204. The control unit 206 includes a CPU, a ROM, a RAM, an EEPROM, and the like. The control unit 206 controls the operations of the camera 202 and the BTI / F 204.

[0044] Server device 300 can communicate with relay device 100 via Internet 150. Server device 300 is managed by the manufacturer of dishwasher / dryer 10. Server device 300 includes input device 302, output device 304, communication I / F 306, and control device 308. Input device 302 is a device for inputting, for example, a mouse, a keyboard, etc. Output device 304 is a device for outputting, for example, a display, a speaker, etc. An operator of server device 300 can operate server device 300 via input device 302 and output device 304. Communication I / F 306 is a wired or wireless I / F for communication via Internet 150. Control device 308 includes a CPU, ROM, RAM, HDD, SSD, etc. Control device 308 controls the operations of input device 302, output device 304, and communication I / F 306. Control device 308 includes memory 310. The control device 308 controls the operation of the server device 300 in accordance with a program 312 stored in a memory 310. The control device 308 will be described in detail later.

[0045] The control device 60 of the dishwasher / dryer 10 will be described in detail with reference to Fig. 4. The control device 60 loads a program 66 stored in memory 64 into the RAM, and uses a CPU to interpret and execute the program 66 loaded into the RAM. As a result, the control device 60 is configured as a computer including, as software modules, a dish placement image acquisition unit 401, an image analysis unit 402, a resolution conversion unit 403, a dish related information acquisition unit 404, a dish placement state estimation unit 405, a washing device control unit 406, a dish stain information acquisition unit 407, and a drying device control unit 408.

[0046] The dish placement image acquisition unit 401 acquires a captured image 409 from the camera 202 of the imaging device 200. The captured image 409 shows a state in which a user places dishes 19 in the dish basket 61. The captured image 409 may be a still image captured at a certain point in time, a set of multiple still images captured over time, or a video captured over a predetermined period of time.

[0047] The image analysis unit 402 performs image analysis based on the captured image 409. The tableware related information acquisition unit 404 acquires tableware related information 410, which is information related to the tableware 19, from the result of the image analysis in the image analysis unit 402. The tableware related information 410 is, for example, information about the size, shape, material, etc. of the tableware 19. In addition, the tableware stain information acquisition unit 407 acquires tableware stain information 411, which is information about stains on the tableware 19, from the result of the image analysis in the image analysis unit 402. The tableware stain information 411 is, for example, information about the type of stain adhering to the tableware 19, the degree of staining, etc.

[0048] The resolution conversion section 403 reduces the resolution of the captured image 409. As a result, the resolution conversion section 403 generates a low-resolution captured image 412.

[0049] The tableware placement state estimation unit 405 inputs a low-resolution captured image 412 obtained by reducing the resolution of the captured image 409 to a trained learning device (neural network 413) that has performed machine learning to estimate the tableware placement state, which is the placement state of the tableware 19. The tableware related information 410 is also input as supplementary information 414. As a result, the tableware placement state estimation unit 405 acquires tableware placement state information 420 from the learning device. The tableware placement state information is, for example, information about the placement position and placement angle of the tableware 19 in the tableware basket 61. By inputting the low-resolution captured image 412 instead of the captured image 409 to the neural network 413, the amount of calculation of the arithmetic processing of the neural network 413 can be reduced, and the calculation load can be reduced. Note that the resolution reduction process by the resolution conversion unit 403 may be omitted. In this case, the tableware placement state estimation unit 405 may input the captured image 409 to the learning device (neural network 413).

[0050] The neural network 413 is configured by combining multiple types of neural networks. For example, the neural network 413 is divided into four parts: a fully connected neural network 415, a convolutional neural network 416, a connection layer 417, and an LSTM network 418. The fully connected neural network 415 and the convolutional neural network 416 are arranged in parallel on the input side, and the fully connected neural network 415 receives supplementary information 414, and the convolutional neural network 416 receives a low-resolution captured image 412. The connection layer 417 connects the outputs of the fully connected neural network 415 and the convolutional neural network 416. The LSTM network 418 receives the output from the connection layer 417 and outputs tableware placement state information 420.

[0051] The fully connected neural network 415 is a so-called multi-layered neural network, and includes, in order from the input side, an input layer 415a, an intermediate layer (hidden layer) 415b, and an output layer 415c. However, the number of layers of the fully connected neural network 415 is not limited to this example, and may be appropriately selected depending on the embodiment.

[0052] Each layer 415a, 415b, 415c of the fully connected neural network 415 includes one or more neurons (nodes). The number of neurons included in each layer may be set appropriately depending on the embodiment. Each neuron included in each layer is connected to all neurons included in the adjacent layer, thereby forming the fully connected neural network 415. A weight (connection weight) is set appropriately for each connection.

[0053] The convolutional neural network 416 is a forward propagation type neural network having a structure in which convolutional layers 416a and pooling layers 416b are alternately connected. In the convolutional neural network 416 according to the present embodiment, a plurality of convolutional layers 416a and pooling layers 416b are alternately arranged on the input side. The output of the pooling layer 416b arranged on the most output side is input to the fully connected layer 416c, and the output of the fully connected layer 416c is input to the output layer 416d.

[0054] The convolution layer 416a is a layer that performs the convolution operation of an image. The convolution of an image corresponds to a process of calculating the correlation between an image and a predetermined filter. Therefore, by performing the convolution of an image, for example, a shading pattern similar to the shading pattern of the filter can be detected from the input image.

[0055] The pooling layer 416b is a layer that performs a pooling process, which discards some of the information from positions where the image had a strong response to the filter, and eliminates the effects of minute position changes of features that appear in the image.

[0056] The fully connected layer 416c is a layer that connects all neurons between adjacent layers. That is, each neuron included in the fully connected layer 416c is connected to all neurons included in the adjacent layer. The fully connected layer 416c may be composed of two or more layers. In addition, the number of neurons included in the fully connected layer 416c may be appropriately set according to the embodiment.

[0057] The output layer 416d is a layer disposed on the most output side of the convolutional neural network 416. The number of neurons included in the output layer 416d may be set appropriately depending on the embodiment. Note that the configuration of the convolutional neural network 416 is not limited to this example, and may be set appropriately depending on the embodiment.

[0058] The connection layer 417 is disposed between the fully connected neural network 415 and the convolutional neural network 416 and the LSTM network 418. The connection layer 417 connects the output from the output layer 415c of the fully connected neural network 415 and the output from the output layer 416d of the convolutional neural network 416. The output of the connection layer 417 is input to the LSTM network 418. The number of neurons included in the connection layer 417 may be set appropriately depending on the number of outputs of the fully connected neural network 415 and the convolutional neural network 416.

[0059] The LSTM network 418 is a recurrent neural network including an LSTM block 418b. A recurrent neural network is a neural network having an internal loop, such as a path from an intermediate layer to an input layer. The LSTM network 418 has a structure in which the intermediate layer of a general recurrent neural network is replaced with the LSTM block 418b.

[0060] The LSTM network 418 includes, in order from the input side, an input layer 418a, an LSTM block 418b, and an output layer 418c, and has a path from the LSTM block 418b back to the input layer 418a in addition to a forward propagation path. The number of neurons included in the input layer 418a and the output layer 418c may be set appropriately depending on the embodiment.

[0061] The LSTM block 418b is a block that includes an input gate and an output gate and is configured to be able to learn the timing of storing and outputting information. The LSTM block 418b may also include a forget gate that adjusts the timing of forgetting information. The configuration of the LSTM network 418 can be appropriately set according to the embodiment.

[0062] A threshold is set for each neuron, and basically, the output of each neuron is determined depending on whether the sum of the products of each input and each weight exceeds the threshold. The tableware placement state estimation unit 405 inputs supplementary information 414 to a fully connected neural network 415, and inputs a low-resolution captured image 412 to a convolutional neural network 416. Then, the neural network 413 performs a firing determination for each neuron included in each layer, starting from the input side. In this way, the neural network 413 performs arithmetic processing on the input. The tableware placement state estimation unit 405 obtains an output value corresponding to tableware placement state information 420 from the LSTM network 418.

[0063] Information indicating the configuration of such neural network 413 (e.g., the number of layers in each network, the number of neurons in each layer, the connection relationships between neurons, and the transfer functions of each neuron), the weights of the connections between each neuron, and the thresholds of each neuron are included in learning result data 419. The tableware placement state estimation unit 405 refers to the learning result data 419 to set up the neural network 413.

[0064] The washing device control unit 406 sets the washing conditions for the washing operation based on the tableware placement state information 420 obtained from the tableware placement state estimation unit 405 and the tableware stain information 411 obtained from the tableware stain information acquisition unit 407. The washing conditions include, for example, the water temperature, water spray range, water spray direction, water spray pressure, operation time in the pre-wash operation, the water temperature, water spray range, water spray direction, water spray pressure, the amount of detergent dispensed by the detergent dispenser mechanism 71, operation time, the water temperature, water spray range, water spray direction, water spray pressure, operation time, and the number of rinses in the rinse operation. The washing device control unit 406 may set the washing conditions for the washing operation based only on the tableware placement state information 420 without using the tableware stain information 411. And / or the washing device control unit 406 may set the washing conditions for the washing operation based also on the tableware-related information 410 obtained from the tableware-related information acquisition unit 404. The washing device control unit 406 controls the washing device 76 based on the set washing conditions.

[0065] The drying device control unit 408 sets the drying conditions for the drying operation based on the tableware placement state information 420 obtained from the tableware placement state estimation unit 405. The drying conditions include, for example, the volume of the drying air during the drying operation, the temperature of the drying air, the inflow position of the drying air into the storage 14, the inflow direction of the drying air into the storage 14, the outflow position of the drying air from the storage 14, the operation time, etc. The drying device control unit 408 may set the drying conditions for the drying operation based on the tableware-related information 410 in addition to the tableware placement state information 420. The drying device control unit 408 controls the drying device 77 based on the set drying conditions.

[0066] The control device 308 of the server device 300 will be described in detail below with reference to Fig. 5. The control device 308 loads a program 312 stored in a memory 310 onto a RAM, and interprets and executes the program 312 loaded onto the RAM by a CPU. As a result, the control device 308 is configured as a computer including a learning data acquisition unit 321 and a learning processing unit 322 as software modules.

[0067] The learning data acquisition unit 321 acquires a set of a low-resolution captured image 323 obtained by reducing the resolution of a captured image 409 acquired from the camera 202 of the imaging device 200, supplementary information 324 consisting of tableware-related information 410, and tableware placement state information 325 corresponding to the low-resolution captured image 323 and supplementary information 324 as learning data 326. The low-resolution captured image 323 and supplementary information 324 correspond to the low-resolution captured image 412 and supplementary information 414 described with reference to FIG. 4, respectively, and are used as input data. The tableware placement state information 325 corresponds to the tableware placement state information 420 in FIG. 4, and is used as teacher data (correct answer data). When the learning processing unit 322 receives the low-resolution captured image 323 and supplementary information 324, it performs machine learning of a learner (neural network 327) so as to output an output value corresponding to the tableware placement state information 325.

[0068] The learning data 326 can be created by linking tableware placement images for various situations in which the user places tableware 19 in the tableware basket 61 with tableware placement states that indicate the placement state of the tableware 19 in the tableware basket 61 at that time. At this time, the tableware placement images for the various situations have the same resolution as the low-resolution captured image 412 in FIG. 4.

[0069] The tableware-related information 410 included in the supplementary information 324 can be obtained by analyzing tableware placement images of various situations in which the user places tableware 19 in the tableware basket 61, and extracting information about the size, shape, material, etc. of the tableware 19. The learning data 326 can also be created by linking this extracted information with the tableware placement state.

[0070] The learning data 326 may be created manually by an operator using the input device 302 (see FIG. 1), or automatically by the processing of the program 312 (see FIG. 1). The learning data 326 may be collected as needed from the control device 60 of the dishwasher / dryer 10. The learning data 326 may also be created by an information processing device other than the server device 300. The number of data items in the learning data 326 may be appropriately determined depending on the embodiment so that machine learning of the neural network 327 can be performed.

[0071] The control device 308 includes a neural network 327. The neural network 327 of the control device 308 includes a fully connected neural network 328, a convolutional neural network 329, a connection layer 330, and an LSTM network 331. The fully connected neural network 328, the convolutional neural network 329, the connection layer 330, and the LSTM network 331 of the control device 308 are configured similarly to the fully connected neural network 415, the convolutional neural network 416, the connection layer 417, and the LSTM network 418 of the control device 60, respectively.

[0072] The learning processing unit 322 constructs a neural network 327 so that when supplementary information 324 is input to a fully connected neural network 328 and a low-resolution captured image 323 is input to a convolutional neural network 329, an output value corresponding to tableware placement state information 325 is output from an LSTM network 331.

[0073] Specifically, first, the learning processing unit 322 prepares a neural network 327. The configuration of the prepared neural network 327, the initial values ​​of the weights of the connections between the neurons, and the initial values ​​of the thresholds of the neurons may be provided by a template or by input by an operator. When re-learning is performed, the control device 308 may prepare the neural network 327 based on the learning result data 419 to be re-learned.

[0074] Next, the learning processing unit 322 uses the low-resolution captured image 323 and the supplementary information 324 included in the learning data 326 as input data, and the tableware placement state information 325 as teacher data (correct answer data), to perform learning processing of the neural network 327. For the learning processing of the neural network 327, a stochastic gradient descent method or the like may be used.

[0075] For example, the learning processing unit 322 inputs the supplementary information 324 to the input layer of the fully connected neural network 328, and inputs the low-resolution captured image 323 to the convolutional layer arranged on the most input side of the convolutional neural network 329. Then, the learning processing unit 322 performs a firing determination for each neuron included in each layer, starting from the input side. As a result, the learning processing unit 322 obtains an output value from the output layer of the LSTM network 331. Next, the learning processing unit 322 calculates the error between the output value obtained from the output layer of the LSTM network 331 and the value corresponding to the tableware placement state information 325. Next, the learning processing unit 322 calculates the error of each of the connection weights between the neurons and the threshold value of each neuron by using the calculated output value error by a diachronic error backpropagation method. Then, the learning processing unit 322 updates the value of each of the connection weights between the neurons and the threshold value of each neuron based on the calculated errors.

[0076] The learning processing unit 322 repeats this series of processes for each piece of learning data 326 until the output value output from the neural network 327 matches the value corresponding to the tableware placement state information 325. In this way, the learning processing unit 322 can construct a neural network 327 that outputs an output value corresponding to the tableware placement state information 325 when the low-resolution captured image 323 and the supplementary information 324 are input.

[0077] Then, the learning processing unit 322 stores information indicating the configuration of the constructed neural network 327, the weights of the connections between the neurons, and the threshold value of each neuron in the memory 310 as learning result data 419. The learning processing unit 322 periodically transmits the learning result data 419 stored in the memory 310 to the control device 60 via the Internet 150. As a result, the learning result data 419 in the control device 60 is periodically updated.

[0078] The operation of the dishwasher / dryer 10 will be described below with reference to Fig. 6. When the user presses the start switch on the operation panel 16 with the container 14 pulled out of the main body 12, the control device 60 executes the process shown in Fig. 6.

[0079] In S2, the control device 60 causes the tableware placement image acquisition section 401 to acquire a tableware placement image from the camera 202 of the imaging device 200. The acquired tableware placement image is treated as a captured image 409 in subsequent processing.

[0080] In S4, the control device 60 acquires tableware related information 410 from the tableware placement image using the image analysis unit 402 and the tableware related information acquisition unit 404. The acquired tableware related information 410 is treated as supplementary information 414 in subsequent processing.

[0081] In S6, the control device 60 causes the image analysis section 402 and the dish stain information acquisition section 407 to acquire dish stain information 411 from the dish placement image.

[0082] In S8, the control device 60 obtains an image in which the resolution of the image with tableware placed on it has been reduced by the resolution conversion section 403. The image with tableware placed on it that has been reduced in resolution is treated as a low-resolution captured image 412 in subsequent processing.

[0083] In S10, the control device 60 estimates the tableware placement state by the tableware placement state estimation unit 405. Specifically, the tableware placement state estimation unit 405 inputs the low-resolution captured image 412 and the supplementary information 414 to the neural network 413. The neural network 413 has acquired learning result data 419 from the control device 308 of the server device 300 via the Internet 150. In other words, the neural network 413 corresponds to a trained learning device that has performed machine learning to estimate the tableware placement state based on the tableware placement image. The neural network 413 provides an output including tableware placement state information 420 in response to an input including the low-resolution captured image 412 and the supplementary information 414.

[0084] In S12, the control device 60 executes a pre-wash operation process by the washing device control section 406. In the pre-wash operation process, the washing device control section 406 sets operation conditions such as water temperature, water spray range, water spray direction, water spray pressure, and operation time for the pre-wash operation based on the dish placement state information 420 acquired in S10, the dish-related information 410 acquired in S4, and the dish stain information 411 acquired in S6. Note that the setting of the operation conditions for the pre-wash operation only needs to be based on the dish placement state information 420, and does not necessarily have to be based on the dish-related information 410 or the dish stain information 411. Then, the washing device control section 406 controls the washing device 76 according to the set operation conditions.

[0085] Specifically, the washing device control unit 406 first rotates the washing pump 27 in the reverse direction to drain the water in the container 14, and then stops the washing pump 27. After that, the washing device control unit 406 opens the water supply valve 41 to supply water into the container 14, and closes the water supply valve 41 when the switch 49 of the water level detector 45 is turned ON. After that, the washing device control unit 406 rotates the washing pump 27 in the forward direction and turns the heater 30 ON. As a result, the water in the container 14 is heated by the heater 30, high-temperature water is sprayed from the washing nozzle 20, and the tableware 19 in the container 14 is pre-washed. At this time, the washing device control unit 406 controls the operation of the switching valve 21, the washing pump 27, and the heater 30 according to the set water temperature, water spray range, water spray direction, and water spray pressure. After the set operation time has elapsed, the washing device control unit 406 stops the washing pump 27, turns off the heater 30, and ends the pre-wash operation.

[0086] In S14, the control device 60 executes a main wash operation process by the washing device control section 406. In the main wash operation process, the washing device control section 406 sets operation conditions such as the water temperature, water spray range, water spray direction, water spray pressure, amount of detergent dispensed by the detergent dispenser mechanism 71, and operation time for the main wash operation based on the dish placement state information 420 acquired in S10, the dish-related information 410 acquired in S4, and the dish stain information 411 acquired in S6. Note that the operation conditions for the main wash operation may be set based on the dish placement state information 420, and may not be based on the dish-related information 410 or the dish stain information 411. Then, the washing device control section 406 controls the washing device 76 according to the set operation conditions.

[0087] Specifically, the washing device control unit 406 first rotates the washing pump 27 in the reverse direction to drain the water in the storage chamber 14, and then stops the washing pump 27. Thereafter, the washing device control unit 406 opens the water supply valve 41 to supply water into the storage chamber 14, and closes the water supply valve 41 when the switch 49 of the water level detector 45 is turned ON. Thereafter, the washing device control unit 406 opens the detergent injection valve 73 to inject a set amount of detergent into the storage chamber 14. Then, the washing device control unit 406 rotates the washing pump 27 in the forward direction and turns the heater 30 ON. As a result, the water in the storage chamber 14 is heated by the heater 30, high-temperature water is sprayed from the washing nozzle 20, and the tableware 19 in the storage chamber 14 is washed. At this time, the washing device control unit 406 controls the operation of the switching valve 21, the washing pump 27, and the heater 30 according to the set water temperature, water spray range, water spray direction, and water spray pressure. When the set operation time has elapsed, the cleaning device control unit 406 stops the cleaning pump 27 and turns off the heater 30, thereby completing the main cleaning operation.

[0088] In S16, the control device 60 executes a rinsing operation process by using the washing device control section 406. In the rinsing operation process, the washing device control section 406 sets operation conditions such as the water temperature, water spray range, water spray direction, water spray pressure, operation time, and number of rinses for the rinsing operation based on the dish placement state information 420 acquired in S10, the dish-related information 410 acquired in S4, and the dish stain information 411 acquired in S6. Note that the operation conditions for the rinsing operation may be set based on the dish placement state information 420, and need not be based on the dish-related information 410 or the dish stain information 411. Then, the washing device control section 406 controls the washing device 76 according to the set operation conditions.

[0089] Specifically, the washing device control unit 406 first rotates the washing pump 27 in the reverse direction to drain the water in the container 14, and then stops the washing pump 27. After that, the washing device control unit 406 opens the water supply valve 41 to supply water into the container 14, and closes the water supply valve 41 when the switch 49 of the water level detector 45 is turned ON. After that, the washing device control unit 406 rotates the washing pump 27 in the forward direction and turns the heater 30 ON. As a result, the water in the container 14 is heated by the heater 30, and high-temperature water is sprayed from the washing nozzle 20 to rinse the tableware 19 in the container 14. At this time, the washing device control unit 406 controls the operation of the switching valve 21, the washing pump 27, and the heater 30 according to the set water temperature, water spray range, water spray direction, and water spray pressure. After the set operation time has elapsed, the washing device control unit 406 stops the washing pump 27 and turns the heater 30 OFF to end one rinsing cycle. Then, the cleaning device control unit 406 repeats the above process the set number of rinses, then drives the cleaning pump 27 to rotate in the reverse direction to drain the water from the storage compartment 14, and then stops the cleaning pump 27 to end the rinsing operation.

[0090] In S18, the control device 60 executes a drying operation process by the drying device control section 408. In the drying operation process, the drying device control section 408 sets the operating conditions, such as the volume of the drying air, the temperature of the drying air, the inflow position of the drying air into the storage 14, the inflow direction of the drying air into the storage 14, the outflow position of the drying air from the storage 14, and the operating time, in the drying operation, based on the tableware placement state information 420 acquired in S10 and the tableware-related information 410 acquired in S4. Note that the setting of the operating conditions for the drying operation only needs to be based on the tableware placement state information 420, and does not necessarily have to be based on the tableware-related information 410. Then, the drying device control section 408 controls the drying device 77 in accordance with the set operating conditions.

[0091] Specifically, the drying device control unit 408 drives and rotates the drying fan 52 and turns on the heater 30. This causes high-temperature air to flow through the storage chamber 14 to dry the tableware 19 in the storage chamber 14. At this time, the drying device control unit 408 controls the operations of the exhaust damper 17, the drying fan 52, the air intake damper 65, and the air intake louver 67 according to the set air volume and temperature of the drying air, the inflow position of the drying air into the storage chamber 14, the inflow direction of the drying air into the storage chamber 14, and the outflow position of the drying air from the storage chamber 14. When the set operation time has elapsed, the drying device control unit 408 stops the drying fan 52 and turns off the heater 30 to end the drying operation.

[0092] The process executed by the server device 300 to update the learning result data 419 of the control device 60 will be described below with reference to FIG.

[0093] In S22, the control device 308 causes the learning data acquisition unit 321 to acquire a set of the low-resolution captured image 323, the supplemental information 324, and the tableware placement state information 325 as learning data 326.

[0094] In S24, the control device 308 causes the learning processing unit 322 to perform learning processing of the neural network 327 using the learning data 326 acquired in S22, so that when the low-resolution captured image 323 and the supplementary information 324 are input, an output value corresponding to the tableware placement state information 325 is output. The learning processing unit 322 acquires, as learning result data 419, information indicating the configuration of the neural network 327 obtained by the learning processing, the connection weights between the neurons, and the threshold value of each neuron.

[0095] In S26, the control device 308 causes the learning processing unit 322 to transmit the acquired learning result data 419 to the control device 60 of the dishwasher / dryer 10. As a result, the learning result data 419 of the control device 60 is updated.

[0096] The above describes a configuration in which the opening 63a through which the dry air flows into storage compartment 14 is switched by switching between opening and closing of air intake damper 65 during the drying operation of dishwasher-dryer 10. Alternatively, for example, a three-way valve may be used to switch opening 63a through which the dry air flows into storage compartment 14.

[0097] In the above, a configuration has been described in which opening 18a through which dry air flows out from storage compartment 14 is switched by switching between opening and closing of exhaust damper 17 during the drying operation of dishwasher-dryer 10. Alternatively, opening 18a through which dry air flows out from storage compartment 14 may be switched using, for example, a three-way valve.

[0098] In the above, a configuration has been described in which dishwasher / dryer 10 connects to Internet 150 via relay device 100. In contrast, if dishwasher / dryer 10 has a Wi-Fi I / F, dishwasher / dryer 10 may connect to Internet 150 without going through relay device 100. Alternatively, if imaging device 200 has a Wi-Fi I / F, dishwasher / dryer 10 may connect to Internet 150 via imaging device 200.

[0099] In the above, we have described the case where the image of the tableware placement is captured by the imaging device 200 attached to the range hood 4 or the ceiling 6. However, alternatively, the image of the tableware placement may be captured by a mobile information terminal with a camera, such as a smartphone, which is capable of communicating with the dishwasher / dryer 10 via wired or wireless communication.

[0100] The above describes the case where the dishwasher / dryer 10 is capable of performing a pre-wash operation, a main wash operation, a rinsing operation and a drying operation of the dishes 19, but the dishwasher / dryer 10 may also be a dishwasher capable of performing at least one of the pre-wash operation, the main wash operation and / or the rinsing operation of the dishes 19, or a dishdryer capable of performing only a drying operation of the dishes 19.

[0101] Although the embodiments of the present invention have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and changes of the specific examples exemplified above.

[0102] The technical elements described in this specification or drawings have technical utility either alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. In addition, the technologies illustrated in this specification or drawings can achieve multiple objectives simultaneously, and achieving one of these objectives is itself technically useful. [Explanation of symbols]

[0103] 2: System 3: System kitchen 4: Range hood 6: Ceiling 10: Dishwasher / dryer 12: Main unit 14: Cleaning tank 14a:Aperture 15: Door 16: Operation panel 17: Exhaust damper 18: Exhaust route 18a:Aperture 18b: Exhaust port 19: Tableware 20: Cleaning nozzle 20a: Injection nozzle 20b: Nozzle 20c: Nozzle 21: Switching valve 27: Cleaning pump 28: Impeller 30: Heater 31: Suction recess 32: Suction passage 33: Back wall 34: Drain hose 36: Drainage channel 38: Water level path 39: Bottom 40: Water supply hose 41: Water supply valve 42: 1st water supply channel 43:Second water supply channel 44: Water level path 45: Water level detector 46: Water level chamber 47: Float 48: Bar 49: Switch 50: Rotational speed sensor 51: Back wall 52: Drying fan 53: Fan 55: Thermistor 56: Lid 58: Water level 60: Control device 61: Tableware basket 62 :BTI / F 63: Air supply route 63a:Aperture 64: Memory 65: Air supply damper 66: Program 67: Air supply louver 70: Sealing material 71: Detergent dispenser mechanism 72: Detergent tank 73: Detergent dispenser valve 74: Detergent supply route 75: Detergent injection path 76: Cleaning equipment 77:Drying equipment 88: Water supply pipe 100: Relay device 102 :BTI / F 104: Wi-Fi I / F 106: Control unit 150: Internet 200: Imaging device 202: Camera 204 :BTI / F 206: Control section 300: Server device 302: Input device 304: Output device 306: Communication I / F 308: Control device 310: Memory 312: Program 321: Learning data acquisition unit 322: Learning processing unit 323: Low resolution image 324: Supplementary information 325: Tableware placement status information 326: Training data 327: Neural Networks 328: Fully connected neural network 329: Convolutional Neural Networks 330: Bonding layer 331: LSTM network 401: Tableware placement image acquisition unit 402: Image analysis unit 403: Resolution conversion section 404: Tableware-related information acquisition unit 405: Tableware placement state estimation unit 406: Cleaning device control section 407: Dish stain information acquisition unit 408: Drying device control unit 409: Captured image 410: Tableware related information 411: Dish stain information 412: Low resolution image 413: Neural Networks 414: Supplementary information 415: Fully connected neural network 415a: Input layer 415b: Middle class 415c: Output layer 416: Convolutional Neural Networks 416a: Convolutional layer 416b: Pooling layer 416c: Fully connected layer 416d: Output layer 417: Bonding layer 418: LSTM network 418a: Input layer 418b: LSTM block 418c: Output layer 419: Learning results data 420: Tableware placement status information

Claims

1. The container includes a placement member on which tableware is placed, and a container for storing the tableware; An imaging device configured to capture a tableware placement image showing a state in which a user places the tableware on the placement member; A dishware appliance comprising a control device, The control device includes: a tableware placement image acquisition unit that acquires the tableware placement image from the imaging device; a tableware related information acquisition unit that acquires tableware related information, which is information related to the tableware, by image analysis based on the tableware placement image; The tableware equipment includes a tableware placement state estimation unit that estimates the tableware placement state by inputting the tableware placement image and the tableware-related information into a trained learning device that has performed machine learning to estimate the tableware placement state, which is the state in which the tableware is placed, based on the tableware placement image and the tableware-related information, and executing calculation processing of the trained learning device.

2. The dishwasher further includes a washing device for washing the tableware stored in the storage unit. The tableware equipment according to claim 1 , wherein the control device further comprises a washing device control unit that sets washing conditions based on the tableware placement state and controls the washing device based on the washing conditions.

3. The control device further includes a dish stain information acquisition unit that acquires dish stain information, which is information regarding stains on the dishes, by image analysis based on the dish placement image, The tableware equipment according to claim 2 , wherein the washing device control unit sets the washing conditions based on the tableware stain information as well, and controls the washing device based on the washing conditions.

4. The dishwasher further includes a drying device for drying the tableware stored in the storage unit. The tableware fixture according to claim 1 , further comprising a drying device control unit that sets drying conditions based on the tableware placement state and controls the drying device based on the drying conditions.

5. The container includes a placement member on which tableware is placed, and a container for storing the tableware; An imaging device configured to capture a tableware placement image showing a state in which a user places the tableware on the placement member; 1. A method of operating a dishware appliance comprising a control device, comprising: The control device acquires the tableware placement image from the imaging device; The control device acquires tableware-related information, which is information related to the tableware, by image analysis based on the tableware placement image; An operating method in which the control device inputs the dish placement image and the dish-related information into a trained learning device that has performed machine learning to estimate the dish placement state, which is the state in which the dishes are placed, based on the dish placement image and the dish-related information, and executes calculation processing of the trained learning device to estimate the dish placement state.

6. The tableware equipment further includes a washing device for washing the tableware stored in the storage compartment, The operating method according to claim 5 , further comprising the step of: the control device setting washing conditions based on the tableware placement state, and controlling the washing device based on the washing conditions.

7. The control device further includes a step of acquiring dish stain information, which is information regarding stains on the dish, by image analysis based on the dish placement image, The operating method of claim 6 , wherein in the step of controlling the operation of the washing device, the control device sets the washing conditions based also on the dish stain information, and controls the washing device based on the washing conditions.

8. The tableware equipment further includes a drying device for drying the tableware stored in the storage compartment, The operating method according to claim 5 , further comprising a step of the control device setting drying conditions based on the tableware placement state and controlling the drying device based on the drying conditions.

9. The container includes a placement member on which tableware is placed, and a container for storing the tableware; An imaging device configured to capture a tableware placement image showing a state in which a user places the tableware on the placement member; A learning device for a tableware appliance comprising a control device, the learning device comprising: a learning data acquisition unit that acquires learning data including the tableware placement image acquired from the imaging device, tableware-related information that is associated with the tableware placement image and is acquired by image analysis based on the tableware placement image, and a tableware placement state that is associated with the tableware placement image and is the placement state of the tableware; The learning device is provided with a learning processing unit that performs machine learning of a learning device to obtain an output including the dish placement state in response to an input including the dish placement image and the dish-related information.

10. The container includes a placement member on which tableware is placed, and a container for storing the tableware; An imaging device configured to capture a tableware placement image showing a state in which a user places the tableware on the placement member; A learning method for a tableware appliance comprising a control device, comprising: A learning device acquires learning data including the tableware placement image acquired from the imaging device, tableware-related information associated with the tableware placement image and acquired by image analysis based on the tableware placement image, and a tableware placement state associated with the tableware placement image and indicating the placement state of the tableware; The learning method includes a step in which the learning device performs machine learning of a learning module to obtain an output including the dish placement state in response to an input including the dish placement image and the dish-related information.

Images