Control system, learning system, device control method and program

Abstract

To provide a control system, a learning system, an apparatus control method, and a program that can appropriately control an apparatus by a gesture operation.SOLUTION: An NN update unit 216 updates a neural network that is selected by a selection unit 212, on the basis of gesture image information and operation identification information corresponding to a determination value calculated from the gesture image information, every time when an image acquisition unit 211 acquires the gesture image information. Every time when a preset NN update period arrives, the NN update unit 216 specifies a neural network that has a least calculation amount among the neural networks each corresponding to each of air conditioners 1A and 1B which are stored by an NN storage unit 231, and updates all of weight coefficient information on the neural network corresponding to each of the air conditioners 1A and 1B to weight coefficient information on the specified neural network.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a control system, a learning system, an equipment control method, and a program. [Background technology]

[0002] There has been proposed an operation determination device that includes an indirect operation unit that generates indirect operation data based on the trajectory of the electronic key when an operator performs an indirect operation to operate an operation target while carrying the electronic key, and a control unit that determines that an indirect operation has been performed if teacher data, which is a template of a graph of the trajectory of the electronic key, matches the indirect operation data within an allowable range. Here, if the control unit determines that an indirect operation has not been performed because the indirect operation data deviates from the allowable range of the teacher data, but a direct operation that can give the same instruction as the indirect operation is performed within a preset period, it determines that the operator has performed the operation again and learns so that the indirect operation can be performed using the indirect data, and updates the teacher data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-197450 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, in an operation determination device that determines the operation content corresponding to an indirect operation, i.e., a gesture operation, as described in Patent Document 1, there is a demand for improving the recognition accuracy of the gesture operation so that the operation target can be appropriately operated.

[0005] The present disclosure has been made in consideration of the above-mentioned reasons, and aims to provide a control system, a learning system, a device control method, and a program that are capable of appropriately controlling a device through gesture operations. [Means for solving the problem]

[0006] In order to achieve the above object, the control system according to the present disclosure comprises: a neural network storage unit that stores a neural network for identifying one type of operation content corresponding to a gesture image of the same type from gesture image information showing a gesture of the same type of user corresponding to each of a plurality of devices; an imaging device provided in each of the plurality of devices and configured to image a user of each of the plurality of devices; an image acquisition unit that acquires, from the imaging device, gesture image information indicating a gesture of the user captured by the imaging device; a selection unit that selects, from the neural networks stored in the neural network storage unit, a neural network corresponding to an apparatus provided with an imaging device that has generated the gesture image information acquired by the image acquisition unit; a neural network calculation unit that calculates a judgment value corresponding to the gesture image information acquired by the image acquisition unit, using the neural network selected by the selection unit; an operation identification unit that identifies operation identification information that identifies an operation on the device associated with the determination value calculated by the neural network calculation unit; a device control unit that controls a device provided with an imaging device that generates the gesture image information, based on the operation identification information identified by the operation identification unit; The neural network update unit updates the neural network selected by the selection unit based on the gesture image information and operation identification information corresponding to a judgment value calculated from the image information each time the image acquisition unit acquires the gesture image information, and identifies, each time a preset neural network update time arrives, a neural network requiring the least amount of calculations from among the neural networks corresponding to each of the plurality of devices stored in the neural network storage unit, and updates all of the neural networks corresponding to each of the plurality of devices to the identified neural network. Effect of the Invention

[0007] According to the present disclosure, the neural network update unit updates the neural network selected by the selection unit based on the gesture image information and the operation identification information corresponding to the judgment value calculated from the image information every time the image acquisition unit acquires gesture image information. Also, the neural network update unit identifies the neural network with the smallest amount of calculation among the neural networks corresponding to each of the multiple devices stored in the neural network storage unit every time a preset neural network update time arrives, and updates all the neural networks corresponding to each of the multiple devices to the identified neural network. As a result, the neural networks corresponding to each of the multiple devices are periodically updated to the neural network with the smallest amount of calculation, i.e., the most computationally efficient, so that it is possible to shorten the time required to identify the operation content corresponding to the gesture made by the user and improve the accuracy of identifying the operation content. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic configuration diagram of a control system according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram showing a hardware configuration of a control system according to an embodiment. [Diagram 3]A block diagram showing the functional configuration of an air conditioner according to an embodiment. [Figure 4] FIG. 1A is a diagram showing an example of an image showing a gesture of a user; FIG. 1B is a diagram showing an example of an image showing a gesture of another user; [Diagram 5] FIG. 1 is a block diagram showing a functional configuration of a cloud server according to an embodiment. [Figure 6] FIG. 1 is an explanatory diagram of the operation of a control system according to an embodiment of the present invention; [Figure 7] FIG. 1 is an explanatory diagram of the operation of a control system according to an embodiment of the present invention; [Figure 8] FIG. 1 is a diagram showing an example of information stored in an operation identification information storage unit according to an embodiment; [Figure 9] FIG. 1 is a sequence diagram showing an operation of a control system according to an embodiment. [Figure 10] FIG. 1 is a sequence diagram showing an operation of a control system according to an embodiment. [Figure 11] FIG. 1 is a sequence diagram showing an operation of a control system according to an embodiment. [Figure 12] A flowchart showing an example of the flow of device control processing executed by an air conditioner according to an embodiment. [Figure 13] A flowchart showing an example of the flow of device control processing executed by an air conditioner according to an embodiment. [Figure 14] 1 is a flowchart showing an example of the flow of an operation determination process executed by a cloud server according to an embodiment. [Figure 15] 1 is a flowchart showing an example of the flow of a weighting coefficient determination process executed by a cloud server according to an embodiment. [Figure 16] 1 is a flowchart showing an example of the flow of an operation determination process executed by a cloud server according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A control system according to an embodiment of the present disclosure will be described below with reference to the drawings. The control system according to the embodiment includes a neural network storage unit, an imaging device, an image acquisition unit, a selection unit, a neural network calculation unit, an operation identification unit, a device control unit, and a neural network update unit. The neural network storage unit stores a neural network for identifying one type of operation content corresponding to a gesture image of the same type from gesture image information indicating a gesture of the same type of user corresponding to each of a plurality of devices. The imaging device is provided in each of the plurality of devices and images the user of each of the plurality of devices. The image acquisition unit acquires, from the imaging device, gesture image information indicating the gesture of the user captured by the imaging device. The selection unit selects, from the neural networks stored in the neural network storage unit, a neural network corresponding to the device provided with the imaging device that generated the gesture image information acquired by the image acquisition unit. The neural network calculation unit calculates a judgment value corresponding to the gesture image information from the gesture image information acquired by the image acquisition unit using the selected neural network. The operation identification unit identifies operation identification information that identifies an operation on a device associated with the determination value calculated by the neural network calculation unit. The device control unit controls a device provided with an imaging device that generates the gesture image information, based on the operation identification information identified by the operation identification unit. The neural network update unit updates the neural network selected by the selection unit, based on the gesture image information and the operation identification information corresponding to the determination value calculated from the image information, every time the image acquisition unit acquires the gesture image information. In addition, the neural network update unit identifies a neural network with the smallest amount of calculation from among the neural networks corresponding to each of the multiple devices stored in the neural network storage unit, and updates all of the neural networks corresponding to each of the multiple devices to the identified neural network, every time a preset neural network update time arrives.

[0010] As shown in FIG. 1, the control system according to the present embodiment includes a plurality of air conditioners 1A, 1B (two in FIG. 1) installed in a building H, and a cloud server 2. Operation devices 6A, 6B for operating the air conditioners 1A, 1B, respectively, are installed in the building H. Here, the air conditioners 1A, 1B are each installed in a room in the building H. Also installed in the building H are a router 82 connected to a local network NW2 constructed in the building H, and a data circuit-terminating device 81 connected to the router 82 and a wide area network NW1. The wide area network NW1 is, for example, the Internet. Also, the local area network NW2 is, for example, a wired LAN (Local Area Network) or a wireless LAN. The data circuit-terminating device 81 is, for example, a modem, a gateway, or the like.

[0011] As shown in FIG. 2, the air conditioners 1A and 1B according to this embodiment have a control unit 100 and an imaging device 181. The air conditioner 1 also has a compressor (not shown) and a blower fan (not shown) that operate based on a control signal input from the control unit 100. The cloud server 2, the imaging devices 181 of the air conditioners 1A and 1B, and the control unit 100 constitute a learning system that updates the weighting coefficients of the neural network, that is, trains the neural network, as described below. The imaging device 181 has a camera using a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The imaging device 181 generates an image signal corresponding to an image captured by the camera and outputs it to an imaging interface 108 of the control unit 100 described below.

[0012] The control unit 100 includes a CPU (Central Processing Unit) 101, a main memory 102, an auxiliary memory 103, a neuro engine 104, a clock 105, a local communication unit 106, a wireless module 107, an imaging interface 108, and a bus 109 that interconnects the above. The main memory 102 is made of a volatile memory and is used as a working area for the CPU 101. The auxiliary memory 103 is made of a non-volatile memory such as a semiconductor flash memory and stores programs for implementing various functions of the control unit 100. The neuro engine 104 is hardware dedicated to arithmetic processing using a neural network having a preset number of nodes and layers. The clock 105 is, for example, a real-time clock. The local communication unit 106 is connected to a local network NW2, and transmits various information notified from the CPU 101 to the local network NW2 and notifies the CPU 101 of various information received from the local network NW2. Wireless module 107 wirelessly communicates with manipulation devices 6A, 6B using a communication method conforming to a short-range wireless communication standard such as Bluetooth (registered trademark), and upon receiving operation information indicating the operation content performed by the user on manipulation devices 6A, 6B from manipulation devices 6A, 6B, notifies CPU 101 of the operation information. Note that wireless module 107 may be one that executes infrared communication with manipulation devices 6A, 6B based on, for example, the IrDA (Infrared Data Association) standard. When an image signal is input from imaging device 181, imaging interface 108 generates image information corresponding to the input and notifies CPU 101 of the image information.

[0013] 3, the CPU 101 reads out a program stored in the auxiliary storage unit 103 into the main storage unit 102 and executes it, thereby functioning as an image acquisition unit 111, an image correction unit 112, an operation reception unit 113, a device control unit 114, an operation identification information transmission unit 115, an operation mode setting unit 116, a user identification unit 117, an image transmission unit 118, and a device setting unit 119. The auxiliary storage unit 103 also has an image storage unit 131, a user information storage unit 132, an operation mode storage unit 133, a device setting storage unit 134, and an operation information storage unit 135. The image storage unit 131 stores gesture image information indicating a gesture of a user captured by the imaging device 181. The image storage unit 131 stores, for example, gesture image information indicating a gesture image GE11 corresponding to the operation of "air conditioner ON", gesture image information indicating a gesture image GE12 corresponding to the operation of "air conditioner OFF", gesture image information indicating a gesture image GE13 corresponding to the operation of "change the temperature setting of the air conditioner", or gesture image information indicating a gesture image GE14 corresponding to the operation of "maintain the temperature setting of the air conditioner", as shown in Fig. 4(A). Alternatively, the image storage unit 131 stores, for example, gesture image information indicating a gesture image GE21 corresponding to the operation of "air conditioner ON", gesture image information indicating a gesture image GE22 corresponding to the operation of "air conditioner OFF", gesture image information indicating a gesture image GE23 corresponding to the operation of "change the temperature setting of the air conditioner", or gesture image information indicating a gesture image GE24 corresponding to the operation of "maintain the temperature setting of the air conditioner", as shown in Fig. 4(B).

[0014] Returning to FIG. 3, the user information storage unit 132 stores image feature information indicating features of the user such as the contour and dimensions that appear in the image of the user captured by the imaging device 181 in association with user identification information for identifying the user. The operation mode storage unit 133 stores operation mode information indicating the operation mode when operating the air conditioners 1A and 1B. Here, the operation modes include a normal operation mode in which the operation settings of the air conditioners 1A and 1B can be changed only by the user's operation on the operation devices 6A and 6B, and a gesture operation mode in which the operation settings of the air conditioners 1A and 1B are changed also in response to the gesture of the user captured by the imaging device 181. The operation modes also include a gesture registration mode for registering a new gesture of the user in the air conditioners 1A and 1B, that is, for updating the correspondence between the operation identification information and the judgment value information stored in the operation identification information storage unit 233 of the cloud server 2 described later. The device setting storage unit 134 stores device setting information indicating device setting parameters of the air conditioners 1A and 1B.

[0015] 3, image acquiring unit 111 acquires gesture image information indicating a gesture image captured by imaging device 181, and stores the acquired image information in image storage unit 131. When operation mode information stored in operation mode storage unit 133 indicates a gesture registration mode, image acquiring unit 111 waits for a waiting time preset based on the time measured by clock unit 105 after operation mode information indicating the gesture registration mode is stored in operation mode storage unit 133, and then acquires gesture image information corresponding to a gesture of a user captured by imaging device 181 from imaging device 181.

[0016] The image correction unit 112 corrects the gesture image information acquired by the image acquisition unit 111. Here, the image correction unit 112 corrects the gesture image information based on imaging conditions such as the performance of the imaging device 181, the installation position of the imaging device 181, the brightness around the imaging device 181, and the background image of the user's gesture image captured by the imaging device 181. For example, the image correction unit 112 roughly sets a boundary between an image portion including the user's gesture image and a background image portion assuming that the user's image is present in the center of the gesture image information, then calculates a new boundary by performing graph cut based on the color distribution of each image portion, and calculates a new boundary by performing graph cut again based on the color distribution of each image portion partitioned by the new boundary, thereby extracting the user's gesture image from the background image. Then, the image correction unit 112 performs angle of view correction based on the installation position of the imaging device 181 on the gesture image information indicating the extracted gesture image of the user, and then performs averaging processing or complementation processing to convert the number of pixels of the gesture image to a preset reference number of pixels based on the number of pixels of the imaging device 181, and then performs brightness correction (e.g., backlight correction) based on the brightness around the imaging device 181 before storing the image in the image storage unit 131.

[0017] When wireless module 107 receives operation information transmitted from operation devices 6A, 6B, operation reception unit 113 accepts the received operation information. Then, if the operation information relates to updating the device setting parameters of air conditioners 1A, 1B, operation reception unit 113 notifies operation identification information transmission unit 115 and device setting unit 119 of the operation information. Furthermore, if the operation information relates to changing the operation mode of air conditioners 1A, 1B, operation reception unit 113 notifies operation mode setting unit 116 of the operation information.

[0018] When the operation information is notified from the operation reception section 113 , the operation mode setting section 116 specifies operation mode information based on the notified operation information, and stores the specified operation mode information in the operation mode storage section 133 .

[0019] The user identification unit 117 extracts features of the user, such as the contour and dimensions, that appear in the image of the user, based on the image information acquired by the image acquisition unit 111. Then, the user identification unit 117 identifies, from the user identification information stored in the user information storage unit 132, user identification information that corresponds to image feature information that shows the same features as the extracted features, as the user identification information of the user of the air conditioners 1A and 1B. The user identification unit 117 also stores the user identification information of the identified users of the air conditioners 1A and 1B in the user information storage unit 132.

[0020] When the operation mode information stored in the operation mode storage unit 133 indicates the gesture operation mode, upon acquiring the operation identification information transmitted from the cloud server 2, the device setting unit 119 calculates device setting parameters based on the acquired operation identification information. Then, the device setting unit 119 stores the device setting information indicating the calculated device setting parameters in the device setting storage unit 134. Furthermore, upon receiving the operation identification information from the operation reception unit 113, the device setting unit 119 calculates device setting parameters based on the operation identification information, and stores the device setting information indicating the calculated device setting parameters in the device setting storage unit 134. Then, the device control unit 114 controls the operation of the compressor and the blower fan based on the device setting information stored in the device setting storage unit 134.

[0021] When the operation identification information transmitting unit 115 acquires operation information from the operation receiving unit 113, it selects operation identification information corresponding to the acquired operation information from the operation identification information stored in the operation information storage unit 135. Then, the operation identification information transmitting unit 115 transmits the selected operation identification information to the cloud server 2, and notifies the image transmitting unit 118 of operation identification information transmission notification information notifying that the operation identification information will be transmitted.

[0022] When the operation mode information stored in the operation mode storage unit 133 indicates the gesture operation mode, and new gesture image information is stored in the image storage unit 131, the image transmission unit 118 transmits the gesture image information, user information corresponding to the gesture image information stored in the user information storage unit 132, and device identification information for identifying the air conditioners 1A and 1B to the cloud server 2. Furthermore, when the operation mode information stored in the operation mode storage unit 133 indicates the gesture registration mode, and the image transmission unit 118 is notified of operation identification information transmission notification information from the operation identification information transmission unit 115, the image transmission unit 118 transmits the newly stored gesture image information stored in the image storage unit 131 and the device identification information for identifying the air conditioners 1A and 1B to the cloud server 2.

[0023] When the operation receiving unit 113 receives a new registration operation for newly registering neural networks corresponding to the air conditioners 1A and 1B in the cloud server 2, the registration request unit 120 identifies the user based on the operation content of the new registration operation. Here, the registration request unit 120 accepts the new registration operation when the user performs a new device registration operation on the operation devices 6A and 6B to register the air conditioners 1A and 1B newly installed in building H, registering device identification information of the newly installed air conditioners 1A and 1B and information about the user. Then, the registration request unit 120 transmits registration request information including the user information of the identified user and the device identification information of the air conditioners 1A and 1B to the cloud server 2.

[0024] Returning to FIG. 2, the cloud server 2 includes a CPU 201, a main memory unit 202, an auxiliary memory unit 203, a clock unit 205, a wide area communication unit 206, and a bus 209 connecting these together. The CPU 201 is, for example, a multi-core processor. The main memory unit 202 is made up of a volatile memory and is used as a working area for the CPU 201. The auxiliary memory unit 203 is made up of a large-capacity non-volatile memory and stores programs for realizing various functions of the cloud server 2. The clock unit 205 is, for example, a real-time clock. The wide area communication unit 206 is connected to the wide area network NW1.

[0025] 5, the CPU 201 functions as an image acquisition unit 211, a selection unit 212, a neural network calculation unit (hereinafter referred to as an "NN calculation unit") 213, an operation identification unit 214, an operation identification information notification unit 215, a neural network update unit (hereinafter referred to as an "NN update unit") 216, and an operation identification information acquisition unit 217. The auxiliary storage unit 203 also has a neural network storage unit (hereinafter referred to as an "NN storage unit") 231, an image storage unit 232, and an operation identification information storage unit 233. The image storage unit 232 stores the gesture image information and user information acquired from the air conditioners 1A and 1B, and the device identification information of the air conditioners 1A and 1B in association with each other.

[0026] The NN storage unit 231 stores a neural network for identifying one type of operation content corresponding to the same type of gesture image from gesture image information indicating the same type of gesture of the user, corresponding to each of the air conditioners 1A and 1B. Specifically, the NN storage unit 231 stores information indicating the structure of a neural network for identifying an operation performed by a user from the gesture image information of the user and information indicating weighting coefficients in the neural network in association with the device identification information of the corresponding air conditioners 1A and 1B. Here, the information indicating the structure of the neural network includes information indicating the number of nodes and layers of the neural network, and weighting coefficients and activation functions corresponding to each node. This neural network has an input layer L10, a hidden layer L20, and an output layer L30 as shown in FIG. 6. The input layer L10 inputs pixel position information and pixel value information included in the gesture image information to the hidden layer L20.

[0027] The hidden layer L20 is composed of N (N is a positive integer) layers including a preset number M[j] of nodes x[j,i] (1≦i≦M[j], M[j] is a positive integer). That is, the hidden layer L20 has a structure in which each node column is connected to each other. Here, the output y[j,i] of each node x[j,i] is expressed by the relational expression of the following formula (1).

number

[0028] Moreover, this neural network is a so-called convolutional neural network (CNN), and the hidden layer L20 includes a convolutional layer, a pooling layer, a fully connected layer, and a judgment layer. In the convolutional layer, a convolution operation is performed using a plurality of convolution filters of a preset width. In this convolutional layer, the degree of conformity of features in a partial area of ​​an image indicated by gesture image information is calculated, and the image is converted into information indicating where an image of a feature is located. In addition, in the convolutional layer, for example, the same number of pieces of information as filters having a two-dimensional array structure are output. In the pooling layer, an average pooling process is performed with a preset width on the output from the convolutional layer. In addition, in the pooling layer, the information having a two-dimensional array structure input from the convolutional layer is averaged and compressed in the two-dimensional direction, so that only information effective for distinguishing a gesture remains. As a result, even if there is a slight difference in the shape of the user's object included in the gesture image, the influence of the difference on the gesture discrimination can be eliminated. Then, the convolution layer and the pooling layer are combined to calculate the feature quantity corresponding to the image information from the image information input from the input layer. In particular, the calculation of the convolution layer and the calculation of the pooling layer are alternately performed multiple times, thereby reducing the amount of calculation. The fully connected layer calculates the sum of values ​​obtained by multiplying each of the multiple feature quantities by a weighting coefficient, converts the calculated sum into a feature variable using an activation function, and outputs it. Here, for example, a sigmoid function is adopted as the activation function. In the fully connected layer, all information output from the pooling layer at the final stage is input and output as information with a one-dimensional structure. In addition, in the fully connected layer, when the number of nodes corresponding to the pooling layer at the final stage increases, the number of divisions of the feature quantity space increases accordingly, and the number of feature functions that characterize each region obtained by dividing the feature quantity space increases accordingly. The judgment layer estimates the operation content corresponding to the image information by converting the feature variables output from the fully connected layer into judgment values ​​within a preset numerical range using, for example, a softmax function.As shown in FIG. 7, the amount of information processed by the NN calculation unit 213 is ND1 in the input layer, but decreases as it progresses to the convolution layer and the pooling layer, and decreases to ND2 just before the fully connected layer. This is because the pooling process averages multiple features and aggregates them into one feature. The information indicating the structure of the neural network includes information indicating the width of the convolution filter and information indicating the width in the average pooling process. The information indicating the weighting coefficient includes information indicating the weighting coefficient of each of the multiple convolution filters used in the convolution layer and information indicating the weighting coefficient in the fully connected layer. The NN storage unit 231 also stores information indicating the initial value of the weighting coefficient used when determining the weighting coefficient of the neural network.

[0029] The operation identification information storage unit 233 stores judgment value information indicating the range of judgment values ​​assigned to various operations performed by the user in association with the operation identification information identifying the operation content performed by the user. For example, as shown in FIG. 8, the range of judgment values ​​corresponding to the operation identification information identifying the operation of "air conditioner ON" is set to NUM11 or more and less than NUM12, and the range of judgment values ​​corresponding to the operation identification information identifying the operation of "air conditioner OFF" is set to NUM21 or more and less than NUM22. In addition, the range of judgment values ​​corresponding to the operation identification information identifying the operation of "change temperature setting of air conditioner" is set to NUM31 or more and less than NUM32, and the range of judgment values ​​corresponding to the operation identification information identifying the operation of "maintain temperature setting of air conditioner" is set to NUM41 or more and less than NUM42. The ranges of judgment values ​​are set so as not to overlap with each other.

[0030] 5, when the image acquisition unit 211 acquires the gesture image information, user information, and device identification information transmitted from the air conditioners 1A, 1B, the image acquisition unit 211 associates these with each other and stores them in the image storage unit 232. The image acquisition unit 211 also notifies the selection unit 212 of the acquired device identification information. This device identification information is information that identifies the air conditioners 1A, 1B in which the imaging device 181 that generated the gesture image information acquired by the image acquisition unit 211 is installed.

[0031] The selection unit 212 selects a neural network corresponding to the device identification information notified by the image acquisition unit 211 from among the neural networks stored in the NN storage unit 231. Specifically, the selection unit 212 selects information indicating the structure of the neural network corresponding to the device identification information notified by the image acquisition unit 211 and information indicating weighting coefficients. Then, the selection unit 212 sets the weighting coefficients of each of the selected convolution filters to the selected convolution filters. In addition, the selection unit 212 sets the weighting coefficients in the fully connected layer of the selected neural network to a portion corresponding to the fully connected layer of the selected neural network.

[0032] The NN calculation unit 213 calculates a judgment value from the gesture image information and the user identification information stored in the image storage unit 232, using the neural network selected by the selection unit 212 based on the device identification information. The NN calculation unit 213 also calculates a feature amount from the gesture image information and the user information by performing an arithmetic process using a portion corresponding to a convolution layer and a pooling layer in the neural network. Here, the NN calculation unit 213 calculates the feature amount by repeating a convolution operation using a plurality of convolution filters and a pooling process for the gesture image information and the user information a preset number of times. Then, the NN calculation unit 213 calculates a judgment value for the feature amount corresponding to the gesture image information and the user information by performing an arithmetic process using a portion corresponding to a fully connected layer and a judgment layer in the neural network. The NN calculation unit 213 uses the neural network selected by the selection unit 212 to calculate a judgment value included in the range of "NUM11" or more and less than "NUM12" from image information showing images GE11 and GE21 of the same type of gesture corresponding to "air conditioner ON" as shown in Figs. 4(A) and (B), and calculates a judgment value included in the range of "NUM21" or more and less than "NUM22" from image information showing images GE12 and GE22 of the same type of gesture corresponding to "air conditioner OFF". The NN calculation unit 213 also uses the neural network selected by the selection unit 212 to calculate a judgment value included in the range of "NUM31" or more and less than "NUM32" from image information showing images GE13 and GE23 of the same type of gesture corresponding to "change temperature setting" as shown in Figs. 4(A) and (B), and calculates a judgment value included in the range of "NUM41" or more and less than "NUM42" from images GE14 and GE24 of the same type of gesture corresponding to "maintain temperature setting". The NN calculation unit 213 notifies the operation identification unit 214 and the NN update unit 216 of judgment value information indicating the calculated judgment value. In this way, even if the users indicated by the gesture image information are different, the NN calculation unit 213 calculates a judgment value that is included in the same range of judgment values ​​when the gestures indicated by the gesture image information correspond to the same type of gesture operation.

[0033] When the operation identification unit 214 is notified of the judgment value information from the NN calculation unit 213, it identifies the operation identification information corresponding to the notified judgment value information from the operation identification information stored in the operation identification information storage unit 233. Then, the operation identification unit 214 notifies the operation identification information transmission unit 215 of the identified operation identification information. Based on the device identification information stored in the image storage unit 232, the operation identification information transmission unit 215 transmits the operation identification information notified from the operation identification unit 214 to the air conditioners 1A, 1B that are the transmission source of the gesture image information corresponding to the operation identification information.

[0034] The NN update unit 216 updates the neural network selected by the selection unit 212 based on the gesture image information and the operation identification information corresponding to the judgment value calculated from the gesture image information every time the image acquisition unit 211 acquires the gesture image information, the user information, and the device identification information. Specifically, the NN update unit 216 first calculates an error between the judgment value notified from the NN calculation unit 213 and the median value of the range of judgment values ​​indicated by the judgment value information corresponding to the operation identification information stored in the operation identification information storage unit 233, and determines the weight coefficient of the neural network by error backpropagation based on the calculated error. Here, the NN update unit 216 determines the weight coefficient of each of the multiple convolution filters used in the convolution layer of the neural network and the weight coefficient of the fully connected layer of the neural network. Then, the NN update unit 216 updates the information indicating the weight coefficient of the neural network selected by the selection unit 212 stored in the NN storage unit 231 with the information indicating the determined weight coefficient.

[0035] Furthermore, each time a preset neural network update time (hereinafter referred to as the "NN update time") arrives, the NN update unit 216 identifies the neural network with the least amount of calculations from among the neural networks corresponding to the air conditioners 1A, 1B stored in the NN storage unit 231. Then, the NN update unit 216 updates all of the neural networks corresponding to the air conditioners 1A, 1B to the identified neural network.

[0036] Next, the operation of the control system according to the present embodiment will be described with reference to Figs. 9 and 10. First, as shown in Fig. 9, it is assumed that the air conditioners 1A and 1B are set to a gesture registration mode by the user (step S1). Next, when the air conditioners 1A and 1B determine that a preset waiting time has elapsed after the operation mode was set to the gesture registration mode (step S2), they identify operation identification information based on the operation performed by the user on the operation devices 6A and 6B during the waiting time (step S3). Next, the air conditioners 1A and 1B acquire gesture image information obtained by capturing an image of a gesture performed by the user to operate the air conditioners 1A and 1B (step S4). At this time, the air conditioners 1A and 1B may notify the user that the user's gesture will be captured by blinking an LED (Light Emitting Diode) lamp (not shown) provided in the air conditioner 1 or sounding a buzzer (not shown) provided in the air conditioners 1A and 1B. Thereafter, the air conditioners 1A, 1B correct the acquired gesture image information (step S5). Next, the air conditioners 1A, 1B identify the users of the air conditioners 1A, 1B based on the corrected gesture image information (step S6). Next, the gesture image information acquired by the air conditioners 1A, 1B, user information indicating the identified user, operation identification information identified by the air conditioners 1A, 1B, and device identification information of the air conditioners 1A, 1B are transmitted from the air conditioners 1A, 1B to the cloud server 2 (step S7).

[0037] On the other hand, when the cloud server 2 acquires the gesture image information, the user information, the operation identification information, and the device identification information, the cloud server 2 associates the acquired gesture image information, the user information, and the device identification information with each other and stores them in the image storage unit 232. Next, the cloud server 2 selects a neural network corresponding to the air conditioner 1A, 1B that is the sender of the gesture image information, the operation identification information, and the device identification information based on the device identification information (step S8). Next, the cloud server 2 calculates a judgment value from the gesture image information and the user information using the selected neural network (step S9). After that, the cloud server 2 stores judgment value information indicating the calculated judgment value in the operation identification information storage unit 233 in association with the operation identification information acquired from the air conditioners 1A, 1B (step S10).

[0038] Next, it is assumed that the user performs an operation mode change operation to change the operation mode of the air conditioners 1A and 1B to the gesture operation mode. In this case, the operation mode of the air conditioner 1 is set to the gesture operation mode (step S11). Next, the air conditioners 1A and 1B acquire gesture image information obtained by capturing an image of a gesture for operating the air conditioners 1A and 1B performed by the user (step S12), and correct the acquired gesture image information (step S13). After that, the air conditioners 1A and 1B identify the user of the air conditioners 1A and 1B based on the gesture image information (step S14). Next, the gesture image information acquired by the air conditioners 1A and 1B, user information indicating the identified user, and device identification information of the air conditioners 1A and 1B are transmitted from the air conditioners 1A and 1B to the cloud server 2 (step S15). Next, the cloud server 2 selects a neural network corresponding to the air conditioner 1A, 1B that transmitted the gesture image information, the operation identification information, and the device identification information, based on the device identification information (step S16). After that, the cloud server 2 calculates a judgment value from the gesture image information and the user information using the selected neural network (step S17).

[0039] Next, the cloud server 2 identifies the operation identification information corresponding to the calculated judgment value from among the operation identification information stored in the operation identification information storage unit 233 (step S18). The identified operation identification information is then transmitted from the cloud server 2 to the air conditioners 1A and 1B that are the transmission sources of the gesture image information, the user information, and the device identification information (step S19). On the other hand, upon acquiring the operation identification information, the air conditioners 1A and 1B calculate device setting parameters corresponding to the acquired operation identification information, generate device setting information indicating the calculated device setting parameters, and update the device setting information stored in the device setting storage unit 131 (step S20). At this time, the air conditioners 1A and 1B control the operation of the compressor and the blower fan based on the updated device setting information stored in the device setting storage unit 134.

[0040] Furthermore, the cloud server 2 determines the weighting coefficient of the selected neural network based on the judgment value corresponding to the specified operation identification information stored in the operation identification information storage unit 233 and the judgment value calculated from the gesture image information, the user information, and the device identification information (step S21). After that, the cloud server 2 updates the weighting coefficient information for the selected neural network stored in the NN storage unit 231 with information indicating the determined weighting coefficient (step S22).

[0041] 10, when a preset NN update time arrives, the cloud server 2 compares the amount of calculations in the neural networks of the air conditioners 1A and 1B stored in the NN storage unit 231 (step S23). Next, the cloud server 2 identifies the neural network with the smallest amount of calculations (step S24). Then, the cloud server 2 updates all of the weighting coefficient information of the neural networks corresponding to the air conditioners 1A and 1B to the weighting coefficient information of the identified neural network (step S25).

[0042] Also, suppose that after the series of processes from steps S12 to S17 described above are executed, the cloud server 2 determines that it has failed to identify the operation identification information (step S26). Here, if the judgment value calculated using the selected neural network is significantly different from the judgment value range indicated by the judgment value information corresponding to each of the operation identification information stored in the operation identification information storage unit 233 and is not included in any of the judgment value ranges, the cloud server 2 determines that it has failed to identify the operation identification information. In this case, the cloud server 2 newly selects a neural network corresponding to the other air conditioners 1A, 1B different from the selected neural network (step S27). Then, the cloud server 2 calculates a judgment value from the gesture image information and user information acquired from the air conditioners 1A, 1B using the newly selected neural network (step S28). Thereafter, if the cloud server 2 is able to identify the operation identification information based on the calculated judgment value (step 29), the identified operation identification information is transmitted from the cloud server 2 to the air conditioners 1A, 1B that are the senders of the gesture image information, user information, and equipment identification information (step S19).

[0043] Furthermore, the cloud server 2 determines a weighting coefficient of the previously selected neural network based on a judgment value corresponding to the specified operation identification information stored in the operation identification information storage unit 233 and a judgment value calculated from the gesture image information, the user information, and the device identification information using the previously selected neural network (step S30). Next, the cloud server 2 updates the weighting coefficient information for the previously selected neural network stored in the NN storage unit 231 with information indicating the determined weighting coefficient (step S31).

[0044] Also, assume that the user performs a new device registration operation to register the air conditioners 1A and 1B newly installed in the building H. In this case, the operation modes of the air conditioners 1A and 1B are set to a new registration mode (step S32). Next, the air conditioners 1A and 1B identify the user based on the operation content of the new registration operation (step S33). After that, the identified user information and device identification information of the air conditioners 1A and 1B are transmitted from the air conditioners 1A and 1B to the cloud server 2 (step S34). On the other hand, when the cloud server 2 acquires the identified user information and device identification information of the air conditioners 1A and 1B, it selects one neural network based on the user information from among the neural networks stored in the NN storage unit 231 (step S35). Next, the cloud server 2 newly stores, in the NN storage unit 231, NN structure information indicating the structure of the selected neural network and weight coefficient information indicating the weight coefficient in association with the acquired device identification information (step S36).

[0045] Also, as shown in FIG. 11, it is assumed that within a preset reference time after the device setting information is updated by executing a series of processes from steps S12 to S22, the user performs a switching operation to switch the air conditioners 1A and 1B from the gesture operation mode to the manual operation mode in order to change the device settings of the air conditioners 1A and 1B. That is, it is assumed that the user performs a correction operation to correct the device settings of the air conditioners 1A and 1B that were set by the gesture operation. In this case, the air conditioners 1A and 1B accept the correction operation to correct the operation content corresponding to the acquired gesture image, and the operation mode of the air conditioners 1A and 1B is set to the manual operation mode (step S37). Next, when the air conditioners 1A and 1B determine that a preset waiting time has elapsed after the operation mode was set to the manual operation mode (step S38), they identify operation identification information based on the operation performed by the user on the operation devices 6A and 6B during the waiting time (step S39). Thereafter, the identified operation identification information and the device identification information of the air conditioners 1A and 1B are transmitted from the air conditioners 1A and 1B to the cloud server 2 (step S40). Meanwhile, upon acquiring the operation identification information and the device identification information of the air conditioners 1A and 1B, the cloud server 2 selects neural networks corresponding to the air conditioners 1A and 1B respectively (step S41). Next, the cloud server 2 determines a weighting coefficient for each of the selected neural networks based on the judgment value corresponding to the identified operation identification information stored in the operation identification information storage unit 233 and the judgment value calculated in step S17 (step S42). Next, the cloud server 2 updates the weighting coefficient information of the neural networks stored in the NN storage unit 231 with information indicating the weighting coefficient determined for each neural network (step S43).

[0046] Next, the device control process executed by the air conditioners 1A, 1B according to this embodiment will be described with reference to Fig. 12 and Fig. 13. This device control process is started, for example, when the air conditioners 1A, 1B are powered on. Furthermore, when the operation reception unit 113 receives operation information for changing the operation mode information from the operation devices 6A, 6B as a result of the user performing an operation for changing the operation mode on the operation devices 6A, 6B, the operation reception unit 113 notifies the operation mode setting unit 116 of the operation information for changing the operation mode information. Then, each time the operation reception unit 113 notifies the operation mode setting unit 116 of the operation information for changing the operation mode information, the operation mode setting unit 116 updates the operation mode information stored in the operation mode storage unit 133 based on the notified operation information.

[0047] First, the operation reception unit 113 judges whether or not a new registration operation for newly registering a neural network corresponding to the air conditioners 1A and 1B in the NN storage unit 231 of the cloud server 2 has been received (step S101). If the operation reception unit 113 judges that a new registration operation has not been received (step S101: No), the process of step S104 described below is executed. On the other hand, if the operation reception unit 113 judges that a new registration operation has been received (step S101: Yes), the registration request unit 120 identifies a user based on the operation content of the new registration operation (step S102). Next, the registration request unit 120 transmits user information of the identified user and device identification information of the air conditioners 1A and 1B to the cloud server 2 (step S103).

[0048] Next, the operation acceptance unit 113 judges whether or not the air conditioner 1 is set to the gesture registration mode by referring to the operation mode information stored in the operation mode storage unit 133 (step S104). If the operation acceptance unit 113 judges that the air conditioners 1A and 1B are not set to the gesture registration mode (step S104: No), the process of step S111 described below is executed. On the other hand, if the operation acceptance unit 113 judges that the air conditioners 1A and 1B are set to the gesture registration mode (step S104: Yes), it judges whether or not a preset waiting time has elapsed since the gesture registration mode was set (step S105). As long as the operation acceptance unit 113 judges that the above-mentioned waiting time has not yet elapsed since the gesture registration mode was set (step S105: No), it repeatedly executes the process of step S105. On the other hand, when operation reception unit 113 determines that the aforementioned waiting time has elapsed after air conditioners 1A, 1B were set to the gesture registration mode (step S105: Yes), if the operation information received from manipulation devices 6A, 6B is related to updating device setting parameters of air conditioners 1A, 1B, it identifies operation identification information corresponding to the operation information (step S106). At this time, operation reception unit 113 notifies operation identification information transmission unit 115 of the identified operation identification information.

[0049] Thereafter, the image acquisition unit 111 acquires gesture image information of the gesture to be registered, which is obtained by imaging the image capture device 181 (step S107). At this time, the image acquisition unit 111 may inform the user that the user's gesture will be imaged by notifying a lighting device (not shown) that lights up the LED lamp of command information for blinking the LED lamp provided in the air conditioner 1, or by notifying a buzzer driving unit (not shown) that drives the buzzer of command information for sounding the buzzer provided in the air conditioner 1. Next, the image correction unit 112 corrects the gesture image information acquired by the image acquisition unit 111 (step S108). At this time, the image correction unit 112 stores the corrected gesture image information in the image storage unit 131. Next, the user identification unit 117 identifies the user based on the gesture image information stored in the image storage unit 131 (step S109). At this time, user identification unit 117 stores the user information of the identified user in user storage unit 132. Thereafter, image transmission unit 118 transmits the gesture image information, user information, and device identification information of air conditioners 1A and 1B to cloud server 2, and operation identification information transmission unit 115 transmits the operation identification information notified by operation reception unit 113 to cloud server 2 (step S110).

[0050] Next, the image acquisition unit 111 judges whether or not the operation mode indicated by the operation mode information stored in the operation mode storage unit 133 is the gesture operation mode (step S111). If the image acquisition unit 111 judges that the operation mode information stored in the operation mode storage unit 133 does not indicate the gesture operation mode (step S111: No), the process of step S101 is executed again. On the other hand, if the image acquisition unit 111 judges that the operation mode information stored in the operation mode storage unit 133 indicates the gesture operation mode (step S111: Yes), the image acquisition unit 111 judges whether or not gesture image information corresponding to a gesture for operating the air conditioners 1A and 1B by the user has been acquired (step S112). Here, if the image acquisition unit 111 judges that gesture image information has not been acquired (step S112: No), the process of step S101 is executed again.

[0051] On the other hand, when the image acquisition unit 111 determines that the gesture image information for the user to operate the air conditioners 1A and 1B has been acquired (step S112: Yes), the image correction unit 112 corrects the gesture image information (step S113). Next, the user identification unit 117 identifies the user based on the gesture image information stored in the image storage unit 131 (step S114). After that, the image transmission unit 118 transmits the gesture image information, the user information, and the device identification information of the air conditioners 1A and 1B to the cloud server 2 (step S115). Next, as shown in FIG. 13, the device setting unit 119 determines whether or not the operation identification information transmitted from the cloud server 2 has been acquired within a preset waiting time (step S116). Here, the waiting time is set to, for example, several seconds. When the device setting unit 119 determines that the operation identification information has not been acquired within the aforementioned waiting time (step S116: No), the process of step S101 is executed again assuming that the gesture operation has not been performed. On the other hand, when the device setting unit 119 determines that it has acquired the operation identification information transmitted from the cloud server 2 (step S116: Yes), it calculates device setting parameters based on the acquired operation identification information, and stores device setting information indicating the calculated device setting parameters in the device setting storage unit 134 (step S117). Then, the device control unit 114 controls the operation of the compressor and the blower fan based on the device setting information stored in the device setting storage unit 134.

[0052] Thereafter, the operation acceptance unit 113 judges whether or not the time elapsed since the device setting information stored in the device setting storage unit 134 was updated has passed a preset reference time based on the time information measured by the clock unit 105 (step S118). When the operation acceptance unit 113 judges that the time elapsed since the device setting information was updated has passed the aforementioned reference time (step S118: Yes), the process of step S101 is executed again. On the other hand, when the operation acceptance unit 113 judges that the time elapsed since the device setting information was updated has not yet passed the aforementioned reference time (step S118: No), the operation acceptance unit 113 judges whether or not the operation mode information stored in the operation mode storage unit 133 has been changed from the gesture operation mode to the manual operation mode (step S119). That is, the operation acceptance unit 113 judges whether or not a correction operation for correcting the operation content corresponding to the gesture image acquired by the image acquisition unit 111 has been accepted. When the operation reception unit 113 determines that the operation mode has not yet been changed to the manual operation mode (step S119: No), the process of step S118 is executed again.

[0053] On the other hand, when the operation reception unit 113 determines that the operation mode has been changed to the manual operation mode (step S119: Yes), it determines whether or not a preset waiting time has elapsed since the operation mode was changed to the manual operation mode (step S120). As long as the operation reception unit 113 determines that the aforementioned waiting time has not yet elapsed since the operation mode was changed to the manual operation mode (step S120: No), it repeatedly executes the process of step S120. On the other hand, when the operation reception unit 113 determines that the aforementioned waiting time has elapsed since the operation mode was changed to the manual operation mode (step S120: Yes), it identifies the operation identification information corresponding to the operation information received from the operation devices 6A and 6B (step S121). At this time, the operation reception unit 113 notifies the operation identification information transmission unit 115 of the identified operation identification information. Next, the operation identification information transmission unit 115 transmits the operation identification information notified from the operation reception unit 113 and the device identification information of the air conditioners 1A and 1B to the cloud server 2 (step S122). Then, the process of step S101 is executed again.

[0054] Next, the operation determination process executed by the cloud server 2 according to this embodiment will be described with reference to Fig. 14 to Fig. 16. This operation determination process is started, for example, when the cloud server 2 is powered on and a program for executing the operation determination process is started. First, the image acquisition unit 211 acquires gesture image information, user information, and device identification information from the air conditioners 1A and 1B, and the operation identification information acquisition unit 117 determines whether or not it has acquired the operation identification information from the air conditioners 1A and 1B (step S201). Here, if it is determined that the image acquisition unit 211 has not acquired the gesture image information, user information, and device identification information from the air conditioners 1A and 1B, or the operation identification information acquisition unit 117 has not acquired the operation identification information from the air conditioners 1A and 1B (step S201: No), the process of step S204 described below is executed. On the other hand, it is assumed that the image acquisition unit 211 acquires gesture image information, user information, and device identification information from the air conditioners 1A and 1B, and that the operation identification information acquisition unit 117 determines that it has acquired the operation identification information from the air conditioners 1A and 1B (step S201: Yes). In this case, the selection unit 212 selects a neural network corresponding to the device identification information notified from the image acquisition unit 211 from the neural networks stored in the NN storage unit 231 (step S202). Next, the NN calculation unit 213 calculates a judgment value from the gesture image information and the user identification information stored in the image storage unit 232 using the neural network selected by the selection unit 212 based on the device identification information, and the operation identification unit 214 stores judgment value information indicating the calculated judgment value in the operation identification information storage unit 233 in association with the operation identification information acquired from the air conditioners 1A and 1B (step S203).

[0055] Next, the image acquisition unit 211 determines whether or not gesture image information, user information, and device identification information have been acquired from the air conditioners 1A and 1B (step S204). If the image acquisition unit 211 determines that gesture image information, user information, and device identification information have not been acquired from the air conditioners 1A and 1B (step S204: No), the process of step S214 described below is executed. On the other hand, it is assumed that the image acquisition unit 211 has determined that gesture image information, user information, and device identification information have been acquired from the air conditioners 1A and 1B (step S204: Yes). In this case, the selection unit 212 selects a neural network corresponding to the device identification information notified from the image acquisition unit 211 from among the neural networks stored in the NN storage unit 231 (step S205). Next, the NN calculation unit 213 calculates a judgment value from the gesture image information and the user identification information stored in the image storage unit 232 using the neural network selected by the selection unit 212 based on the device identification information (step S206). Next, the operation identification unit 214 judges whether or not the operation identification information has been successfully identified based on the calculated judgment value (step S207). Here, if the operation identification unit 214 judges that the identification of the operation identification information has failed (step S207: No), the selection unit 212 judges whether or not a selectable neural network exists among the neural networks corresponding to the other air conditioners 1A and 1B different from the selected neural network (step S208). Here, the "selectable neural network" refers to a neural network corresponding to another air conditioner of the same type as the air conditioners 1A and 1B corresponding to the neural network that has failed to identify the operation identification information, and corresponds to a neural network with no history selected by the selection unit 212 after the identification of the operation device identification information has failed in step S207. Then, when the selection unit 212 determines that there is no selectable neural network (step S208: No), the process of identifying the operation identification information ends, and the process of step S214 described below is executed.On the other hand, if the selection unit 212 determines that a selectable neural network exists (step S208: Yes), it selects a new neural network corresponding to the other air conditioners 1A, 1B that is different from the previously selected neural network (step S209). After that, the NN calculation unit 213 uses the newly selected neural network to calculate a new judgment value from the gesture image information and user information acquired from the air conditioners 1A, 1B (step S210). Next, the process of step S207 is executed again.

[0056] On the other hand, if the operation identification unit 214 determines in step S207 that the operation identification information has been successfully identified based on the calculated determination value (step S207: Yes), the operation identification information transmission unit 215 transmits the identified operation identification information to the air conditioners 1A and 1B (step S211). After that, the NN update unit 216 and the NN calculation unit 213 execute a weighting coefficient determination process (step S212).

[0057] Here, the weighting coefficient determination process executed by the cloud server 2 according to the present embodiment will be described in detail with reference to FIG. 15. First, the NN calculation unit 213 executes a calculation for repeating a convolution operation using a plurality of convolution filters to which the weighting coefficients selected by the selection unit 212 are set and a pooling process for the gesture image information stored in the image storage unit 131 a preset number of times (step S301). As a result, the NN calculation unit 213 calculates a feature value corresponding to the gesture image information stored in the image storage unit 131. Next, the NN calculation unit 213 calculates a judgment value by executing a calculation of a part corresponding to the fully connected layer and the judgment layer to which the weighting coefficients selected by the selection unit 212 are set for the calculated feature value (step S302). Next, the NN update unit 216 calculates an error between the calculated judgment value and the median value of the range of judgment values ​​indicated by the judgment value information stored in the operation identification information storage unit 233 (step S303). After that, the NN update unit 216 determines new weight coefficients of the neural network by backpropagation based on the calculated error (step S304).

[0058] Returning to FIG. 14, next, the NN update unit 216 updates the weight coefficient information of the neural network selected by the selection unit 212 stored in the NN storage unit 231 with the weight coefficient information indicating the weight coefficient determined by the weight coefficient determination process (step S213). Next, the NN update unit 216 determines whether or not a preset NN update time has arrived based on the time information measured by the clock unit 205 (step S214). If the NN update unit 216 determines that the NN update time has not yet arrived (step S214: No), the process of step S201 is executed again. On the other hand, if the NN update unit 216 determines that the NN update time has arrived (step S214: Yes), as shown in FIG. 16, the NN update unit 216 compares the amount of calculation in the neural networks of the air conditioners 1A and 1B stored in the NN storage unit 231 (step S215). After that, the NN update unit 216 identifies the neural network with the smallest amount of calculation (step S216). Next, the NN update unit 216 updates all of the neural networks corresponding to the air conditioners 1A and 1B to the identified neural network (step S217). Specifically, the NN update unit 216 updates all of the weighting coefficient information of the neural networks corresponding to the air conditioners 1A and 1B to the weighting coefficient information of the identified neural network.

[0059] Next, the operation identification information acquisition unit 217 judges whether or not the operation identification information and the device identification information have been acquired from the air conditioners 1A and 1B (step S218). Here, if the operation identification information acquisition unit 217 judges that the operation identification information and the device identification information have not been acquired from the air conditioners 1A and 1B (step S218: No), the processing of the above-mentioned step S201 is executed. On the other hand, it is assumed that the operation identification information acquisition unit 217 judges that the operation identification information and the device identification information have been acquired from the air conditioners 1A and 1B (step S218: Yes). In this case, the selection unit 212 selects all of the neural networks stored in the NN storage unit 231 (step S219). After that, the NN update unit 216 and the NN calculation unit 213 execute the weighting coefficient determination processing for all of the neural networks (step S220). Next, the NN update unit 216 updates the weight coefficient information of the corresponding neural network stored in the NN storage unit 231 with the weight coefficient information indicating the weight coefficient of each neural network determined by the weight coefficient determination process (step S221). Then, the process of step S201 is executed again.

[0060] As described above, in the control system according to the present embodiment, the NN update unit 216 updates the weighting coefficient information of the neural network selected by the selection unit 212 based on the gesture image information and the operation identification information corresponding to the judgment value calculated from the gesture image information, every time the image acquisition unit 111 acquires gesture image information. In addition, the NN update unit 216 identifies the neural network with the least amount of calculation among the neural networks corresponding to the air conditioners 1A and 1B stored in the NN storage unit 231, every time a preset NN update time arrives, and updates all of the weighting coefficient information of the neural networks corresponding to the air conditioners 1A and 1B to the weighting coefficient information of the identified neural network. As a result, the weighting coefficients of the neural networks corresponding to the air conditioners 1A and 1B are periodically updated to the weighting coefficients of the neural network with the least amount of calculation, i.e., the most efficient calculation, so that it is possible to shorten the time required to identify the operation content corresponding to the gesture made by the user and improve the accuracy of identifying the operation content.

[0061] However, with the development of deep learning technology, neural networks have come to be used for voice recognition, image recognition, and the like, and are also being used in devices equipped with pattern recognition functions, automatic estimation functions, and the like. In addition, with the recent emergence of CPUs dedicated to neural network calculations, it has become possible to mount the calculation function of the neural network on the edge side, i.e., on the device, and a system that executes the calculation of the neural network on the device has also become possible to be constructed. In addition, in the past, it was common for a user to operate the device by pressing an operation button on an operation device connected to the device, but in recent years, devices that can perform operations by voice input and gesture operations by the user using the above-mentioned pattern recognition function and automatic estimation function are also increasing. The control system according to this embodiment is this type of system, in which the air conditioners 1A and 1B transmit gesture image information obtained by photographing a user with the imaging device 181 mounted thereon to the cloud server 2, and the cloud server 2 calculates a judgment value from the gesture image information using a neural network, and specifies the operation identification information based on the calculated judgment value. Systems are being provided in which the identified operation identification information is transmitted to the air conditioners 1A, 1B to operate the air conditioners 1A, 1B. In such control systems, there is a demand for improving the learning effect of the neural network used by the cloud server 2 to improve the accuracy of identifying the operation identification information. In this embodiment, the control system employs a so-called neural network battle method in which the performance of the neural networks corresponding to the air conditioners 1A, 1B, which have the same contents of gesture operations, is compared, specifically, the amount of calculation therein, and the one with the least amount of calculation and the best efficiency in calculating the judgment value is identified, and the other neural network is updated to it, thereby improving the learning effect of the neural networks.

[0062] Furthermore, when the operation identification unit 214 fails to identify the operation identification information associated with the judgment value calculated by the NN calculation unit 213, the selection unit 212 according to this embodiment newly selects a neural network corresponding to the other air conditioners 1A and 1B. Then, it is assumed that the operation identification unit 214 succeeds in identifying the operation identification information associated with the judgment value newly calculated from the gesture image information using the neural network newly selected by the NN calculation unit 213. In this case, the NN update unit 216 updates the weighting coefficient information of the neural network previously selected by the selection unit 212 based on the gesture image information and the operation identification information corresponding to the newly calculated judgment value. This allows the neural network to learn, that is, the weighting coefficient information to be updated, using the gesture image information for which the operation identification unit 214 failed to identify the operation identification information as teacher data, thereby facilitating the optimization of the weighting coefficients of the neural network.

[0063] Furthermore, when the operation receiving unit 113 of the air conditioners 1A, 1B receives the above-mentioned correction operation, the NN update unit 216 according to this embodiment updates all of the weight coefficient information of the neural network of each of the air conditioners 1A, 1B based on the gesture image information and the operation identification information corrected by the correction operation. This makes it possible to learn the neural network using the gesture image information that requires correction of the correspondence with the operation identification information as teacher data, that is, to update the weight coefficient information, thereby facilitating the optimization of the weight coefficients of the neural network.

[0064] Furthermore, when operation receiving unit 113 receives the above-mentioned new registration operation, selection unit 212 according to this embodiment selects one neural network from the neural networks for air conditioners 1A, 1B already stored in NN storage unit 231. Then, NN update unit 216 stores the selected one neural network in NN storage unit 231 as the neural network for the newly registered air conditioner. This makes it possible to quickly optimize the weighting coefficient of the neural network for the newly registered air conditioner, thereby improving the accuracy in identifying operation identification information based on the user's gesture image information from the beginning of use of the air conditioner.

[0065] Furthermore, in this embodiment, the image correcting unit 112 of the air conditioners 1A, 1B corrects the gesture image information acquired by the image acquiring unit 111 based on the imaging conditions of the imaging device 181. Then, the NN calculating unit 213 of the cloud server 2 calculates a judgment value corresponding to the corrected gesture image information from the gesture image information corrected by the image correcting unit 112, using the neural network selected by the selecting unit 212. This can improve the accuracy in identifying the operation identification information by the operation identifying unit 214.

[0066] Furthermore, when operation reception unit 113 of air conditioners 1A, 1B receives the above-mentioned gesture registration operation, NN update unit 216 according to this embodiment updates the neural network based on image information corresponding to at least one type of gesture by the user acquired by image acquisition unit 111 and operation identification information identifying the content of the operation accepted by operation reception unit 113. This allows the user to freely change the correspondence between the gesture and the content of the operation, thereby improving user convenience.

[0067] Although the embodiment of the present disclosure has been described above, the present disclosure is not limited to the above-mentioned embodiment. For example, the NN calculation unit 213 may use the neural network selected by the selection unit 212 based on the device identification information to calculate the judgment value only from the gesture image information stored in the image storage unit 232. In this case, there is no need to identify the user in the air conditioners 1A and 1B, which has the advantage of reducing the processing load in the air conditioners 1A and 1B.

[0068] In the embodiment, an example has been described in which all calculations of the convolution layer, pooling layer, fully connected layer, and judgment layer are executed in the cloud server 2. However, this is not limited to this, and for example, only the calculations of the convolution layer and pooling layer may be executed in the cloud server 2, and the calculations of the fully connected layer and judgment layer may be executed in the air conditioners 1A and 1B. In this case, the NN update unit 216 of the cloud server 2 may specify, for example, a neural network with the smallest amount of calculation in at least one of the convolution layer and the pooling layer from among the neural networks corresponding to the air conditioners 1A and 1B stored in the NN storage unit 231 each time the NN update time arrives.

[0069] According to this configuration, even if the cloud server 2 is configured to execute only the calculations of the convolution layer and pooling layer of the neural network, it is possible to promote optimization of the weight coefficients of the neural network.

[0070] In the embodiment, the imaging device 181 may include a thermal image sensor. In this case, the user information storage unit 132 may store heat dissipation area information indicating the position of the area where heat dissipation is large on the surface of the user calculated from the temperature distribution of the surface of the user shown in the thermal image of the user obtained by the imaging device 181 in association with the user identification information. The user identification unit 117 may specify the area where heat dissipation is large on the surface of the user from the temperature distribution of the surface of the user shown in the thermal image included in the image information acquired by the image acquisition unit 111, and specify the user of the air conditioner 1A, 1B based on the heat dissipation area information stored in the user information storage unit 132 and the position of the specified area where heat dissipation is large. In the embodiment, the imaging device 181 may be, for example, a camera mounted on a mobile terminal or the like. In this case, a well-known image processing technique may be adopted as a method for extracting the image of the user from the image captured by the imaging device 181.

[0071] In the embodiment, an example has been described in which the neural network is made to perform so-called supervised learning in the processing of steps S212 and S213 by setting the teacher data to the median value of the range of judgment values ​​indicated by the judgment value information stored in the operation identification information storage unit 233. However, the present invention is not limited to this, and so-called unsupervised learning may be adopted in which, for example, the NN calculation unit 213 clusters judgment values ​​calculated in the past to automatically generate teacher data for the judgment values.

[0072] In the embodiment, the air conditioners 1A and 1B may be manufactured by different manufacturers.

[0073] In the embodiment, an example has been described in which the neural network used by the NN calculation unit 213 is a convolutional neural network, but this is not limited thereto, and the neural network used by the NN calculation unit 213 may be a neural network of another structure, such as a regression neural network.

[0074] Furthermore, the various functions of the air conditioners 1A, 1B and cloud server 2 according to the present disclosure may be realized by software, firmware, or a combination of software and firmware. In this case, the software or firmware may be written as a program, and the program may be stored and distributed in a computer-readable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc), or an MO (Magneto-Optical Disc), and the program may be read and installed in a computer to configure a computer capable of realizing each of the above-mentioned functions. In addition, when each function is realized by sharing the work between an OS (Operating System) and an application, or by cooperation between the OS and an application, only the parts other than the OS may be stored in the recording medium.

[0075] Furthermore, each program can be superimposed on a carrier wave and distributed via a network. For example, the program can be posted on a bulletin board (BBS, Bulletin Board System) on the network and distributed via the network. These programs can then be started and run under the control of the OS in the same way as other application programs, thereby enabling the above-mentioned processing to be performed. [Industrial Applicability]

[0076] The present disclosure is suitable for a control system capable of controlling an air conditioner in accordance with gestures performed by a user. [Explanation of symbols]

[0077] 1A, 1B Air conditioner, 2 Cloud server, 6A, 6B Operation device, 81 Data circuit terminating device, 82 Router, 100 Control unit, 101, 201 CPU, 102, 202 Main memory unit, 103, 203 Auxiliary memory unit, 104 Neuro engine, 105, 205 Timer unit, 106 Local communication unit, 107 Wireless module, 108 Imaging interface, 109, 209 Bus, 111, 211 Image acquisition unit, 112 Image correction unit, 113 Operation reception unit, 114 Device control unit, 115, 215 Operation identification information transmission unit, 116 Operation mode setting unit, 117 User identification unit, 118 Image transmission unit, 119 Device setting unit, 120 Registration request unit, 131, 232 Image storage unit, 132 User information storage unit, 133 Operation mode storage unit, 134 device setting storage unit, 135 operation information storage unit, 181 imaging device, 212 selection unit, 213 NN calculation unit, 214 operation identification unit, 216 NN update unit, 217 operation identification information acquisition unit, 231 NN storage unit, 233 operation identification information storage unit, H building, L10 input layer, L20 hidden layer, L30 output layer, NW1 wide area network, NW2 local network

Claims

1. a neural network storage unit that stores a neural network for identifying one type of operation content corresponding to a gesture image of the same type from gesture image information showing a gesture of the same type of user corresponding to each of a plurality of devices; an imaging device provided in each of the plurality of devices and configured to image a user of each of the plurality of devices; an image acquisition unit that acquires, from the imaging device, gesture image information indicating a gesture of the user captured by the imaging device; a selection unit that selects, from the neural networks stored in the neural network storage unit, a neural network corresponding to an apparatus provided with an imaging device that has generated the gesture image information acquired by the image acquisition unit; a neural network calculation unit that calculates a judgment value corresponding to the gesture image information acquired by the image acquisition unit, using the neural network selected by the selection unit; an operation identification unit that identifies operation identification information that identifies an operation on the device associated with the determination value calculated by the neural network calculation unit; a device control unit that controls a device provided with an imaging device that generates the gesture image information, based on the operation identification information identified by the operation identification unit; a neural network update unit that updates the neural network selected by the selection unit based on the gesture image information and operation identification information corresponding to a determination value calculated from the image information every time the image acquisition unit acquires the gesture image information, and that identifies a neural network with the least amount of calculation from among the neural networks corresponding to each of the plurality of devices stored in the neural network storage unit every time a preset neural network update time arrives, and updates all of the neural networks corresponding to each of the plurality of devices to the identified neural network. Control system.

2. the selection unit, when the operation identification unit fails to identify the operation identification information for the device associated with the determination value, selects a new neural network corresponding to another device; the neural network calculation unit recalculates a judgment value corresponding to the gesture image information from the gesture image information by using a newly selected neural network; The operation identification unit identifies operation identification information associated with the determination value recalculated by the neural network calculation unit, the neural network update unit updates a neural network corresponding to an apparatus provided with an imaging device that has generated the gesture image information acquired by the image acquisition unit, based on the gesture image information and the operation identification information corresponding to the recalculated determination value. The control system of claim 1 .

3. an operation receiving unit that receives the content of the operation performed by the user on the operation device, when the operation receiving unit receives a correction operation for correcting an operation content corresponding to the gesture image acquired by the image acquiring unit, the neural network updating unit updates all of the neural networks of each of the plurality of devices based on the gesture image information and the operation identification information after the correction operation.

3. A control system according to claim 1 or 2.

4. The system further includes an operation receiving unit that receives a new registration operation for the user to newly register a neural network corresponding to a new device to the operation device, when the operation reception unit receives the new registration operation, the selection unit selects at least one neural network from among the neural networks for the multiple devices already stored in the neural network storage unit; the neural network update unit stores the at least one neural network selected by the selection unit in the neural network storage unit as a neural network for each of the at least one device to be newly registered; A control system according to any one of claims 1 to 3.

5. the neural network is a convolutional neural network; the neural network update unit, every time a time comes to update the neural network, identifies a neural network having the smallest amount of calculation in at least one of a convolution layer and a pooling layer from among the neural networks corresponding to each of the plurality of devices stored in the neural network storage unit, and updates all of the neural networks corresponding to each of the plurality of devices with the identified neural network; A control system according to any one of claims 1 to 4.

6. an image correction unit that corrects the gesture image information acquired by the image acquisition unit based on an image capture condition including at least one of a performance of the image capture device, an installation position of the image capture device, an ambient brightness of the image capture device, and a background image of a gesture image of a user captured by the image capture device; the neural network calculation unit calculates a judgment value corresponding to the gesture image information after the correction by the image correction unit, using the neural network selected by the selection unit, from the gesture image information after the correction by the image correction unit; A control system according to any one of claims 1 to 5.

7. an operation receiving unit that receives the content of the operation performed by the user on the operation device, When the operation reception unit receives a gesture registration operation for registering image information corresponding to a new gesture, the neural network update unit updates the neural network based on image information corresponding to at least one type of gesture of the user acquired by the image acquisition unit and operation identification information identifying the content of the operation accepted by the operation reception unit. A control system according to any one of claims 1 to 6.

8. the imaging device includes a thermal image sensor; A user identification unit that identifies a user based on a thermal image of the user captured by the imaging device, The neural network calculation unit calculates a judgment value corresponding to the gesture image information from the gesture image information acquired by the image acquisition unit and user identification information for identifying the user identified by the user identification unit, using the neural network selected by the selection unit. A control system according to any one of claims 1 to 7.

9. a neural network storage unit that stores a neural network for identifying one type of operation content corresponding to a gesture image of the same type from gesture image information showing a gesture of the same type of user corresponding to each of a plurality of devices; an imaging device provided in each of the plurality of devices and configured to image a user of each of the plurality of devices; an image acquisition unit that acquires, from the imaging device, gesture image information indicating a gesture of the user captured by the imaging device; a selection unit that selects, from the neural networks stored in the neural network storage unit, a neural network corresponding to an apparatus provided with an imaging device that has generated the gesture image information acquired by the image acquisition unit; a neural network calculation unit that calculates a judgment value corresponding to the gesture image information acquired by the image acquisition unit, using the neural network selected by the selection unit; an operation identification unit that identifies operation identification information that identifies an operation on the device associated with the determination value calculated by the neural network calculation unit; a device control unit that controls a device provided with an imaging device that generates the gesture image information for identifying the operation identification information based on the operation identification information identified by the operation identification unit; a neural network update unit that updates the neural network selected by the selection unit based on the gesture image information and operation identification information corresponding to a determination value calculated from the image information every time the image acquisition unit acquires the gesture image information, and that identifies a neural network with the least amount of calculation from among the neural networks corresponding to each of the plurality of devices stored in the neural network storage unit every time a preset neural network update time arrives, and updates all of the neural networks corresponding to each of the plurality of devices to the identified neural network. Learning system.

10. a step in which the control system acquires, from an imaging device provided in each of a plurality of devices and configured to image a user of each of the plurality of devices, gesture image information indicating a gesture of the user captured by the imaging device; a step in which the control system selects, from among neural networks stored in a neural network storage unit that stores neural networks for identifying one type of operation content corresponding to a gesture image of the same type of user from gesture image information showing the same type of gesture of the user, the neural network corresponding to each of the plurality of devices, a neural network corresponding to a device provided with an imaging device that generated the acquired gesture image information; A step of calculating a judgment value corresponding to the acquired gesture image information by the control system using the selected neural network from the gesture image information; A step of the control system specifying operation identification information for identifying an operation on the device associated with the calculated determination value; a step of controlling an apparatus provided with an imaging device that generates the gesture image information, based on the identified operation identification information by the control system; updating the selected neural network based on the gesture image information and the operation identification information corresponding to the determination value calculated from the image information, every time the control system acquires the gesture image information; and a step of the control system identifying a neural network with the least amount of calculation from among the neural networks corresponding to the plurality of devices stored in the neural network storage unit, and updating all of the neural networks corresponding to the plurality of devices to the identified neural network, each time a preset neural network update time arrives. Device control method.

11. Computer, a neural network storage unit that stores a neural network for identifying one type of operation content corresponding to a gesture image of the same type from gesture image information showing a gesture of the same type of user corresponding to each of a plurality of devices; an imaging device provided in each of the plurality of devices and configured to image a user of each of the plurality of devices; an image acquisition unit that acquires, from the imaging device, gesture image information indicating a gesture of the user captured by the imaging device; a selection unit that selects, from the neural networks stored in the neural network storage unit, a neural network corresponding to an apparatus provided with an imaging device that has generated the gesture image information acquired by the image acquisition unit; a neural network calculation unit that calculates a judgment value corresponding to the gesture image information acquired by the image acquisition unit, using the neural network selected by the selection unit; an operation identification unit that identifies operation identification information that identifies an operation on the device associated with the determination value calculated by the neural network calculation unit; a device control unit that controls a device provided with an imaging device that generates the gesture image information, based on the operation identification information identified by the operation identification unit; a neural network update unit that updates the neural network selected by the selection unit based on the gesture image information and operation identification information corresponding to a determination value calculated from the image information, every time the image acquisition unit acquires the gesture image information, and that identifies a neural network requiring the least amount of calculation from among the neural networks corresponding to each of the plurality of devices stored in the neural network storage unit, and updates all of the neural networks corresponding to each of the plurality of devices to the identified neural network, every time a preset neural network update time arrives; A program to function as a

Images