Learning Model Update Method, Server Device, Edge Device, and Meter Reading Device

Abstract

To provide a learning model update method, a server device, an edge device, and a meter reader that can additionally learning a learning model according to an environment in which the edge device or the meter reader is installed, and improve inference accuracy of the edge device or the meter reader.SOLUTION: A system for updating a learning model includes an edge device 100 that stores a learning model 107 and performs inference processing related to image recognition, and a server device 200 that is connected to the edge device 100 via a communication network N. A learning model update method comprises a step of updating the learning model 107 stored in the edge device 100 according to the environment in which the edge device 100 is installed.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a learning model update method, a server device, an edge device, and a meter reading device for updating a learning model stored in an edge device that stores a learning model and performs inference into an updated learning model suitable for the environment in which the edge device is installed.

Background Art

[0002] In recent years, with the progress of IoT (Internet of Things), in addition to the cloud side, there has been an increasing expectation of performing inference processing based on a learning model also on the edge side.

[0003] However, edge devices provided on the edge side in a communication network may be installed in various environments (referring to environments such as location, temperature, brightness, atmospheric pressure, etc.). If the same learning model is necessarily applied to a plurality of edge devices, it is not always possible for all of the plurality of edge devices to output expected inference results.

[0004] Also, in claim 1 of Patent Document 1, the following invention is disclosed. "An edge device machine learning model switching system that causes an edge device to determine the placed environment and switch from another machine learning model that has already been applied to a machine learning model suitable for the placed environment, the edge device including: a first acquisition means for acquiring sensor data indicating the placed environment; a determination means for causing the edge device to analyze the acquired sensor data and determine the placed environment; a determination means for causing the edge device to determine a machine learning model suitable for the determined environment; a second acquisition means for causing the edge device to acquire a machine learning model suitable for the determined environment from the cloud; and a switching means for causing the edge device to switch from another machine learning model that has already been applied to the acquired machine learning model."

Prior Art Documents

Patent Document

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the prior art, the edge device only selects an existing machine learning model, and often there are cases where sufficient accuracy inference cannot be made with the pre-prepared machine learning model.

Means for Solving the Problems

[0007] Therefore, as a means for solving the above problems, the learning model update method according to the present invention is a learning model update method in which an edge device that stores a learning model and performs inference processing related to image recognition and a server device connected to the edge device via a communication network update the learning model stored in the edge device according to the environment in which the edge device is installed. The server device includes: a receiving step of receiving, from the edge device, failure image information that has fallen below a predetermined threshold in the process of performing the inference processing using the image information in the environment acquired by the edge device as an input; an input step of receiving an input of a correct label of the failure image information; a creating step of performing additional learning on the learning model using the failure image information to which the correct label is assigned as teacher data to create an updated learning model; and an updating step of transmitting the updated learning model to the edge device and causing the edge device to change the learning model used for the inference processing from the learning model to the updated learning model.

[0008] In addition, a learning model update method according to another aspect of the present invention is a learning model update method in which an edge device that stores a learning model and performs inference processing related to image recognition and a server device connected to the edge device via a communication network update the learning model stored in the edge device according to the environment in which the edge device is installed. The edge device includes a transmission step of transmitting, to the server device, failure image information that has fallen below a predetermined threshold in the process of performing the inference processing using, as input, image information in the environment acquired by the edge device, and an update step of receiving, from the server device, an updated learning model and changing the learning model used for inference processing in the edge device from the learning model to the updated learning model. The updated learning model is a learning model created by performing additional learning on the learning model using teacher data to which a correct label is assigned for the failure image information received by the server device.

[0009] Alternatively, the teacher data may be created by the server device or the edge device displaying a failure image on the server device or the edge device based on the failure image information, and the server device or the edge device on which the failure image is displayed receiving an input of the correct label for the failure image information and associating the failure image information with the correct label.

[0010] Alternatively, the inference processing related to image recognition performed by the edge device may be multi-class classification using the softmax function as an activation function and having 10 classes, and the threshold may be 0.4 to 0.6 in terms of the softmax value.

[0011] Note that the "softmax value" refers to the largest value among a plurality of values obtained as the output of the softmax function (softmax function, softmax function) used as the activation function of the learning model. For example, in multi-class classification for classifying three classes, when 0.3, 0.3, and 0.4 are output as the output of the softmax function, the softmax value at this time is 0.4, which is the largest value.

[0012] The additional learning may be performed using, as teacher data, a data set including a combination of the failure image information and extended failure image information, which is information different from the failure image information and generated based on the failure image information.

[0013] Note that the "extended failure image information" is, for example, information different from the failure image information generated based on the failure image information obtained by data augmentation (data duplication), and is generated for the purpose of increasing the volume of the data set of the teacher data used for the additional learning.

[0014] In addition, the edge device is a meter reading device that reads the measured value from an existing meter device that measures the usage amount of energy such as water, gas, and electricity and displays the measured value on a display surface, and the meter reading device may acquire the image information by imaging the display surface.

[0015] Moreover, another learning model update method according to the present invention is a method for updating a learning model by a meter reading device that stores a learning model and performs inference processing related to image recognition to read a measurement value from an existing meter device capable of measuring the usage amount of energy such as water, gas, and electricity and displaying a multi-digit measurement value on a display surface, and a server device connected to the meter reading device via a communication network. The method updates the learning model stored in the meter reading device according to the environment where the meter reading device is installed. The meter reading device is configured to capture an image of the display surface and obtain image information, and is fixedly installed at a position where the display surface can be imaged. The meter reading device performs the inference process that takes the image information as input and outputs the recognition value of each digit of the measurement value to recognize the measurement value. The inference step includes: a transmission step in which the meter reading device transmits failure image information whose output of the inference process is below a predetermined threshold to the server device; and an update step in which the meter reading device receives an updated learning model from the server device and changes the learning model used in the inference process in the meter reading device from the learning model to the updated learning model. The updated learning model is a learning model created by performing additional learning on a digit-by-digit basis on the learning model using teacher data with a correct label containing the measurement value for the failure image information received by the server device.

[0016] Also, a server device according to the present invention is a server device configured to be connectable to an edge device that stores a learning model and performs inference processing related to image recognition via a communication network, and failure image information that has fallen below a predetermined threshold in the process of performing the inference processing using the image information acquired by the edge device as an input is received from the edge device, a label reception unit that receives a correct label of the failure image information, a creation unit that performs additional learning on the learning model of the edge device using the failure image information to which the correct label is assigned as teacher data to create an updated learning model, and a transmission unit that transmits the updated learning model to the edge device in order to cause the edge device to update the learning model.

[0017] Also, an edge device according to the present invention is an edge device configured to be connectable to a server device via a communication network, stores a learning model, and performs inference processing related to image recognition, an image acquisition unit that acquires image information, an inference unit that performs the inference processing that outputs a predetermined value using the image information as an input, a determination unit that determines the image information as failure image information when the predetermined value output by the inference processing falls below a predetermined threshold, a transmission unit that transmits the failure image information to the server device, a reception unit that receives, from the server device, an updated learning model created by additional learning using the failure image information as teacher data for the learning model, and an update unit that changes the learning model used for the inference processing related to image recognition to the updated learning model.

[0018] Also, a meter reading device according to the present invention is a meter reading device that stores a learning model used to read a measured value from an existing meter device that is configured to be connectable to a server device via a communication network, measures the usage amount of energy such as water, gas, and electricity, and can display a multi-digit measured value on a display surface, a fixing unit that fixes the main body at a position where the display surface can be imaged, and an image acquisition unit that acquires image information by imaging the display surface. An inference means for recognizing the measured value by performing the inference process that takes the image information as input and outputs recognition values for each digit of the measured value, a transmission means for transmitting failure image information whose output of the inference process is below a predetermined threshold to the server device, and an update means for receiving an updated learning model from the server device and changing the learning model used for the inference process in the meter reading device from the learning model to the updated learning model, wherein the updated learning model is a learning model created by performing additional learning on the learning model digit by digit using teacher data with the measured value as the correct label assigned to the failure image information received by the server device.

[0019] Note that in the above description, the phrase "fall below a predetermined threshold in the process of performing the inference process" means that when the inference process is performed using the learning model, a predetermined value obtained as the output of the inference process is below a predetermined threshold arbitrarily set. As such a predetermined value, for example, a softmax value or a sigmoid value described later can be adopted.

[0020] Note that in the above description, the term "learning model" is not limited to meaning a single learning model, and may be referred to as a "learning model" as an aggregate of a plurality of learning models. Also, the term "learning model" may be appropriately read as a classifier.

Advantages of the Invention

[0021] According to the present invention, it is possible to additionally train a learning model according to the environment in which the edge device or the meter reading device is installed, and improve the inference accuracy of the edge device or the meter reading device.

[0022] Thereby, an optimal learning model for the environment in which the edge device or the meter reading device is installed can be applied to the edge device or the meter reading device.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted. In addition, all the content described below and the content of the drawings represent an example for implementing the present invention, and do not limit the scope of the present invention.

[0025] In the following description, the term "learning model" used may be read as "classifier" as appropriate, and similarly, the term "learning unit" may be read as "learning device" as appropriate.

[0026] (Embodiment 1) In Embodiment 1, a learning model update method for updating the learning model 107 stored in the edge device 100 using the server device 200 and the edge device 100, the server device 200 used therefor, and the edge device 100 will be described. Here, the learning model 107 stored in the edge device 100 is a learning model used for performing inference processing related to image recognition. The learning model 107 stored in the edge device 100 is preferably a machine learning model learned using a method such as a neural network. Further, the learning model 107 includes neural network models created by various methods such as the gradient descent method, the error backpropagation method, and transfer learning. Further, as the activation function of the learning model 107, it is preferable to use a sigmoid function (sigmoid function, sigmoid function) or a softmax function that outputs a value that can be interpreted as a probability.

[0027] As shown in FIG. 1, the server device 200 and the edge device 100 are connected via a communication network N. Only one edge device 100 may be connected to the server device 200 via the communication network N, or a plurality of edge devices 100 may be connected to the server device 200 via the communication network N. Further, when a plurality of edge devices 100 are connected to the server device 200, each edge device 100 may have the same configuration as each other, or each edge device 100 may have a different configuration from each other. In addition, the edge device 100 may be directly connected to the WAN (Wide Area Network) in order to be connected to the server device 200, or may be connected to the server device 200 via a relay device such as a router. Also, a plurality of edge devices 100 may be installed at one of locations such as buildings, factories, and houses, and a part or all of these plurality of edge devices 100 may be connected to the server device 200 via a relay device such as a router.

[0028] Next, with reference to FIG. 2, the functional units included in the edge device 100 and the server device 200 will be described. As shown in FIG. 2, the edge device 100 preferably includes a control unit 101, a storage unit 102, an image acquisition unit 103, an inference unit 104, a determination unit 105, and a communication unit 106. Also, a learning model 107 used for inference processing is stored in the storage unit 102. All of these functional units are realized by specific means in which the hardware and software of the edge device 100 cooperate. Further, the edge device 100 is installed in any environment where there is an object of inference processing related to image recognition. The edge device 100 may be a dedicated device specially configured according to the installation environment, or may be a general-purpose computer.

[0029] The control unit 101 can be configured by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processor Unit). The control unit 101 can read and execute an OS (Operating System), middleware, software, or an application in cooperation with the storage unit 102. The control unit 101 controls the edge device 100 and can realize various functional units described below.

[0030] The storage unit 102 can be configured by, for example, a RAM (Random Access Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like. The storage unit 102 stores the OS read by the control unit 101, other software, and information used for various processes. In particular, the storage unit 102 stores the learning model 107 used for inference processing in the edge device 100. The storage unit 102 preferably stores the image information acquired by the image acquisition unit 103. The storage unit 102 can store the updated learning model received from the server device 200. This updated learning model will be used for inference processing in the edge device 100 instead of the previously used learning model 107. Further, the storage unit 102 may store the threshold value used for determination by the determination unit 105.

[0031] The image acquisition unit 103 can be configured by, for example, imaging means such as a camera provided in the edge device 100 itself, or external imaging means independent of the edge device 100 installed in the same environment as the edge device 100. When configured by external imaging means, the edge device 100 is configured to be able to acquire image information from the external imaging means. The image acquisition unit 103 acquires image information in the environment where the edge device 100 is installed. This image information will be used as the input information for inference processing.

[0032] The inference unit 104 is configured such that, for example, the control unit 101 and the storage unit 102 cooperate to perform inference using the learning model 107. The inference unit 104 performs inference processing related to image recognition using the learning model 107 stored in the storage unit 102. In the inference unit 104, inference processing is performed with the image information acquired by the image acquisition unit 103 as the input and the result of image recognition of the image as the output. In addition, as for image recognition, it is preferable to perform object recognition. However, it is not limited to object recognition, and object detection or the like may be performed in combination with object recognition. Further, as for image recognition, it is preferable to handle an identification problem of which class the content of an image belongs to, such as binary classification or multi-class classification, and it is more preferable to handle multi-class classification in particular. In addition, the result of the inference process related to image recognition is preferably output as a value interpretable as a probability of the possibility that the image belongs to each class. For example, a value interpretable as a probability can be output by a sigmoid function, a softmax function, or the like.

[0033] The determination unit 105 determines whether or not the output by the inference process is less than a predetermined threshold in the process of performing the inference process related to image recognition. When the output by the inference process is less than the predetermined threshold, the inference process is determined to have failed, and the image information input in the inference process is handled as failed image information. The determination unit 105 can be realized, for example, by combining a CPU and a RAM and performing processing. The determination unit 105 determines whether the inference process has failed or the probability of failure is high by comparing the result of the inference process with the threshold value stored in the storage unit 102 in advance. When the inference process has failed or the probability of failure is high, the determination unit 105 causes the various functional units to handle the image information that was input information in the inference process as failed image information. For the determination by the determination unit 105, it is preferable to use the sigmoid value that is the largest value among the output values of the sigmoid function interpretable as a probability used during the inference process or the softmax value that is the largest value among the output values of the softmax function. In this case, a process of comparing the sigmoid value or the softmax value with a predetermined threshold is performed, and when these output values are less than the predetermined threshold, it is determined that the inference process has failed. In this case, it is preferable that the threshold value is decreased as the number of classes, which is the number of classes in image recognition, increases, and the threshold value is increased as the number of classes decreases. In particular, when the number of classes is more than 10, the threshold value may be set to less than 0.5, and when the number of classes is less than 10, the threshold value may be set to more than 0.5. Further, when the number of classes is 10, it is preferable to determine the threshold value in the range of approximately 0.4 to 0.6. Furthermore, when the number of classes is 10, it is particularly preferable that the threshold value is 0.5. Also, when the threshold value is decreased, the number of pieces of image information handled as failure image information decreases, and when the threshold value is increased, the number of pieces of image information handled as failure image information increases. Therefore, the value of the threshold value can be appropriately adjusted according to the learning plan for additional learning and the computer resources of the server device 200 for performing additional learning.

[0034] The communication unit 106 can be configured by, for example, a communication module, a communication antenna, or a communication circuit. Also, the communication method in the communication unit 106 is not limited to wired or wireless. The communication unit 106 connects the edge device 100 to the server device 200 via the communication network N. The communication unit 106 cooperates with the control unit 101 to enable transmission and reception of information with the server device 200. In particular, the communication unit 106 transmits failure image information to the server device 200 and receives an updated learning model from the server device 200. Further, the communication unit 106 may transmit the learning model 107 to the server device 200 in order to cause the server device 200 to store and manage the learning model 107 for each edge device 100.

[0035] Next, as shown in FIG. 2, the server device 200 preferably includes a control unit 201, a communication unit 202, a storage unit 203, a learning unit 204, a display unit 205, and an input unit 206. These functional units are all realized by specific means in which the hardware and software of the server device 200 cooperate. Also, the server device 200 may be configured by a single information processing device, or may be configured as an aggregate of a plurality of information processing devices that transmit and receive information to and from each other.

[0036] The control unit 201 can be configured by, for example, a CPU, an MPU, or the like. The control unit 201 can read and execute an OS, middleware, software, an application, or the like in cooperation with the storage unit 203. The control unit 201 controls the server device 200 and can realize various functional units described below.

[0037] The communication unit 202 can be configured by, for example, a communication module, a communication antenna, or a communication circuit. Further, the communication method by the communication unit 202 is not limited to wired or wireless. The communication unit 202 connects the server device 200 to the edge device 100 via the communication network N. The communication unit 202 enables information transmission and reception with the edge device 100 in cooperation with the control unit 201. In particular, the communication unit 202 receives the failed image information transmitted by the edge device 100 and transmits the updated learning model to the edge device 100. Further, the communication unit 202 may receive the learning model 107 used by the edge device 100.

[0038] The storage unit 203 can be configured by, for example, a RAM, an HDD, or an SSD. The storage unit 203 stores an OS read by the control unit, other software, and information used for various processes. In particular, the storage unit 203 can store the failed image information transmitted by the edge device 100 and the correct label assigned thereto in association with each other. The storage unit 203 may store the learning model 107 used by one or more edge devices 100 when receiving the learning model 107 from the edge device 100, and may store and manage the learning model 107 together with an identifier for identifying each edge device 100. Further, the storage unit 203 may store the updated learning model generated by the learning unit 204.

[0039] The learning unit 204 is configured such that, for example, the control unit 201 and the storage unit 203 cooperate to execute various machine learning methods such as deep learning. The learning unit 204 performs additional learning on the learning model 107 used by the edge device 100 by using teacher data consisting of a plurality of failure image information received from the edge device 100 and the correct labels associated therewith. Note that the data set used as the teacher data may also include images generated by data augmentation (data boosting). This additional learning can be performed by methods such as incremental learning and relearning, for example.

[0040] The display unit 205 can be configured by, for example, various displays and the like. The display unit 205 displays a screen for use in operating the server device 200 and the like. Further, the display unit 205 can display a failure image based on the failure image information received from the edge device 100. A user using the server device 200 can view this failure image and input the correct label suitable for the failure image into the server device 200.

[0041] The input unit 206 can be configured by, for example, a keyboard, a touch panel, and the like. The input unit 206 is used to input information for use in operating the server device 200 and the like. Further, the input unit 206 can be used to input the correct label corresponding to the failure image displayed by the display unit 205.

[0042] FIG. 3 is a sequence diagram showing the processing flow between the edge device 100 and the server device 200 in Embodiment 1. As shown in FIG. 3, the edge device 100 and the server device 200 cooperate with hardware and software by executing various steps to realize specific means for updating the learning model 107 stored in the edge device 100 according to the environment in which the edge device 100 is installed. These steps will be described in detail below.

[0043] The image acquisition step S151 is executed by the edge device 100. In the image acquisition step S151, the edge device 100 uses the image acquisition unit 103 to acquire image information in the environment where the edge device 100 is installed. In the image acquisition step S151, for example, image information can be acquired by imaging means such as a camera provided in the edge device 100. The image information acquired in the environment where the edge device 100 is installed will have characteristics specific to that environment. By using a plurality of such image information as training data and performing additional learning described later, it becomes possible to generate an updated learning model suitable for the environment where the edge device 100 is installed. Also, it is preferable that the plurality of image information acquired by one edge device 100 consists only of a set of images captured from a specific viewpoint in a specific direction. In other words, it is preferable that the image acquisition unit 103 is fixedly installed so as to capture images in a specific direction. In this case, additional learning described later can be performed using image information having specific characteristics determined by predetermined imaging conditions (conditions such as a specific viewpoint and a specific direction), and the accuracy of the updated learning model can be made even higher. Here, the specific characteristics refer to specific characteristics determined by predetermined imaging conditions, and examples thereof include characteristics of the image boundary such as a tree being reflected at a predetermined position, and characteristics of changes in brightness and darkness such as turning off the light at a predetermined time zone.

[0044] The image recognition step S152 is executed by the edge device 100. In the image recognition step S152, the edge device 100 uses the inference unit 104 and the learning model 107 stored in the storage unit 102 to perform inference processing related to image recognition. In the image recognition step S152, inference processing related to image recognition using the image information acquired in the image acquisition step S151 as input information is performed, and a recognition result is output. In the inference processing, it is preferable that the determination unit 105 outputs a value that can be compared with a threshold value during the process of the processing. With such a configuration, in the learning model 107, it is preferable to output an output value that can be interpreted as a probability.

[0045] The determination step S153 is executed by the edge device 100. In the determination step S153, the edge device 100 uses the determination unit 105 to determine whether the image information input in the inference process is below a predetermined threshold value in the inference process. Thereby, it is determined whether the inference process has failed or is likely to have failed in the image recognition step S152. In the determination step S153, when the inference result is below the threshold value, the process proceeds to the next failed image information transmission step S154. On the other hand, when the inference result is not below the threshold value, it is determined that the image recognition is likely to have been successful, the next failed image information transmission step S154 is skipped, and the process proceeds to the updated learning model reception step S155.

[0046] Here, with reference to FIG. 4, a specific example of the processing in the image recognition step S152 and the determination step S153 will be described. Here, the case of performing multi-class classification for classifying an image into five classes from class a to class e will be described. In FIG. 4, as the processing in the image recognition step S152, predetermined image information is input to the inference unit 104. Then, the inference unit 104 performs an inference process related to image recognition on the image information using the learning model 107 stored in the storage unit 102. Further, the inference unit 104 outputs the calculation result by the softmax function set as the activation function as the result of the inference process. As a result of the processing so far, it is assumed that the inference unit 104 determines that the probability that the image represented by the input image information belongs to class a is 0.3, the probability that it belongs to class b is 0.1, the probability that it belongs to class c is 0.1, the probability that it belongs to class d is 0.25, and the probability that it belongs to class e is 0.25. In this case, it can be interpreted that the inference unit 104 determines that the input image is the class a with the highest probability and that the probability is 0.3. Note that the softmax value at this time is 0.3 for class a. At this time, in determination step S153, a process of comparing the highest probability of 0.3 with the set threshold value is performed. Here, if the threshold value is assumed to be 0.5, as shown in FIG. 4, the inference result will be below the threshold value, and it is determined that the inference process has failed. In this case, the image information that was input information in the inference process is handled as failed image information and will be transmitted to the server device 200 in the next step. In addition, if necessary, the probabilities that are the outputs for each class may be transmitted to the server device 200 and used for data analysis of recognition accuracy, formulation of learning plans for additional learning, and the like.

[0047] The failed image information transmission step S154 is executed by the edge device 100. In the failed image information transmission step S154, the edge device 100 uses the communication unit 106 to transmit the failed image information determined to be a failure in the determination step S153 to the server device 200.

[0048] The failed image information reception step S251 is executed by the server device 200. In the failed image information reception step S251, the server device 200 uses the communication unit 202 to receive the failed image information from the edge device 100. Also, when a plurality of edge devices 100 are connected to one server device 200 via the communication network N, it is preferable for the server device 200 to store the failed image information in the storage unit 203 of the server device 200 so that it can identify the edge device 100 that is the transmission source of the failed image information. In this case, for example, the server device 200 can store the received failed image information in association with the identification information for identifying each edge device 100 in the storage unit 203 of the server device 200.

[0049] The correct label assignment step S252 is executed by the server device 200. In the correct label assignment step S252, the server device 200 assigns a correct label to the received failure image information. More specifically, the server device 200 associates the class name corresponding to the content of the received failure image information as the correct label with the received failure image information, and stores this in the storage unit 203. As such a means, for example, the server device 200 can display the failure image on the display unit 205 based on the received failure image information, and receive the input of the correct label from the user of the server device 200 who has confirmed the failure image via the input unit 206. Also, in the correct label assignment step S252, for example, based on the output obtained by inputting the received failure image information to the learning model used for label assignment held by the server device 200 with different parameter adjustments from the learning model 107 stored in the edge device 100, a correct label may be automatically assigned to the failure image information. Also, in the correct label assignment step S252, for example, the server device 200 transmits failure image display information, which is information for displaying a failure image, to an edge device (not shown) equipped with the display unit 205, causes the edge device that has received the failure image display information to display the failure image, allows the user of the edge device who has confirmed the failure image to input the correct label, and causes the edge device to transmit this correct label to the server device 200. Also, for example, the edge device 100 may perform the assignment of the correct label corresponding to the failure image information before the failure image information transmission step S154, and transmit the correct label together with the failure image information to the server device 200, thereby omitting this correct label assignment step S252.

[0050] The learning determination step S253 is executed by the server device 200. In the learning determination step S253, it is determined whether the number of pieces of failure image information received and stored by the server device 200 exceeds a predetermined number required for additional learning by the server device 200, that is, whether additional learning can be performed. Such a predetermined number is preferably several hundred or more, for example, 300 or more. In the learning determination step S253, if it is determined that the number of pieces of failure image information exceeds a predetermined number, the server device 200 proceeds to the next additional learning step S255. In the learning determination step S253, if it is determined that the number of pieces of failure image information is less than the predetermined number, the server device 200 proceeds to the data expansion step S254. Note that when the number of pieces of failure image information is the same as the predetermined number, the user of the present disclosure may arbitrarily determine whether to proceed to the additional learning step S255 or the data expansion step S254.

[0051] The data expansion step S254 is executed by the server device 200. In the data expansion step S254, the server device 200 generates extended failure image information, which is image information different from the failure image information, based on the failure image information. The extended failure image information generated in the data expansion step S254 is used as teacher data in the subsequent additional learning step S255 together with the failure image information. In the data expansion step S254, in order to perform data expansion, for example, processes such as arbitrarily rotating an image, enlarging or reducing it, shifting it in an arbitrary direction, arbitrarily inverting it, or converting the color tone (RGB conversion) can be performed. Further, these processes may be executed in appropriate combinations. Furthermore, in order to perform data expansion, for example, image generation using a GAN (Generative Adversarial Network) or the like may be performed.

[0052] The additional learning step S255 is executed by the server device 200. In the additional learning step S255, the learning unit 204 of the server device 200 performs additional learning on the learning model 107 of the edge device 100 using the failed image information with the correct label as teacher data, and creates an updated learning model. When performing additional learning, if data augmentation was performed in the previous step, it is preferable to use the augmented failed image information as teacher data in addition to the failed image information. The server device 200 may hold a copy of the learning model 107 stored in the server device 200 in the storage unit 203 in order to perform the processing in the additional learning step S255, or may receive the learning model 107 from the edge device 100 when performing the processing of the additional learning step S255. As the additional learning, it is preferable to perform incremental learning using the failed image information or, if there is augmented failed image information, the failed image information and the augmented failed image information. However, it is not limited to this, and various learning methods such as retraining generally used can be adopted.

[0053] The updated learning model transmission step S256 is executed by the server device 200. In the updated learning model transmission step S256, the server device 200 uses the communication unit 202 to transmit the updated learning model created in the additional learning step S255 to the edge device 100.

[0054] The updated learning model reception step S155 is executed by the edge device 100. In the updated learning model reception step S155, the edge device 100 uses the communication unit 106 to receive the updated learning model from the server device 200.

[0055] The learning model change step S156 is executed by the edge device 100. In the learning model change step S156, the edge device 100 changes the learning model 107 used for inference processing from the previously used learning model 107 to an updated learning model. Specifically, for example, the received updated learning model can be stored in the storage unit of the edge device 100, and the newly stored updated learning model can be configured to be used for inference processing by the inference unit 104. A program for realizing such processing related to the change or switching of the learning model 107 may be stored in the edge device 100 in advance, or may be provided by the server device 200 when necessary.

[0056] The learning model update method including the above steps, and the server device 200 and the edge device 100 including the above configuration have the following effects.

[0057] According to Embodiment 1, the learning model 107 can be additionally learned according to the environment in which the edge device 100 is installed, and the inference accuracy of the edge device 100 can be improved. Thereby, the optimal learning model 107 for the environment in which the edge device 100 is installed can be applied to the edge device 100.

[0058] According to Embodiment 1, the edge device 100 performs inference processing related to image recognition based on the image information in the installed environment. When the determination unit 105 determines that it is a failure, the failure image information is transmitted to the server device 200. Then, the learning model 107 used for inference processing is changed to an updated learning model that has been additionally learned based on the failure image information. Thereby, the learning model stored in the edge device 100 can maintain a state corresponding to the newly acquired image information. In particular, the failure image information has unique features in the environment where the edge device 100 is installed. Therefore, when performing additional learning using the failure image information with unique features to generate an updated learning model, the features of the installed environment for each edge device 100 are reflected in the parameters of the updated learning model. In this case, it becomes possible to update the learning model 107 according to the installation environment of each edge device 100 in particular, and the inference accuracy related to image recognition in the edge device 100 can be improved.

[0059] In Embodiment 1, the threshold value can be arbitrarily set. Therefore, by reducing the threshold value, the number of image information handled as failure image information decreases, and by increasing the threshold value, the number of image information handled as failure image information increases. Thus, by setting the threshold value for each edge device 100 connected to the server device 200 while considering the inference accuracy, the learning load in the additional learning of the server device 200 can be minimized.

[0060] In Embodiment 1, in the data augmentation step S254, augmented failure image information can be generated based on the failure image information. Therefore, even when additional learning cannot be performed due to insufficient teacher data, augmented failure image information can be generated by performing data augmentation processing, and additional learning can be performed using the failure image information and the augmented failure image information as teacher data. Therefore, even when there is little failure image information used for additional learning, additional learning can be effectively performed. In particular, when the augmented failure image information has the same unique features as the failure image information, optimal additional learning according to the environment where the edge device 100 is installed can be performed.

[0061] Also, when generating extended failure image information, extended failure image information belonging to the same class as the class of the correct label assigned to the original failure image information can be generated by rotating, enlarging / reducing, shifting, inverting, tone-converting, image quality-converting, etc. of the image related to the failure image information. In this case, it is preferable that the extended failure image information is automatically assigned the same correct label as the correct label assigned to the failure image information used as the generation source. That is, in this case, the assignment of the correct label is simplified, and the labor required for additional learning can be reduced. Also, when generating extended failure image information by GAN, for example, Conditional GAN is adopted, and image generation can be performed by specifying a class when generating extended failure image information.

[0062] Further, when a plurality of edge devices 100 are connected to a predetermined server device 200, and thus the server device 200 and the edge devices 100 are connected in a one-to-many relationship, the following effects occur. That is, among the plurality of edge devices 100, those with low inference accuracy related to image recognition will send a large number of failure image information to the server device 200, and the server device 200 will perform additional learning frequently. On the other hand, in the edge device 100 where the parameters of the learning model 107 are tuned through additional learning and the inference accuracy has increased, the frequency of sending failure image information to the server device 200 decreases, and the frequency of additional learning decreases. For this reason, the server device 200 will preferentially perform additional learning on the learning model 107 related to the edge device 100 with low image recognition accuracy. Therefore, even when the computer resources of the server device 200 are insufficient compared to the number of edge devices 100, additional learning can be performed in a reasonable and preferable order. Thereby, even when implementing the update method of the learning model 107 according to the present disclosure by connecting a large number of edge devices 100 to a small number of server devices 200, the computer resources of the server device 200 can be minimized, and the update method of the learning model 107 according to the present disclosure can be implemented.

[0063] (Embodiment 2) In Embodiment 2, a learning model update method, a system, a server device 400 used therefor, and a meter reading device 300 for updating a learning model 307 stored in the meter reading device 300 using the server device 400 and the meter reading device 300 will be described. The meter reading device 300 stores the learning model 307 and performs an inference process related to image recognition, thereby measuring the usage amount of energy such as water, gas, and electricity from an existing meter device 500 capable of measuring the usage amount and displaying a multi-digit measured value on a display surface, and reading the measured value. Such a learning model 307 can use, for example, a neural network learned by machine learning as teacher data by associating image information of a meter display surface 501 that displays a measured value measured by the meter device 500 with the value of the measured value in the image information. Here, the meter reading device 300 is understood as a specific aspect of the edge device described in detail in Embodiment 1. The learning model update method, the server device 400, and the meter reading device 300 described below have the same steps and configurations as those in Embodiment 1 except for the parts specifically described.

[0064] As shown in FIG. 5, the server device 400 and the meter reading device 300 are connected via a communication network N. A plurality of meter reading devices 300 may be connected to one server device 400. The meter reading device 300 is installed in the same environment as the meter device 500. Thereby, the meter reading device 300 can use the image acquisition unit 303 to acquire an image of the meter display surface 501 and perform an inference process related to image recognition.

[0065] As shown in FIG. 6, the meter reading device 300 preferably includes a control unit 301, a storage unit 302, an image acquisition unit 303, an inference unit 304, a determination unit 305, and a communication unit 306. Further, the storage unit 302 stores a learning model 307 used for the inference process. The meter reading device 300 is configured such that the image acquisition unit 303 captures an image of the meter display surface 501 of the meter device 500 and acquires image information of the measured value displayed on the meter display surface 501. As such an image acquisition unit 303, various imaging means, such as a camera, can be used.

[0066] The meter device 500 displays a multi-digit measured value on the meter display surface 501. In such a mode, the measured value will be displayed by characters such as numbers, alphabets, and symbols. In the present embodiment, as an example, a meter device 500 that displays a six-digit measured value composed of decimal numbers from 0 to 9 will be used for the description.

[0067] In addition, in this embodiment, the learning model 307 stored in the storage unit 302 of the meter reading device 300 has a plurality of unit models that are inference means for recognizing the value of each digit for each digit of the measured value displayed by the meter device 500, and is configured as a set of the plurality of unit models. Specifically, the learning model 307 includes the number of unit models corresponding to the number of digits of the measured value. When image information including the digit area corresponding to each unit model is input, each unit model outputs the recognized value of each digit. As a whole, the learning model 307 outputs the recognized value corresponding to the measured value of the meter device 500 by arranging the recognized values of each unit model in the order of digits. Such a learning model 307 may be created by transfer learning based on the learning model 307 of another meter reading device 300 when the meter reading device 300 is installed. For example, it may be created by learning an open dataset such as MNIST as teacher data. In addition, each of the plurality of unit models can be said to be a small unit learning model that can recognize the value of a specific digit on the meter display surface 501 by inference processing related to image recognition. Further, a combination of these unit models constitutes a learning model 307 that recognizes a measured value of a plurality of digits as a whole. In order to output the recognized value of each digit, it is preferable that the unit model performs processing such as calculation of the softmax function in order to perform processing related to multi-class classification, for example.

[0068] Also, as shown in FIG. 6, it is preferable that the server device 400 includes a control unit 401, a communication unit 402, a storage unit 403, a learning unit 404, a display unit 405, and an input unit 406. Further, the server device 400 may be configured by a single information processing device, or may be configured as an aggregate of information processing devices by multiple information processing devices transmitting and receiving information to and from each other.

[0069] Next, an example of the appearance and configuration of the meter reading device 300 and the meter device 500 will be described with reference to FIG. 7. FIG. 7(a) shows a state where the meter reading device 300 is installed on the meter device 500, and FIG. 7(b) shows a state where the meter reading device 300 and the meter device 500 are separated.

[0070] The meter reading device 300 includes a device main body 310 configured as a hollow block body, an attachment 311 configured to cover the outer periphery of the lower end portion of the device main body 310, and a fixing portion 312 disposed at the lower end of the attachment 311 for fixing the attachment 311 and the device main body 310 to the meter device 500. Further, a camera for imaging the meter display surface 501 formed on the upper surface of the meter device 500 is provided on the lower surface of the device main body 310. Further, inside the device main body 310, a CPU, a RAM, an auxiliary storage device, a communication module, a battery, etc. for realizing various functions such as the control and inference described above are accommodated. Since the meter reading device 300 is fixedly installed with respect to the meter device 500, every time the meter display surface 501 is imaged, the same direction and position are imaged from the same viewpoint. According to such a configuration, it is possible to prevent the positions where scratches, dirt, cracks, or differences in brightness and darkness occur on the meter display surface 501 from changing every time imaging is performed. Thereby, the meter reading device 300 can perform additional learning described later using a set of image information having the characteristics of the specific meter display surface 501 determined by the same imaging conditions.

[0071] Next, based on FIG. 8, the processing flow between the meter reading device 300 and the server device 400 will be described. Also in the following description of the processing, the description of the same parts as in the first embodiment will be omitted. As shown in FIG. 8, the meter reading device 300 and the server device 400 update the learning model 307 stored in the meter reading device 300 according to the installation environment in order to read the measured value displayed on the meter display surface 501 by the meter device 500 by executing various steps.

[0072] The imaging step S351 is executed by the meter reading device 300. In the imaging step S351, the meter reading device 300 uses an image acquisition unit 303, for example, a camera fixedly installed facing the meter display surface 501, to image the meter display surface 501 of the meter device 500.

[0073] The image recognition step S352 is executed by the meter reading device 300. In the image recognition step S352, the meter reading device 300 uses the inference unit 304 and the learning model 307 stored in the storage unit 302 to perform inference processing related to image recognition, thereby recognizing the measured value displayed on the meter display surface 501. The inference processing uses the image information of the meter display surface 501 acquired by the image acquisition unit 303 as input information and the recognition result of the measured value of the meter device 500 as output information. More specifically, each of the plurality of unit models already described recognizes each digit of the measured value and outputs a measured value consisting of a plurality of digits as a whole. Such a unit model can be understood as a learning model 307 related to image recognition that recognizes a one-digit value. Each unit model outputs a recognition result regarding the digit of the measured value recognized by each unit model and an output value that can be interpreted as the probability of belonging to each predetermined class. As such an output value, for example, a softmax value can be adopted. Here, it is preferable that the region of the image of each digit recognized by each unit model has its coordinates specified in the image captured in advance by the imaging means. Specifically, with the meter reading device 300 fixed to the meter device 500 by the fixing unit 312, it is preferable to capture a test image by the imaging means and set the coordinate range of the region where the measured value is displayed in the test image. Further, for example, it is preferable that the coordinate range is set for each display region of each digit, such as the region where the first digit is displayed and the region where the second digit is displayed. Note that the range of coordinates where the measured value to be recognized is displayed is not limited to the case where the coordinates are set in advance before the inference processing. For example, every time the meter reading device 300 performs inference processing related to image recognition, it may perform processes such as specifying the position of the display region of the measured value, dividing the display region into regions for each digit, and recognizing the value for each digit.

[0074] The determination step S353 is executed by the meter reading device 300. In the determination step S353, the meter reading device 300 uses the determination unit 305 to determine whether the image information input in the inference process is below a predetermined threshold in the inference process. In particular, in the determination step S353, for each digit of the measured value displayed on the meter display surface 501, a process of comparing the softmax value obtained as the output value of the inference process related to image recognition with a predetermined threshold is performed. Then, when it is determined that the softmax value of any one of the digits of the recognition values of the plurality of digits indicating the measured value is below the predetermined threshold, it is determined that the inference process related to the image recognition has failed. Thus, when the softmax value of any one of the digits is below the predetermined threshold, the image information input as the input information in the inference process related to the image recognition is treated as failed image information in the following steps. Note that since the recognition of the value of each digit of the measured value in the second embodiment is a 10-class classification from 0 to 9, as described in the first embodiment, the predetermined threshold value to be compared with the softmax value is preferably a value between 0.4 and 0.6. In particular, the predetermined threshold value is preferably 0.5.

[0075] Here, with reference to FIG. 9, a specific example of the processing in the image recognition step S352 and the determination step S353 will be described. The meter display surface 501 is configured to be able to display, for example, a six-digit measured value. From the first-digit region 501f provided on the far right side of the meter display surface 501 towards the left, it includes the second-digit region 501e, the third-digit region 501d, the fourth-digit region 501c, the fifth-digit region 501b, and the leftmost sixth-digit region 501a. Here, the value "5" is displayed in the first-digit region 501f, the value "4" is displayed in the second-digit region 501e, the value "3" is displayed in the third-digit region 501d, the value "2" is displayed in the fourth-digit region 501c, the value "1" is displayed in the fifth-digit region 501b, and the value "0" is displayed in the sixth-digit region 501a respectively. That is, at the time of performing the processes of the image recognition step S352 and the determination step S353, the meter device 500 is displaying "012345" as the measured value on the meter display surface 501. Here, it is preferable that the range of coordinates in the image has been set in advance by the user of the meter reading device 300 for each digit region of the meter display surface 501 surrounded by each dashed line. The meter reading device 300 includes an image acquisition unit 303 configured to image the meter display surface 501 facing the meter display surface 501. Further, the meter reading device 300 stores a learning model 307 in a storage unit 302. The learning model 307 includes six unit models. Specifically, a unit model A307a, a unit model B307b, a unit model C307c, a unit model D307d, a unit model E307e, and a unit model F307f are included in the learning model 307. The unit model A307a corresponds to the sixth digit area 501a, the unit model B307b corresponds to the fifth digit area 501b, the unit model C307c corresponds to the fourth digit area 501c, the unit model D307d corresponds to the third digit area 501d, the unit model E307e corresponds to the second digit area 501e, and the unit model F307f corresponds to the first digit area 501f, and the values displayed in the respective digit areas are recognized by inference processing related to image recognition. And as a whole of the learning model 307, the inference result of the unit model A307a is used as the sixth digit, the inference result of the unit model B307b is used as the fifth digit, the inference result of the unit model C307c is used as the fourth digit, the inference result of the unit model D307d is used as the third digit, the inference result of the unit model E307e is used as the second digit, and the inference result of the unit model F307f is used as the first digit, thereby outputting a six-digit recognition value corresponding to the measured value of the meter device 500. Although not shown in the second embodiment, similar to that described in the first embodiment, in the determination step S353, a process of comparing the softmax value output in the process of the inference process of each unit model with a predetermined threshold value is performed. And when it is determined that the softmax value of any unit model is lower than the predetermined threshold value, the inference process is determined to be a failure, and the image information input as input information to the inference process is treated as failure image information. The meaning of the learning unit 404 shown below in FIG. 9 will be described later together with the description of the additional learning step S455.

[0076] The failure image information transmission step S354 is executed by the meter reading device 300. In the failure image information transmission step S354, the meter reading device 300 uses the communication unit 306 to transmit the failure image information to the server device 400.

[0077] The failure image information reception step S451 is executed by the server device 400. In the failure image information reception step S451, the server device 400 uses the communication unit 402 to receive the failure image information transmitted by the meter reading device 300.

[0078] The correct label assignment step S452 is executed by the server device 400. In the correct label assignment step S452, the server device 400 assigns a correct label to the received failure image information. Specifically, the server device 400 assigns the correct class name as the correct label to each digit of the measured value measured by the meter device 500 for the received failure image information, and associates and stores the failure image information and the correct label. For example, taking a virtual example based on the example in FIG. 9, assume that the measured value of the meter device 500 is "012345", the recognized value of the meter reading device 300 is "013395", and the softmax value of any digit is below the threshold. In this case, in the determination step S353, the inference process of the meter reading device 300 is determined to be a failure, and the failure image information is transmitted to the server device 400. Then, in the correct label assignment step S452, based on the received failure image information, the failure image is displayed on the display unit 405 of the server device 400. The server device 400 receives an input from the user of the server device 400 who has confirmed the failure image that the measured value of the meter device 500 represented by the failure image is "012345". The server device 400 that has received this input associates and stores the failure image information and the correct label with the first digit of the measured value included in the failure image information being "5", the second digit being "4", the third digit being "3", the fourth digit being "2", the fifth digit being "1", and the sixth digit being "0".

[0079] The learning determination step S453 is executed by the server device 400. In the learning determination step S453, it is determined whether the number of failure image information received and stored by the server device 400 exceeds a predetermined number necessary for the server device 400 to perform additional learning, that is, whether additional learning can be performed. In the learning determination step S453, if it is determined that the number of failure image information exceeds the predetermined number, the server device 400 proceeds to the additional learning step S455. In the learning determination step S453, if it is determined that the number of failure image information is less than the predetermined number, the server device 400 proceeds to the data expansion step S454.

[0080] The data expansion step S454 is executed by the server device 400. In the data expansion step S454, the server device 400 generates extended failure image information, which is image information different from the failure image information, based on the failure image information.

[0081] The additional learning step S455 is executed by the server device 400. In the additional learning step S455, the learning unit 404 of the server device 400 performs additional learning on the learning model 307 of the meter reading device 300 using the failure image information with correct labels as teacher data, and creates an updated learning model. When performing additional learning, if data expansion has been performed in the previous step, it is preferable to use the extended failure image information as teacher data in addition to the failure image information. As the additional learning, it is preferable to perform incremental learning using the failure image information or, if the extended failure image information exists, using the failure image information and the extended failure image information. However, it is not limited to this, and various learning methods such as retraining generally used can be adopted. To perform additional learning, it is preferable for the server device 400 to store in advance a copy of the learning model 307 stored in the meter reading device 300. It is preferable to perform additional learning for each unit model corresponding to each digit of the measured value of the meter device 500. That is, since the inference unit 304 provided in the meter reading device 300 outputs a recognized value as a result of inference processing using a plurality of unit models, it is preferable to perform additional learning for each digit area of the meter display device and adjust parameters for each unit model. Therefore, in the disclosure of the second embodiment as shown in FIG. 9, the learning unit 404 performs additional learning for the learning model 307 stored in the meter reading device 300 for each unit model. Here, for the image information of each digit area of the failure image information received by the server device 400, the correct label corresponding to the digit can be combined and used as teacher data for additional learning. For example, in the inference process described in FIG. 9, it is assumed that the output of the softmax values from the unit models corresponding to the second and fourth digits is below a predetermined threshold. In this case, in the additional learning step S455, it is preferable to perform additional learning for the unit model E307e and the unit model C307c, which are the unit models corresponding to the second and fourth digits.

[0082] The updated learning model transmission step S456 is executed by the server device 400. In the updated learning model transmission step S456, the server device 400 uses the communication unit 402 to transmit the updated learning model created in the additional learning step S455 to the meter reading device 300.

[0083] The updated learning model reception step S355 is executed by the meter reading device 300. In the updated learning model reception step S355, the meter reading device 300 uses the communication unit 306 to receive the updated learning model from the server device 400.

[0084] The learning model change step S356 is executed by the meter reading device 300. In the learning model change step S356, the meter reading device 300 changes the learning model 307 used in the inference process from the previously used learning model 307 to the updated learning model.

[0085] The learning model update method comprising the above steps, the server device 400 having the above configuration, and the meter reading device 300 have the following effects.

[0086] According to Embodiment 2, even when there are causes of image recognition inhibition such as scratches, cracks, and dirt in a specific digit area of the meter display surface 501 of the meter device 500, the learning model 307 can be updated to an updated learning model adjusted according to the situation for each digit, and a meter reading device 300 and a server device 400 that enable this can be provided. That is, since the meter device 500 is generally installed outdoors or the like, it is easily affected by sunlight, temperature, etc. For this reason, cracks, dirt adhesion, etc. often occur on the meter display surface 501. In such a case, even if the learning model 307 that performs inference processing related to general image recognition for numbers, characters, etc. is used, sufficient recognition accuracy cannot be obtained for the digits affected by the causes of image recognition inhibition. Therefore, according to the present invention, it becomes possible to perform additional learning on the unit models existing for each digit area of the failure image information, with each digit area being assigned a correct label. Thereby, even when there is a cause of image recognition inhibition in a specific digit area, the learning model 307 can be updated to an updated learning model in which parameter adjustment is performed by additionally learning such a digit. As a result, it is possible to surely recognize the value even for the digits affected by the causes of image recognition inhibition, and the inference accuracy related to image recognition can be improved.

[0087] The needle detection of the meter device 500 is not generally always or frequently performed, and imaging of images is often performed at regular intervals. For this reason, the failure image information used for additional learning is often insufficient. According to Embodiment 2, in such a situation, in a method or configuration that enables data augmentation in the present disclosure, by means such as rotation, enlargement, reduction, shift, inversion, color tone conversion, image quality conversion, or GAN of the image, the augmented failure image information used as teacher data can be generated. Therefore, in the present disclosure, even when the number of failure image information is insufficient for additional learning, by generating augmented failure image information, additional learning can be performed early. As a result, the inference accuracy related to the image recognition of the meter reading device 300 can be improved early. In particular, as a method or configuration for data augmentation, it is preferable to perform color tone conversion. In particular, by changing the brightness and darkness of the image by color tone conversion, an image captured during the day can be converted into an image captured after sunset, and an image captured after sunset can be converted into an image captured during the day. It is also preferable to perform image quality conversion on the image. Since the meter reading device 300 is fixedly installed with respect to the meter device 500, when performing additional learning to update the learning model 307 of the meter reading device 300, it is preferable to use, as teacher data, an image approximated to the image acquired by the meter reading device 300 in the actually fixed state. From this perspective, since the color tone conversion related to the change in brightness and darkness approximates the difference in imaging time, and the image quality conversion approximates the difference in defocus, it exhibits characteristics approximated to the image actually acquired by the meter reading device 300. Therefore, in a method or configuration for performing data augmentation by color tone conversion or image quality conversion, or both color tone conversion and image quality conversion, additional learning can be performed using teacher data specialized for the characteristics specific to the environment where the individual meter reading devices 300 are installed, and the inference accuracy in the meter reading device 300 can be improved. Furthermore, based on the features in such data augmentation, it may be possible to generate data that cannot actually be acquired by the meter reading device 300. In this case, it may be acceptable not to perform operations such as rotation, enlargement, reduction, shift, and inversion of the image, which are means of data augmentation.

[0088] (Other Embodiments) In other embodiments, methods and configurations related to other image recognition that can be appropriately combined by those skilled in the art with Embodiment 1 and Embodiment 2, or Embodiment 1 or Embodiment 2, the update of the learning model when using such other image recognition methods, the preprocessing of image data related to image recognition, and the End2End image recognition for collectively recognizing multiple-digit values will be described.

[0089] (Classification into 20 Classes Using Half-Digits) In Embodiment 2, a multi-class classification model for classifying each digit of a decimal measured value into 10 classes of all digits was described. However, by identifying the state where the gaps between the digits of a drum-type meter displaying decimal numbers are displayed using half-digits, it may be possible to classify each digit of the decimal measured value into 20 classes. Note that in other embodiments, the term "all digits" refers to the type of digits in a state where predetermined digits are displayed on the front of the meter display surface 501, and the term "half-digits" refers to the type of digits in a state where blanks, which are the gaps between the digits, are displayed on the front of the meter display surface 501.

[0090] As the meter device, a drum-type meter (also referred to as a rotary meter) is widely and commonly used. In a drum-type meter, characters from 0 to 9 are sequentially displayed along the circumferential direction on the circumferential surface of a cylindrical drum that rotates around an axis. As the drum rotates around the axis, the numbers are sequentially displayed through the display window. In such a drum-type meter, when the drum rotates 36 degrees, the value to be displayed increases or decreases by one. Due to such characteristics of the drum-type meter, there may be a gap between adjacent numbers displayed in the circumferential direction on the meter display surface 501 of the drum-type meter. In this case, on the meter display surface, the lower end of one number is displayed at the upper end of the meter display surface 501, and the upper end of the other number is displayed at the lower end of the meter display surface, and the middle part in the vertical direction of the meter display surface becomes blank. In such a case, it is preferable that the state in which the gap between the numbers is displayed can also be recognized. For example, when the drum-type meter displays the gap between 1 and 2, it is preferably interpreted as indicating the range between 1.0 and 2.0 or approximately 1.5.

[0091] Therefore, in another embodiment, an example is shown in which, for such a drum-type meter, by adding 10 half-digit identifiers indicating between numbers, the value of each digit of the measured value is classified into 20 classes.

[0092] In another embodiment, such half-digits are recognized by 10 identifiers from "a" to "j". That is, the state in which the blank between 0 and 1 of the drum of the drum-type meter is displayed is "a", between 1 and 2 is "b", between 2 and 3 is "c", between 3 and 4 is "d", between 4 and 5 is "e", between 5 and 6 is "f", between 6 and 7 is "g", between 7 and 8 is "h", between 8 and 9 is "i", and between 9 and 0 is "j". By identifying in this way, the state of the gap between the numbers can be recognized. Here, "a" to "j" are used as identifiers, but the identifiers are not limited to this, and various other alphabets, kana characters, symbols, etc. can be used.

[0093] For the generation of a learning model that recognizes the gap between such numbers by half-digits, for example, for the image information of the meter display surface 501 on which the gap between the numbers is displayed, the teacher data with the half-digit identifier corresponding to the state of the image of the meter display surface 501 attached as a label can be used for the generation of the learning model. Specifically, for example, for the image information with a predetermined number displayed on the front of the meter display surface 501, data with labels for all the numbers related to the number are attached, and for the image information of the meter display surface 501 where the blank spaces between the numbers are displayed, data with a semi - numeric identifier corresponding to the state of the image of the meter display surface 501 attached as a label are combined. By using this data set as teacher data to train the parameters of the neural network, it is possible to generate a learning model that can identify semi - numeric characters in addition to all - numeric characters. Also, for example, a learning model for identifying all - numeric characters and a learning model for identifying semi - numeric characters are generated separately, stored in the edge device or the meter reading device, and the edge device 100 or the meter reading device 300 is made to determine whether the image to be identified displays all - numeric characters or semi - numeric characters, and perform inference processing using the learning model according to the determination result. Also, it is possible to perform the determination between all - numeric characters and semi - numeric characters using a learning model that enables binary classification of all - numeric characters and semi - numeric characters. Furthermore, as a configuration for determining all - numeric characters and semi - numeric characters, based on the feature that when all - numeric characters are displayed, the upper and lower ends of the meter display surface 501 are blank, and when semi - numeric characters are displayed, the middle part in the vertical direction of the meter display surface 501 is blank, it may be based on a rule - based determination program.

[0094] With the above configuration, when recognizing the measured value of the meter device by classifying it into 20 classes by adding 10 classes of semi - numeric characters to 10 classes of all - numeric characters, compared with the case of reading each digit of the measured value by 10 - class classification, the measured value of the meter device 500 can be recognized in more detail and accurately.

[0095] Also, even in the case of a meter reading device that stores a learning model for classifying each digit of the measured value into 20 classes by combining all - numeric characters and semi - numeric characters, similar to the meter reading device 300 described in Embodiment 2, the failure image information whose output of the inference processing is below a predetermined threshold is transmitted to the server device 400, and by performing additional learning by the server device 400, the learning model can be updated. In this case, similar to the second embodiment, softmax values can be output as the output of the inference process, and in this case, the threshold value is preferably 0.5. However, for example, when the inference accuracy is low, the number of failed images used as teacher data in additional learning may be increased by setting the threshold value higher than 0.5. When the inference accuracy is sufficiently high, the number of failed image information used as teacher data in additional learning may be decreased by setting the threshold value lower than 0.5.

[0096] (Preprocessing) Also, the area of the meter display surface 501 on which the measured value is displayed in the meter device 500 may be specified by the learning model based on the coordinates in the captured image.

[0097] The specification of the coordinates in such a captured image can be performed by a learning model generated by machine learning using, as teacher data, the image information related to the image captured by the meter reading device 300 and the coordinate information related to the range of coordinates to be the target of the inference process for image recognition in the image. In the learning model learned by such image information and coordinate information, by inputting the captured image for specifying the coordinates, the coordinate information indicating the range of coordinates including the measured value can be output by the inference process related to regression. In this case, it is not necessary to specify the coordinates including the measured value by a human hand, and the range of coordinates including the measured value can be predicted and specified from the captured image by the meter reading device 300.

[0098] Here, as an example of a method for specifying the range of coordinates including the measured value from the captured image by the meter reading device 300, an example using a regression learning model learned by machine learning is given. However, the meter reading device 300 may be fixed to the meter device 500, and the range of coordinates to be the target of image recognition may be set in advance so that the range of coordinates to be recognized does not move from the preset range of coordinates.

[0099] Further, when the image captured by the meter reading device 300 is tilted by a predetermined angle from the horizontal direction, the tilt may be corrected to be horizontal.

[0100] Such tilt correction may be performed by a learning model stored in the meter reading device 300. The display window for displaying the measured value in the meter device 500 is generally formed in a horizontally long rectangle. Therefore, the position of such a display window can be specified by the learning model, and the tilt can be corrected so that the display window is horizontal. Also, in specifying the coordinates described above, a bounding box may be generated, and based on the tilt of this bounding box, the tilt may be corrected so that the bounding box is horizontal. Such tilt correction can be realized, for example, by rotating the extracted image related to the coordinate range by -x degrees when the coordinate range of the object to be image-recognized is tilted by x degrees from the horizontal.

[0101] Here, it is assumed that the meter reading device 300 corrects the tilt by the learning model. However, the tilt correction is not limited to being performed by the learning model. The meter reading device 300 may be fixed to the meter device 500, and the tilt of the image acquired by the fixed image acquisition unit 303 may be set in advance, so that the images captured thereafter are uniformly tilt-corrected.

[0102] In this way, by making the tilt of the image acquired by the image acquisition unit 303 horizontal, the accuracy of recognizing the measured value by the meter reading device 300 can be improved.

[0103] (Batch recognition of multiple digits) In addition, in the second embodiment, an example of recognizing measurement values consisting of multiple digits digit by digit was described. However, it is also possible to recognize measurement values consisting of multiple digits collectively using a single learning model without a unit model. Generally, water meters and electric meters are configured to be able to display measurement values of around 5 digits, and gas meters are configured to be able to display measurement values of around 7 digits. Therefore, in order to recognize measurement values that take values from 100,000 to 10 million, it is preferable to use the Extreme Multi-class Classification method that can solve multi-class classification problems in an extremely large number of classes.

[0104] In addition, as another method of recognizing multiple-digit measurement values collectively using such a single learning model, the following methods may be adopted. First, within the coordinate range where the measurement value is displayed in the image, check the display content from either the left or the right, search for the positions of the blanks that appear between each digit and digit, determine that the blank positions are the boundaries of each digit's value, and divide the coordinate range digit by digit. Then, perform inference processing related to 10-class classification from 0 to 9 for each of the divided digit regions. As a result, the value of the measurement value displayed by the meter device can be recognized using a single learning model without a unit model through this inference processing.

[0105] In this way, even when using a learning model for collective recognition of multiple digits, similar to the second embodiment, it is possible to output a softmax value as the output of the inference processing, and in this case, the threshold is preferably 0.5. However, for example, when the inference accuracy is low, it is also possible to increase the number of failure image information used as teacher data in additional learning by setting the threshold higher than 0.5, and when the inference accuracy is sufficiently high, it is also possible to decrease the number of failure image information used as teacher data in additional learning by setting the threshold lower than 0.5.

[0106] In addition, the 20-class classification using semi-numerals, preprocessing, and collective recognition of multiple digits described above can be combined as appropriate. Furthermore, these learning methods and learning models may also be realized by end-to-end learning using only the input and output as teacher data.

[0107] Those skilled in the art will be able to understand from the description of the embodiments of the meter reading device according to the present invention that the present invention can be used as a reading device for a decimal display that displays multiple-digit decimal numbers at specific positions. For example, such a decimal display reading device is a reading device that stores a learning model used to read a display value from a decimal display capable of displaying multiple-digit decimal numbers on a display surface and configured to be connectable to a server device via a communication network, and includes a fixing portion that fixes the main body at a position where the display surface can be imaged, an image acquisition means that acquires image information by imaging the display surface, an inference means that recognizes the display value by performing inference processing by the learning model that takes the image information as input and outputs recognition values for each digit of the measured value, a transmission means that transmits failure image information whose output of the inference processing is below a predetermined threshold to the server device, and an update means that receives an updated learning model from the server device and changes the learning model used for the inference processing in the reading device from the learning model to the updated learning model. The updated learning model is a learning model created by performing additional learning on a digit-by-digit basis on the learning model using teacher data with a correct label having the display value assigned to the failure image information received by the server device.

[0108] Also, any of the several embodiments already described or modes that can be appropriately modified thereof are merely examples of specific embodiments of the present invention, and it should not be construed that the scope of the present invention is limited only to such modifications.

[0109] The distributed and centralized modes of the system configured to include a server device and an edge device or a server device and a meter reading device in the present invention are not limited to those already described. That is, the degree of distribution and centralization of the system according to the present invention can be adjusted or changed as appropriate, and any such adjustment or change is included in the scope of the disclosure described herein.

Explanation of Signs

[0110] 100 Edge device 101 Control unit 102 Storage unit 103 Image acquisition unit 104 Inference unit 105 Judgment unit 106 Communication unit 107 Learning model 200 Server device 201 Control unit 202 Communication unit 203 Storage unit 204 Learning unit 205 Display unit 206 Input unit S151 Image acquisition step S152 Image recognition step S153 Judgment step S154 Failed image information transmission step S155 Updated learning model reception step S156 Learning model change step S251 Failed image information reception step S252 Correct label assignment step S253 Learning judgment step S254 Data augmentation step S255 Additional learning step S256 Updated learning model transmission step N Communication network 300 Meter reading device 301 Control unit 302 Storage unit 303 Image acquisition unit 304 Inference unit 305 Judgment Unit 306 Communication Unit 307 Learning Model 307a Unit Model A 307b Unit Model B 307c Unit Model C 307d Unit Model D 307e Unit Model E 307f Unit Model F 310 Device Body 311 Attachment 312 Fixing Part 400 Server Device 401 Control Unit 402 Communication Unit 403 Memory Unit 404 Learning Unit 405 Display Unit 406 Input Unit 500 Meter Device 501 Meter Display Surface 501f First - digit Region 501e Second - digit Region 501d Third - digit Region 501c Fourth - digit Region 501b Fifth - digit Region 501a Sixth - digit Region S351 Imaging Step S352 Image Recognition Step S353 Judgment Step S354 Failed Image Information Sending Step S355 Updated Learning Model Receiving Step S356 Learning Model Changing Step S451 Failed Image Information Receiving Step S452 Correct Label Assigning Step S453 Learning Judgment Step S454 Data Expansion Step S455 Additional Learning Step S456 Updated Learning Model Sending Step

Claims

1. A learning model update method for updating a learning model stored in an edge device according to the environment in which the edge device is installed, by an edge device that stores the learning model and performs inference processing related to image recognition, and a server device connected to the edge device via a communication network, comprising: The edge device is a meter reading device that reads a measured value from an existing meter device that measures the usage amount of energy such as water, gas, and electricity and displays the measured value on a display surface; A plurality of the meter reading devices are connected to the server device via the communication network; The meter reading device includes an image acquisition unit fixedly installed so that the image acquisition unit images a specific direction; An imaging step of imaging the display surface from a specific viewpoint in a specific direction as a predetermined imaging condition by the image acquisition unit; An image acquisition step in which the meter reading device acquires, from the image acquisition unit, image information of the display surface having unique features determined by the predetermined imaging condition; A reception step in which the server device receives, from the meter reading device, failure image information that falls below a predetermined threshold in the process of performing the inference process using the image information in the environment acquired by the meter reading device as an input; A step of storing, in a storage unit of the server device, the failure image information based on the image information acquired by one of the plurality of meter reading devices, associated with identification information for identifying the one meter reading device that is the transmission source of the failure image information; An input step in which the server device receives an input of a correct label of the failure image information; A creation step of creating an updated learning model by performing additional learning on the learning model using, as teacher data, the failure image information associated with the identification information to which the correct label is assigned; An update step in which the server device transmits the updated learning model to the one meter reading device and causes the one meter reading device to change the learning model used for the inference process from the learning model to the updated learning model. A learning model update method characterized by the above.

2. A learning model update method for updating a learning model stored in an edge device according to the environment in which the edge device is installed, by an edge device that stores the learning model and performs inference processing related to image recognition, and a server device connected to the edge device via a communication network, comprising: The edge device is a meter reading device that reads a measured value from an existing meter device that measures the usage amount of energy such as water, gas, and electricity and displays the measured value on a display surface; A plurality of the meter reading devices are connected to the server device via the communication network; The meter reading device includes an image acquisition unit, and the image acquisition unit is fixedly installed to image a specific direction; An imaging step of imaging the display surface from a specific viewpoint in a specific direction as a predetermined imaging condition by the image acquisition unit; An image acquisition step in which the meter reading device acquires, from the image acquisition unit, image information of the display surface having unique features determined by the predetermined imaging condition; A transmission step in which the meter reading device transmits, to the server device, failure image information that has fallen below a predetermined threshold in the process of performing the inference process using the image information in the environment acquired by the meter reading device as an input; A step of storing, in a storage unit of the server device, the failure image information based on the image information acquired by one of the plurality of meter reading devices, being associated with identification information for identifying the one meter reading device that is the transmission source of the failure image information; An update step in which the meter reading device receives an updated learning model from the server device and changes the learning model used for inference processing in the meter reading device from the learning model to the updated learning model; The updated learning model is a learning model created by performing additional learning on the learning model with teacher data having a correct label assigned to the failure image information associated with the identification information received by the server device; The learning model update method is characterized in that.

3. The teacher data is: The server device or the meter reading device displays a failure image on the server device or the meter reading device based on the failure image information; The server device or the meter reading device on which the failure image is displayed receives an input of the correct label for the failure image information, and creates an association between the failure image information and the correct label. The learning model update method according to claim 1 or 2, characterized in that.

4. The inference process related to the image recognition performed by the meter reading device is a multi-class classification using the softmax function as an activation function and having 10 classes. The threshold value is 0.4 to 0.6 in the softmax value, characterized in that. The learning model update method according to any one of claims 1 to 3.

5. The additional learning is performed using, as teacher data, a data set including a combination of the failure image information and extended failure image information that is information different from the failure image information and is generated based on the failure image information. The learning model update method according to any one of claims 1 to 4, characterized in that.

6. A learning model update method in which a learning model is stored and an inference process related to image recognition is performed, and a meter reading device reads a measurement value from an existing meter device capable of measuring the usage amount of energy such as water, gas, and electricity and displaying a multi-digit measurement value on a display surface, and a server device connected to the meter reading device via a communication network updates the learning model stored in the meter reading device according to the environment in which the meter reading device is installed. A plurality of the meter reading devices are connected to the server device via the communication network. The meter reading device includes an image acquisition unit fixedly provided so that the image acquisition unit images a specific direction. An imaging step of imaging the display surface from a specific viewpoint in a specific direction as a predetermined imaging condition by the image acquisition unit. An image acquisition step in which the meter reading device acquires, from the image acquisition unit, image information of the display surface having unique features determined by the predetermined imaging condition. An inference step in which the meter reading device performs the inference process that takes the image information as an input and outputs a recognition value for each digit of the measurement value, thereby recognizing the measurement value. A transmission step in which the meter reading device transmits failure image information whose output of the inference process is below a predetermined threshold value to the server device. A step in which failure image information based on the image information acquired by one of the plurality of the meter reading devices is stored in a storage unit of the server device, associated with identification information for identifying the one meter reading device that is the source of the failure image information; An update step in which the one meter reading device receives an updated learning model from the server device and changes the learning model used for the inference process in the meter reading device from the learning model to the updated learning model, including; The updated learning model is a learning model created by performing additional learning on the learning model digit by digit using teacher data with a correct label containing the measured value assigned to the failure image information associated with the identification information received by the server device. A learning model update method characterized by the above.

7. A server device configured to be connectable via a communication network to an edge device that stores a learning model and performs an inference process related to image recognition, The edge device is a meter reading device that reads a measured value from an existing meter device that measures the usage amount of energy such as water, gas, and electricity and displays the measured value on a display surface, A plurality of the meter reading devices are connected to the server device via the communication network, The meter reading device includes an image acquisition unit fixedly installed so that the image acquisition unit images a specific direction, Receiving means for receiving, from the meter reading device, failure image information that has fallen below a predetermined threshold value in the process of the meter reading device performing the inference process with input image information having unique features determined by the predetermined imaging conditions, the image information being acquired by imaging the display surface from a specific viewpoint in a specific direction as the predetermined imaging conditions; Means for storing, in a storage unit of the server device, the failure image information based on the image information acquired by one of the plurality of the meter reading devices, associated with identification information for identifying the one meter reading device that is the source of the failure image information; Label receiving means for receiving a correct label of the failure image information associated with the identification information. Creation means for performing additional learning on the learning model of the one meter reading device using the failed image information with the correct label as teacher data and creating an updated learning model; Transmission means for transmitting the updated learning model to the one meter reading device in order to update the learning model of the one meter reading device, comprising: A server device characterized by this.

8. A meter reading device configured to be connectable to a server device via a communication network, storing a learning model and performing inference processing related to image recognition, The meter reading device is a meter reading device that reads the measured value from an existing meter device that measures the usage amount of energy such as water, gas, and electricity and displays the measured value on a display surface, A plurality of the meter reading devices are connected to the server device via the communication network, The meter reading device includes an image acquisition unit, and the image acquisition unit is fixedly installed so as to image a specific direction, Inference means for performing inference processing that takes, as input, the image information acquired by imaging the display surface from a specific viewpoint in a specific direction as predetermined imaging conditions and having unique features determined by the predetermined imaging conditions, and outputs a predetermined value; Determination means for determining the image information as failed image information when the predetermined value output by the inference processing is below a predetermined threshold; Transmission means for transmitting the failed image information to the server device; Means for storing, in a storage unit of the server device, the failed image information based on the image information acquired by one of the plurality of meter reading devices in association with identification information for identifying the one meter reading device that is the transmission source of the failed image information; Receiving means for receiving, from the server device, an updated learning model created by additional learning using the failed image information associated with the identification information as teacher data for the learning model; Updating means for changing the learning model used in the inference processing related to image recognition to the updated learning model, comprising: A meter reading device characterized by this.

9. A meter reading device that stores a learning model used to read measurement values from an existing meter device configured to be connectable to a server device via a communication network, measure the usage amount of energy such as water, gas, and electricity, and display multiple-digit measurement values on a display surface. A plurality of the meter reading devices are connected to the server device via the communication network. Image acquisition means for acquiring image information by imaging the display surface. A fixing part for fixedly installing the main body at a position where the image acquisition means can image the display surface from a specific direction. Inference means for recognizing the measurement value by performing inference processing by the learning model that takes the image information acquired by imaging the display surface from a specific viewpoint in a specific direction as a predetermined imaging condition and having unique features determined by the predetermined imaging condition as an input, and outputs recognition values for each digit of the measurement value. Transmission means for transmitting failure image information whose output of the inference processing is below a predetermined threshold value to the server device. Means for storing the failure image information based on the image information acquired by one of the plurality of meter reading devices in the storage unit of the server device in association with identification information for identifying the one meter reading device that is the transmission source of the failure image information. Update means for receiving an updated learning model from the server device and changing the learning model used for the inference processing in the one meter reading device from the learning model to the updated learning model. The updated learning model is a learning model created by performing additional learning on a digit-by-digit basis on the learning model using teacher data with a correct label containing the measurement value assigned to the failure image information associated with the identification information received by the server device. A meter reading device characterized by the above.

Images