Processing method, processing system, and processing program
The system addresses accuracy maintenance in edge computing by using a model cascade with automatic relearning, ensuring efficient resource utilization and accuracy through a lightweight and high-precision model combination with adaptive retraining based on load and accuracy changes.
Patent Information
- Application Number
- JP2022564857
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-11-24
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-11-24
AI Technical Summary
Existing edge computing systems face challenges in maintaining model accuracy over time due to the need for complex manual processes in retraining models, which are resource-constrained and require significant computational load, leading to potential accuracy deterioration.
A processing system that employs a model cascade using a lightweight model (DNN1) on edge devices and a high-precision model (DNN2) on server devices, with automatic relearning determination based on load changes and inference accuracy, utilizing a confidence threshold to decide model execution location and generating domain-specific relearning data.
This approach allows for efficient and automated relearning of models at the edge and cloud, maintaining accuracy by dynamically adjusting processing to resource availability and accuracy needs, reducing administrative burden.
Smart Images

Figure 0007586191000001 
Figure 0007586191000002 
Figure 0007586191000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a processing method, a processing system, and a processing program. [Background technology]
[0002] The amount of data collected by IoT devices, such as sensors, is enormous, and when the collected data is aggregated and processed by cloud computing, a huge amount of communication traffic occurs. For this reason, attention is being paid to edge computing, which processes collected data in edge devices close to users.
[0003] However, the computational load and memory resources of the edge devices are poor compared to devices other than the edge devices (hereinafter, for simplicity, referred to as clouds) that are located physically and logically farther from users than the edge devices. For this reason, when a process with a large computational load is performed by the edge device, it may take a long time to complete the process, or it may take a long time to complete other processes that do not require a large computational load.
[0004] Here, one type of processing that requires a large amount of calculation is processing related to machine learning. Non-Patent Document 1 proposes the application of so-called adaptive learning to edge clouds. That is, the method described in Non-Patent Document 1 deploys a trained model trained using general-purpose training data in the cloud to an edge device, and re-trains the model trained in the cloud using data acquired by the edge device, thereby realizing an operation that makes the most of the advantages of the cloud and the edge device. [Prior art documents] [Non-patent literature]
[0005] [Non-Patent Document 1] :Okoshi et al., “Proposal and Evaluation of a DNN Model Operation Method Using Cloud-Edge Collaboration”, Proceedings of the 80th National Conference 2018(1), 3-4, 2018-03-13. Summary of the Invention [Problem to be solved by the invention]
[0006] Here, as operation continues, the accuracy of the model may deteriorate over time. For this reason, it is necessary to maintain the required accuracy by executing re-learning on the models placed on the edge device and the cloud. However, in order to re-learn the model, the system administrator needs to carry out cumbersome processing, such as checking all data acquired during operation, determining which data to use for each model and at what timing to execute the re-learning of the model, and arranging the re-learning process of the model.
[0007] The present invention has been made in consideration of the above, and aims to provide a processing method, a processing system, and a processing program that can properly perform re-learning of models placed on the edge and the cloud, respectively, and maintain the accuracy of the models. [Means for solving the problem]
[0008] In order to solve the above-mentioned problems and achieve the object, the processing method of the present invention is a processing method executed by a processing system that performs a first inference in an edge device and a second inference in a server device, and is characterized by having a determination step of determining whether or not a trend of a target data group for inference has changed in at least one of the edge device and the server device based on a fluctuation in load or a decrease in inference accuracy in at least one of the edge device and the server device, and a re-learning step of executing re-learning of at least one of a first model for performing the first inference and a second model for performing the second inference if it is determined in the determination step that the trend of the target data group has changed. Effect of the Invention
[0009] According to the present invention, it is possible to appropriately perform re-learning of models placed on the edge and in the cloud, respectively, and to maintain the accuracy of the models. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a diagram for explaining an outline of a processing method of a processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of DNN1 and DNN2. [Diagram 3] FIG. 3 is a diagram illustrating an example of a configuration of a processing system according to an embodiment. [Figure 4] FIG. 4 is a diagram showing the relationship between the offload rate and the overall accuracy. [Diagram 5] FIG. 5 is a flowchart illustrating a processing procedure of the learning data generation process according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing the processing procedure of the re-learning determination process for the DNN1 in the embodiment. [Figure 7] FIG. 7 is a flowchart showing the processing procedure of the re-learning determination process for DNN2 in the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a computer that realizes an edge device and a server device by executing a program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are indicated by the same reference numerals.
[0012] [Embodiment Mode] [Outline of the embodiment] An embodiment of the present invention will be described. In the embodiment of the present invention, a processing system that performs inference processing using a trained high-precision model and a lightweight model will be described. Note that in the processing system of the embodiment, a case where a DNN (Deep Neural Network) is used as a model used in the inference processing will be described as an example. In the processing system of the embodiment, a neural network other than a DNN may be used, and low-computational-amount signal processing and high-computational-amount signal processing may be used instead of a trained model.
[0013] FIG. 1 is a diagram for explaining an outline of a processing method of a processing system according to an embodiment. The processing system according to the embodiment configures a model cascade using a high-precision model and a lightweight model. The processing system according to the embodiment controls whether to execute processing in an edge device using a high-speed, low-precision lightweight model (e.g., DNN1 (first model)) or in a cloud (server device) using a low-speed, high-precision high-precision model (e.g., DNN2 (second model)). For example, a server device is a device located in a place that is physically and logically far from a user. An edge device is an IoT device and various terminal devices located in a place that is physically and logically close to a user, and has fewer resources than a server device.
[0014] DNN1 and DNN2 are models that output inference results based on input data to be processed. In the example of Figure 1, DNN1 and DNN2 take an image as input and infer the probability of each class of object appearing in the image. Note that the two images shown in Figure 1 are both the same image.
[0015] As shown in FIG. 1, the processing system obtains a confidence level for the inference of the class classification of the DNN1 for the object in the input image. The confidence level is the degree of likelihood that the result of the object recognition by the DNN1 is correct. For example, the confidence level may be the probability of the class of the object in the image output by the DNN1, for example, the probability of the highest class.
[0016] In the processing system, if the acquired confidence level is, for example, equal to or greater than a predetermined threshold, the inference result of DNN1 is adopted. In other words, the inference result of the lightweight model is output as the final estimation result of the model cascade. On the other hand, in the processing system, if the confidence level is less than a predetermined threshold, the inference result obtained by inputting the same image into DNN2 is output as the final inference result.
[0017] In this way, the processing system according to the embodiment selects an edge device or a server device based on the degree of certainty as to whether the data to be processed should be processed in the edge device or the server device, and processes the data to be processed.
[0018] [Lightweight and high-precision models] Next, we will explain DNN1 and DNN2. Figure 2 is a diagram for explaining an example of DNN1 and DNN2. A DNN has an input layer where data is input, multiple intermediate layers that convert data input from the input layer in various ways, and an output layer that outputs so-called inference results such as probability and likelihood. The output values output from each layer may be non-reversible if the input data needs to be kept anonymous.
[0019] As shown in Figure 2, the processing system may use independent DNN1 and DNN2. For example, after DNN2 is trained in a known manner, DNN1 may be trained using the learning data used in training DNN2.
[0020] Here, DNN1 only needs to solve the same problem as DNN2 and be lighter than DNN2. For example, in the case of the example in FIG. 3, DNN1 has the first intermediate layer to the P-th intermediate layer (P < S) with fewer layers than the first intermediate layer to the S-th intermediate layer of DNN2. In this way, DNN1 and DNN2 may be designed such that DNN2 has more layers than DNN1. Also, darknet19, which is the backend model of YOLOv2 that is relatively lightweight and fast (hereinafter referred to as YOLOv2), may be selected as DNN1, and darknet53, which is the backend model of YOLOv3 that is relatively highly accurate (hereinafter referred to as YOLOv3), may be selected as DNN2. In a simple example, in the same NN, DNN1 and DNN2 may be configured to have different depths. DNN1 and DNN2 may use any network respectively. For example, a CNN may be used.
[0021] In this embodiment, a system is proposed that determines the timing of retraining of DNN1 or / and DNN2 and automatically executes the retraining of DNN1 and DNN2. And in this embodiment, data for retraining is automatically selected and retraining is executed. Thereby, according to this embodiment, while reducing the burden on the administrator regarding the retraining process of the model, the retraining of the models respectively arranged on the edge and the cloud can be appropriately executed, and the accuracy of the model can be maintained.
[0022] [Processing System] Next, the configuration of the processing system will be described. FIG. 3 is a diagram schematically showing an example of the configuration of the processing system according to the embodiment.
[0023] The processing system 100 according to the embodiment includes a server device 20 and an edge device 30. The server device 20 and the edge device 30 are connected via a network N. The network N is, for example, the Internet. For example, the server device 20 is a server provided in a cloud environment. The edge device 30 is, for example, an IoT device and various terminal devices. In the present embodiment, a case will be described in which a target data group to be processed in the server device 20 and the edge device 30 is an image group.
[0024] The server device 20 and the edge device 30 are realized by loading a predetermined program into a computer or the like including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and the CPU executes the predetermined program. Also, so-called accelerators such as a GPU, a VPU (Vision Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a dedicated AI (Artificial Intelligence) chip are also used. The server device 20 and the edge device 30 each have a NIC (Network Interface Card) or the like, and can communicate with other devices via electric communication lines such as a LAN (Local Area Network) or the Internet.
[0025] 3, the server device 20 includes an inference unit 21 that performs inference (second inference) using a DNN2 that is a trained high-precision model. The DNN2 includes information such as model parameters.
[0026] The inference unit 21 uses the DNN2 to perform inference processing on the image output from the edge device 30. The inference unit 21 takes the image output from the edge device 30 as input for the DNN2. The inference unit 21 uses the DNN2 to perform inference processing on the input image. The inference unit 21 obtains an inference result (e.g., the probability for each class of object appearing in the image) as the output of the DNN2. The input image is assumed to be an image with an unknown label. In addition, when the inference result is to be returned to the user, the inference result obtained by the inference unit 21 may be transmitted to the edge device 30 and returned from the edge device 30 to the user.
[0027] Here, the server device 20 and the edge device 30 constitute a model cascade. For this reason, the inference unit 21 does not always perform inference. The inference unit 21 accepts input of a segmented image for which it has been determined in the edge device 30 that the server device 20 should execute the inference process, and performs inference using DNN2. Here, an image is described, but it may be a feature extracted from an image rather than the image itself.
[0028] The edge device 30 has an inference unit 31 having a DNN1 which is a trained lightweight model, and a judgment unit 32.
[0029] The inference unit 31 inputs an image to be processed to the DNN 1 and obtains an inference result. The inference unit 31 executes an inference process (first inference) for the input image using the DNN 1. The inference unit 31 accepts an input of the image to be processed, processes the image to be processed, and outputs an inference result (for example, the probability for each class of an object appearing in the image).
[0030] The determination unit 32 determines whether to adopt the inference result of the edge device 30 or the server device 20 by comparing the confidence level with a predetermined threshold value. In this embodiment, the edge device 30 determines whether to adopt the inference result inferred by the edge device 30, and when it is determined not to adopt this inference result, the inference result of the server device 20 is adopted.
[0031] If the confidence level is equal to or higher than a predetermined threshold, the determination unit 32 outputs the inference result inferred by the inference unit 31. If the confidence level is lower than a predetermined threshold, the determination unit 32 determines to output the image to be processed to the server device 20 and to have the DNN 2 arranged in the server device 20 execute the inference process.
[0032] In the processing system 100, as functions related to the re-learning process for the DNN1 and the DNN2, for example, a learning data generation unit 22, a learning data management unit 23, and a re-learning unit 24 are provided in the server device 20. Note that the learning data generation unit 22, the learning data management unit 23, and the re-learning unit 24 are not limited to being provided in the server device 20, and may be provided in other devices that can communicate with the server device 20 and the edge device 30.
[0033] The learning data generation unit 22 generates learning data for each of DNN1 and DNN2 to be used when re-learning DNN1 and DNN2. The learning data generation unit 22 generates, as re-learning data, data that contributes greatly to load fluctuations or deterioration of inference accuracy from among a group of images on which inference processing was actually performed during operation. The learning data generation unit 22 has a generation unit 221 and a correction unit 222.
[0034] The generation unit 221 generates data in which an image on which inference has been performed in the DNN2 is associated with an inference result by the DNN2 as a label, as edge re-learning data for the DNN1 of the edge device 30. The label of this learning data is given by automatic annotation. The generation unit 221 may generate separate learning data to be used when re-learning the DNN1 and test data. All data determined to be used for inference on the server side may be used as the re-learning data for the DNN1.
[0035] The correction unit 222 accepts input of corrections to the inference result by DNN2 of the input image. This correction is a so-called manual annotation, in which an administrator distinguishes the image to be processed and corrects the inference result. Alternatively, the correction is a process of correcting the inference result by executing an inference process using a mechanism different from DNN2.
[0036] Then, the correction unit 222 generates data in which an image on which inference has been performed in the DNN2 is associated with a corrected inference result (corrected label) in which label correction has been applied to the inference result of the DNN2 for the image, as cloud re-learning data for the DNN2 of the server device 20. The correction unit 222 may generate separate learning data to be used when re-learning the DNN2 and test data.
[0037] The learning data management unit 23 manages learning data for re-learning of the DNN1 and DNN2 generated by the learning data generation unit 22. The learning data management unit 23 includes a storage unit 231 and a selection unit 232.
[0038] The storage unit 231 stores the edge re-learning data for DNN1 generated by the learning data generation unit 22 in an edge re-learning data database (DB) 251. When there are multiple DNN1s, the storage unit 231 stores the edge re-learning data separately for each DNN1. The storage unit 231 stores the cloud re-learning data for DNN2 generated by the learning data generation unit 22 in a cloud re-learning data DB 252. When there are multiple DNN2s, the storage unit 231 stores the cloud re-learning data separately for each DNN2.
[0039] When the selection unit 232 is requested to output re-learning data by the re-learning unit 24 described later, the selection unit 232 extracts the re-learning data corresponding to the request from the edge re-learning data DB 251 or the cloud re-learning data DB 252 and outputs it to the re-learning unit 24.
[0040] The relearning unit 24 executes relearning of at least one of DNN1 and DNN2. The relearning unit 24 has a relearning determination unit 241 (determination unit) that determines whether or not to execute relearning of DNN1 or DNN2, and a relearning execution unit 242 (relearning unit).
[0041] The re-learning determination unit 241 determines whether or not the tendency of the image group to be inferred has changed in at least one of the edge device 30 and the server device 20 based on a change in load or a decrease in inference accuracy in at least one of the edge device 30 and the server device 20. If the re-learning determination unit 241 determines that the tendency of the image group has changed, it determines to execute re-learning of DNN1 or DNN2. The re-learning determination unit 241 determines to execute re-learning of DNN1 or DNN2 depending on a change from the set value of the off-load rate (processing rate in the server device 20), a decrease in inference accuracy, or the amount of learning data held. In addition, when the off-load rate decreases, the system administrator determines whether to execute re-learning based on the inference accuracy. This is because a decrease in the off-load rate does not necessarily mean that DNN1 needs to be re-learned. An increase in the off-load rate may be used as a trigger to re-learn DNN1. In the server device 20, according to the necessity of re-learning thus determined, an instruction to execute re-learning is given, and according to the instruction, the re-learning determination unit 241 executes re-learning of DNN1 or DNN2. Note that since DNN2 often performs inference on data offloaded from multiple DNN1s, it is preferable to perform inference based on the correction rate rather than the offload rate.
[0042] When the relearning determination unit 241 determines that the tendency of the image group has changed, the relearning execution unit 242 executes relearning of at least one of DNN1 and DNN2. The relearning execution unit 242 executes relearning of at least one of DNN1 and DNN2 using data from the image group that contributes greatly to load fluctuations or deterioration of inference accuracy.
[0043] The relearning execution unit 242 executes relearning of DNN1 using the edge relearning data as learning data. The relearning execution unit 242 executes relearning of DNN2 using the cloud relearning data as learning data. The relearning execution unit 242 transmits the re-learned DNN1 (or a model equivalent to DNN1) to the edge device 30 and arranges it as an edge-side model. The relearning execution unit 242 outputs the re-learned DNN2 (or a model equivalent to DNN2) to the inference unit 21 and arranges it as a cloud-side model. Note that the DNN1 and DNN2 to be used for relearning and the DNN1 and DNN2 after relearning may be held in the server device 20, or may be held in another device capable of communicating with the edge device 30 and the server device 20.
[0044] [Confidence threshold and offload rate] The method of determining the confidence threshold and the offload rate will be described. FIG. 4 is a diagram showing the relationship between the offload rate and the overall accuracy. FIG. 4 was obtained by determining the fluctuation in the overall accuracy of the inference result accompanying the fluctuation in the offload rate based on the inference result during operation. The threshold is linked to the offload rate, and the confidence threshold is increased when the offload rate is reduced. In FIG. 4, "Offload rate 0" is a state in which all data is processed by the edge device 30 and the accuracy (acc_origin) is low, and "Offload rate 1" is a state in which all data is processed by the server device 20 and the accuracy (acc_origin) is high.
[0045] Furthermore, when the offload rate exceeds 0.4 (threshold is 0.5), the improvement in accuracy is small even if the offload rate is increased, i.e., the confidence threshold is lowered. For this reason, it is considered that a balance between the offload rate (0.4) and accuracy (0.75) can be achieved by setting the confidence threshold to 0.5. In other words, when the offload rate is 0.4, the confidence threshold is set to 0.5. In this way, by setting the threshold according to the balance between the offload rate and accuracy, it becomes possible to adjust the offload rate and overall accuracy according to each use case.
[0046] It is possible to take statistics on the offload rate during operation. The amount of data transmitted from DNN1 to DNN2, that is, from the edge device 30 to the server device 20, may be used as an index value to take statistics on the offload rate. For example, if the edge device 30 processes inference per unit time, for example, at 5 frames per second, and a transmission amount equivalent to 2 frames occurs, the offload rate can be estimated to be 0.4. In this way, it is possible to take statistics on the offload rate and detect changes in the offload rate.
[0047] [Relearning judgment section processing] The re-learning determination unit 241 determines whether to perform re-learning of DNN1 or DNN2 based on the change from the set value of the offload rate, the decrease in inference accuracy, and the amount of learning data.
[0048] [DNN1 re-learning decision] The re-learning determination unit 241 determines whether to perform re-learning of the DNN1 in the edge device 30 in the following cases.
[0049] First, the relearning determination unit 241 determines whether or not to perform relearning of the DNN1 when the offload rate has changed from the set value. In this case, it is considered that the offload rate has increased due to a change in the tendency of the image group to be inferred, and the number of processes in the server device 20 has increased. In other words, it is detected that the overall calculation cost has changed due to the increase in the number of processes in the server device 20. In such a case, it is considered that the accuracy of the DNN1 has decreased because there are many inference results in which the confidence of the inference result of the DNN1 in the edge device 30 is below a predetermined threshold. Note that the above set value may be within a set range, and it may be determined whether or not to perform relearning in either case of a value above the set range or a value below the set range.
[0050] In addition, the re-learning determination unit 241 determines whether to perform re-learning of the DNN1 when the inference accuracy of the DNN1 falls below a predetermined accuracy. In this case, the system administrator determines that the inference accuracy of the DNN1 has fallen, and instructs to perform re-learning of the DNN1. In addition, the re-learning determination unit 241 determines whether to perform re-learning of the DNN1 when the re-learning data for edges reaches the batch amount.
[0051] The re-learning determination unit 241 executes re-learning of the DNN2 in the server device 20 in the following cases. Specifically, the re-learning determination unit 241 determines to execute re-learning of the DNN2 when the inference accuracy of the DNN2 has decreased below a predetermined accuracy. In this case, the system administrator determines that the inference accuracy of the DNN2 has decreased and instructs to execute re-learning of the DNN2.
[0052] Furthermore, the re-learning determination unit 241 determines whether to perform re-learning of the DNN2 when the correction rate of the inference result of the DNN2 by the correction unit 222 becomes equal to or greater than a predetermined rate. This is because it is determined that the inference accuracy of the DNN2 has decreased. Furthermore, the re-learning determination unit 241 determines whether to perform re-learning of the DNN2 when the re-learning data for the cloud reaches the batch amount.
[0053] [Learning data generation process] Next, a description will be given of a learning data generation process in the server device 20. Fig. 5 is a flowchart showing a processing procedure of the learning data generation process in the embodiment.
[0054] 5, in the server device 20, the generation unit 221 acquires the inference result of DNN2 and an image on which inference has been performed in DNN2 (step S11). Next, the generation unit 221 generates data in which the image on which inference has been performed in DNN2 is associated with the inference result of DNN2 for the image as a label as edge re-learning data (step S12), and instructs the storage unit 231 to store the data in the edge re-learning data DB 251 (step S13).
[0055] Then, the learning data generation unit 22 determines whether or not an input of correction to the inference result of the DNN2 is received (step S14). If the learning data generation unit 22 has not received an input of correction to the inference result of the DNN2 for the input image (step S14: No), the learning data generation unit 22 returns to step S11.
[0056] When an input of correction to the inference result by DNN2 for the input image is accepted (step S14: Yes), the correction unit 222 generates data associating the image on which inference was performed by DNN2 with the corrected inference result (correct label) of the image as cloud re-learning data for DNN2 of the server device 20 (step S15). Then, the correction unit 222 instructs the storage unit 231 to store this data in the cloud re-learning data DB 252 (step S16).
[0057] [DNN1 re-learning decision process] Next, a description will be given of the re-learning determination process for the DNN 1. Fig. 6 is a flowchart showing the processing procedure of the re-learning determination process for the DNN 1 in the embodiment.
[0058] As shown in FIG. 6, the relearning determination unit 241 determines whether the offload rate has increased from the set value (step S21). If the offload rate has not increased from the set value (step S21: No), the relearning determination unit 241 determines whether the inference accuracy of the DNN1 has decreased below a predetermined accuracy (step S22). If the inference accuracy of the DNN1 has not decreased below a predetermined accuracy (step S22: No), the relearning determination unit 241 determines whether the edge re-learning data has reached the batch amount (step S23). If the edge re-learning data has not reached the batch amount (step S23: No), the relearning determination unit 241 returns to step S21 and performs a determination on the change in the offload rate.
[0059] If the offload rate increases from the set value (step S21: Yes), or if the inference accuracy of DNN1 drops below a predetermined accuracy (step S22: Yes), or if the re-learning data for edges reaches the batch amount (step S23: Yes), the re-learning judgment unit 241 judges whether to perform re-learning of DNN1 (step S24).
[0060] Next, the re-learning execution unit 242 requests the selection unit 232 to output the edge re-learning data, so that the selection unit 232 selects the edge re-learning data (step S25) and outputs it to the re-learning execution unit 242. The re-learning execution unit 242 executes re-learning of the DNN1 using this edge re-learning data as learning data (step S26).
[0061] The re-learning execution unit 242 performs an accuracy test using test data corresponding to DNN1 (step S27), and if the accuracy improves (step S28: Yes), sets an offload rate and a confidence threshold corresponding to the offload rate, and places the re-learned DNN1 as a model of the edge device 30 (step S29). If the accuracy of the re-learned DNN1 does not improve (step S28: No), it is assumed that the inference accuracy of DNN2 has also decreased. In such a case, the re-learning execution unit 242 returns to step S24 and re-labels the heuristic, or re-learns DNN1 using data re-labeled with a DNN different from DNN2 (for example, a DNN with a higher load and higher accuracy). In such a case, re-learning should be performed for DNN2 as well.
[0062] [DNN2 re-learning decision process] Next, a description will be given of the re-learning determination process for the DNN 2. Fig. 7 is a flowchart showing the processing procedure of the re-learning determination process for the DNN 2 in the embodiment.
[0063] As shown in FIG. 7, the relearning determination unit 241 determines whether or not the correction rate of the inference result of the DNN2 by the correction unit 222 is equal to or greater than a predetermined rate (step S31). If the correction rate of the inference result of the DNN2 by the correction unit 222 is not equal to or greater than the predetermined rate (step S31: No), the relearning determination unit 241 determines whether or not the inference accuracy has decreased below a predetermined accuracy (step S32). If the inference accuracy has not decreased below the predetermined accuracy (step S32: No), the relearning determination unit 241 determines whether or not the re-learned data for the cloud has reached the batch amount (step S33). If the re-learned data for the cloud has not reached the batch amount (step S33: No), the relearning determination unit 241 returns to step S31 and performs a determination on the change in the offload rate.
[0064] If the correction rate of the DNN2 inference result by the correction unit 222 becomes equal to or greater than a predetermined rate (step S31: Yes), or if the inference accuracy falls below a predetermined accuracy (step S32: Yes), or if the re-learning data for the cloud reaches the batch amount (step S33: Yes), the re-learning judgment unit 241 judges whether to perform re-learning of DNN2 (step S34).
[0065] Next, the re-learning execution unit 242 requests the selection unit 232 to output the re-learned data for the cloud, so that the selection unit 232 selects the re-learned data for the cloud (step S35) and outputs it to the re-learning execution unit 242. The re-learning execution unit 242 executes re-learning of the DNN2 using the re-learned data for the cloud as learning data (step S36). The re-learning execution unit 242 performs an accuracy test using test data corresponding to the DNN2 (step S37), and if the accuracy has improved (step S38: Yes), the re-learned DNN2 is placed as a model of the server device 20 (step S39). If the accuracy has not improved (step S38: No), the re-learning execution unit 242 proceeds to step S34 and executes re-learning.
[0066] [Effects of the embodiment] In this manner, in the processing system 100 according to the present embodiment, at least one of the edge device 30 and the server device 20 determines whether or not the tendency of the image group (target data group) to be inferred has changed based on a change in load or a decrease in inference accuracy in at least one of the edge device and the server device. Then, in the processing system 100, when it is determined that the tendency of the image group has changed, re-learning of at least one of DNN1 and DNN2 is executed. Therefore, according to the processing system 100, the timing of re-learning for each of DNN1 and DNN2 can be determined, and re-learning of DNN1 and DNN2 can be automatically executed.
[0067] In the processing system 100, re-learning of at least one of DNN1 and DNN2 is performed using data that contributes greatly to load fluctuations or to a decrease in inference accuracy among the image group processed during operation of the system, so that DNN1 and DNN2 that can deal with load fluctuations or a decrease in inference accuracy can be constructed by re-learning. In the processing system 100, by disposing these DNN1 and DNN2 in the edge device 30 and the server device 20, it is possible to maintain the accuracy of the models disposed on the edge and the cloud, respectively.
[0068] In the processing system 100, among the images processed during the operation of the system, the images on which inference processing has actually been performed in the DNN2 and the inference results of the images by the DNN2 are used as learning data to perform re-learning of the DNN1. In other words, in the processing system 100, images on which inference has actually been performed in the DNN1 and to which the inference results of the DNN2, which are more accurate than the DNN1, are labeled as edge re-learning data, and the DNN1 is re-learned using the edge re-learning data. Therefore, the DNN1 becomes a domain-specific model every time it is re-learned, and the accuracy required for the edge device 30 can be appropriately maintained.
[0069] In the processing system 100, among the images processed during the operation of the system, the images on which the inference process was actually performed in the DNN2 and the corrected inference result obtained by correcting the inference result of the DNN2 for the image are used as training data to perform re-learning of the DNN2. That is, in the DNN2, an image in which the inference performed in the DNN2 was incorrect and to which a correct answer label is attached is generated as re-learning data for the cloud, and the DNN2 is re-learned using the re-learning data for the cloud, thereby improving the accuracy of the DNN2.
[0070] In this way, according to the processing system 100, it is possible to appropriately perform re-learning of models deployed on the edge and the cloud while reducing the burden on the administrator regarding the model re-learning process, thereby maintaining the accuracy of the models.
[0071] In this embodiment, there may be a plurality of edge devices 30 or a plurality of server devices 20, or there may be a plurality of both edge devices 30 and server devices 20. In this case, edge re-learning data is generated for each edge device 30, and cloud re-learning data is generated for each server device 20, and re-learning of each model is performed using the corresponding learning data.
[0072] [System configuration, etc.] Each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.
[0073] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.
[0074] [program] 8 is a diagram showing an example of a computer in which the edge device 30 and the server device 20 are realized by executing a program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 may also have the accelerator described above to assist in calculations. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0075] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.
[0076] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes of the edge device 30 and the server device 20 are implemented as program modules 1093 in which computer-executable codes are written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configurations in the edge device 30 and the server device 20 are stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).
[0077] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes them.
[0078] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.
[0079] Although the embodiments of the present invention have been described above, the present invention is not limited by the descriptions and drawings that form part of the disclosure of the present invention according to the present embodiments. In other words, other embodiments, examples, and operation techniques made by those skilled in the art based on the present embodiments are all included in the scope of the present invention. [Explanation of symbols]
[0080] 20 Server device 21,31 Reasoning part 22 Learning data generation unit 23 Learning Data Management Department 24 Re-learning section 30 Edge Device 32 Judgment section 100 Processing Systems 221 Generation part 222 Correction section 231 Storage Unit 232 Selection section 241 Relearning Judgment Unit 242 Re-learning execution unit 251 Edge Re-learning Data DB 252 Cloud Re-Learning Data DB
Claims
1. A processing method executed by a processing system that performs a first inference in an edge device and a second inference in a server device, the method comprising: a determination step of determining whether or not a trend of a target data group to be inferred has changed in at least one of the edge device and the server device based on a change in load or a decrease in inference accuracy in at least one of the edge device and the server device; a re-learning step of executing re-learning of at least one of a first model performing the first inference and a second model performing the second inference when it is determined in the determination step that the trend of the target data group has changed; a generating step of generating data in which an inference result of the second inference is associated as a label with the target data on which inference has been performed in the second inference among the target data group, as edge re-learning data for the first model; a correction process of receiving an input of a correction to an inference result in the second inference of the target data on which the second inference has been executed among the target data group, and generating data in which a corrected inference result in which a label correction has been applied to the inference result in the second inference of the target data is associated with the target data on which the second inference has been executed, as cloud re-learning data for the second model; Including, The determination step determines whether to perform re-learning of the first model when a processing rate in the server device increases from a set value, when an inference accuracy of the first model falls below a predetermined accuracy, or when the retained re-learning data for edges reaches a batch amount, The processing method is characterized in that the determination step determines whether to perform re-learning of the second model when the correction rate for the inference result in the second inference becomes equal to or greater than a predetermined rate, when the inference accuracy of the second model falls below a predetermined accuracy, or when the retained re-learning data for the cloud reaches a batch amount.
2. A processing system that performs a first inference in an edge device and a second inference in a server device, a determination unit that determines whether or not a trend of a target data group for inference has changed in at least one of the edge device and the server device based on a change in load or a decrease in inference accuracy in at least one of the edge device and the server device; a re-learning unit that executes re-learning of at least one of a first model that performs the first inference and a second model that performs the second inference when the determination unit determines that a trend of the target data group has changed; a generation unit that generates data in which an inference result of the second inference of the target data is associated as a label with the target data on which inference has been performed in the second inference among the target data group, as edge re-learning data for the first model; a correction unit that receives an input of a correction to an inference result in the second inference of the target data on which the second inference has been executed among the target data group, and generates data as cloud re-learning data for the second model, in which the target data on which the second inference has been executed is associated with a corrected inference result in which a label correction has been applied to the inference result in the second inference of the target data; having the determination unit determines whether to execute re-learning of the first model when a processing rate in the server device increases from a set value, when an inference accuracy of the first model falls below a predetermined accuracy, or when the retained re-learning data for the edge reaches a batch amount, The processing system is characterized in that the determination unit determines to perform re-learning of the second model when a correction rate for the inference result in the second inference becomes equal to or greater than a predetermined rate, when the inference accuracy of the second model falls below a predetermined accuracy, or when the retained re-learning data for the cloud reaches a batch amount.
3. a determining step of determining whether or not a trend of a target data group to be inferred has changed in at least one of the edge device and the server device based on a change in load or a decrease in inference accuracy in at least one of the edge device and the server device; a re-learning step of executing re-learning of at least one of a first model for performing a first inference in the edge device and a second model for performing a second inference in the server device when it is determined in the determination step that a trend of the target data group has changed; a generating step of generating data in which an inference result of the second inference is associated as a label with the target data on which inference has been performed in the second inference among the target data group, as edge re-learning data for the first model; a correction step of receiving an input of a correction to an inference result in the second inference of the target data on which the second inference has been executed among the target data group, and generating data in which a corrected inference result in which a label correction has been applied to the inference result in the second inference of the target data is associated with the target data on which the second inference has been executed, as cloud re-learning data for the second model; Run the following on your computer: The determination step determines whether to perform re-learning of the first model when a processing rate in the server device increases from a set value, when an inference accuracy of the first model falls below a predetermined accuracy, or when the retained re-learning data for edges reaches a batch amount, The determination step determines whether to perform re-learning of the second model when a correction rate for an inference result in the second inference becomes equal to or greater than a predetermined rate, when the inference accuracy of the second model falls below a predetermined accuracy, or when the retained re-learning data for cloud reaches a batch amount. Processing program.
Citation Information
Patent Citations
Machine learning management program, machine learning management device and machine learning management method
JP2018010475A
Recognition device, recognition system, recognition method, and program
JP2018045369A
Image classifier and program
JP2020024534A
