Processing System and Processing Method

Abstract

The edge device (30) of the processing system (100) has: an inference unit (32) that processes the data to be processed and outputs the processing result of the data to be processed; a determination unit (33) that determines to cause the server device (20) to execute processing related to the data to be processed when the evaluation value satisfies the condition, and determines that the evaluation value is a value included in the range to be processed by the edge device (30) when the processing result of the data to be processed satisfies a specified evaluation, and outputs the processing result of the data to be processed obtained by the processing of the inference unit (32), where the evaluation value is a value for evaluating which of the edge device (30) and the server device (20) should process the data to be processed; and a communication unit (34) that, when the determination unit (33) determines to cause the server device (20) to execute processing related to the data to be processed, transmits data for causing the server device (20) to execute processing related to the data to be processed.

Description

Technical Field

[0001] The present invention relates to a processing system and a processing method. Background Art

[0002] The amount of data collected by IoT devices represented by sensors is huge. Therefore, when aggregating and processing the collected data using cloud computing, a huge amount of communication traffic is generated. Accordingly, edge computing for processing the collected data using an edge device close to a user has attracted attention.

[0003] However, compared with a device other than the edge device (hereinafter, simply referred to as cloud) that is configured at a physically and logically more distant position from the user relative to the edge device, the amount of computation and resources such as memory of the device used in the edge device are limited. Therefore, when performing a process with a large computational load using the edge device, there is a case where it takes a very long time until the process is completed, or it also takes time before other processes with a small amount of computation are completed.

[0004] Here, as one of the processes with a large amount of computation, a process related to machine learning is cited. In Non-Patent Document 1, an application of so-called adaptive learning to an edge cloud is proposed. That is, the method described in Non-Patent Document 1 realizes the following operation: a learned model obtained by learning using general learning data in the cloud is deployed to the edge device, and the data obtained by the edge device is used to learn the model learned in the cloud again, thereby effectively utilizing the advantages of the cloud and the edge device.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Non-Patent Document 1: Osakoshi et al., "Proposal and Evaluation of DNN Model Operation Method by Cloud-Edge Collaboration", Proceedings of the 80th National Conference of the Information Processing Society of Japan 2018(1), 3-4, 2018-03-13. Summary of the Invention

[0008] Problems to be Solved by the Invention

[0009] However, the method described in Non-Patent Document 1 has not studied inference processing. For inference, the more complex the data of the inference object to be processed and the more difficult the problem to be solved, the greater the amount of computation for inference. It is preferably considered to perform such a process with a large amount of computation in the cloud. However, in order to determine the process with a large amount of computation that should be performed in the cloud, it is necessary to determine the complexity of the data of the inference object and the difficulty of the problem to be solved in the edge device.

[0010] In addition, from a perspective different from the difficulty of the problem to be solved, there are the inference accuracy and response required by the user. That is, there are cases where the user requires a relatively low inference accuracy but a quick response, and cases where the response is slow but the inference accuracy is high. However, a method for determining, by the edge device, a computationally intensive process that should be processed in the cloud while considering the inference accuracy and response required by the user is not described in Non-Patent Document 1.

[0011] The present invention has been completed in view of the above circumstances, and an object thereof is to provide a processing system and a processing method capable of controlling the execution of processing in which an edge device collaborates with a cloud according to user expectations.

[0012] Means for Solving the Problem

[0013] To solve the above problems and achieve the object, a processing system of the present invention is a processing system using an edge device and a server device, characterized in that the edge device has: an edge processing unit that processes processing target data and outputs a processing result of the processing target data; a determination unit that determines to cause the server device to execute processing related to the processing target data when an evaluation value satisfies a condition, and determines that the evaluation value is a value within a range that should be processed by the edge device when the processing result of the processing target data satisfies a specified evaluation, and outputs the processing result of the processing target data obtained by processing by the edge processing unit, where the evaluation value is a value for evaluating which one of the edge device and the server device should process the processing target data; and a transmission unit that transmits data for causing the server device to execute processing related to the processing target data when the determination unit determines to cause the server device to execute processing related to the processing target data.

[0014] In addition, the processing method of the present invention is a processing method executed by a processing system using an edge device and a server device, characterized in that the processing method includes: a processing step in which the edge device processes the data to be processed and outputs a processing result of the data to be processed; a determination step in which the edge device determines to cause the server device to execute processing related to the data to be processed when an evaluation value satisfies a condition, and determines the evaluation value to be a value included in the range that should be processed by the edge device when the processing result of the data to be processed satisfies a specified evaluation, and outputs the processing result of the data to be processed obtained by processing in the processing step, where the evaluation value is a value for evaluating which one of the edge device and the server device should process the data to be processed; and a sending step in which, when it is determined in the determination step that the server device is to execute processing related to the data to be processed, the edge device sends data for causing the server device to execute processing related to the data to be processed.

[0015] Effects of the Invention

[0016] According to the present invention, it is possible to control the execution of processing in which an edge device and a cloud cooperate according to a user's expectation, and it is possible to make the entire system composed of the device and the cloud operate efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a diagram for explaining an outline of the processing method of the processing system according to Embodiment 1.

[0018] Figure 2 It is a diagram for explaining an example of DNN1 and DNN2.

[0019] Figure 3 It is a diagram for explaining an example of DNN1 and DNN2.

[0020] Figure 4 It is a diagram schematically showing an example of the structure of the processing system according to Embodiment 1.

[0021] Figure 5 It is a timing diagram showing the processing flow of the processing system according to Embodiment 1.

[0022] Figure 6 It is a diagram showing a structural example of a learning device that learns a lightweight model and a high-precision model.

[0023] Figure 7 It is a diagram showing an example of the loss in each case.

[0024] Figure 8It is a flowchart showing the learning process of a high-precision model.

[0025] Figure 9 It is a flowchart showing the learning process of a lightweight model.

[0026] Figure 10 It is a diagram schematically showing an example of the structure of the processing system according to Embodiment 2.

[0027] Figure 11 It is a timing diagram schematically showing the process flow of the processing system according to Embodiment 2.

[0028] Figure 12 It is a diagram schematically showing another example of the structure of the processing system according to Embodiment 2.

[0029] Figure 13 It is a diagram schematically showing an example of the structure of the processing system according to Embodiment 3.

[0030] Figure 14 It is schematically showing Figure 13 a diagram of an example of the edge device shown.

[0031] Figure 15 It is schematically showing Figure 13 a diagram of an example of the server device shown.

[0032] Figure 16 It is a timing diagram showing the process flow of the processing system according to Embodiment 3.

[0033] Figure 17 It is a diagram schematically showing an example of the structure of the processing system according to Embodiment 4.

[0034] Figure 18 It is schematically showing Figure 17 a diagram of an example of the edge device shown.

[0035] Figure 19 It is a timing diagram showing the process flow of the processing system according to Embodiment 3.

[0036] Figure 20 It is a diagram showing an overview of the processing system in the modification examples of Embodiments 1 to 4.

[0037] Figure 21 It is a diagram showing the variations in the functions of the DNN, determination unit, encoding unit, decoding unit, and the variations in the communication content.

[0038] Figure 22 It is a diagram showing an example of a computer that realizes an edge device and a server device by executing a program. Detailed implementation

[0039] A description will be given in detail below of an embodiment of the present invention with reference to the accompanying drawings. In addition, the present invention is not limited to this embodiment. In the description of the drawings, the same reference numerals are assigned to the same parts.

[0040] [Embodiment 1]

[0041] [Outline of Embodiment 1]

[0042] An embodiment of the present invention will be described. In Embodiment 1 of the present invention, a processing system that performs inference processing using a learned high-precision model and a lightweight model will be described. In addition, in the processing system of the embodiment, as an example of the model used in the inference processing, the case of using a DNN (Deep Neural Network) is described. In the processing system of the embodiment, a neural network other than the DNN may be used, and signal processing with a low amount of computation and signal processing with a high amount of computation may be used instead of the learned model.

[0043] Figure 1 This is a diagram for explaining the outline of the processing method of the processing system according to Embodiment 1. In the processing system according to Embodiment 1, a high-precision model and a lightweight model form a model cascade. In the processing system according to Embodiment 1, an evaluation value is used to control which of the edge device and the cloud (server device) performs the processing, where the edge device uses a high-speed and low-precision lightweight model (for example, DNN1), and the cloud (server device) uses a low-speed and high-precision high-precision model (for example, DNN2). For example, the server device is a device configured at a location physically and logically far from the user. The edge device is an IoT device and various terminal devices configured at a location physically and logically close to the user, and has fewer resources than the server device.

[0044] DNN1 and DNN2 are models that output inference results based on the input data. In Figure 1 's example, DNN1 and DNN2 take an image as input and infer the probability of each category of the object presented in the image. In addition, Figure 1 The two images shown are the same image. For example, DNN1 is a model trained in consideration of which model among DNN1 and DNN2 can be used for inference to obtain greater benefits desired by the user. DNN1 and DNN2 are optimized so that the evaluation value can obtain the best value.

[0045] The user's expectations are, for example, high-precision inference results, reduced data communication volume, high-speed calculation processing, resource optimization of edge devices, etc. The evaluation value is a value for evaluating which of the edge device and the server device should process the data to be processed while meeting the user's expectations. The more difficult the processing of the data to be processed is, the more likely the evaluation value is to fall within the range where the server device should perform the evaluation.

[0046] As Figure 1 shown, in the processing system, an evaluation value related to the inference of the class classification of the object presented in the input image by DNN1 is obtained. Then, in the processing system, when the obtained evaluation value satisfies a specified condition, for example, satisfies a specified value, the inference result of DNN1 is adopted. That is, the inference result of the lightweight model is output as the final estimation result of the model cascade. On the other hand, in the processing system, when the evaluation value does not satisfy the specified value, the inference result obtained by inputting the same image into DNN2 is output as the final inference result. The cases where the specified value is satisfied include, for example, whether the condition based on a preset threshold is satisfied and whether it is included within a specified range.

[0047] In this way, in the processing system of Embodiment 1, based on the evaluation value, the edge device or the server device is selected to process the data to be processed, where the evaluation value is a value for evaluating which of the edge device and the server device should process the data to be processed according to the user's expectations. Therefore, in the processing system of Embodiment 1, it is possible to control which of the edge device and the cloud performs the processing according to the user's expectations.

[0048] [Lightweight Model and High-Precision Model]

[0049] Next, DNN1 and DNN2 will be described. Figure 2 And Figure 3 are diagrams for explaining an example of DNN1 and DNN2. A DNN has an input layer for inputting data, one or more intermediate layers for performing various conversions on the data input from the input layer, and an output layer for outputting so-called inference results such as probabilities or likelihoods. When it is necessary to keep the output values of the input data anonymous, the output values output from each layer can also be made irreversible.

[0050] As Figure 2As shown, the processing system may also use separate and independent DNN1a and DNN2a. For example, after training DNN2a using known methods, DNN1a is trained considering which of DNN1a and DNN2a can be used for inference to provide greater benefits desired by the user. DNN1a is trained in such a way that it outputs a value related to the evaluation value. DNN1a outputs the output value of the intermediate layer of DNN1a, i.e., the intermediate output value, as the value related to the evaluation value. The evaluation value can be a value calculated based on the intermediate output value or the intermediate output value itself. Regarding the intermediate output value, the result obtained by inputting the intermediate output value of a specified intermediate layer into a cost function that can perform learning more satisfying the user's expectations can be used as the intermediate output value. This cost function, for example, obtains the correlation between the intermediate output value and the likelihood. In addition, the output of any intermediate layer of the learned model designed using only the problem to be solved as the cost function can also be used as the intermediate output value. This is because, for example, in the case of using a network (such as a CNN) that easily has the property of reflecting the characteristics of the input data in higher-order intermediate layers, the effective features that can be used for the problem to be solved have been extracted into the output value of the higher-order intermediate layer. Additionally, the same task with different accuracies and performances can be assigned to DNN1a and DNN2a, or different tasks can be assigned to DNN1a and DNN2a.

[0051] In addition, as Figure 3 shown, the processing system divides DNN3, which is a DNN trained as a single entity between the R-th layer and the (R + 1)-th layer using a specified criterion, into DNN1b and DNN2b. Then, the processing system can apply the front-stage DNN1b to the edge device and the back-stage DNN1b to the server device. In this case, DNN1b outputs the intermediate output value from the R-th intermediate layer as the evaluation value. Additionally, DNN1b can also output the intermediate output value from a layer before the R-th intermediate layer as the evaluation value.

[0052] In addition, the evaluation value is not limited to the intermediate output value output from DNN1a or DNN1b. For example, the evaluation value can be the inference error output from DNN1a, or a value based on the inference error. For example, the evaluation value can also be a value representing the confidence level indicating whether the result of the processing by the edge device is the correct answer. In addition, the evaluation value can also be a value determined based on any of the following items: the time required until the processing result of the processing target data is obtained; the acquisition deadline of the processing result of the processing target data; the usage status of the resources of the edge device when determining which of the edge device and the server device should process the processing target data; whether the processing target data is data in which an event has occurred compared to other data. In addition, the usage status of the resources of the edge device can be the usage rate of the CPU or memory of the edge device itself, the power consumption, etc., or the difference in the workload or resource usage rate between the edge device and other edge devices. In addition, an event refers to a situation where the object frame has changed by more than a desired size compared to the previous frame, or a situation where an object to be estimated in detail has been generated. In addition, it can also be designed to transmit the object that has been calculated in the edge device and the data representing the result to the server device, and the server device only performs calculations on the objects that have not been calculated in the edge device. Specifically, the coordinate values of the bounding box, the class classification result, and its reliability can be sent together, and the server device only performs calculations on the objects that do not meet the reliability.

[0053] [Processing System]

[0054] Next, the structure of the processing system will be described. Figure 4 It is a diagram schematically showing an example of the structure of the processing system of Embodiment 1.

[0055] The processing system 100 of the embodiment includes a server device 20 and an edge device 30. In addition, the server device 20 and the edge device 30 are connected via a network N. The network N is the Internet, for example. In this case, the server device 20 can be a server set in a cloud environment. In addition, the edge device 30 can also be an IoT device and various terminal devices.

[0056] The server device 20 and the edge device 30 are respectively implemented by loading a prescribed program into a computer including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and causing the CPU to execute the prescribed program. In addition, so-called accelerators represented by a GPU, a VPU (Vision Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a dedicated AI (Artificial Intelligence) chip can also be used. The server device 20 and the edge device 30 can also respectively have an NIC (Network Interface Card), etc., so as to communicate with other devices via an electrical communication line such as a LAN (Local Area Network) or the Internet.

[0057] As Figure 4 shown, the server device 20 stores the DNN2 as a learned high-precision model. The DNN2 includes information such as model parameters. In addition, the server device 20 has an inference unit 22.

[0058] The inference unit 22 inputs inference data (data to be processed) to the DNN2 and obtains an inference result (processing result). The inference unit 22 accepts the input of the inference data and outputs the inference result. Assume that the inference data is data with an unknown label. For example, the inference data is an image. In addition, when returning the inference result to the user, the inference result obtained by the inference unit 22 can also be transmitted to the edge device and returned to the user from the edge device.

[0059] Here, the server device 20 and the edge device 30 form a model cascade. Therefore, the inference unit 22 does not always perform inference on the inference data. When it is determined to cause the server device 20 to execute the inference process related to the inference data, the inference unit 22 performs inference based on the DNN2.

[0060] The edge device 30 stores the DNN1 as a lightweight model that has completed learning. The DNN1 contains information such as model parameters. The DNN1 is trained considering which of the DNN1 and DNN2 models can be used for inference to bring greater benefits desired by the user. The following parameters are set in the DNN1: By pre-learning this parameter, the cascaded model including the DNN1 and DNN2 is optimized considering whether the benefits desired by the user are large. In addition, the edge device 30 has an inference unit 32 (edge processing unit), a determination unit 33, and a communication unit 34 (transmission unit).

[0061] The inference unit 32 inputs inference data (data to be processed) to the DNN1 and obtains an inference result. The inference unit 32 accepts the input of the inference data, processes the inference data, and outputs an inference result (the processing result of the data to be processed).

[0062] The determination unit 33 determines whether an evaluation value satisfies a specified value, where the evaluation value is designed to reflect the user's expectation and is used to evaluate which of the edge device 30 and the server device 20 should process the inference data.

[0063] When the evaluation value satisfies the specified value, the determination unit 33 considers that the inference result of the inference data satisfies the specified evaluation, determines that the evaluation value is within the range that should be processed by the edge device 30, and outputs the inference result inferred by the inference unit 32. When the evaluation value does not satisfy the specified value, the determination unit 33 considers that the evaluation value is within the range that should be evaluated by the server device 20, and determines that the server device 20 performs processing (inference processing) related to the inference data. As described above, the evaluation value is an intermediate output value, an inference error, a confidence level, etc. In addition, the determination unit 33 can also narrow down the data to be processed that is the transmission target. For example, the determination unit 33 narrows down the data to be processed to the data of the nodes required for the processing of the DNN2. Here, a criterion for narrowing down in the case where the inference data is an image is exemplified. When an event occurs in a partial area of the image, the determination unit 33 can narrow down the data to be processed to that partial area, or narrow down the data to be processed to the area required for making an estimate related to the event. In addition, when determining whether to process by the edge device or the server device for each area of the image, the determination unit 33 only needs to narrow down the data to be processed to the area to be processed by the server device. Although an example of narrowing down from a spatial perspective is exemplified, the determination unit 33 can also narrow down from a temporal perspective.

[0064] The communication unit 34 communicates with other devices (e.g., the server device 20) via the network N. When the determination unit 33 determines to cause the server device 20 to execute an inference process related to inference data, the communication unit 34 sends the processing data for causing the server device 20 to execute the inference process to the server device 20. When the evaluation value is an intermediate output value, the communication unit 34 sends the intermediate output value to the server device 20.

[0065] [Processing steps of the processing system]

[0066] Figure 5 is a timing chart showing the process flow of the processing system of Embodiment 1. As Figure 5 shown, first, in the edge device 30, when the inference unit 32 receives the input of inference data (step S1), it inputs the inference data to DNN1 (step S2).

[0067] The determination unit 33 obtains the intermediate output value of DNN1 (steps S3, S4), and obtains the evaluation value (step S5). The determination unit 33 determines whether the evaluation value satisfies a specified value (step S6).

[0068] When the evaluation value satisfies the specified value (step S6: Yes), the determination unit 33 inputs the intermediate output value to the next intermediate layer of the layer in the intermediate layer of DNN1 that output the intermediate output value (step S7). The inference unit 32 obtains the inference result of DNN1 (step S8), and outputs the obtained inference result of DNN1 (step S9).

[0069] In contrast, when the evaluation value does not satisfy the specified value (step S6: No), the determination unit 33 sends the processing data for causing the server device 20 to execute the inference process to the server device 20 via the communication unit 34 (steps S10, S11). For example, the processing data is the inference data and the confidence of DNN1. Alternatively, the processing data is the intermediate output value.

[0070] In the server device 20, the inference unit 22 inputs the processing data into the DNN2 (step S11) and obtains the inference result of the DNN2 (steps S12 and S13). The inference result of the DNN2 is sent to the edge device 30 (steps S14 and S15) and output from the edge device 30 (step S16). In addition, in the present embodiment, a structure for returning the inference result to the user is assumed, and it is configured to output the final inference result from the edge device 30. However, when the final inference result is used on the server device 20 side, it may be configured to output the inference result of the DNN2 from the server device 20, or the inference result of the DNN2 may be kept as it is in the server device 20. When using the inference result of the DNN1, when using the inference result on the server device 20 side, the edge device 30 only needs to send the inference result to the server device 20.

[0071] [Effect of Embodiment 1]

[0072] Thus, according to Embodiment 1, the edge device or the server device is selected based on the evaluation value to process the data to be processed, where the evaluation value is a value that evaluates which of the edge device and the server device should process the data to be processed according to the user's expectation. Therefore, in the processing system of Embodiment 1, it is possible to control which of the edge device and the cloud executes the processing according to the user's expectation.

[0073] In addition, in the present Embodiment 1, the case where the number of edge devices 30 and the number of server devices 20 are both single has been described. However, the number of edge devices 30 or server devices 20 may also be multiple. In addition, both the edge devices 30 and the server devices 20 may be multiple.

[0074] [Application Example]

[0075] The following is an example: applying the present Embodiment 1 for the expectation of high-precision inference results, and using the confidence as the evaluation value. First, the learning of the lightweight model and the high-precision model for realizing high-precision inference results will be described.

[0076] Figure 6 is a diagram showing a structural example of a learning device for learning the lightweight model and the high-precision model. As Figure 2 shown, the learning device 10 accepts the input of the learning data and outputs the learned high-precision model information and the learned lightweight model information. In addition, the learning device 10 has a high-precision model learning unit 11 and a lightweight model learning unit 12.

[0077] The high-precision model learning unit 11 includes an estimation unit 111, a loss calculation unit 112, and an update unit 113. In addition, the high-precision model learning unit 11 stores high-precision model information 114. The high-precision model information 114 is information such as parameters for constructing a high-precision model. The learning data is data with known labels. For example, the learning data is a combination of an image and a label (the correct category).

[0078] The estimation unit 111 inputs the learning data to the high-precision model constructed based on the high-precision model information 114 and obtains an estimation result. The estimation unit 111 accepts the input of the learning data and outputs the estimation result.

[0079] The loss calculation unit 112 calculates a loss based on the estimation result obtained by the estimation unit 111. The loss calculation unit 112 accepts the input of the estimation result and the label, and outputs the loss. For example, the loss calculation unit 112 calculates a loss, where the smaller the confidence in the label in the estimation result obtained by the estimation unit 111, the larger the loss. For example, the confidence is the degree of the possibility that the estimation result is the correct solution. For example, the confidence can be the probability output by the aforementioned multi-class classification model. Specifically, the loss calculation unit 112 can calculate the softmax cross-entropy described later as the loss.

[0080] The update unit 113 updates the parameters of the high-precision model so that the loss is optimized. For example, if the high-precision model is a neural network, the update unit 113 updates the parameters of the high-precision model by the error backpropagation method or the like. Specifically, the update unit 113 updates the high-precision model information 114. The update unit 113 accepts the input of the loss calculated by the loss calculation unit 112 and outputs the information of the updated model.

[0081] The lightweight model learning unit 12 includes an estimation unit 121, a loss calculation unit 122, and an update unit 123. In addition, the lightweight model learning unit 12 stores lightweight model information 124. The lightweight model information 124 is information such as parameters for constructing a lightweight model.

[0082] The estimation unit 121 inputs the learning data to the lightweight model constructed based on the lightweight model information 124 and obtains an estimation result. The estimation unit 121 accepts the input of the learning data and outputs the estimation result.

[0083] Here, the high-precision model learning unit 11 performs learning of the high-precision model based on the output of the high-precision model. On the other hand, the lightweight model learning unit 12 performs learning of the lightweight model based on the outputs of both the high-precision model and the lightweight model.

[0084] The loss calculation unit 122 calculates the loss based on the estimation result obtained by the estimation unit. The loss calculation unit 122 receives the estimation result of the high-precision model, the estimation result of the lightweight model, and the input of the label, and outputs the loss. The estimation result of the high-precision model can be the estimation result obtained by further inputting the learning data into the high-precision model that has been learned by the high-precision model learning unit 11. More specifically, the lightweight model learning unit 12 receives the input related to whether the estimation result of the high-precision model is the correct answer. For example, if the category with the highest probability output by the high-precision model is consistent with the label, its estimation result is the correct answer.

[0085] In addition to aiming at maximizing the estimation accuracy in the lightweight model itself, the loss calculation unit 122 also calculates the loss aiming at maximizing the profit in the case of forming a model cascade. Here, it is assumed that the higher the estimation accuracy, the greater the profit, and the smaller the calculation cost, the greater the profit.

[0086] For example, the high-precision model has the characteristics of high estimation accuracy but large calculation cost. In addition, for example, the lightweight model has the characteristics of low estimation accuracy but small calculation cost. Here, the loss calculation unit 122 calculates the loss Loss as shown in Equation (1). Where w is the weight, which is a parameter set in advance.

[0087]

Equation 1

[0088] Loss = L classifier + wL cascade …(1)

[0089] Here, L classifier is the softmax cross-entropy in the multi-class classification model. In addition, L classifier is an example of the first term. The smaller the confidence of the estimation result of the lightweight model with respect to the correct answer, the larger L classifier . L classifier is represented by Equation (2). Where N is the number of samples. In addition, k is the number of classes. In addition, y is the label indicating the class of the correct answer. In addition, q is the probability output by the lightweight model. i is the number identifying the sample. In addition, j is the number identifying the class. In the i-th sample, if the j-th class is the correct answer, the label y i,j is 1, and if the j-th class is not the correct answer, the label y i,j is 0.

[0090]

Equation 2

[0091]

[0092] In addition, L cascade is the term used to maximize the profit in the case of forming a model cascade. L cascadeIndicates the loss when the estimation results of the high-precision model and the lightweight model are adopted based on the confidence of the lightweight model for each sample. Here, the loss includes a penalty for inappropriate confidence and the cost of using the high-precision model. In addition, the loss is divided into four categories according to the combination of whether the estimation result of the high-precision model is the correct solution and whether the estimation result of the lightweight model is the correct solution. When the estimation of the high-precision model is not the correct solution and the confidence of the lightweight model is low, the penalty increases, which will be described in detail later. On the other hand, when the estimation of the lightweight model is the correct solution and the confidence of the lightweight model is high, the penalty decreases. L cascade Is represented by Equation (3).

[0093]

Equation 3

[0094]

[0095] 1 fast Is the following indicator function: returns 0 if the estimation result of the lightweight model is the correct solution, and returns 1 if the estimation result of the lightweight model is not the correct solution. In addition, 1 acc Is the following indicator function: returns 0 if the estimation result of the high-precision model is the correct solution, and returns 1 if the estimation result of the high-precision model is not the correct solution. COST acc Is the cost consumed by the estimation using the high-precision model and is a preset parameter.

[0096] max j q i,j Is the maximum value of the probability output by the lightweight model and is an example of confidence. It can be said that when the estimation result is the correct solution, the greater the confidence, the higher the estimation accuracy. On the other hand, it can be said that when the estimation result is not the correct solution, the greater the confidence, the lower the estimation accuracy.

[0097] (The max in Equation (3)) j q i,j 1 fast Is an example of the second term. When the estimation result of the lightweight model is not the correct solution, the greater the confidence of the estimation result of the lightweight model, the greater the max j q i,j 1 fast Moreover, (1 - max in Equation (3)) j q i,j )1 acc Is an example of the third term. When the estimation result of the high-precision model is not the correct solution, the smaller the confidence of the estimation result of the lightweight model, the greater the (1 - max j q i,j )1 acc Moreover, (1 - max in Equation (3)) j q i,j)COST acc is an example of Item 4. The smaller the confidence of the estimation result of the lightweight model, the larger (1 - max j q i,j )COST acc becomes. In this case, the minimization of the loss achieved by the update unit 123 is equivalent to the optimization of the loss.

[0098] The update unit 123 updates the parameters of the lightweight model so that the loss is optimized. That is, the update unit 123 updates the parameters of the lightweight model based on the estimation result of the lightweight model and the estimation result obtained by inputting the learning data to the high-precision model, so that the model cascade including the lightweight model and the high-precision model is optimized, where the high-precision model is a model that outputs an estimation result based on the input data, and its processing speed is slower than that of the lightweight model and its estimation accuracy is higher than that of the lightweight model. The update unit 123 receives the input of the loss calculated by the loss calculation unit 122 and outputs the information of the updated model.

[0099] Figure 7 is a diagram showing an example of the loss in each case. The vertical axis is the value of L cascade . In addition, the horizontal axis is the value of max j q i,j . In addition, let COST acc = 0.5. max j q i,j is the confidence of the estimation result of the lightweight model, which is simply referred to as the confidence here.

[0100] Figure 7 The "□" of is the value of L cascade corresponding to the confidence when the estimation results of both the lightweight model and the high-precision model are correct answers. In this case, the larger the confidence, the smaller the value of L cascade . This is because if the estimation result of the lightweight model is the correct answer, the larger the confidence, the more likely the lightweight model is to be adopted.

[0101] Figure 7 The "◇" of is the value of L cascade corresponding to the confidence when the estimation result of the lightweight model is the correct answer and the estimation result of the high-precision model is not the correct answer. In this case, the larger the confidence, the smaller the value of L cascade . In addition, compared with the case of "□", the maximum value of L cascade and the degree of decrease are larger. This is because when the estimation result of the high-precision model is not the correct answer and the estimation result of the lightweight model is the correct answer, the larger the confidence, the stronger the tendency to adopt the lightweight model becomes.

[0102] Figure 7The blackened square is the value of L corresponding to the confidence level when the estimation result of the lightweight model is a non-positive solution and the estimation result of the high-precision model is a positive solution. cascade In this case, the greater the confidence level, the greater the value of L. cascade This is because when the estimation result of the lightweight model is a non-positive solution, the smaller the confidence level, the more difficult it is to adopt the estimation result.

[0103] Figure 7 The "◆" is the value of L corresponding to the confidence level when the estimation results of both the lightweight model and the high-precision model are non-positive solutions. cascade In this case, the greater the confidence level, the smaller the value of L. cascade However, compared with the case of "□", the value of L is larger. cascade This is because since the estimation results of both models are non-positive solutions, the loss is always large. In such a situation, the lightweight model should be able to make accurate estimations.

[0104] [Learning Process]

[0105] Figure 8 is a flowchart showing the learning process of the high-precision model. As Figure 8 shown, first, the estimation unit 111 estimates the category of the learning data using the high-precision model (step S101).

[0106] Next, the loss calculation unit 112 calculates the loss based on the estimation result of the high-precision model (step S102). Then, the update unit 113 updates the parameters of the high-precision model so that the loss is optimized (step S103). In addition, the learning device 10 can also repeat the processing of steps S101 to S103 until the end condition is satisfied. The end condition can be that the processing has been repeated a predetermined number of times, or the update width of the parameters has converged.

[0107] Figure 9 is a flowchart showing the learning process of the lightweight model. As Figure 9 shown, first, the estimation unit 121 estimates the category of the learning data using the lightweight model (step S201).

[0108] Next, the loss calculation unit 122 calculates the loss based on the estimation result of the lightweight model, the estimation result of the high-precision model, and the cost of estimation using the high-precision model (step S202). Then, the update unit 123 updates the parameters of the lightweight model so that the loss is optimized (step S203). In addition, the learning device 10 can also repeat the processing of steps S201 to S203 until the end condition is satisfied.

[0109] In this way, the estimation unit 121 inputs the learning data into the lightweight model that outputs the estimation result based on the input data, and obtains the first estimation result. In addition, the update unit 123 updates the parameters of the lightweight model based on the first estimation result and the second estimation result obtained by inputting the learning data into the high-precision model, so as to optimize the model cascade including the lightweight model and the high-precision model. Here, the high-precision model is a model that outputs the estimation result based on the input data, and its processing speed is slower than that of the lightweight model, and its estimation accuracy is higher than that of the lightweight model. In this way, in the learning device 10, in the model cascade composed of the lightweight model and the high-precision model, by enabling the lightweight model to perform an estimation suitable for the model cascade, the performance of the model cascade can be improved. As a result, according to the learning device 10, not only can the accuracy of the model cascade be improved, but also the computational cost and the consumption of computational resources can be suppressed. Moreover, in the first embodiment, since the loss function is changed, there is no need to change the model architecture, and there are no restrictions on the model or optimization method to be applied.

[0110] The update unit 123 updates the parameters of the lightweight model to minimize the loss calculated based on the loss function, where the loss function includes: a first term that becomes larger as the confidence of the first estimation result relative to the correct answer becomes smaller; a second term that becomes larger as the confidence of the first estimation result becomes larger when the first estimation result is not the correct answer; a third term that becomes larger as the confidence of the first estimation result becomes smaller when the second estimation result is not the correct answer; and a fourth term that becomes larger as the confidence of the first estimation result becomes smaller. As a result, in the first embodiment, in the model cascade composed of the lightweight model and the high-precision model, the estimation accuracy of the model cascade can be improved based on the cost considering the estimation result using the high-precision model.

[0111] In the processing system 100, when performing inference using the learned high-precision model and lightweight model obtained by the learning device 10, the edge device 30 inputs the inference data into the lightweight model (DNN1) and obtains the confidence. When the confidence is above the threshold, the estimation result of the lightweight model is adopted. In addition, when the confidence is less than the threshold, the edge device 30 sends the processing data to the server device 20. Then, the processing system adopts the estimation result of the high-precision model obtained by inputting the inference data into the high-precision model (DNN2) of the server device 20.

[0112] In addition, in the first embodiment, an example of learning the DNN is described, but a machine learning mechanism other than the DNN can also be used.

[0113] [Embodiment 2]

[0114] Next, Embodiment 2 will be described. In Embodiment 2, the edge device encodes the processing data and then sends it to the server device.

[0115] Figure 10 FIG. is a diagram schematically showing an example of the structure of the processing system of Embodiment 2. The processing system 200 of Embodiment 2 has a server device 220 instead of Figure 4 the server device 20 shown, and has an edge device 230 instead of the edge device 30.

[0116] Compared with the edge device 30, the edge device 230 has an encoding unit 235. The encoding unit 235 encodes the data sent to the server device 220 by the communication unit 34. For example, the encoding unit 235 compresses the data to be sent to reduce the communication volume. In addition, when the output value of the intermediate layer of DNN1 is used as the data to be sent to the server device 220, even if the data is eavesdropped, the eavesdropper cannot interpret the meaning of the sent data, so the security can also be guaranteed.

[0117] Regarding the intermediate output value, among the multiple intermediate output values of DNN1 output in the process of outputting the inference result of the inference data, a value that is easier to encode than other intermediate output values is selected. Being easier to encode means having a smaller entropy or a higher sparsity compared to other intermediate output values. For example, the intermediate output value is the intermediate output value of the intermediate layer of the learned DNN1 that is learned in such a way that the entropy of the output value of the desired intermediate layer becomes smaller. The intermediate output value is the intermediate output value of the intermediate layer of the learned DNN1 that is learned in such a way that the sparsity of the output value of the desired intermediate layer becomes higher.

[0118] Compared with the server device 20, the server device 220 has a decoding unit 223. The decoding unit 223 decodes the processing data encoded by the encoding unit 235 and outputs it to the inference unit 22.

[0119] Here, when DNN1 and DNN2 are models obtained by dividing a DNN3 ( Figure 3 refer to) that is a single DNN trained using a specified standard into DNN1b and DNN2b, it is preferable to construct an encoding unit 235 that has less distortion of the inference result and is efficient.

[0120] For example, when learning all the data in the training set, each node in the intermediate layer to be transmitted can see the maximum value and the occurrence frequency of zero. Therefore, the encoding unit 235 is designed to perform corresponding encoding processing. The encoding processing can be a process of reducing the dimension of the representation space of the encoding object by estimating the influence of nodes with a high occurrence frequency of zero to be smaller, or a process of determining the value range of each node and thus selecting a method that reflects its tendency or determining the quantization granularity.

[0121] In addition, the encoding unit 235 can also perform encoding based on the method of vector quantization. In this case, the encoding unit 235 does not quantize the values of the nodes individually, but regards the values of all the nodes as vectors, clusters these values in the vector space, and performs encoding.

[0122] In addition, by finding the layer with a small entropy and splitting the DNN3 at this layer, the encoding unit 235 can obtain an intermediate output value with a small entropy.

[0123] In addition, the encoding unit 235 and the decoding unit 223 can adopt an encoding / decoding method based on known rules, or can also adopt a learning method based on AE (AutoEncoder) and VAE (Variational Auto Encoder).

[0124] Moreover, the encoding unit 235 can switch the encoding method for the processing data according to the intermediate output value and the destination DNN2 among multiple encoding methods. The decoding unit 223 decodes the data in a manner corresponding to the encoding method executed by the encoding unit 235.

[0125] [Processing Steps of the Processing System]

[0126] Figure 11 It is a timing diagram showing the processing flow of the processing system according to Embodiment 2. Figure 11 The steps S21 to S29 shown are the same processing as Figure 5 the steps S1 to S9 shown.

[0127] When the evaluation value does not satisfy the specified value (step S26: No), the encoding unit 235 encodes the processing data for causing the server device 220 to perform inference processing (step S30), and sends the encoded data to the server device 220 via the communication unit 34 (steps S31, S32). In the server device 220, the decoding unit 223 decodes the encoded data (step S33), and outputs the decoded processing data to the inference unit 22 (step S34). Steps S35 to S40 are Figure 5 the steps S11 to S16 shown.

[0128] [Effect of Embodiment 2]

[0129] In this way, in Embodiment 2, the edge device 230 encodes the processing data and then sends it to the server device 220. As a result, it is possible to send the processing data while maintaining security, to send the processing data in a data form with less distortion of the inference result, or to send the processing data efficiently.

[0130] In addition, in Embodiment 2, the structure in which the edge device 230 has an encoding unit 236 and the server device 220 has a decoding unit 223 is described, but it is not limited thereto. Figure 12 It is a diagram schematically showing another example of the structure of the processing system of Embodiment 2. As Figure 12 shown, an encoding unit 235 may be provided in the NW device 240A on the edge device 230A side between the edge device 230A and the server device 220A, and a decoding unit 223 may be provided in the NW device 250A on the server device 220A side.

[0131] In addition, in this Embodiment 2, there may be multiple edge devices 230 or server devices 220, and there may also be multiple edge devices 230 and server devices 220.

[0132] [Embodiment 3]

[0133] Next, Embodiment 3 will be described. Figure 13 It is a diagram schematically showing an example of the structure of the processing system of Embodiment 3. As Figure 13 shown, the processing system 300 of Embodiment 3 has the following structure: A plurality of edge devices 330-1, 330-2 are connected to one server device 320 via a network N. The number of edge devices is an example, and it may also be 3 or more. When collectively referring to the edge devices 330-1, 330-2, they are denoted as edge devices 330.

[0134] Figure 14 It is schematically showing Figure 13 an example of the edge device 330-1 shown in Figure 14 As shown, compared with the edge device 30, the edge device 330-1 has an additional unit 336. The additional unit 336 attaches a symbol for identifying this device to the processing data. The communication unit 34 sends the symbol for identifying this device together with the intermediate output value as the processing data to the server device 320.

[0135] In addition, the edge device 330-2 also has the same structure as the edge device 330-1. At this time, the DNN1 possessed by each edge device 330 may be the same model.

[0136] In addition, the DNN1 possessed by each edge device 330 may also be a model formed through multi-task learning. By reaching a consensus among the models, these models are the same up to a specified intermediate layer. Reaching a consensus among the models means, for example, reaching a consensus and performing learning among the intermediate layers of the same level in multiple models. That is, it can be said that two cost items are optimized simultaneously. One cost item is the cost item related to the problem set for itself when different learning data is given to each model, and the other cost item is the cost item for reaching a consensus among the intermediate layers of the same level in other models. As a result, the DNN1 possessed by each edge device 330 can be a model obtained by learning in such a way that the weights from the input layer to the specified intermediate layer are the same. For example, in the DNN1 possessed by each edge device 330, it is the same up to the feature quantity extraction layer of the sound signal, and the subsequent layers perform different processes respectively. In this case, the intermediate output values output by each edge device 330 are set as the output values from the same layer. Of course, the edge device 330 may also send the output values of different intermediate layers to the server device 320.

[0137] Figure 15 is schematically shown Figure 13 a diagram showing an example of the server device 320 shown. As Figure 15 shown, compared with the server device 20, the server device 320 has a storage unit 324 and an inference result database (DB) 325. The storage unit 324 stores the result (inference result) obtained by the inference unit 22 parsing the intermediate output value and the symbol for determining the edge device 330 that sent the processing data in the inference result DB 325 in a corresponding manner.

[0138] In the processing system 300, the processing performed in the edge device 330 and the processing performed in the server device 320 are optimized so that inference processing is performed on the data sent from any one of the multiple edge devices 330. For example, the DNN2 of the server device 320 is optimized so that it can process the processing data sent from any edge device 330.

[0139] [Processing Steps of the Processing System]

[0140] Figure 16 is a timing diagram showing the process flow of the processing system of Embodiment 3. Figure 16 The steps S41 to S49 shown are the same processing as Figure 5 the steps S1 to S9 shown.

[0141] When the evaluation value does not satisfy the specified value (step S46: No), the additional unit 336 attaches the symbol for identifying this device to the processing data (step S50). The communication unit 34 sends the symbol for identifying this device and the intermediate output value as the processing data to the server device 320 together (steps S51 and S52).

[0142] Figure 16 Steps S53 to S58 shown are Figure 5 Steps S11 to S16 shown. In the server device 320, the storage unit 324 stores the inference result in association with the symbol of the edge device 330 that sent the processing data in the inference result DB325 (steps S59 to S61).

[0143] [Effects of Embodiment 3]

[0144] In this way, in Embodiment 3, even when connected to multiple edge devices 330, the DNN2 of the server device 320 is optimized so that processing data sent from any of the edge devices 330 can be processed. Also, the edge device 330 sends the symbol for identifying this device and the intermediate output value as the processing data to the server device 320. Therefore, in the DNN2 of the server device 320, by identifying from which edge device 330 the processing data is sent, the inference process using the processing data can be appropriately executed.

[0145] Alternatively, the encoding unit 235 and the decoding unit 223 described in Embodiment 2 may be provided in the processing system 300.

[0146] [Embodiment 4]

[0147] Next, Embodiment 4 will be described. Figure 17 is a diagram schematically showing an example of the structure of the processing system of Embodiment 4. As Figure 14 shown, the processing system 300 of Embodiment 4 has the following structure: The edge device 430 is connected to a plurality of server devices 420-1 and 420-2 via the network N. The number of server devices is an example, and it may be three or more. When collectively referring to the server devices 420-1 and 420-2, they are referred to as the server device 420.

[0148] The DNNs 2 of the server devices 420 perform different tasks, respectively. For example, the DNN 2 of the server device 420-1 classifies the type of the object data (whether it is an image or an audio signal). The DNN 2 of the server device 420-2 classifies the nature of the object data (in the case where the task is object recognition, whether it is a person or a vehicle, etc.). In addition, the DNNs 2 of the other server devices 420 classify the processing content of the object data to be processed (whether it is an object recognition task or a sound source enhancement task). For example, when the DNN 1 of the edge device 430 performs feature extraction of data, the DNNs 2 of the respective server devices 420 are dedicated to the respective tasks provided by the respective server devices 420. In addition, in the case of performing different tasks, so-called multi-task learning can be used. Specifically, the layers up to a specified intermediate layer are arranged in the edge device 430, and the layers after the specified intermediate layer are arranged in the server device 420, where learning has been performed for task 1 and task 2 such that the weights from the input layer to the above-specified intermediate layer are the same. Thus, it can be configured such that, regardless of what task it is, the model arranged in the edge device 430 is the same model, and the models arranged in any server device can perform processing. In addition, different tasks may also be tasks with the same purpose but different estimation accuracies. For example, it can also be configured such that the estimation accuracy of the edge device 430 < the estimation accuracy of the server device 420-1 < the estimation accuracy of the server device 420-2.

[0149] Figure 18 is a diagram schematically showing Figure 17 an example of the edge device 330-1 shown in the figure. As Figure 18 shown, compared with the edge device 30, the edge device 430 has a selection unit 437. The selection unit 437 selects the server device 420 to which the processing data is to be sent from among the plurality of server devices 420 according to the purpose of processing the inference data.

[0150] [Processing steps of the processing system]

[0151] Figure 19 is a timing diagram showing the processing flow of the processing system of Embodiment 4. Figure 19 The steps S71 to S79 shown are the same processing as Figure 5 the steps S1 to S9 shown.

[0152] When the evaluation value does not satisfy the specified value (step S76: No), the selection unit 437 selects the server device 420 as the transmission destination according to the purpose and accuracy of the processing inference data (step S80). The communication unit 34 transmits the processing data to the server device 420 (for example, server device 420-1) selected by the selection unit 437 (steps S81, S82). Figure 19 The steps S83 to S88 shown are the same processing as Figure 5 the steps S11 to S16 shown. In addition, the selection unit 437 that selects the transmission destination (step S80) can be physically and / or logically configured on the edge device side, or can be configured on the server device side. In addition, the selection unit 437 can also be located on the network (a position where it is impossible to distinguish between the server side and the edge side).

[0153] [Effect of Embodiment 4]

[0154] In this way, in Embodiment 4, even when the edge device 430 is connected to a plurality of server devices 420, the server device 420 as the transmission destination can be selected according to the purpose of the processing inference data, so that the inference processing can be appropriately executed.

[0155] In addition, in this Embodiment 4, there may be a plurality of edge devices 430. In addition, in the processing system 400, the selection unit 437 may be provided in the NW device between the edge device and the server device. In addition, the encoding unit 235 and the decoding unit 223 described in Embodiment 2 may be provided in the processing system 400. At this time, the configuration location of the selection unit 237 may be before or after the encoding unit 235.

[0156] [Modification Example]

[0157] Next, modification examples of Embodiments 1 to 4 will be described. Figure 20 is a diagram showing an outline of a processing system in modification examples of Embodiments 1 to 4. Next, with reference to Figure 21 for Figure 20 the functions of the DNN1, DNN2, determination unit 33, encoding unit 235, decoding unit 223, and DNN2 shown, and the deformation of the communication content will be described.

[0158] Figure 21 is a diagram showing the deformation of the functions of the DNN1, DNN2, determination unit 33, encoding unit 235, and decoding unit 223 and the deformation of the communication content. In [[ID=In each of the columns shown, (1-A) to (1-H) represent variations in the functions of DNN1 and DNN2. In addition, (2-A) to (2-G) represent variations in the functions of the determination unit 33. In addition, (3-A) to (3-F) represent variations in the communication content between the encoding unit 235, the decoding unit 223, and the edge device and the server device. Each functional unit and communication content can be set according to the content shown in each column.

[0159] In addition, it is also possible to combine and use each functional unit and communication content. For example, in the case of using separately independent DNN1a and DNN2a (refer to ​ ), it is possible to combine the functions and communication content of (1-D), (2-C), and (3-A). In addition, in the case of using DNN1b and DNN2b obtained by dividing the integrated DNN3 (refer to ​ ), it is possible to combine any variation of (1-D) and (1-G), any variation of (2-C) and (2-D), and any variation of (3-B), (3-C), (3-D-1), (3-D-2), and (3-D-3).

[0160] The present invention can be applied to various situations where the expectations of users are diverse. Several specific examples are given.

[0161] [Autopilot]

[0162] The following example is given: An arithmetic device such as a DSP (Digital Signal Processor) arranged in an automobile is used as an edge device, and these arithmetic devices cooperate with the cloud. For example, it can be configured that in the server device, processing such as navigation considering traffic congestion, which is likely to increase both the amount of calculation and the amount of transmission but can have a slower response, is performed, while event detection related to the direct control of the automobile or determination of the control of the automobile corresponding to the detected event requires a certain degree of accuracy and response speed, so the processing is performed on the edge device side.

[0163] [Change Detection]

[0164] In the case of using a time-series image signal as an object, it is possible to detect in the edge device whether there is a change in the edge device compared to normal times or the previous frame, and estimate in the server device what kind of change has occurred.

[0165] The time-series image signal can be surveillance camera data, satellite images, or aerial photos. In the case of surveillance camera data, the situation where a person passes in front of the surveillance camera can be detected as a change in the edge device, and in the server device, it can be estimated what kind of person passes in front of the surveillance camera. In the case of satellite images, changes in the edges or textures of buildings, the passage of ships or cars, etc. can be detected as changes in the edge device, and in the server device, it can be estimated what the building condition or the constructed building is like, what the passing ship is like, etc. In this case, the computing device configured on the aircraft or satellite can also be regarded as the edge device.

[0166] [Security]

[0167] In the edge device, relatively simple / lightweight reasoning is performed (counting the number of people, estimating gender / age, making a rough determination of clothing, etc.), and more complex reasoning with a greater load (person identification, pose estimation, detection of suspicious persons, etc.) is performed in the cloud (server device).

[0168] In addition, in the edge device, the detection of known persons who require immediate responsiveness, such as VIPs (Virtual IPs), repeaters, complainants, etc., is performed, and the detection of more ordinary persons, the feature extraction of such persons, and database creation, etc., which can take time, are performed in the cloud.

[0169] [Agriculture]

[0170] For an unmanned controlled tractor, in the edge device (the tractor itself), it is confirmed whether there are no obstacles in front, and in the cloud, reasoning / planning is performed considering what the obstacles are and how to respond.

[0171] [Inference-based Vision]

[0172] Images from the camera are received at the site, image processing (usual two-layer reasoning) is performed, and the processing results are transferred to the cloud for more advanced processing or aggregation processing (multi-level reasoning). When the resources of a certain site A are exhausted and the resources of its adjacent site B are surplus, control is performed from the cloud to transfer a part of the processed data of site A to site B to perform the remaining processing. Thus, effective resource utilization (robustness of the service, effective utilization of resources) can be achieved. This means that the computing device configured on the site, etc., can be controlled as a so-called edge cloud.

[0173] [Control of Drone Camera Group]

[0174] On the cloud side, control / indicate the configuration of each UAV camera in the overall photography plan accompanied by multiple UAV camera groups, the recovery support between cameras corresponding to the situation, etc., and perform reasoning / judgment on the UAV (edge device) side related to coping with the situation inherent to each UAV camera (such as dodging when an obstacle suddenly appears in front of the camera). In this example, the 1-to-many Embodiment 3 can be applied.

[0175] In addition, an application example of the one-to-many Embodiment 4 will be described. At the edge (DNN1), feature quantization is performed on one camera image, and the feature quantity is transferred to multiple clouds in parallel for common utilization to perform various task processes (such as people counting, person identification, category classification, pose estimation, etc.). The above application example is a one-to-many case, and coding processing can be applied for privacy protection.

[0176] [System Structure, etc.]

[0177] Each structural element of each device shown in the figure is a functional conceptual structural element, and it is not necessarily required to be physically configured as shown in the figure. That is, the specific manner of dispersion or integration of each device is not limited to that shown in the figure, and all or part of the devices can be dispersed or integrated functionally or physically in any unit according to various loads, usage conditions, etc. And all or any part of the processing functions performed in each device are implemented by a CPU and a program analyzed and executed by the CPU, or can be implemented by hardware based on wiring logic.

[0178] In addition, in each process described in this embodiment, all or part of the process described as being automatically performed can also be performed manually, or all or part of the process described as being performed manually can be automatically performed using a known method. In addition, regarding the information including processing steps, control steps, specific names, various data or parameters shown in the above paragraphs or the drawings, it can be arbitrarily changed unless otherwise specifically described.

[0179] [Program]

[0180] ​ It is a diagram showing an example of a computer that realizes the edge devices 30, 230, 330, 430 and the server devices 20, 220, 320, 420 by executing a program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. In addition, the computer 1000 may also have the aforementioned accelerator to assist in the operation. In addition, the computer 1000 has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0181] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System), for example. The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. A detachable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100, for example. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0182] The hard disk drive 1090 stores an OS (Operating System) 1091, application programs 1092, program modules 1093, and program data 1094, for example. That is, programs defining the respective processes of the edge devices 30, 230, 330, 430 and the server devices 20, 220, 320, 420 are installed as program modules 1093 that describe code executable by a computer. The program modules 1093 are stored in the hard disk drive 1090, for example. For example, program modules 1093 for executing processes having the same functional structures as those in the edge devices 30, 230, 330, 430 and the server devices 20, 220, 320, 420 are stored in the hard disk drive 1090. In addition, the hard disk drive 1090 can be replaced with an SSD (Solid State Drive).

[0183] In addition, setting data used in the processes of the above-described embodiments is stored as program data 1094 in the memory 1010 or the hard disk drive 1090, for example. And, as needed, the CPU 1020 reads out the program modules 1093 and program data 1094 stored in the memory 1010 or the hard disk drive 1090 to the RAM 1012 and executes them.

[0184] In addition, the program modules 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090. For example, they can also be stored in a detachable storage medium and read out by the CPU 1020 via the disk drive 1100 or the like. Or, the program modules 1093 and program data 1094 can be stored in other computers connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). And, the program modules 1093 and program data 1094 can be read out by the CPU 1020 from other computers via the network interface 1070.

[0185] Above, embodiments applying the invention completed by the present inventor have been described. However, the present invention is not limited by the descriptions and drawings that form part of the disclosure of the present invention based on this embodiment. That is, other embodiments, examples, and application techniques obtained by those skilled in the art based on this embodiment are all included in the scope of the present invention.

[0186] Description of Reference Numerals

[0187] 10 Learning device

[0188] 11 High-precision model learning unit

[0189] 12 Lightweight model learning unit

[0190] 30, 230, 230A, 330, 430 Edge device

[0191] 20, 220, 220A, 320, 420 Server device

[0192] 100, 200, 300, 400 Processing system

[0193] 111, 121 Estimation unit

[0194] 112, 122 Loss calculation unit

[0195] 113, 123 Update unit

[0196] 114 High-precision model information

[0197] 124 Lightweight model information

[0198] 22, 32 Inference unit

[0199] 33 Judgment unit

[0200] 34 Communication unit

[0201] 223 Decoding unit

[0202] 235 Encoding unit

[0203] 240A, 250A NW device

[0204] 324 Storage unit

[0205] 325 Inference result database (DB)

[0206] 336 Attachment unit

[0207] 437 Selection unit

Claims

1. A processing system that uses an edge device and a server device, characterized in that: The edge device has: An edge processing unit that processes the data to be processed and outputs the processing result of the data to be processed; A determination unit that, when the evaluation value satisfies a condition, determines to cause the server device to execute processing related to the data to be processed, and when the processing result of the data to be processed satisfies a specified evaluation, determines that the evaluation value is a value within the range that should be processed by the edge device, and outputs the processing result of the data to be processed obtained by the edge processing unit. The evaluation value is a value for evaluating which of the edge device and the server device should process the data to be processed; and A sending unit that, when the determination unit determines to cause the server device to execute processing related to the data to be processed, sends data for causing the server device to execute processing related to the data to be processed, The evaluation value is a value that reflects either a reduction in data traffic or resource optimization of the edge device.

2. The processing system according to claim 1, characterized in that: The more difficult the processing of the data to be processed is, the more likely the evaluation value is to fall within the range that should be evaluated by the server device.

3. The processing system according to claim 1, characterized in that: The evaluation value is a value representing the confidence level of whether the result obtained by processing the data to be processed by the edge device is the correct answer.

4. The processing system according to claim 1, characterized in that: The evaluation value is determined based on any of the following: the time required to obtain the processing result of the data to be processed; the acquisition deadline of the processing result of the data to be processed; the usage status of the resources of the edge device when the determination unit makes a determination; whether the data to be processed is data that has occurred an event compared to other data.

5. A processing system that uses an edge device and a server device, characterized in that: The edge device has: An edge processing unit that processes the data to be processed and outputs the processing result of the data to be processed; A determination unit that, when the evaluation value satisfies a condition, determines to cause the server device to execute processing related to the data to be processed, and when the processing result of the data to be processed satisfies a specified evaluation, determines that the evaluation value is a value within the range that should be processed by the edge device, and outputs the processing result of the data to be processed obtained by the edge processing unit. The evaluation value is a value for evaluating which of the edge device and the server device should process the data to be processed; and A sending unit that, when the determination unit determines to cause the server device to perform processing related to the processing target data, sends data for causing the server device to perform processing related to the processing target data. The evaluation value is calculated based on an intermediate output value of a process for outputting a processing result of the processing target data by the edge processing unit. The sending unit sends the intermediate output value to the server device.

6. The processing system according to claim 5, wherein: The processing system further includes an encoding unit that encodes data to be sent by the sending unit to the server device. The intermediate output value is selected from among a plurality of intermediate output values output in a process of outputting a processing result of the processing target data, and is a value that is easier to encode than other intermediate output values.

7. The processing system according to claim 5, wherein: There are a plurality of the edge devices. The processing performed by the edge devices and the processing performed by the server device are optimized such that the server device performs processing related to the processing target data regardless of data sent from any one of the plurality of edge devices.

8. The processing system according to any one of claims 5 to 7, wherein: The process of outputting a processing result of the processing target data is inference based on a learned neural network. The intermediate output value is an output value of an intermediate layer of the learned neural network.

9. The processing system according to any one of claims 5 to 7, wherein: There are a plurality of the server devices. The edge device further includes a selection unit that selects, according to the purpose of processing the processing target data, a server device to which data for causing the server device to perform processing related to the processing target data is to be sent from among the plurality of server devices.

10. The processing system according to claim 7, wherein: The intermediate output value is irreversible with respect to the processing target data. The edge device sends the intermediate output value together with a symbol for identifying the device itself. The server device stores, in association with each other, a result obtained by analyzing the intermediate output value and the symbol for identifying the edge device.

11. A processing method, which is executed by a processing system using an edge device and a server device, characterized in that, The processing method includes: A processing step in which the edge device processes processing target data and outputs a processing result of the processing target data. A determination step in which the edge device determines to cause the server device to perform processing related to the processing target data when the evaluation value satisfies a condition, and determines that the evaluation value is a value within a range that should be processed by the edge device when the processing result of the processing target data satisfies a specified evaluation, and outputs the processing result of the processing target data obtained in the processing step, where the evaluation value is a value for evaluating which of the edge device and the server device should process the processing target data. Transmission step: When it is determined in the determination step that the server device is to execute processing related to the data to be processed, the edge device transmits data for causing the server device to execute processing related to the data to be processed. The evaluation value is a value that reflects either a reduction in data traffic or optimization of the resources of the edge device.

Images