Processing system, processing method, and recording medium

Abstract

The processing system (100) is a processing system that uses an edge device (30) and a server device (20) for processing. The edge device (30) has: an inference unit (31) that uses a first model to extract feature quantities of the processing object data and performs inference processing on the processing object data based on the extracted feature quantities; and a determination unit (32) that outputs the inference result of the inference unit (31) when the confidence level of the inference result exceeds a threshold, and outputs the feature quantities of the processing object data to the server device (20) when the confidence level is below the threshold. The server device (20) has an inference unit (21) that uses a second model with higher inference accuracy than the first model and performs inference processing on the processing object data based on the feature quantities of the processing object data output from the edge device (30).

Description

Technical Field

[0001] The present invention relates to a processing system, a processing method, and a recording medium. Background Art

[0002] The amount of data collected by IoT devices representing sensors is huge. Therefore, when aggregating and processing the data collected through cloud computing, a huge amount of communication traffic is generated. Thus, even for edge devices close to users, edge computing for processing the collected data has attracted attention.

[0003] However, the computing power, memory, and other resources of the devices used in edge devices are inferior to those of devices other than edge devices (hereinafter, simply referred to as cloud) that are physically and logically configured farther from users relative to edge devices. Therefore, when performing a processing with a large computing load in an edge device, it sometimes takes a long time until the processing is completed, or it also takes time until other processing with a small computing amount is completed. For example, a situation may occur where other processing is pending during the processing with a large computing amount in the edge device.

[0004] Here, as one of the processes with a large computing amount, a process related to machine learning can be cited. The application of a so-called adaptive cloud was proposed in Non-Patent Document 1. That is, in the method described in Non-Patent Document 1, a learned model that has been learned using general learning data in the cloud is loaded into the edge device, and the model learned in the cloud is re-learned using the data obtained in the edge device, thereby enabling effective utilization of the advantages of the cloud and the edge device.

[0005] Prior Art Documents

[0006] Non-Patent Documents

[0007] Non-Patent Document 1: "Otsuki et al., 'Proposal and Evaluation of DNN Model Operation Method by Cloud-Edge Collaboration', Proceedings of the 80th National Conference Lecture Collection 2018(1), 3-4, 2018-03-13." Summary of the Invention

[0008] Problems to be Solved by the Invention

[0009] The above-mentioned edge-cloud network is expected to be applied to applications such as automatic analysis of surveillance camera images, autonomous driving, and smart speakers. In these applications, both accuracy and real-time performance are important, but the communication cost and latency between the edge and the cloud become problems.

[0010] The present invention was made in view of the above circumstances, and its object is to provide a processing system, processing method and processing program that can reduce the amount of data transmitted from the edge device to the server device and reduce latency.

[0011] Methods for solving problems

[0012] To address the aforementioned issues and achieve the objective, the processing system of the present invention utilizes an edge device and a server device for processing. The edge device comprises: a first inference unit that extracts feature quantities of the object data to be processed using a first model and performs inference processing on the object data based on the extracted feature quantities; and a determination unit that outputs the inference result of the first inference unit if the confidence level of the inference result exceeds a threshold, and outputs the feature quantities of the object data to the server device if the confidence level is below the threshold. The server device has a second inference unit that uses a second model with higher inference accuracy than the first model to perform inference processing on the object data based on the feature quantities of the object data output from the edge device.

[0013] Invention Effects

[0014] According to the present invention, it is possible to reduce the amount of data transmitted from the edge device to the server device and to reduce latency. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating the outline of the processing method of the processing system of the implementation method.

[0016] Figure 2-1 This is a diagram illustrating an example of DNN1 and DNN2.

[0017] Figure 2-2 This is a diagram illustrating an example of DNN1 and DNN2.

[0018] Figure 3 This is a diagram that schematically illustrates an example of the structure of a processing system according to an implementation method.

[0019] Figure 4 This is a diagram illustrating a selected example of the basic model of DNN.

[0020] Figure 5 This is a diagram showing a summary of the structure of YOLOv3.

[0021] Figure 6-1 This is a diagram illustrating an example of the structure of a DNN.

[0022] Figure 6-2 This is a diagram illustrating an example of the structure of a DNN.

[0023] Figure 7 This is a sequence diagram illustrating the processing flow of the processing system of an embodiment.

[0024] Figure 8 This is a graph showing the processing time of each selected DNN in the processing system as an example.

[0025] Figure 9 This is a distribution diagram of the entropy of the test data obtained based on the inference results of DNN.

[0026] Figure 10 This is a graph showing the relationship between the split rate and the overall accuracy.

[0027] Figure 11 This is a diagram illustrating an example of a computer that implements edge devices and server devices by executing programs. Detailed Implementation

[0028] Hereinafter, one embodiment of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to this embodiment. Furthermore, in the accompanying drawings, the same reference numerals are used to indicate the same parts.

[0029] [Implementation Method]

[0030] [Summary of Implementation Methods]

[0031] Embodiments of the present invention will be described. In these embodiments, a processing system for inference processing using a learned high-precision model and a lightweight model will be described. Furthermore, in the processing system of the embodiments, the case where a DNN (Deep Neural Network) is used as the model in the inference processing will be described as an example. In the processing system of the embodiments, any neural network can be used, and low-computational-load signal processing and high-computational-load signal processing can be used instead of a learned model.

[0032] Figure 1 This is a diagram illustrating the outline of the processing method of the processing system in this embodiment. In the processing system of this embodiment, a high-precision model and a lightweight model constitute a model cascade. In the processing system of this embodiment, reliability is used to control whether processing is performed on an edge device using a high-speed, low-precision lightweight model (e.g., DNN1 (first model)) or on a cloud (server device) using a low-speed, high-precision high-precision model (e.g., DNN2 (second model)). For example, a server device is a device configured physically and logically far from the user. Edge devices are IoT devices and various terminal devices configured physically and logically close to the user, and have fewer resources compared to server devices.

[0033] DNN1 and DNN2 are models that process input data and output inference results. Figure 1 In the example, DNN1 takes an image as input and, based on the features extracted from the image, infers the probability that the objects in the image belong to each category. DNN2 takes the features extracted by DNN1 (intended to output the intermediate layer, hereinafter referred to as the feature map) as input and infers the probability that the objects in the image belong to each category. Furthermore, both DNN1 and DNN2 perform inference on the same image. In the processing system, at the edge device, a comparison between confidence (described later) and a threshold determines which inference result from DNN1 or DNN2 should be used. Confidence is a value used to determine whether the data to be processed should be processed in the edge device or the server device. Confidence can also be calculated, for example, based on the entropy of the inference result from DNN1. Confidence can simply be set to a value that decreases as the entropy of the inference result from DNN1 increases. Specific examples are described later.

[0034] like Figure 1 As shown, in the processing system, the confidence level of DNN1's inference for classifying objects in the input image is obtained. Then, in the processing system, if the obtained confidence level exceeds a predetermined threshold, the inference result of DNN1 is adopted. That is, the inference result of the lightweight model is output as the final inference result of the model cascade. On the other hand, in the processing system, if the confidence level is below the predetermined threshold, the inference result obtained by inputting the feature map extracted by DNN1 into DNN2 is output as the final inference result.

[0035] Thus, in the processing system of this embodiment, the selection of whether to process the object data on the edge device or the server device is based on a comparison of confidence level and threshold. Therefore, the processing system of this embodiment can control which device, the edge device or the cloud, performs the processing.

[0036] [Lightweight Model and High-Precision Model]

[0037] Next, DNN1 and DNN2 will be explained. Figure 2-1 and Figure 2-2 This diagram illustrates an example of DNN1 and DNN2. A DNN has an input layer that takes data as input, multiple intermediate layers that perform various transformations on the data from the input layer, and an output layer that outputs the results of inference, such as probabilities and likelihoods. When the input data needs to be anonymized, the output values ​​from each layer can be set to be irreversible.

[0038] As described above, the processing system uses DNN2 on the cloud side and DNN1 on the edge side. As DNN2′ before being configured on the cloud side, a high-precision model is adopted, which has a feature extraction layer Bf2 that extracts features from the input image and outputs them as feature maps, and a layer that processes the extracted features (e.g., a layer that detects objects reflected in the input image or infers the probability that objects reflected in the input image belong to various categories (hereinafter collectively referred to as the detection layer Bd2)) (Second Execution Unit) (Ref.) Figure 2-1 In the processing system, DNN2′ is trained in advance using training data before configuration.

[0039] Furthermore, in the processing system, before configuring to the edge side, the feature extraction layer Bf1 of the trained DNN2′ can be directly configured before the lightweight detection layer Bd1 on the edge side, replacing the feature extraction layer Bf1 of the DNN1′ before edge side configuration (see [reference]). Figure 2-1 Arrow Y1), thus setting the DNN composed of feature extraction layer Bf2 (first extraction unit) and detection layer Bd1 (first execution unit) as the edge-side DNN1 (refer to). Figure 2-2 The detection layer Bd1 uses the feature maps (feature values ​​of a certain layer) extracted by the feature extraction layer Bf2 of DNN1 to infer the probability that the objects reflected in the input image belong to each category. Here, the structures of DNN1 configured on the edge side and DNN2 configured on the cloud side are described. For DNN1 configured on the edge side, the feature extraction layer Bf1 is removed from the original DNN1 (see...). Figure 2-1 And configure the trained DNN2′ feature extraction layer Bf2 to replace the feature extraction layer Bf1 (refer to...). Figure 2-2 That is, DNN1, which is composed of the feature extraction layer Bf2 of DNN2′ and the detection layer Bd1 of DNN1′, is configured on the edge side. Alternatively, it can also be... Figure 2-2 The feature extraction layer Bf2 in the DNN1 shown is fixed and learned again. Additionally, the DNN2 configured on the server side becomes the original DNN2' (refer to...). Figure 2-1 The structure of the feature extraction layer Bf2 was deleted (see reference). Figure 2-2 ).

[0040] For DNN1, the parameters of the feature extraction layer Bf2 are fixed as the parameters of the original DNN2′ after training, and the subsequent detection layer Bd1 is trained using the training data. Alternatively, for DNN1, both the feature extraction layer Bf2 and the detection layer Bd1 can be trained using the training data. Furthermore, besides DNN2 and DNN1 learning independently, they can also learn collaboratively. For example, DNN2 can be retrained using the training data used by DNN1. Alternatively, training can be performed using training data shared by both DNN1 and DNN2.

[0041] In the processing system of this embodiment, the feature extraction layer Bf2 in the initial DNN2′ is configured as the feature extraction layer Bf2 of the edge-side DNN1. Therefore, the detection layer Bd2 of the cloud-side DNN2 can perform inference processing using the feature map output by the feature extraction layer Bf2 of the edge-side DNN1 (see reference). Figure 2-2 (arrow Y2). Therefore, in the processing system of this embodiment, it can be said that the same feature map can be shared between the edge and the cloud, while inference processing is performed on the edge side and the cloud side respectively.

[0042] Therefore, when inference is performed on the cloud side, the feature extraction process can be omitted, thus reducing computation time and achieving low latency. Furthermore, the data output from the edge side to the cloud side is not the image itself, but rather a feature map extracted from the image, thus reducing the amount of data transmitted from the edge side to the cloud side. Moreover, since the same feature maps can be shared between the edge and the cloud, the minimum condition that the feature extraction layer Bf2 of DNN1 actually configured on the edge side should satisfy is that the connected layers have the same size. This is for the purpose of relearning the detection layer of DNN1. Furthermore, the parameter values ​​in the feature extraction layers Bf2 of DNN1 and DNN2' can be different.

[0043] [Processing System]

[0044] Next, the structure of the processing system will be explained. Figure 3 This is a diagram that schematically illustrates an example of the structure of a processing system according to an implementation method.

[0045] The processing system 100 of this embodiment includes a server device 20 and an edge device 30. Furthermore, the server device 20 and the edge device 30 are connected via a network N. The network N is, for example, the Internet. For example, the server device 20 is a server located in a cloud environment. Additionally, the edge device 30 is, for example, an IoT device and various terminal devices.

[0046] Server device 20 and edge device 30 respectively read a predetermined program from a computer, including ROM (Read Only Memory), RAM (Random Access Memory), and CPU (Central Processing Unit), and the CPU executes the predetermined program. Additionally, so-called accelerators, such as GPUs, VPUs (Vision Processing Units), FPGAs (Field Programmable Gate Arrays), ASICs (Application Specific Integrated Circuits), and dedicated AI (Artificial Intelligence) chips, are used. Server device 20 and edge device 30 each have NICs (Network Interface Cards), and can communicate with other devices via electrical communication lines such as LANs (Local Area Networks) and the Internet.

[0047] like Figure 3 As shown, the server device 20 has an inference unit 21 (second inference unit) that performs inference using a learned high-precision model, namely DNN2. DNN2 contains information such as model parameters. As described above, DNN2 has a detection layer Bd2.

[0048] The inference unit 21 uses DNN2 to perform inference processing on the input image based on the feature map of the input image output from the edge device 30. The inference unit 21 returns the quantized feature map output from the edge device 30 to FP32 as input to the detection layer Bd2 of DNN2. The inference unit 21 obtains the inference result (e.g., the probability that an object in the image belongs to each category) as the output of DNN2. The inference unit 21 accepts the inference data, i.e., the output value of the feature extraction layer Bf2 of DNN1, and outputs the inference result. The feature map is the feature quantity of data with unknown labels. For example, the inference data is an image. Alternatively, when returning the inference result to the user, the inference result obtained by the inference unit 21 can be transmitted to the edge device 30 and returned to the user from the edge device 30.

[0049] Here, the server device 20 and the edge device 30 form a model cascade. Therefore, the inference unit 21 is not always performing inference. The inference unit 21 accepts the input of the quantized feature map when it is determined that the server device 20 should perform inference processing, and performs inference through the detection layer Bd2 of the DNN2.

[0050] The edge device 30 includes: an inference unit 31 (first inference unit) having a learned lightweight model, namely DNN1; a decision unit 32; and a quantization unit 33.

[0051] The inference unit 31 uses the trained lightweight model, DNN1, for inference. DNN1 contains information such as model parameters. As mentioned above, DNN1 has a trained DNN2′ (see reference). Figure 2-1 The DNN 1 extracts features from the input image into a feature map and performs inference processing on the input image based on the extracted feature map.

[0052] The inference unit 31 inputs an image as the processing object into the DNN1 and obtains an inference result. The inference unit 31 uses the DNN1 to extract feature quantities of the processing object data and performs inference processing on the processing object data based on the extracted feature quantities. The inference unit 31 accepts the input of an image as the processing object, processes the image as the processing object, and outputs an inference result (e.g., the probability that the object reflected in the image belongs to each category).

[0053] The decision unit 32 determines which inference result to use, the edge device 30 or the server device 20, by comparing the confidence level with a specified threshold.

[0054] If the confidence level exceeds a predetermined threshold, the determination unit 32 outputs the reasoning result inferred by the reasoning unit 31. If the confidence level is below the predetermined threshold, the determination unit 32 outputs the output of the feature extraction layer Bf2 connected to the edge-side DNN1, i.e., the feature map, to the server device 20, and determines that the reasoning process is performed by the DNN2 configured in the server device 20.

[0055] As mentioned above, the confidence level can be set to a value that decreases as the entropy of the inference result of DNN1 increases. The entropy of the inference result of DNN1 can be obtained from equation (1). C is the set of all labels c output by DNN1, and y is the probability of each label. The confidence level is calculated based on the entropy obtained in this way, for example, confidence level = 1 / entropy.

[0056] [Mathematical Expression 1]

[0057]

[0058] When the determination unit 32 determines that the server device 20 is performing inference processing, the quantization unit 33 quantizes the feature map extracted by the feature extraction layer Bf2 of the DNN1 and outputs the quantized feature map to the server device 20.

[0059] [Example of model selection for DNN1 and DNN2]

[0060] Figure 4 This diagram illustrates selected examples of the basic models for DNN1 and DNN2. In this embodiment, the model representing the probability of an object belonging to each category is as follows: Figure 4 As shown, the relatively lightweight and high-speed YOLOv2 backend model, Darknet 19 (hereinafter referred to as YOLOv2), is selected as the base model for the edge device 30, and the higher-precision YOLOv3 backend model, Darknet 53 (hereinafter referred to as YOLOv3), is selected as the base model for the server device 20. The selected neural network (NN) is just an example; any NN that is both a high-precision and a high-speed NN, but with lower precision than the high-precision NN, and that can share feature extraction layers, can be arbitrarily chosen. In a simple example, the same NN can also be configured with different depths in the edge device 30 and the server device 20.

[0061] Figure 5 This is a diagram illustrating the general structure of YOLOv3. It also shows selected examples of the base models of DNN1 and DNN2. YOLOv3 has: a convolutional layer (feature extraction layer) Bf-1, which performs feature extraction with a residual block Bf-1; and a detection unit Bd2-1, which serves as the object detection network (FPN). Here, the following focuses on sharing feature maps between edges and clouds.

[0062] First, high accuracy is maintained in YOLOv3 by incorporating the residual block Bf-1. Therefore, it is desirable to avoid compromising the structure within the residual block Bf-1. Next, in YOLOv3, it is preferable to receive feature maps before the detection unit Bd2-1, enabling the detection unit Bd2-1 to directly use the feature maps for detection. Furthermore, the deeper (more) the layers, the slower the model computation and the more parameters are used. Therefore, to achieve lightweight and high speed, it is preferable that the DNN1 of the edge device 30 is a shallow model.

[0063] [An example of the structure of DNN1 and DNN2]

[0064] Figure 6-1 and Figure 6-2 This is a diagram illustrating an example of the structure of DNN1 and DNN2. To avoid compromising the structure within the YOLOv3 Residual block Bf-1, the feature extraction layer Bf2-1, which is the result of learning the Residual block Bf-1, is directly placed in the preceding stage of the YOLOv2 model selected as the base model for the edge device 30. Figure 6-1Arrow Y11) completes the DNN1 of the edge device 30. Therefore, in the edge device 30, using YOLOv2 reconstructed in this way, feature maps are extracted from the image in the feature extraction layer Bf2-1, and the extracted feature maps are used to infer the probability that the objects reflected in the input image belong to each category in the detection layer Bd1-1.

[0065] Then, in the edge device 30, if the confidence level of the result (Result1) of the detection layer Bd1-1 exceeds a predetermined threshold, Result1 is considered reliable, and Result1 is output (refer to...). Figure 6-2 (Arrow Y11) indicates the end of processing. Here, as described above, the feature extraction layer Bf2-1 in the edge device 30 has a common structure with the feature extraction layer Bf2-1 of YOLOv3, which serves as the base model of the server device 20. Therefore, the feature map output from the feature extraction layer Bf2-1 can also be shared in the detection unit Bd2-1 of the server device 20. Furthermore, in the server device 20, the YOLOv3 structure after removing the feature extraction layer Bf2-1 is applied as DNN2.

[0066] Therefore, in edge device 30, if Result1 is below the threshold, Result1 is considered unreliable, and the feature map of feature extraction layer Bf2-1 is quantized and output to edge device 30 (see reference). Figure 6-2 (arrow Y12). Then, in server device 20, the feature map is returned to FP32, the feature map is input to the detection unit Bd2-1 of the reconstructed YOLOv3, and the inference result (Result2) of the detection unit Bd2-1 is output (see reference). Figure 6-2 (arrow Y13). Therefore, the computational range in the server device 20 is the same as the computational range of the detection unit Bd2-1. In other words, in the server device 20, the computation performed in the feature extraction layer Bf2-1 can be omitted.

[0067] [Processing steps of the processing system]

[0068] Figure 7 This is a sequence diagram illustrating the processing flow of the processing system in the implementation method. For example... Figure 7 As shown, firstly, in the edge device 30, when the inference unit 31 receives an image input (step S1), it inputs the input image to the DNN1. In the DNN1, the feature extraction layer Bf2 extracts the feature values ​​of the input image as a feature map (step S2) and outputs it to the decision unit 32 (step S3). In the DNN1, the detection layer Bd1 performs inference processing, such as detection processing, on the input image based on the feature map (step S4) and outputs it to the decision unit 32 (step S5).

[0069] The determination unit 32 calculates the credibility based on the reasoning result of the reasoning unit 31 (step S6), compares the calculated credibility with a predetermined threshold, and determines whether the credibility is below the predetermined threshold (step S7).

[0070] If the confidence level is not below the threshold (step S7: No), that is, if the confidence level exceeds the threshold, the determination unit 32 outputs the reasoning result inferred by the DNN1 of the reasoning unit 31 (step S8).

[0071] On the other hand, if the confidence level is below the threshold (step S7: Yes), the determination unit 32 outputs the feature map to the quantization unit 33 (step S9), the quantization unit 33 quantizes the feature map (step S10), and sends it to the server device 20 (step S11).

[0072] In server device 20, inference unit 21 returns the quantized feature map output from edge device 30 to FP32 as input to detection layer Bd2 of DNN2. Detection layer Bd2 performs inference processing, such as detection processing, on the input image based on the feature map output from edge device 30 (step S12). Server device 20 sends the inference result of DNN2 to edge device 30 (step S13) and outputs it from edge device 30 (step S14). In this embodiment, the structure for returning the inference result to the user is envisioned as a structure where the final inference result is output from edge device 30. However, when using the final inference result on server device 20, it is also possible to configure the server device 20 to output the inference result of DNN2, or to directly store the inference result of DNN2 in server device 20. When using the inference result of DNN1, if the inference result is used on server device 20, edge device 30 can simply send the inference result to server device 20.

[0073] [Evaluation Experiment 1]

[0074] The processing time of DNN1 on the edge device 30 and the processing time of DNN2 on the server device were evaluated. The task was classification, and the test data was ImageNet (300 images per class, totaling 3000 images). The platform was NVIDIA GeForce RTX 2070 + AMD 3600. Figure 8 This is a graph showing the processing time of each DNN selected as an example in the processing system 100.

[0075] exist Figure 8In the diagram, the edge device's DNN1 is a model trained by configuring the learned feature extraction layer Bf2-1, which stores the residual block Bf-1, in front of YOLOv2. The server device's DNN2 is a learned YOLOv3. For comparison, the processing time is also shown for the base model of the edge device 30, namely YOLOv2.

[0076] like Figure 8 As shown, the edge device's DNN1 deepens the feature extraction layer Bf2-1 compared to YOLOv2, thus increasing the processing time. However, it shows an improvement in inference accuracy. Furthermore, the edge device's DNN1 processes each image approximately twice as fast as the server device's DNN2 (YOLOv3). Thus, even when the YOLOv3 feature extraction layer Bf2-1 is used instead of the YOLOv2 feature extraction layer, the inference accuracy of the edge device's DNN1 can be improved while maintaining high processing speed.

[0077] [Evaluation Experiment 2]

[0078] Using DNN1 with edge device 30, the distribution of entropy in the test data is visualized. The task, test data, and HW are related to... Figure 8 Under the same conditions. Figure 9 This is a distribution diagram of the entropy of the test data obtained based on the inference results of DNN1.

[0079] The higher the entropy, the less reliable the inference results of DNN1. Therefore, it can be said that using reliability as an evaluation value for determining whether processing is performed on the edge device or the server device in the model cascade is appropriate. Furthermore, since the entropy of most data is below 0.5, it is sufficient to set 0.5 as the baseline for the entropy threshold used to determine whether processing is performed on the edge device or the server device.

[0080] [Evaluation Experiment 3]

[0081] Therefore, evaluation experiment 3 was conducted, in which the change in the overall accuracy of the inference result along with the change in the threshold (splitting rate) of entropy was determined. Furthermore, the threshold is linked to the splitting rate; the threshold is increased when the splitting rate is decreased. In this evaluation experiment, the feature maps extracted by the feature extraction layer Bf2 of the edge device 30 were quantized in int4, int6, and int8. Additionally, the task, test data, and HW are related to... Figure 8 Under the same conditions. Figure 10 This is a graph showing the relationship between the split rate and overall accuracy. In Figure 10In the diagram, "Offload rate 0" represents a low original precision (acc_origin) when all data is processed by the edge device 30 without quantization, and "Offload rate 1" represents a high original precision (acc_origin) when all data is processed by the server device 20 without quantization. The feature map of the feature extraction layer Bf2 based on the edge device 30 is quantized using int4, int6, and int8 methods.

[0082] Among them, such as Figure 10 As shown, when the feature map of the feature extraction layer Bf2 based on edge device 30 is quantized using int4, int6, and int8 quantization, the accuracy is almost not reduced, while the amount of transmitted data is reduced by 75% compared to the case without quantization. Therefore, it is desirable to apply int8 quantization to the quantization of feature maps.

[0083] Additionally, a cost term can be set to make the feature map output by the feature extraction layer Bf2 more sparse. As a result, when int6 and / or int4 show... Figure 10 When the shunt rate and / or overall precision of int8 are close, you can choose the smaller of int6 or int4, which indicates the shunt rate and / or overall precision that are close to int8.

[0084] Alternatively, the quantized feature map can be compressed. When compression is required, both the feature map and the quantized feature map possess the same properties as the natural image contained in the object image; therefore, image encoding methods such as HEVC and VVC, or general compression methods such as ZIP, can be used. Furthermore, the properties similar to those of a natural image mean that there are more cases where adjacent pixels have high correlation—a general property used in image encoding. Additionally, inverse quantization can be performed when quantization is performed, and decoding can be performed when compression is performed. Quantization and / or compression can be performed on an edge device, or on other devices physically or logically located closer to the edge device (compared to the server device). Inverse quantization and / or decoding can be performed on the server device, or on other devices physically or logically located closer to the server device (compared to the edge device).

[0085] Furthermore, when the split rate exceeds 0.4 (with an entropy threshold of 0.5), even increasing the split rate (i.e., lowering the entropy threshold) will result in less improvement in accuracy. Therefore, setting the threshold to 0.5 is considered to achieve a balance between split rate and accuracy. In this way, by setting the threshold based on the balance between split rate and accuracy, adjustments to the split rate and overall accuracy corresponding to each use case can be made.

[0086] Here, we further explain the rationale for determining the entropy threshold. Lowering the entropy threshold means a higher splitting rate. That is, it can also be said that by lowering the entropy threshold, the amount of data for inference on the server device side will increase.

[0087] Here, we consider the threshold used to determine whether inference should be performed on the edge device side or the server device side. As an example, consider overall accuracy as a benchmark. Ideally, this threshold should be such that if the accuracy remains largely unchanged regardless of whether inference is performed on the edge device side or the server device side, then processing can be performed on the edge device side; if accuracy decreases when processing on the edge device side, then processing can be performed on the server device side. For example, in... Figure 10 In this case, it can be seen that when the quantization bit rate is set to 8 (acc_int8), the threshold for the shunt rate can be set to 0.4. Then, the threshold for the entropy of the inference result on the edge side can be determined by setting the shunt rate to 0.4. Figure 10 In this case, an entropy threshold of around 0.5 is sufficient.

[0088] [Effects of the Implementation Method]

[0089] Thus, in this implementation, by configuring the feature extraction layer Bf2 in the DNN2′ before configuration to the server device 20 as the feature extraction layer Bf2 of the DNN1 of the edge device 30, the feature map output by the edge device 30 can also be shared in the server device 20. That is, the detection layer Bd2 of the DNN2 of the server device 20 can use the feature map output by the feature extraction layer Bf2 of the DNN1 of the edge device 30 to perform inference processing.

[0090] Therefore, when inference is performed on the server device 20, the feature extraction process can be omitted, thus shortening the overall system computation time and achieving low latency. Furthermore, the output data from the edge device 30 to the server device 20 is not the image being processed, but a feature map extracted from the image, thus reducing the amount of data transmitted from the edge device 30 to the server device 20. Additionally, since the edge device sends a feature map to the server device that is neither the image being processed nor the encoded image, nor a commonly used feature quantity like a frequency signal, confidentiality against third parties can be improved. To reliably ensure confidentiality, during learning, a constraint can be further imposed that makes the relationship between the feature map and the input data (the image being processed) irreversible.

[0091] In this implementation, a DNN1 is proposed that configures the feature extraction layer Bf2 in DNN2′ before it is configured to the server device 20, and places it before the detection layer Bd1 in the edge device 30. Evaluations show that this DNN1 performs well in classification tasks, combining accuracy and speed, and is effective as a lightweight model.

[0092] Furthermore, in this embodiment, the confidence level used to determine which of the edge device 30 and server device 20 to use for the inference result is formulated as the entropy of the inference result of DNN1, and a threshold is set, thereby enabling the adjustment of the split rate and overall accuracy corresponding to the actual use case.

[0093] Furthermore, in this embodiment, images are used as an example of the data to be processed, but it is not limited to images; it can also be detection results detected by various sensors. Additionally, in this embodiment, YOLOv2 and YOLOv3 are used as examples of the base models for DNN1 and DNN2, but the base models for DNN1 and DNN2 can be appropriately set according to the task.

[0094] In addition, in this embodiment, there may be multiple edge devices 30 or server devices 20, and there may also be multiple edge devices 30 and multiple server devices 20.

[0095] [System architecture, etc.]

[0096] The constituent elements of the devices illustrated are functional conceptual elements and do not necessarily need to be physically configured as shown in the illustration. That is, the specific methods of distributing or combining the devices are not limited to those shown in the illustration. They can be configured by distributing or combining all or part of them in any functional or physical manner, depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be implemented by a CPU and the program parsed and executed by that CPU, or can be implemented as hardware based on wiring logic.

[0097] Furthermore, in each of the processes described in this embodiment, all or part of the processes described as automatically performed may be performed manually, or all or part of the processes described as manually performed may be performed automatically using known methods. In addition, the processing steps, control steps, specific names, and information containing various data or parameters shown in the above text or figures can be arbitrarily changed, unless specifically stated otherwise.

[0098] [program]

[0099] Figure 11This diagram illustrates an example of a computer that implements the edge device 30 and the server device 20 by executing programs. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. Furthermore, it may include the aforementioned accelerator to assist in computation. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected via a bus 1080.

[0100] The memory 1010 includes a ROM (Read-Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). A hard disk drive interface 1030 is connected to a hard disk drive 1090. A disk drive interface 1040 is connected to a disk drive 1100. For example, a removable storage medium such as a hard disk or optical disk is inserted into the disk drive 1100. A serial port interface 1050 is connected, for example, to a mouse 1110 and a keyboard 1120. A video adapter 1060 is connected, for example, to a monitor 1130.

[0101] The hard disk drive 1090 stores, for example, an operating system (OS) 1091, application programs 1092, program modules 1093, and program data 1094. That is, programs defining the processing of the edge device 30 and the server device 20 are installed as program modules 1093, which contain code executable by a computer. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, program modules 1093 for performing processing with the same functional structure as those in the edge device 30 and the server device 20 are stored in the hard disk drive 1090. Alternatively, the hard disk drive 1090 can be replaced by an SSD (Solid State Drive).

[0102] Furthermore, the setting data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. Moreover, CPU 1020 reads program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 and executes them as needed.

[0103] Furthermore, program module 1093 and program data 1094 are not limited to being stored in hard disk drive 1090; for example, they can also be stored in removable storage media and read by CPU 1020 via disk drive 1100, etc. Alternatively, program module 1093 and program data 1094 can also be stored in other computers connected via a network (LAN, WAN, etc.). Moreover, program module 1093 and program data 1094 can also be read by CPU 1020 from other computers via network interface 1070.

[0104] The embodiments of the invention made by the inventors have been described above, but the present invention is not limited to the descriptions and drawings that constitute a part of the present invention. That is, all other embodiments, examples, and techniques applied based on the present invention are included within the scope of the present invention.

[0105] Label Explanation

[0106] 20 server devices

[0107] 21, 31 Reasoning Department

[0108] 30 edge devices

[0109] 32 Judgment Department

[0110] 33 Quantitative Department

[0111] 100 processing system

Claims

1. A processing system that processes using an edge device and a server device, characterized by the edge device including a first inference section that extracts a feature amount of processing target data using a first model, and performs inference processing on the processing target data based on the extracted feature amount; and a determination section that outputs an inference result of the first inference section in a case where a reliability of the inference result of the first inference section exceeds a threshold value, and outputs a feature amount of the processing target data to the server device in a case where the reliability is below the threshold value, the server device having a second inference section that performs inference processing on the processing target data based on the feature amount of the processing target data output from the edge device using a second model that has higher inference accuracy than the first model, the first model having a first extraction section that extracts a feature amount of the processing target data; and a first execution section that performs first inference processing based on the feature amount of the processing target data, the second model having a second execution section that performs second inference processing based on the feature amount of the processing target data, the second execution section performing the second inference processing based on the feature amount of the processing target data extracted by the first extraction section, a model obtained by deleting a feature extraction layer in the first model before being configured to the edge device, and configuring a feature extraction layer of a second model trained before being configured to the server device to a front stage of the first execution section of the edge device as the first extraction section, is configured as the first model, a model obtained by deleting the feature extraction layer from the second model before being configured to the server device is configured as the second model.

2. The processing system according to claim 1, characterized by the determination section outputs the quantized feature amount of the processing target data to the server device.

3. The processing system according to claim 1, characterized by the reliability is based on an entropy of the inference result of the first inference section.

4. A processing method executed by a processing system that processes using an edge device and a server device, characterized by the processing method having a first inference process in which the edge device extracts a feature amount of processing target data using a first model, and performs inference processing on the processing target data based on the extracted feature amount; and a determination process in which the edge device outputs an inference result in the first inference process in a case where a reliability of the inference result exceeds a threshold value, and outputs a feature amount of the processing target data to the server device in a case where the reliability is below the threshold value; and a second inference process in which the server device performs inference processing on the processing target data based on the feature amount of the processing target data output from the edge device using a second model that has higher inference accuracy than the first model, the first model having a first extraction section that extracts a feature amount of the processing target data; and a first execution section that performs first inference processing based on the feature amount of the processing target data, the second model having a second execution section that performs second inference processing based on the feature amount of the processing target data, the second execution section performing the second inference processing based on the feature amount of the processing target data extracted by the first extraction section. a first execution unit that executes first inference processing based on the feature amount of the processing target data, the second model has: a second execution unit that executes second inference processing based on the feature amount of the processing target data, the second execution unit executes the second inference processing based on the feature amount of the processing target data extracted by the first extraction unit, as the first model, a model obtained by deleting a feature extraction layer in the first model before being configured to the edge device and configuring a feature extraction layer of a second model trained before being configured to the server device to a stage preceding the first execution unit of the edge device, as the second model, a model obtained by deleting the feature extraction layer from the second model before being configured to the server device.

5. A recording medium storing a processing program that causes a computer to execute a method, characterized by the processing program causes a computer that is an edge device to execute: a first inference step of extracting a feature amount of processing target data using a first model and executing inference processing on the processing target data based on the extracted feature amount; and a determination step of outputting an inference result in the first inference step when reliability of the inference result exceeds a threshold value, and outputting the feature amount of the processing target data to a server device when the reliability is below the threshold value, the processing program causes a computer that is a server device to execute a second inference step of executing inference processing on the processing target data based on the feature amount of the processing target data output from the edge device using a second model having higher inference accuracy than the first model, the first model has: a first extraction unit that extracts a feature amount of the processing target data; and a first execution unit that executes first inference processing based on the feature amount of the processing target data, the second model has: a second execution unit that executes second inference processing based on the feature amount of the processing target data, the second execution unit executes the second inference processing based on the feature amount of the processing target data extracted by the first extraction unit, as the first model, a model obtained by deleting a feature extraction layer in the first model before being configured to the edge device and configuring a feature extraction layer of a second model trained before being configured to the server device to a stage preceding the first execution unit of the edge device, as the second model, a model obtained by deleting the feature extraction layer from the second model before being configured to the server device.

Images