Inference device and inference method
The inference device clusters classes into higher classes, determining optimal resolutions for each class to enhance prediction accuracy and speed in image classification.
Patent Information
- Application Number
- JP2024069432
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-04-23
- Publication Date
- 2025-11-05
AI Technical Summary
Existing neural networks struggle to predict the optimal resolution for input images consistently, leading to varying inference accuracy across different images.
An inference device and method that clusters classes into higher classes, determines a resolution for each class, converts the image resolution accordingly, and performs classification using a neural network.
Maintains high prediction accuracy across all input images by optimizing resolution for each class, balancing inference speed and accuracy.
Smart Images
Figure 2025165427000001_ABST
Abstract
Description
[Technical Field]
[0001] The present disclosure relates to an inference device and an inference method that convert the resolution of an input image to perform inference. [Background technology]
[0002] Non-Patent Document 1 describes a neural network that converts (resizes) the resolution of an input image to reduce the computational load when performing inference on the input image. The resolution after conversion is a resolution that can maintain inference accuracy. Inference on an input image includes classification of objects (samples) in the input image. In general, resolution conversion reduces the resolution while maintaining inference accuracy. Non-Patent Document 1 introduces that samples such as pandas can be predicted correctly even at low resolution, but samples that easily blend into the background, such as thread dragonflies, cannot be classified correctly unless they are at high resolution.
[0003] The neural network described in Non-Patent Document 1 includes a resolution predictor and an image classifier. The resolution predictor is trained in advance using images with various resolutions. In the inference phase, the resolution predictor predicts the minimum resolution of an input image at which the image classifier can perform inference without reducing inference accuracy. After the resolution of the input image is converted to the predicted resolution, the resolution-converted input image is input to the image classifier. The image classifier performs inference based on the input image. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Mingjian Zhu, et al., "Dynamic Resolution Network", NeurIPS 2021 Conference Submissions Summary of the Invention [Problem to be solved by the invention]
[0005] In the neural network described in Non-Patent Document 1, it is difficult for the resolution predictor to predict the optimal resolution for all input images. This is because it is difficult for the resolution predictor to learn the appropriate resolution for each image across all of the various images. In other words, it is difficult to maintain high prediction accuracy for resolution prediction across all input images.
[0006] An object of the present invention is to provide an inference device and an inference method that can maintain high prediction accuracy in resolution prediction across all input images. [Means for solving the problem]
[0007] An inference device based on the present disclosure is an inference device that performs inference by converting the resolution of an input image, and includes a clustering means that clusters multiple classes to be classified into multiple higher classes, a resolution determination means that determines the resolution corresponding to each of the multiple higher classes, a prediction means that predicts the higher class to which the class to be classified in the input image belongs, a resolution conversion means that converts the resolution of the input image to a resolution that corresponds to the predicted higher class, and a classification means that performs class classification on the input image whose resolution has been converted.
[0008] An inference method based on the present disclosure is an inference method that performs inference by converting the resolution of an input image, clustering multiple classes to be classified into multiple higher classes, determining the resolution corresponding to each of the multiple higher classes, predicting the higher class to which the class to be classified in the input image belongs, converting the resolution of the input image to the resolution corresponding to the predicted higher class, and performing class classification using the input image with the converted resolution as the target.
[0009] An inference program based on the present disclosure is an inference program for converting the resolution of an input image to perform inference, and causes a computer to perform the following processes: clustering multiple classes to be classified into multiple higher classes; determining the resolution corresponding to each of the multiple higher classes; predicting the higher class to which the class to be classified in the input image belongs; converting the resolution of the input image to the resolution corresponding to the predicted higher class; and performing class classification using the input image with its resolution converted. [Effects of the Invention]
[0010] According to the present invention, high prediction accuracy of resolution prediction can be maintained across all input images. [Brief explanation of the drawings]
[0011] [Figure 1] FIG. 2 is a block diagram showing an example of the configuration of an inference device. [Figure 2] FIG. 10 is an explanatory diagram showing an example of inference accuracy for each input resolution of a classifier. [Figure 3] 10 is a flowchart illustrating an example of the operation of the inference device in a learning phase. [Figure 4] 10 is a flowchart showing an example of the operation of the inference device in an inference phase. [Figure 5] 10 is a flowchart showing another example of the operation of the inference device in the learning phase. [Figure 6] FIG. 10 is a block diagram showing another example configuration of the inference device. [Figure 7] 10 is a flowchart showing another example of the operation of the inference device in the inference phase. [Figure 8] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a wireless sensing system. [Figure 9] FIG. 10 is an explanatory diagram showing a method for adjusting the number of dimensions of the output of a classifier. [Figure 10] FIG. 1 is a block diagram illustrating an example of the configuration of an information processing device. [Figure 11] FIG. 2 is a block diagram showing the main parts of the inference device. DETAILED DESCRIPTION OF THE INVENTION
[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0013] Embodiment 1. Fig. 1 is a block diagram showing an example of the configuration of an inference device according to an embodiment. The inference device 100 shown in Fig. 1 includes a higher class determination unit 101, a resolution predictor 102 configured, for example, by a neural network, a learning unit 103, a resolution converter 111, and a classifier 112 configured, for example, by a neural network. The inference device 100 is a device that primarily classifies objects in an input image, i.e., objects appearing in the image.
[0014] Note that the arrows in Figure 1 simply indicate the direction of signal (data) flow, but do not exclude bidirectionality. This also applies to other block diagrams. In Figure 1, dashed arrows indicate the flow of signals (data) in the learning phase, and solid arrows indicate the flow of signals (data) in the inference phase.
[0015] First, the concept of the present disclosure will be explained. Fig. 2 is an explanatory diagram showing an example of inference accuracy for each input resolution for classifier 112 in inference device 100. The input resolution in Fig. 2 is the resolution of the input image input to classifier 112.
[0016] In FIG. 2(A), class n (n=0 to 5) corresponds to the classification target. Examples of classification targets include dogs, cats, airplanes, and apples. The rows corresponding to each of a plurality of resolutions (512, 256, 128, 64, 32, and 16 in the example of FIG. 2) illustrate the inference accuracy of each class. The average inference accuracy is the average of the inference accuracy of classes 0 to 5. For example, a resolution of m (m=512, 256, 128, 64, 32, or 16) means that, assuming that the input image is square, the number of pixels in the width direction and the number of pixels in the height direction of the image are m. Furthermore, the resolution of m may be defined as the number of pixels per predetermined unit (e.g., inch) in the width direction and the height direction. In any case, a resolution with a larger value is higher than a resolution with a smaller value. For example, the resolution of 512 is the same as the resolution of the input image to the inference device 100.
[0017] Fig. 2(B) shows a graph of the inference accuracy for each input resolution exemplified in Fig. 2(A). In general, the lower the input resolution, the higher the processing speed (inference speed) of the classifier 112.
[0018] Therefore, input resolution can be thought of as inference speed.
[0019] In the example shown in Figure 2, the inference accuracy does not decrease significantly even if the input resolution is lowered for classes 0, 2, 4, and 5. For classes 1 and 3, the inference accuracy decreases as the input resolution is lowered.
[0020] Therefore, if classes 0 to 5 are clustered into multiple clusters and one resolution is assigned to each cluster, it may be possible to increase the inference speed while preventing a decrease in inference accuracy. For example, for classes 0, 2, 4, and 5, the inference accuracy does not decrease significantly even if the input resolution is lowered, so they are clustered into one cluster and the resolution for that cluster is lowered. Also, for classes 1 and 3, the inference accuracy decreases when the input resolution is lowered, so they are clustered into another group and the resolution corresponding to that cluster is increased. Hereinafter, the clusters will be referred to as higher classes.
[0021] The upper class consisting of classes 0, 2, 4, and 5 is called upper class A. The upper class consisting of classes 1 and 3 is called upper class B. As an example, the resolution for upper class A is set to 16, and the resolution for upper class B is set to 256. Hereinafter, the resolution for the upper classes may be referred to as upper resolution. In this example, the number of upper classes is 2, but the number of upper classes may be 3 or more. For example, if a user desires higher inference accuracy, the number of upper classes is increased. For example, if a user desires to prioritize inference speed, the number of upper classes is decreased.
[0022] In Figure 2(B), the average input resolution and average inference accuracy are shown by stars when the upper class consisting of classes 0, 2, 4, and 5 is designated as upper class A, the resolution for upper class A is set to 16, the upper class consisting of classes 1 and 3 is designated as upper class B, and the resolution for upper class B is set to 256. The average input resolution is (98 + 95 + 67 + 95 + 79 + 83) / 6 = 86.1. The average inference accuracy is (16 × 4 + 256 × 2) / 6 = 96. As illustrated in Figure 2(B), the average inference accuracy when each class is clustered into the upper class is higher than the average inference accuracy when the input resolution is the same. In other words, the average input resolution when clustered is lower than the input resolution corresponding to the average inference accuracy when the inference accuracy is the same.
[0023] [Learning phase processing] A plurality of resolution candidates output by the resolution predictor 102 is set in advance by, for example, a user. The resolution candidates are, for example, 512, 256, 128, 64, 32, and 16.
[0024] Furthermore, the inference accuracy for each input resolution when each of the resolution candidates is used as the input resolution is calculated in advance. For example, the classifier 112 calculates the inference accuracy of each class for each resolution candidate. In this example, when the resolution candidates are 512, 256, 128, 64, 32, and 16, the inference accuracy for each input resolution shown in FIG. 2B is obtained. Note that if the inference accuracy for each input resolution in the classifier 112 has already been obtained, there is no need to reevaluate the inference results by the classifier 112.
[0025] In inference device 100, higher class determination unit 101 determines a higher class. Determining a higher class involves determining a class to which each of a plurality of higher classes belongs. Take the example where the resolution candidates are 512, 256, 128, 64, 32, and 16, and the number of higher classes is two. Higher class determination unit 101 determines a higher class for each combination of resolution candidates (two resolution candidates in this example). In this example, there are six types of resolution, so there are 6C2 = 15 possible combinations of resolution candidates.
[0026] The following method is an example of a method for determining a higher class for a combination of resolution candidates. In the following example, it is assumed that higher class A is a higher class with a lower resolution, and higher class B is a higher class with a higher resolution.
[0027] As an example, the upper class determination unit 101 clusters, in a combination of resolution candidates (two resolutions in this example), a class whose inference accuracy calculated by the classifier 112 does not fall below a predetermined threshold even when the input resolution is lowered, into upper class A. Furthermore, the upper class determination unit 101 clusters, in a combination of resolution candidates (two resolutions in this example), a class whose inference accuracy falls below a threshold when the input resolution is lowered into upper class B.
[0028] For example, if the combination of resolution candidates is 256 and 16, and the inference accuracy for each input resolution is obtained as shown in Figure 2, when the input resolution is less than 128, classes 1 and 3, which are below the predetermined threshold of 60, are clustered into higher class A, and classes 0, 2, 4, and 5 are clustered into higher class B.
[0029] The upper class determination unit 101 performs the above process for all combinations of resolution candidates (15 combinations of resolution candidates in the above example). As a result, clustering for upper classes is performed for each combination of resolution candidates. Note that upper classes A and B at this stage are not final upper classes, but are candidates for the upper classes.
[0030] Furthermore, in the above example, the upper class determination unit 101 performs clustering for the upper classes using a threshold value, but it may also perform clustering for the upper classes using other methods, for example, an evaluation function.
[0031] The upper class determination unit 101 may perform clustering of upper classes based on prior knowledge. For example, if a class (e.g., dogs) and another class (e.g., cats) are similar to each other and therefore difficult to distinguish, the classes are clustered into the same upper class. Whether the classes are similar or not is determined based on, for example, feature vectors.
[0032] The upper class determination unit 101 may perform clustering for the upper classes using the K-means method. When using the K-means method, for example, the upper class determination unit 101 targets combinations of resolution candidates and utilizes the difference in inference accuracy between the classes.
[0033] Furthermore, in order to determine the resolution for each upper class, the upper class determination unit 101 performs the following process for each of all combinations of resolution candidates.
[0034] That is, the upper class determination unit 101 calculates the average input resolution and average inference accuracy of all classes (e.g., classes 0 to 5) for each combination of resolution candidates. As a result, the average input resolution and average inference accuracy are obtained for all combinations of resolution candidates. Note that the inference accuracy for each resolution of each class used to calculate the average input resolution and average inference accuracy has been calculated by the classifier 112.
[0035] An example of the average input resolution and average inference accuracy when the number of top classes is 2, the resolution for top class A is 16, and the resolution for top class B is 256 corresponds to the average input resolution and average inference accuracy indicated by stars in Figure 2. For example, if the number of combinations of resolution candidates is 15, 15 stars are plotted on the graph, following the example graph shown in Figure 2(B).
[0036] The upper class determination unit 101 selects one combination of resolution candidates from all combinations of resolution candidates. The upper class determination unit 101 determines each resolution candidate in the selected combination of resolution candidates as the final resolution combination to be provided to the resolution predictor 102. In this manner, the resolution combination is determined.
[0037] The following method is an example of a method for selecting a combination of resolution candidates.
[0038] As an example, the superior class determination unit 101 selects the combination with the highest average inference accuracy from among the combinations with an average input resolution equal to or less than a predetermined value. Following the graph illustrated in FIG. 2B, the star with the highest average inference accuracy is selected from among the star symbols with an average input resolution equal to or less than a predetermined value. Note that, as described above, the star symbol is based on the average input resolution and average inference accuracy of the multiple resolutions (e.g., two resolutions) that make up the candidate combination of resolutions.
[0039] As another example, the upper class determination unit 101 uses an evaluation function such as (input resolution+λ inference accuracy) to select a combination of resolution candidates that maximizes the value of the evaluation function.
[0040] The multiple resolutions that make up the determined combination of resolutions correspond to higher classes. For example, if the determined combination of resolutions is 256 and 16, the resolution 256 corresponds to higher class B, and the resolution 16 corresponds to higher class A. Therefore, the higher class determination unit 101 may supply a label of the higher class to the resolution predictor 102 as information that can identify the resolution. Below, an example will be given in which the higher class determination unit 101 supplies a label of the higher class to the resolution predictor 102.
[0041] After the above-described process of determining the higher class and the process of selecting the resolution combination (determination process) are completed, the learning unit 103 instructs the resolution predictor 102 to start learning for classifying the image into one of the multiple higher classes. The resolution predictor 102 is a type of learning model (prediction model).
[0042] In inference device 100, a large number of training data sets (training data sets) stored in advance in training data storage unit 200 are sequentially supplied to upper class determination unit 101 and resolution predictor 102. Each training data set is assigned a label. For example, the label corresponds to one of classes 0 to 5 described above.
[0043] Specifically, in response to an instruction from the learning unit 103, the resolution predictor 102 sequentially reads out the learning data from the learning data storage unit 200. In addition, the upper class determination unit 101 reads out from the learning data storage unit 200 the label corresponding to the learning data read out by the resolution predictor 102.
[0044] The upper class determination unit 101 supplies the label of the upper class to which the label (i.e., class) of the training data belongs to the resolution predictor 102. The resolution predictor 102 learns to which of the multiple upper classes an image read from the training data storage unit 200 corresponds. In other words, the resolution predictor 102 performs learning to classify an image into one of the multiple upper classes.
[0045] Next, the operation of the upper class determination unit 101 and the resolution predictor 102 in the learning phase will be described with reference to the flowchart of FIG.
[0046] For all of the resolution candidates set in advance, the inference accuracy of each class is calculated (step S101). As described above, the inference accuracy of each class for all of the resolution candidates is calculated by, for example, the classifier 112.
[0047] The higher class determination unit 101 selects one combination of resolution candidates from all of the resolution candidates, and executes the above-mentioned higher class determination process for the selected combination of resolution candidates (step S102). Note that, as described above, the higher class determined at this stage is a higher class candidate.
[0048] When the process of determining a higher class has been executed for all of the resolution candidates, the process proceeds to step S104 (step S103). If the process of determining a higher class has not yet been executed for all of the resolution candidates, the process returns to step S102, and the process of determining a higher class is executed for another combination of resolution candidates.
[0049] In step S104, the upper class determination unit 101 determines a final combination of resolutions. As described above, the upper class determination unit 101 calculates the average input resolution and average inference accuracy of all classes for each candidate combination of resolutions, and determines the final combination of resolutions based on the calculation results.
[0050] Next, the resolution predictor 102 performs learning to classify the image into one of a plurality of higher classes, as described above (step S105).
[0051] (Processing in the inference phase) Next, processing in the inference phase will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the operation of inference device 100 in the inference phase.
[0052] In reasoning apparatus 100, an input image (eg, an image with a resolution of 512) is input to resolution predictor 102 and resolution converter 111.
[0053] In inference apparatus 100, resolution predictor 102 predicts to which higher class an object in an input image input to inference apparatus 100 belongs (step S111).
[0054] The resolution predictor 102 outputs the resolution corresponding to the predicted upper class to the resolution converter 111. The resolution converter 111 converts the resolution of the input image input to the inference device 100 to the resolution corresponding to the predicted upper class (step S112). In the process of step S112, the resolution converter 111 reduces the resolution if a higher class with a resolution lower than the resolution of the input image is predicted. In the process of step S112, the resolution converter 111 can use pixel thinning or max-pooling techniques to reduce the resolution.
[0055] The classifier 112 predicts a class using the input image with reduced resolution (step S113). Then, the classifier 112 outputs the prediction result. The prediction result is output to, for example, a storage device or a display device.
[0056] It should be noted that a general learning model that has already been trained can be used as the classifier 112. Furthermore, the learning model as the classifier 112 may be trained using images with a resolution that can be used as training data.
[0057] As described above, inference device 100 of this embodiment clusters multiple classes into multiple clusters, assigns one resolution to each cluster, i.e., upper class, and converts the input image to a resolution corresponding to the cluster before performing prediction (e.g., classification). Furthermore, for each of the multiple resolutions set, inference device 100 determines a resolution (resolution corresponding to the cluster) that increases the average inference accuracy of the multiple classes (average inference accuracy). Therefore, inference device 100 can increase the inference speed while preventing a decrease in inference accuracy.
[0058] [Another example of the first embodiment] FIG. 5 is a flowchart showing another example of the operation of inference device 100 in the learning phase.
[0059] In the above embodiment, the upper class determination unit 101 performs processing to determine an upper class, and the resolution predictor 102 performs learning regarding the resolution corresponding to each upper class. After that, the resolution predictor 102 performs processing to predict a resolution appropriate for the input image without performing new learning (training). However, as shown in Fig. 5, the resolution predictor 102 may perform learning at a predetermined timing in the learning phase.
[0060] In this example, similar to the above embodiment, the inference accuracy of each class is calculated for all preset resolution candidates (step S101). The classifier 112 is configured to calculate and output the inference accuracy of the class for each resolution at a predetermined timing in the learning phase. The predetermined timing is during the process of determining the upper class or after learning of the resolution predictor.
[0061] Furthermore, the upper class determination unit 101 selects one combination of resolution candidates from all of the resolution candidates, and executes the above-described upper class determination process for the selected combination of resolution candidates (step S102).
[0062] The resolution predictor 102 performs learning to classify an image into one of a plurality of higher classes (step S121). The process of step S121 corresponds to the process of step S105 in the above embodiment.
[0063] Next, for each resolution constituting the combination of resolution candidates selected in step S102, the average inference accuracy and average inference speed that will be obtained by the classifier 112 are calculated (step S122). In step S122, unlike the above embodiment, for example, the actually trained resolution predictor 102 and classifier 112 are operated to obtain the average inference accuracy and average inference speed.
[0064] Once the processes of steps S102, S121, and S122 have been executed for all combinations of resolution candidates, the process proceeds to step S124. If there is still a combination of resolution candidates for which the processes of steps S102, S121, and S122 have not been executed, the process returns to step S102, and the processes of steps S102, S121, and S122 are executed for another combination of resolution candidates.
[0065] In step S124, the upper class determination unit 101 determines a final combination of resolutions. For each candidate combination of resolutions, the upper class determination unit 101 determines the final combination of resolutions based on the values of the average input resolution and average inference accuracy calculated in the processing of step S122. As one example, the upper class determination unit 101 selects the combination with the highest average inference accuracy from among combinations with an average input resolution equal to or greater than a predetermined value. As another example, the upper class determination unit 101 uses an evaluation function such as (input resolution + λ inference accuracy) to select the candidate combination of resolutions that maximizes the value of the evaluation function.
[0066] The operation of the inference device 100 in this example in the inference phase is the same as the operation in the above embodiment.
[0067] In this example, the upper class determination unit 101 determines the final resolution combination based on the average resolution and average inference accuracy calculated using the prediction results of the resolution predictor, so it is expected that the accuracy of the inference process (prediction process) by the classifier 112 will improve.
[0068] Embodiment 2. Fig. 6 is a block diagram showing an example configuration of an inference device. The inference device 110 shown in Fig. 6 includes a superclass determination unit 101, a resolution predictor 102 configured, for example, by a neural network, a learning unit 103, a resolution converter 111, a classifier 112 configured, for example, by a neural network, and a model switching unit 113. The inference device 110 is a device that mainly classifies objects in an input image, i.e., objects appearing in the image. The configurations and functions of the superclass determination unit 101, the resolution predictor 102, the learning unit 103, and the resolution converter 111 are the same as those in the first embodiment.
[0069] In the first embodiment, classifier 112 in inference device 100 is implemented as one learning model. For example, if the number of upper classes is two and the number of classes is six, classifier 112 is a six-class classification model that supports two types of resolution.
[0070] However, the classifier 112 may be configured with multiple learning models. For example, if the number of upper classes is two and the number of classes is six, the classifier 112 may be configured with two learning models corresponding to the respective upper classes. The reason for this configuration is that the classes to be predicted are determined for each resolution.
[0071] Based on the above-mentioned concept, in this embodiment, the classifier 112 is configured with a plurality of learning models.
[0072] As an example, suppose that the resolution corresponding to upper class A is 16 and the resolution corresponding to upper class B is 256. Also, suppose that four classes (e.g., classes 0, 2, 4, and 5) belong to upper class A, and two classes (e.g., classes 1 and 3) belong to upper class B.
[0073] In this case, the classifier 112 is configured with a four-class classification model corresponding to one resolution (for example, 16) and a two-class classification model corresponding to the other resolution (for example, 256).
[0074] The model switching unit 113 receives the resolution predicted by the resolution predictor 102. The model switching unit 113 instructs the classifier 112 to use a learning model corresponding to the received resolution. Specifically, the model switching unit 113 switches the learning model to be used when the resolution changes. In the classifier 112, the learning model after the switch executes prediction (e.g., classification).
[0075] The processing of the learning phase of inference device 110 is the same as the operation of inference device 100 of the first embodiment (see FIGS. 3 and 5).
[0076] Next, processing in the inference phase will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the operation of inference device 110 in the inference phase.
[0077] The processes of steps S111 and S112 are the same as those in the first embodiment shown in Fig. 4. In this embodiment, the model switching unit 113 switches the learning model used in the classifier 112 when the resolution changes (step S131).
[0078] The processing in step S113 is basically the same as the processing in the first embodiment shown in FIG. 4, but in this embodiment, the switched learning model in the classifier 112 predicts the class using the input image whose resolution has been converted.
[0079] In this embodiment, the number of classes to be classified by the learning model in the classifier 112 is reduced, which is expected to reduce the difficulty of the task and improve the inference accuracy. In addition, it becomes possible to simplify the architecture of the learning model.
[0080] (Variation 1) In the above embodiment, the resolution predictor 102 performs learning before the start of the inference phase of the classifier 112 or at a predetermined timing during the inference phase. When performing learning, the resolution predictor 102 may include a function for searching for the architecture (neural network structure: number of layers, number of channels, how layers are connected, etc.) of a prediction model (included in the resolution predictor 102) that predicts to which higher class an object in an input image belongs.
[0081] The resolution predictor 102 performs architecture search, for example, as follows.
[0082] As an example, a neural network structure similar to that of the training model in the classifier 112 is trained, and architecture search is performed by referring to the inference accuracy of the classifier 112. For example, the resolution predictor 102 gradually simplifies the neural network structure of a prediction model similar to that of the training model in the classifier 112. The simplification involves reducing the number of layers and channels. Then, the resolution predictor 102 stops the simplification when the inference accuracy falls below a predetermined tolerance. The resolution predictor 102 uses the neural network structure at that time as the new neural network structure of the predictor.
[0083] As another example, the resolution predictor 102 may use techniques such as Bayesian optimization to optimize neural network structures, such as hyperparameters.
[0084] Furthermore, as another example, the resolution predictor 102 may optimize the neural network structure using a general NAS (Neural Architecture Search) method.
[0085] (Variation 2) The inference devices 100, 110 of the above embodiments may be applied to a wireless sensing system. A wireless sensing system is a system that identifies an object using received radio waves. For example, there is a type of wireless sensing system that identifies a transmitter in a radio wave environment. There is also a wireless sensing system that detects an object based on the radio wave conditions that are affected by electromagnetic waves hitting the object. Below, we will take as an example a wireless sensing system that identifies a transmitter.
[0086] 8 is an explanatory diagram showing an example of the configuration of a wireless sensing system. The wireless sensing system 500 shown in FIG. 8 includes a receiving unit 501, a feature element generating unit 502, an inference device 503, a matching unit 504, and a template storage unit 600.
[0087] The receiving unit 501 receives electromagnetic waves. The feature element generating unit 502 extracts feature elements (characteristic elements) from signals based on the received electromagnetic waves. The receiving unit 501 can extract feature elements from, for example, the spectrum of the received waves or intensity changes in the spectrogram.
[0088] As the inference device 503, the inference device 100 of the first embodiment or the inference device 110 of the second embodiment can be used.
[0089] The matching unit 504 estimates the transmitter by comparing the prediction result of the inference device 503 with the templates stored in the template storage unit 600. In other words, the matching unit 504 can identify the transmitter by template matching.
[0090] When inference device 100 of the first embodiment is used as inference device 503, classifier 112 can handle multiple input resolutions and the number of dimensions of its output is always constant, making template matching easy.
[0091] When the inference device 110 of the second embodiment is used, the learning model is switched. Since the number of output dimensions of the inference unit differs for each learning model, the number of output dimensions of the classifier 112 needs to be adjusted to match the number of dimensions of the templates stored in the template storage unit 600. In other words, the number of output dimensions of the classifier 112 needs to be adjusted.
[0092] 9 is an explanatory diagram showing a method for adjusting the number of dimensions of the output of the classifier 112. In Fig. 9, "y" represents an element of the vector output by the classifier 112. "x" represents the output from the layer immediately before the output layer of the learning model included in the classifier 112.
[0093] 9(A), multiple template storage units are prepared according to the resolution, and the template storage unit is switched according to the predicted resolution, similar to the switching of the learning model in the classifier 112. In this case, a template storage unit switching unit is provided in the wireless sensing system 500.
[0094] In the method shown in FIG. 9(B), the classifier 112 outputs the prediction result by setting the dimensions corresponding to non-corresponding classes to 0 so that the prediction result by the classifier 112 is equal to the number of dimensions of the template.
[0095] In the method shown in Figure 9(C), the number of nodes in the layer immediately before the output layer in each learning model (n in the example shown in Figure 9(C)) is unified, and the classifier 112 outputs the value of that layer in the inference phase.
[0096] Fig. 10 is a block diagram showing an example configuration of an information processing device (computer) capable of implementing the functions of the inference devices 100, 110 of the above-described embodiments. The information processing device shown in Fig. 10 includes one or more processors such as CPUs (Central Processing Units), a program memory 1002, and a memory 1003. Fig. 4 shows an example of an information processing device having one processor 1001.
[0097] The program memory 1002 is, for example, a non-transitory computer-readable medium. Non-transitory computer-readable media include various types of tangible storage media. For example, the program memory 1002 can be a semiconductor storage medium such as a flash ROM (Read Only Memory) or a magnetic storage medium such as a hard disk. The program memory 1002 stores an inference program for implementing the functions of each block (the upper class determination unit 101, the resolution predictor 102, the learning unit 103, the resolution converter 111, the classifier 112, and the model switching unit 113) in the inference devices 100 and 110 of the above-described embodiments.
[0098] Processor 1001 realizes the functions of inference devices 100 and 110 by executing processing in accordance with an inference program stored in program memory 1002. If multiple processors are installed, the functions of inference devices 100 and 110 can also be realized by the multiple processors working together.
[0099] For example, a RAM (Random Access Memory) can be used as the memory 1003. The memory 1003 stores temporary data generated when the inference devices 100, 110 are executing processing. It is also possible to imagine a configuration in which an inference program is transferred to the memory 1003, and the processor 1001 executes processing based on the inference program in the memory 1003. Note that the program memory 1002 and the memory 1003 may be integrated.
[0100] Fig. 11 is a block diagram showing the main components of an inference device. The inference device 10 (corresponding to inference device 100 or 110) shown in Fig. 11 includes clustering means 11 (implemented by upper class determination unit 101 in the embodiment) that clusters multiple classes to be classified into multiple upper classes, resolution determination means 12 (implemented by upper class determination unit 101 in the embodiment) that determines a resolution corresponding to each of the multiple upper classes, prediction means 13 (implemented by resolution predictor 102 in the embodiment) that predicts the upper class to which the class to be classified in the input image belongs, resolution conversion means 14 (implemented by resolution converter 111 in the embodiment) that converts the resolution of the input image to a resolution corresponding to the predicted upper class, and classification means 15 (implemented by classifier 112 in the embodiment) that classifies the resolution-converted input image.
[0101] In the inference device 10, the classification means 15 includes a plurality of learning models that perform class classification on input images converted to a resolution corresponding to one of a plurality of higher classes, and the inference device 10 may also be provided with a switching means that switches the learning model used for class classification (for example, changes the learning model used to the learning model corresponding to the changed higher class B) when the higher class predicted by the prediction means 13 is changed (for example, changed from higher class A to higher class B).
[0102] Although part or all of the above-described embodiments can be described as follows, the present invention is not limited to the following configurations.
[0103] (Supplementary Note 1) An inference device that performs inference by converting the resolution of an input image, a clustering means for clustering a plurality of classes to be classified into a plurality of higher classes; a resolution determination means for determining a resolution corresponding to each of the plurality of higher classes; a prediction means for predicting a higher class to which a class to be classified in an input image belongs; a resolution conversion means for converting the resolution of the input image into a resolution corresponding to the predicted higher class; a classification means for performing class classification on the input image whose resolution has been converted; An inference device comprising:
[0104] (Supplementary Note 2) The classification means includes a plurality of learning models that perform class classification on the input image converted into a resolution corresponding to one of the plurality of higher classes, and a switching means for switching the learning model used for class classification when the higher class predicted by the prediction means is changed. 2. The inference apparatus of claim 1.
[0105] (Supplementary Note 3) A calculation means is provided for calculating the inference accuracy of each of the plurality of classes for each of a plurality of resolutions that can be used; The clustering means performs clustering using the inference accuracy of the classes at each of a plurality of resolutions. 10. The inference device of claim 1 or 2.
[0106] (Supplementary Note 4) The resolution determination means determines a resolution corresponding to each of the plurality of higher classes based on the average inference accuracy of the plurality of classes at resolutions corresponding to the number of higher classes selected from the plurality of resolutions that can be used, using the inference accuracy. 10. The inference apparatus of claim 3.
[0107] (Supplementary Note 5) The prediction means includes a prediction model; A search unit is provided for performing an architecture search for the prediction model when learning the higher class to which the class to be classified in the input image belongs. 10. The inference device of claim 1 or 2.
[0108] (Appendix 6) Incorporated into wireless sensing system 10. The inference device of claim 1 or 2.
[0109] (Appendix 7) An inference method for converting the resolution of an input image and performing inference, Clustering the classes to be classified into multiple higher classes; determining a resolution corresponding to each of the plurality of superclasses; Predicting a higher class to which the class to be classified in the input image belongs; converting the resolution of the input image to a resolution corresponding to the predicted higher class; Classification is performed on the input image whose resolution has been converted. Reasoning method.
[0110] (Supplementary Note 8) When the predicted upper class is changed, the learning model for classifying the input image converted to a resolution corresponding to one of the plurality of upper classes is switched. The inference method described in Appendix 7.
[0111] (Appendix 9) An inference program for converting the resolution of an input image and performing inference, On the computer, A process of clustering a plurality of classes to be classified into a plurality of higher classes; determining a resolution corresponding to each of the plurality of superclasses; A process of predicting a higher class to which the class to be classified in the input image belongs; converting the resolution of the input image to a resolution corresponding to the predicted higher class; A process of performing class classification on the input image whose resolution has been converted. An inference program for executing the above.
[0112] (Appendix 10) To the computer, When the predicted upper class is changed, a process of switching a learning model for classifying the input image converted into a resolution corresponding to one of the plurality of upper classes is executed. 9. The inference program according to claim 9.
[0113] (Appendix 11) The inference method according to appendix 7 or appendix 8, implemented by a computer.
[0114] Some or all of the configurations described in Supplementary Notes 3 to 6, which are subordinate to Supplementary Note 1, may be subordinate to Supplementary Notes 7 and 9 in the same subordinate relationship as Supplementary Notes 3 to 6. Furthermore, not limited to Supplementary Notes 1, 7, and 8, some or all of the configurations described as the above Supplements may be subordinate to various hardware, software, various recording means for recording software, or systems, provided that they do not deviate from the above-described embodiments. [Explanation of symbols]
[0115] 10 Reasoning device 11 Clustering Methods 12 Resolution determination method 13 Prediction methods 14 Resolution conversion method 15 Classification means 100, 110 Reasoning device 101 Upper Class Decision Section 102 Resolution Predictor 103 Learning Department 111 Resolution Converter 112 Classifier 113 Model switching unit 200 Learning data storage unit 500 Wireless Sensing System 501 Receiving unit 502 Feature element generation unit 503 Reasoning device 504 Matching Unit 600 Template Storage 1001 processor 1002 program memory 1003 memory
Claims
1. An inference device that performs inference by converting the resolution of an input image, a clustering means for clustering a plurality of classes to be classified into a plurality of higher classes; a resolution determination means for determining a resolution corresponding to each of the plurality of higher classes; a prediction means for predicting a higher class to which a class to be classified in an input image belongs; a resolution conversion means for converting the resolution of the input image into a resolution corresponding to the predicted higher class; a classification means for performing class classification on the input image whose resolution has been converted; An inference device comprising:
2. the classification means includes a plurality of learning models that perform class classification on the input image converted to a resolution corresponding to one of the plurality of higher classes; and a switching means for switching the learning model used for class classification when the higher class predicted by the prediction means is changed. The inference device according to claim 1 .
3. a calculation means for calculating an inference accuracy of each of the plurality of classes for each of a plurality of resolutions that can be used; The clustering means performs clustering using the inference accuracy of the classes at each of a plurality of resolutions.
3. The inference device according to claim 1 or 2.
4. The resolution determination means determines a resolution corresponding to each of the plurality of higher classes based on the average inference accuracy of the plurality of classes at resolutions corresponding to the number of higher classes selected from the plurality of resolutions that can be used, using the inference accuracy. The inference device according to claim 3 .
5. the prediction means includes a prediction model; A search unit is provided for performing an architecture search for the prediction model when learning the higher class to which the class to be classified in the input image belongs.
3. The inference device according to claim 1 or 2.
6. Incorporated into wireless sensing systems 3. The inference device according to claim 1 or 2.
7. An inference method for performing inference by converting the resolution of an input image, Clustering the classes to be classified into multiple higher classes; determining a resolution corresponding to each of the plurality of superclasses; Predicting a higher class to which the class to be classified in the input image belongs; converting the resolution of the input image to a resolution corresponding to the predicted higher class; Classification is performed on the input image whose resolution has been converted. Reasoning method.
8. When the predicted upper class is changed, a learning model is switched to perform class classification on the input image converted to a resolution corresponding to one of the plurality of upper classes. The inference method of claim 7.
9. An inference program for converting the resolution of an input image and performing inference, On the computer, A process of clustering a plurality of classes to be classified into a plurality of higher classes; determining a resolution corresponding to each of the plurality of superclasses; A process of predicting a higher class to which the class to be classified in the input image belongs; converting the resolution of the input image to a resolution corresponding to the predicted higher class; A process of performing class classification on the input image whose resolution has been converted. An inference program for executing the above.
10. On the computer, When the predicted upper class is changed, a process of switching a learning model for classifying the input image converted into a resolution corresponding to one of the plurality of upper classes is executed. The inference program according to claim 9.