Inference processing device, imaging device, inference processing method and program

Abstract

To reduce the time for inference arithmetic processing while maintaining inference accuracy required for an inference application in an inference processing apparatus that performs inference using an NN.SOLUTION: An inference processing apparatus includes: inference arithmetic means that is formed of a neural network in which an input layer that receives input of an input image, a first intermediate layer that extracts a feature quantity included in the input image, a first output layer that outputs a first classification result based on the feature quantity extracted by the first intermediate layer, a second intermediate layer that extracts a feature quantity with the output from the first intermediate layer as input, and a second output layer that outputs a second classification result based on the feature quantity extracted by the second intermediate layer are hierarchically connected; and abstraction level determination means that determines the level of abstraction of the classification results. According to the level of abstraction of the first classification result determined by the abstraction level determination means. The inference processing apparatus switches whether to stop arithmetic executed by the inference arithmetic means with the first classification result or execute arithmetic until the second classification result is obtained.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an inference processing device, an imaging device, an inference processing method, and a program. [Background technology]

[0002] Conventionally, inference processing devices that perform inference using neural networks (hereinafter also simply referred to as "NNs") have been known. In particular, convolutional neural networks (hereinafter also simply referred to as "CNNs") are often used as NNs in inference processing devices that perform image recognition.

[0003] In an inference processing device that performs image recognition using CNN, image data input to the input layer (hereinafter simply referred to as the "input image") is sequentially processed in the intermediate layer and the fully connected layer to obtain a final inference result that recognizes the object contained in the input image. In the intermediate layer, multiple processing layers for extracting features are hierarchically connected, and each processing layer performs convolution, activation, and pooling on the input data (hereinafter simply referred to as the "input") input from the previous layer. In this way, the intermediate layer extracts high-dimensional features contained in the input image by repeating the convolution, activation, and pooling processes in each processing layer. In the fully connected layer, the calculation result data of the intermediate layer (i.e., the features extracted by the intermediate layer, hereinafter simply referred to as the "output of the intermediate layer") is combined, classification is performed based on the features, and a final inference result (classification result) is output. Hereinafter, the fully connected layer is also simply referred to as the "output layer."

[0004] Such multi-layer neural networks, which have an intermediate layer composed of multiple processing layers hierarchically connected, are also called deep neural networks (DNNs). DNNs have attracted much attention in recent years due to their excellent image recognition capabilities.

[0005] Neural network calculations (processing) are performed in two stages: a learning phase and an inference phase. In the learning phase, a large amount of image data is used as training data to extract trends and perform learning (learning calculations) to improve the recognition accuracy of image recognition. A trained neural network (hereinafter simply referred to as a "trained neural network" or "trained NN model") is obtained through the learning performed in the learning phase. In the inference phase, inference (inference calculations) is performed to derive an answer (i.e., an inference result) for unknown image data based on the learning results (i.e., the trained NN model) obtained in the learning phase.

[0006] For example, there are two types of layers with weights that are modified during CNN training: the convolutional layer and the fully connected layer (output layer). These weights are used as filter values ​​in the convolutional layer, and become calculation parameter values ​​adjusted as weight coefficients and biases in the fully connected layer (output layer). Hereinafter, these weights will be collectively referred to as "calculation parameters." CNN training is the process of modifying calculation parameters, and calculation parameters are modified by CNN training. Hereinafter, calculation parameters modified by CNN training will be referred to as "learned calculation parameters." The configuration of CNN (middle layer and output layer) and the combination of learned calculation parameters to derive a certain answer are called a "dictionary," and it is common to use multiple dictionaries for each object or group containing objects in image recognition.

[0007] On the other hand, when using NN inference for the autofocus (AF) of an imaging device, there are cases where the level of abstraction of the classification is switched during inference. For example, if you want to focus only on the eyes of a specific animal, such as a cat or a dog, you need to specifically classify the subject, which is the target of image recognition. Also, if you want to focus on animals regardless of species, you need to use abstract classification so that the answer (inference result) is not biased towards a specific animal.

[0008] As a related technology, for example, the technology disclosed in Patent Document 1 has been proposed. In the technology disclosed in Patent Document 1, in order to improve the recognition accuracy of a recognition target, an attention area is set based on the recognition result (classification result) of a recognition process using an inference calculation of a neural network, and a detailed recognition process that is more detailed than the recognition process is performed on the set attention area. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Publication No. 2019-96006 Summary of the Invention [Problem to be solved by the invention]

[0010] However, in the technology of Patent Document 1, dictionaries are switched according to the switching of the abstraction level of the classification, and multiple dictionaries must be prepared for each abstraction level of the classification of the recognition target, which results in a huge amount of calculation for learning the NN.Furthermore, in the technology of Patent Document 1, in order to perform detailed recognition processing, the inference calculation of the NN must be executed again, which causes a problem of delay in the processing time of the inference calculation.

[0011] When the technology of Patent Document 1 is used in an application requiring real-time performance, such as an AF application in an imaging device, such a delay in processing time is undesirable, particularly in continuous AF shooting operations.

[0012] Furthermore, a NN has the property that the more intermediate layers it has, the higher the accuracy of image recognition. However, as the number of intermediate layers increases, the amount of calculations performed by the NN becomes enormous, resulting in increased processing time and power consumption. In particular, when a NN with many layers is used for applications that require real-time performance, such as autofocus in an imaging device, high-speed and ample hardware is required to complete the NN calculations within a specified time, which increases costs.

[0013] An object of the present invention is to shorten the processing time of inference calculations while maintaining the inference accuracy required for the inference application in an inference processing device that performs inference using a neural network. [Means for solving the problem]

[0014] In order to achieve the above object, an inference processing device of the present invention comprises an inference calculation means configured by a neural network in which an input layer for inputting an input image, a first intermediate layer for extracting features included in the input image, a first output layer for outputting a first classification result based on the features extracted by the first intermediate layer, a second intermediate layer for extracting features using an output of the first intermediate layer as an input, and a second output layer for outputting a second classification result based on the features extracted by the second intermediate layer are hierarchically connected; an imaging means; and a control means for controlling the imaging means to operate in any one of a plurality of imaging modes including a specific imaging mode for photographing a specific subject, wherein the inference calculation means accepts an image obtained by imaging using the imaging means as an input image of the neural network, and when the first classification result outputted by the inference calculation means in the specific imaging mode is a classification result including the specific subject, performs inference calculation until the second classification result is obtained. Performance of and if the first classification result output by the inference calculation means in the specific shooting mode is not a classification result that includes the specific subject, the calculation by the inference calculation means is terminated with the first classification result. [Effects of the Invention]

[0015] According to the present invention, in an inference processing device that performs inference using a neural network, it is possible to reduce the processing time for inference calculations while maintaining the inference accuracy required for the inference application. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing an example of the configuration of an imaging device having an inference processing device according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a diagram for explaining the flow of inference (inference calculation) by a trained neural network in the inference processing device according to the first embodiment. [Figure 3] FIG. 2 is a diagram for explaining a trained NN model obtained by training (training calculation) in the inference processing device according to the first embodiment. [Figure 4] FIG. 3 is a diagram showing an example of an abstraction level table of the inference processing device according to the first embodiment. [Figure 5] FIG. 10 is a block diagram showing an example of the configuration of an imaging device having an inference processing device according to a second embodiment of the present invention. [Figure 6] 10 is a flowchart showing the flow of inference (inference calculation) in an inference processing device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in each of the embodiments.

[0018] First Embodiment An inference processing device that performs inference using a neural network according to a first embodiment of the present invention will now be described with reference to the drawings. Fig. 1 is a block diagram showing an example of the configuration of an imaging device that has an inference processing device according to the first embodiment of the present invention.

[0019] 1, in the imaging device 101, a CPU (Central Processing Unit) 112, a ROM (Read Only Memory) 113, a memory 114, an imaging unit 115, a display unit 118, an input unit 117, and an inference processing device 102 are connected to a system bus 111. The inference processing device 102 has an abstraction level determination unit 116 as an abstraction level determination means, and an inference calculation unit 119 as an inference calculation means. The units connected to the system bus 111 are configured to be able to exchange data with each other via the system bus 111.

[0020] The CPU 112 controls the imaging unit 115, the display unit 118, the abstraction level determination unit 116, and the inference calculation unit 119 in accordance with a program based on a control signal corresponding to the user operation, which is generated by the input unit 117 that has received the user operation. The CPU 112 serves as control means and controls the operation of the entire imaging device 101 that includes the inference processing device 102. Under the control of the CPU 112, the imaging unit 115, the display unit 118, the abstraction level determination unit 116, and the inference calculation unit 119 perform processing and / or operations corresponding to the user operation, respectively.

[0021] The ROM 113 stores various programs and data for the CPU 112 to perform control. Note that the various programs and data for the CPU 112 to perform control may be stored in an information storage device other than the ROM. For example, the various programs and data for the CPU 112 to perform control may be stored in an external storage device such as a hard disk drive.

[0022] The memory 114 is configured by, for example, a RAM (random access memory). The CPU 112 performs control in accordance with a program read from the ROM 113, using the memory 114 as a work memory.

[0023] The imaging unit 115 has a lens (not shown), a shutter (not shown) with an aperture function, an imaging means (not shown), and an image processing unit (not shown) that performs various image processing such as exposure control and distance measurement control on a signal output from the imaging means. The imaging unit 115 acquires an image by performing imaging processing according to the imaging mode.

[0024] The imaging means is composed of an imaging element or the like that converts an optical image into an electrical signal. The imaging element is composed of a solid-state imaging element such as a CCD image sensor or a CMOS image sensor. Under the control of the CPU 112, the imaging unit 115 can perform imaging processing in various imaging modes in response to user operations performed via the input unit 117 (hereinafter, the imaging processing will be simply referred to as "imaging").

[0025] The abstraction level determination unit 116 determines the level of abstraction of the classification result required when performing inference using a neural network. The abstraction level of the classification result referred to in the present invention is defined as follows: the more specific the classification, the lower the level of abstraction of the classification result, and the more abstract the classification, the higher the level of abstraction of the classification result. For example, if the classification result is a specific classification of only one specific animal type, such as cats or dogs, the level of abstraction of the classification result is low. On the other hand, if the answer (classification result) is not biased toward a specific animal but is an abstract classification of small animals such as cats or dogs or pets, the level of abstraction of the classification result is high. In the inference processing device 102 according to the first embodiment, the abstraction level determination unit 116 is configured to determine the level of abstraction of specific classifications to be low and the level of abstract classifications to be high.

[0026] Regarding the method of determining the abstraction level of the classification result, for example, the CPU 112 may determine the abstraction level of the classification result in the abstraction level determination unit 116 in response to a user operation performed via the input unit 117. Note that the method of determining the abstraction level of the classification result is not limited to the above-described method, and the abstraction level of the classification result may be determined based on an abstraction level table that is prepared in advance and stored in the ROM 213 and that associates the classification result with the operation mode (shooting mode) of the imaging device.

[0027] The input unit 117 receives a user operation, generates a control signal according to the received user operation, and transmits the generated control signal to the CPU 112. For example, the input unit 117 has physical operation buttons, a touch panel, or the like as input means for receiving the user operation. Note that the touch panel is an input device that outputs coordinate information according to the position of contact with a planar input unit, for example.

[0028] Display unit 118 is configured, for example, by a display, and includes a mechanism for outputting a display signal for displaying an image on the display. When a touch panel is used as input unit 117, input unit 117 and the display (display unit 118) can be configured as an integrated unit. For example, the touch panel is configured so that its light transmittance does not interfere with the display on the display, and is attached to the upper layer of the display surface of the display. Then, input coordinates on the touch panel can be associated with display coordinates on the display.

[0029] The inference calculation unit 119 has the configuration of a neural network calculation unit, and analyzes still images or moving images (hereinafter, still images and moving images will be collectively referred to as "images" without any particular distinction) acquired by the imaging unit 115, and performs calculations such as detection and classification of objects included in the images. That is, the inference calculation unit 119 is configured as a neural network, and performs inference by executing calculation processing based on a trained neural network (trained NN model).

[0030] As described above, neural network calculations (processing) are performed in two phases: a learning phase and an inference phase. In the inference processing device 102 according to the first embodiment, the learning of the neural network constituting the inference calculation unit 119 is performed, for example, on a cloud server (not shown). A neural network learning method of the inference processing device 102 according to the first embodiment will be described later with reference to FIG. 3.

[0031] The neural network constituting the inference calculation unit 119 is configured, for example, by a graphics processing unit (hereinafter simply referred to as "GPU"). A GPU is a computing device capable of performing specialized processing for computer graphics calculations, and is generally capable of processing calculations required for neural networks, such as matrix calculations, in a shorter time than a CPU. Furthermore, the neural network constituting the inference calculation unit 119 does not necessarily have to be configured by a single GPU, but may be configured by multiple GPUs. Furthermore, the neural network constituting the inference calculation unit 119 does not necessarily have to be configured by a GPU; any configuration capable of performing neural network calculations can be applied to a neural network.

[0032] FIG. 2 is a diagram for explaining the flow of inference (inference calculation) by a trained neural network in the inference processing device 102 according to the first embodiment.

[0033] 2, in the trained neural network in the inference processing device 102 according to the first embodiment, an input layer 251, a first hidden layer 252, a second hidden layer 253, a first output layer 254, and a second output layer 255 are hierarchically connected. In the imaging device 101, the inference calculation unit 119 is configured to perform inference by executing calculation processes on the input layer 251, the first hidden layer 252, the second hidden layer 253, the first output layer 254, and the second output layer 255. That is, the inference calculation unit 119 is configured to perform inference (inference calculation) under the control of the CPU 112 of the imaging device 101. The inferences performed by the inference calculation unit 119 include a first inference and a second inference, which will be described later.

[0034] The inference calculation unit 119 performs a first inference by executing calculation processing from the input layer 251 to the first hidden layer 252, and outputs the output of the first hidden layer 252 obtained by the first inference to two layers, the first output layer 254 and the second hidden layer 253. Under the control of the CPU 112 of the imaging device 101, the inference calculation unit 119 is configured to perform the first inference after an image acquired by the imaging unit 115 is input to the input layer 251.

[0035] FIG. 2 shows an example of a first inference in which the first hidden layer 252 performs calculations on a seven-layer neural network including the input layer 251. In the example of the first inference in FIG. 2, the number of layers from the input layer 251 to the first hidden layer 252 is seven, but the number of layers is not limited to seven and can be changed to another number of layers. However, it is necessary to configure the NN from the input layer 251 to the first hidden layer 252 (i.e., to determine the number of layers of the NN) so that the first classification result (described later) obtained in the first inference and output from the first output layer 254 satisfies the inference accuracy required for the purpose of the inference. A method for determining the number of layers will be described later using FIG. 3.

[0036] The first output layer 254 outputs an inference result by the first inference based on the feature amount extracted by the first hidden layer 252 (i.e., the output of the first hidden layer 252). In the first embodiment, the inference result by the first inference refers to the classification of each object included in the input image, as described above. This classification result (the inference result by the first inference) output from the first output layer 254 is defined as the first classification result.

[0037] If the first classification result obtained by the first inference satisfies the inference accuracy required for the inference application, the inference processing device 102 determines that the first classification result is a candidate for termination, and terminates the calculation by the inference calculation unit 119 under the control of the CPU 112. On the other hand, if the first classification result obtained by the first inference does not satisfy the inference accuracy required for the inference application, the inference processing device 102 determines that the first classification result is not a candidate for termination, and proceeds to the second inference.

[0038] The inference calculation unit 119 performs a second inference by executing calculation processing in the second hidden layer 253, and outputs the output of the second hidden layer 253 obtained by the second inference to the second output layer 255.

[0039] 2 shows an example of a second inference in which the second hidden layer 253 performs calculations on a seven-layer neural network using the output of the first hidden layer 252 as input. In the example of the second inference in FIG. 2, the number of layers in the second hidden layer 253 is seven, but the number of layers in the second hidden layer 253 is not limited to seven and can be changed to another number of layers.

[0040] The second output layer 255 outputs an inference result by the second inference based on the feature amount extracted by the second intermediate layer 253 (i.e., the output of the second intermediate layer 253). In the first embodiment, the inference result by the second inference refers to the classification of each object included in the input image, as described above. This classification result (the inference result by the second inference) output from the second output layer 255 is defined as the second classification result.

[0041] As described above, in the trained neural network, the inference calculation unit 119 extracts high-dimensional features contained in the input image by repeating processing at each intermediate layer. Therefore, in order to extract high-dimensional features, the number of intermediate layers is important for the accuracy of the classification results. Therefore, in the inference processing device 102 according to the first embodiment, the first classification result is a classification with a high level of abstraction, and the second classification result is a classification with a low level of abstraction.

[0042] In the example mentioned above, the second classification result would be a specific classification of only one specific type of animal, such as cats or dogs, while the answer (classification result) would not be biased towards a specific animal, and the first classification result would be an abstract classification of small animals or pets such as cats and dogs.

[0043] In the following, the first embodiment will be described with reference to an inference processing device 102, assuming that five classes are possible based on the first classification result and that 20 classes are possible based on the second classification result. Note that the number of classes that can be classified based on the assumed first classification result and second classification result (hereinafter simply referred to as the "number of classifications") is a number set simply to make the description of the inference processing device 102 easier to understand. During training of the neural network, which will be described later, the number of classifications based on the first classification result and the number of classifications based on the second classification result may be freely set. Furthermore, if inference accuracy is not a concern, the number of classifications based on the first classification result and the number of classifications based on the second classification result may be set to be the same.

[0044] In the inference processing device 102 according to the first embodiment, the features of the learning performed in the learning phase of the neural network that constitutes the inference calculation unit 119 will be described with reference to FIG.

[0045] FIG. 3 is a diagram illustrating a trained NN model obtained by training (learning calculation) in the inference processing device according to the first embodiment. In the embodiment of the present invention, training of the neural network constituting the inference calculation unit 119 is performed in advance, for example, by a high-performance computer or GPU (not shown). Training of the neural network constituting the inference calculation unit 119 is divided into a first training and a second training. That is, a feature of training of the neural network constituting the inference calculation unit 119 is that the second training is performed after the first training. FIG. 3(A) is a diagram illustrating the first training of the neural network constituting the inference calculation unit 119, and FIG. 3(B) is a diagram illustrating the second training of the neural network constituting the inference calculation unit 119.

[0046] In an embodiment of the present invention, first, in the initial learning, a neural network configured with an input layer 251, an intermediate layer 301, and an output layer 302 shown in FIG. 3(A) is trained to determine the configurations and calculation parameters of the intermediate layer 301 and the output layer 302. Hereinafter, the neural network configured with the input layer 251, the intermediate layer 301, and the output layer 302 shown in FIG. 3(A) is also simply referred to as a "first NN" or a "first learned NN model." As described above, in the first embodiment, the configurations and calculation parameters of the intermediate layer 301 and the output layer 302 are determined by learning the first NN so that the five-class classification as the first classification result obtained by the trained first NN satisfies the inference accuracy required for the inference application. Hereinafter, the configurations and calculation parameters of the intermediate layer 301 and the output layer 302 determined by learning the first NN are also simply referred to as a "first trained NN model." The first trained NN model obtained by the initial training (i.e., the configurations and operation parameters of the hidden layer 301 and the output layer 302 determined by the initial training) becomes the configurations and operation parameters of the first hidden layer 252 and the first output layer 254 shown in FIG. 2. In this way, the five-class classification as the first classification result by a seven-layer inferencing device based on the first trained NN model (i.e., a trained first NN configured with seven layers, i.e., the input layer 251 and the hidden layer 301) can satisfy the inference accuracy required for the inference application. Incidentally, if the five-class classification as the first classification result by the seven-layer inferencing device does not satisfy the inference accuracy required for the inference application, various methods are possible to make this five-class classification satisfy the inference accuracy required for the inference application. For example, the number of layers (seven) of the input layer 251 and the hidden layer 301 may be changed to another number of layers (e.g., eight or nine layers), or the number of classifications (five) by the output layer 302 may be reduced. As described above, the configuration and calculation parameters of the first hidden layer 252 and the first output layer 254 shown in FIG. 2 are determined by the learning result of the first learning (first trained NN model) shown in FIG. 3(A).

[0047] Next, in the second learning, a neural network composed of the input layer 251, first hidden layer 252, hidden layer 311, and output layer 312 shown in FIG. 3(B) is trained to determine the configurations and operation parameters of the hidden layer 311 and the output layer 312. Hereinafter, the neural network composed of the input layer 251, first hidden layer 252, hidden layer 311, and output layer 312 shown in FIG. 3(B) is also simply referred to as the "second NN" or "second learning NN model." In the second learning, the configuration and operation parameters of the first hidden layer 252 determined in the first learning are used as they are, and learning of the second NN begins from the output of the first hidden layer 252. In other words, in the second learning, the configurations and operation parameters of the hidden layer 311 and the output layer 312 are determined by learning the second NN, but the configuration and operation parameters of the first hidden layer 252 are not changed from the learning results of the first learning shown in FIG. 3(A). That is, the configuration and calculation parameters of the first hidden layer 252 determined in the initial learning are used as the configuration and calculation parameters of the first hidden layer 252 of the second NN. As described above, in the first embodiment, the configurations and calculation parameters of the hidden layer 311 and the output layer 312 are determined by learning the second NN so that the 20-class classification as the second classification result obtained by the trained second NN satisfies the inference accuracy required for the inference application. Hereinafter, the configuration and calculation parameters of the first hidden layer 252 of the second NN and the configurations and calculation parameters of the hidden layer 311 and the output layer 312 determined by learning the second NN are collectively referred to as the "second trained NN model." The configurations and calculation parameters of the hidden layer 311 and the output layer 312 determined by the second learning become the configurations and calculation parameters of the second hidden layer 253 and the second output layer 255 shown in FIG. 2. In this way, the 20-class classification as the second classification result by the 14-layer inference device based on the second trained NN model can now satisfy the inference accuracy required for the inference application. Incidentally, the 14-layer inference device based on the second trained NN model is a trained second NN composed of a total of 14 layers, including the input layer 251, the first hidden layer 252, and the hidden layer 311.

[0048] In this manner, in the embodiment of the present invention, by performing the first learning and the second learning, it is possible to configure inference machines with different levels of classification abstraction (the above-mentioned 7-layer inference machine and 14-layer inference machine) with completely identical configurations and calculation parameters up to the middle. Specifically, the configuration and calculation parameters of the first hidden layer 252 determined in the first learning are used in both the first inference for obtaining a first classification result with a high level of abstraction and the second inference for obtaining a second classification result with a low level of abstraction.

[0049] Next, a specific example of how to use inference by the inference processing device 102 in the imaging device 101 will be described using AF use of the imaging device as a specific example with reference to Fig. 4. Fig. 4 is a diagram showing an example of an abstraction level table of the inference processing device 102 according to the first embodiment.

[0050] The abstraction level table shown in FIG. 4 is an abstraction level table that associates the first classification result obtained by the first inference performed by the inference processing device 102 with the shooting mode of the imaging device 101. As shown in FIG. 4, the first classification result is classified into five classes, for example, people 401, animals 402, plants 403, sky 404, and food 405. As described above, the imaging device 101 can capture images in various shooting modes. For simplicity, however, it is assumed that the imaging device 101 has a people shooting mode 411, a landscape shooting mode 412, and a food shooting mode 413. The input unit 117 accepts a user operation specifying a desired shooting mode, generates a control signal in accordance with the accepted user operation, and the CPU 112, upon receiving the control signal, selects the desired shooting mode. Note that in the embodiment of the present invention, the method for selecting a shooting mode is not limited to the above-described selection method, and other selection methods may be used. For example, the CPU 112 of the imaging device 101 may automatically select an optimal shooting mode. As described above, the abstraction level table shown in Fig. 4 may be stored in advance in the ROM 213. Also, the abstraction level table shown in Fig. 4 may be freely changed by the user.

[0051] The flow of inference by the inference processing device 102 in the image capture device 101 will be described based on the abstraction level table (hereinafter simply referred to as "table") shown in FIG.

[0052] When photography is performed in portrait photography mode 411, if the first classification result from the first inference is a person 401, the inference processing device 102 proceeds to the second inference based on the table, performs the second inference, outputs the second classification result from the second inference, and terminates the inference.

[0053] On the other hand, when photography is performed in portrait mode 411, if the first classification result is animal 402, plant 403, sky 404, or food 405, the inference processing device 102 terminates the calculation based on the table, outputs the first classification result, and ends the inference.

[0054] When photography is performed in landscape photography mode 412, if the first classification result is plants 403 or sky 404, the inference processing device 102 proceeds to a second inference based on the table, performs the second inference, outputs the second classification result based on the second inference, and terminates the inference.

[0055] On the other hand, when shooting in landscape shooting mode 412, if the first classification result is a person 401, an animal 402, or a food 405, the inference processing device 102 terminates the calculation based on the table, outputs the first classification result, and ends the inference.

[0056] When photography is performed in food photography mode 413, if the first classification result from the first inference is food 405, the inference processing device 102 proceeds to the second inference based on the table, performs the second inference, outputs the second classification result from the second inference, and terminates the inference.

[0057] On the other hand, when photography is performed in food photography mode 413, if the first classification result is person 401, animal 402, plant 403, or sky 404, the inference processing device 102 terminates the calculation based on the table, outputs the first classification result, and ends the inference.

[0058] The specific example described above is an example in which the inference processing device 102 performs inference based on a table linking the first classification results to three shooting modes of the imaging device 101. However, the inference processing device 102 according to the first embodiment is not limited to this specific example. For example, the inference processing device 102 according to the first embodiment can perform inference based on a table linking the first classification results to various shooting modes of the imaging device 101. Furthermore, the inference processing device 102 according to the first embodiment may be configured to switch between terminating the inference calculation at the first classification result and performing the inference calculation up to the second classification result, depending on whether the inference processing device 102 is performing continuous shooting or video capture or single shooting. During continuous shooting or video capture for the first time, the inference processing device 102 performs calculation up to the second classification result for focusing. However, as long as the first classification result remains unchanged from the first time, the inference processing device 102 determines that the focus target has not changed based on the first classification result, thereby reducing the processing time for the calculation to obtain the second classification result.

[0059] Furthermore, the use of inference by the inference processing device 102 according to the first embodiment is not limited to the above-described AF use, and may be other uses of the imaging device 101. That is, the inference processing device 102 according to the first embodiment may be configured to switch between terminating the inference calculation at the first classification result and performing the inference calculation up to the second classification result, depending on uses other than AF use of the imaging device 101. For example, the inference processing device 102 according to the first embodiment may be configured to perform the inference calculation up to the second classification result when performing inference to tag a captured image, and to terminate the inference calculation at the first classification result when performing inference for AF without tagging.

[0060] As described above, the inference processing device 102 according to the first embodiment can configure inference devices with different levels of classification abstraction by making their configurations and calculation parameters completely identical up to a certain point. Therefore, it is possible to shorten the time required for inference when a classification with a low level of abstraction (corresponding to the second classification result in the first embodiment) is required. Furthermore, it is possible to abort the inference calculation based on the result of a classification with a high level of abstraction (corresponding to the first classification result in the first embodiment), thereby shortening the processing time for the inference calculation without performing unnecessary inference.

[0061] In this way, in the first embodiment, multiple levels of abstraction can be expressed using one inference device (the trained neural network shown in FIG. 2) configured based on two trained NN models (the above-mentioned 7-layer inference device and 14-layer inference device) obtained through two rounds of learning shown in FIG. 3. Therefore, it is possible to provide an inference processing device that reduces the impact on continuous shooting operations.

[0062] As described above, the trained neural network in the inference processing device 102 according to the first embodiment is configured with an inference unit capable of inferring two classifications with different levels of abstraction, the first classification result and the second classification result, as shown in FIG. 2. However, the present invention is not limited to this. By using a learning method similar to the learning method shown in FIG. 3, it is possible to configure an inference unit capable of inferring four classifications with different levels of abstraction by setting the levels of abstraction in a stepwise manner, such as the third classification result and the fourth classification result. In other words, the trained neural network in the inference processing device according to the embodiment of the present invention may be configured with an inference unit capable of inferring four classifications with different levels of abstraction, the first classification result, the second classification result, the third classification result, and the fourth classification result.

[0063] Second Embodiment An inference processing device according to a second embodiment of the present invention, which performs inference using a neural network taking into account the reliability of the inference result, will be described below. Fig. 5 is a block diagram showing an example of the configuration of an imaging device having an inference processing device according to the second embodiment of the present invention.

[0064] 5, in the imaging device 501, a CPU 512, a ROM 513, a memory 514, an imaging unit 515, a display unit 518, an input unit 517, and an inference processing device 502 are connected to a system bus 511. The inference processing device 502 has an abstraction level determination unit 516 as an abstraction level determination means, an inference calculation unit 519 as an inference calculation means, and a reliability determination unit 520 as a reliability determination means. The units connected to the system bus 511 are configured to be able to exchange data with each other via the system bus 511.

[0065] The configurations and operations of the system bus 511, CPU 512, ROM 513, memory 514, and imaging unit 515 are similar to those of the system bus 111, CPU 112, ROM 113, memory 114, and imaging unit 115, and therefore descriptions thereof will be omitted. Also, the configurations and operations of the abstraction level determination unit 516, input unit 517, display unit 518, and inference calculation unit 519 are similar to those of the abstraction level determination unit 116, input unit 117, display unit 118, and inference calculation unit 119, and therefore descriptions thereof will be omitted.

[0066] The reliability determination unit 520 determines the reliability of the inference result (classification result) obtained by inference using the trained neural network in the inference processing device 502. The reliability determination unit 520 may also be configured as a circuit (e.g., ASIC). Here, a method for displaying the classification result obtained by inference using a trained NN model (trained inference device) will be described. Inference using a trained NN is generally performed in the form of determining which class among the possible classes the result most closely resembles. Specifically, for example, when inference is performed on unknown image data based on an inference device (trained NN model) capable of classifying the data into five categories—people, animals, plants, sky, and food—the inference result (classification result) is displayed as 90% people, 3% animals, and so on, totaling 100%. Here, 90% people means that the inference result (classification result) was inferred to be a person with a 90% probability, i.e., the inference result (classification result) corresponds to the person class with a 90% probability. Also, Animal 3% means that the inference result (classification result) was inferred to be an animal with a 3% probability, that is, the inference result (classification result) falls into the animal class with a 3% probability. Furthermore, if the inference result (classification result) does not fall into any of the classes of person, animal, plant, sky, or food, it is displayed in a way that makes it 100% including others.

[0067] The reliability of the inference result (classification result) displayed in the form of a probability as described above is determined by the reliability determination unit 520. The reliability determination method used by the reliability determination unit 520 will be described below. A threshold is set for the probability, and if the probability of the inference result (classification result) exceeds the set threshold (a predetermined percentage), the reliability of the inference result (classification result) is determined to be high, and if the probability of the inference result is equal to or lower than the set threshold, the reliability is determined to be low. Note that a high reliability of the inference result (classification result) means that the reliability required for the inference purpose is achieved, and a low reliability of the inference result (classification result) means that the reliability required for the inference purpose is not achieved.

[0068] The reliability determination method used by the reliability determination unit 520 is not limited to the above-described reliability determination method, and other methods may be used. For example, a method may be used in which the reliability is determined to be high when the same inference result (classification result) is obtained from multiple consecutive images, and the reliability is determined to be low when the inference results (classification results) obtained from multiple consecutive images are different. The reliability determination unit 520 may also be configured with a reliability table, which is prepared in advance and stored in the ROM 213, linking the inference results (classification results) to the reliability of the inference results (classification results). It may be determined based on such a reliability table whether the reliability required for the inference purpose is obtained.

[0069] As described above, in the inference calculation unit of the inference processing device according to an embodiment of the present invention, a trained neural network extracts high-dimensional features contained in an input image by repeating processing at each intermediate layer. Therefore, in order to extract high-dimensional features, the number of intermediate layers is important for the accuracy of the classification results. In the first embodiment, the number of classifications based on the first classification result obtained by the first trained neural network with a small number of layers is different from the number of classifications based on the second classification result obtained by the second trained neural network with a large number of layers. This allows both the first and second neural networks to be trained to achieve the required inference accuracy. However, some input images are difficult to infer, and there are cases where, at the time the first classification result is obtained, it is impossible to determine whether the first classification result satisfies the inference accuracy required for the inference application. This applies, for example, when the probabilities of the two classes in the first classification result are nearly identical, such as 45% for people and 43% for animals.

[0070] We will now describe an inference processing device according to a second embodiment of the present invention, which maintains the inference accuracy required for the inference application by performing inference up to the second classification result depending on the reliability of the inference result even in the above-mentioned case.

[0071] 6 is a flowchart showing the flow of inference (inference calculation) in an inference processing device 502 according to a second embodiment of the present invention, which performs inference using a neural network that takes into account the reliability of the inference result. In the following description, "S" denotes a step. The inference processing device 502 is configured to perform processes from S601 to S605 under the control of the CPU 512, which serves as control means for the imaging device 501. The inference performed by the inference calculation unit 519 includes a first inference and a second inference.

[0072] When an unknown image is input to the input layer 251, under the control of the CPU 512 of the imaging device 501, the inference processing device 502 performs the processes from S601 to S605 shown in the flowchart of FIG. 6 as follows.

[0073] First, in S601, the inference calculation unit 519 performs a first inference by executing calculation processing from the input layer 251 to the first hidden layer 252, and outputs the output of the first hidden layer 252 obtained by the first inference to two layers, the first output layer 254 and the second hidden layer 253. Incidentally, the output of the first hidden layer 252 is the first classification result.

[0074] Next, in S602, the inference processing device 502 determines the reliability of the first classification result using the reliability determination unit 520, and determines whether the reliability required for inference purposes is achieved based on the determined reliability of the first classification result. Specifically, in S602, if the determined reliability of the first classification result is high, the inference processing device 502 determines that the reliability required for inference purposes is achieved, and proceeds to S603. On the other hand, if the determined reliability of the first classification result is low, the inference processing device 502 determines that the reliability required for inference purposes is not achieved, and proceeds to S605.

[0075] Next, in S603, the inference processing device 502 determines whether the first classification result is a candidate for termination. Incidentally, the method for determining whether the first classification result is a candidate for termination has already been described in the first embodiment, and therefore will not be described here. Specifically, in S603, if the inference processing device 502 determines that the first classification result is a candidate for termination, i.e., if it determines that there is no need to perform the second inference, it proceeds to S604. On the other hand, in S603, if the inference processing device 502 determines that the first classification result is not a candidate for termination, i.e., if it determines that there is a need to perform the second inference, it proceeds to S605.

[0076] In S604, under the control of the CPU 512, the inference processing device 502 terminates the calculation by the inference calculation unit 519, sets the first classification result output to the first output layer 254 as the final classification result, and ends the inference (inference calculation).

[0077] Meanwhile, in S605, the inference calculation unit 519 performs a second inference by executing calculation processing in the second hidden layer 253, and outputs the output of the second hidden layer 253 obtained by the second inference to the second output layer 255. Incidentally, the output of the second output layer 255 is a second classification result. In S605, the inference processing device 502 regards the second classification result output to the second output layer 255 as the final classification result, and ends the inference (inference calculation).

[0078] As described above, in the second embodiment, by taking into consideration the reliability of the inference result, it is possible to switch between performing calculations by the inference calculation unit 519 up to the first inference or performing calculations by the inference calculation unit 519 up to the second inference, thereby maintaining the inference accuracy required for the inference application. The inference processing device 502 according to the second embodiment is an inference processing device that utilizes the characteristic of NN that the recognition accuracy of image recognition increases as the number of intermediate layers increases. Therefore, according to the inference processing device 502, even an input image that cannot be inferred by the first inference can be inferred by proceeding up to the second inference, so that by performing inference calculations only when necessary, it is possible to maintain the inference accuracy required for the inference application.

[0079] In this way, in the second embodiment, multiple levels of abstraction can be expressed using one inference device (the trained neural network shown in FIG. 2) configured based on two trained NN models (the 7-layer inference device and the 14-layer inference device described above) obtained through two rounds of learning shown in FIG. 3. Therefore, it is possible to provide an inference processing device that minimizes the impact on continuous shooting operations. Furthermore, by taking into account the reliability of the inference results, it is possible to configure an inference processing device that does not reduce inference accuracy even in calculations with a high level of abstraction.

[0080] As described above, the trained neural network in the inference processing device 502 according to the second embodiment is configured with an inference unit capable of inferring two classifications with different levels of abstraction, the first classification result and the second classification result, as shown in FIG. 2. However, the present invention is not limited to this. By using a learning method similar to the learning method shown in FIG. 3, it is possible to configure an inference unit capable of inferring four classifications with different levels of abstraction by setting the levels of abstraction in a stepwise manner, such as the third classification result and the fourth classification result. In other words, the trained neural network in the inference processing device according to the embodiment of the present invention may be configured with an inference unit capable of inferring four classifications with different levels of abstraction, the first classification result, the second classification result, the third classification result, and the fourth classification result.

[0081] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and variations are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. [Explanation of symbols]

[0082] 102 Inference Processing Unit 116 Abstraction level determination part 119 Inference calculation section 251 Input Layer 252 The First Middle Class 253 The Second Middle Class 254 First output layer 255 second output layer 502 Inference Processing Unit 516 Abstraction level determination part 519 Inference calculation section 520 Reliability determination unit

Claims

1. An input layer that inputs an input image; a first intermediate layer that extracts features included in the input image; a first output layer that outputs a first classification result based on the feature amount extracted by the first intermediate layer; a second intermediate layer that receives the output of the first intermediate layer as an input and extracts features; a second output layer that outputs a second classification result based on the feature quantity extracted by the second intermediate layer; and an inference calculation means configured of a neural network in which these are hierarchically connected. An imaging means; a control means for controlling the imaging means to operate in any one of a plurality of imaging modes including a specific imaging mode for photographing a specific subject; Equipped with the inference calculation means receives an image obtained by capturing an image with the imaging means as an input image for the neural network; When the first classification result output by the inference calculation means in the specific photographing mode is a classification result including the specific subject, calculation is performed until the second classification result is obtained; An inference processing device characterized in that, if the first classification result output by the inference calculation means in the specific shooting mode is not a classification result that includes the specific subject, the calculation by the inference calculation means is terminated at the first classification result.

2. The learning of the neural network is carried out in two stages: a first learning and a second learning; In the first learning, a first neural network configured with the input layer, the first intermediate layer, and the first output layer is trained to determine configurations and operation parameters of the first intermediate layer and the first output layer; 2. The inference processing device according to claim 1, wherein in the second learning, the configuration and operation parameters of the first hidden layer determined in the first learning are used as they are, and learning of a second neural network consisting of the input layer, the first hidden layer, the second hidden layer, and the second output layer is started from the output of the first hidden layer, and the configurations and operation parameters of the second hidden layer and the second output layer are determined.

3. 3. The inference processing device according to claim 2, wherein in the trained neural network, the number of layers of the first hidden layer is different from the number of layers of the second hidden layer.

4. the inference processing device performs inference based on the trained neural network to obtain the classification result, 4. The inference processing device according to claim 2, wherein the classification results include the first classification result and the second classification result.

5. The inference processing device comprises: performing a first inference based on the trained first neural network to obtain the first classification result; The inference processing device according to claim 4, wherein the second classification result is obtained by performing a second inference based on the second neural network that has already been trained.

6. The inference processing device comprises: If the first classification result satisfies the inference accuracy required for the application of the inference, determining that the first classification result is a candidate for termination and terminating the calculation; On the other hand, if the first classification result does not satisfy the inference accuracy required for the application, the inference processing device determines that the first classification result is not a candidate for termination and proceeds to the second inference.

7. Further, a reliability determination means for determining the reliability of the classification result is provided, 7. The inference processing device according to claim 5, wherein the reliability determining means determines whether to terminate the calculation at the first classification result or to continue the calculation until the second classification result is obtained, depending on the reliability of the first classification result determined by the reliability determining means.

8. The inference processing device comprises: If the reliability of the first classification result is high, it is determined that the reliability required for the purpose of the inference is achieved, and further, it is determined whether or not the first classification result is a candidate for termination, and if it is determined that the first classification result is a candidate for termination, it terminates the calculation, and on the other hand, if it is determined that the first classification result is not a candidate for termination, it proceeds to the second inference, 8. The inference processing device according to claim 7, wherein when the reliability of the first classification result is low, it is determined that the reliability required for the intended use is not achieved, and the device proceeds to the second inference.

9. An input layer that inputs an input image; a first intermediate layer that extracts features included in the input image; a first output layer that outputs a first classification result based on the feature amount extracted by the first intermediate layer; a second intermediate layer that receives the output of the first intermediate layer as an input and extracts features; a second output layer that outputs a second classification result based on the feature extracted by the second intermediate layer, and a second output layer that outputs a second classification result based on the feature extracted by the second intermediate layer. a control step of controlling the imaging device to operate in any one of a plurality of imaging modes including a specific imaging mode for photographing a specific subject, the inference processing device receives an image obtained by capturing an image with the imaging device as an input image of the neural network; If the first classification result output by the inference processing device in the specific photographing mode is a classification result including the specific subject, performing calculations until the second classification result is obtained; An inference processing method characterized in that, if the first classification result output by the inference processing device in the specific shooting mode is not a classification result that includes the specific subject, the calculation by the inference processing device is terminated at the first classification result.

10. 9. A program for causing a computer to execute each means of the inference processing device according to claim 1.

Images