Inference apparatus

Abstract

To provide an inference device capable of obtaining a plurality of classification results having different abstraction degrees in a short time.SOLUTION: An inference device comprises a neural network and control means. The neural network comprises an input layer which inputs image data, N intermediate layers, and N output layers. A first intermediate layer extracts a first feature quantity from data output by the input layer. A M-th intermediate layer can extract a M-th feature quantity from a (M-1)th feature quantity output by a (M-1)th intermediate layer. A first output layer can classify an object included in the image at a first abstraction degree according to the first feature quantity output by the first intermediate layer. A M-th output layer can classify the object at a M-th abstraction degree lower than a (M-1)th abstraction degree according to the M-th feature quantity output by the M-th intermediate layer. The control means controls inference by the neural network.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an inference device, and more particularly to an inference device used in an imaging device. [Background technology]

[0002] Inference devices that perform inference using neural networks have been known in the past. In particular, so-called convolutional neural networks (CNNs) are often used in inference devices that perform image recognition.

[0003] In CNN, input image data is processed sequentially in the intermediate layer and the fully connected layer, resulting in a final inference result that recognizes the object contained in the image. In the intermediate layer, multiple feature extraction processing layers are connected hierarchically, and in each processing layer, convolution calculations, activation processing, and pooling processing are performed on the input data input from the previous layer. In the intermediate layer, the processing in each processing layer is repeated to extract high-dimensional features contained in the input image data. In the fully connected layer, the calculation results data from the intermediate layer are combined to obtain the final inference result.

[0004] The calculations of such neural networks are broadly composed of a training phase and an inference phase. In the training phase, a large amount of image data is used to extract trends and improve the accuracy of image recognition (learning), while in the inference phase, a process is carried out to derive answers based on the learning results for new, unknown image data (inference). For example, there are two types of layers with weights that are modified during CNN learning: convolutional layers and fully connected layers. These weights are parameter values ​​for calculations that are used as filter values ​​in convolutional layers and as weight coefficients or biases in fully connected layers.

[0005] Consider a case where an imaging device classifies subjects by performing inference using a neural network and performs autofocus (AF) based on the classification results. In this case, inference may be performed by switching the level of abstraction of the classification. For example, if you want to focus only on the eyes of a specific animal, you need to classify the subject in detail (specifically). On the other hand, if you want to focus on all animals regardless of species, you need to classify the subject roughly (abstractly) so that the focus is not biased towards a specific animal.

[0006] Patent Document 1 discloses that a recognition process is performed using a first CNN and a detailed recognition process is performed using a second CNN. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2019-96006 Summary of the Invention [Problem to be solved by the invention]

[0008] However, if the neural network is switched in response to a change in the level of abstraction of the classification, the inference calculation must be performed from the beginning each time the level of abstraction is changed, and it takes time to obtain the classification result of the new level of abstraction. Since continuity of operation is required in AF and the like, it is preferable to obtain the classification result of the desired level of abstraction in a short time.

[0009] The present invention aims to make it possible to obtain multiple classification results with different levels of abstraction in a short time. Let's say. [Means for solving the problem]

[0010] The inference device of the present invention has an input layer that inputs and outputs image data, N intermediate layers from a first intermediate layer to an Nth intermediate layer (N is an integer of 2 or more), and N output layers from a first output layer to an Nth output layer, wherein the first intermediate layer of the N intermediate layers extracts a first feature amount from the data output by the input layer and outputs the first feature amount, and an Mth intermediate layer of the N intermediate layers (M is an integer of 2 or more and N or less) extracts an Mth feature amount from the M-1th feature amount output by the M-1th intermediate layer and outputs the Mth feature amount. a first output layer among the N output layers is capable of classifying objects included in the image at a first level of abstraction in accordance with the first feature amount output by the first intermediate layer, and outputting a first classification result that is a result of the classification; and an Mth output layer among the N output layers is capable of classifying the objects at an Mth level of abstraction that is lower than an M-1th level of abstraction in accordance with the Mth feature amount output by the Mth intermediate layer, and outputting an Mth classification result that is a result of the classification. The control means controls to switch whether to continue the inference based on the Pth classification result (P is an integer of 1 or more and N-1 or less). do. [Effects of the Invention]

[0011] According to the present invention, it is possible to obtain a plurality of classification results with different levels of abstraction in a short period of time. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating an example of the configuration of an imaging device according to a first embodiment. [Figure 2] FIG. 1 is a schematic diagram illustrating an example of a trained model according to the first and second embodiments. [Figure 3] FIG. 2 is a schematic diagram illustrating an example of a learning model according to the first and second embodiments. [Figure 4] 4 is a flowchart illustrating an example of the operation of the imaging device according to the first embodiment. [Figure 5] 4 is a schematic diagram showing a specific example of the operation of the imaging device according to the first embodiment. FIG. [Figure 6]4 is a schematic diagram showing a specific example of the operation of the imaging device according to the first embodiment. FIG. [Figure 7] 4 is a schematic diagram showing a specific example of the operation of the imaging device according to the first embodiment. FIG. [Figure 8] FIG. 10 is a block diagram illustrating an example of the configuration of an imaging device according to a second embodiment. [Figure 9] FIG. 10 is a schematic diagram illustrating an example of a method for determining reliability according to the second embodiment. [Figure 10] 10 is a flowchart illustrating an example of the operation of the imaging device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Example 1 A first embodiment of the present invention will be described below.

[0014] (composition) 1 is a block diagram showing an example of the configuration of an image capturing apparatus 100 according to Example 1. In the image capturing apparatus 100, a CPU 101, a ROM 102, a memory 103, an image capturing unit 104, an input unit 105, a display unit 106, a calculation unit 107, and an abstraction level determination unit 108 are connected to a system bus 109. The units connected to the system bus 109 can exchange data with each other via the system bus 109.

[0015] The CPU 101 controls the imaging unit 104, the display unit 106, the calculation unit 107, and the abstraction level determination unit 108 in accordance with a program stored in the ROM 102. When a user performs an operation using the input unit 105, a control signal corresponding to the user operation is supplied from the input unit 105 to the CPU 101. The CPU 101 controls the imaging unit 104, the display unit 106, the calculation unit 107, and the abstraction level determination unit 108 in accordance with the control signal supplied from the input unit 105. This allows the CPU 101 to cause the imaging unit 104, the display unit 106, the calculation unit 107, and the abstraction level determination unit 108 to operate in accordance with the user operation.

[0016] The ROM 102 is a non-volatile storage medium that stores various programs and the like for the operation of the CPU 101. The storage medium that stores various programs and the like for the operation of the CPU 101 is not limited to the ROM, and may be, for example, a hard disk or the like.

[0017] The memory 103 is a volatile storage medium, such as a RAM. When the CPU 101 operates according to the program stored in the ROM 102, the memory 103 is used as a work memory.

[0018] The imaging unit 104 includes an imaging sensor unit configured with a lens, a shutter with an aperture function, and a CCD or CMOS element that converts an optical image into an electrical signal. The imaging unit 104 also includes an imaging control unit that performs various controls, such as exposure control and distance measurement control, based on signals from the imaging sensor unit, and can perform a series of controls related to imaging. The imaging unit 104 also includes an image processing unit that performs a series of image processes related to imaging. The imaging unit 104 can also perform imaging processing, such as storing signals from the imaging sensor unit as an image file in a storage medium (not shown), such as a memory card. The CPU 101 controls the imaging unit 104 in response to user operations using the input unit 105, allowing the imaging unit 104 to capture images in response to user operations.

[0019] The input unit 105 accepts a user operation, generates a control signal in response to the user operation, and supplies the control signal to the CPU 101. For example, the input unit 105 has physical operation buttons, a touch panel, etc. as input devices that accept the user operation. The touch panel has, for example, a planar input surface, and outputs coordinate information corresponding to the position on the input surface that the user touches.

[0020] The display unit 106 generates a display signal for displaying an image on the display and outputs it to the display. The display unit 106 may include a display. When a touch panel is used as the input unit 105, the input unit 105 and the display can be configured as an integrated unit. For example, the touch panel is configured so that its light transmittance does not interfere with the display on the display, and is attached to the upper layer of the display surface of the display. Then, input coordinates on the touch panel are associated with display coordinates on the display.

[0021] The calculation unit 107 performs inference using a neural network. For example, the calculation unit 107 acquires and analyzes images (still images or video images) captured by the imaging unit 104, and detects and classifies objects (subjects) contained in the images. Neural network calculations are broadly divided into a learning phase and an inference phase. For example, learning is performed on a cloud server (not shown), and inference is performed on the calculation unit 107 of the imaging device 100. Details of learning and inference will be described later. Note that learning may be performed in the imaging device 100, and inference may be performed on the cloud.

[0022] The arithmetic unit 107 is, for example, a graphics processing unit (hereinafter referred to as GPU). A GPU is a computing device capable of performing specialized processing for computer graphics calculations, and is generally capable of performing calculations required for neural networks, such as matrix calculations, in a shorter time than the CPU 101. The arithmetic unit 107 may be configured with one GPU or multiple GPUs. The arithmetic unit 107 need only have a configuration capable of performing neural network calculations, and does not have to be a GPU.

[0023] The abstraction level determination unit 108 is a circuit that determines one or more levels of abstraction when the calculation unit 107 classifies the object. For example, the CPU 101 controls the abstraction level determination unit 108 in response to a user operation using the input unit 105, so that the abstraction level determination unit 108 determines the abstraction level in response to the user operation. A classification with a low level of abstraction can be considered as a detailed (concrete) classification, and a classification with a high level of abstraction can be considered as a detailed (concrete) classification. A category can be thought of as a broad (abstract) classification. For example, "cats" and "dogs" are less abstract than "small animals" and "pets."

[0024] The method for determining the abstraction level is not limited to the above method. For example, a table indicating the relationship between the operation mode of the imaging device 100 (the purpose of the classification result) and the abstraction level of the classification may be stored in the ROM 102 in advance, and the abstraction level determination unit 108 may determine different abstraction levels depending on the operation mode of the imaging device 100. A classification result with a higher abstraction level can be obtained in a shorter time than a classification result with a lower abstraction level. Therefore, when the classification result is used for autofocus (AF), which requires continuous operation, the abstraction level determination unit 108 may determine one or more abstraction levels, including a high abstraction level, so that the classification result can be obtained in a shorter time. The abstraction level determination unit 108 may determine a high abstraction level (e.g., the second abstraction level) when the classification result is used as information (tag) to be added to a captured (recorded) image, and may determine a low abstraction level (e.g., the first abstraction level) when the classification result is used for AF without using it as a tag. The abstraction level determination unit 108 may determine different abstraction levels for continuous shooting or video shooting and for single shooting.

[0025] (Pre-trained model) 2(A) and 2(B) are schematic diagrams showing an example of a trained model 200 used by the calculation unit 107. The trained model 200 represents a neural network having an input layer 210, N intermediate layers 220 from the first intermediate layer to the Nth intermediate layer (N is an integer equal to or greater than 2), and N output layers 230 from the first output layer to the Nth output layer. FIG. 2(B) shows the case where N=2.

[0026] The input layer 210 receives and outputs image data (image data). A first intermediate layer 221 of the N intermediate layers 220 extracts a first feature amount from the image data output by the input layer 210 and outputs the first feature amount. A first output layer 231 of the N output layers 230 is capable of classifying objects included in an image (still image or video) at a first level of abstraction according to the first feature amount output by the first intermediate layer 221, and outputting a first classification result that is the result of the classification.

[0027] An Mth intermediate layer (M is an integer greater than or equal to 2 and less than or equal to N) among the N intermediate layers 220 can extract an Mth feature from the M-1th feature output by the M-1th intermediate layer, and output the Mth feature. For example, the second intermediate layer 222 can extract a second feature from the first feature output by the first intermediate layer 221, and output the second feature.

[0028] The Mth output layer of the N output layers 230 is capable of classifying objects at an Mth level of abstraction according to the Mth feature amount output by the Mth hidden layer, and outputting an Mth classification result that is the result of the classification. For example, the second output layer 232 is capable of classifying objects at a second level of abstraction according to the second feature amount output by the second hidden layer 222, and outputting a second classification result that is the result of the classification.

[0029] In a neural network, by repeating processing in each processing layer of the intermediate layer, features contained in input image data are extracted in a high dimension. Since the accuracy of classification depends on the number of intermediate layers, in Example 1, the Mth level of abstraction is set to be lower than the M-1th level of abstraction. For example, the first classification result is "small animal" or "pet," and the second classification result is "cat" or "dog," etc.

[0030] 2(A) and 2(B) show an example in which each hidden layer 220 performs neural network calculations using seven processing layers, but the number of processing layers in each hidden layer 220 may be more or less than seven. The number of processing layers may differ among multiple hidden layers 220. The number of processing layers in the intermediate layer 220 is determined, for example, so that the output layer 230 connected to the intermediate layer 220 can perform classification with a desired accuracy.

[0031] (Learning method) An example of a learning method for constructing a trained model 200 (neural network) will be described. Learning is performed in advance, for example, on a high-performance PC such as a cloud server. In learning, for example, multiple image data are each used as input data to the input layer, and the correct classification results for each image data are used as training data. The configurations and calculation parameters of the intermediate layer and output layer are then adjusted so that the classification results obtained by the output layer approach the correct classification results. Here, a learning method is described that enables five classifications at the first abstraction level and 20 classifications at the second abstraction level. Note that the number and types of classifications are not particularly limited and can be freely set during learning. If classification accuracy is not a concern, the number of classifications may be the same for the first and second abstraction levels.

[0032] First, a learning model 310 shown in Fig. 3(A) is prepared. The learning model 310 has an input layer 210, an intermediate layer 311, and an output layer 312. The input layer 210 inputs image data and outputs it to the intermediate layer 311. The intermediate layer 311 extracts features from the image data output by the input layer 210 and outputs the extracted features to the output layer 312. The output layer 312 classifies objects included in the image according to the features output by the intermediate layer 311 and outputs the classification results.

[0033] Then, by performing learning using the learning model 310, the configurations and calculation parameters of the intermediate layer 311 and the output layer 312 are determined. Here, it is assumed that by setting the number of processing layers in the intermediate layer 311 to seven, five classifications can be performed with the desired accuracy. If the desired accuracy cannot be achieved even when the number of processing layers in the intermediate layer 311 is set to seven, the number of processing layers in the intermediate layer 311 may be increased to eight, nine, etc., or the number of classifications in the output layer 312 may be reduced to four, three, etc. When learning using the learning model 310 is complete, the intermediate layer 311 can be used as the first intermediate layer 221 shown in FIGS. 2(A) and 2(B), and the output layer 312 can be used as the first output layer 231 shown in FIGS. 2(A) and 2(B).

[0034] Next, a learning model 320 shown in FIG. 3(B) is prepared. The learning model 320 has an input layer 210, a first hidden layer 221, a hidden layer 321, and an output layer 322. The input layer 210 inputs image data and outputs it to the first hidden layer 221. The first hidden layer 221 extracts features from the image data output by the input layer 210 and outputs the extracted features to the hidden layer 321. The hidden layer 321 extracts higher-dimensional features from the features output by the first hidden layer 221 and outputs the extracted features to the output layer 322. The output layer 322 classifies objects included in the image according to the features output by the hidden layer 321 and outputs the classification results.

[0035] Then, by performing learning using the learning model 320, the configurations and calculation parameters of the intermediate layer 321 and the output layer 322 are determined. Here, the calculation parameters of the first intermediate layer 221 are fixed. That is, in learning using the learning model 320, the configurations and calculation parameters of the intermediate layer 321 and the output layer 322 are adjusted, but the calculation parameters of the first intermediate layer 221 are not changed from the results of learning using the learning model 310. Here, it is assumed that by setting the number of processing layers in the intermediate layer 321 to 7 (the total number of processing layers including the first intermediate layer 221 and the intermediate layer 321 is 14), 20 classifications can be performed with the desired accuracy. If the desired accuracy cannot be achieved even when the number of processing layers in the intermediate layer 321 is set to 7, the number of processing layers in the intermediate layer 321 may be increased to 8, 9, etc., or the number of classifications in the output layer 322 may be reduced to 18, 15, 12, 10, etc. When learning using the learning model 320 is completed, the intermediate layer 321 is used as the second intermediate layer 222 shown in FIGS. 2(A) and 2(B). The output layer 322 can be used as the second output layer 232 shown in FIGS. 2(A) and 2(B).

[0036] If N = 2, learning is completed here. If N > 2, the same process as above is repeated while adding one intermediate layer at a time to the learning model until the parameters of the Nth intermediate layer and the Nth output layer are determined.

[0037] By performing such learning, it is possible to make the calculation parameters the same up to some point among multiple calculations for obtaining multiple classification results with different levels of abstraction. For example, the first hidden layer 221 is used in both the calculation for obtaining the first classification result (classification result with the first level of abstraction) and the calculation for obtaining the second classification result (classification result with the second level of abstraction). Thus, it is possible to obtain one trained model 200 that can obtain multiple classification results with different levels of abstraction.

[0038] (operation) An example of the operation of the imaging device 100 will be described. Here, an example in which the classification results are used for AF will be described. When focusing only on the eyes of a specific animal, it is necessary to classify the subject in detail (specifically). To perform detailed classification, it is necessary to extract high-dimensional features using many intermediate layers. However, increasing the number of intermediate layers increases the computational load of inference by the neural network, lengthening the inference time (the time until the classification results are obtained). A longer inference time delays AF, increasing the likelihood that the user will miss a photo opportunity. When the classification results are used for AF, it is preferable to obtain the classification results in a short time so as not to delay AF. Therefore, in the first embodiment, when obtaining a classification result with a desired level of abstraction, the CPU 101 controls the calculation unit 107 so that classification results with multiple levels of abstraction equal to or higher than the desired level are obtained sequentially. A classification result with a level of abstraction higher than the desired level can be obtained in a shorter time than a classification result with the desired level of abstraction. Furthermore, in the first embodiment, an increase in computation time due to switching between abstraction levels can be suppressed. For example, part of the calculations for obtaining a classification result with a high level of abstraction (a level of abstraction higher than the desired level) and part of the calculations for obtaining a classification result with the desired level of abstraction are common. Therefore, after obtaining a classification result with a high level of abstraction, it is not necessary to perform the calculations for obtaining a classification result with the desired level of abstraction from the beginning, and the classification result with the desired level of abstraction can be obtained in a short time.

[0039] Fig. 4 is a flowchart showing an example of the operation of the imaging device 100. For example, when a user operates the input unit 105 to instruct the execution of AF, the CPU 101 starts the operation of Fig. 4 from the processing of step S401. In the operation of Fig. 4, the CPU 101 controls the calculation unit 107 to classify objects included in an unknown image, and controls the imaging unit 104 to perform AF based on the classification result. The CPU 101 may repeatedly perform the operation of Fig. 4.

[0040] In step S401, the CPU 101 determines whether the focus setting of the imaging device 100 is manual focus (MF) or autofocus (AF). If the CPU 101 determines that the focus setting is MF, inference is not necessary, and the CPU 101 ends the operation of FIG. 4. If the CPU 101 determines that the focus setting is AF, the process proceeds to step S402. The CPU 101 can switch the focus setting in response to a user operation using the input unit 105.

[0041] In step S402, the CPU 101 selects one abstraction level equal to or higher than the abstraction level (target abstraction level) determined by the abstraction level determination unit 108 from among the abstraction levels of classification that can be performed by the calculation unit 107. For example, the CPU 101 repeats the process of step S402, selecting abstraction levels one by one from the highest abstraction level to the target abstraction level. First, the first abstraction level is selected, and then the second abstraction level is selected. After the CPU 101 selects one abstraction level, the process returns to step S402. Proceed to step S403.

[0042] In step S403, CPU 101 controls calculation unit 107 to obtain a classification result for the abstraction level selected in step S402. For example, if the abstraction level selected in step S402 is the first extraction level, CPU 101 controls calculation unit 107 to perform calculations up to the first hidden layer 221 and the first output layer 231, thereby obtaining a first classification result. Similarly, if the abstraction level selected in step S402 is the second extraction level, CPU 101 controls calculation unit 107 to perform calculations up to the second hidden layer 222 and the second output layer 232, thereby obtaining a second classification result. After CPU 101 obtains a classification result for the abstraction level selected in step S402, the process proceeds to step S404.

[0043] In step S404, CPU 101 controls image capture unit 104 to perform AF based on the classification result obtained in step S403. For example, CPU 101 controls image capture unit 104 to repeatedly perform the process of step S404 and repeatedly perform AF while sequentially switching the classification result to be used from the first classification result onwards. After CPU 101 controls image capture unit 104 to perform AF based on the classification result obtained in step S403, the process proceeds to step S405.

[0044] In step S405, CPU 101 determines whether to continue the inference. If CPU 101 determines to continue the inference, the process proceeds to step S402, where the next level of abstraction is selected. If CPU 101 determines to suspend (not continue) the inference, CPU 101 ends the operation of FIG. 4. For example, CPU 101 determines to suspend the inference when the selected level of abstraction is the target level of abstraction. CPU 101 may determine to suspend the inference when the classification result obtained in step S403 (Pth classification result (P is an integer greater than or equal to 1 and less than or equal to N-1)) is a specific classification result. CPU 101 may use, as the specific classification result, a classification result that differs depending on the operation mode of imaging device 100, or may use a classification result that is independent of the operation mode. CPU 101 may determine to suspend the inference when the classification result obtained in step S403 (Pth classification result) is the same as the previous inference. In this case, the CPU 101 may also consider the Qth classification result (Q is an integer greater than or equal to P+1 and less than or equal to N) to be the same as the previous inference. By doing so, the Qth classification result can be obtained in a shorter time. The previous inference is an inference performed by inputting the image captured immediately before to the calculation unit 107.

[0045] Although the example has been described in which the abstraction level determination unit 108 determines one target abstraction level and the CPU 101 controls the calculation unit 107 so as to obtain classification results of one or more abstraction levels equal to or higher than the target abstraction level, the present invention is not limited to this. For example, the abstraction level determination unit 108 may determine one or more abstraction levels, and the CPU 101 may control the calculation unit 107 so as to obtain only classification results of the abstraction levels determined by the abstraction level determination unit 108. In this case, in step S402, the CPU 101 selects one of the abstraction levels determined by the abstraction level determination unit 108. For example, the CPU 101 repeats the process of step S402, and selects the abstraction levels determined by the abstraction level determination unit 108 one by one, starting from the highest abstraction level.

[0046] A specific example of the operation of the imaging device 100 will be described using Figures 5(A) to 5(C). Figures 5(A) to 5(C) are schematic diagrams showing an example of how the AF target changes depending on the change in the classification result. Here, it is assumed that five classifications are possible at the first abstraction level, 20 classifications are possible at the second abstraction level, and 50 classifications are possible at the third abstraction level. It is also assumed that the first abstraction level is high, and one of the five classifications at the first abstraction level is "person." It is also assumed that the second abstraction level is medium (second abstraction level<first abstraction level), and one of the 20 classifications at the second abstraction level is "face." It is also assumed that the third abstraction level is low (third abstraction level<second abstraction level), and one of the 50 classifications at the third abstraction level is "eyes." Then, the abstraction level determination unit 108 sets the third abstraction level as the target abstraction level. Let us assume that the degree of

[0047] First, in step S402 of Fig. 4, CPU 101 selects a first level of abstraction. In step S403, CPU 101 obtains "person" as the first classification result (classification result at the first level of abstraction). Then, in step S404, CPU 101 focuses on person 501, as shown in Fig. 5(A). Thereafter, in step S405, CPU 101 determines to continue inference, and the process proceeds to step S402.

[0048] Next, in step S402, CPU 101 selects a second level of abstraction. In step S403, CPU 101 obtains "face" as the second classification result (classification result at the second level of abstraction). Then, in step S404, CPU 101 focuses on face 502 of person 501, as shown in FIG. 5(B). Thereafter, in step S405, CPU 101 determines to continue inference, and the process proceeds to step S402.

[0049] Next, in step S402, CPU 101 selects a third level of abstraction. In step S403, CPU 101 obtains "eyes" as a third classification result (classification result of the third level of abstraction). Then, in step S404, CPU 101 focuses on eyes 503 of face 502, as shown in FIG. 5(C). Thereafter, in step S405, CPU 101 determines to interrupt inference, and the operation in FIG. 4 ends.

[0050] After obtaining "eyes" as the third classification result, it becomes possible to focus on the eyes when taking a photo. Furthermore, the timing for starting AF (such as mechanical AF drive) can be advanced, allowing you to take photos without missing a photo opportunity. Specifically, by focusing on a person when the first classification result is obtained and on a face when the second classification result is obtained, it becomes possible to take photos without missing a photo opportunity even if the third classification result has not been obtained.

[0051] Another specific example of the operation of the imaging device 100 will be described with reference to FIG. 6. FIG. 6 is a schematic diagram (table) showing an example of the relationship between the shooting mode, the first classification result (the classification result at the first level of abstraction), and whether to continue inference (whether to perform classification at the second level of abstraction). A table showing the relationship between the operation mode, the classification result, and the continuation / interruption of inference as shown in FIG. 6 is stored in advance in the ROM 102, for example. Such a table may be fixed or may be freely changeable by the user. The portrait shooting mode is a mode for shooting people specifically, the landscape shooting mode is a mode for shooting landscapes specifically, and the food shooting mode is a mode for shooting food specifically. The CPU 101 can switch the shooting mode in response to a user operation using the input unit 105, or can switch the shooting mode automatically.

[0052] When photographing in portrait mode, if the first classification result is "person," the imaging device 100 focuses on the part classified as "person" and continues inference. If the first classification result is not "person" (if the first classification result is "animal," "plant," "sky," or "food"), the imaging device 100 focuses on the part classified as "animal," "plant," "sky," or "food" and suspends inference. If the first classification result is not "person," the imaging device 100 may suspend inference without performing AF.

[0053] When photographing in landscape photography mode, if the first classification result is "plant" or "sky," the imaging device 100 focuses on the part classified as "plant" or "sky" and continues inference. If the first classification result is neither "plant" nor "sky" (if the first classification result is "person," "animal," or "food"), the imaging device 100 focuses on the part classified as "person," "animal," or "food" and suspends inference. If the first classification result is neither "plant" nor "sky," the image capturing apparatus 100 may discontinue inference without performing AF.

[0054] When photographing in food photography mode, if the first classification result is "food," imaging device 100 focuses on the part classified as "food" and continues inference. If the first classification result is not "food" (if the first classification result is "person," "animal," "plant," or "sky"), imaging device 100 focuses on the part classified as "person," "animal," "plant," or "sky" and suspends inference. If the first classification result is not "food," imaging device 100 may suspend inference without performing AF.

[0055] Note that the processing (predetermined processing) performed based on the classification results is not limited to AF. The CPU 101 can switch the shutter speed setting between auto and manual in response to a user operation using the input unit 105. For example, when the shutter speed setting is auto, the CPU 101 may automatically change the shutter speed based on the classification results. FIG. 7 is a schematic diagram (table) showing an example of the relationship between the first classification result (classification result of the first abstraction level), the second classification result (classification result of the second abstraction level), the third classification result (classification result of the third abstraction level), and the processing based on the classification results. The table showing the relationship between the classification results and the processing, as shown in FIG. 7, is stored in advance in the ROM 102, for example. Such a table may be fixed or may be freely changeable by the user.

[0056] Here, it is assumed that five classifications are possible at the first abstraction level, 20 classifications are possible at the second abstraction level, and 50 classifications are possible at the third abstraction level. It is also assumed that the first abstraction level is high, and one of the five classifications at the first abstraction level is "animal." It is also assumed that the second abstraction level is medium (second abstraction level<first abstraction level), and the 20 classifications at the second abstraction level include "dog," "bird," and "turtle." It is also assumed that the third abstraction level is low (third abstraction level<second abstraction level), and the 50 classifications at the third abstraction level include "Chihuahua," "Doberman," "swimming turtle," and "land turtle." It is also assumed that the abstraction level determination unit 108 determines the third abstraction level as the target abstraction level.

[0057] Furthermore, because the specific value of shutter speed changes depending on conditions such as external light and aperture, we will use abstract (relative) terms such as fast, medium, and slow shutter speeds here. Fast, medium, and slow refer to differences in shutter speeds under the same shooting conditions. For example, under the same shooting conditions, a fast shutter speed is faster than a medium shutter speed, which is faster than a slow shutter speed.

[0058] If the first classification result is "animal", the CPU 101 controls the shutter speed to a faster value, and then controls the CPU 101 to continue inference (to perform classification at the second abstraction level).

[0059] If the second classification result is "dog", CPU 101 controls the shutter speed to an intermediate value. If the second classification result is "bird", CPU 101 controls the shutter speed to maintain a fast value. If the second classification result is "turtle", CPU 101 controls the shutter speed to a slow value. If the second classification result is "bird", CPU 101 controls the inference to be interrupted, and if the second classification result is "dog" or "turtle", CPU 101 controls the inference to continue (perform classification at a third level of abstraction).

[0060] If the second classification result is "dog", the following control is performed based on the third classification result. If the third classification result is "Chihuahua", the CPU 101 controls to maintain an intermediate shutter speed. If the third classification result is "Doberman", the CPU 101 Controls the shutter speed to a fast value.

[0061] If the second classification result is "turtle," the following control is performed based on the third classification result. If the third classification result is "swimming turtle," CPU 101 controls the shutter speed to an intermediate value. If the third classification result is "land turtle," CPU 101 controls to maintain a slow shutter speed.

[0062] In this way, by using the classification results, the shutter speed can be automatically changed to a value appropriate for the subject. Furthermore, the timing for starting shutter speed control can be advanced, making it possible to take pictures without missing a photo opportunity. For example, by controlling the shutter speed to a value appropriate for animals when the first classification result is obtained, it becomes possible to take pictures without missing a photo opportunity even when the second or third classification result has not yet been obtained.

[0063] Here, an example has been described in which the first classification result is "animal," but CPU 101 may control the shutter speed in a similar manner even when the first classification result is not "animal." Also, an example has been described in which CPU 101 controls the shutter speed to a slow speed when the second classification result is "turtle," but this is not limiting. For example, if it is desired to minimize image blur before inference is completed, CPU 101 may control the shutter speed to an intermediate speed or the like when the second classification result is "turtle." CPU 101 may control the shutter speed so as to maintain or slow the shutter speed in response to a decrease in the level of abstraction of the classification result (not to increase the shutter speed in response to a decrease in the level of abstraction of the classification result).

[0064] Based on the classification result, automatic white balance, exposure, and the like may be performed. Multiple functions of the imaging device 100 (such as AF, automatic shutter speed control, and automatic white balance) may be executed based on the classification result. In this case, the abstraction level determination unit 108 determines the abstraction level taking the multiple functions into consideration. For example, the abstraction level determination unit 108 individually determines the abstraction level required for each function. The CPU 101 then controls the calculation unit 107 to obtain a classification result with the abstraction level determined by the abstraction level determination unit 108. Each time the CPU 101 obtains a classification result, it executes one of the multiple functions that requires the classification result. The CPU 101 then individually determines whether to suspend inference for each function, and controls the calculation unit 107 to suspend inference if it determines to suspend inference for all functions. When the CPU 101 continues inference and obtains a new classification result, it may not execute the function for which it has determined to suspend inference. The abstraction level determination unit 108 may individually determine one target abstraction level for each function. In this case, for example, CPU 101 controls calculation unit 107 so as to obtain classification results of one or more levels of abstraction that include all target abstraction levels and are equal to or greater than the maximum target abstraction level. Each time CPU 101 obtains a classification result, it executes, from among the multiple functions, a function whose classification result is equal to or less than the target abstraction level. CPU 101 then determines whether to suspend inference for each function, and controls calculation unit 107 to suspend inference if it determines to suspend inference for all functions.

[0065] (summary) As described above, according to the first embodiment, it is possible to obtain a plurality of classification results with different levels of abstraction in a short time. For example, part of the calculations for obtaining a classification result with a high level of abstraction (a level of abstraction higher than a desired level of abstraction) and part of the calculations for obtaining a classification result with a desired level of abstraction are common. Therefore, after obtaining a classification result with a high level of abstraction, it is not necessary to perform the calculations for obtaining a classification result with the desired level of abstraction from the beginning, and the classification result with the desired level of abstraction can be obtained in a short time. Furthermore, by interrupting inference in response to a classification result with a high level of abstraction, it is possible to reduce the processing load, processing time, etc., caused by unnecessary inference.

[0066] Furthermore, when performing a predetermined process (such as AF) based on the classification result, by performing the predetermined process when a classification result with a high level of abstraction is obtained, it becomes possible to take a photograph without missing a photo opportunity even if a classification result with the desired level of abstraction is not obtained.

[0067] Thus, according to Example 1, an inference device can be provided that can perform inference with reduced impact on the continuity of behavior based on classification results using a single neural network (trained model) that can obtain multiple classification results with different levels of abstraction.

[0068] <Example 2> A second embodiment of the present invention will be described below. Differences from the first embodiment (configuration, processing, etc.) will be described in detail, and explanations of similarities to the first embodiment will be omitted as appropriate. In the second embodiment, whether to continue inference is determined based on the reliability of the classification result.

[0069] (composition) 8 is a block diagram showing an example of the configuration of an imaging device 800 according to Example 2. In the imaging device 800, a CPU 101, a ROM 102, a memory 103, an imaging unit 104, an input unit 105, a display unit 106, a calculation unit 107, an abstraction level determination unit 108, and a reliability determination unit 801 are connected to a system bus 109. The units connected to the system bus 109 can exchange data with each other via the system bus 109.

[0070] The reliability determination unit 801 is a circuit that determines the reliability of the classification result obtained by the calculation unit 107. Inference using a neural network generally classifies an object by inferring which of multiple classifications (clusters) the object most closely corresponds to. Consider a case where unknown image data is input to an inference device capable of seven classifications: "person," "animal," "plant," "sky," "food," "other," and "not applicable." In this case, the calculation results obtained are "person: 90%," "animal: 6%," "plant: 2%," "sky: 1%," "food: 1%," "other: 0%," and "not applicable," totaling 100%. The values ​​indicated by "%" indicate the probability that the object corresponds to the corresponding classification. If the above calculation result is obtained, it is estimated that the probability that the object is a person is 90%. The reliability determination unit 801 determines the reliability of the classification result based on, for example, such a calculation result. If the above calculation result is obtained, the probability that the object is a person (90%) is the highest, and the calculation unit 107 determines that the object is a person. Then, the reliability determination unit 801 determines the reliability of the classification result that the object is a person based on the possibility (90%) that the object is a person. For example, if the possibility (%) of the classification result is equal to or greater than a predetermined threshold, the reliability determination unit 801 determines that the reliability of the classification result is high, and if the possibility (%) of the classification result is less than the predetermined threshold, the reliability determination unit 801 determines that the reliability of the classification result is low.

[0071] The accuracy rate of the calculation unit 107 depends on the classification (cluster), e.g., the accuracy rate for people and animals is high and the accuracy rate for sky is low. Therefore, the reliability determination unit 801 may use different values ​​as the above-mentioned predetermined threshold depending on the classification result. FIG. 9 is a schematic diagram (table) showing an example of the relationship between the first classification result, the possibility (%) of the first classification result, and the reliability of the first classification result. A table showing the relationship between the classification result, the possibility (%) of the classification result, and the classification result, as shown in FIG. 9, is stored in advance in the ROM 102, for example. Such a table may be fixed or may be freely changeable by the user.

[0072] In the example shown in FIG. 9, if the first classification result is "person," "animal," or "plant," and the possibility of the first classification result is 90% or more, the reliability determination unit 801 determines that the reliability of the first classification result is high. If the possibility of the first classification result is less than 70%, the reliability determination unit 801 determines that the reliability of the first classification result is low. If the possibility of the first classification result is 70% or more but less than 90%, the reliability determination unit 801 determines that the reliability of the first classification result is medium. If the first classification result is "sky", and the possibility of the first classification result is 60% or more, the reliability determination unit 801 determines that the reliability of the first classification result is high. If the possibility of the first classification result is less than 40%, the reliability determination unit 801 determines that the reliability of the first classification result is low. If the possibility of the first classification result is 40% or more but less than 60%, the reliability determination unit 801 determines that the reliability of the first classification result is medium. If the first classification result is "food", and the possibility of the first classification result is 70% or more, the reliability determination unit 801 determines that the reliability of the first classification result is high. If the possibility of the first classification result is less than 30%, the reliability determination unit 801 determines that the reliability of the first classification result is low. When the possibility of the first classification result is equal to or greater than 30% and less than 70%, the reliability determination unit 801 determines that the reliability of the first classification result is medium.

[0073] Furthermore, the method of determining the reliability is not limited to the above method. For example, if a plurality of classification results obtained by inputting a plurality of images (a predetermined number of images) captured within a predetermined period up to the present into the calculation unit 107 are the same, the reliability determination unit 801 may determine that the reliability of the current classification result is high. If a plurality of classification results obtained by inputting a plurality of images (a predetermined number of images) captured within a predetermined period up to the present into the calculation unit 107 are non-uniform, the reliability determination unit 801 may determine that the reliability of the current classification result is low.

[0074] As explained in the first embodiment, the accuracy of classification depends on the number of intermediate layers. In the first embodiment, learning is performed so that classification can be performed with a desired accuracy. However, depending on the image, it may be difficult to classify the object. In such a case, the object may have the same probability of falling into multiple categories, such as "person: 45%" and "animal: 43%," and more intermediate layers are required. Therefore, in the second embodiment, the CPU 101 controls the calculation unit 107 to continue inference when it is determined that the reliability of the classification result is low.

[0075] (operation) Fig. 10 is a flowchart showing an example of the operation of the imaging device 100. For example, when a user instructs the execution of AF by operating the input unit 105, the CPU 101 starts the operation of Fig. 10 from the processing of step S1001. In the operation of Fig. 10, the CPU 101 controls the calculation unit 107 to classify objects included in an unknown image, and controls the imaging unit 104 to perform AF based on the classification result. The CPU 101 may repeatedly perform the operation of Fig. 10.

[0076] The processing in steps S1001 to S1004 is the same as the processing in steps S401 to S404 in Fig. 4. After step S1004, the processing proceeds to step S1005.

[0077] In step S1005, CPU 101 controls reliability determination unit 801 to determine whether the reliability of the classification result obtained in step S1003 is high. If CPU 101 determines that the reliability is high, the process proceeds to step S1006. If CPU 101 determines that the reliability is low, the process proceeds to step S1002, where inference continues (the next abstraction level is selected). However, if the selected abstraction level is the target abstraction level, the process proceeds to step S1006.

[0078] If CPU 101 determines that the reliability is medium, the process may proceed to either step S1002 or step S1006. For example, the process proceeds to step S1002 whether CPU 101 determines that the reliability is medium or low. If CPU 101 determines that the reliability is low, CPU 101 performs control to issue a predetermined notification (such as a warning display) to the user.

[0079] The process in step S1006 is the same as the process in step S405 in FIG.

[0080] Although the example in which CPU 101 determines the reliability each time it obtains a classification result has been described, this is not limiting. For example, CPU 101 may determine the reliability only for a specific classification result, such as the first classification result.

[0081] (summary) As described above, according to the second embodiment, whether to continue inference is determined based on the reliability of the classification result. This allows inference to be continued until a highly reliable classification result is obtained, thereby eliminating unnecessary inference with a high degree of accuracy.

[0082] It should be noted that Examples 1 and 2 are merely examples, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of Examples 1 and 2 within the scope of the gist of the present invention, and configurations obtained by appropriately combining the configurations of Examples 1 and 2. For example, although an example in which the present invention is applied to an imaging device has been described, the present invention can be applied to various inference devices that can be controlled to perform inference using a neural network, for example.

[0083] <Other Examples> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0084] 100, 800: Imaging device 101: CPU 107: Calculation unit 200: Trained model 210: Input layer 220: Intermediate layer 221: First intermediate layer 222: Second intermediate layer 230: Output layer 231: First output layer 232: Second output layer 310: Learning model 311: Hidden layer 312: Output layer 320: Learning model 321: Intermediate layer 322: Output layer

Claims

1. An input layer that inputs and outputs image data; N intermediate layers from a first intermediate layer to an Nth intermediate layer (N is an integer of 2 or more); N output layers from the first output layer to the Nth output layer; and the first hidden layer among the N hidden layers extracts a first feature from the data output by the input layer and outputs the first feature; an M-th intermediate layer (M is an integer of 2 or more and N or less) among the N intermediate layers is capable of extracting an M-th feature amount from the M-1-th feature amount output by the M-1-th intermediate layer, and outputting the M-th feature amount; a first output layer among the N output layers is capable of classifying objects included in the image at a first level of abstraction in accordance with the first feature amount output by the first intermediate layer, and outputting a first classification result that is a result of the classification; The Mth output layer of the N output layers is capable of classifying the object at an Mth level of abstraction, which is lower than the M-1th level of abstraction, according to the Mth feature amounts output by the Mth intermediate layers, and outputting an Mth classification result that is the result of the classification. a control means for controlling the neural network to perform inference; The control means controls to switch whether or not to continue the inference based on the Pth classification result (P is an integer greater than or equal to 1 and less than or equal to N-1). An inference device characterized by:

2. further comprising a determining means for determining one or more levels of abstraction; The control means controls the determination means to obtain the classification result of the abstraction level determined by the determination means.

2. The inference device according to claim 1.

3. the determining means determines one level of abstraction; The control means performs control so as to obtain classification results of one or more abstraction levels equal to or higher than the abstraction level determined by the determination means.

3. The inference device according to claim 2.

4. The determining means determines different abstraction levels depending on the use of the classification result obtained by the inference.

4. The inference device according to claim 2 or 3.

5. The control means controls the inference so as to be interrupted when the classification result of the P is a specific classification result.

5. The inference device according to claim 1, wherein:

6. The control means uses different classification results as the specific classification results depending on the operation mode of the inference device.

6. The inference device according to claim 5.

7. The control means controls the inference so as to interrupt the inference if the classification result of the P is the same as that of the previous inference.

7. The inference device according to claim 1, wherein:

8. If the classification result of P is the same as that of the previous inference, the control means considers the classification result of Q (Q is an integer greater than or equal to P+1 and less than or equal to N) to be the same as that of the previous inference.

8. The inference device according to claim 7.

9. The control means controls to switch whether or not to continue the inference based on the reliability of the classification result of the P.

9. The inference device according to claim 1, wherein:

10. The control means controls the inference to continue when it is determined that the reliability of the classification result of the P is low.

10. The inference device according to claim 9.

11. further comprising a processing means for executing a predetermined process based on the classification result obtained by the inference; The processing means repeatedly executes the predetermined process while sequentially switching the classification result to be used from the first classification result. The inference device according to any one of claims 1 to 10.

12. acquiring image data; an input layer that inputs and outputs the image data; N intermediate layers from a first intermediate layer to an Nth intermediate layer (N is an integer of 2 or more); N output layers from the first output layer to the Nth output layer; and the first hidden layer among the N hidden layers extracts a first feature from the data output by the input layer and outputs the first feature; an M-th intermediate layer (M is an integer of 2 or more and N or less) among the N intermediate layers is capable of extracting an M-th feature amount from the M-1-th feature amount output by the M-1-th intermediate layer, and outputting the M-th feature amount; a first output layer among the N output layers is capable of classifying objects included in the image at a first level of abstraction in accordance with the first feature amount output by the first intermediate layer, and outputting a first classification result that is a result of the classification; The Mth output layer of the N output layers is capable of classifying the object at an Mth level of abstraction, which is lower than the M-1th level of abstraction, according to the Mth feature amounts output by the Mth intermediate layers, and outputting an Mth classification result that is the result of the classification. a step of controlling inference by a neural network; a step of controlling whether or not to continue the inference based on the Pth classification result (P is an integer of 1 or more and N-1 or less); have An inference method characterized by:

13. A program for causing a computer to function as each means of the inference device according to any one of claims 1 to 11.

Images