Information processing device and control method for information processing device

Abstract

To provide an information processing apparatus which can reduce a calculation amount while having less influence on inference accuracy, in processing of identifying an object in a moving image.SOLUTION: The information processing apparatus includes: acquisition means which acquires continuous frames of a moving image; and identification means which identifies an object from each of the frames of the moving image by using a calculator including a plurality of hierarchies, in which a degree of influence being a product sum of outputs from individual nodes included in a first hierarchy to a prescribed node in a second hierarchy and weights respectively corresponding to the outputs is calculated and is inputted to the prescribed node. On the basis of a first degree of influence inputted to the prescribed node for a first frame and a partial calculation result of calculation of a second degree of influence inputted to the prescribed node for a second frame after the first frame, the identification means determines whether or not calculation of the second degree of influence is interrupted.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to an information processing apparatus and a control method for the information processing apparatus. [Background technology]

[0002] In recent years, research into deep learning and inference and its application to the real world have progressed, leading to the use of large-scale neural networks. Multi-class classifiers capable of identifying around 1,000 types of objects have also been realized.

[0003] For example, a multi-class classifier that can distinguish about 1,000 types of objects calculates the probability that each object exists in an image, but in the real world, the number of types of objects that can exist in a given image at one time is generally limited. As the number of types of objects that can be distinguished increases, the ratio of the calculation volume for objects that do not exist in the image to the overall calculation volume of the multi-class classifier increases, and the efficiency of identifying objects in the image decreases.

[0004] As a method for reducing the amount of calculation, for example, Patent Document 1 discloses a method for distinguishing between useful and useless information in a later layer of a neural network and replacing the feature quantities or weights of parts that are determined to be useless with zero. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-200648 [Non-patent literature]

[0006] [Non-Patent Document 1] S. Haykin, “Neural Networks A Comprehensive Foundation 2nd Edition”, Prentice Hall, pp.156-255, July 1998 Summary of the Invention [Problem to be solved by the invention]

[0007] The technology described in Patent Document 1 does not take into account correlation in the time direction, and therefore is not expected to be effective in reducing the amount of calculations required for time-sequential images such as videos. Furthermore, even if the amount of calculations is reduced in the latter stages of the neural network, when a multi-class classifier is used, it is not expected to be effective in suppressing the increase in the amount of calculations that accompanies an increase in the number of object types.

[0008] Therefore, an object of the present invention is to provide an information processing device that can reduce the amount of calculation while minimizing the impact on inference accuracy in the process of identifying objects in moving images. [Means for solving the problem]

[0009] The information processing device of the present invention comprises: an acquisition means for acquiring successive frames of a video; an identification means for identifying an object from each frame of the video using a computing unit including a plurality of layers, the computing unit calculating an influence degree which is the sum of the product of an output from each node included in a first layer to a predetermined node in a second layer and a weight corresponding to each of the outputs, and inputting the calculated influence degree to the predetermined node; The discrimination means determines whether or not to suspend the calculation of the second influence based on a first influence input to the predetermined node in a first frame and a partial calculation result of a second influence input to the predetermined node in a second frame subsequent to the first frame. judge It is characterized by: [Effects of the Invention]

[0010] According to the present invention, in the process of identifying objects in moving images, it is possible to reduce the amount of calculation while minimizing the impact on inference accuracy. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of an information processing device. [Figure 2] FIG. 10 is a diagram illustrating an example of the structure of a neural network that identifies object classes. [Figure 3] FIG. 1 is a diagram illustrating a feature detection process and a feature integration process of a CNN. [Figure 4] FIG. 1 is a schematic diagram of a neural network model structure. [Figure 5] 10 is a flowchart illustrating an example of an inference process using an inference model. [Figure 6] 10 is a flowchart illustrating an inference calculation for a first frame image. [Figure 7] A diagram explaining the computing units used in the nodes of the inference model. [Figure 8] FIG. 2 is a diagram illustrating a configuration of a product-sum calculator; [Figure 9] 10 is a graph showing an example of a result of object class identification. [Figure 10] 10 is a flowchart illustrating an inference calculation for a frame other than the first frame. [Figure 11] 10 is a graph showing an example of changes in influence degree in Modification 1. DETAILED DESCRIPTION OF THE INVENTION

[0012] <Embodiment> Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In this embodiment, an information processing device reduces the amount of unnecessary calculations when a neural network model is used for multi-class classification.

[0013] 1 is a block diagram illustrating an example of the configuration of an information processing device 100. The information processing device 100 includes a CPU 101, a ROM 102, a memory 103, an image acquisition unit 104, and a recognition unit 105. Each component included in the information processing device 100 is connected to a system bus 110, and can exchange data with each other via the system bus 110.

[0014] The CPU 101 controls the image acquisition unit 104 and the recognition unit 105 by reading out a program stored in the ROM 102 into the memory 103 and executing the program.

[0015] The ROM 102 stores various programs for the operation of the CPU 101. The various programs for the operation of the CPU 101 are not limited to being stored in the ROM 102, but may also be stored in, for example, a hard disk.

[0016] The memory 103 is, for example, a RAM, and is used as a work memory for executing various programs by the CPU 101. The CPU 101 can read out programs stored in the ROM 102 into the memory 103 and execute them.

[0017] The image acquisition unit 104 acquires successive frame images of a moving image from which an object (subject) is to be identified. The image acquisition unit 104 can acquire the moving image, for example, from an imaging unit (not shown) included in the information processing device 100. Alternatively, the image acquisition unit 104 may acquire the moving image from an imaging device connected to the information processing device 100.

[0018] The identification unit 105 identifies (detects) an object from the frame image acquired by the image acquisition unit 104 using an inference model (trained model) of a neural network. The identification unit 105 identifies (detects) an object from the frame image acquired by the image acquisition unit 104 using an inference model (trained model) of a neural network based on the calculation result for the previous frame and a part of the calculation result for the current frame. It is determined whether or not to suspend the calculation for the current frame. By not executing the calculation after the suspension, the amount of calculation for identifying the object is reduced.

[0019] The basic configuration of a neural network inference model will be explained using a Convolutional Neural Network (CNN) as an example with reference to Figures 2 and 3. Figure 2 shows the basic configuration of a CNN that identifies object classes from an input image, which is two-dimensional image data.

[0020] CNN includes multiple layers, each of which includes two layers called the feature detection layer (S layer) and the feature integration layer (C layer). In the example of Figure 2, the input image input to the CNN is processed in order from the first layer to the Xth layer.

[0021] In CNN, first, in layer S, features of the input image are detected based on the features detected in the previous layer. Next, the features detected in layer S are integrated in layer C, and the detection results of the current layer are input to the next layer.

[0022] The S layer contains multiple feature detection cell planes, each of which detects a different feature. The C layer contains multiple feature integration cell planes, and pools the detection results of the feature detection cell planes in the S layer. In the example of Figure 2, the final layer, the output layer (Xth layer), is composed of the S layer without using the C layer. The feature detection cell planes and feature integration cell planes are collectively referred to as cell planes.

[0023] The feature detection process at the feature detection cell plane and the feature integration process at the feature integration cell plane will be described in detail with reference to Figure 3. In Figure 3, rectangles represent cell planes. The Lth layer S includes multiple feature detection cell planes, and the L-1th layer C and the Lth layer C each include multiple feature integration cell planes.

[0024] The feature detection cell surface is composed of multiple feature detection neurons, which are connected to the C layer of the previous layer in a predetermined structure. The feature integration cell surface is composed of multiple feature integration neurons, which are connected to the S layer of the same layer in a predetermined structure.

[0025] In the m-th cell plane of the S layer of the Lth hierarchy, the output value of the feature detection neuron at position (ξ,ζ) is expressed as y m LS (ξ,ζ), the feature of the position (ξ,ζ) in the mth cell plane of the C layer of the Lth layer The output value of the integrating neuron is y m LC (ξ,ζ). The connection coefficients of each neuron The number w mLS (n,u,v), w m LC Assuming (u,v), each output value can be expressed as follows:

number

[0026] In Equation 1, f is an activation function, which can be a sigmoid function such as a logistic function or a hyperbolic tangent function, and is realized by, for example, a tanh function. m LS (ξ,ζ) is the L-level The output value of the feature detection neuron shown in Equation 1 is the internal state u of the feature detection neuron at position (ξ,ζ) on the m-th cell plane of the layer S of the eye. m LS It is calculated by transforming (ξ,ζ) with the activation function f.

[0027] The output value of the feature integration neuron shown in Equation 2 is calculated by the coupling coefficient w without using an activation function. m LC It is calculated by a simple linear sum of (u,v) and the mth output value of the Sth layer of the Lth hierarchy. If no quantization function is used, the internal state u of the feature integration neuron m LC (ξ,ζ) and output value y m LC (ξ,ζ) are equal. Also, y in Eq. n L-1C (ξ+u,ζ+v), y in Eq. m LS (ξ+u, ζ+v) are called the output values ​​of the feature integration neuron and the feature detection neuron, respectively.

[0028] The following explains ξ, ζ, u, v, and n in Equations 1 and 2. The position (ξ, ζ) corresponds to the position coordinates in the input image. y m LS (ξ,ζ) is the output of the feature detection neuron at other positions. If the force value is higher than the force value, it means that the feature detected in the m-th cell plane of the S layer of the Lth hierarchy is likely to exist at the pixel position (ξ, ζ) of the input image.

[0029] The n in Equation 1 means the n-th cell plane in the C layer of the L-1th layer, and is called the target feature number. m LS (ξ,ζ) is the number of layers in the C layer of the L-1th layer. For the cell surface, the coupling coefficient w m LS (n,u,v) and the output value of the feature integration neuron y n L-1C It is calculated by multiplying and adding (ξ+u,ζ+v).

[0030] (u,v) is the relative position coordinate of the coupling coefficient, and the product-sum operation is performed within a finite range (u,v) depending on the size of the feature to be detected. The finite range (u,v) is called the receptive field. The size of the receptive field is expressed as the number of horizontal pixels x the number of vertical pixels in the coupled range, and is hereafter referred to as the receptive field size.

[0031] In Equation 1, L=1, that is, the first S layer, y n L-1C (ξ+u,ζ+v) is the input Force Image in_image (ξ+u,ζ+v) or input position map y in_posi_map (ξ+u, The distribution of neurons and pixels is discrete, and the connection feature numbers are also discrete. Therefore, ξ, ζ, u, v, and n are not continuous variables, but take discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, and u and v are integers, all of which have a finite range.

[0032] w in Equation 1 m LS (n,u,v) is the distribution of coupling coefficients for detecting a given feature. By adjusting the connection coefficients to appropriate values, it becomes possible to detect specific features. In the construction (learning) of CNN, various test patterns are presented and y m LS (ξ,ζ) is The coupling coefficients are iteratively adjusted to achieve the appropriate output value.

[0033] w in Equation 2 m LC (u,v) is expressed as Equation 3 using a two-dimensional Gaussian function. .

number

[0034] (u,v) is a finite range, and as in the explanation of feature detection neurons, the finite range (u,v) is called the receptive field, and the size of the receptive field range is called the receptive field size. The receptive field size can be set to an appropriate value depending on the size of the feature detected on the m-th cell plane of the S layer of the Lth hierarchy. In Equation 3, σ is the feature size factor, and is set to an appropriate constant depending on the receptive field size. Specifically, the feature size factor σ is preferably set to a value that allows the value of the coupling coefficient at the outermost edge of the receptive field to be considered nearly 0.

[0035] By performing the above calculations at each layer of the neural network, it becomes possible to identify object classes at the final layer, S.

[0036] A specific learning method for the neural network will be described. The number of neurons is adjusted by supervised learning. In supervised learning, test patterns are given to actually obtain the output values ​​of the neurons, and the connection coefficients w m LS (n,u,v) is the actual neuron The coupling coefficients can be corrected based on the relationship between the output value of the neuron and the teacher signal (the desired output value that the neuron should output). For example, the coupling coefficients can be corrected using the least squares method in the feature detection layer at the final layer, and the backpropagation method in the feature detection layer at the intermediate layer. The coupling coefficients can be corrected using the least squares method, the backpropagation method, or other known methods such as those disclosed in Non-Patent Document 1.

[0037] When training a neural network in advance, many test patterns are prepared for training, including specific patterns to be detected and patterns that should not be detected. When the activation function is set to a tanh function and a specific pattern to be detected is presented, a teacher signal is given to neurons in the area of ​​the feature detection cell plane at the final layer where the specific pattern exists so that the output value becomes 1. Conversely, when a pattern that should not be detected is presented, a teacher signal is given to neurons in the area where the pattern that should not be detected exists so that the output value becomes -1.

[0038] By the above method, a neural network that can identify object classes from input images is constructed. The actual detection (identification) process is performed using the connection coefficients w m LS (n If the output value of a neuron on the feature detection cell plane of the final layer is equal to or greater than a predetermined value, it is determined that the object of the object class to be identified is present in the position (area) corresponding to the neuron.

[0039] The model structure of a neural network will be explained with reference to Figure 4. Figure 4 is a schematic diagram of the model structure of a neural network. In the explanation of Figure 4, neurons are called nodes, connections between neurons are called edges, and connection coefficients that represent the strength of connections between neurons are called edge weights. Nodes are represented by circles, and edges connecting nodes are represented by straight lines.

[0040] The inference model 400 (computing unit) shown in Figure 4 has a model structure including an input layer 401, a hidden layer 403, and an output layer 405. The input layer 401 has N nodes, the hidden layer 403 has M nodes, and the output layer 405 has 10 nodes, and the inference model 400 is a multi-class classifier that can distinguish between 10 object classes. The nodes in the hidden layer 403 perform calculations using an arbitrary activation function. The group of edges connecting the input layer 401 and the hidden layer 403 is called the first layer, and the group of edges connecting the hidden layer 403 and the output layer 405 is called the second layer.

[0041] The inference process for identifying the object class will be described with reference to Figures 5 to 9. Figure 5 is a flowchart illustrating the inference process using the inference model 400. In step S501, the CPU 101 determines whether the image (frame image) acquired by the image acquisition unit 104 is the first frame.

[0042] If CPU 101 determines that the acquired image is the first frame, the process proceeds to step S503. If CPU 101 determines that the acquired image is not the first frame, the process proceeds to step S505.

[0043] In the following description, CPU 101 switches the processing depending on whether it is the first frame (first frame), but this is not limiting. CPU 101 may determine whether the subject has changed based on, for example, the difference in pixel values ​​between the previous frame (first frame) and the current frame (second frame). CPU 101 may then proceed to step S503 if it determines that the subject has changed, or proceed to step S505 if it determines that the subject has not changed. Depending on the change in the subject, the calculation may be changed. Since the reference value for determining whether or not to interrupt is updated, the CPU 101 can further reduce unnecessary calculations.

[0044] In step S503, the CPU 101 executes inference calculation 1. The processing flow of inference calculation 1 will be described with reference to Fig. 6. Fig. 6 is a flowchart illustrating inference calculation 1 for the first frame image.

[0045] In step S601, CPU 101 executes inference operation 11. Inference operation 11 is a process of first calculating the input (influence) to each node in hidden layer 403, and then passing the calculated influence through an activation function to obtain an output value from the node.

[0046] First, the CPU 101 calculates the input to each node included in the hidden layer 403 of the inference model 400 in Fig. 4. With reference to Figs. 7 and 8, the input a to each node of the hidden layer 403 j (1) (j=1, 2, 3, ..., M). j indicates the number of each node included in the hidden layer 403. In the example of FIG. 4, the hidden layer 403 has M nodes. The input a to each node of the hidden layer 403 is j (1) "(1)" in j (1) indicates that this is the calculation result in the first layer. The input to node 421 is a1 (1) It is expressed as:

[0047] 7 is a diagram illustrating the computing units used in the nodes of the inference model 400. FIG. 7 shows an example of computation for the nodes included in the hidden layer 403. First, the CPU 101 uses a multiply-accumulate computing unit 701 to compute the input a j (1) (j=1, 2, 3, ..., M) is calculated. The input a to the j-th node of the hidden layer 403 is j (1) When calculating, the input to the product-sum calculator 701 is the output values ​​x from the N nodes of the input layer 401. i (1) (i=1,2,3,…,N) and edge weight w j,i (1) (i=1, 2, 3, ..., N). The multiply-accumulate calculator 701 calculates a j (1) as x i(1) and w j,i (1) For example, in node 421 (j=1), the product-sum calculator 701 calculates the sum of products of x i (1) and w 1,i (1) The sum of products with a1 (1) It is calculated as:

[0048] 8 is a diagram illustrating the configuration of the product-sum calculator 701 shown in FIG. 7. For each node in the hidden layer 403, the CPU 101 calculates x i (1) and w j,i (1) Repeatedly multiplying and adding the multiplied results, the input a to each node j (1) Calculates (j=1,2,3,...,M). Input a to the node obtained by the operation in 701 j (1) is stored in the memory 103 as the degree of influence.

[0049] Next, the CPU 101 calculates the input a j (1) are passed through the activation function calculator 702, and the output value x j (2) For example, the output value from node 421 is a1 (1) is passed through the activation function calculator 702. (2) The activation function calculator 702 uses a sigmoid function, a tanh function, or the like as an activation function. The CPU 101 calculates the input a j (1) is passed through the activation function calculator 702 to obtain x j (2) (j=1,2,3,…,M).

[0050] 4 shows an example in which there is one hidden layer 403, but there may be multiple hidden layers. When there are multiple hidden layers, CPU 101 executes inference operation 11 for each hidden layer. This determines the input to each node included in the multiple hidden layers.

[0051] 6, CPU 101 executes inference operation 12. Inference operation 12 is a process of first calculating the input (influence) to each node of output layer 405, and then passing the calculated influence through an activation function to obtain an output value from the node.

[0052] First, the CPU 101 calculates the input a to each node included in the output layer 405 of the inference model 400 shown in FIG. k (2) (k=1, 2, 3, . . . , 10) is calculated by the multiply-and-accumulate calculator 7 shown in FIG. Calculation is performed using 01.

[0053] Input a to the k-th node of the output layer 405 k (2) When calculating, the input to the product-sum calculator 701 is the output values ​​x from the M nodes of the hidden layer 403. j (2) (j=1,2,3,…,M) and edge weight w k,j (2) (j=1, 2, 3, ..., M). The multiply-accumulate calculator 701 calculates a k (2) as x j (2) and w k,j (2) For example, in node 423, the multiply-and-accumulate operator 701 calculates the sum of products with x j (2) and w 1,j (2) The sum of products with a1 (2) The input a to the node obtained by the calculation in the multiply-add calculator 701 is k (2) is stored in the memory 103 as the degree of influence.

[0054] The CPU 101 receives input a k(2) are passed through the activation function calculator 702. In the output layer 405, the activation function calculator 702 is, for example, a Softmax function. The Softmax function is expressed by the following equation 4. In equation 4, e is Napier's constant.

number

[0055] 9 is a graph showing an example of the classification result of the object class. k The k-th node of the output layer 405 (hereinafter referred to as node k) The output y from k is the presence of objects in the object class corresponding to node k. In Figure 9, the output y6 from node 6 in the output layer 405 is the probability that the output y6 from other nodes The probability that an object of the object class corresponding to node 6 exists is higher than the probability that an object of the object class corresponding to the other nodes exists.

[0056] In FIG. 9, the output y6 from node 6 is followed by the output y2 from node 2 and the output y7 from node 7, and the outputs from the other nodes are lower than y2 and y7. Therefore, the probability of existence of a subject of the corresponding subject class is highest at node 6, and decreases in the order of node 2, node 7, and the other nodes.

[0057] In step S505 of Fig. 5, CPU 101 executes inference calculation 2, which is processing when it is determined that the image acquired by image acquisition unit 104 is not the first frame. The processing flow of inference calculation 2 will be described with reference to Fig. 10. Fig. 10 is a flowchart illustrating inference calculation 2 for a frame other than the first frame.

[0058] In step S1001, the CPU 101 executes the inference operation 11. The inference operation 11 in step S1001 is the same as the inference operation 11 in step S601 in Fig. 6, and therefore a description thereof will be omitted.

[0059] In step S1003, the CPU 101 executes the inference operation 21. The inference operation 21 executes the inference operation 21. k (2) This is a process that calculates (k=1,2,3,...,10) halfway. k (2) is the output value x of M nodes in the hidden layer 403 (first layer) as shown in the following equation 5. j (2) and the corresponding w k,j (2) In the inference calculation 21, the calculation of Equation 5 is divided into two as shown in Equation 6, and the CPU 101 executes the calculation of the first term of Equation 6.

[0060]

number

[0061] In steps S1005 and S1007, CPU 101 determines whether or not to perform the calculation of the second term of equation 6, and reduces the amount of calculation by not performing calculations that do not affect the subject identification result and suspending the calculation of the influence degree.

[0062] In step S1005, CPU 101 determines whether the difference between a partial calculation result (hereinafter also referred to as partial influence) of the influence (second influence) in the current frame and a reference value is within a predetermined range. The reference value is a value corresponding to the sum of products (the first term in Equation 6) for some nodes of hidden layer 403 in the first frame (first frame), and is, for example, a k (2) It can be set to a value obtained by multiplying (first influence degree) by 0.5.

[0063] The influence is calculated by Equation 6. k (2) The partial influence is the calculation result of the first term of Equation 6. In this case, the CPU 101 calculates a k (2) It is determined whether the difference between 0.5 times (reference value) and the calculation result of the first term of Equation 6 (partial influence degree) is within a predetermined range.

[0064] When temporally consecutive frames, such as a moving image, are input to the image acquisition unit 104, there is generally a high correlation between the consecutive frames. For this reason, there is a possibility that a subject detected in a temporally consecutive previous frame will also be detected in the current frame, and a subject not detected in a consecutive previous frame will not be detected in the current frame either.

[0065] If the difference between the reference value and the calculation result of the first term of Equation 6 is within a predetermined range, the CPU 101 can determine that the first frame and the current frame capture the same subject characteristics. The predetermined range may be, for example, the influence degree in the first frame (a stored in the memory 103). k (2) ) is set based on the influence level. Specifically, the predetermined range may be 3% of the influence level or 5% of the reference value. By increasing the predetermined range, the number of cases where the calculation of the second term in Equation 6 is not executed increases, thereby further reducing the amount of calculation.

[0066] The value of m that separates the first and second terms in Equation 6 is set to about half of M, but is not limited to this and may be set to about 1 / α (0<1 / α<1) of M. The reference value is the value of a for the first frame when m is set to about half of M. j (2) However, if m is set to approximately 1 / α of M, a j (2) It is sufficient to set it to about 1 / α of the above.

[0067] In step S1005, if the CPU 101 determines that the difference between the reference value and the partial impact is within a predetermined range, the process proceeds to step S1007. If it is determined that the difference between the reference value and the partial influence is not within the predetermined range, the process proceeds to step S1009. If the difference between the reference value and the partial influence is not within the predetermined range, the CPU 101 determines that there is no correlation between the first frame and the current frame, and calculates a k (2) Calculate the following.

[0068] In step S1007, CPU 101 determines whether the partial influence is equal to or less than a threshold value. The partial influence is, for example, the calculation result of the first term of Equation 6. Since the influence is large when the characteristics of the subject are well captured, CPU 101 also calculates the second term of Equation 6 when the partial influence is greater than the threshold value.

[0069] On the other hand, the influence level is small when no features of the subject are detected, so the CPU 101 does not execute the calculation of the second term of Equation 6 when the partial influence level is equal to or less than the threshold value. k (2) The previous frame, the first frame, a k (2) Alternatively, a predetermined value such as the value of the first term, which is the calculation result at the time of interruption, or zero may be set. k (2) The value set in is input to node k of the output layer 405 as the influence.

[0070] The operation is interrupted and k (2) When the value of the first term of Equation 6 or a predetermined value is set to a, the value a stored in the memory 103 in the inference calculation 1 of the first frame is k (2) may be used as the influence for determining the reference value in the next frame without being updated. In this case, a stored in the memory 103 k (2) is not updated until the next inference operation 1 (step S503) is executed.

[0071] The threshold value to be compared with the partial influence is set, for example, as follows: k (2) The weight w k,j (2) (j=1,2,3,…,M) and the output x from the node j (2) etc. are normalized to the range of 0 to 1. The threshold is, for example, a k (2) It can be set to a value equal to or greater than 10% of M, which is the maximum value of

[0072] The percentage of M that the threshold should be set to is determined by the number of product sum terms in the first term of Equation 6. k (2) The threshold value may be set according to the percentage of M that the threshold occupies in the number of terms M. The threshold value may be set based on the accuracy and speed of identifying the object class. The threshold value is not limited to the maximum value M when the weights are normalized, and may be set according to the percentage of M that the threshold occupies in the number of terms M. k (2) may be set based on the maximum value that can be taken.

[0073] If the CPU 101 determines in step S1007 that the partial influence is equal to or less than the threshold, the process proceeds to step S1011. If the CPU 101 determines that the partial influence is greater than the threshold, the process proceeds to step S1009. If the partial influence is greater than the threshold, the CPU 101 determines that the output y k (the probability of existence of the corresponding object) is determined to be high, and the second Including the term a k (2) Calculate the following.

[0074] In step S1009, the CPU 101 executes the inference operation 22. The inference operation 22 executes the calculation process of the second term of the formula 6 to obtain the input a k (2) The CPU 101 calculates a, passes the calculation result through the activation function, and obtains the output value from the node. k (2) is passed through the activation function in the same manner as in the inference operation 12 in step S603.

[0075] The CPU 101 receives the obtained a k (2) may be stored in the memory 103 as the influence for determining the reference value in the next frame. k (2) is not stored, and the influence a stored in the memory 103 is not stored until the next inference operation 1 (step S503) is executed. k (2) The reference value may be determined using

[0076] In step S1011, the CPU 101 calculates the input a k (2) The CPU 101 determines whether the calculation of the input a k (2) If the calculation has been executed, it can be determined that the calculation has ended. If the CPU 101 determines that the calculation has ended, the processing of inference calculation 2 shown in Fig. 10 ends. If the CPU 101 determines that the calculation has not ended, the processing returns to step S1003.

[0077] In step S1005 of the inference calculation 2 in FIG. 10, the reference value is a stored in the memory 103 in the first frame. k (2)Although an example in which the reference value is set based on the influence degree is shown, the present invention is not limited to this. The reference value is set based on the influence degree stored in the memory 103 in the frame immediately before the current frame. k (2) It may be set based on

[0078] In this embodiment, the first frame a k (2) However, the CPU 101 may execute inference calculation 1 (step S503) for every predetermined number of frames, as in the first frame. When inference calculation 1 is executed, the CPU 101 stores a calculated in inference calculation 12 (step S603) as the influence. k (2) is set as the new influence degree, and the influence degree stored in the memory 103 is updated.

[0079] In addition, in Equation 5 and Equation 6, the input a k (2) Although an example of reducing the amount of calculation by interrupting part of the calculation (the calculation in the second layer in FIG. 4) has been described, it is also possible to reduce the amount of calculation in the first layer in a similar manner. j (1) In the calculation of (the calculation in the first layer of Figure 4), a is used as in Eq. j (1) is divided into the first and second terms, and the same processing as that from step S1003 to step S1009 in FIG. 10 is executed. j (1) By not performing the calculation of the second term, the amount of calculation can be reduced.

[0080] Furthermore, when there are multiple hidden layers, the amount of calculation may be reduced by interrupting some of the calculations of inputs to each node in the hidden layer next to hidden layer 403, similar to the calculations of inputs to each node in output layer 405.

[0081] According to the above embodiment, the information processing device 100 determines whether to suspend the calculation of the influence based on the influence in a frame prior to the current frame and the calculation results of part of the influence in the current frame. This allows the information processing device 100 to reduce the amount of calculation while suppressing the impact on inference accuracy. The information processing device 100 can shorten the inference calculation time by early suspending the calculation of a node whose influence is equal to or less than a threshold.

[0082] (Variation 1) In the inference calculation 12 in step S603 of FIG. 6, the influence is a k (2) In the first modification, the influence b k (2) As shown above, the influence is calculated by the output x j (2) and edge weight w k,j (2) may be configured to be an unsigned value that monotonically increases.

number

[0083] In this case, the inference model 400 also uses x j (2) and w k,j (2) When the influence is configured as in Equation 7, as shown in Figure 11, the influence b k (2) Since the value of increases monotonically, it does not decrease after exceeding the threshold. In the example of FIG. 11, the CPU 101 performs m multiply-accumulate operations, which is about half of M multiply-accumulate operations. Therefore, CPU 101 can determine whether a partial calculation result of the calculation of the influence degree (partial influence degree) is equal to or greater than the threshold value in the process of step S1007 in FIG.

[0084] (Variation 2) Each node in the output layer 405, for example, the input a k (2) is the input a to each node of the hidden layer 403 in the first frame (or a frame before the current frame). j (1) The calculation may be performed based on the magnitude of the influence (degree of influence). CPU 101 performs the product-sum calculation by prioritizing nodes in hidden layer 403 that have a greater influence in the first frame than other nodes. This shortens the time until the threshold is exceeded, allowing CPU 101 to more quickly determine whether to perform inference calculation 22.

[0085] For example, in the inference operation 21 in step S1003 of FIG. k (2) If the intermediate calculation result exceeds the threshold, the CPU 101 determines that the influence will be large, and proceeds to step S1009 to execute the inference calculation 22. The threshold here can be determined in the same way as the threshold in step S1007.

[0086] Instead of the determinations in steps S1005 and S1007, the CPU 101 prioritizes nodes in the hidden layer 403 in the previous frame that have a greater influence than other nodes. k (2) This allows the CPU 101 to more quickly determine whether the threshold value has been exceeded.

[0087] Note that the nodes in hidden layer 403 do not need to be sorted strictly in descending order of influence, but rather, nodes with greater influence than other nodes may be prioritized. For example, CPU 101 may classify nodes into four groups according to the magnitude of their influence, and perform calculations starting with nodes in groups with greater influence than other groups. In this case, the magnitude of influence within a group need not be taken into consideration. The grouping according to influence may be, for example, into four groups with the highest influence of 100% to 75%, 75% to 50%, 50% to 25%, and 25% to 0%. The number of groups is not limited to four, and may be determined depending on the processing load caused by grouping or sorting.

[0088] In addition, the CPU 101 selects the nodes in the hidden layer 403 with influence a j (1) The nodes with a larger value are given priority for multiplication and addition, but the influence is j (1) In the second modification, the input a to the node of the output layer 405 is k (2) The influence of determining the order of calculation is the output value x in the first frame. j (2) and edge weight w k,j (2) In this case, the CPU 101 calculates the output value x from each node of the hidden layer 403 in the first frame. j (2) and the corresponding weight w k,j (2) The product of the output value and the weight in the first frame is used as the influence, and the CPU 101 prioritizes nodes with a higher influence than other nodes and calculates the input a to the nodes in the output layer 405 in the current frame. k (2) Calculation is performed.

[0089] (Variation 3) When applying a filter to the entire frame image to extract features, the information processing device 100 may thin out a portion of the frame image and input it to the inference model 400. For example, the CPU 101 may apply a filter to lines 0-2 of the frame image, then thin out the scan with a filter (such as lines 1-3) starting from lines 1 to 5, and apply the filter to lines 6-8. The CPU 101 may apply the filter to the end of the frame image while thinning out, and then return to lines 1-3 and apply the filter again. Note that the frame image is not limited to being thinned line by line in the vertical direction, and may be thinned horizontally or both vertically and horizontally.

[0090] By thinning out a portion of the frame image and scanning the filter in this way, the CPU 101 can roughly grasp the characteristics of the entire image. Even with an image whose characteristics are roughly grasped, the CPU 101 can interrupt the calculation early and reduce the amount of calculation based on the influence degree in the previous frame and the calculation result of part of the influence degree in the current frame.

[0091] When handling image signals as input, the input x i (1) can be an unsigned 8-bit RGB or offset YUV signal. A YUV signal can represent color information by combining a luminance signal (Y), the difference between the luminance signal and the blue component (U), and the difference between the luminance signal and the red component (V).

[0092] While the present invention has been described above with reference to preferred embodiments, it is not limited to these embodiments and various modifications and variations are possible within the scope of the present invention. Furthermore, the features described in the embodiments can be combined as appropriate.

[0093] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0094] 100: Information processing device, 101: CPU, 104: Image acquisition unit, 105: Recognition unit, 400: Inference model

Claims

1. an acquisition means for acquiring successive frames of a video; an identification means for identifying an object from each frame of the video using a computing unit including a plurality of layers, the computing unit calculating an influence degree which is the sum of the product of an output from each node included in a first layer to a predetermined node in a second layer and a weight corresponding to each of the outputs, and inputting the calculated influence degree to the predetermined node; The discrimination means determines whether or not to suspend the calculation of the second influence based on a first influence input to the predetermined node in a first frame and a partial calculation result of a second influence input to the predetermined node in a second frame subsequent to the first frame.

1. An information processing device comprising:

2. The discrimination means suspends calculation of the second influence level input to the predetermined node when a difference between a product sum for the part of nodes in the first layer in the second frame and a reference value that is a value corresponding to the product sum for the part of nodes in the first frame is within a predetermined range and the product sum for the part of nodes in the first layer in the second frame is equal to or smaller than a threshold value.

2. The information processing apparatus according to claim 1, wherein:

3. The predetermined range is set based on the first influence degree.

3. The information processing apparatus according to claim 2, wherein:

4. The threshold is set based on the maximum possible sum of the product of the output from each node included in the first layer to a predetermined node in the second layer and the weight corresponding to each of the outputs.

4. The information processing apparatus according to claim 2, wherein the information processing apparatus is a computer.

5. The identifying means inputs the calculation result or a predetermined value at the time when the calculation of the second influence is interrupted to the predetermined node as the second influence.

5. The information processing device according to claim 1, wherein the information processing device is a computer.

6. The identification means calculates the sum of products of the second influences in the second frame by giving priority to nodes in the first layer that have a greater influence than other nodes in the first frame.

6. The information processing device according to claim 1, wherein the information processing device is a computer.

7. The information processing device according to claim 6, characterized in that the identification means classifies each node of the first hierarchy into a plurality of groups based on the magnitude of its influence in the first frame, and calculates the sum of the products of the second influence in the second frame, starting with nodes in groups with greater influence than other groups.

8. The discrimination means sets the weights used in the calculation of the first influence degree and the second influence degree as unsigned values.

8. The information processing device according to claim 1, wherein the information processing device is a computer.

9. The discrimination means thins out some of the frame images of the moving image and inputs them to the computing unit.

9. The information processing device according to claim 1, wherein the information processing device is a computer.

10. The identification means updates the first influence degree every predetermined number of frames of the moving image.

10. The information processing device according to claim 1, wherein the information processing device is a computer.

11. an acquisition step of acquiring successive frames of a video; an identification step of identifying an object from each frame of the video using a computing unit including a plurality of layers, the computing unit calculating an influence degree which is the sum of the product of an output from each node included in a first layer to a predetermined node in a second layer and a weight corresponding to each of the outputs, and inputting the calculated influence degree to the predetermined node; Has, In the identifying step, it is determined whether or not to suspend the calculation of the second influence based on a first influence input to the predetermined node in a first frame and a partial calculation result of a second influence input to the predetermined node in a second frame subsequent to the first frame.

2. A method for controlling an information processing apparatus comprising:

12. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 10.

Images