Information classification device, learning device, and program

Abstract

To provide an information classification device, a learning device, and a program which allow for improvement of learning effect and increase in classification accuracy of moving images with less increase of computational complexity.SOLUTION: The information classification device comprises: a tensorizing unit which converts input moving image data to a tensor; a base model which classifies information included in the moving image data in accordance with the tensorized moving image data and consists of a plurality of layers; and a time correlation block which receives an intermediate tensor outputted from an n-th layer of the base model and generates a time-correlation feature integrated intermediate tensor including correlation information with all frames of the moving image data, from the intermediate tensor and outputs the time-correlation feature integrated intermediate tensor to an (n+1)th layer of the base model. Classification results are outputted from an output layer of the base model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information classification device, a learning device, and a program, and more particularly to an information classification device that classifies the type of information included in video data, a learning device therefor, and a program therefor. [Background technology]

[0002] In recent years, a lot of video is generated and recorded every day. For example, broadcasting stations have accumulated a large amount of video, including past broadcast footage and recorded materials. In order to reuse these efficiently, technology is essential to automatically assign metadata to video for classifying and searching the video content. Face recognition and object recognition technologies can identify the "names of people and objects in the video," but technology to assign more diverse metadata is required to understand the video content in detail. One such technology is video classification technology to identify "movements" and "actions."

[0003] The use of neural networks has been considered as a method for classifying or analyzing videos. For example, a behavior recognition system using a three-dimensional convolutional neural network with video and its optical flow data as input has been proposed (Patent Document 1). In addition, a system has been proposed for analyzing behavior from video, using a convolutional neural network and a recurrent neural network to recognize driving behavior based on video captured from the driver's viewpoint of the vehicle (Patent Document 2). In this system, a recurrent neural network is used to integrate temporal information.

[0004] Also, a video classification model called "X3D" is known as a method for classifying videos using three-dimensional convolution (Non-Patent Document 1). A conventional information classification device that classifies videos using this three-dimensional convolution will be described.

[0005] FIG. 5 is a block diagram showing an example of a conventional information classification device. The information classification device in FIG. 5 takes video data as input, discriminates the actions of the people shown in the video data, and outputs the type of the action as a classification result. The information classification device is composed of a tensorization unit 10 and a base model 20.

[0006] The tensorization unit 10 tensorizes (converts to a tensor) the input video data to obtain a tensor V.

[0007] The base model 20 is a learnable device to which a neural network is applied, and is composed of N layers 21 to 26. For example, a known model X3D (Non-Patent Document 1) can be used for the structure of the base model. When the tensor V of the video data is input to the base model 20, various processes such as 3D convolution are performed in each layer, and the tensor V is converted. The intermediate n-th layer (n < N) 23 outputs an intermediate tensor X n and inputs this intermediate tensor X n to the (n + 1)-th layer 24. Through the processes in each layer, the N-th layer, which is the output layer, outputs a probability value that is the type of the video (action), and the classification result of the input video can be obtained.

[0008] Also, in the prior art, the tensor V obtained from the video data with correct labels was input to the base model 20, and the classification result was compared with the correct label to perform learning of the base model 20.

Prior Art Documents

Patent Documents

[0009]

Patent Document 1

Patent Document 1

Non-Patent Documents

[0010]

Non-Patent Document 1

[0011] In general, 3D convolution used in video classification models only considers local time information. In order to obtain wide-range time information that is useful for capturing the movements throughout the entire video, it is necessary to expand the kernel size in the time direction of the convolution or to repeatedly apply multiple convolutions. There are also methods that use recurrent neural networks, but both methods have the problem of increasing the amount of calculations. In particular, video classification models tend to have larger input data and classification models than image classification models, so the increased amount of calculations increases the learning time, and in some cases, problems such as insufficient memory on the computer used for learning occur, making learning impossible.

[0012] Therefore, in consideration of the above-mentioned problems, an object of the present invention is to provide an information classification device, a learning device, and a program that obtain time information across an entire video with a small amount of calculation, improve the learning effect, and increase the accuracy of video classification. [Means for solving the problem]

[0013] In order to solve the above problems, the information classification device according to the present invention includes a tensorization unit that converts input video data into tensors, a base model consisting of multiple layers that classifies information contained in the tensorized video data, and a temporal correlation block that receives an intermediate tensor output from the n-th layer of the base model as input, generates a temporal correlation feature integrated intermediate tensor from the intermediate tensor that includes correlation information with all frames of the video data, and outputs the intermediate tensor to the n+1-th layer of the base model, and outputs a classification result from the output layer of the base model.In the information classification device, the time correlation block includes a time correlation calculation unit that calculates a correlation in the time direction between a reference signal and the intermediate tensor and outputs a time correlation tensor, a time correlation integration unit that integrates the time correlation tensor into one tensor to generate an integrated tensor, a time correlation feature calculation unit that calculates a time correlation feature tensor indicating a time correlation feature of the video data based on the integrated tensor, and a time correlation feature integration unit that integrates the time correlation feature tensor and the intermediate tensor into one tensor to generate a time correlation feature integrated intermediate tensor. It is characterized by:

[0015] In addition, it is preferable that in the information classification device, the base model and the time correlation block are configured using a neural network.

[0016] In addition, it is preferable that the information in the information classification device is a type of motion of a person included in the video data.

[0017] In order to solve the above problems, the learning device according to the present invention includes a tensorization unit that converts video data as learning data into tensors, a base model consisting of multiple layers that classifies information contained in the video data from the tensorized video data, a time correlation block that receives an intermediate tensor output from the n-th layer of the base model, generates a time correlation feature integrated intermediate tensor from the intermediate tensor including correlation information with all frames of the video data, and outputs the time correlation block to the n+1-th layer of the base model, a loss calculation unit that compares the classification result output from the base model with a correct label of the learning data to calculate a loss, and a learning termination determination unit that determines the termination of learning based on a termination determination criterion, and learns a model of the base model and the time correlation block based on the loss calculation to generate an information classification model. In the learning device, the time correlation block includes a time correlation calculation unit that calculates a correlation in the time direction between a reference signal and the intermediate tensor and outputs a time correlation tensor, a time correlation integration unit that integrates the time correlation tensor into one tensor to generate an integrated tensor, a time correlation feature calculation unit that calculates a time correlation feature tensor indicating a time correlation feature of the video data based on the integrated tensor, and a time correlation feature integration unit that integrates the time correlation feature tensor and the intermediate tensor into one tensor to generate a time correlation feature integrated intermediate tensor. It is characterized by:

[0019] In addition, it is preferable that the base model and the time correlation block of the learning device are configured using a neural network.

[0020] In order to solve the above problems, the present invention provides a program that causes a computer to function as the information classification device.

[0021] In order to solve the above problems, the present invention provides a program for causing a computer to function as the learning device. Effect of the Invention

[0022] According to the information classification device, learning device, and program of the present invention, it is possible to improve the learning effect and the accuracy of video classification while suppressing an increase in the amount of calculation. [Brief description of the drawings]

[0023] [Figure 1] 1 is a block diagram showing an example of an overall configuration of an information classification device and a learning device according to an embodiment. [Diagram 2] FIG. 2 is a conceptual diagram illustrating an example of a base model. [Diagram 3] FIG. 2 is a block diagram showing an example of a time correlation block of an information classification device. [Figure 4] 11 is a flowchart illustrating an example of a learning algorithm of the learning device according to an embodiment. [Diagram 5] FIG. 1 is a block diagram showing an example of a conventional information classification device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0025] (Embodiment) FIG. 1 is a block diagram showing an example of the overall configuration of an information classification device and a learning device according to an embodiment of the present invention.

[0026] The information classification device 100 of the present invention receives video data as input, and outputs the type of information contained in the video data (for example, the type of motion of a person appearing in the video data) as a classification result. The information classification device 100 includes a tensorization unit 10, a base model 20, and a time correlation block 30. The information classification device 100 can be configured as a whole using a single computer and program.

[0027] Moreover, the learning device 200 of the present invention is a learning device that generates a model (information classification model) of the information classification device 100 that classifies the types of information included in video data by performing machine learning. The learning device 200 is configured by adding a loss calculation unit 110 and a learning end determination unit 120 to the tensorization unit 10, the base model 20, and the time correlation block 30 (i.e., the information classification device 100). The learning device 200 can be configured as a whole by one computer and a program. Note that, if necessary, the loss calculation unit 110 and the learning end determination unit 120 may be configured by separate computers.

[0028] In the 3D convolution used in many video classification models, only local time information is considered, but in the present invention, a time correlation block 30 that can obtain time information for the entire video with a small amount of calculation is incorporated into the video classification model (information classification model). The inclusion of this time correlation block 30 is what distinguishes our invention from the conventional technology (Fig. 5).

[0029] In the present invention, the intermediate tensor X n The time correlation feature integration intermediate tensor X n corr The time correlation feature integration intermediate tensor X n corr contains correlation information with all frames of video data, i.e., correlation information in the time direction of the entire video. This time correlation feature integration intermediate tensor X n corr By inputting this to the (n+1)th layer 24 of the base model 20, a model is constructed that obtains classification results using features that take into account the time information of the entire video.

[0030] 1, one time correlation block 30 is provided, but the time correlation block 30 may be added to multiple locations in the base model 20. AI (neural network) is applied to the base model 20 and the time correlation block 30. Details of the time correlation block 30 will be described later.

[0031] In the present invention, in order to capture the time information of the entire video, instead of expanding the kernel size in the time direction of the 3D convolution or repeatedly applying a plurality of convolutions, by using the temporal correlation block 30, it is possible to enhance the learning effect while suppressing an increase in the amount of calculation. According to the information classification device 100 of the present invention, compared with a conventional information classification device, the classification accuracy for determining what kind of information is shown in the video is improved, and metadata can be more accurately added to the video.

[0032] Each component of the information classification device 100 will be described.

[0033] The tensorization unit 10 tensorizes the input video data to obtain a tensor (input tensor) V. Known means can be used for tensorization. For example, when the number of frames of the input video is T, the number of channels (e.g., R, G, B, etc.) is C, the height is H, and the width is W, the input tensor V is generated as a tensor of T×C×H×W in which pixel values are arranged for each frame.

[0034] The base model 20 is configured using a feed forward neural network (FFNN). A known model (X3D, SlowFast, etc.) can be used for the structure of the base model 20. FIG. 2 is a conceptual diagram showing an example of the base model 20. The base model 20 is composed of N layers 21 to 26. For example, the architecture of X3D is composed of a plurality of blocks named conv1, res2 to res5, conv5, pool5, fc1, fc2, etc., and each block includes a plurality of data layers.

[0035] In FIG. 2, the input tensor V obtained by tensorizing the input video data by the tensorization unit 10 is input to the input layer 21. Various processes such as 3D convolution are performed in each layer to convert the input tensor V. The intermediate n-th layer (n < N) 23 outputs an intermediate tensor X n and outputs this intermediate tensor Xn The time correlation feature integration intermediate tensor X n corr Assuming that the output is n corr is input to the n+1th layer 24 of the base model 20. After processing in each layer, a probability value (score) that the input video corresponds to the corresponding type is output to an output unit corresponding to the type of action (e.g., walking, running, drinking, etc.) in the output layer (Nth layer) 26. In this way, the classification result of the input video can be obtained.

[0036] Next, the time correlation block 30 will be described. Fig. 3 is a block diagram showing an example of the time correlation block 30 of the information classification device 100. The time correlation block 30 is provided with reference signals 40, 41, and includes time correlation calculation units 50, 51, a time correlation integration unit 60, a time correlation feature calculation unit 70, and a time correlation feature integration unit 80. Of these, a forward propagation neural network (FFNN) is applied to the time correlation feature calculation unit 70. Also, AI (neural network) can be applied to the time correlation integration unit 60 and the time correlation feature integration unit 80 as necessary.

[0037] The time correlation calculation units 50 and 51 calculate the intermediate tensor X n The correlation between the n-th layer 23 of the base model 20 and the intermediate tensor X n Let T be a tensor with the number of frames, C with the number of channels, H with the height, and W with the width. Let X be the number of frames represented by the tensor with the number of channels, C with the height, and W with the width. n [1], X n [2], …, X n Let's say [T].

[0038] Also, M reference signals 1 (40) to M (41) are prepared. The reference signals may be preset fixed signals or may be learnable parameters. The reference signals 40, 41 can be provided to the time correlation calculation units 50, 51 by, for example, reading a preset reference signal from a memory or generating a reference signal from a reference signal learning and generation block. The mth (1≦m≦M) reference signal is a numeric sequence of length T, which is referred to as C. m [1], C m [2], …, C m Let's say [T].

[0039] The m-th time correlation calculation unit calculates the m-th reference signal and the intermediate tensor X n The correlation in the time direction with is calculated using the following equation (1). The calculation result is the time correlation tensor with the mth reference signal.

[0040]

number

[0041] Similarly, in each of the time correlation calculation units 50 and 51, the reference signals (1 to M) and the intermediate tensor X n The time correlation with is calculated, and the calculation result is output to the time correlation integrating unit 60. The reference signal can also be interpreted as a weighting for each of all frames (T frames).

[0042] The time correlation integration unit 60 integrates the M correlations (time correlation tensors) output from the time correlation calculation units 50 and 51 into one tensor, and outputs the integrated tensor to the time correlation feature calculation unit 70. The integration method may be a known method using a combination of tensors, an element-by-element sum, an element-by-element product, or a model with learnable parameters. When there is only one time correlation calculation unit 50, the time correlation integration unit 60 may be omitted.

[0043] The time correlation feature calculation unit 70 calculates a tensor (time correlation feature tensor) s indicating the time correlation feature of the input video based on an integrated tensor obtained by integrating M correlations. For this calculation, a convolution layer or a pooling layer of a neural network is used. The time correlation feature calculation unit 70 can improve the calculation model so that more appropriate time correlation features can be obtained by a learning algorithm described later. The calculated time correlation feature tensor s is output to the time correlation feature integration unit 80.

[0044] The time correlation feature integration unit 80 calculates the time correlation feature tensor s, which indicates the time correlation feature of the video data output from the time correlation feature calculation unit 70, and the intermediate tensor X n The integration method can be a combination of tensors, element-wise sum, element-wise product, or a known method using a model with learnable parameters. The integration result is the time-correlation feature integration intermediate tensor X n corr which is the output of the time correlation block 30.

[0045] The information classification device 100 uses the time correlation feature integrated intermediate tensor X n corr is input to the n+1-th layer 24 of the base model 20. Thereafter, a classification result is obtained by processing in the n+1-th layer 24 to the N-th layer 26 of the base model 20. Note that a plurality of time correlation blocks 30 may be provided and inserted into a plurality of locations in the base model 20.

[0046] The above is an overview of the configuration and operation of information classification device 100.

[0047] Next, the learning device 200 of the present invention will be described.

[0048] 1, the learning device 200 includes the information classification device 100 (tensorization unit 10, base model 20, and time correlation block 30), and further includes a loss calculation unit 110 and a learning end determination unit 120. In the learning mode, the learning device 200 corrects the model (base model 20 and time correlation block 30) configured with a neural network or the like of the information classification device 100 based on the calculation result of the loss calculation unit 110, and determines the end time of learning of the base model 20 and the time correlation block 30 based on the result of the learning end determination unit 120.

[0049] Loss calculation unit 110 performs loss calculation by comparing the classification result based on the learning data output from information classification device 100 with the correct label. The loss (error) can be calculated using a general loss calculation method such as cross entropy.

[0050] In the learning mode, for example, for each batch N of learning data, learning (parameter correction) of the models of the base model 20 and the time correlation block 30 (e.g., a time correlation feature calculation unit 70 to which a neural network is applied) is performed so as to reduce the loss (error) calculated by the loss calculation unit 110.

[0051] The learning end determination unit 120 determines whether to continue or end learning in the learning device 200. For example, the learning end determination unit 120 obtains the output accuracy of the information classification device 100 from the classification result of the learning data (or verification data) and the correct answer data, evaluates the learning state, and determines whether to end the learning based on whether a predetermined learning state has been achieved. Alternatively, the learning end determination may be based on other end determination criteria, such as the number of times learning is performed based on the learning data.

[0052] In the learning mode, the learning device 200 learns (modifies parameters) the base model 20 and the time correlation block 30 models, and ends the learning when a predetermined end criterion is met. In this way, an information classification model of the information classification device 100 is generated.

[0053] 4 is a flowchart showing an example of a learning algorithm for the learning device 200 of this embodiment. Each step will be described below.

[0054] Step S1: Video data as learning data is input to the learning device 200.

[0055] Step S2: Learning is performed using the input learning data (video data). Specifically, the classification result based on the learning data is compared with the correct label, and based on the result, learning (parameter correction) of the models of the base model 20 and the time correlation block 30 is performed so as to reduce the error.

[0056] Step S3: The learning end determination unit 120 determines whether to end learning of the learning device 200. If the learning end condition is not met, the process returns to step S1, and if the learning end condition is met, the process proceeds to the next step S4. The learning end condition can be set appropriately according to the desired information classification device 100, for example, when the accuracy of the classification result reaches a predetermined level, or when a predetermined number of learnings have been performed (when a predetermined number of epochs have been completed in which all learning data is used once for learning).

[0057] Step S4: The trained base model 20 and the time correlation block 30 are output or saved.

[0058] After going through the above steps, the flowchart of the learning algorithm is completed.

[0059] According to the learning device 200 of the present invention, in addition to learning the conventional base model 20, a model based on the time information of the entire video can be learned by learning the time correlation block 30, thereby improving classification accuracy. Furthermore, since the information classification model of the present invention can extract the time information of the entire video using a simple calculation method, it is possible to shorten the learning time and reduce resources such as memory used for learning compared to video classification models in which the time information is expanded using conventional methods.

[0060] (Examples and verification of effects) An embodiment of the information classification device 100 and learning device 200 according to the present invention was configured as follows, and the effects were verified.

[0061] [Dataset] As input video data, we used 5,100 videos from the HMDB51 video dataset (with correct answer labels) for training data, and 1,530 videos from the video dataset for evaluation data.

[0062] [Base model] X3D-S (see Non-Patent Document 1) was used as the structure of the base model 20. For the initial values ​​of the network part of the base model, parameters of X3D-S trained by Kinetics 400 were used. Note that these parameters are publicly available.

[0063] [Time correlation block] A time correlation block 30 was inserted after either the conv1 block, the res4 block, or the res5 block of X3D-S.

[0064] The reference signals 40 and 41 use a sine function and a cosine function with the number of video frames as one wavelength. The output of the nth intermediate layer (intermediate tensor) X n X frame t n t Then, the time correlation calculation unit performed the calculation of the following equation (2).

[0065]

number

[0066] Since the sin and cos functions are orthogonal to each other, X sin and X cos can be regarded as two independent time-correlated features across the entire video. The correlation between the sine function and the cosine function used in this embodiment corresponds to extracting low-frequency components of the temporal features across the entire video.

[0067] In the time correlation integration unit 60, the obtained X sin and X cos were alternately combined in the channel direction to form an integrated tensor.

[0068] The time correlation feature calculation unit 70 is composed of three convolution layers. sin and X cos Apply group convolution with group number C to the combined tensor, X n conv X n conv is a tensor of size C×1×H×W. Furthermore, this X n conv The convolution and transformation process of the following equation (3) was performed on the image to obtain the weight (s) for each channel and image position.

[0069]

number

[0070] Both Conv1 and Conv2 are convolution layers with a kernel size of 1×1 and the number of output channels being C / 2 and C, respectively, which have the effect of improving generalization performance. In addition, ReLU (Rectified Linear Unit) is a function in which the output value is always 0 when the input value to the function is 0 or less, and the output value is the same as the input value when the input value is above 0. In addition, σ is a sigmoid function. From this calculation formula, s becomes a tensor with a size of C×1×H×W. This tensor s is set as the time correlation feature tensor s, which is the output of the time correlation feature calculation unit 70.

[0071] The time correlation feature integration unit 80 integrates the time correlation feature tensor s and the intermediate tensor X n and are integrated into one tensor to create the time correlation feature integrated intermediate tensor X n corr The time correlation feature integration intermediate tensor X n corr X frame t n corr [t], the tth frame of the intermediate tensor X nWhen [t] and the time correlation feature tensor is s, * is the product of each element and is calculated using the following equation (4).

[0072]

number

[0073] [output] The output layer of the base model 20 results in a score indicating the probability of falling into one of the 51 categories. The output of the information classification device 100 is the category with the maximum score, which is the determination result (classification result).

[0074] [Learning and Evaluation] An information classification model using only the base model based on the conventional X3D-S method and an information classification model using X3D-S and a time correlation (TC) block 30 according to an embodiment of the present invention were trained, and the accuracy and the amount of calculation required for product-sum operations were compared. Each model was trained for the same number of epochs. Each model was evaluated based on the accuracy rate.

[0075] Table 1 shows the results when the temporal correlation (TC) block 30 is inserted after each of conv1, res4, and res5 of X3D-S. The highest accuracy was achieved when the TC block was inserted after conv1 and res5, and compared to the conventional model that does not use the temporal correlation block, the accuracy rate improved by 1.4%, while the increase in the amount of calculation was suppressed to 0.2%.

[0076] [Table 1]

[0077] From the above comparison results, it was confirmed that the information classification model of the present invention, which takes into account the temporal correlation of image features throughout an entire video, improves classification accuracy compared to conventional techniques while reducing the amount of calculations.

[0078] In the above embodiment, the configuration and operation of the information classification device 100 have been described, but the present invention is not limited to this, and may be configured as a classification method for classifying information from video data. That is, the present invention may be configured as a classification method including each process from converting video data into tensors to outputting the classification result according to the data flow in FIG. 1.

[0079] Furthermore, in the above embodiment, the configuration and operation of learning device 200 have been described, but the present invention is not limited to this, and may be configured as a learning method for an information classification device that classifies information from video data. That is, the present invention may be configured as a learning method including each process up to learning an information classification model for an information classification device to which a neural network is applied, according to the data flow in FIG. 1.

[0080] A computer can be suitably used to function as the above-mentioned information classification device 100 or learning device 200, and such a computer can be realized by storing in a storage unit of the computer a program describing the processing contents for realizing each function of the information classification device 100 or learning device 200, and reading and executing this program by the CPU of the computer. This program can be recorded on a computer-readable recording medium.

[0081] Although the above-mentioned embodiment has been described as a representative example, it is clear to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or changes can be made without departing from the scope of the claims. For example, the functions included in each block, each step, etc. described in the embodiment can be rearranged so as not to be logically inconsistent, and multiple configuration blocks, steps, etc. can be combined into one or divided. [Explanation of symbols]

[0082] 10 Tensorization section 20 Base model 30 Time Correlation Blocks 40,41 Reference signal 50,51 Time correlation calculation unit 60 Time correlation integration unit 70 Time correlation feature calculation unit 80 Time correlation feature integration unit 100 Information classification device 110 Loss calculation section 120 Learning completion determination unit 200 Learning Device

Claims

1. a tensorizer for converting input video data into a tensor; A base model consisting of multiple layers that classifies information contained in the video data from the tensorized video data; a time correlation block that receives an intermediate tensor output from the n-th layer of the base model, generates a time correlation feature integration intermediate tensor from the intermediate tensor, the time correlation feature integration intermediate tensor including correlation information with all frames of the video data, and outputs the time correlation feature integration intermediate tensor to the n+1-th layer of the base model; In an information classification device that outputs a classification result from an output layer of the base model, The time correlation block comprises: A time correlation calculation unit that calculates a correlation in the time direction between a reference signal and the intermediate tensor and outputs a time correlation tensor; A time correlation integration unit that integrates the time correlation tensors into one tensor to obtain an integrated tensor; a time correlation feature calculation unit that calculates a time correlation feature tensor that indicates a time correlation feature of the video data based on the integrated tensor; a time correlation feature integration unit that integrates the time correlation feature tensor and the intermediate tensor into one tensor to generate a time correlation feature integrated intermediate tensor; An information classification device comprising:

2. 2. The information classification device according to claim 1, An information classification device, characterized in that the base model and the time correlation block are constructed using a neural network.

3. 3. The information classification device according to claim 1, An information classification device, characterized in that the information is a type of human movement contained in the video data.

4. A tensorization unit that converts video data as learning data into tensors; A base model consisting of multiple layers that classifies information contained in the video data from the tensorized video data; A temporal correlation block that receives an intermediate tensor output from the n-th layer of the base model, generates a temporal correlation feature integration intermediate tensor from the intermediate tensor, the temporal correlation feature integration intermediate tensor including correlation information with all frames of the video data, and outputs the intermediate tensor to the n+1-th layer of the base model; a loss calculation unit that compares the classification result output from the base model with a correct label of the training data to calculate a loss; a learning completion determination unit that determines whether learning has been completed based on a completion determination criterion; A learning device that learns the base model and the time correlation block model based on the loss calculation and generates an information classification model, The time correlation block comprises: A time correlation calculation unit that calculates a correlation in the time direction between a reference signal and the intermediate tensor and outputs a time correlation tensor; A time correlation integration unit that integrates the time correlation tensors into one tensor to obtain an integrated tensor; a time correlation feature calculation unit that calculates a time correlation feature tensor that indicates a time correlation feature of the video data based on the integrated tensor; a time correlation feature integration unit that integrates the time correlation feature tensor and the intermediate tensor into one tensor to generate a time correlation feature integrated intermediate tensor; A learning device comprising:

5. 5. The learning device according to claim 4, A learning device, wherein the base model and the time correlation block are configured using a neural network.

6. A program causing a computer to function as the information classification device according to any one of claims 1 to 3.

7. A program for causing a computer to function as the learning device according to claim 4 or 5.

Images