Video processing device, method, and program

Abstract

A video processing device according to one embodiment comprises: a video feature extraction unit that extracts feature information indicating a feature of a video; a linguistic information extraction unit that extracts linguistic information related to a behavioral intention of a subject who appears in the video; and an inference unit that infers, on the basis of the feature information and the linguistic information, a region which receives attention from the subject in the video.

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a video processing device, method, and program. [Background technology]

[0002] First-person video gaze area estimation technology is a technology that estimates the area that a person is looking at while they are moving. This technology estimates and visualizes the gaze of an expert, and is being researched as a means of skill transfer, such as helping non-experts acquire the skills of experts. For example, Non-Patent Document 1 discloses a configuration including a video feature extraction unit, a gaze area estimation unit, and an action label estimation unit. In this configuration, the video feature extraction unit extracts video features from an input video, and the gaze area estimation unit estimates a gaze area from the video features. Finally, the feature amount obtained by superimposing the video feature and the estimated gaze area is input to the activity label estimation unit, and the activity label is estimated. In this way, by estimating the gaze area together with the activity label estimation, the subject's activity can be estimated with high accuracy. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kyle Min and Jason J. Corso, “Integrating human gaze into attention for egocentric activity recognition”, in Proc. Of WACV2021. [Non-patent document 2] Joao Carreira and Andrew Zisserman, “Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset.” In Proc. of CVPR2017. [Non-patent document 3] Thomas N. Kipf and Max Welling, “Semi-supervised classification with graph convolutional networks.” in Proc. of ICLR2017. [Non-patent document 4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin, “Attention is all you need.” in Proc. of Neurips2017. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the gaze area estimated by the gaze area estimation unit disclosed in Non-Patent Document 1 is limited to gaze areas associated with pre-set behavioral labels, and gaze areas that are relatively less related to the subject's current behavior may be missed from the detection targets.

[0005] This invention has been made in light of the above circumstances, and its purpose is to provide a video processing device, method, and program that can estimate a gaze area taking into account a person's behavioral intentions, and can also estimate a gaze area that has little correlation with the current behavior. [Means for solving the problem]

[0006] The video processing device according to one aspect of the present invention includes a video feature extraction unit that extracts feature information indicating features of a video, and a target person appearing in the video. by Linguistic information related to behavioral intentions and a knowledge graph feature that is a tensor indicating the feature of the node based on information including a node that is linguistic information explaining a specific section of the video and an edge that indicates a relationship between the linguistic information of the node. Extract Knowledge Graph Features an extraction unit, the feature information and the Knowledge Graph Features Based on the above, in the video The aforementioned An estimation unit that estimates an area that is attracting attention from a subject.

[0007] A video processing method according to one aspect of the present invention is performed by a video processing device. circle The method includes: extracting feature information indicating features of the video by an image feature extraction unit of the video processing device; Knowledge Graph Features The extraction unit extracts the target person who appears in the video. by Linguistic information related to behavioral intentions and a knowledge graph feature that is a tensor indicating the feature of the node based on information including a node that is linguistic information explaining a specific section of the video and an edge that indicates a relationship between the linguistic information of the node. and an estimation unit of the video processing device extracts the feature information and the Knowledge Graph Features Based on the above, in the video The aforementioned Estimate the areas that are of interest to the subject. [Effects of the Invention]

[0008] According to the present invention, a gaze area is estimated taking into consideration a person's behavioral intention, and it is possible to estimate even a gaze area that has little relevance to the current behavior. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an application example of a video processing device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of the concept of each function of a video processing device according to an embodiment of the present invention. [Figure 3] FIG. 3 is a flowchart showing a first example of a learning processing routine executed by the video processing device. [Figure 4] FIG. 4 is a flowchart showing a second example of the learning processing routine executed by the video processing device. [Figure 5] FIG. 5 is a flowchart showing a first example of an inference processing routine executed by the video processing device. [Figure 6] FIG. 6 is a flowchart showing a second example of the inference processing routine executed by the video processing device. [Figure 7] FIG. 7 is a block diagram showing an example of the hardware configuration of a video processing device according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of the present invention will be described below with reference to the drawings. <Configuration> First, the configuration of a video processing device according to an embodiment of the present invention will be described. Fig. 1 is a diagram showing an application example of a video processing device according to an embodiment of the present invention.

[0011] The video processing device 100 shown in FIG. 1 is configured as a computer having a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory) in which a program for executing the video processing routine described below is stored, and is functionally configured as follows.

[0012] As shown in FIG. 1, the video processing device 100 according to this embodiment includes a memory unit 1, a video feature extraction unit 2, a knowledge graph feature extraction unit 3, an intermediate feature extraction unit 4, a gaze area estimation unit 5, a behavioral intention estimation unit 6, a parameter update unit 7, and an output unit 8.

[0013] The memory unit 1 stores an image x, an external knowledge graph y, a correct gaze area z, a correct behavioral intention w, a neural network of an image feature extractor A, a neural network of a knowledge graph feature extractor B, a neural network of an intermediate feature extractor C, a neural network of a gaze area estimator D, and a neural network of a behavioral intention estimator E.

[0014] FIG. 2 is a diagram showing an example of the concept of each function of a video processing device according to an embodiment of the present invention. Figure 2 shows an example of the conceptual configuration of video x, external knowledge graph y, correct gaze area z, correct behavioral intention w, the neural network of video feature extractor A, the neural network of knowledge graph feature extractor B, the neural network of intermediate feature extractor C, the neural network of gaze area estimator D, and the neural network of behavioral intention estimator E.

[0015] The image x may be any data that constitutes certain image data, such as a collection of images of multiple frames. The image x may also be provided with depth information for each frame and optical flow information that indicates changes in appearance between frames.

[0016] In the external knowledge graph y, nodes are linguistic information that explain specific sections of video x, and edges are labels that indicate the relationships between the linguistic information of each node, and the data composed of these nodes and edges can be any linguistic information related to the information on the subject's behavioral intentions that appear in video x. For example, if the behavior shown in video x is related to cooking, then the nodes of the external knowledge graph y are sentences that show the cooking steps, and the edges are labels that indicate the relationships between the cooking steps that correspond to each node.

[0017] The correct gaze area z may be any numerical data indicating the area in the video x that the subject person focused on when the data of the video x was acquired. For example, when video x is a set of images with N frames, correct gaze area z is a set of N maps with the same size as the images and a range of [0, 1].

[0018] The correct behavioral intention w may be any numerical data indicating each node in the external knowledge graph y related to a specific section related to video x. For example, if the external knowledge graph y has 10 nodes and each node is assigned an ID (identification) from 0 to 9, the correct behavioral intention w is a binary vector of length 10. When a specific section of video x is related to a node in the external knowledge graph y with an ID of 2, the value of "index 2" of the correct behavioral intention w is "1," and the rest are 0.

[0019] The image feature extractor A may be any neural network that inputs an image x and outputs a tensor that indicates the features of the image x, such as the first five layers of the I3D (Two-Stream Inflated 3D ConvNet) disclosed in the above-mentioned Non-Patent Document 2.

[0020] The image feature a may be any tensor that indicates the characteristics of the image x, for example, a tensor that is output when the image x is input to the first five layers of I3D disclosed in Non-Patent Document 2.

[0021] The knowledge graph feature extractor B may be any neural network that inputs an external knowledge graph y and outputs tensors that indicate the features of each node in the external knowledge graph y, such as the GCN (Graph Convolutional Networks) disclosed in the above-mentioned non-patent document 3.

[0022] The knowledge graph feature b can be any tensor that indicates the features of each node in the external knowledge graph y, and is, for example, a tensor that is output when the external knowledge graph y is input to the GCN disclosed in Non-Patent Document 3.

[0023] The intermediate feature extractor C may be any neural network that receives the video feature a and the knowledge graph feature b, estimates the strength of the relationship between the video feature a and the knowledge graph feature b, and extracts features while weighting the stronger relationship. The intermediate feature extractor C is, for example, the Transformer Network disclosed in Non-Patent Document 4.

[0024] The intermediate feature c may be any tensor that indicates a feature extracted while weighting the part of the strong relationship between the video feature a and the knowledge graph feature b. The intermediate feature c is, for example, a tensor that is output when the video feature a and the knowledge graph feature b are input to the Transformer Network disclosed in Non-Patent Document 4.

[0025] The gaze area estimator D may be any neural network that receives the intermediate feature c as input and estimates the area in the video x that the person is paying attention to. For example, when the intermediate feature c is a tensor of size "w'×h'×d'×f'", the gaze area estimator D is a neural network that inputs a tensor of size "w'×h'×d'×f'" and outputs a tensor of size "w''×h''×f''. In this case, the sizes "w'' and h''" indicate the horizontal and vertical lengths of the image, and f' indicates the number of frames.

[0026] The gaze estimation result d is the output when the intermediate feature c is input to the gaze area estimator D, and can be any numerical data that indicates the area in the video x where a person is paying attention. For example, the gaze estimation result d is a tensor of size "w'' × h'' × f'', where w'' and h'' indicate the horizontal and vertical lengths of the video. Furthermore, f' indicates the number of frames, and is a map with a range of [0,1] for each frame.

[0027] The behavioral intention estimator E may be any neural network that inputs intermediate feature c and estimates nodes related to video x in the external knowledge graph y, i.e., the intention of a person's behavior in video x. For example, when intermediate feature c is a tensor of size "w' x h' x d' x f'" and the number of nodes in the external knowledge graph y is 10, the behavioral intention estimator E is a neural network that inputs a tensor of size "w' x h' x d' x'" and outputs a vector of size 10. In this case, each value of the output vector of size 10 indicates the strength of the association between each node in the external knowledge graph y and video x.

[0028] The behavioral intention estimation result e may be any numerical data indicating a node related to video x in the external knowledge graph y. For example, when the number of nodes in the external knowledge graph y is 10, the behavioral intention estimation result e is a vector with a size of 10 and a range of [0, 1]. In this case, each value of the vector of the behavioral intention estimation result e indicates the strength of association with video x.

[0029] The video feature extraction unit 2 receives the video x and the video feature extractor A from the storage unit 1, inputs the video x to the video feature extractor A, and outputs the video feature a.

[0030] The knowledge graph feature extraction unit 3 receives the external knowledge graph y and the knowledge graph feature extractor B from the storage unit 1, inputs the external knowledge graph y to the knowledge graph feature extractor B, and outputs the knowledge graph feature b.

[0031] The intermediate feature extraction unit 4 receives the intermediate feature extractor C from the memory unit 1, receives the video feature a from the video feature extraction unit 2, and receives the knowledge graph feature b from the knowledge graph feature extraction unit 3, inputs the video feature a and the knowledge graph feature b to the intermediate feature extractor C, and outputs the intermediate feature c.

[0032] The gaze area estimation unit 5 receives the gaze area estimator D from the storage unit 1, receives the intermediate feature c from the intermediate feature extraction unit 4, inputs the intermediate feature c to the gaze area estimator D, and outputs the gaze area estimation result d.

[0033] The behavioral intention estimation unit 6 receives the behavioral intention estimator E from the storage unit 1, receives the intermediate feature c from the intermediate feature extraction unit 4, inputs the intermediate feature c to the behavioral intention estimator E, and outputs the behavioral intention estimation result e.

[0034] The parameter update unit 7 receives the video feature extractor A, the knowledge graph feature extractor B, the intermediate feature extractor C, the gaze area estimator D, the behavioral intention estimator E, the correct gaze area z, and the correct behavioral intention w from the memory unit 1, receives the gaze area estimation result d from the gaze area estimation unit 5, and receives the behavioral intention estimation result e from the behavioral intention estimation unit 6, and updates the parameters of each neural network so as to satisfy the following two constraints:

[0035] The first constraint is to update the parameters of each neural network so that the gaze area estimation result d and the correct gaze area z approach or become the same, and any learning method configured to satisfy this constraint is acceptable. The difference between the gaze area estimation result d and the correct gaze area z is also called gaze area loss. In this case, each neural network may be a video feature extractor A, a knowledge graph feature extractor B, an intermediate feature extractor C, a gaze area estimator D, and a behavioral intention estimator E; a knowledge graph feature extractor B, an intermediate feature extractor C, a gaze area estimator D, and a behavioral intention estimator E; a video feature extractor A, an intermediate feature extractor C, a gaze area estimator D, and a behavioral intention estimator E; an intermediate feature extractor C, a gaze area estimator D, and a behavioral intention estimator E; or an intermediate feature extractor C and a gaze area estimator D.

[0036] For example, the parameter update unit 7 calculates the squared error between the gaze area estimation result d and the correct gaze area z, and updates the parameters of the video feature extractor A, the knowledge graph feature extractor B, the intermediate feature extractor C, the gaze area estimator D, and the behavioral intention estimator E so as to reduce this error or set it to zero.

[0037] The second constraint is to update the parameters of each neural network so that the behavioral intention estimation result e and the correct behavioral intention w approach or become the same, and any learning method configured to satisfy this constraint is acceptable. The difference between the behavioral intention estimation result e and the correct behavioral intention w is also called behavioral intention loss. In this case, each neural network may be composed of a video feature extractor A, a knowledge graph feature extractor B, an intermediate feature extractor C, a gaze area estimator D, and a behavioral intention estimator E, or a knowledge graph feature extractor B, an intermediate feature extractor C, a gaze area estimator D, and a behavioral intention estimator E, or a video feature extractor A, an intermediate feature extractor C, a gaze area estimator D, and a behavioral intention estimator E, or an intermediate feature extractor C, a gaze area estimator D, and a behavioral intention estimator E, or an intermediate feature extractor C and a behavioral intention estimator E.

[0038] For example, the parameter update unit 7 calculates the cross-entropy loss between the behavioral intention estimation result e and the correct behavioral intention w according to the following formula (1), and updates the parameters of the video feature extractor A, the knowledge graph feature extractor B, the intermediate feature extractor C, the gaze area estimator D, and the behavioral intention estimator E so as to reduce or eliminate this error.

[0039]

number

[0040] Here, m in Equation (1) is the index of the behavioral intention estimation result e and the correct behavioral intention w, and y m is a value in the behavioral intention estimation result e, and t m is the value in the correct behavioral intention w. m In this example, only the index value for the correct class is "1", and the values ​​for other indexes are "0".

[0041] The output unit 8 receives the gaze area estimation result d from the gaze area estimation unit 5 and the behavioral intention estimation result e from the behavioral intention estimation unit 6, or receives only the gaze area estimation result d from the gaze area estimation unit 5, and outputs the estimation result. The estimation result may be only the gaze area estimation result d, or may include the behavioral intention estimation result e in addition to the gaze area estimation result d.

[0042] <Function of the image processing device> Next, the operation of the video processing device 100 in this embodiment will be described. The video processing device 100 executes a learning processing routine and an inference processing routine, which will be described below.

[0043] <<Learning processing routine>> First, a first example of the learning process routine will be described below: Fig. 3 is a flowchart showing a first example of the learning process routine executed by the video processing device. First, in step S101, the video feature extraction unit 2 receives the video x and the video feature extractor A from the storage unit 1. The video feature extractor A receives the video x and outputs the video feature a, which is then transmitted to the intermediate feature extractor 4 .

[0044] In step S102, the knowledge graph feature extraction unit 3 receives the external knowledge graph y and the knowledge graph feature extractor B from the storage unit 1. The knowledge graph feature extractor B receives the external knowledge graph y as an input, outputs the knowledge graph feature b, and transmits it to the intermediate feature extractor 4 .

[0045] In step S103, the intermediate feature extraction unit 4 receives the intermediate feature extractor C from the storage unit 1, receives the video feature a from the video feature extraction unit 2, and receives the knowledge graph feature b from the knowledge graph feature extraction unit 3. The intermediate feature extractor C receives the video feature a and the knowledge graph feature b, outputs the intermediate feature c, and transmits it to the gaze area estimation unit 5 and the behavioral intention estimation unit 6.

[0046] In step S104, the gaze area estimation unit 5 receives the gaze area estimator D from the storage unit 1 and receives the intermediate feature c from the intermediate feature extraction unit 4. The gaze area estimator D receives the intermediate feature c and outputs the gaze area estimation result d, which is transmitted to the parameter update unit 7.

[0047] In step S105, the behavioral intention estimation unit 6 receives the behavioral intention estimator E from the storage unit 1 and receives the intermediate feature c from the intermediate feature extraction unit 4. The behavioral intention estimator E receives the intermediate feature c, outputs the behavioral intention estimation result e, and transmits it to the parameter update unit 7.

[0048] In step S106, the parameter update unit 7 receives the video feature extractor A, the knowledge graph feature extractor B, the intermediate feature extractor C, the gaze area estimator D, the behavioral intention estimator E, the correct gaze area z, and the correct behavioral intention w from the storage unit 1, receives the gaze area estimation result d from the gaze area estimation unit 5, and receives the behavioral intention estimation result e from the behavioral intention estimation unit 6. The parameter update unit 7 updates the parameters of each neural network so as to satisfy the above two constraints.

[0049] Next, a second example of the learning process routine will be described below: Fig. 4 is a flowchart showing the second example of the learning process routine executed by the video processing device. First, steps S101, S102, S103, and S104 described in the first example of the learning processing routine are performed in this order.

[0050] In step S107, the parameter update unit 7 receives the video feature extractor A, the knowledge graph feature extractor B, the intermediate feature extractor C, the gaze area estimator D, and the correct gaze area z from the memory unit 1, receives the gaze area estimation result d from the gaze area estimation unit 5, and updates the parameters of each neural network so as to satisfy the first constraint above, which is the constraint regarding gaze area estimation.

[0051] <<Inference processing routine>> Next, a first example of an inference processing routine will be described below: Fig. 5 is a flowchart showing a first example of an inference processing routine executed by a video processing device. First, steps S101, S102, S103, S104, and S105 described in the first example of the learning processing routine are performed in this order.

[0052] Next, in step S206, the output unit 8 receives the gaze area estimation result d from the gaze area estimation unit 5 and the behavioral intention estimation result e from the behavioral intention estimation unit 6, and outputs these received gaze area estimation result d and behavioral intention estimation result e.

[0053] Next, a second example of the inference processing routine will be described below: Fig. 6 is a flowchart showing the second example of the inference processing routine executed by the video processing device. First, steps S101, S102, S103, and S104 described in the first example of the learning processing routine are performed in this order.

[0054] Next, in step S207, the output unit 8 receives the gaze area estimation result d from the gaze area estimation unit 5, and outputs the received gaze area estimation result d.

[0055] In one embodiment of the present invention, behavioral intention information is input to a gaze area estimator along with video features. Therefore, according to one embodiment of the present invention, in relation to a person's behavioral intentions, such as "I want to do XX" or "Let's do XX," and the person's gaze area in the video, it is possible to estimate even gaze areas that have a weak correlation with the person's current behavior, thereby improving estimation accuracy. That is, in one embodiment of the present invention, a gaze area is estimated taking into consideration a person's behavioral intention, and it is possible to estimate even a gaze area that has little relevance to the current behavior.

[0056] FIG. 7 is a block diagram showing an example of the hardware configuration of a video processing device according to an embodiment of the present invention. 7, the video processing device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU. A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to the hardware processor 111A via a bus 115.

[0057] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from the communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0058] The input / output interface 113 is connected to an input device 200 and an output device 300 that are attached to the video processing device 100 and used by a user or the like. The input / output interface 113 can take in operation data input by a user or the like through an input device 200 such as a keyboard, a touch panel, a touchpad, or a mouse, and can also process output data to an output device 300 including a display device using a liquid crystal or an organic electroluminescence (EL) display, etc. Note that the input device 200 and the output device 300 may be devices built into the video processing device 100, or may be input devices and output devices of other information terminals that can communicate with the video processing device 100 via a network NW.

[0059] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and a non-volatile memory such as a ROM, and can store programs required to execute various control processes, etc., according to one embodiment.

[0060] The data memory 112 is a tangible storage medium, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM, and can be used to store various data or information acquired and created during various processes.

[0061] A video processing device 100 according to an embodiment of the present invention can be configured as a data processing device having the units shown in FIG. 1 as software-based processing function units.

[0062] Each information storage unit used as a work memory or the like by each unit of the video processing device 100 can be configured by using the data memory 112 shown in Fig. 7. However, these configured storage areas are not essential components within the video processing device 100, and may be areas provided in an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0063] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0064] The methods described in each embodiment may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), or may be transmitted and distributed via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by having the operation controlled by this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0065] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention. [Explanation of symbols]

[0066] 100...Video processing device 1...Storage section 2...Video feature extraction unit 3...Knowledge graph feature extraction unit 4...Intermediate feature extraction unit 5... Gaze area estimation unit 6...Behavioral intention estimation unit 7...Parameter update section 8...Output section

Claims

1. a video feature extraction unit that extracts feature information indicating features of the video; a knowledge graph feature extraction unit that extracts knowledge graph features, which are tensors that indicate the characteristics of nodes, based on linguistic information related to the behavioral intention of the subject that appears in the video, the linguistic information including nodes that explain specific sections of the video and edges that indicate relationships between the linguistic information of the nodes; an estimation unit that estimates an area in the video that is focused on by the subject based on the feature information and the knowledge graph feature; A video processing device comprising:

2. The estimation unit Inferring an intention of a behavior by the subject in the video based on the feature information and the knowledge graph features; The video processing device according to claim 1 .

3. a relation feature extraction unit that extracts features based on the strength of the relationship between the feature information and the knowledge graph features; The estimation unit estimating an area in the video that is focused on by the subject based on the feature based on the strength of the relationship; The video processing device according to claim 1 .

4. The estimation unit inputting the feature information and the knowledge graph features into a neural network, and estimating an area in the video that is focused on by the subject based on the input result; an updating unit that updates parameters of the neural network so that an area in the video that is focused on by the subject, estimated using the neural network, approaches correct information about an area in the video that is focused on by the subject; The video processing device according to claim 1 .

5. The estimation unit inputting the feature information and the knowledge graph features into a neural network, and estimating the intention of the action of the subject in the video based on the result of this input; an updating unit that updates parameters of the neural network so that the intention of the behavior of the subject in the video, estimated using the neural network, approaches correct information of the intention of the behavior of the subject in the video; The video processing device according to claim 2 .

6. A method performed by a video processing device, comprising: extracting feature information indicating features of the video by a video feature extraction unit of the video processing device; A knowledge graph feature extraction unit of the video processing device extracts knowledge graph features, which are tensors indicating the characteristics of the nodes, based on information including linguistic information related to the behavioral intention of the subject that appears in the video, the linguistic information being nodes that explain specific sections of the video, and edges that indicate the relationship between the linguistic information of the nodes; an estimation unit of the video processing device estimates an area in the video that is focused on by the subject based on the feature information and the knowledge graph feature; Image processing method.

7. The estimation unit Inferring an intention of a behavior by the subject in the video based on the feature information and the knowledge graph features; The video processing method according to claim 6.

8. A video processing program that causes a processor to function as each unit of the video processing device according to any one of claims 1 to 5.

Images