Video understanding method, system and device based on global and local changes and medium

Abstract

The invention discloses a video understanding method, system and device based on global and local changes and a medium. The method comprises the following steps: constructing graph representation containing a target object on different frame pictures of a video; training the graph representation containing the target object in the video frame through a graph neural network to obtain action transfer information between adjacent graph representations; splicing the motion transfer information representations of all the frame pictures of the training video into a unified motion change matrix; and inputting the action change matrix into a Transform model, and extracting a change relationship between different implicit representations of the video frame pictures. According to the method, a target object on a whole video frame picture is utilized to construct a corresponding graph representation; after the action change trend on a single frame of picture is obtained by utilizing graph neural network training, the action relationship on different frames of pictures is obtained through a Transform model, focusing on the moment when the action change is most obvious, and the accuracy of video understanding is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video understanding method, system, device and medium based on global and local changes. Background Art

[0002] At present, there are many applications involving video understanding technology, such as pedestrian action recognition, gesture recognition, etc. How to effectively understand human behavior and actions in video clips has become the key to video understanding technology. Some studies, such as the invention application with patent publication number CN111898458A, proposed a violent video recognition method based on attention mechanism-based dual-modal task learning, in which a deep neural network with an attention mechanism module is used to extract the features of violent videos in single-frame images and motion optical flows, and a post-fusion average fusion strategy is used to achieve the final classification decision of violent and bloody videos; the invention application with patent publication number CN110532960A proposed a target-assisted action recognition method based on graph neural network, in which the features of each target bounding box corresponding to the target in the video frame are extracted to construct a graph model; then a mapping function is constructed to automatically calculate the similarity between any two nodes, and the similarity is used to control the iterative update of the graph nodes, and then the target features and the features of the video frame sequence are used to classify the action.

[0003] However, these existing technical means still have some shortcomings: the methods for video understanding and video recognition only consider the motion relationship between different video frames, or the relationship between targets within a single video frame, but do not combine the two well. Summary of the invention

[0004] The purpose of the present invention is to provide a video understanding method, system, device and medium based on global and local changes to solve the above-mentioned shortcomings of existing video understanding methods.

[0005] To achieve the above purpose, the technical solution provided by the present invention is:

[0006] The first aspect of the present application provides a video understanding method based on global and local changes, comprising the following steps:

[0007] Construct a graph representation containing the target object on different frames of the video;

[0008] The graph representation of the target object in the video frame is trained through the graph neural network to obtain the action transfer information between adjacent graph representations;

[0009] The motion transfer information representations of all frames of the training video are concatenated into a unified motion change matrix;

[0010] The action change matrix is ​​input into the Transformer model to extract the change relationship between different implicit representations of the video frame images.

[0011] To optimize the above technical solutions, the specific measures taken also include:

[0012] When the action change matrix is ​​input into the Transformer model, a multi-layer perceptron is used to suppress possible noise and interference.

[0013] The Transformer model obtains the corresponding video label through the softmax module.

[0014] The construction of a graph representation containing the target object on different frames of the training video is specifically as follows:

[0015] Get video frame (X i ,y i ) and store them in a bucket, with a total of Z i target objects; then calculate the similarity between all the target objects and remove the targets with too high similarity, so that the number of targets in the entire bucket reaches the specified number K, and the insufficient ones are filled with the diagonal matrix I;

[0016] Each frame is T means that each training video clip has T frames, and a specific target object extracted from different frames is represented as Calculate the specified target With other targets The similarity ε is:

[0017]

[0018] Among them, t represents the tth frame in T frames, Z i represents the number of targets, exp() represents the natural exponential function, and σ represents the heat core value;

[0019] When the objects on all video frames are placed in the bucket and processed, a graph representation of each frame of video is constructed:

[0020]

[0021] in, The graph representation of the t-th frame image, D represents the specified target object obtained in each frame, and E represents the set of edges;

[0022] By filling the diagonal matrix I into the graph, the obtained graph representation can maximize the collection of local action information on a single video frame and utilize the target information in the entire video.

[0023] The graph neural network is used to train the graph representation of the target object in the video frame to obtain the action transfer information between adjacent graph representations, specifically:

[0024] By extracting the relationship between adjacent graph representations, the trend of action changes can be captured:

[0025]

[0026] in, represents the trend of action change, Φ() represents the action change capturer, GNN() represents the graph neural network, The graph representation of the t-th frame image, The graph representation of the t+1th frame image;

[0027] The optimal graph representation for the t-th frame of the graph neural network encoding output is Q, The optimal graph of the t+1th frame is represented as G.

[0028] Assume that there is a mapping matrix R such that Q = GR + E, where E is the transfer noise between different graph representations, and construct the constraint function:

[0029]

[0030] sqF T R=I

[0031] Where F represents the F norm and I represents the diagonal matrix; then we can solve R = UV T , where U and V are matrices G T The left and right singular value matrices obtained after SVD decomposition of Q are obtained by the above formula

[0032] The motion transfer information representation of all frames of the training video is spliced ​​into a unified motion change matrix, specifically:

[0033] All action change trends The action transfer matrix A is integrated into a whole:

[0034]

[0035] Where i represents the video frame, T represents each training video segment has T frames, Represents the action transfer matrix between the average fusion graph over all graph representations and the graph representation constructed for the target object in the bucket;

[0036] In order to ensure the construction of a reasonable action transfer matrix A, the number of frames extracted from the training data is set to n 2 , n is a positive integer.

[0037] The action change matrix is ​​input into the Transformer model to extract the change relationship between different implicit representations of the video frame pictures, specifically: the action transfer matrix A is processed to obtain the global action focus, and the corresponding action representation H is output. i , where the input token is the change trend of all actions that constitute the action transfer matrix

[0038] A second aspect of the present application provides a video understanding system based on global and local changes, comprising:

[0039] The video frame graph representation construction module is used to construct graph representations containing target objects on different frames of the video;

[0040] The action transfer information acquisition module is used to train the graph representation of the target object in the video frame through the graph neural network to obtain the action transfer information between adjacent graph representations;

[0041] The motion extraction module is used to concatenate the motion transfer information representations of all frames of the training video into a unified motion change matrix;

[0042] A change relationship acquisition module, used to input the action change matrix into the Transformer model to extract the change relationship between different implicit representations of the video frame pictures;

[0043] The global and local change based video understanding system is used to execute the global and local change based video understanding method as described above.

[0044] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the video understanding method based on global and local changes as described in any one of claims 1 to 7 is implemented.

[0045] A fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the video understanding method based on global and local changes as described above.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention constructs a graph representation containing the target object on different frames of the training video, trains the graph representation containing the target object in the video frame through a graph neural network, obtains the action transfer information between adjacent graph representations, and then splices the action transfer representations of all frames into a unified action change matrix and inputs it into the Transformer model to extract the change relationship between different implicit representations. The present invention also suppresses possible noise and interference through a multi-layer perceptron, and obtains the corresponding video label through a softmax module.

[0048] In general video understanding methods, only the target object on a single video frame is used to construct the corresponding graph representation. The present invention uses the target object on the entire video frame to construct the corresponding graph representation; after using graph neural network (GNN) training to obtain the action change trend on a single frame image, the action relationship on different frame images is obtained through the Transformer model, so that the Transformer model can focus on the moment when the action change is most obvious, thereby improving the accuracy of video understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 : Flow chart of the video understanding method based on global and local changes of the present invention.

[0050] Figure 2 : An example diagram of constructing a graph representation in the present invention.

[0051] Figure 3 : An example diagram of the transfer and change of the target object's pre-action and post-action in the present invention. DETAILED DESCRIPTION

[0052] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0053] Some of the terms in the present invention are explained as follows:

[0054] GNN (Graph Neural Network): A neural network used to process graph-structured data, where entities are nodes and the relationships between entities are represented by edges. The goal of GNN is to learn useful representations from graph-structured data and use these representations for various tasks.

[0055] In one embodiment, the present invention provides a video understanding method based on global and local changes, such as Figure 1As shown in the figure, a graph representation containing the target object is constructed on different frames of the video; the graph representation containing the target object in the video frame is trained through a graph neural network to obtain the action transfer information between adjacent graph representations; the action transfer information representation of all frame images of the training video is spliced ​​into a unified action change matrix; the action change matrix is ​​input into the Transformer model to extract the change relationship between different implicit representations of the video frame images.

[0056] When inputting the action change matrix into the Transformer model, a multi-layer perceptron is used to suppress possible noise and interference.

[0057] The Transformer model obtains the corresponding video label through the softmax module.

[0058] The present invention is further described in detail below in conjunction with a specific embodiment:

[0059] (1) Data collection and processing

[0060] The training data sets used in this embodiment are all from public data sets, and 16 frames of data are extracted from each video clip and standardized to a uniform size.

[0061] (2) Construction of graph representation

[0062] In general video understanding methods, only the target object on a single video frame is used to construct the corresponding graph representation. In this application, the target object on the entire video frame is used to construct the corresponding graph representation, specifically:

[0063] Use a general target detector to obtain the training dataset video frames (X i ,y i ) and store it in a bucket, such as Figure 2 As shown, there are a total of Z i Then, the similarity between all the targets is calculated and some targets with too high similarity are removed, so that the number of targets in the entire bucket reaches the specified number K, and the insufficient ones are filled with the diagonal matrix I. Each frame of the picture is and Indicates that each training video clip has T frames, and the target objects extracted from different frames The similarity between the target object and other targets is calculated as:

[0064]

[0065] t represents the tth frame in T frames, Z iRepresents the number of targets, exp() represents the natural exponential function, and σ represents the heat kernel value. When all targets on all video frames are placed in the bucket and processed, a graph representation of each frame of video is constructed:

[0066]

[0067] in, It represents the graph representation of the t-th frame image, D represents the specified target object obtained on each frame, that is, the target object obtained by the Detector on each frame. Since the target object extracted on each frame of video is not complete relative to the target in the entire video clip, this application uses a diagonal matrix I to fill in the graph. The graph representation obtained in this way can maximize the collection of local action information on a single video frame, while also utilizing the target information in the entire video; E represents a set of edges.

[0068] (3) Generate action transfer matrix

[0069] Generally speaking, the actions between adjacent video frames may have more repetitive problems, that is, the action changes are relatively smooth. For a series of continuous video clips, it can be analogized to the corresponding quadratic function, X i =ay i 2 +by i +c, its first-order derivative can be regarded as the corresponding action function. If its first-order derivative ay i +b indicates a specific action, which will lead to problems such as unclear action extraction and noise inclusion.

[0070] Since there is a certain temporal action connection between different video frames, that is, the graph representation of different video frames There is also a corresponding action transfer between them, that is, this application captures the trend of action changes by extracting the relationship between adjacent graph representations:

[0071]

[0072] in, represents the trend of action change, Φ() represents the action change capturer, GNN() represents the graph neural network, The graph representation of the t-th frame image, The graph representation of the t+1th frame image;

[0073] Specifically, for the optimal graph representation of GNN encoding output Assuming that there is a mapping matrix R, such that Q = GR + EE is the transfer noise between different graph representations, the constraint function can be constructed:

[0074]

[0075] sqF T R=I

[0076] F represents the F norm, I represents the diagonal matrix; then we can solve R = UV T , where U and V are matrices G T The left and right singular value matrices of Q are obtained after SVD decomposition (the correctness of this solution has been discussed in detail in Zhang C, Fu H, Hu Q, et al. Generalized latent multi-view subspace clustering [J]. IEEE transactions on pattern analysis and machine intelligence, 2018, 42 (1): 86-99), which can be obtained by the above formula "ST" is an abbreviation of "such that" in English. It is used to indicate that what follows is a condition or a group of conditions stated previously.

[0077] Since the action corresponding to the actual label in the video clip may only exist in some intermediate frames, such as Figure 1 As shown in , the information on the previous and next frames is only some pre- and post-action preparations, and the corresponding action changes are also relatively smooth, which interferes with the acquisition of the real video action features. To obtain the action representation of the entire training video, just like obtaining the video function X i =ay i 2 +by i +c on the second-order derivative, this application uses all the action changes to represent The action transfer matrix A is integrated into a whole:

[0078]

[0079] Represents the action transfer matrix between the average fusion graph on all graph representations and the graph representation constructed by the target object in the bucket. To ensure the construction of a reasonable action transfer matrix A, the number of frames extracted from the training data is set to n 2 , n is a positive integer.

[0080] (4) Extracting global action representation

[0081] Use the Transformer module to obtain the relationship between different action change trends, that is, process the action transfer matrix A to obtain the global action focus and output the corresponding action representation H i , where the input token is the change trend of all actions that constitute the action transfer matrix

[0082] (5) Output the label and calculate the loss with the true label.

[0083] like Figure 3 As shown, in a video of a shot put, the present application can obtain the transfer changes of the preceding action and the following action in the continuous action video frame pictures of the shot put, so as to focus on the moment when the action change is most obvious and improve the accuracy of video understanding.

[0084] The present invention obtains the local object relationship on a single video frame by constructing a graph representation of the object on a single video frame, and obtains the global object relationship on the entire video clip by constructing a graph representation of the object on all videos. The corresponding action change trend is obtained by extracting the change relationship between the graph representations on adjacent video frames. Obtain the global action transfer matrix by extracting the target object changes between all local graph representations and the global graph representation

[0085] Encoding all action change trends through the Transformer model and the global action transfer matrix The global relationship between them is used to obtain the action representation of the entire training video clip. This representation integrates the local and global action change trends on the training dataset, combined with the transfer changes of the preceding action and the following action, and can better focus on the area where the real action changes, thereby improving the accuracy of video understanding.

[0086] In another embodiment, the present invention proposes a video understanding system based on global and local changes, comprising:

[0087] The video frame graph representation construction module is used to construct graph representations containing target objects on different frames of the video;

[0088] The action transfer information acquisition module is used to train the graph representation of the target object in the video frame through the graph neural network to obtain the action transfer information between adjacent graph representations;

[0089] The motion extraction module is used to concatenate the motion transfer information representations of all frames of the training video into a unified motion change matrix;

[0090] The change relationship acquisition module is used to input the action change matrix into the Transformer model and extract the change relationship between different implicit representations of video frame images;

[0091] In another embodiment, the present invention proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned video understanding method based on global and local changes is implemented.

[0092] In another embodiment, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the above-mentioned video understanding method based on global and local changes.

[0093] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0094] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A video understanding method based on global and local changes, characterized in that: The following steps are involved: Construct a graph representation containing the target object on different frames of the video; The graph representation of the target object in the video frame is trained through the graph neural network to obtain the action transfer information between adjacent graph representations; The motion transfer information representations of all frames of the training video are concatenated into a unified motion change matrix; The action change matrix is ​​input into the Transformer model to extract the change relationship between different implicit representations of the video frame images.

2. The video understanding method based on global and local changes according to claim 1, characterized in that: When the action change matrix is ​​input into the Transformer model, a multi-layer perceptron is used to suppress possible noise and interference.

3. The video understanding method based on global and local changes according to claim 1, characterized in that: The Transformer model obtains the corresponding video label through the softmax module.

4. The video understanding method based on global and local changes according to claim 1, characterized in that: The construction of a graph representation containing the target object on different frames of the training video is specifically as follows: Get video frame (X i ,y i ) and store them in a bucket, with a total of Z i target objects; then calculate the similarity between all the target objects and remove the targets with too high similarity, so that the number of targets in the entire bucket reaches the specified number K, and the insufficient ones are filled with the diagonal matrix I; Each frame is T means that each training video clip has T frames, and a specific target object extracted from different frames is represented as Calculate the specified target With other targets The similarity ε is: Among them, t represents the tth frame in T frames, Z i represents the number of targets, exp( ) represents the natural exponential function, and σ represents the heat core value; When the objects on all video frames are placed in the bucket and processed, a graph representation of each frame of video is constructed: in, The graph representation of the t-th frame image, D represents the specified target object obtained in each frame, and E represents the set of edges; By filling the diagonal matrix I into the graph, the obtained graph representation can maximize the collection of local action information on a single video frame and utilize the target information in the entire video.

5. The video understanding method based on global and local changes according to claim 1, characterized in that: The graph neural network is used to train the graph representation of the target object in the video frame to obtain the action transfer information between adjacent graph representations, specifically: By extracting the relationship between adjacent graph representations, the trend of action changes can be captured: in, represents the trend of action change, Φ( ) represents the action change capturer, GNN( ) represents the graph neural network, The graph representation of the t-th frame image, The graph representation of the t+1th frame image; The optimal graph representation for the t-th frame of the graph neural network encoding output is Q, The optimal graph of the t+1th frame is represented as G. Assume that there is a mapping matrix R such that Q = GR + E, where E is the transfer noise between different graph representations, and construct the constraint function: s.t.R T R=I Where F represents the F norm and I represents the diagonal matrix; then we can solve R = UV T , where U and V are matrices G T The left and right singular value matrices obtained after SVD decomposition of Q are obtained by the above formula 6. The video understanding method based on global and local changes according to claim 1, characterized in that: The motion transfer information representation of all frames of the training video is spliced ​​into a unified motion change matrix, specifically: All action change trends The action transfer matrix A is integrated into a whole: Where i represents the video frame, T represents each training video segment has T frames, Represents the action transfer matrix between the average fusion graph over all graph representations and the graph representation constructed for the target object in the bucket; In order to ensure the construction of a reasonable action transfer matrix A, the number of frames extracted from the training data is set to n 2 , n is a positive integer.

7. The video understanding method based on global and local changes according to claim 1, characterized in that: The action change matrix is ​​input into the Transformer model to extract the change relationship between different implicit representations of the video frame pictures, specifically: the action transfer matrix A is processed to obtain the global action focus, and the corresponding action representation H is output. i , where the input token is the change trend of all actions that constitute the action transfer matrix 8. A video understanding system based on global and local changes, characterized in that: include: The video frame graph representation construction module is used to construct graph representations containing target objects on different frames of the video; The action transfer information acquisition module is used to train the graph representation of the target object in the video frame through the graph neural network to obtain the action transfer information between adjacent graph representations; The motion extraction module is used to concatenate the motion transfer information representations of all frames of the training video into a unified motion change matrix; A change relationship acquisition module, used to input the action change matrix into the Transformer model to extract the change relationship between different implicit representations of the video frame pictures; The global and local change based video understanding system is used to execute the global and local change based video understanding method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the video understanding method based on global and local changes as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the video understanding method based on global and local changes according to any one of claims 1 to 7.

Images