Video missing information completion method and device based on high-order structure intervention

Abstract

The invention provides a video missing information complementing method and device based on high-order structure intervention, and the method comprises the steps: carrying out the detection of video data, and obtaining a missing information position; expressing a space-time relationship in the video number by using a high-order graph structure to obtain a high-order tensor; learning the high-order structure correlation between the nodes in the high-order tensor and the feature correlation between the nodes, and intervening the feature correlation by using the node structure correlation to obtain a structure intervention attention tensor; simultaneous transmission of time and space messages is carried out on the high-order tensor based on the structure intervention attention, and final node feature representation is obtained; and performing missing information calculation based on the final node feature representation, and complementing the video data based on a missing information calculation result and a missing information position. According to the method, the spatio-temporal information in the video data can be captured more accurately by learning the high-order structure correlation and the feature correlation in the video data, so that the accuracy of video missing information completion is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video data completion, and in particular to a method and device for completing missing video information based on high-order structure intervention. Background Art

[0002] Image recognition technology is a key application and research focus in the fields of computer engineering, multimedia information processing, machine learning, and pattern recognition. It is also one of the main challenges in the fields of computer science and artificial intelligence. This technology covers multiple research directions, among which the recovery of missing information in video image data is a core issue. Video data can be converted into advanced graph structure data. With the help of advanced structural intervention technology, we can conduct in-depth analysis of this data, reveal the complex structural information hidden in it, and then achieve efficient and accurate completion of missing information in the video.

[0003] In the process of video data acquisition, due to the influence of various factors (such as camera failure, sensor delay, network packet loss, etc.), video data often has missing information. Therefore, it is necessary to complete these missing information to restore the integrity of the video. Video data can be regarded as a collection of multi-frame image data, and each frame of image can be converted into a data representation of a graph structure. Based on this, graph neural network technology is used to effectively complete the missing information in video data. At present, graph neural network technology has been able to complete the missing information in video data. However, most of the existing graph neural network technologies are based on traditional graph neural network methods, which mainly model the spatial relationship between image feature blocks through spatial message passing. In fact, there are high-order structural correlations in the graph data structure sequence composed of video images, which will also have different effects on the results of video completion, thus resulting in the low accuracy of the video data completion method in the prior art. Summary of the invention

[0004] The present invention aims to at least solve the technical problem of low accuracy of the video data completion method in the prior art, and particularly innovatively proposes a method and device for completing missing video information based on high-order structural intervention.

[0005] In order to achieve the above-mentioned object of the present invention, the present invention provides a method for completing missing video information based on high-order structure intervention, the method comprising:

[0006] S1, collecting video data, and detecting the video data to obtain the position of missing information;

[0007] S2. Using a high-order graph structure to represent the spatiotemporal relationship in the video data to obtain a high-order tensor;

[0008] S3, learning the high-order structural correlation between nodes and the feature correlation between nodes in the high-order tensor, and intervening the feature correlation using the node structural correlation to obtain a structural intervention attention tensor;

[0009] S4, based on the structural intervention attention tensor, the high-order tensor is simultaneously transmitted in time and space to obtain a final node feature representation;

[0010] S5. Calculate the missing information based on the final node feature representation, and complete the video data based on the missing information calculation result and the missing information position.

[0011] As an optional embodiment of the present invention, optionally, obtaining the high-order tensor in step S2 includes:

[0012] S201, extracting frame image data from the video data;

[0013] S202, detecting objects in the frame image data based on an object detection algorithm, taking the objects as nodes, and extracting node features;

[0014] S203, constructing edges based on the spatial relationship between the nodes;

[0015] S204, constructing an adjacency matrix of the frame image data based on the node features and edges;

[0016] S205 , sorting the adjacency matrix in each frame of image data in chronological order to obtain a high-order tensor.

[0017] As an optional embodiment of the present invention, optionally, in step S3, the expression for learning the high-order structural correlation between the nodes in the high-order tensor is:

[0018]

[0019] in, represents the adjacency tensor reconstructed by CPTF, C, P and Q all represent factor matrices, which are used to learn the dynamic structural patterns in the adjacency tensor. Indicates that the tensor is decomposed into three factor matrices, R represents the number of factor vectors, r represents a natural number, represents a tensor of rank r, c r 、p r and q r Both represent the rth factor matrix, represents the tensor outer product operation, ||||2 represents the L2 norm calculation, A represents the adjacent tensor, represents the structural correlation between node i and node j in the tth data frame, SoftMax() represents normalization, represents the element at position (i, j, t) in the adjacency tensor A, exp() represents the exponential function, k represents any node in the direct neighbor set, N(i) represents the direct neighbor set of node i, {i} represents the node set consisting of node i, Represents the element at position (i,j,k) in the adjacency tensor A.

[0020] As an optional embodiment of the present invention, optionally, in step S3, the expression for learning the feature correlation between the nodes in the high-order tensor is:

[0021]

[0022] in, represents the feature correlation between node i and node j in the tth data frame, exp() represents the exponential function, σ() represents the activation function, represents the neighbor node set of node i in the tth video data frame, W t Represents the node feature transformation matrix of the tth video data frame, represents the neighbor node set of node j in the tth video data frame, [||] represents the feature vector concatenation operation, u T represents the transpose of the learning vector u, k represents any node in the direct neighborhood set, N(i) represents the direct neighborhood set of node i, {i} represents the node set composed of node i, Represents the set of neighbor nodes of node k in the tth video data frame.

[0023] As an optional embodiment of the present invention, optionally, the expression for obtaining the structural intervention attention in step S3 is:

[0024]

[0025] in, represents the importance measure of structural correlation in the tth data frame, exp() represents the exponential function, and Both represent the learnable vectors of the t-th data frame, represents the importance measure of feature correlation in the tth data frame, represents the structural intervention attention of node i and node j in the tth data frame, represents the structural correlation between node i and node j in the tth data frame, represents the feature correlation of the t-th data frame between node i and node j, k represents any node in the direct neighbor set, N(i) represents the direct neighbor set of node i, {i} represents the node set composed of node i, represents the structural correlation between node i and node k in the tth data frame, Represents the feature correlation between node i and node k in the tth data frame.

[0026] As an optional embodiment of the present invention, optionally, in step S4, the high-order tensor is simultaneously transmitted in time and space based on the structural intervention attention tensor, and the expression for obtaining the final node feature representation is:

[0027]

[0028]

[0029] Among them, H represents the embedding representation tensor of the node, σ() represents the activation function, O represents the structural intervention attention tensor, N represents the input original node feature tensor, W represents the node feature conversion vector, * represents the tensor multiplication operation, and represents different information tensors after time message transmission, τ() represents information aggregation in the time dimension of the information tensor, ×3 represents modulo three operation, M represents triangular mixed moment, M t,k represents the element value at position (t, k) in M, max() represents the maximum value, t represents the time position of the data frame, k represents any node in the direct neighbor set, and z represents the time window. represents the final feature vector of node i in the tth data frame, j represents the node, N(i) represents the direct neighborhood set of node i, {i} represents the node set composed of node i itself, represents the structural intervention attention of node i and node j in the tth data frame, represents the neighbor node set of node j in the tth video data frame, W t Represents the node feature transformation matrix of the t-th video data frame in the weight tensor W.

[0030] On the other hand, the present invention further provides a video missing information completion device based on high-order structure intervention, the device includes the video missing information completion method based on high-order structure intervention; the device also includes:

[0031] A receiving module, used for acquiring video data;

[0032] A data storage module, connected to the receiving module, used to store the video data and detect the video data to obtain the position of missing information;

[0033] A video high-order graph data extraction module, connected to the data storage module, for extracting high-order tensors from the video data;

[0034] A structural intervention module, connected to the video high-order graph data extraction module, is used to learn the high-order structural correlation between nodes and the feature correlation between nodes in the high-order tensor, and use the node structural correlation to intervene in the feature correlation to obtain a structural intervention attention tensor;

[0035] A high-order graph convolution module, connected to the structure intervention module, for simultaneously transmitting time and space messages to the high-order tensor according to the structure intervention attention tensor to obtain a final node feature representation;

[0036] A missing information calculation module is connected to the high-order graph convolution module and is used to calculate the missing information according to the final node feature representation, and to complete the video data based on the missing information calculation result and the missing information position.

[0037] As another optional embodiment of the present invention, optionally, the data storage module includes:

[0038] A video data storage unit, used for storing the video data;

[0039] A video missing information detection unit, used to detect the video data and obtain the missing information position;

[0040] The completed video data storage unit is used to store the completed video data.

[0041] As another optional embodiment of the present invention, optionally, the video high-order graph data extraction module includes:

[0042] A video data frame extraction unit, used to extract frame image data from the video data;

[0043] A node construction unit, connected to the video data frame extraction unit, for constructing nodes in the high-order tensor;

[0044] An edge construction unit, connected to the node construction unit, and configured to construct edges in the high-order tensor according to the spatial relationship between the nodes;

[0045] An information tensor construction unit is connected to the edge construction unit and is used to construct a high-order tensor according to the nodes and edges.

[0046] As another optional embodiment of the present invention, optionally, the structural intervention module includes:

[0047] A high-order structure correlation learning unit, connected to the video high-order graph data extraction module, for learning the high-order structure correlation between the nodes in the high-order tensor;

[0048] A node feature correlation learning unit, connected to the video high-order graph data extraction module, for learning the feature correlation between nodes in the high-order tensor;

[0049] The structure intervention unit is connected to the high-order structure correlation learning unit and the node feature correlation learning unit, and is used to intervene in the feature correlation by utilizing the node structure correlation to obtain structure intervention attention.

[0050] Beneficial effects of the present invention: By learning high-order structural correlation and feature correlation in video data, the present invention can more accurately capture the spatiotemporal information in video data, thereby improving the accuracy of completing missing video information. Through the structural intervention attention mechanism, the present invention can effectively intervene in the feature correlation, so that the completed video data is closer to the actual situation of the original video data. The use of high-order graph convolutional networks to process video data can better capture the spatiotemporal features in the video and improve the accuracy and naturalness of the completion effect. The structural intervention module can intervene in the feature correlation according to the high-order structural correlation, so that the model can pay more attention to key information during the completion process and improve the completion quality.

[0051] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0053] Figure 1 It is a flowchart of a method for completing missing video information based on high-order structural intervention according to Example 1 of the present invention.

[0054] Figure 2 It is a flowchart of a method for completing missing video information based on high-order structural intervention according to Embodiment 1 of the present invention.

[0055] Figure 3 This is a comparison chart of the accuracy of video missing information completion with high-order structural intervention before and after the application of Example 1 of the present invention.

[0056] Figure 4 It is a structural diagram of a video missing information completion device based on high-order structural intervention according to Example 2 of the present invention. DETAILED DESCRIPTION

[0057] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0058] Example 1

[0059] like Figure 1 As shown, a method for completing missing video information based on high-order structure intervention includes:

[0060] S1, collecting video data, and detecting the video data to obtain the position of missing information;

[0061] It should be noted that in step S1, collecting video data can be achieved in a variety of ways, such as real-time collection by a camera or reading a pre-recorded video file from a storage device. Detecting video data to obtain the location of missing information can be achieved through image processing techniques, such as edge detection, motion detection, and the like.

[0062] S2. Using a high-order graph structure to represent the spatiotemporal relationship in the video data to obtain a high-order tensor;

[0063] It should be noted that in step S2, firstly, the data frames in the video are extracted, and then the detection algorithm is used to obtain the objects in each frame, and the features of the corresponding objects are extracted; then each detected object is taken as a node, and the corresponding object features are taken as node features to obtain the node feature matrix N t ∈R K×F ; Then, the edges between nodes are constructed. Through the relative position relationship between nodes in each data frame, the connecting edges are constructed to obtain the adjacency matrix A between nodes in each data frame. t ∈R K×K , is a matrix of K rows and K columns, where K is the number of nodes; finally, the node feature matrix and adjacency matrix in each frame are constructed in chronological order to form the feature tensor X and adjacency tensor A∈R K×K×T ,in:

[0064] The number of nodes K is the number of nodes (objects) in the data frame, which is a positive integer.

[0065] The number of data frames T is the total number of data frames extracted from the video, which is a positive integer.

[0066] The feature dimension F is the number of node feature dimensions extracted from the video data frame, which is a positive integer.

[0067] S3, learning the high-order structural correlation between nodes and the feature correlation between nodes in the high-order tensor, and intervening the feature correlation using the node structural correlation to obtain a structural intervention attention tensor;

[0068] It should be noted that in step S3, the canonical multivariate tensor decomposition (CPTF) method is first used to decompose the adjacency tensor A to capture high-order structural patterns, reconstruct the adjacency tensor through the sum of rank tensors, and learn dynamic structural patterns. Then the factor matrix is ​​learned using the L2 norm calculation, and the normalized structural correlation is calculated using the SoftMax equation to form a structural correlation tensor S. The feature correlation between nodes is then calculated using a feedforward neural network, and the feature correlation in the feature tensor F is obtained through a learnable vector and a nonlinear activation function. Combining structural correlation and feature correlation, an adaptive fusion strategy is used to intervene the feature correlation with high-order structural information to form a high-order structural intervention attention tensor O.

[0069] S4, based on the structural intervention attention tensor, the high-order tensor is simultaneously transmitted in time and space to obtain a final node feature representation;

[0070] It should be noted that in step S4, in this embodiment, the graph tensor convolution is first implemented by tensor operation, and the time message passing of the video data is implemented by the tensor M conversion operation. The tensor M is a lower triangular mixing matrix whose size is determined by the time window z, and then the tensor slice multiplication operation is performed. and It can aggregate node spatial neighbor information and realize spatial message passing. Finally, tensors are used Complete node feature vector The final feature vector of node i in the tth data frame is obtained by transformation.

[0071] S5. Calculate the missing information based on the final node feature representation, and complete the video data based on the missing information calculation result and the missing information position.

[0072] It should be noted that in step S5, the missing information calculation can adopt a variety of algorithms, such as a method based on a generative adversarial network (GAN), in which the generator attempts to generate video frames of the missing parts, and the discriminator attempts to distinguish between real video frames and generated video frames. Through this adversarial process, the generator gradually learns how to generate more realistic video frames to fill in the missing information. In addition, a spatiotemporal interpolation technology based on deep learning can also be used to intelligently predict and fill in the missing frames by analyzing the spatiotemporal relationship between video frames.

[0073] like Figure 1 and 2As shown, in summary, this embodiment can more accurately capture the spatiotemporal information in the video data by learning the high-order structural correlation and feature correlation in the video data, thereby improving the accuracy of completing the missing information in the video. Through the structural intervention attention mechanism, the present invention can effectively intervene in the feature correlation, so that the completed video data is closer to the actual situation of the original video data. The use of a high-order graph convolutional network to process video data can better capture the spatiotemporal features in the video and improve the accuracy and naturalness of the completion effect. The structural intervention module can intervene in the feature correlation according to the high-order structural correlation, so that the model can pay more attention to key information during the completion process and improve the completion quality.

[0074] As an optional embodiment of the present invention, optionally, obtaining the high-order tensor in step S2 includes:

[0075] S201, extracting frame image data from the video data;

[0076] It should be noted that, in step S201, first, the frame image data in the video file is extracted frame by frame by using video decoding technology. In order to improve processing efficiency, multi-threading or parallel processing technology can be used to divide the video file into multiple parts and perform frame extraction at the same time. The extracted frame image data will serve as the basic data source for subsequent processing.

[0077] S202, detecting objects in the frame image data based on an object detection algorithm, taking the objects as nodes, and extracting node features;

[0078] It should be noted that in step S202, the extracted frame image data is analyzed frame by frame using target detection algorithms in deep learning, such as YOLO, SSD or Faster R-CNN. These algorithms can efficiently identify and locate multiple objects in the image, and generate bounding boxes and category labels for each detected object. The results of object detection will be used as nodes in the graph network, and the features of each node are obtained by extracting the image area within the bounding box and performing feature extraction. Feature extraction methods include convolutional neural networks (CNN), such as ResNet or VGG, which can extract rich visual features from images.

[0079] S203, constructing edges based on the spatial relationship between the nodes;

[0080] It should be noted that, in step S203, the edges in the graph network are constructed based on the spatial position relationship between the detected objects. The spatial relationship can be determined by calculating the center point distance, angle or other geometric features of the object bounding box. For example, if the center point distance between two objects is less than a preset threshold, it can be considered that there is a spatial connection between the two objects, and an edge is established between them in the graph network. In this way, a graph structure reflecting the spatial relationship between objects in the video frame can be constructed.

[0081] S204, constructing an adjacency matrix of the frame image data based on the node features and edges;

[0082] It should be noted that, in step S204, an adjacency matrix reflecting the relationship between objects in the video frame can be constructed using the node features and the constructed edge information. The adjacency matrix is ​​a two-dimensional matrix, in which the elements represent the connection relationship between nodes, and 0 and 1 are usually used to represent no connection and connection. When constructing the adjacency matrix, the similarity of node features can be considered, for example, by calculating the cosine similarity between feature vectors to determine the connection strength between nodes. In this way, the adjacency matrix not only reflects the spatial relationship between nodes, but also contains the similarity information of node features, providing rich information for subsequent graph convolution operations.

[0083] S205 , sorting the adjacency matrix in each frame of image data in chronological order to obtain a high-order tensor.

[0084] It should be noted that in step S205, the adjacency matrix in each frame image data is sorted in chronological order to ensure the temporal continuity of the video data. The sorted adjacency matrix will form a high-order tensor, which contains not only spatial information but also temporal information. In this way, the frame image data in the video data can be converted into a high-order tensor, thereby providing a basis for subsequent graph convolution network processing. The construction of high-order tensors enables the model to process temporal and spatial information at the same time, which is crucial for capturing complex dynamic features in video data. By sorting the adjacency matrix in chronological order, the model can better understand the spatiotemporal relationship between video frames, so that it is more accurate and natural when completing missing information. After constructing the high-order tensor, the video data is processed using a graph convolution network. The graph convolution network can effectively process graph structure data, capture complex relationships between nodes, and extract useful feature representations. By applying graph convolution operations on high-order tensors, the model can learn the spatiotemporal features in the video data and generate more accurate video completion results.

[0085] like Figure 2As shown above, in summary, the present invention can effectively capture the spatiotemporal information in the video and improve the accuracy of video missing information completion by constructing a high-order tensor and using a graph convolutional network to process video data. Through the combination of the structural intervention attention mechanism and the high-order graph convolutional network, this embodiment can not only better process the complex dynamic features in the video data, but also generate more realistic completion results, thereby improving the accuracy of video restoration.

[0086] As an optional embodiment of the present invention, optionally, in step S3, the expression for learning the high-order structural correlation between the nodes in the high-order tensor is:

[0087] In this embodiment, in order to calculate the high-order structural correlation between nodes, the canonical multivariate tensor decomposition (CPTF) is used to capture the high-order structural pattern. CPTF decomposes the adjacency tensor A into the sum of multiple rank tensors, which is expressed as:

[0088]

[0089] in, represents the adjacency tensor after CPTF reconstruction;

[0090] C, P, and Q all represent factor matrices, which are used to learn dynamic structural patterns in adjacency tensors. C = [c1, c2, ..., c R ]∈R N×R , P=[P1,P2,...,P R ]∈R N×R , Q=[Q1,Q2,...,Q R ]∈R T×R ; C, P, and Q are factor matrices consisting of factor vectors used to learn dynamic structural patterns in the adjacency tensor A.

[0091] Represents decomposing the tensor into three factor matrices;

[0092] R represents the number of factor vectors;

[0093] r represents a natural number;

[0094] represents a tensor of rank r;

[0095] c r 、p r and q r Both represent the rth factor matrix;

[0096] Represents tensor outer product operation;

[0097] According to the complete dynamic structure information retained in the above adjacency tensor A, in order to capture the high-order structural pattern from it, the factor matrix is ​​learned in the following way:

[0098]

[0099] Among them, || ||2 represents L2 norm calculation;

[0100] A represents the adjacency tensor;

[0101] →Indicates that the limit approaches a certain value;

[0102] Finally, according to the factor matrix calculation results, the SoftMax(·) equation is used to obtain the normalized structural correlation:

[0103]

[0104] in, Represents the structural correlation between node i and node j in the tth data frame;

[0105] SoftMax() means normalization;

[0106] Represents the element at position (i, j, t) in the adjacency tensor A;

[0107] exp() represents the exponential function;

[0108] k represents any node in the direct neighborhood set;

[0109] N(i) represents the direct neighborhood set of node i, N(i) = {j∈V t :(i,j)∈E t}, the structural correlation matrix of the tth data frame is S t ∈R N×N , the structural correlation tensor of the entire video data high-order graph is S∈R N×N×T .

[0110] {i} represents the node set consisting of node i itself;

[0111] Represents the element at position (i,j,k) in the adjacency tensor A.

[0112] As an optional embodiment of the present invention, optionally, in step S3, the expression for learning the feature correlation between the nodes in the high-order tensor is:

[0113]

[0114] in, represents the feature correlation between node i and node j in the tth data frame, is the learnable vector u∈R 2F The parameterized single-layer feedforward neural network was calculated;

[0115] exp() represents the exponential function;

[0116] σ() represents the activation function;

[0117] Represents the neighbor node set of node i of the tth video data frame;

[0118] W t Represents the node feature transformation matrix of the tth video data frame;

[0119] Represents the neighbor node set of node j in the tth video data frame;

[0120] [||] indicates feature vector concatenation operation;

[0121] u T represents the transpose of the learning vector u;

[0122] k represents any node in the direct neighborhood set;

[0123] N(i) represents the direct neighborhood set of node i;

[0124] {i} represents the node set consisting of node i;

[0125] Represents the set of neighbor nodes of node k in the tth video data frame.

[0126] As an optional embodiment of the present invention, optionally, the expression for obtaining the structural intervention attention in step S3 is:

[0127]

[0128] It should be noted that this embodiment adopts an adaptive fusion strategy to intervene the high-order structural information on the feature correlation and introduces two learnable vectors and Adaptively learn a relative importance measure of correlations in each data frame.

[0129]

[0130] in, Represents the importance measure of the t-th data frame of structural relevance;

[0131] exp() represents the exponential function;

[0132] and Both represent the learnable vectors of the t-th data frame;

[0133] Represents the importance measure of feature correlation in the tth data frame;

[0134] Represents the structural intervention attention of node i and node j in the tth data frame;

[0135] Represents the structural correlation between node i and node j in the tth data frame;

[0136] Represents the feature correlation between node i and node j in the tth data frame;

[0137] k represents a node;

[0138] N(i) represents the direct neighborhood set of node i;

[0139] {i} represents the node set consisting of node i;

[0140] Represents the structural correlation between node i and node k in the tth data frame;

[0141] Represents the feature correlation between node i and node k in the tth data frame.

[0142] It should be noted that by calculating the high-order structure intervention attention of all data frames, the attention tensor O∈R N×N×T From the above, we can see that O is the attention calculated by intervening feature correlation based on structural correlation.

[0143] As an optional embodiment of the present invention, optionally, in step S4, the high-order tensor is simultaneously transmitted in time and space based on the structural intervention attention tensor, and the expression for obtaining the final node feature representation is:

[0144] Tensor multiplication operation realizes graph tensor convolution: H = σ (O*N*W)

[0145] Among them, H represents the embedding representation tensor of the node;

[0146] σ() represents the activation function;

[0147] O represents the structural intervention attention tensor;

[0148] N represents the original node feature tensor of the input;

[0149] W represents the node feature transformation vector;

[0150] * represents tensor multiplication operation;

[0151] In graph tensor convolution, temporal message passing in the high-order graph of video data is achieved through tensor M transformation operations:

[0152]

[0153] in, and Represents different information tensors after time message passing,

[0154] τ() represents the information aggregation of the time dimension of the information tensor;

[0155] ×3 represents modulo 3 operation;

[0156] M represents the triangular mixing moment;

[0157]

[0158] Among them, M t,k Represents the element value at position (t,k) in M;

[0159] max() means taking the maximum value;

[0160] t represents the time position of the data frame;

[0161] k represents any node in the direct neighborhood set;

[0162] z represents the time window;

[0163]

[0164] in, Represents the final feature vector of node i in the tth data frame;

[0165] j represents a node;

[0166] N(i) represents the direct neighborhood set of node i;

[0167] {i} represents the node set consisting of node i;

[0168] Represents the structural intervention attention of node i and node j in the tth data frame;

[0169] Represents the neighbor node set of node j in the tth video data frame;

[0170] W tRepresents the node feature transformation matrix of the t-th video data frame in the weight tensor W.

[0171] like Figure 3 As shown, in order to verify the performance of the above-mentioned video data hidden feature extraction method, this embodiment uses this method on a PC (configuration: INTEL i7-6700, 3.4GHz processor, 32G memory), and runs a simulation experiment for example analysis. In the example analysis, the video data used comes from high-definition video data of a monitoring system. The example analysis uses the mean square error RMSE as the evaluation index of the accuracy of video data missing information completion. The lower the RMSE, the higher the accuracy of the completion.

[0172] Figure 3 This is a comparison of the accuracy of completing missing information for high-definition video data before and after applying the embodiments of the present invention. Figure 3 After applying this embodiment, the accuracy of missing information completion is greatly improved. In practical applications, it can better provide safe, reliable and practically guaranteed missing information completion services for the video data field.

[0173] Example 2

[0174] like Figure 4 As shown, a video missing information completion device based on high-order structure intervention, the device includes the video missing information completion method based on high-order structure intervention; the device also includes:

[0175] A receiving module, used for acquiring video data;

[0176] In this embodiment, the receiving module is a wireless or wired communication module, which is used to receive video data collected by the video acquisition device and instruct the storage module to store the collected video data.

[0177] A data storage module, connected to the receiving module, used to store the video data and detect the video data to obtain the position of missing information;

[0178] The data storage module includes a video data storage unit, a completed video data storage unit and a missing information detection unit. The video data storage unit is used to store the collected video data; the missing information detection unit is used to detect missing information in the video, and the completed video data storage unit is used to store the completed video data.

[0179] A video high-order graph data extraction module, connected to the data storage module, for extracting high-order tensors from the video data;

[0180] The video high-order graph data extraction module is used to use high-order graph data to represent the spatiotemporal relationship in video data. Nodes represent specific locations or objects in the video, and edges represent the spatial relationship between these nodes. A node is created for each key point (such as an object) in the video, and then edges are created between nodes based on the spatial proximity relationship.

[0181] A structural intervention module, connected to the video high-order graph data extraction module, is used to learn the high-order structural correlation between nodes and the feature correlation between nodes in the high-order tensor, and use the node structural correlation to intervene in the feature correlation to obtain a structural intervention attention tensor;

[0182] The structural intervention module is used to capture the inter-node correlation contained in the high-order structure, intervene in the correlation between node features, and consider the structural correlation and node feature correlation between nodes at the same time to obtain a finer-grained spatial correlation between nodes.

[0183] A high-order graph convolution module, connected to the structure intervention module, for simultaneously transmitting time and space messages to the high-order tensor according to the structure intervention attention tensor to obtain a final node feature representation;

[0184] The high-order graph convolution module is used to perform graph convolution operations on high-order graph data, and further optimizes and refines node features by combining structural intervention attention tensors. This module can effectively extract spatiotemporal features in video data by simultaneously transmitting messages in the time and space dimensions, thereby obtaining a more accurate node feature representation.

[0185] A missing information calculation module is connected to the high-order graph convolution module and is used to calculate the missing information according to the final node feature representation, and to complete the video data based on the missing information calculation result and the missing information position.

[0186] The missing information calculation module is used to calculate the missing information based on the final node feature representation, and complete the video data by combining the missing information position. This module uses the node feature representation obtained by the high-order graph convolution module to infer and fill in the missing parts to achieve complete processing of the video data.

[0187] The result output unit completes the missing video information according to the obtained node feature tensor and stores it in the completed video storage unit in the video data module.

[0188] like Figure 4As shown, in this embodiment, the receiving module first obtains the video data, and then the data storage module stores the video data and detects missing information. Next, the video high-order graph data extraction module extracts the high-order tensor in the video data to represent the spatiotemporal relationship in the video data. The structural intervention module learns the high-order structural correlation between each node in the high-order tensor and the feature correlation between the nodes, and uses the node structural correlation to intervene in the feature correlation to obtain the structural intervention attention tensor. The high-order graph convolution module transmits the high-order tensor in time and space simultaneously according to the structural intervention attention tensor to obtain the final node feature representation. The missing information calculation module calculates the missing information according to the final node feature representation, and completes the video data based on the missing information calculation result and the missing information position. Finally, the result output unit completes the missing information of the video according to the obtained node feature tensor, and stores it in the completed video storage unit in the video data module. Through this embodiment, the missing information in the video data can be effectively completed, and the integrity and availability of the video data can be improved. In practical applications, the device can be widely used in monitoring systems, video analysis, intelligent transportation and other fields, providing more accurate and reliable video data support for related fields.

[0189] As another optional embodiment of the present invention, optionally, the data storage module includes:

[0190] A video data storage unit, used for storing the video data;

[0191] A video missing information detection unit, used to detect the video data and obtain the missing information position;

[0192] The completed video data storage unit is used to store the completed video data.

[0193] As another optional embodiment of the present invention, optionally, the video high-order graph data extraction module includes:

[0194] A video data frame extraction unit, used to extract frame image data from the video data;

[0195] The video data frame extraction unit extracts images from the video frame by frame, then detects objects in each frame using an object detection algorithm, and extracts features (such as position, category, and visual features) of each detected object.

[0196] A node construction unit, connected to the video data frame extraction unit, for constructing nodes in the high-order tensor;

[0197] The node construction unit treats each detected object as a node. The node features include the object category, location coordinates, visual features, etc. The node features in the node of the t-th video frame can form a third-order tensor N of size F×Tt , F is the feature dimension, and T is the total number of data frames.

[0198] An edge construction unit, connected to the node construction unit, and configured to construct edges in the high-order tensor according to the spatial relationship between the nodes;

[0199] The edge construction unit is used to construct the edges between nodes. Through the relative position relationship between the nodes in the tth data frame, the connection edges can be constructed to obtain the adjacency matrix A. t ∈R K×K , is a matrix of K rows and K columns, where K is the number of nodes.

[0200] An information tensor construction unit is connected to the edge construction unit and is used to construct a high-order tensor according to the nodes and edges.

[0201] The information tensor construction unit arranges the graph data composed of each frame of the video in chronological order to obtain the corresponding third-order tensor representation, where the node features can form a third-order tensor N of size K×F×T, where T is the total number of image data frames, and the adjacency matrix of different data frames {A 1 ,A 2 ,...,A T} Arranged in chronological order, a third-order tensor A∈R of size K×K×T can be formed K×K×T .

[0202] As another optional embodiment of the present invention, optionally, the structural intervention module includes:

[0203] A high-order structure correlation learning unit, connected to the video high-order graph data extraction module, for learning the high-order structure correlation between the nodes in the high-order tensor;

[0204] The high-order structural correlation learning unit is used to analyze the high-order structural correlation in video data and extract the spatial relationship between nodes through deep learning algorithms. The unit learns high-order structural correlation by analyzing the proximity relationship and relative position between nodes, thus providing a basis for subsequent structural intervention.

[0205] A node feature correlation learning unit, connected to the video high-order graph data extraction module, for learning the feature correlation between nodes in the high-order tensor;

[0206] The node feature correlation learning unit is used to analyze the node feature correlation in the video data and extract the correlation between node features through a feedforward neural network. This unit learns the correlation between node features by analyzing the visual features, categories and other information of the nodes, thus providing a basis for subsequent structural intervention.

[0207] The structure intervention unit is connected to the high-order structure correlation learning unit and the node feature correlation learning unit, and is used to intervene in the feature correlation by utilizing the node structure correlation to obtain structure intervention attention.

[0208] The structure intervention unit is used to integrate high-order structure correlation and node feature correlation, intervene in feature correlation through deep learning algorithm, and generate structure intervention attention tensor. This unit generates an attention mechanism by combining the outputs of high-order structure correlation learning unit and node feature correlation learning unit to highlight important features and suppress irrelevant features, thereby improving the model's understanding and processing capabilities of video data.

[0209] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for completing missing video information based on high-order structure intervention, characterized in that: The method comprises: S1, collecting video data, and detecting the video data to obtain the position of missing information; S2. Using a high-order graph structure to represent the spatiotemporal relationship in the video data to obtain a high-order tensor; S3, learning the high-order structural correlation between nodes and the feature correlation between nodes in the high-order tensor, and intervening the feature correlation using the node structural correlation to obtain a structural intervention attention tensor; S4, based on the structural intervention attention tensor, the high-order tensor is simultaneously transmitted in time and space to obtain a final node feature representation; S5. Calculate the missing information based on the final node feature representation, and complete the video data based on the missing information calculation result and the missing information position.

2. The method for completing missing video information based on high-order structure intervention as claimed in claim 1, characterized in that: Obtaining the high-order tensor in step S2 includes: S201, extracting frame image data from the video data; S202, detecting objects in the frame image data based on an object detection algorithm, taking the objects as nodes, and extracting node features; S203, constructing edges based on the spatial relationship between the nodes; S204, constructing an adjacency matrix of the frame image data based on the node features and edges; S205 , sorting the adjacency matrix in each frame of image data in chronological order to obtain a high-order tensor.

3. The method for completing missing video information based on high-order structure intervention as claimed in claim 1, characterized in that: In step S3, the expression for learning the high-order structural correlation between the nodes in the high-order tensor is: in, represents the adjacency tensor reconstructed by CPTF, C, P and Q all represent factor matrices, which are used to learn the dynamic structural patterns in the adjacency tensor. Indicates that the tensor is decomposed into three factor matrices, R represents the number of factor vectors, r represents a natural number, represents a tensor of rank r, c r 、p r and q r Both represent the rth factor matrix, represents the tensor outer product operation, || ||2 represents the L2 norm calculation, A represents the adjacent tensor, represents the structural correlation between node i and node j in the tth data frame, SoftMax() represents normalization, represents the element at position (i, j, t) in the adjacency tensor A, exp( ) represents the exponential function, k represents any node in the direct neighbor set, N(i) represents the direct neighbor set of node i, {i} represents the node set consisting of node i, Represents the element at position (i,j,k) in the adjacency tensor A.

4. The method for completing missing video information based on high-order structure intervention as claimed in claim 1, characterized in that: In step S3, the expression for learning the feature correlation between nodes in the high-order tensor is: in, represents the feature correlation between node i and node j in the tth data frame, exp( ) represents the exponential function, σ( ) represents the activation function, represents the neighbor node set of node i in the tth video data frame, W t Represents the node feature transformation matrix of the tth video data frame, represents the neighbor node set of node j in the tth video data frame, [||] represents the feature vector concatenation operation, u T represents the transpose of the learning vector u, k represents any node in the direct neighborhood set, N(i) represents the direct neighborhood set of node i, {i} represents the node set composed of node i, Represents the set of neighbor nodes of node k in the tth video data frame.

5. The method for completing missing video information based on high-order structure intervention as claimed in claim 2, characterized in that: The expression for obtaining the structural intervention attention in step S3 is: in, represents the importance measure of structural correlation in the tth data frame, exp( ) represents the exponential function, and Both represent the learnable vectors of the t-th data frame, represents the importance measure of feature correlation in the tth data frame, represents the structural intervention attention of node i and node j in the tth data frame, represents the structural correlation between node i and node j in the tth data frame, represents the feature correlation between node i and node j in the tth data frame, k represents any node in the direct neighbor set, N(i) represents the direct neighbor set of node i, {i} represents the node set composed of node i, represents the structural correlation between node i and node k in the tth data frame, Represents the feature correlation between node i and node k in the tth data frame.

6. The method for completing missing video information based on high-order structure intervention as claimed in claim 1, characterized in that: In step S4, the high-order tensor is simultaneously transmitted in time and space based on the structural intervention attention tensor, and the expression for obtaining the final node feature representation is: H=σ(O*N*W) Among them, H represents the embedding representation tensor of the node, σ( ) represents the activation function, O represents the structural intervention attention tensor, N represents the input original node feature tensor, W represents the node feature conversion vector, * represents the tensor multiplication operation, and represents different information tensors after time message transmission, τ( ) represents the information aggregation of the information tensor in the time dimension, ×3 represents the modulo three operation, M represents the triangular mixed moment, M t,k represents the element value at position (t, k) in M, max() represents the maximum value, t represents the time position of the data frame, k represents any node in the direct neighbor set, z represents the time window, represents the final feature vector of node i in the tth data frame, j represents the node, N(i) represents the direct neighborhood set of node i, {i} represents the node set composed of node i itself, represents the structural intervention attention of nodes i and j in the tth data frame, represents the neighbor node set of node j in the tth video data frame; W t Represents the node feature transformation matrix of the t-th video data frame in the weight tensor W.

7. A video missing information completion device based on high-order structure intervention, characterized in that: The device comprises the video missing information completion method based on high-order structure intervention as described in any one of claims 1 to 6; the device also comprises: A receiving module, used for acquiring video data; A data storage module, connected to the receiving module, used to store the video data and detect the video data to obtain the position of missing information; A video high-order graph data extraction module, connected to the data storage module, for extracting high-order tensors from the video data; A structural intervention module, connected to the video high-order graph data extraction module, is used to learn the high-order structural correlation between nodes and the feature correlation between nodes in the high-order tensor, and use the node structural correlation to intervene in the feature correlation to obtain a structural intervention attention tensor; A high-order graph convolution module, connected to the structure intervention module, for simultaneously transmitting time and space messages to the high-order tensor according to the structure intervention attention tensor to obtain a final node feature representation; A missing information calculation module is connected to the high-order graph convolution module and is used to calculate the missing information according to the final node feature representation, and to complete the video data based on the missing information calculation result and the missing information position.

8. The apparatus for completing missing video information based on high-order structure intervention as claimed in claim 7, characterized in that: The data storage module comprises: A video data storage unit, used for storing the video data; A video missing information detection unit, used to detect the video data and obtain the missing information position; The completed video data storage unit is used to store the completed video data.

9. The video missing information completion device based on high-order structure intervention as claimed in claim 7, characterized in that: The video high-order graph data extraction module includes: A video data frame extraction unit, used to extract frame image data from the video data; A node construction unit, connected to the video data frame extraction unit, for constructing nodes in the high-order tensor; An edge construction unit, connected to the node construction unit, and configured to construct edges in the high-order tensor according to the spatial relationship between the nodes; An information tensor construction unit is connected to the edge construction unit and is used to construct a high-order tensor according to the nodes and edges.

10. The video missing information completion device based on high-order structure intervention as claimed in claim 7, characterized in that: The structural intervention module includes: A high-order structure correlation learning unit, connected to the video high-order graph data extraction module, for learning the high-order structure correlation between the nodes in the high-order tensor; A node feature correlation learning unit, connected to the video high-order graph data extraction module, for learning the feature correlation between nodes in the high-order tensor; The structure intervention unit is connected to the high-order structure correlation learning unit and the node feature correlation learning unit, and is used to intervene in the feature correlation by utilizing the node structure correlation to obtain structure intervention attention.

Images