Video target detection method and system based on space-time diagram convolution

Abstract

The invention discloses a video target detection method and system based on space-time diagram convolution, and the method comprises the steps: obtaining a to-be-detected video, inputting all frame images in the to-be-detected video into a trained target detection model in batches, and obtaining the target detection results of all frame images; wherein the trained target detection model performs feature extraction on each frame of image to obtain image features of each frame; extracting global aggregation features and local aggregation features from the image features of each frame, and mixing the global aggregation features and the local aggregation features to obtain mixed features of each frame; and identifying and positioning the mixed features of each frame to obtain a target detection result of each frame. According to the method, a large number of experiments are carried out on an ImageNet VID data set to prove the effectiveness of the proposed model.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a video target detection method and system based on spatiotemporal graph convolution. Background Art

[0002] In recent years, video object detection has made significant progress thanks to convolutional neural networks and visual transformers. Convolutional neural networks are good at extracting local features, but have difficulty extracting global features. In contrast, visual transformers are able to capture features at a global scale, but face challenges in representing local feature details. Existing video object detection methods rely solely on convolutional neural networks or visual transformers for feature aggregation, which hinders their ability to exploit both global and local information, thus limiting detection performance.

[0003] Video object detection aims to predict the location box and category label of each object in the video. In the past few decades, with the explosive growth of video data, video object detection has become a key task in the fields of autonomous driving, intelligent monitoring, intelligent transportation, and drone navigation. Traditional image object detection methods are mainly used for static image processing, while in practical applications, video object detection not only needs to identify the object in each frame, but also must process the temporal information between frames to improve detection accuracy and robustness.

[0004] Convolutional neural networks extract local features of images through layer-by-layer convolution operations and have excellent local perception capabilities. However, the inherent local perspective of convolutional neural networks limits their ability to process long-distance dependencies and global context information. This is particularly evident when processing dynamic videos, because complex spatiotemporal dependencies may not be fully captured by local feature extraction, resulting in reduced detection accuracy in complex scenes such as occlusion and blur.

[0005] Transformer has been gradually introduced into visual tasks in recent years due to its success in natural language processing. Unlike convolutional neural networks, Transformer can capture global feature dependencies through the self-attention mechanism and is particularly good at extracting global information. However, Transformer also faces certain challenges in video object detection, especially when processing short-distance and local detail features, it performs worse than convolutional neural networks.

[0006] Therefore, there is an urgent need for a new method that can combine the local feature extraction capability of convolutional neural networks and the global relationship capture capability of Transformer, while effectively overcoming the limitations of existing technologies to improve the accuracy and real-time performance of video object detection. Summary of the invention

[0007] In order to address the shortcomings of the prior art, the present invention provides a video object detection method and system based on spatiotemporal graph convolution; first, a spatiotemporal Transformer module is developed to aggregate global context information to form a global representation with long-range feature dependencies. Secondly, a spatiotemporal graph convolution module is introduced, which utilizes local spatial and temporal relationships to aggregate features and generate local features that are complementary to the spatiotemporal Transformer output. Third, a global-local feature hybrid module is designed to adaptively couple the Transformer-based global representation and the graph convolution-based local representation. The present invention conducts a large number of experiments on the ImageNet VID dataset to demonstrate the effectiveness of the proposed model.

[0008] On the one hand, a video object detection method based on spatiotemporal graph convolution is provided, including:

[0009] Obtain the video to be detected, input all frame images in the video to be detected into the trained target detection model in batches, and obtain the target detection results of all frame images;

[0010] Among them, the trained target detection model extracts features from each frame of the image to obtain the image features of each frame; extracts global aggregation features and local aggregation features from the image features of all frames respectively, mixes the global aggregation features and the local aggregation features to obtain the mixed features of each frame; identifies and locates the mixed features of each frame to obtain the target detection results of each frame.

[0011] On the other hand, a video object detection system based on spatiotemporal graph convolution is provided, including:

[0012] The target detection module is configured to: obtain a video to be detected, input all frame images in the video to be detected into the trained target detection model in batches, and obtain target detection results for all frame images;

[0013] Among them, the trained target detection model extracts features from each frame of the image to obtain the image features of each frame; extracts global aggregation features and local aggregation features from the image features of all frames respectively, mixes the global aggregation features and the local aggregation features to obtain the mixed features of each frame; identifies and locates the mixed features of each frame to obtain the target detection results of each frame.

[0014] On the other hand, there is also provided an electronic device, comprising:

[0015] a memory for non-transitory storage of computer-readable instructions; and

[0016] a processor for executing the computer readable instructions,

[0017] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0018] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the method described in the first aspect is executed.

[0019] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0020] The above technical solution has the following advantages or beneficial effects:

[0021] The present invention overcomes the limitations of the prior art by rationally combining the advantages of convolutional neural networks and visual transformers. Convolutional neural networks excel in extracting local information, while visual transformers are suitable for capturing global features. This patent effectively utilizes the local perception capabilities of convolutional neural networks and the global perception capabilities of transformers to achieve more comprehensive feature extraction and information aggregation, thereby improving the accuracy and efficiency of target detection.

[0022] This patented technology has a wide range of impacts on multiple industries, especially in the field of visual information that requires high-precision, real-time processing. For example, in intelligent monitoring and security systems, it can improve the reliability of public safety through more accurate target recognition; in autonomous driving and intelligent transportation, this technology improves the safety and response speed of the system through more comprehensive scene perception. In augmented reality (AR) and virtual reality (VR) applications, this technology helps to improve the accuracy of scene recognition and target tracking and enhance user experience. Through the application of this technology, these industries will see significant improvements in accuracy, efficiency and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 is a flow chart of the method of embodiment 1;

[0025] Figure 2 Schematic diagram of the internal structure of the spatiotemporal Transformer module of Example 1;

[0026] Figure 3 Schematic diagram of the internal structure of the spatiotemporal graph convolution module of Example 1;

[0027] Figure 4 Schematic diagram of the connection relationship between the Transformer decoder and the feedforward network of Example 1. DETAILED DESCRIPTION

[0028] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0029] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0030] In this embodiment, all data is obtained in compliance with laws and regulations and based on the user's consent, and is used legally.

[0031] Embodiment 1

[0032] This embodiment provides a video object detection method based on spatiotemporal graph convolution;

[0033] The video object detection method based on spatiotemporal graph convolution includes:

[0034] Obtain the video to be detected, input all frame images in the video to be detected into the trained target detection model in batches, and obtain the target detection results of all frame images;

[0035] Among them, the trained target detection model extracts features from each frame of the image to obtain the image features of each frame; extracts global aggregation features and local aggregation features from the image features of all frames respectively, mixes the global aggregation features and the local aggregation features to obtain the mixed features of each frame; identifies and locates the mixed features of each frame to obtain the target detection results of each frame.

[0036] Furthermore, the training process of the trained target detection model includes:

[0037] Constructing a training set, wherein the training set is a video frame image with a known target detection label;

[0038] The training set is input into the target detection model to train the model. When the loss function value of the model no longer decreases or the number of iterations reaches the set number, the training is stopped to obtain the trained target detection model.

[0039] Furthermore, the training set refers to: given a video data set, the data set contains several videos, each video contains several video frames The video object detection method of the present invention takes N consecutive frames (the default setting is 25 frames) as input and outputs the detection results of all input frames at one time.

[0040] Furthermore, the trained target detection model includes:

[0041] Several parallel backbone networks are connected. The input end of each backbone network inputs the frame image to be detected. The output end of each backbone network outputs the image features of the corresponding frame image (referred to as frame features). The image features and corresponding position features of each frame image are input to the input end of the spatiotemporal Transformer module. At the same time, the image features and corresponding position features of each frame are input to the input end of the spatiotemporal graph convolution module.

[0042] The output of the spatiotemporal Transformer module is connected to the input of the global-local feature mixing module, and the output of the spatiotemporal graph convolution module is connected to the input of the global-local feature mixing module;

[0043] The output end of the global-local feature mixing module outputs the mixed features of each frame of the image; the mixed features of each frame are input to the input end of the corresponding decoder, the output end of the decoder is connected to the input end of the corresponding multi-layer perceptron, and the output end of the multi-layer perceptron outputs the target detection result of each frame of the image.

[0044] Furthermore, if Figure 2 As shown, the spatiotemporal Transformer module includes:

[0045] N parallel branches, each branch includes an adder, a spatial multi-head self-attention mechanism layer and a first addition and normalization module connected in sequence, the input end of the first addition and normalization module is also connected to the output end of the adder; the input end of the adder of the nth branch is used to input the frame features and corresponding position features of the nth frame; the output end of the first addition and normalization module of the nth branch outputs the first intermediate features of the nth frame image, and the first intermediate features of the N frames of image are input into the temporal multi-head self-attention mechanism layer, the temporal multi-head self-attention mechanism layer inputs the output value into the second addition and normalization module, the second addition and normalization module outputs the global aggregation features of each frame of image; the input end of the second addition and normalization module also inputs the first intermediate features.

[0046] Furthermore, if Figure 3 As shown, the spatiotemporal graph convolution module includes:

[0047] There are N parallel branches, each of which includes an adder and a spatial graph convolution unit connected in sequence; the input end of the adder of the nth branch is used to input the frame features and corresponding position features of the nth frame; the output end of the spatial graph convolution unit of the nth branch outputs the second intermediate features of the nth frame image, and the second intermediate features of the N frames of image are input into the temporal graph convolution unit, which outputs the local aggregation features of each frame of image.

[0048] Furthermore, the trained object detection model extracts features from each frame of the image to obtain the image features of each frame, including:

[0049] Several backbone networks are used to extract features from several frames of images, where one backbone network corresponds to one frame of image, and finally, the image features of all frames are obtained. The present invention uses ResNet-101 as the backbone network and DETR as the baseline network, and adjusts the total stride of the last stage of ResNet-101 (i.e., Conv5) from 32 to 16.

[0050] Furthermore, global aggregation features and local aggregation features are extracted from the image features of all frames, including:

[0051] Through the spatiotemporal Transformer module, the global aggregate features of each frame are extracted from the image features of all frames;

[0052] Through the spatiotemporal graph convolution module, the local aggregate features of each frame are extracted from the image features of all frames.

[0053] Furthermore, the spatiotemporal Transformer module extracts the global aggregate features of each frame from the image features of all frames, including:

[0054] Given a backbone network extracts the nth frame feature Where c, h, and w represent the dimension, height, and width of the frame features, respectively;

[0055] Add the features of each frame and the corresponding position features together to get the added features And the added features Input into the spatial multi-head self-attention mechanism layer to explore the long-range spatial dependencies within the frame and perform intra-frame feature aggregation from a global perspective;

[0056] Among them, the position feature is calculated using sine and cosine coding, and its formula is:

[0057]

[0058]

[0059] Among them, pos represents the position, i represents the dimension, sine encoding is used for even channels, and cosine encoding is used for odd channels.

[0060] The calculation formula of the spatial multi-head self-attention mechanism layer (SMHSA) is:

[0061]

[0062] Where q∈Ω q Represents a query element, whose characteristics are k∈Ω k Represents a key element, whose characteristics are D stands for z q and x k The characteristic dimension of q and Ω k denotes the set of query and key elements respectively. T denotes the total number of attention heads, K denotes the total number of key elements, and represents the learnable mapping weight, D v =D / T. The symbol “·” indicates a scalar multiplication operation. D v represents the dimension of each attention head. tqk The calculation formula is:

[0063]

[0064] Among them, O tqk represents the self-attention weight of the k-th sampled key element in the t-th attention head, and is normalized over all key elements. ∝ indicates proportional to the operation. (·) T Represents a transpose operation. and represents the learnable mapping weights.

[0065] Next, the output features of the spatial multi-head self-attention mechanism layer SMHSA are sequentially passed through the addition and normalization modules to generate the intermediate features of the current frame.

[0066] The addition and normalization module ensures the flow of information and stable training through residual connection and normalization. The implementation method is:

[0067] AddLN=LayerNorm(x+SMHSA)#(5)

[0068] This process first performs residual connection and then uses layer normalization LayerNorm to ensure numerical stability and avoid gradient disappearance or explosion problems.

[0069] Afterwards, the intermediate features of each frame are input into the temporal multi-head self-attention mechanism layer to mine the long-range temporal dependencies between frames and perform inter-frame feature aggregation from a global perspective;

[0070] The computation of the temporal multi-head self-attention mechanism layer (TMHSA) is expressed as:

[0071]

[0072] Among them, x n represents the intermediate feature, n represents the index of the input frame, O tnqk represents the self-attention weight of the k-th sampled key element in the t-th attention head on the n-th frame.

[0073] O tnqk The mathematical formula for this is:

[0074]

[0075] Its normalization method is

[0076] Finally, the output features of the temporal multi-head self-attention mechanism layer are sequentially operated through addition and normalization modules (in the same way as the spatial multi-head self-attention) to generate global aggregate features.

[0077] The number of attention heads for both spatial and temporal Transformer is set to 6.

[0078] Furthermore, the spatiotemporal graph convolution module extracts local aggregate features of each frame from the image features of all frames, including:

[0079] (1-1) Add the frame features and corresponding position features of the nth frame, and input the addition result into the spatial graph convolution unit; the spatial graph convolution unit is used to explore the short-range spatial dependency within the frame and perform intra-frame feature aggregation from a local perspective;

[0080] (1-2) The spatial graph convolution unit first constructs an undirected graph. In the process of constructing the undirected graph, each input feature is taken as a vertex, and the edge is determined according to the correlation between the vertices. Based on the constructed edges, the adjacency matrix is ​​determined.

[0081] (1-3) constructing an adjacency tensor based on the adjacency matrix; selecting two adjacency matrices from the adjacency tensor; obtaining a final pruned adjacency matrix through the two adjacency matrices; determining a pruned undirected graph based on the final pruned adjacency matrix; obtaining the nth frame intermediate features output by the spatial graph convolution unit based on the vertex feature matrix of the undirected graph;

[0082] (1-4) The intermediate features of all frames are input into the temporal graph convolution unit to obtain local aggregated features.

[0083] The number of dynamic graph convolution layers in spatial and temporal graph convolution is set to 2.

[0084] Furthermore, (1-2) the spatial graph convolution unit first constructs an undirected graph; in the process of constructing the undirected graph, each input feature is taken as a vertex, and the edge is determined according to the association between the vertices; based on the constructed edges, the adjacency matrix is ​​determined, which specifically includes:

[0085] First, build a fully connected undirected graph in, ε and Represent the vertex set, edge set and adjacency matrix respectively. In the construction of undirected graph, each input feature (i.e., the sum of each frame feature and position feature) is defined as a graph vertex v i , each edge e ij is defined as the vertex v i and v j The pairwise relationship between the edges. ij The calculation is:

[0086]

[0087] in, represents a mapping function consisting of two fully connected layers, Π[·,·,·] represents a concatenation operation. Euc(·,·) represents the normalized Euclidean distance, Cos(·,·) represents the cosine similarity. Sec(·,·) represents the semantic similarity, Sec(r i ,r j )pass Calculated.

[0088] Based on the constructed edge ij , the adjacency matrix A is obtained by the softmax function, and each element A in the adjacency matrix A ij is calculated as:

[0089] Since the goal of the spatiotemporal graph convolution module is to aggregate features from a local perspective of graph convolution, directly using a fully connected graph cannot well model the local relationship between vertex features. In this case, performing graph convolution may lead to the aggregation of useless information (such as inter-class targets and backgrounds) and introduce a lot of additional computational overhead.

[0090] To alleviate these problems, a graph pruning mechanism is proposed, aiming to sparsify the topological structure of the graph by removing useless or weakly connected edges, so as to focus on the useful local relationships between vertices. Most of the previous methods use a single threshold to prune the normalized adjacency matrix, and these methods lead to the problem that the feature representations are difficult to distinguish because they eliminate the matrix elements below the threshold. Therefore, the present invention proposes a novel graph pruning mechanism.

[0091] Furthermore, the (1-3) constructs an adjacency tensor based on the adjacency matrix, specifically including:

[0092] The graph pruning mechanism uses S real thresholds, denoted as Γ = [θ1, θ2, …, θ S , θ i < θ j , and θ i , θ j ∈ [0, 1] holds for all i < j.

[0093] Construct an adjacency tensor This tensor consists of a set of adjacency matrices where,

[0094] Set A 1 to the identity matrix E. For each s ≥ 2, The mathematical calculation can be defined as:

[0095]

[0096] where, P ij is an element in the probability matrix P, which can be calculated by P ij = λ · A ij / d i The λ (set to 0.3) represents the scalar weight, is an element in the diagonal matrix .

[0097] Furthermore, the (1-3) selects two adjacency matrices from the adjacency tensor, specifically including:

[0098] Through two 1×1 convolutions, select two adjacency matrices Q1 and Q2 from the adjacency tensor , where the non-negative weights are obtained by the softmax function:

[0099]

[0100] where, and represent two different 1×1 convolutional layers. and represents a learnable parameter.

[0101] Furthermore, the (1-3) obtains a final pruned adjacency matrix through the two adjacency matrices, and determines a pruned undirected graph based on the final pruned adjacency matrix; specifically includes:

[0102] The final pruned adjacency matrix pass ψ(·) represents the Laplace normalization operation, calculate.

[0103] Furthermore, in (1-3), based on the vertex feature matrix of the pruned undirected graph, the n-th frame intermediate feature output by the spatial graph convolution unit is obtained, including:

[0104] Based on the pruned graph, the graph convolutional network is extended from a static setting to a dynamic setting, and a dynamic graph convolution module with residual connections is designed to effectively aggregate features from local neighboring locations. The proposed dynamic graph convolution module (DGCB) is defined as:

[0105] DGCB(H)=DGCL2(DGCL1(H))+ρ·H#(11)

[0106] in, is the vertex feature matrix of the pruned graph, ρ represents the residual constant, DGCL1(·) and DGCL2(·) represent two consecutive dynamic graph convolutional layers;

[0107] The goal of dynamic graph convolution is to dynamically adjust the adjacency matrix or edge weights of the graph so that the graph structure can be adaptively adjusted according to the input features of each layer to capture more complex relationships. The calculation formulas for DGCL1 and DGCL2 are:

[0108]

[0109] Among them, H (i) Represents the vertex feature matrix of the i-th layer. Express The Laplace normalized adjacency matrix is ​​used to represent the connection relationship between vertices. (i) represents the learnable weight matrix of the i-th layer. σ represents the activation function. H 0 is the vertex feature matrix of the pruned graph, H 1 is the output of the first layer of dynamic graph convolution, H 2 is the output of the second layer of dynamic graph convolution.

[0110] Different from the graph convolution using fixed adjacency matrix in traditional graph convolutional networks, the dynamic graph convolution proposed in this paper helps to capture dynamic graph structure during graph convolution and improve the effectiveness of graph convolution. The output of the spatial graph convolution unit is represented as an intermediate feature.

[0111] Spatial graph convolution and temporal graph convolution have the same workflow, but the difference between them lies in the different input features. Specifically, in spatial graph convolution, the input features are the element-by-element addition of frame features and position features. In temporal graph convolution, the input features are the multi-frame intermediate features output by spatial graph convolution.

[0112] Furthermore, the intermediate features of all frames in (1-4) are input into the temporal graph convolution unit to obtain local aggregate features, which specifically include:

[0113] In the temporal graph convolution, each vertex This corresponds to the multi-frame intermediate features output by the spatial graph convolution. Each edge in the temporal graph convolution Defined as a vertex and relationship, side The calculation of refers to formula (8).

[0114] First, construct the adjacency matrix in the same way as spatial graph convolution The adjacency matrix B is obtained through the softmax function. Further graph pruning operations are performed to construct an adjacency tensor This tensor consists of a set of adjacency matrices composition.

[0115] in, From formula (9), we get

[0116] in, is the probability matrix P * An element of λ (set to 0.3) represents the scalar weight,

[0117] Similarly, two adjacency matrices are selected by formula (10). The adjacency matrix after pruning is pass Calculated.

[0118] Finally, the feature matrix is ​​obtained by the formula DGCB(H)=DGCL2(DGCL1(H))+ρ·H, where H is the vertex feature matrix of the pruned graph.

[0119] The intermediate features of each frame pass through the temporal graph convolution module to mine the short-range temporal dependencies between frames and aggregate inter-frame features from a local perspective to generate local aggregated features that are complementary to the output of the spatiotemporal Transformer module.

[0120] Given the uncertain differences between Transformer global features and graph convolution local features, simply combining (e.g., concatenating or adding) these features is almost ineffective in significantly improving performance. To this end, the present invention customizes a global-local feature mixing module whose mixing weights are dynamically balanced according to the input Transformer global features and graph convolution local features. Compared with using simple addition or concatenation methods, the global-local feature mixing module proposed in the present invention is more flexible and effective. With the help of the global-local feature mixing module designed by the present invention, the object detection method can enhance the collaborative representation from the Transformer and graph convolution networks, thereby improving the performance of video object detection.

[0121] Furthermore, the global aggregation features and the local aggregation features are mixed to obtain the mixed features of each frame, which specifically include:

[0122]

[0123] in, represents the mixed feature matrix of N frames, and They represent the global aggregate feature matrix output by the spatiotemporal Transformer and the local aggregate feature matrix output by the spatiotemporal graph convolutional network, respectively. Represents an element-wise aware multiplication operation. and They represent the mixer weights respectively, and the formula is:

[0124] α GF ,α LF =softmax(W α Π[G,J])#(14)

[0125] in, represents the trainable parameters of a linear mapping layer, and Π[·,·] represents the concatenation operation between features.

[0126] like Figure 4 As shown, further, the mixed features of each frame are identified and located to obtain the target detection results of each frame, including:

[0127] The mixed features and target query of each frame are input into the Transformer decoder, and after the output of the Transformer decoder, they pass through the feedforward network. The two parallel feedforward networks predict the target bounding box and target category respectively to obtain the target detection result of each frame.

[0128] The number of target queries per frame is set to 80. Target queries are a set of learnable vectors used to generate the final detection results. These query vectors do not depend on the specific content in the image, but interact with image features through the Transformer structure to obtain detection information of each target such as (position, category, etc.).

[0129] Furthermore, the internal structure of the Transformer decoder includes:

[0130] A self-attention mechanism layer, a third addition and normalization module, a cross attention mechanism layer, a fourth addition and normalization module, a feed-forward neural network, and a fifth addition and normalization module connected in sequence;

[0131] The input end of the self-attention mechanism layer is also connected to the input end of the third addition and normalization module;

[0132] The output end of the third addition and normalization module is also connected to the input end of the fourth addition and normalization module;

[0133] The output terminal of the fourth addition and normalization module is also connected to the input terminal of the fifth addition and normalization module.

[0134] Furthermore, the self-attention mechanism layer includes:

[0135] First, a self-attention operation is performed on the input target query, where each target query interacts with other target queries and obtains additional useful information from them;

[0136] The mathematical formula for self-attention is:

[0137]

[0138] Among them, Q (query element), K (key element), and V (value element) all come from the target query's own representation.

[0139] The core task of the decoder is to interact the target query with the image features output by the encoder through cross-attention.

[0140] The cross attention formula is:

[0141]

[0142] The formula for criss-cross attention is similar to self-attention, but Q comes from the target query, while V comes from the mixed features, and K is the result of adding the mixed features and the position encoding. The feed-forward network is applied to each position separately and identically, consisting of two layers of 1×1 convolutions, and the ReLU activation function is applied after each layer. After these two layers, there is also summation and normalization. The third, fourth, and fifth summation and normalization modules are the same as the previous formula.

[0143] The video object detection method proposed in this paper makes full use of the advantages of Transformer and graph convolutional networks through three key technical improvements, while making up for their limitations, combining their advantages to explore local and global information at the same time. Its overall architecture is as follows Figure 1 Given an input video sequence, each video frame is first extracted by a weight-sharing backbone network to obtain the corresponding frame features, and then these frame features and the corresponding position features generated by sine and cosine encoding (hereinafter referred to as position features) are added and sent to the spatiotemporal Transformer module as shown in Figure 2 The spatiotemporal graph convolution module is shown in Figure 3 As shown in , the global and local aggregate features are output respectively. Next, the global and local aggregate features are input into the global-local feature mixing module to generate mixed features. Finally, the mixed features are input into the Transformer decoder together with the target query to predict the detection results for each input frame as shown in Figure 4 In view of the module advantages of the present invention, a parallel detection method is adopted in the framework, which can detect targets on all input frames at the same time, so that the video target detection method has the ability of real-time reasoning.

[0144] Embodiment 2

[0145] This embodiment provides a video object detection system based on spatiotemporal graph convolution, including:

[0146] The target detection module is configured to: obtain a video to be detected, input all frame images in the video to be detected into the trained target detection model in batches, and obtain target detection results for all frame images;

[0147] Among them, the trained target detection model extracts features from each frame of the image to obtain the image features of each frame; extracts global aggregation features and local aggregation features from the image features of all frames respectively, mixes the global aggregation features and the local aggregation features to obtain the mixed features of each frame; identifies and locates the mixed features of each frame to obtain the target detection results of each frame.

[0148] It should be noted here that the above target detection module corresponds to the step of obtaining the video to be detected in Embodiment 1, inputting all frame images in the video to be detected into the trained target detection model in batches, and obtaining the target detection results of all frame images. The examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0149] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0150] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0151] Embodiment 3

[0152] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

[0153] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0154] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0155] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0156] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0157] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0158] Embodiment 4

[0159] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video object detection method based on spatiotemporal graph convolution, characterized by: include: Obtain the video to be detected, input all frame images in the video to be detected into the trained target detection model in batches, and obtain the target detection results of all frame images; Among them, the trained target detection model extracts features from each frame of the image to obtain the image features of each frame; extracts global aggregation features and local aggregation features from the image features of all frames respectively, mixes the global aggregation features and the local aggregation features to obtain the mixed features of each frame; identifies and locates the mixed features of each frame to obtain the target detection results of each frame.

2. The video target detection method based on spatiotemporal graph convolution as claimed in claim 1, characterized in that: The trained object detection model includes: Several parallel backbone networks, each backbone network inputs the frame image to be detected, and each backbone network outputs the image features of the corresponding frame image, and the frame features and corresponding position features of each frame image are input to the input end of the spatiotemporal Transformer module; at the same time, the frame features and corresponding position features of each frame image are input to the input end of the spatiotemporal graph convolution module; The output of the spatiotemporal Transformer module is connected to the input of the global-local feature mixing module, and the output of the spatiotemporal graph convolution module is connected to the input of the global-local feature mixing module; The output end of the global-local feature mixing module outputs the mixed features of each frame of the image; the mixed features of each frame are input to the input end of the corresponding decoder, the output end of the decoder is connected to the input end of the corresponding feedforward network, and the output end of the feedforward network outputs the target detection result of each frame of the image.

3. The video target detection method based on spatiotemporal graph convolution as claimed in claim 2, characterized in that: The spatiotemporal Transformer module comprises: N parallel branches, each branch includes an adder, a spatial multi-head self-attention mechanism layer and a first addition and normalization module connected in sequence, the input end of the first addition and normalization module is also connected to the output end of the adder; the input end of the adder of the nth branch is used to input the frame features and corresponding position features of the nth frame; the output end of the first addition and normalization module of the nth branch outputs the first intermediate features of the nth frame image, and the first intermediate features of the N frames of image are input into the temporal multi-head self-attention mechanism layer, the temporal multi-head self-attention mechanism layer inputs the output value into the second addition and normalization module, the second addition and normalization module outputs the global aggregation features of each frame of image; the input end of the second addition and normalization module also inputs the first intermediate features.

4. The video target detection method based on spatiotemporal graph convolution as claimed in claim 1, characterized in that: The global aggregation features and local aggregation features are extracted from the image features of all frames, including: Through the spatiotemporal Transformer module, the global aggregate features of each frame are extracted from the image features of all frames; Through the spatiotemporal graph convolution module, the local aggregate features of each frame are extracted from the image features of all frames.

5. The video target detection method based on spatiotemporal graph convolution as claimed in claim 4, characterized in that: The spatiotemporal Transformer module extracts the global aggregate features of the i-th frame from the image features of all frames, including: Given a backbone network extracts the nth frame feature Where c, h, and w represent the dimension, height, and width of the frame features, respectively; Add the features of each frame and the corresponding position features together to get the added features And the added features Input into the spatial multi-head self-attention mechanism layer to explore the long-range spatial dependencies within the frame and perform intra-frame feature aggregation from a global perspective; Among them, the position feature is calculated using sine and cosine coding, and its formula is: Among them, pos represents the position, i represents the dimension, sine encoding is used for even channels, and cosine encoding is used for odd channels; The calculation formula of the spatial multi-head self-attention mechanism layer SMHSA is: Where q∈Ω q Represents a query element, whose characteristics are k∈Ω k Represents a key element, whose characteristics are D stands for z q and x k The characteristic dimension of q and Ω k denote the set of query and key elements respectively; T denotes the total number of attention heads, K denotes the total number of key elements, and represents the learnable mapping weight, D v =D / T; the symbol "·" indicates a scalar multiplication operation; D v represents the dimension of each attention head; O tqk The calculation formula is: Among them, O tqk represents the self-attention weight of the k-th sampled key element in the t-th attention head, and is normalized over all key elements; ∝ means proportional to the operation; Represents the transpose operation; and represents the learnable mapping weights; Next, the output features of the spatial multi-head self-attention mechanism layer SMHSA are sequentially passed through the addition and normalization modules to generate the intermediate features of the current frame; The addition and normalization module ensures the flow of information and stable training through residual connection and normalization. The implementation method is: AddLN=LayerNorm(x+SMHSA)#(5) This process first performs residual connection, and then uses layer normalization LayerNorm to ensure numerical stability and avoid gradient disappearance or explosion problems; Afterwards, the intermediate features of each frame are input into the temporal multi-head self-attention mechanism layer to mine the long-range temporal dependencies between frames and perform inter-frame feature aggregation from a global perspective; The calculation of the temporal multi-head self-attention mechanism layer TMHSA is expressed as: Among them, x n represents the intermediate feature, n represents the index of the input frame, O tnqk represents the self-attention weight of the k-th sampled element in the t-th attention head on the n-th frame; O tnqk The mathematical formula for this is: The normalization method is Finally, the output features of the temporal multi-head self-attention mechanism layer are sequentially operated through addition and normalization modules to generate global aggregate features.

6. The video target detection method based on spatiotemporal graph convolution as claimed in claim 4, characterized in that: The spatiotemporal graph convolution module extracts local aggregate features of each frame from the image features of all frames, including: (1-1) Add the frame features and corresponding position features of the nth frame, and input the addition result into the spatial graph convolution unit; the spatial graph convolution unit is used to explore the short-range spatial dependency within the frame and perform intra-frame feature aggregation from a local perspective; (1-2) The spatial graph convolution unit first constructs an undirected graph. In the process of constructing the undirected graph, each input feature is taken as a vertex, and the edge is determined according to the correlation between the vertices. Based on the constructed edges, the adjacency matrix is ​​determined. (1-3) constructing an adjacency tensor based on the adjacency matrix; selecting two adjacency matrices from the adjacency tensor; obtaining a final pruned adjacency matrix through the two adjacency matrices; determining a pruned undirected graph based on the final pruned adjacency matrix; obtaining the nth frame intermediate features output by the spatial graph convolution unit based on the vertex feature matrix of the undirected graph; (1-4) The intermediate features of all frames are input into the temporal graph convolution unit to obtain local aggregated features.

7. The video object detection method based on spatiotemporal graph convolution as claimed in claim 1, characterized in that: The global aggregation features and local aggregation features are mixed to obtain the mixed features of each frame, including: in, represents the mixed feature matrix of N frames, and They represent the global aggregate feature matrix output by the spatiotemporal Transformer and the local aggregate feature matrix output by the spatiotemporal graph convolutional network, respectively. Represents an element-wise multiplication operation; and They represent the mixer weights respectively, and the formula is: a GF ,a LF =softmax(W α Π[G,J])#(9) in, represents the trainable parameters of a linear mapping layer, and Π[·,·] represents the concatenation operation between features.

8. A video object detection system based on spatiotemporal graph convolution, characterized by: include: The target detection module is configured to: obtain a video to be detected, input all frame images in the video to be detected into the trained target detection model in batches, and obtain target detection results for all frame images; Among them, the trained target detection model extracts features from each frame of the image to obtain the image features of each frame; extracts global aggregation features and local aggregation features from the image features of all frames respectively, mixes the global aggregation features and the local aggregation features to obtain the mixed features of each frame; identifies and locates the mixed features of each frame to obtain the target detection results of each frame.

9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 7 is executed.

10. A storage medium, characterized in that: The computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the method according to any one of claims 1 to 7 is performed.

Images