Video target detection method and system based on spatio-temporal graph convolution

Abstract

The application discloses a video target detection method and system based on a space-time graph convolution, and the method comprises the following steps: acquiring a video to be detected; inputting all frame images in the video to be detected into a trained target detection model in batches to obtain target detection results of all frame images; wherein the trained target detection model extracts features of each frame image to obtain image features of each frame; global aggregation features and local aggregation features are extracted from the image features of each frame respectively, the global aggregation features and the local aggregation features are mixed to obtain mixed features of each frame; and the mixed features of each frame are recognized and positioned to obtain the target detection results of each frame. A large number of experiments on an ImageNet VID dataset prove the effectiveness of the proposed model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly to a video target detection method and system based on spatio-temporal graph convolution. BACKGROUND

[0002] In recent years, video object detection has made significant progress thanks to the convolutional neural network and visual Transformer. Convolutional neural networks are good at extracting local features, but it is difficult to extract global features. On the contrary, visual Transformer can capture features in a global range, but it faces challenges in representing local feature details. Existing video object detection methods only rely on convolutional neural networks or visual Transformers for feature aggregation, hindering their ability to utilize both global and local information, thus limiting the detection performance.

[0003] Video object detection aims to predict the location box and class label of each object in a video. In the past few decades, with the explosive growth of video data, video object detection has become a key task in the fields of autonomous driving, intelligent monitoring, intelligent transportation, unmanned aerial vehicle navigation, etc. Traditional image object detection methods are mainly used for static image processing, while in practical applications, video object detection not only needs to identify the objects in each frame, but also must handle the temporal information between frames to improve the detection accuracy and robustness.

[0004] Convolutional neural networks extract local features of images through layer-by-layer convolution operations, with excellent local perception ability. However, the inherent local perspective of convolutional neural networks limits its ability to handle long-distance dependencies and global context information. This is particularly evident when dealing with dynamic videos, as complex spatio-temporal dependencies may not be fully captured through local feature extraction, resulting in decreased detection accuracy in complex scenarios such as occlusion and blur.

[0005] Transformers have gradually been introduced into visual tasks in recent years due to their success in natural language processing. Unlike convolutional neural networks, Transformers can capture global feature dependencies through self-attention mechanisms, and are particularly good at extracting global information. However, Transformers also face certain challenges in video object detection, especially when dealing with short-range, local detail features, which are not as good as convolutional neural networks.

[0006] Therefore, there is an urgent need for a new method that can combine the local feature extraction ability of convolutional neural networks and the global relationship capturing ability of Transformers, while effectively overcoming the limitations of existing technologies to improve the accuracy and real-time performance of video object detection. SUMMARY

[0007] In order to solve the problems in the prior art, the application provides a video target detection method and system based on spatio-temporal graph convolution. First, a spatio-temporal Transformer module is developed to aggregate global context information and form a global representation with long-range feature dependency. Second, a spatio-temporal graph convolution module is introduced to aggregate features using local spatial and temporal relationships to generate local features complementary to the spatio-temporal Transformer output. Third, a global-local feature mixing module is designed to adaptively couple the Transformer-based global representation and the graph convolution-based local representation. Extensive experiments on the ImageNet VID dataset demonstrate the effectiveness of the proposed model.

[0008] In one aspect, a video target detection method based on spatio-temporal graph convolution is provided, comprising:

[0009] Obtaining a video to be detected, inputting all frame images in the video to be detected into a trained target detection model in batches to obtain target detection results of the frame images.

[0010] The trained target detection model extracts features of each frame image to obtain image features of each frame, extracts global aggregation features and local aggregation features from the image features of all frames, mixes the global aggregation features and the local aggregation features to obtain mixed features of each frame, and identifies and locates the mixed features of each frame to obtain the target detection results of each frame.

[0011] In another aspect, a video target detection system based on spatio-temporal graph convolution is provided, comprising:

[0012] A target detection module configured to obtain a video to be detected, input all frame images in the video to be detected into a trained target detection model in batches to obtain target detection results of the frame images.

[0013] The trained target detection model extracts features of each frame image to obtain image features of each frame, extracts global aggregation features and local aggregation features from the image features of all frames, mixes the global aggregation features and the local aggregation features to obtain mixed features of each frame, and identifies and locates the mixed features of each frame to obtain the target detection results of each frame.

[0014] In still another aspect, an electronic device is also provided, comprising:

[0015] A memory for non-transitory storage of computer readable instructions; and

[0016] A processor for running the computer readable instructions,

[0017] The computer readable instructions, when executed by the processor, perform the method of the first aspect.

[0018] In another aspect, a storage medium is also provided, which stores non-transitory computer readable instructions, wherein when the non-transitory computer readable instructions are executed by a computer, the method of the first aspect is performed.

[0019] In another aspect, a computer program product is also provided, which includes a computer program for implementing the method of the first aspect when executed on one or more processors.

[0020] The above technical solution has the following advantages or beneficial effects:

[0021] The present application overcomes the limitations of the prior art by rationally combining the advantages of convolutional neural networks and visual Transformers. Convolutional neural networks perform excellently in extracting local information, while visual Transformers are suitable for capturing global features. This patent effectively utilizes the local perception ability of convolutional neural networks and the global perception ability of Transformers, achieving more comprehensive feature extraction and information aggregation, thereby improving the accuracy and efficiency of target detection.

[0022] This patent technology has a wide impact on multiple industries, especially in the field of visual information requiring high precision and real-time processing. For example, in intelligent monitoring and security systems, the reliability of public safety can be improved through more accurate target recognition; in autonomous driving and intelligent transportation, this technology improves the safety and response speed of the system through more comprehensive scene perception. In augmented reality (AR) and virtual reality (VR) applications, this technology helps to improve the accuracy of scene recognition and target tracking, enhancing user experience. These industries will significantly improve in terms of accuracy, efficiency and safety through the application of this technology. BRIEF DESCRIPTION OF DRAWINGS

[0023] The drawings accompanying this specification form a part of the specification. Such drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The embodiments illustrated in the drawings are intended to explain the aspects of the present application and are not limiting of the present application in any way.

[0024] Figure 1 The method flowchart of Example One;

[0025] Figure 2 The internal structure diagram of the space-time Transformer module of Example One;

[0026] Figure 3 The internal structure diagram of the space-time graph convolution module of Example One;

[0027] Figure 4 Figure 1 is a schematic diagram of the connection relationship between the Transformer decoder and the feedforward network of Embodiment 1. DETAILED DESCRIPTION

[0028] It should be noted that the following detailed description is exemplary and is intended to further explain the present application. Unless otherwise indicated, all technical and scientific terms used in the present application have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0029] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0030] All data acquisition of the present embodiment is based on compliance with laws and regulations and user consent, and legal application of data.

[0031] Embodiment 1

[0032] The present embodiment provides a video target detection method based on spatio-temporal graph convolution;

[0033] The video target detection method based on spatio-temporal graph convolution comprises:

[0034] Obtaining a video to be detected, inputting all frame images in the video to be detected into a trained target detection model in batches to obtain target detection results of all frame images;

[0035] Wherein, the trained target detection model extracts features of each frame image to obtain image features of each frame; global aggregation features and local aggregation features are extracted from the image features of all frames, respectively, the global aggregation features and the local aggregation features are mixed to obtain mixed features of each frame; the mixed features of each frame are recognized and positioned to obtain target detection results of each frame.

[0036] Further, the training process of the trained target detection model comprises:

[0037] Constructing a training set, the training set being video frame images with known target detection labels;

[0038] Inputting the training set into the target detection model to train the model, stopping the training when the loss function value of the model no longer decreases, or the number of iterations reaches a set number of times, to obtain the trained target detection model.

[0039] Further, the training set refers to: given a video data set, the data set contains a plurality of videos, and each video contains a plurality of video frames The video target detection method of the present application takes continuous N frames (default setting is 25 frames) as input and outputs detection results of all input frames at one time.

[0040] Further, the trained target detection model comprises:

[0041] a plurality of parallel backbone networks, an input end of each backbone network inputs a frame image to be detected, and an output end of each backbone network outputs image features (referred to as frame features) of the corresponding frame image; the image features and the corresponding position features of each frame image are jointly input to an input end of a space-time Transformer module; meanwhile, the image features and the corresponding position features of each frame are jointly input to an input end of a space-time graph convolution module;

[0042] an output end of the space-time Transformer module is connected with an input end of a global-local feature mixing module, and an output end of the space-time graph convolution module is connected with the input end of the global-local feature mixing module;

[0043] an output end of the global-local feature mixing module outputs mixed features of each frame image; the mixed features of each frame are input to an input end of a corresponding decoder, an output end of the decoder is connected with an input end of a corresponding multi-layer perception machine, and an output end of the multi-layer perception machine outputs a target detection result of each frame image.

[0044] Further, as shown in Figure 2 the space-time Transformer module comprises:

[0045] N parallel branches, each branch comprises a adder, a spatial multi-head self-attention mechanism layer and a first addition and normalization module connected in sequence, an input end of the first addition and normalization module is also connected with an output end of the adder; an input end of the adder of the nth branch is used for inputting frame features of the nth frame and corresponding position features; an output end of the first addition and normalization module of the nth branch outputs first intermediate features of the nth frame image; the first intermediate features of the N frame images are all input to a time multi-head self-attention mechanism layer, the time multi-head self-attention mechanism layer inputs an output value to a second addition and normalization module, and the second addition and normalization module outputs global aggregation features of each frame image; an input end of the second addition and normalization module also inputs the first intermediate features.

[0046] Further, as shown in Figure 3 the space-time graph convolution module comprises:

[0047] N parallel branches, each branch includes a adder and a spatial graph convolution unit connected in turn; the input end of the adder of the nth branch is used for inputting the frame feature and the corresponding position feature of the nth frame; the output end of the spatial graph convolution unit of the nth branch outputs the second intermediate feature of the nth frame image; the second intermediate features of the N frame images are all input into a time graph convolution unit, and the time graph convolution unit outputs the local aggregation feature of each frame image.

[0048] Further, the trained target detection model extracts features of each frame of image to obtain image features of each frame, including:

[0049] A plurality of backbone networks are used to extract features of a plurality of frames of images, wherein one backbone network corresponds to one frame of image, and finally, image features of all frames are obtained. ResNet-101 is used as a backbone network in the application, DETR is used as a baseline network, and the total stride of the last stage (i.e. Conv5) of ResNet-101 is adjusted from 32 to 16.

[0050] Further, global aggregation features and local aggregation features are extracted from the image features of all frames, specifically including:

[0051] The global aggregation feature of each frame is extracted from the image features of all frames through a space-time Transformer module;

[0052] The local aggregation feature of each frame is extracted from the image features of all frames through a space-time graph convolution module.

[0053] Further, the global aggregation feature of each frame is extracted from the image features of all frames through the space-time Transformer module, including:

[0054] Given the nth frame feature extracted by a backbone network Where c, h and w represent the dimension, height and width of the frame feature respectively;

[0055] The features of each frame and the corresponding position features are added together to obtain the added features And the added features are input into a spatial multi-head self-attention mechanism layer to explore the long-distance spatial dependency within the frame and aggregate the features within the frame from a global perspective;

[0056] Wherein the position feature is calculated by using the sine-cosine encoding, and the formula is:

[0057]

[0058]

[0059] where pos denotes the position, i denotes the dimension, the sine encoding is used for even channels, and the cosine encoding is used for odd channels.

[0060] The calculation formula of the spatial multi-head self-attention mechanism layer (SMHSA) is as follows:

[0061]

[0062] where q e Q q denotes the query element, and its features are k e Q k denotes the key element, and its features are D denotes the feature dimension of z q and x k , Q q and Q k denote the set of query and key elements respectively. T denotes the total number of attention heads, K denotes the total number of key elements, and denote the learnable mapping weights, D v =D / T. The symbol “·” denotes the scalar multiplication operation. D v denotes the dimension of each attention head. O tqk The calculation formula is as follows:

[0063]

[0064] where O tqk denotes the self-attention weight of the k-th sampled key element in the t-th attention head, and is normalized over all key elements. a denotes the proportional operation. T denotes the transpose operation. and denote the learnable mapping weights.

[0065] Next, the output features of the spatial multi-head self-attention mechanism layer SMHSA are sequentially subjected to an addition and normalization module to generate the intermediate features of the current frame.

[0066] The addition and normalization module is used to ensure the flow of information and stable training through residual connection and normalization, and the implementation is as follows:

[0067] AddLN = LayerNorm (x + SMHSA) # (5)

[0068] This process first performs residual connection, and then uses layer normalization LayerNorm to ensure numerical stability and avoid gradient vanishing or explosion problems.

[0069] Then, the intermediate features of each frame are input into the temporal multi-head self-attention mechanism layer to mine the long-distance temporal dependency between frames and perform inter-frame feature aggregation from a global perspective.

[0070] The calculation of the temporal multi-head self-attention mechanism layer (TMHSA) is represented as:

[0071]

[0072] where x n represents the intermediate feature, n represents the index of the input frame, O tnqk represents the self-attention weight of the key element of the k-th sample in the t-th attention head on the n-th frame.

[0073] O tnqk The mathematical formula for calculating O

[0074]

[0075] The normalization method is

[0076] Finally, the output features of the temporal multi-head self-attention mechanism layer are sequentially operated by the addition and normalization modules (in the same way as the spatial multi-head self-attention), to generate global aggregated features.

[0077] The number of attention heads of the spatial and temporal Transformers is set to 6.

[0078] Further, the local aggregated features of each frame are extracted from the image features of all frames by the spatio-temporal graph convolution module, including:

[0079] (1-1) Adding the frame features of the n-th frame and the corresponding position features, and inputting the addition result into the spatial graph convolution unit; exploring the short-distance spatial dependency within the frame through the spatial graph convolution unit, and performing intra-frame feature aggregation from a local perspective;

[0080] (1-2) The spatial graph convolution unit first constructs an undirected graph; during the construction of the undirected graph, each input feature is taken as a vertex, and the edges are determined according to the association between the vertices; based on the constructed edges, an adjacency matrix is determined;

[0081] (1-3) Based on the adjacency matrix, an adjacency tensor is constructed; two adjacency matrices are selected from the adjacency tensor; the final pruned adjacency matrix is obtained through the two adjacency matrices; based on the final pruned adjacency matrix, a pruned undirected graph is determined; based on the vertex feature matrix of the undirected graph, the intermediate features of the n-th frame output by the spatial graph convolution unit are obtained;

[0082] (1-4) The intermediate features of all frames are input into a time graph convolution unit to obtain local aggregation features.

[0083] The number of dynamic graph convolution layers in the spatial and temporal graph convolution is set to 2.

[0084] Further, the (1-2) spatial graph convolution unit first constructs an undirected graph; in the construction process of the undirected graph, each input feature is taken as a vertex, and edges are determined according to the relevance between vertices; based on the constructed edges, an adjacency matrix is determined, specifically including:

[0085] First, a fully connected undirected graph is constructed wherein, ε and respectively represent a vertex set, an edge set and an adjacency matrix. In the construction of the undirected graph, each input feature (that is, the addition of each frame feature and position feature) is defined as a graph vertex v i , and each edge e ij is defined as a pair relationship between vertices v i and v j . The calculation of the edge e ij is:

[0086]

[0087] wherein, denotes a mapping function composed of two fully connected layers, Π[·,·,·] denotes a series operation. Euc(·,·) denotes a standardized Euclidean distance, Cos(·,·) denotes a cosine similarity. Sec(·,·) denotes semantic similarity, Sec(r i ,r j ) is obtained by calculation.

[0088] Based on the constructed edge e ij , the adjacency matrix A is obtained by a softmax function, and each element A ij in the adjacency matrix A is calculated as:

[0089] Since the goal of the spatio-temporal graph convolution module is to perform feature aggregation from the local perspective of graph convolution, directly using a fully connected graph cannot well model the local relationship between vertex features, and in this case, graph convolution may cause aggregation of useless information (such as inter-class targets and backgrounds) and introduce a large amount of additional computational overhead.

[0090] To alleviate these problems, a graph pruning mechanism is proposed to sparsify the topology of the graph by removing useless or weakly connected edges to focus on useful local relations between vertices. Previous methods mostly use a single threshold to prune the normalized adjacency matrix, which causes the problem of difficult-to-distinguish feature representations because the matrix elements below the threshold are eliminated. To this end, a novel graph pruning mechanism is proposed.

[0091] Further, the (1-3) constructs an adjacency tensor based on the adjacency matrix, specifically comprising:

[0092] The graph pruning mechanism uses S real number thresholds, denoted as Γ=[θ1,θ2,…,θ S ],θ i <θ j , and θ i , θ j ∈[0,1] holds for all i<j.

[0093] An adjacency tensor is constructed The tensor is composed of a set of adjacency matrices , where,

[0094] Set A 1 to the identity matrix E, for each s≥2, The mathematical calculation can be defined as:

[0095]

[0096] Where P ij is an element in the probability matrix P, which can be calculated by P ij =λ·A ij / d i . λ (set to 0.3) represents a scalar weight, is an element in the diagonal matrix .

[0097] Further, the (1-3) selects two adjacency matrices from the adjacency tensor, specifically comprising:

[0098] Two adjacency matrices Q1 and Q2 are selected from the adjacency tensor by two 1×1 convolutions, where the non-negative weights are obtained by the softmax function:

[0099]

[0100] Where, and represent two different 1×1 convolution layers. and denote learnable parameters.

[0101] Further, the (1-3) obtains a final pruned adjacency matrix through the two adjacency matrices, and determines a pruned undirected graph based on the final pruned adjacency matrix; specifically comprising:

[0102] Final pruned adjacency matrix by is calculated. ψ(·) denotes a Laplacian normalization operation, and is calculated by .

[0103] Further, the (1-3) obtains the n-th frame intermediate feature output by the spatial graph convolution unit based on the vertex feature matrix of the pruned undirected graph, comprising:

[0104] Based on the pruned graph, the graph convolution network is extended from a static setting to a dynamic setting, and a dynamic graph convolution module with residual connection is designed to effectively aggregate features from local adjacent positions. The proposed dynamic graph convolution module (DGCB) is defined as:

[0105] DGCB(H)=DGCL2(DGCL1(H))+ρ·H#(11)

[0106] wherein, is the vertex feature matrix of the pruned graph, and ρ represents a residual constant, and DGCL1(·) and DGCL2(·) represent two consecutive dynamic graph convolution layers.

[0107] The goal of dynamic graph convolution is to dynamically adjust the adjacency matrix or edge weight of the graph, so that the graph structure can be adaptively adjusted according to the input features of each layer, thereby capturing more complex relationships. The calculation formula of DGCL1 and DGCL2 is:

[0108]

[0109] wherein, H (i) denotes the vertex feature matrix of the i-th layer. denotes the adjacency matrix of Laplacian normalization on , which is used to represent the connection relationship between vertices. W (i) denotes the learnable weight matrix of the i-th layer. σ denotes an activation function. H 0 is the vertex feature matrix of the pruned graph, H 1 is the output of the first layer dynamic graph convolution, and H 2 is the output of the second layer dynamic graph convolution.

[0110] Unlike the graph convolution using a fixed adjacency matrix in the traditional graph convolutional network, the dynamic graph convolution proposed by the application helps to capture the dynamic graph structure in the graph convolution process, improving the effectiveness of the graph convolution. The output of the spatial graph convolution unit is represented as the intermediate feature.

[0111] The spatial graph convolution and the temporal graph convolution have the same workflow, but they differ in input features. Specifically, in the spatial graph convolution, the input feature is the element-wise addition of the frame feature and the position feature. In the temporal graph convolution, the input feature is the multi-frame intermediate feature output by the spatial graph convolution.

[0112] Further, the (1-4) inputs the intermediate features of all frames to the temporal graph convolution unit to obtain the local aggregation feature, specifically including:

[0113] In the temporal graph convolution, each vertex corresponds to the multi-frame intermediate feature output by the spatial graph convolution, and each edge in the temporal graph convolution is defined as the relationship between vertices and The calculation of the edge refers to formula (8).

[0114] First, the adjacency matrix is constructed in the same way as the spatial graph convolution. Further, the graph pruning operation is performed to construct an adjacency tensor The tensor is composed of a group of adjacency matrices .

[0115] wherein, is obtained from formula (9)

[0116] wherein, is an element in the probability matrix P * , which can be calculated by . Lambda (set to 0.3) represents a scalar weight,

[0117] Similarly, two adjacency matrices are selected from formula (10). The pruned adjacency matrix is obtained by .

[0118] Finally, the feature matrix is obtained from the formula DGCB(H) = DGCL2(DGCL1(H)) + p·H. Wherein, H is the pruned graph vertex feature matrix.

[0119] The intermediate features of each frame are processed by the temporal graph convolution module to uncover short-range temporal dependencies between frames, and inter-frame features are aggregated from a local perspective to generate local aggregated features that are complementary to the output of the spatiotemporal Transformer module.

[0120] Given the uncertain differences between Transformer global features and graph convolutional local features, simply combining (e.g., concatenating or adding) these features yields almost no significant performance improvement. Therefore, this invention customizes a global-local feature fusion module whose fusion weights are dynamically balanced based on the input Transformer global features and graph convolutional local features. Compared to using simple addition or concatenation methods, the proposed global-local feature fusion module is more flexible and effective. With the help of the global-local feature fusion module designed in this invention, object detection methods can enhance the collaborative representation from Transformer and graph convolutional networks, thereby improving the performance of video object detection.

[0121] Furthermore, the global aggregated features and local aggregated features are blended to obtain the blended features for each frame, specifically including:

[0122]

[0123] in, Represents the mixed feature matrix of N frames. and Let denot represent the global aggregated feature matrix output by the spatiotemporal Transformer and the local aggregated feature matrix output by the spatiotemporal graph convolutional network, respectively. This indicates element-aware multiplication. and These represent the mixer weights, and their formulas are as follows:

[0124] α GF ,α LF =softmax(W α Π[G,J])#(14)

[0125] in, Π[·,·] represents the trainable parameters of a linear mapping layer, and Π[·,·] represents the concatenation operation between features.

[0126] like Figure 4 As shown, further, the mixed features of each frame are identified and located to obtain the target detection results for each frame, including:

[0127] The mixed features of each frame and the target query are input into a Transformer decoder, and after the output of the Transformer decoder, a feedforward network is passed through, two parallel feedforward networks respectively predict the bounding box and target category of the target to obtain the target detection result of each frame.

[0128] wherein the number of target queries of each frame is set to 80. The target query is a set of learnable vectors used to generate the final detection result. These query vectors do not depend on the specific content in the image, but interact with the image features through the Transformer structure to obtain the detection information of each target such as (position, category, etc.).

[0129] Further, the Transformer decoder has an internal structure comprising:

[0130] a self-attention mechanism layer, a third addition and normalization module, a cross-attention mechanism layer, a fourth addition and normalization module, a feedforward neural network and a fifth addition and normalization module connected in sequence;

[0131] The input end of the self-attention mechanism layer is also connected to the input end of the third addition and normalization module;

[0132] The output end of the third addition and normalization module is also connected to the input end of the fourth addition and normalization module;

[0133] The output end of the fourth addition and normalization module is also connected to the input end of the fifth addition and normalization module.

[0134] Further, the self-attention mechanism layer comprises:

[0135] First, perform a self-attention operation on the input target query, each target query interacts with other target queries and obtains additional useful information therefrom;

[0136] The mathematical formula of self-attention is:

[0137]

[0138] wherein Q (query element), K (key element) and V (value element) are all from the self-representation of the target query.

[0139] The core task of the decoder is to interact the target query with the image features output by the encoder through cross-attention.

[0140] The cross-attention formula is:

[0141]

[0142] The formula for cross-attention is similar to that for self-attention, but Q comes from the target query, while V comes from the mixed features, and K is the result of adding the mixed features and the positional encoding. The feedforward network is applied separately and identically to each position, consisting of two 1×1 convolutional layers, with a ReLU activation function applied after each layer. After these two layers, there are addition and normalization steps. The third, fourth, and fifth addition and normalization modules use the same formulas as before.

[0143] The video object detection method proposed in this invention fully leverages the advantages of Transformer and Graph Convolutional Networks while overcoming their limitations through three key technological improvements, combining their strengths to explore both local and global information. Its overall architecture is as follows: Figure 1 As shown. Given an input video sequence, each video frame is first processed by a backbone network with shared weights to extract corresponding frame features. These frame features are then added to the corresponding positional features generated by sine and cosine coding (hereinafter referred to as positional features) and sent together to the spatiotemporal Transformer module, as shown. Figure 2 As shown, the spatiotemporal graph convolution module is as follows: Figure 3 As shown, global and local aggregated features are output separately. Next, the global and local aggregated features are input into the global-local feature fusion module to generate blended features. Finally, the blended features are input along with the target query into the Transformer decoder to predict the detection results for each input frame, as shown. Figure 4 As shown. Given the modular advantages of this invention, the framework employs a parallel detection method, enabling simultaneous detection of targets on all input frames, thus giving the video target detection method real-time inference capabilities.

[0144] Example 2

[0145] This embodiment provides a video target detection system based on spatiotemporal graph convolution, including:

[0146] The object detection module is configured to: acquire the video to be detected, input all frame images in the video to be detected in batches into the trained object detection model, and obtain the object detection results of all frame images;

[0147] The trained target detection model extracts features from each frame of the image to obtain the image features of each frame; it extracts global aggregate features and local aggregate features from the image features of all frames respectively, mixes the global aggregate features and local aggregate features to obtain the mixed features of each frame; and it identifies and locates the mixed features of each frame to obtain the target detection result of each frame.

[0148] It should be noted that the target detection module corresponds to the obtaining the video to be detected in Embodiment One, and all frame images in the video to be detected are input into the trained target detection model in batches to obtain the target detection result of all frame images. The module and the corresponding steps achieve the same examples and application scenarios, but are not limited to the content disclosed in Embodiment One. It should be noted that the module can be executed in a computer system such as a set of computer executable instructions as part of the system.

[0149] The description of each embodiment in the above embodiments has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0150] The proposed system can be implemented in other ways. For example, the system embodiments described above are only illustrative, for example, the division of the above modules is only a logical function division, and in actual implementation, there can be another division method, for example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0151] Embodiment Three

[0152] The embodiment also provides an electronic device, including one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method described in Embodiment One.

[0153] It should be understood that in the embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSPs, application-specific integrated circuits ASICs, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0154] The memory can include read-only memory and random access memory, and provide instructions and data for the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0155] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software.

[0156] The method in the embodiment one can be directly embodied as being completed by a hardware processor, or being completed by a combination of hardware and software modules in the processor. The software modules can be located in a storage medium in the art, such as a random access memory, a flash memory, a read only memory, a programmable read only memory, an electrically erasable programmable memory, a register, or the like. The storage medium is located in the storage, and the processor reads information in the storage, and combines the hardware to complete the steps of the above method. To avoid repetition, no further detailed description is given here.

[0157] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0158] Embodiment four

[0159] The embodiment also provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in embodiment one.

[0160] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A video object detection method based on spatiotemporal graph convolution, characterized by: include: The video to be detected is acquired, and all frame images in the video are input into the trained target detection model in batches to obtain the target detection results of all frame images. The trained target detection model extracts features from each frame of the image to obtain the image features of each frame; it extracts global aggregate features and local aggregate features from the image features of all frames respectively, mixes the global aggregate features and local aggregate features to obtain the mixed features of each frame; and it identifies and locates the mixed features of each frame to obtain the target detection result of each frame. The trained object detection model includes: Several parallel backbone networks are used. The input of each backbone network is the frame image to be detected, and the output of each backbone network is the image features of the corresponding frame image. The frame features and corresponding position features of each frame image are input to the input of the spatiotemporal Transformer module. At the same time, the frame features and corresponding position features of each frame image are input to the input of the spatiotemporal graph convolution module. The output of the spatiotemporal Transformer module is connected to the input of the global-local feature mixing module, and the output of the spatiotemporal graph convolution module is connected to the input of the global-local feature mixing module. The output of the global-local feature fusion module is the fused features of each frame of the image. The fused features of each frame are input to the input of the corresponding decoder. The output of the decoder is connected to the input of the corresponding feedforward network. The output of the feedforward network is the target detection result of each frame of the image.

2. The video target detection method based on spatiotemporal graph convolution as described in claim 1, characterized in that, The spatiotemporal Transformer module includes: Each of the following parallel branches includes an adder, a spatial multi-head self-attention mechanism layer, and a first addition and normalization module connected in sequence. The input of the first addition and normalization module is also connected to the output of the adder. The input terminal of the adder in the branch is used to input the first... Frame features and corresponding positional features of the frame; the first The output of the first adder and normalization module of the branch outputs the first... The first intermediate feature of the frame image will The first intermediate features of each frame image are input into the temporal multi-head self-attention mechanism layer. The temporal multi-head self-attention mechanism layer inputs the output value into the second addition and normalization module. The second addition and normalization module outputs the global aggregated features of each frame image. The first intermediate features are also input into the second addition and normalization module.

3. The video target detection method based on spatiotemporal graph convolution as described in claim 1, characterized in that, Global aggregated features and local aggregated features are extracted from the image features of all frames, specifically including: The spatiotemporal Transformer module extracts the global aggregated features of each frame from the image features of all frames. The spatiotemporal graph convolution module extracts the local aggregated features of each frame from the image features of all frames.

4. The video target detection method based on spatiotemporal graph convolution as described in claim 3, characterized in that, The method involves using the spatiotemporal Transformer module to extract the first [frame's] image features from all frames. Global aggregate features of the frame include: Given a backbone network extracting the first... Frame features ,in , and These represent the dimensions, height, and width of the frame feature, respectively. The features of each frame and its corresponding location features are added together to obtain the summed features. and the features after addition The input is fed into the spatial multi-head self-attention mechanism layer to explore long-range spatial dependencies within the frame and to aggregate intra-frame features from a global perspective; The location features are calculated using sine and cosine coding, and the formula is as follows: in, Indicates location, To represent the dimension, sine coding is used for even channels, and cosine coding is used for odd channels; Spatial multi-head self-attention mechanism layer The calculation formula is: in, This represents the query element, characterized by: , Represents a key element, characterized by: ; express and Feature dimensions, and These represent the sets of query elements and key elements, respectively. This represents the total number of heads of attention. Indicates the total number of key elements. and Represents the learnable mapping weights. ;symbol" " indicates a scalar multiplication operation; The dimension representing each attention head; The calculation formula is: in, Indicates the first The first one in the attention. The self-attention weights of each sampled key element are normalized over all key elements. This indicates that it is proportional to the operation; Indicates the transpose operation; and Represents the learnable mapping weights; Next, the output features of the Spatial Multi-Head Self-Attention (SMHSA) layer are sequentially processed by addition and normalization modules to generate the intermediate features of the current frame. The addition and normalization modules ensure information flow and stable training through residual connections and normalization. The implementation method is as follows: This process first involves residual connections, followed by layer normalization. To ensure numerical stability and avoid gradient vanishing or exploding problems; Subsequently, the intermediate features of each frame are input into the temporal multi-head self-attention mechanism layer to mine long-distance temporal dependencies between frames and perform inter-frame feature aggregation from a global perspective. Time-based multi-head self-attention mechanism layer The calculation is expressed as: in, Indicates intermediate features, Indicates the index of the input frame. Indicates the first The first one in the attention. The sampling element at the th ... Self-attention weights on frames; The mathematical formula for calculation is: The normalization method is ; Finally, the output features of the temporal multi-head self-attention mechanism layer are sequentially processed by addition and normalization modules to generate global aggregate features.

5. The video target detection method based on spatiotemporal graph convolution as described in claim 3, characterized in that, The process of extracting local aggregated features for each frame from the image features of all frames using the spatiotemporal graph convolution module includes: (1-1) The first The frame features and corresponding location features of a frame are added together, and the result is input into a spatial graph convolutional unit. The spatial graph convolutional unit is used to explore short-range spatial dependencies within the frame and to aggregate intra-frame features from a local perspective. (1-2) Spatial graph convolutional unit: First, an undirected graph is constructed; during the construction of the undirected graph, each input feature is treated as a vertex, and the edges are determined based on the correlation between vertices; based on the constructed edges, the adjacency matrix is ​​determined. (1-3) Construct an adjacency tensor based on the adjacency matrix; select two adjacency matrices from the adjacency tensor; obtain the final pruned adjacency matrix through the two adjacency matrices; determine the pruned undirected graph based on the final pruned adjacency matrix; obtain the first vertex feature matrix output by the spatial graph convolution unit based on the vertex feature matrix of the undirected graph. Frame mid-frame features; (1-4) Input the intermediate features of all frames into the temporal map convolution unit to obtain local aggregated features.

6. The video target detection method based on spatiotemporal graph convolution as described in claim 1, characterized in that, The global aggregated features and local aggregated features are blended to obtain the blended features for each frame, specifically including: in, express N The mixed feature matrix of the frame, and Let these represent the global aggregated feature matrix output by the spatiotemporal Transformer and the local aggregated feature matrix output by the spatiotemporal graph convolutional network, respectively. This indicates an element-aware multiplication operation; and These represent the mixer weights, and their formulas are as follows: in, This represents the trainable parameters of a linear mapping layer. This indicates a chain operation between features.

7. A video target detection system based on spatiotemporal graph convolution, characterized in that, include: The object detection module is configured to: acquire the video to be detected, input all frame images in the video to be detected in batches into the trained object detection model, and obtain the object detection results of all frame images; The trained target detection model extracts features from each frame of the image to obtain the image features of each frame; it extracts global aggregate features and local aggregate features from the image features of all frames respectively, mixes the global aggregate features and local aggregate features to obtain the mixed features of each frame; and it identifies and locates the mixed features of each frame to obtain the target detection result of each frame. The trained object detection model includes: Several parallel backbone networks are used. The input of each backbone network is the frame image to be detected, and the output of each backbone network is the image features of the corresponding frame image. The frame features and corresponding position features of each frame image are input to the input of the spatiotemporal Transformer module. At the same time, the frame features and corresponding position features of each frame image are input to the input of the spatiotemporal graph convolution module. The output of the spatiotemporal Transformer module is connected to the input of the global-local feature mixing module, and the output of the spatiotemporal graph convolution module is connected to the input of the global-local feature mixing module. The output of the global-local feature fusion module is the fused features of each frame of the image. The fused features of each frame are input to the input of the corresponding decoder. The output of the decoder is connected to the input of the corresponding feedforward network. The output of the feedforward network is the target detection result of each frame of the image.

8. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-6.

9. A storage medium, characterized in that, Non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method of any one of claims 1-6 is performed.

Images