Video target detection method and system based on alternating decoupling

Abstract

The application discloses an alternating decoupling-based video target detection method and system. The method comprises the following steps: acquiring a video to be detected; performing frame division on the video to be detected to obtain a plurality of frame images; performing target detection on each frame image by using a trained target detection model to obtain a target detection result of each frame image; wherein the process of performing target detection on each frame image by using the trained target detection model comprises the following steps: extracting frame features of the frame image; encoding the frame features to obtain encoded features; extracting space-time features and time-space features from the encoded features; coupling the space-time features and the time-space features to obtain coupled features of the frame image; and identifying the coupled features to obtain the target detection result of the frame image. The application realizes accurate detection of video targets.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a video target detection method and system based on alternating decoupling. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, video object detection has attracted widespread attention in the field of computer vision. Currently, most existing video object detection methods either use learnable modules to aggregate spatial and temporal features or follow a continuous spatial-temporal paradigm to aggregate features. However, the potential of these methods in extracting spatial and temporal information has not been fully explored, resulting in limited detection performance. More importantly, these feature aggregation methods may lead to feature collapse problems, which in turn can cause false positives or false negatives. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a video target detection method and system based on alternating decoupling, which improves the accuracy of target detection.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] Firstly, a video target detection method based on alternating decoupling is proposed, including:

[0007] Obtain the video to be tested;

[0008] The video to be tested is divided into frames to obtain multiple frame images;

[0009] The trained object detection model is used to perform object detection on each frame of the image to obtain the object detection result for each frame. The process of the trained object detection model performing object detection on each frame of the image is as follows: extracting frame features of the frame image; encoding the frame features to obtain encoded features; extracting spatial-temporal features and temporal-space features from the encoded features; coupling the spatial-temporal features and temporal-space features to obtain the coupled features of the frame image; and recognizing the coupled features to obtain the object detection result of the frame image.

[0010] Furthermore, the object detection model extracts spatial-temporal features from the encoded features through a spatial-temporal decoupled Transformer decoder; the process of extracting spatial-temporal features through a spatial-temporal decoupled Transformer decoder includes: extracting spatial features from the encoded features through a spatial decoupled Transformer decoder, and extracting temporal features from these spatial features through a temporal decoupled Transformer decoder, thus obtaining spatial-temporal features; the process of extracting spatial-temporal features from the encoded features through a temporal decoupled Transformer decoder includes: extracting temporal features from the encoded features through a temporal decoupled Transformer decoder, and extracting spatial features from these temporal features through a spatial decoupled Transformer decoder, thus obtaining spatial features.

[0011] Furthermore, the spatially decoupled Transformer decoder includes a spatial mask self-attention mechanism, a spatial deformable self-attention mechanism, and a feedforward network module connected in sequence. The spatial mask self-attention mechanism processes the features of the input spatially decoupled Transformer decoder to obtain processed features. The spatial deformable self-attention mechanism interacts and fuses the processed features obtained by the spatial mask self-attention mechanism with the encoded features to obtain interactively fused features. The feedforward network module processes the interactively fused features obtained by the spatial deformable self-attention mechanism to finally output spatial features.

[0012] Furthermore, the temporally decoupled Transformer decoder includes a temporal mask self-attention mechanism, a temporally deformable attention mechanism, and a feedforward network module connected in sequence. The temporal mask self-attention mechanism processes the features of the input temporally decoupled Transformer decoder to obtain processed features. The temporally deformable attention mechanism enables the processed features obtained by the temporal mask self-attention mechanism and the encoded features to interact and fuse, resulting in interact-fused features. The feedforward network module processes the interact-fused features obtained by the temporally deformable attention mechanism to finally output temporal features.

[0013] Furthermore, the spatial-temporal features and temporal-spatial features are coupled through the multi-view structured feature coupling module to obtain coupled features. The process includes:

[0014] Construct multi-view undirected graphs from different perspectives based on space-time and time-space characteristics;

[0015] The edge-aware multi-graph fusion processor merges all undirected graphs to obtain the fused graph.

[0016] The vertex features of the fused graph are extracted by a hierarchical adaptive graph convolutional network, and the fused graph is updated using the vertex features to obtain the updated graph.

[0017] Based on the updated graph, the coupling characteristics are obtained.

[0018] Furthermore, during the training phase of the object detection model, the pre-labeled category tags in the training data are encoded to obtain text features. A text-driven feature imitation learning module is added to the object detection model, and the input of the text-driven feature imitation learning module is the text features and coupled features. A quality score is calculated based on the text features and coupled features. Based on the quality score, the coupled features are divided into example features and non-example features. The parameters of the object detection model are updated and trained based on the loss between non-example features and example features belonging to the same category in feature imitation learning.

[0019] Secondly, a video target detection system based on alternating decoupling is proposed, including:

[0020] The video acquisition unit is used to acquire the video to be detected.

[0021] The frame segmentation unit is used to divide the video to be detected into frames and obtain multiple frame images.

[0022] The object detection unit is used to perform object detection on each frame of the image using a trained object detection model, and obtain the object detection result for each frame of the image. The process of the trained object detection model performing object detection on each frame of the image is as follows: extracting frame features of the frame image; encoding the frame features to obtain encoded features; extracting spatial-temporal features and temporal-space features from the encoded features; coupling the spatial-temporal features and temporal-space features to obtain the coupled features of the frame image; and recognizing the coupled features to obtain the object detection result of the frame image.

[0023] Thirdly, a computer device is proposed, the device comprising:

[0024] A processor, adapted to execute computer programs;

[0025] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the video target detection method based on alternating decoupling proposed in the first aspect.

[0026] Fourthly, a computer-readable storage medium is proposed, which stores a computer program adapted to be loaded by a processor and executed by the video target detection method based on alternating decoupling proposed in the first aspect.

[0027] Fifthly, a computer program product is proposed, which includes a computer program that, when executed by a processor, implements the video target detection method based on alternating decoupling proposed in the first aspect.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] This invention proposes a video target detection method and system based on alternating decoupling. After acquiring the video to be detected, the method first divides the video into frames, then performs target detection on each frame. During target detection in each frame, spatial-temporal features and temporal-spatial features are extracted. These features are then coupled to obtain the coupled features of the frame. The coupled features are then identified and located to obtain the target detection result for the frame. This invention, in acquiring the coupled features, alternately explores spatial and temporal information using a divide-and-conquer approach, achieving a full mining of the spatial and temporal information of the frame, thereby promoting more effective feature aggregation, more fully and comprehensively aggregating spatiotemporal information, and improving the accuracy of target detection.

[0030] In the training phase of the object detection model, the method of the present invention also sets up a text-driven feature imitation learning module, which generates more discriminative features through the supervision of high-quality features, thereby enhancing the performance of low-quality features, alleviating the feature collapse problem, improving the training accuracy of the object detection model, and further ensuring the accuracy of object detection.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0033] Figure 1 Here is a flowchart of the video target detection method based on alternating decoupling disclosed in the embodiments;

[0034] Figure 2 This is a schematic diagram of the internal structure of the Transformer encoder disclosed in the embodiment;

[0035] Figure 3 This is a schematic diagram of the internal structure of the alternating decoupled Transformer decoder module disclosed in the embodiment;

[0036] Figure 4This is a schematic diagram of the internal structure of the space-time decoupled Transformer decoder disclosed in the embodiment;

[0037] Figure 5 The flowchart of the time-space decoupling Transformer decoder method disclosed in the embodiment is shown below;

[0038] Figure 6 This is a schematic diagram of the internal structure of the multi-view structured feature coupling module disclosed in the embodiment;

[0039] Figure 7 This is a schematic diagram of the internal structure of the text-driven feature imitation learning module disclosed in the embodiment. Detailed Implementation

[0040] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0041] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0042] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0043] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0044] Example 1

[0045] Since its inception, the Transformer architecture has been a significant breakthrough in the field of deep learning. Through its self-attention mechanism, it can capture global dependencies in sequential data and exhibits significant advantages over traditional recurrent neural networks and convolutional neural networks when processing long sequences. However, training and optimizing Transformer models typically require substantial data and computational resources, placing high demands on hardware and limiting its application in resource-constrained scenarios.

[0046] Decoupling is a strategy widely used in computer vision and deep learning. Its core is to break down complex problems into multiple independent sub-problems, thereby reducing the complexity of the problem and improving the performance and interpretability of the model.

[0047] To leverage the advantages of Transformers to fully extract temporal and spatial information while effectively overcoming the limitations of existing technologies and improving the accuracy and robustness of video object detection, this invention proposes a video object detection method based on alternating decoupling. This method constructs a novel alternating decoupling approach from the perspective of feature aggregation, enabling more comprehensive and complete aggregation of spatiotemporal information. Furthermore, a feature imitation learning module is designed to alleviate the feature collapse problem, which has not yet been thoroughly investigated in the field of video object detection.

[0048] This embodiment provides a detailed description of the video object detection method based on alternating decoupling. First, it proposes an alternating decoupling Transformer decoder module. This module explores spatial and temporal information alternately using a divide-and-conquer approach, thereby promoting more effective feature aggregation. Second, it designs a multi-view structured feature coupling module, which adaptively couples spatiotemporal features by leveraging the advantages of graph learning. Third, it designs a text-driven feature imitation learning module, which generates more discriminative features through the supervision of high-quality features, thereby enhancing the performance of low-quality features. The feature imitation learning module is only installed during the training phase and does not reduce the speed during the testing phase.

[0049] Specifically, the video target detection method based on alternating decoupling disclosed in this embodiment, such as... Figures 1-7 As shown, it includes:

[0050] Obtain the video to be tested;

[0051] The video to be tested is divided into frames to obtain multiple frame images;

[0052] The trained object detection model is used to perform object detection on each frame of the image to obtain the object detection result for each frame. The process of the trained object detection model performing object detection on each frame of the image is as follows: extracting frame features of the frame image; encoding the frame features to obtain encoded features; extracting spatial-temporal features and temporal-space features from the encoded features; coupling the spatial-temporal features and temporal-space features to obtain the coupled features of the frame image; and recognizing the coupled features to obtain the object detection result of the frame image.

[0053] In this embodiment, after acquiring the video to be detected, all frame images in the video to be detected are input into the trained target detection model in batches to obtain the target detection results of all frame images.

[0054] Specifically, the object detection model extracts spatial-temporal features from the encoded features through a spatial-temporal decoupled Transformer decoder; it also extracts temporal-spatial features from the encoded features through a temporal-spatial decoupled Transformer decoder. The process of extracting spatial-temporal features through the spatial-temporal decoupled Transformer decoder includes: extracting spatial features from the encoded features through the spatial decoupled Transformer decoder, and then extracting temporal features from these spatial features through the temporal decoupled Transformer decoder, thus obtaining the spatial-temporal features.

[0055] The spatially decoupled Transformer decoder comprises a spatial mask self-attention mechanism, a spatial deformable self-attention mechanism, and a feedforward network module connected in sequence. The spatial mask self-attention mechanism processes the features of the input spatially decoupled Transformer decoder to obtain processed features. The spatial deformable self-attention mechanism interacts and fuses the processed features obtained by the spatial mask self-attention mechanism with the encoded features to obtain interactively fused features. The feedforward network module processes the interactively fused features obtained by the spatial deformable self-attention mechanism to finally output spatial features.

[0056] The temporally decoupled Transformer decoder comprises a temporally masked self-attention mechanism, a temporally deformable attention mechanism, and a feedforward network module connected in sequence. The temporally masked self-attention mechanism processes the features of the input temporally decoupled Transformer decoder to obtain processed features. The temporally deformable attention mechanism enables the processed features obtained by the temporally masked self-attention mechanism to interact and fuse with the encoded features to obtain interactively fused features. The feedforward network module processes the interactively fused features obtained by the temporally deformable attention mechanism to finally output temporal features.

[0057] The spatial-temporal features and temporal-spatial features are coupled using a multi-view structured feature coupling module to obtain coupled features. The process includes:

[0058] Construct multi-view undirected graphs from different perspectives based on space-time and time-space characteristics;

[0059] The edge-aware multi-graph fusion processor merges all undirected graphs to obtain the fused graph.

[0060] The vertex features of the fused graph are extracted by a hierarchical adaptive graph convolutional network, and the fused graph is updated using the vertex features to obtain the updated graph.

[0061] Based on the updated graph, the coupling characteristics are obtained.

[0062] The target detection model disclosed in this embodiment is described in detail. The target detection model disclosed in this embodiment includes multiple parallel backbone networks. The input end of each backbone network is the frame image to be detected, and the output end of each backbone network outputs the frame features of the corresponding frame image. The frame features and the corresponding position features of each frame image are added together and then input to the Transformer encoder.

[0063] The Transformer encoder outputs the encoded features of each frame; the encoded features are then input into the alternately decoupled Transformer decoder module.

[0064] The Transformer decoder module is alternately decoupled. The spatial-temporal decoupled Transformer decoder outputs spatial-temporal features, and the temporal-spatial decoupled Transformer decoder outputs temporal-spatial features. The spatial-temporal features and the temporal-spatial features are then input into the multi-view structured feature coupling module.

[0065] The multi-view structured feature coupling module outputs the coupling features of each frame of the image; the coupling features of each frame of the image are used as the input of the feedforward network, and the output of the feedforward network outputs the target detection result of each frame of the image.

[0066] This embodiment uses several backbone networks to extract features from several frames of images. One backbone network corresponds to one frame of image, and finally, the frame features of all frames are obtained. This embodiment uses ResNet-101 as the backbone network, DETR as the baseline network, and CLIP as the text encoder.

[0067] The process of encoding frame features using a Transformer encoder to obtain encoded features includes:

[0068] For a given backbone network, the frame features extracted from the nth frame are... Where c, h, and w represent the dimension, height, and width of the frame feature, respectively;

[0069] The frame features and corresponding location features of each frame are added together to obtain the summed features. and the features after addition The input is fed into the Transformer encoder. The position feature PE is calculated using sine and cosine coding, and the formula is as follows:

[0070]

[0071] Where pso represents position, i represents the index of the feature dimension, and d model This refers to the dimension of the model. Sine coding is used for even channels, and cosine coding is used for odd channels.

[0072] The Transformer encoder consists of K=6 stacks of the same level, such as Figure 2 As shown. Each layer has two sublayers: the first sublayer is a multi-head self-attention mechanism, and the second sublayer is a feedforward network. Residual connections are used around each of the two sublayers, followed by layer normalization. The output of each sublayer is LayerNorm(x+Sublayer(x)). Here, Sublayer(x) is the function implemented by the sublayer itself. x represents the vector (or tensor) input to the current sublayer, and LayerNorm(·) represents layer normalization, which normalizes the features of each sample.

[0073] like Figure 3 As shown, the alternating decoupled Transformer decoder module includes a spatial-temporal decoupled Transformer decoder and a temporal-spatial decoupled Transformer decoder; wherein, the spatial-temporal decoupled Transformer decoder comprises a spatial decoupled Transformer decoder and a temporal decoupled Transformer decoder connected in sequence; the temporal-spatial decoupled Transformer decoder comprises a temporal decoupled Transformer decoder and a spatial decoupled Transformer decoder connected in sequence; N frames of target queries are input to the input terminals of the spatial decoupled Transformer decoder in the spatial-temporal decoupled Transformer decoder and the temporal decoupled Transformer decoder in the temporal-spatial decoupled Transformer decoder, respectively; the temporal decoupled Transformer decoder in the spatial-temporal decoupled Transformer decoder outputs spatial-temporal features; the spatial decoupled Transformer decoder in the temporal-spatial decoupled Transformer decoder outputs temporal-spatial features.

[0074] like Figure 4As shown, the space-time decoupled Transformer decoder includes a spatial decoupled Transformer decoder and a temporal decoupled Transformer decoder. The spatial decoupled Transformer decoder comprises N parallel branches, each branch including a spatial mask self-attention mechanism, a spatial deformable self-attention mechanism, and a feedforward network module connected in sequence. The input of the spatial mask self-attention mechanism in the nth branch is used to input the target query of the nth frame; the output of the spatial mask self-attention mechanism in the nth branch is connected to the input of the spatial deformable self-attention mechanism. The spatially deformable self-attention mechanism receives the encoded features of the nth frame as input; its output is connected to the input of a feedforward network, which outputs the spatial features of the nth frame. The spatial features of N frames are then input to a temporally decoupled Transformer decoder, which includes a temporally masked self-attention mechanism, a temporally deformable attention mechanism, and a feedforward network module connected in sequence. The temporally masked self-attention mechanism receives the spatial features of N frames as input; its output is connected to the input of the temporally deformable attention mechanism, which also receives the encoded features of N frames; its output is connected to the input of the feedforward network; and the feedforward network outputs the spatial-temporal features of N frames.

[0075] The process of inputting encoded features into a spatial-temporal decoupled Transformer decoder and outputting spatial-temporal features includes:

[0076] First, the target query for each frame is input into a spatial mask self-attention layer to explore intra-frame spatial dependencies. The target query is a set of learnable vectors used to generate the final detection results. These query vectors do not depend on the specific content in the image, but interact with image features through a Transformer structure to obtain detection information for each target.

[0077] Most adjacent frames contain similar appearance information. Directly using the native self-attention mechanism not only introduces redundant features and reduces model performance, but also incurs additional costs in terms of computation and memory. Therefore, this embodiment designs a spatial mask self-attention mechanism. This mechanism processes the features of the input spatially decoupled Transformer decoder to obtain processed features. The spatial mask self-attention mechanism, SpatMSelfA, has the following formula:

[0078]

[0079] Where, q∈Ω q Represents the query element, k∈Ωk Ω represents the key element. q and Ω k These represent the collections of query elements and key elements, respectively. and Let represent the features of query element q and key element k, respectively, where D is the feature dimension. s represents the index of the attention head, S represents the total number of attention heads, and K represents the total number of key elements. and D represents the learnable mapping weights. v =D / S. The symbol "·" indicates a scalar multiplication operation. M sqk ∈{0,1} represents the mask weight of the k-th sampled key element in the s-th attention head. Its value is 1 when the key element is within the defined local window; otherwise, its value is 0. B sqk ∈[0,1] represents the attention weight of the k-th sampled key element in the s-th attention head, which is normalized over all key elements. B sqk The calculation formula is:

[0080]

[0081] Where ∝ represents proportionality to the operation, This indicates the transpose operation. and This represents the learnable mapping weights.

[0082] Next, the processed features output from the spatial mask self-attention layer are input together with the encoded features into the spatially deformable attention layer. The aim is to explore the spatial dependencies between them and fuse the encoded features with the target query. Specifically, the spatially deformable attention layer uses a spatially deformable self-attention mechanism to interactively fuse the processed features obtained from the spatial mask self-attention mechanism and the encoded features to obtain the interactively fused features. The spatially deformable self-attention mechanism SpatDeformA can be represented as:

[0083]

[0084] Where, p q Δp represents the two-dimensional reference point for the query element q. sqk It is the sampling offset of the k-th sampling key element in the s-th attention head. Since p q +Δp sqk Since it is in fractional form, bilinear interpolation is used to calculate x(p). q +Δp sqk Finally, the output of the spatially deformable attention layer is fed into a feedforward network to generate spatial features.

[0085] Since spatial information alone is insufficient to handle challenges such as ambiguity or interference from similar objects, it is necessary to introduce temporal information to enrich the representation of spatial features. To this end, spatial features from multiple frames are input into a temporal mask self-attention layer to explore cross-frame temporal dependencies. Specifically, the temporal mask self-attention layer processes the features of the input temporally decoupled Transformer decoder through a temporal mask self-attention mechanism to obtain processed features. The temporal mask self-attention mechanism, TempMselfA, can be defined as:

[0086]

[0087] Where n represents the index of the input frame, and N is the total number of input frames. M snqk ∈{0,1) represents the mask weight of the k-th sampled key element in the s-th attention header of the n-th frame. It takes the value 1 when the key element is within the defined local window; otherwise, it takes the value 0. B snqk ∈[0,1] represents the attention weight of the k-th sample key element in the s-th attention header of the n-th frame. B snqk The calculation formula is:

[0088]

[0089] in, The intermediate feature is represented by n, which represents the index of the input frame. The interpretation of the other symbols can be found in formula (4). The normalization method samples NK elements from the N-frame feature map, thereby enabling the modeling of multi-frame dependencies.

[0090] Subsequently, the output features of the temporal mask self-attention layer are input into the temporally deformable attention layer, while the encoded features are also input into the temporally deformable attention layer. The aim is to explore the temporal dependencies between them and fuse the encoded features with the target query to obtain interactively fused features. Specifically, the temporally deformable attention layer uses a temporally deformable attention mechanism to interactively fuse the processed features obtained from the temporal mask self-attention mechanism with the encoded features to obtain interactively fused features. The temporally deformable attention mechanism TempDeformA can be represented as:

[0091]

[0092] The meanings of the symbols in formula (8) can be found in formulas (6) and (7). Finally, the output of the time-deformable attention layer is input into the feedforward network to generate spatial-temporal features.

[0093] like Figure 5 As shown, the time-space decoupled Transformer decoder module includes:

[0094] The temporally decoupled Transformer decoder and the spatially decoupled Transformer decoder have the same internal structure as described above, such as... Figure 4 As shown; the temporally decoupled Transformer decoder and the spatially decoupled Transformer decoder are connected in sequence; the input of the temporally decoupled Transformer decoder is the target query of N frames; the output of the temporally decoupled Transformer decoder is connected to the input of the spatially decoupled Transformer decoder; the spatially decoupled Transformer decoder outputs the temporal-spatial features of N frames.

[0095] The process of inputting the encoded features into the time-space decoupled Transformer decoder to obtain time-space features includes:

[0096] The target query for each frame is input to the temporally decoupled Transformer decoder, and then the output of the temporally decoupled Transformer decoder is input to the spatially decoupled Transformer decoder. This achieves the opposite workflow to the spatially-temporally decoupled Transformer decoder in terms of mining spatial and temporal information. The goal of the temporally-spatially decoupled Transformer decoder is to generate new feature representations that are complementary to the output of the spatially-temporally decoupled Transformer decoder.

[0097] The multi-view structured feature coupling module comprises T undirected graphs with different perspectives, an edge-aware multi-graph fusion module, and a hierarchical adaptive graph convolutional network, connected sequentially. The input to the first undirected graph is the N frames of spatial-temporal and temporal-spatial features output by the alternating decoupled Transformer decoder module. The output of the Tth undirected graph is connected to the input of the edge-aware multi-graph fusion module. The first and Tth undirected graphs share the same set of nodes but provide diverse relational representations through different edge sets and adjacency matrices. The edge-aware multi-graph fusion module outputs the fused graph, which serves as the input to the hierarchical adaptive graph convolutional module. The hierarchical adaptive graph convolutional module outputs an updated graph containing enhanced spatial-temporal and temporal-spatial feature information in different dimensions. Finally, the enhanced spatial-temporal and temporal-spatial features are element-wise added to generate coupled features.

[0098] In this embodiment, time-space features and space-time features are input into the multi-view structured feature coupling module. The specific process of obtaining coupled features includes:

[0099] Construct T undirected multi-views with different perspectives based on time-space characteristics and space-time characteristics. in Represents the vertex set, ε t and A t Let represent the edge set and the adjacency matrix in the t-th view, respectively. When constructing multiple views, each input feature is treated as a vertex, and edges are determined based on the relationships between vertices.

[0100] Unlike methods that directly integrate features using graph learning operations on multiple views, this embodiment proposes a simple and efficient edge-aware multi-graph fusion engine that merges multiple views into a single graph, requiring graph learning operations to be performed only once. The computation of the edge-aware multi-graph fusion engine can be represented as:

[0101]

[0102] in, Represents vertex v in the merged graph i and v j The edges between them. It is a multilayer perceptron, and Concat(·,·,·) represents the connection operation. These represent vertices v in the information view, semantic view, and distance view, respectively. i and v j The edges between them. According to formula (9), using To connect the vertices and obtain the final merged graph, its adjacency matrix can be calculated by using the Softmax function on all edges.

[0103] Based on the fused graph, this embodiment improves the native graph convolutional network and designs a hierarchical adaptive graph convolutional network, recalculating the adjacency matrix according to the newly learned vertex features of each graph convolutional layer. The hierarchical adaptive graph convolutional network designed in this patent employs a two-layer adaptive graph convolutional network with a ReLU activation function, and the enhanced vertex feature matrix G can be calculated as follows:

[0104]

[0105] Where X represents the input feature matrix, namely the spatial-temporal features and temporal-spatial features of N frames. and Θ represents the normalized graph Laplacian regularization matrix for the first and second layers, respectively. (1) and Θ (2) Let represent the weight matrices of the first and second layers, respectively. The hierarchical adaptive graph convolutional network outputs an enhanced feature matrix G, which contains information about the enhanced spatial-temporal and temporal-space features in different dimensions. Finally, the enhanced spatial-temporal and temporal-space features are added element-wise to generate coupled features.

[0106] In this embodiment, after constructing the object detection model, the model is trained using a training set, where the images in the training set are video frame images labeled with object detection tags. The training set is then input into the object detection model to train it. Training is stopped when the loss function value of the model no longer decreases or the number of iterations reaches a set number, resulting in a well-trained object detection model.

[0107] The training set refers to a given video dataset containing several videos, each containing several video frames. The video target detection method in this embodiment takes N consecutive frames (30 frames by default) as input and outputs the detection results of all input frames at once.

[0108] In this embodiment, during the training phase of the object detection model, the pre-labeled category tags in the training data are encoded to obtain text features. A text-driven feature imitation learning module is added to the object detection model, and the input of the text-driven feature imitation learning module is the text features and coupled features. A quality score is calculated based on the text features and coupled features. Based on the quality score, the coupled features are divided into example features and non-example features. The parameters of the object detection model are updated and trained based on the loss between non-example features and example features belonging to the same category in feature imitation learning, so as to improve the performance of the model and the discriminative ability of the features.

[0109] The text-driven feature imitation learning module includes a feature quality indicator and an embedding space. The input to the feature quality indicator is coupled features and text features. The text features are formed by a text encoder based on the category label data of video frames. The output of the feature quality indicator is sample features and non-sample features, which are mapped to the embedding space and then updated through backpropagation.

[0110] The data processing procedure of the text-driven feature imitation learning module disclosed in this embodiment includes:

[0111] First, textual features and coupled features are input into a feature quality indicator to generate example features and non-example features. To better construct example features, the feature quality indicator employs a simple yet effective metric learning method to evaluate feature quality scores. These quality scores characterize the quality of coupled features, and coupled features with quality scores above a threshold are selected as example features, forming the example feature set Ω. e , can be represented as:

[0112]

[0113]

[0114] Among them, ci y represents the i-th coupled feature output by the multi-view structured feature coupling module. j Let represent the j-th text feature generated by the text encoder processing the category labels of the video frame. FQ(·,·) represents the feature quality score, which serves as an indicator to capture high-quality coupled features with rich information. θ represents a scalar threshold. γ and ∏ represent the learning parameters of the two linear mapping layers. The transpose operation is represented by ρ(·), the nonlinear activation function is represented by ρ(·,·), and the concat(·,·) operation is represented by concatenation. According to formula (12), coupled features with quality scores below the threshold θ are considered as non-sample features, forming a non-sample feature set.

[0115] After obtaining example features and non-example features, the text-driven feature imitation learning module employs a cost-negligible single-layer perceptron to map them into the embedding space, reducing the feature dimension to 128 and alleviating memory burden. Subsequently, this embodiment designs a feature imitation learning loss function to support the mapping of non-example features to the vicinity of their associated example features, while simultaneously separating them from example features of other classes. This facilitates the generation of multiple discriminative feature representations, where associated example features refer to example features belonging to the same class as the non-example features. The feature imitation learning loss can be formulated as:

[0116]

[0117] in, Ω is the loss between the i-th non-sample feature and the j-th sample feature. e and These represent the sample feature set and the non-sample feature set, respectively. Represents the cosine similarity function, o i and r j Let i and j represent the i-th non-example feature and the j-th example feature, respectively, belonging to the same category. τ represents the temperature parameter. Using formula (13), the proposed alternating decoupled Transformer mimicry network can improve feature discriminability by using high-quality features to guide the learning of low-quality features. The text-driven feature mimicry learning module is only installed during the training phase and does not reduce the speed of testing.

[0118] The video target detection method based on alternating decoupling disclosed in this embodiment improves video target detection performance by proposing a novel alternating decoupling method from the perspective of feature aggregation through three key technological improvements. This method more fully and comprehensively aggregates spatiotemporal information. Its overall architecture diagram is shown below. Figure 1As shown, given an input video sequence, each video frame first undergoes feature extraction by a shared-weight backbone network to obtain corresponding frame features. These frame features are then added to their corresponding location features and input into a Transformer encoder, which outputs the encoded features for each frame. Next, the encoded features are input into an alternating decoupled Transformer decoder module, which outputs spatial-temporal and temporal-spatial features. Subsequently, the spatial-temporal and temporal-spatial features are input into a multi-view structured feature coupling module to obtain the coupled features for each frame. Finally, the coupled features for each frame are input into a feedforward network for recognition and localization, yielding the target detection result for each frame. Given the advantages of the modules designed in this embodiment, the framework employs a parallel detection method, enabling simultaneous detection of targets on all input frames, thus giving the video target detection method real-time inference capabilities.

[0119] The video object detection method based on alternating decoupling disclosed in this embodiment also designs a feature imitation learning paradigm in the training stage of the object detection model to alleviate the feature collapse problem, which has not been studied in depth in the field of video object detection.

[0120] The video target detection method based on alternating decoupling disclosed in this embodiment has broad application prospects in multiple fields, especially in scenarios with high requirements for high precision and real-time processing, such as intelligent surveillance and virtual reality. For example, in intelligent surveillance, it can improve target recognition accuracy and enhance public safety; in virtual reality, it can improve scene recognition and target tracking accuracy, enhancing user experience. Furthermore, this technology can also be applied to fields such as robotics and drones, improving their autonomy and execution efficiency. Through the application of this technology, these industries will see significant improvements in accuracy, efficiency, and security.

[0121] It should be noted that all data acquisition is conducted in accordance with laws and regulations and with user consent, and the data is used legally.

[0122] Example 2

[0123] In this embodiment, a video target detection system based on alternating decoupling is disclosed, comprising:

[0124] The video acquisition unit is used to acquire the video to be detected.

[0125] The frame segmentation unit is used to divide the video to be detected into frames and obtain multiple frame images.

[0126] The object detection unit is used to perform object detection on each frame of the image using a trained object detection model, and obtain the object detection result for each frame of the image. The process of the trained object detection model performing object detection on each frame of the image is as follows: extracting frame features of the frame image; encoding the frame features to obtain encoded features; extracting spatial-temporal features and temporal-space features from the encoded features; coupling the spatial-temporal features and temporal-space features to obtain the coupled features of the frame image; and recognizing the coupled features to obtain the object detection result of the frame image.

[0127] The present invention also discloses a computer device, the device comprising:

[0128] A processor, adapted to execute computer programs;

[0129] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the video target detection method based on alternating decoupling disclosed in Embodiment 1.

[0130] The present invention also discloses a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the video target detection method based on alternating decoupling disclosed in Embodiment 1.

[0131] The present invention also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the video target detection method based on alternating decoupling disclosed in Embodiment 1.

[0132] The method disclosed in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0133] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for video object detection based on alternating decoupling, characterized in that, The method comprises the following steps: acquiring a video to be detected; frame division is performed on the video to be detected to obtain a plurality of frame images; target detection is performed on each frame image by using a trained target detection model to obtain a target detection result of each frame image; wherein the process of target detection performed on each frame image by using the trained target detection model comprises the following steps: frame features of the frame image are extracted; the frame features are encoded to obtain encoded features; spatial-temporal features and temporal-spatial features are extracted from the encoded features; the spatial-temporal features and the temporal-spatial features are coupled to obtain coupled features of the frame image; and the coupled features are identified to obtain the target detection result of the frame image; the target detection model extracts the spatial-temporal features from the encoded features by using a spatial-temporal decoupling Transformer decoder; and extracts the temporal-spatial features from the encoded features by using a temporal-spatial decoupling Transformer decoder; the process of extracting the spatial-temporal features by using the spatial-temporal decoupling Transformer decoder comprises the following steps: spatial features are extracted from the encoded features by using a spatial decoupling Transformer decoder, and temporal features are extracted from the spatial features by using a temporal decoupling Transformer decoder, and the temporal features are taken as the spatial-temporal features; the process of extracting the temporal-spatial features by using the temporal-spatial decoupling Transformer decoder comprises the following steps: temporal features are extracted from the encoded features by using a temporal decoupling Transformer decoder, and spatial features are extracted from the temporal features by using a spatial decoupling Transformer decoder, and the spatial features are taken as the temporal-spatial features.

2. The method of claim 1, wherein, The spatial decoupling Transformer decoder comprises a spatial mask self-attention mechanism, a spatial deformable self-attention mechanism and a feedforward network module which are connected in sequence; the spatial mask self-attention mechanism is used to process features input into the spatial decoupling Transformer decoder to obtain processed features; the spatial deformable self-attention mechanism is used to interactively fuse the processed features obtained by the spatial mask self-attention mechanism and the encoded features to obtain interactively fused features; and the feedforward network module is used to process the interactively fused features obtained by the spatial deformable self-attention mechanism to finally output the spatial features. 3.The method of claim 1, wherein, The temporal decoupling Transformer decoder comprises a temporal mask self-attention mechanism, a temporal deformable attention mechanism and a feedforward network module which are connected in sequence; the temporal mask self-attention mechanism is used to process features input into the temporal decoupling Transformer decoder to obtain processed features; the temporal deformable attention mechanism is used to interactively fuse the processed features obtained by the temporal mask self-attention mechanism and the encoded features to obtain interactively fused features; and the feedforward network module is used to process the interactively fused features obtained by the temporal deformable attention mechanism to finally output the temporal features.

4. The method of claim 1, wherein, The spatial-temporal features and the temporal-spatial features are coupled by using a multi-view structured feature coupling module to obtain the coupled features, and the process comprises the following steps: Different view multi-view undirected graphs are constructed according to the space-time features and the time-space features; All the undirected graphs are merged by an edge-aware multi-graph fusioner to obtain a fused graph; Vertex features of the fused graph are extracted by a hierarchical adaptive graph convolution network, and the fused graph is updated by using the vertex features to obtain an updated graph; According to the updated graph, coupled features are obtained.

5. The method of claim 1, wherein, In the target detection model training stage, the class labels pre-labeled in the training data are encoded to obtain text features; A text-driven feature imitation learning module is added to the target detection model, and the input of the text-driven feature imitation learning module is the text features and the coupled features; the quality score is calculated according to the text features and the coupled features; and the coupled features are divided into sample features and non-sample features according to the quality score; According to the loss between the non-sample features and the sample features of the same class in the feature imitation learning, the parameters of the target detection model are updated and trained.

6. A video object detection system based on alternating decoupling, characterized in that It comprises: a video acquisition unit configured to acquire a video to be detected; a frame division unit configured to divide the video to be detected into a plurality of frame images; a target detection unit configured to detect targets in each frame image by using a trained target detection model to obtain a target detection result of each frame image; wherein the process of detecting targets in each frame image by using the trained target detection model comprises: extracting frame features of the frame image; encoding the frame features to obtain encoded features; extracting space-time features and time-space features from the encoded features; coupling the space-time features and the time-space features to obtain coupled features of the frame image; and identifying the coupled features to obtain the target detection result of the frame image; the target detection model extracts the space-time features from the encoded features by using a space-time decoupling Transformer decoder; and extracts the time-space features from the encoded features by using a time-space decoupling Transformer decoder; the process of extracting the space-time features by using the space-time decoupling Transformer decoder comprises: extracting space features from the encoded features by using a space decoupling Transformer decoder, and extracting time features from the space features by using a time decoupling Transformer decoder, as the space-time features; the process of extracting the time-space features by using the time-space decoupling Transformer decoder comprises: extracting time features from the encoded features by using a time decoupling Transformer decoder, and extracting space features from the time features by using a space decoupling Transformer decoder, as the time-space features.

7. An electronic device, comprising: The device comprises: a processor adapted to execute a computer program; a computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by the processor, implements the video target detection method based on alternative decoupling according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by the processor and execute the video target detection method based on the alternating decoupling according to any one of claims 1-5.

9. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the video target detection method based on the alternating decoupling according to any one of claims 1-5.

Images