Self feedback method and apparatus for temporal action detection

The self-feedback method and device for DETR-based models address the temporal collapse issue by enhancing attention relationships using guidance maps and objective functions, improving DETR-based model performance in temporal behavior detection.

KR103046296B1Active Publication Date: 2026-11-10RES & BUSINESS FOUND SUNGKYUNKWAN UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
KR1020230164287
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-11-23
Publication Date
2026-11-10
Estimated Expiration
2043-11-23

AI Technical Summary

Technical Problem

DETR-based models suffer from the temporal collapse phenomenon, where the attention mechanism focuses on a few important elements, ignoring temporal features and degrading performance by concentrating on specific sections, leading to information loss and hindered learning.

Method used

A self-feedback method and device using an artificial neural network with an encoder and decoder, incorporating self-attention and cross-attention layers, apply guidance maps and objective functions to enhance attention relationships, reducing spatial dimensions and maintaining temporal features through 3D CNN pooling.

Benefits of technology

Overcomes the temporal collapse by improving model performance in temporal behavior detection, enhancing attention mechanisms to include all relevant features, thereby reducing information loss and improving detection accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112023130960918-PAT00072_ABST
    Figure 112023130960918-PAT00072_ABST
Patent Text Reader

Abstract

The present invention relates to a self-feedback technology for temporal behavior detection. A self-feedback device based on an artificial neural network comprising an encoder, a decoder, and one or more queries for the decoder receives a video image as input, extracts one or more video features based on the video image, extracts one or more first self-attention maps and one or more encoder features through an encoder based on one or more video features, extracts one or more cross-attention maps and one or more second self-attention maps through a decoder based on one or more encoder features and one or more queries, extracts one or more first guidance maps for the encoder and one or more second guidance maps for the decoder based on one or more cross-attention maps and the transpose of one or more cross-attention maps, and a first objective function including one or more first guidance maps and one or more first self-attention maps A second objective function including one or more second guidance maps and one or more second self-attention maps can be applied to an encoder and applied to a decoder.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a self-feedback method and apparatus for temporal action detection, and more specifically, to a technology for overcoming the temporal collapse phenomenon of DETR-based models and improving the detection performance of artificial intelligence models. Background Technology

[0002] Temporal Action Detection (TAD) refers to not only classifying actions in input images but also predicting the temporal boundaries of the images. Conventional techniques developed point-by-point learning to predict the probabilities of start and end boundaries in input images in an attempt to improve the object detection rate problem in object detection. While grouping each pair of start and end boundaries allowed for higher object detection rates through more flexible action detection, the generation of boundary pairs of various lengths resulted in the loss of objects depending on the rank and order of detection. To address this, reliance on ranking and ordering techniques in post-processing steps, such as Non-Maximum Suppression (NMS), became necessary to handle low-precision action detection.

[0003] Similar to the detection transformer (DETR) that has had a significant impact on object detection, DETR-based models for video images are also being introduced recently. In temporal behavior detection, DETR models and DETR-based models learn how to classify and localize feature maps utilizing the temporal characteristics of video images from query vectors.

[0004] Since there is no pre-established mapping between query vectors and ground-truth data, bipartite matching can connect them with minimum cost to assign labels. This approach can perform the task end-to-end through an objective function without heuristic algorithms such as non-maximum suppression.

[0005] However, conventional DETR-based models suffer from various issues with video images, resulting in poor performance in the domain of temporal behavior detection; it has been hypothesized that these problems lead to the failure of the dense attention mechanism. Dense attention refers to a standard attention mechanism that relates all elements without inductive biases, such as those found in convolution. To address this problem, conventional DETR-based models in the field of temporal behavior detection modified standard attention into boundary-sensitive modules, deformable attention, and query relational attention. However, attempts to improve DETR-based models by focusing on standard attention still tended to cause the models to regress significantly in the domain of temporal behavior detection. The problem to be solved

[0006] The technical problem that the embodiments of this specification aim to solve is to overcome the phenomenon of temporal collapse in the field of temporal behavior detection, where the attention mechanism of a model focuses only on a few important elements during the process of input image sequences to DETR models and DETR-based models, thereby ignoring the temporal features or elements of the input image and concentrating only on specific sections, resulting in information loss. Furthermore, it aims to overcome the problem of hindering attention relationship learning and degrading the performance of models caused by the temporal collapse phenomenon of DETR models and DETR-based models. means of solving the problem

[0007] To solve the above technical problem, a self-feedback method for temporal behavior detection according to an embodiment of the present invention comprises: a step of a self-feedback device based on an artificial neural network comprising an encoder, a decoder, and one or more queries for said decoder receiving a video image and extracting one or more video features based on said video image; a step of the self-feedback device extracting one or more first self-attention maps and one or more encoder features through said encoder based on said one or more video features, and extracting one or more cross-attention maps and one or more second self-attention maps through said decoder based on said one or more encoder features and said one or more queries. The self-feedback device may include the step of extracting one or more first guidance maps for the encoder and one or more second guidance maps for the decoder based on one or more cross-attention maps and the transpose of one or more cross-attention maps; the step of the self-feedback device applying a first objective function to the encoder that includes one or more first guidance maps and one or more first self-attention maps; and the step of the self-feedback device applying a second objective function to the decoder that includes one or more second guidance maps and one or more second self-attention maps. Additionally, the artificial neural network may be characterized by including the encoder that includes a self-attention sublayer and the decoder that includes a cross-attention layer and a self-attention sublayer.Additionally, the step of extracting one or more image features may be characterized by performing global average pooling through a 3D CNN (3-dimensional convolutional neural network) within the artificial neural network based on the image, thereby reducing information regarding spatial dimensions and extracting one or more image features that are information regarding temporal dimensions. Additionally, the step of extracting one or more first guidance maps for the encoder and one or more second guidance maps for the decoder may be characterized by extracting the first guidance map that considers all one or more image features input to the encoder and the second guidance map that considers all one or more encoder features input to the decoder by performing an element-wise square root operation based on one or more cross-attention maps and the transpose of one or more cross-attention maps. Additionally, the step of applying the first objective function to the encoder may be characterized by obtaining an aggregated self-attention map by performing consecutive matrix multiplications on one or more first self-attention maps according to the number of layers of the encoder, and applying the first objective function, which is a Kullback-Leibler divergence calculated based on the aggregated self-attention map and the first guidance map, to the encoder.Additionally, the step of applying the second objective function to the decoder may be characterized by applying the second objective function, which is a Kullback-Leibler divergence calculated according to the number of layers of the decoder based on the one or more second self-attention maps and the one or more second guidance maps, to the decoder.

[0008] Furthermore, the following provides a computer-readable recording medium that records a program for executing the self-feedback method for temporal behavior detection described above on a computer.

[0009] To solve the above technical problem, a self-feedback device for temporal behavior detection according to an embodiment of the present invention is a self-feedback device based on an artificial neural network comprising an encoder, a decoder, and one or more queries for the decoder, wherein the device comprises: a processor; and a memory in which one or more instructions executed by the processor are stored, wherein the one or more instructions include: receiving a video image as input and extracting one or more video features based on the video image; extracting one or more first self-attention maps and one or more encoder features through the encoder based on the one or more video features, and extracting one or more cross-attention maps and one or more second self-attention maps through the decoder based on the one or more encoder features and the one or more queries. The method may include the step of extracting one or more first guidance maps for the encoder and one or more second guidance maps for the decoder based on one or more cross-attention maps and the transpose of one or more cross-attention maps; the step of applying a first objective function comprising one or more first guidance maps and one or more first self-attention maps to the encoder; and the step of applying a second objective function comprising one or more second guidance maps and one or more second self-attention maps to the decoder. Additionally, the artificial neural network may be characterized by including the encoder including a self-attention sublayer and the decoder including a cross-attention layer and a self-attention sublayer.Additionally, the step of extracting one or more image features may be characterized by performing global average pooling through a 3D CNN (3-dimensional convolutional neural network) within the artificial neural network based on the image, thereby reducing information regarding spatial dimensions and extracting one or more image features that are information regarding temporal dimensions. Additionally, the step of extracting one or more first guidance maps for the encoder and one or more second guidance maps for the decoder may be characterized by extracting the first guidance map that considers all one or more image features input to the encoder and the second guidance map that considers all one or more encoder features input to the decoder by performing an element-wise square root operation based on one or more cross-attention maps and the transpose of one or more cross-attention maps. Additionally, the step of applying the first objective function to the encoder may be characterized by obtaining an aggregated self-attention map by performing consecutive matrix multiplications on one or more first self-attention maps according to the number of layers of the encoder, and applying the first objective function, which is a Kullback-Leibler divergence calculated based on the aggregated self-attention map and the first guidance map, to the encoder.Additionally, the step of applying the second objective function to the decoder may be characterized by applying the second objective function, which is a Kullback-Leibler divergence calculated according to the number of layers of the decoder based on the one or more second self-attention maps and the one or more second guidance maps, to the decoder.

[0010] To solve the above technical problem, a self-feedback system for temporal behavior detection according to one embodiment of the present invention comprises: an image acquisition device for acquiring an image; and a self-feedback device based on the artificial neural network, which receives the image from the image acquisition device, applies a first objective function to the encoder of the artificial neural network, and applies a second objective function to the decoder of the artificial neural network; wherein the self-feedback device receives the image, extracts one or more video features based on the image, extracts one or more first self-attention maps and one or more encoder features through the encoder based on the one or more video features, extracts one or more cross-attention maps and one or more second self-attention maps through the decoder based on the one or more encoder features and one or more queries, extracts one or more first guidance maps for the encoder and one or more second guidance maps for the decoder based on the one or more cross-attention maps and the transpose of the one or more cross-attention maps, and the one or more first guidance maps and The method may be characterized by applying the first objective function, which includes one or more first self-attention maps, to the encoder, and applying the second objective function, which includes one or more second guidance maps and one or more second self-attention maps, to the decoder. Additionally, the artificial neural network may be characterized by including the encoder, which includes a self-attention sublayer, and the decoder, which includes a cross-attention layer and a self-attention sublayer.In addition, when extracting one or more image features, the method may be characterized by performing global average pooling through a 3D CNN (3-dimensional convolutional neural network) within the artificial neural network based on the image, thereby reducing information regarding spatial dimensions and extracting one or more image features that are information regarding temporal dimensions. In addition, when extracting one or more first guidance maps for the encoder and one or more second guidance maps for the decoder, the method may be characterized by extracting the first guidance map that considers all one or more image features input to the encoder and the second guidance map that considers all one or more encoder features input to the decoder by performing an element-wise square root operation based on one or more cross-attention maps and the transpose of one or more cross-attention maps. Additionally, when applying the first objective function to the encoder, the method may be characterized by obtaining an aggregated self-attention map by performing consecutive matrix multiplications on one or more first self-attention maps according to the number of layers of the encoder, and applying the first objective function, which is a Kullback-Leibler divergence calculated based on the aggregated self-attention map and the first guidance map, to the encoder. Additionally, when applying the second objective function to the decoder, the method may be characterized by applying the second objective function, which is a Kullback-Leibler divergence calculated according to the number of layers of the decoder based on one or more second self-attention maps and one or more second guidance maps, to the decoder. Effects of the invention

[0011] According to the embodiments of the present specification described above, a self-feedback method and model are proposed to overcome the temporal collapse phenomenon occurring in DETR models and DETR-based models by generating a guidance map utilizing a cross-attention map between the encoder and decoder of the model. Through this, the problem of information loss caused by the model's attention mechanism focusing only on a few important elements during the process of input image sequences to DETR models and DETR-based models, thereby ignoring temporal features or elements of the input image and focusing only on specific sections, can be overcome. Furthermore, the problem of hindering the learning of DETR models and DETR-based models and degrading model performance due to the temporal collapse phenomenon can be overcome. Brief explanation of the drawing

[0012] Figure 1 is a diagram showing the temporal decay phenomenon occurring in the field of temporal behavior detection. FIG. 2 is a block diagram illustrating a self-feedback system for temporal behavior detection according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a self-feedback device based on an artificial neural network for temporal behavior detection according to an embodiment of the present invention. FIG. 4 is a diagram illustrating the application of a first objective function to an encoder according to one embodiment of the present invention. FIG. 5 is a diagram illustrating the application of a second objective function to a decoder according to one embodiment of the present invention. FIG. 6 is a flowchart illustrating a self-feedback method for temporal behavior detection according to an embodiment of the present invention. FIG. 7 is a block diagram illustrating a self-feedback device for temporal behavior detection according to another embodiment of the present invention. Specific details for implementing the invention

[0013] Before specifically describing the embodiments of this specification, we will introduce the objectives appearing in the technical field of applying self-feedback for temporal action detection in which the embodiments of this specification are implemented, and the technical means and configurations that can be considered to solve these objectives.

[0014] End-to-end object detection with Transformers was the first to treat object detection as a direct set prediction problem, eliminating the need for non-maximum suppression (NMS), a heuristic process. However, detection Transformers (DETR) models required more training time than conventional optimization methods due to the difficulty of optimizing Hungarian matching with dense attention. To address this issue, Deformable DETR introduced sparse attention to localize elements, thereby directing attention to only certain elements. Deformable attention restored locality to DETR, significantly reducing training time while improving performance. Subsequent DETR-based models further improved query representation by explicitly encoding boundary box information, which proved effective in stabilizing training.

[0015] Transformer-based models utilizing dense attention inherently require significant computational cost during the model's computation process. To further reduce the computational cost of dense attention models, Sparse DETR introduced learnable sparsity into encoder features. To achieve this, sparsity was induced by generating binary masks using encoder-decoder cross-attention maps.

[0016] DETR was introduced relatively recently in the field of temporal action detection (TAD) because DETR-based models achieved new top performance in object detection. RTD-Net (Relaxed Transformer Decoder-network) pointed out a problem where the self-attention layer exhibited an over-smoothing effect due to the uniform distribution resulting from the use of dense attention in the DETR encoder. To mitigate this smoothing effect, RTD-Net replaced the Transformer encoder with a boundary detection module. On the other hand, TadTR devised a temporally deformable attention inspired by Deformable DETR to address the same problem as RTD-Net. In particular, regarding query relationships in the decoder, ReAct developed a new relationship matching method to establish a strong correlation between low-nesting queries and high feature similarity. Because this method only allows predefined relationships, it alleviated the query reduction problem in the decoder's self-attention.

[0017] However, a problem still remains in that standard attention was bypassed by modifiable attention or heuristic query relationships.

[0018] Figure 1 illustrates the temporal collapse phenomenon occurring in the field of temporal behavior detection. Referring to Figure 1, self-attention maps based on a DETR model method in Object Detection and Temporal Behavior Detection (TAD) are shown. The maps of the encoder and decoder (Figures 1 (a) to (f)) were extracted from the last layer of the encoder and decoder, respectively. Referring to Figures 1 (b) and (e), the self-attention maps of the encoder and decoder in temporal behavior detection are concentrated on a small number of keys, but the present invention shows a high correlation with adjacent features (Figures 1 (a), (c)) or the query itself (Figures 1 (d), (f)). That is, Figures 1 (b) and (e) illustrate the occurrence of temporal collapse, a phenomenon in which the model's attention mechanism focuses only on a few important elements in the image input to the model, thereby ignoring other temporal characteristics or elements.

[0019] Based on this recognition of the problem, the present invention aims to mitigate the temporal decay problem by generating a correct self-attention relationship by applying an objective function obtained through the cross-attention map of the decoder within the DETR, as illustrated in FIG. 1 (a) and (c) and (d) and (f).

[0020] Embodiments of the present specification will be described in detail below with reference to the drawings. However, detailed descriptions of known functions or configurations that may obscure the essence of the embodiments in the following description and the attached drawings are omitted. Additionally, throughout the specification, the term 'comprising' a component means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0021] Additionally, terms such as "first," "second," etc., may be used to describe various components, but the components should not be limited by these terms. The terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the rights of this specification, the first component may be named the second component, and similarly, the second component may be named the first component.

[0022] The terms used herein are merely for describing specific embodiments and are not intended to limit the specification. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “comprising” or “comprising” are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0023] Unless specifically defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this specification pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0024] FIG. 2 is a block diagram illustrating a self-feedback system for temporal behavior detection according to an embodiment of the present invention.

[0025] Referring to FIG. 2, a self-feedback system (10) according to one embodiment of the present invention may include an image acquisition device (100) and a self-feedback device (200).

[0026] The image acquisition device (100) can acquire an image transmitted from a shooting device such as a camera. The image acquisition device (100) can transmit the image to a self-feedback device (200), and each device can be connected to a network to communicate with each other.

[0027] The term "network" as used in the present invention may refer to a core network integrated with a wired public network, a wireless mobile communication network, or the mobile internet, and may refer to a global open computer network structure that provides TCP / IP protocols (Transmission Control Protocol), UDP (User Datagram Protocol), and various services existing at the upper layer, such as HTTP (Hyper Text Transfer Protocol), HTTPS (Hyper Text Transfer Protocol Secure), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), MQTT Protocol (Message Queueing Telemetry Transport), IPFS (Inter Planetary File System), etc., and may comprehensively refer to a data communication network capable of transmitting and receiving data in various forms, without being limited to these examples.

[0028] The self-feedback device (200) can receive an image from the image acquisition device (100). A self-feedback device (200) receives a video image as input, extracts one or more video features based on the video image, extracts one or more first self-attention maps and one or more encoder features through an encoder based on one or more video features, extracts one or more cross-attention maps and one or more second self-attention maps through a decoder based on one or more encoder features and one or more queries, extracts one or more first guidance maps for an encoder and one or more second guidance maps for a decoder based on one or more cross-attention maps and the transpose of one or more cross-attention maps, applies a first objective function including one or more first guidance maps and one or more first self-attention maps to an encoder, and applies a second objective function including one or more second guidance maps and one or more second self-attention maps to a decoder. The description of the operation of the self-feedback device (200) of Fig. 1 will be supplemented through the drawings to be described later.

[0029] FIG. 3 is a diagram illustrating a self-feedback device based on an artificial neural network for temporal behavior detection according to an embodiment of the present invention. FIG. 4 is a diagram illustrating the application of a first objective function to an encoder according to an embodiment of the present invention. FIG. 5 is a diagram illustrating the application of a second objective function to a decoder according to an embodiment of the present invention.

[0030] Referring to FIG. 3, the self-feedback device (200) can receive a video image (1). Additionally, the self-feedback device (200) may be configured based on a detection transformer (DETR), which is an artificial neural network comprising an encoder, a decoder, and one or more queries for the decoder.

[0031] Generally, a DETR-based model may include two main components: an encoder and a decoder. Input features to the encoder can be used to learn global relationships. The DETR may use image features extracted from a CNN within the DETR as inputs to the encoder. The encoder within the DETR may contain at least one layer that progressively refines the input features. The encoder may consist of a self-attention module containing at least one self-attention sublayer, and a multi-layer perceptron (MLP) that performs layer normalization and skip connections. On the other hand, the decoder may be configured to learn the relationship between encoder features and learnable embedding vectors. In this case, embedding vectors may be object queries capable of learning positional encoding for object detection. That is, the decoder can be configured to learn position encodings to learn the relationships between its own inputs, also known as object queries for object detection. Additionally, the decoder may include multiple layers by including a self-attention module containing at least one self-attention sublayer, similar to the encoder; however, among the multiple layers, it may additionally include a cross-attention module containing at least one cross-attention layer that focuses on the relationship between encoder features and object queries.That is, the artificial neural network that forms the basis of the self-feedback device (200) may include the encoder including a self-attention sublayer and the decoder including a cross-attention layer and a self-attention sublayer.

[0032] The self-feedback device (200) can perform global average pooling through a pre-trained 3D CNN (3-dimensional convolutional neural network) within an artificial neural network based on a video image (1). At this time, the global average pooling of the self-feedback device (200) is performed only in spatial dimensions, thereby reducing information about spatial dimensions and maintaining information about temporal dimensions, so that one or more video features, which are information about temporal dimensions, can be extracted.

[0033] The self-feedback device (200) can extract one or more first self-attention maps and one or more encoder features through one or more self-attention layers in the encoder based on one or more video features, and extract one or more cross-attention maps and one or more second self-attention maps through one or more cross-attention layers and self-attention layers in the decoder based on one or more encoder features and one or more queries.

[0034] The encoder and decoder of the self-feedback device (200) based on an artificial neural network may include a self-attention module composed of at least one self-attention layer. In this case, each encoder and decoder may require at least three variables for at least one feature received from a 3D CNN. These three variables are each a query , key is and value Thus, each encoder and decoder can project each of the three variables into three latent spaces based on three linear layers. Each encoder and decoder is and Attention scores can be calculated by performing matrix multiplication between the transposes and using the softmax activation function, and the attention scores are and It can mean the similarity between them. Each encoder and decoder uses attention scores. By pooling and linearly projecting, one or more self-attention maps, which are the outputs of the self-attention module, can be extracted and obtained, and in the case of an encoder, encoder features can be obtained. At this time, the self-attention map If so, it is equal to Mathematical Formula 1 below.

[0035]

[0036] Referring to Equation 1, in the self-attention module , and is extracted from the same input features, and means decoder query, and represents encoder features, and Is It means the preposition of, can represent the number of channels. When Equation 1 is used in the cross-attention module described later to obtain the cross-attention map, is extracted from the decoder query embedding, and and It may be extracted from encoder features.

[0037] The self-feedback device (200) can extract one or more first guidance maps for an encoder and one or more second guidance maps for a decoder based on one or more cross-attention maps and the transpose of one or more cross-attention maps.

[0038] Generally, the primary goal of cross-attention may be to represent encoder features using similarity scores between the decoder query and encoder features. Therefore, each feature map is the decoder query and encoder features It can be considered as a cross relation between. That is, the self-feedback device (200) can extract a first guidance map that considers all of the one or more image features input to the encoder and a second guidance map that considers all of the one or more encoder features input to the decoder by performing an element-wise square root operation based on one or more cross-attention maps and the transpose of one or more cross-attention maps. At this time, the cross-attention map Let's call it, and the second guidance map If we say, It is equal to mathematical formula 2 below.

[0039]

[0040] Referring to mathematical formula 2, means element-wise square root operation, and Is It can imply the preposition of. Also, It may include information about the number of decoder queries and encoder features, and It may only contain information about the number of decoder queries.

[0041] In addition, identical to Equation 2, the first guidance map If we say, It is equal to mathematical formula 3 below.

[0042]

[0043] Referring to mathematical formula 3, It may only contain information about the number of encoder features.

[0044] Referring to FIGS. 4 and 5, the self-feedback device (200) may apply a first objective function including one or more first guidance maps and one or more first self-attention maps to an encoder, and apply a second objective function including one or more second guidance maps and one or more second self-attention maps to a decoder.

[0045] Referring to FIG. 4, the self-feedback device (200) may aggregate one or more first self-attention maps to apply a first objective function to an encoder within the self-feedback device (200). The self-feedback device (200) may perform average pooling or matrix multiplication to aggregate one or more first self-attention maps. However, since the first objective function must also be applied to the encoder even under the assumption that there are no multilayer perceptrons (MLPs) and skip connections in the encoder within the self-feedback device (200), it may be preferable for the feedback device (200) to perform matrix multiplication. The aggregated self-attention map If we say, It is equal to mathematical formula 4 below.

[0046]

[0047] Referring to mathematical formula 4, The range of is is, silver Defined as. Is It refers to the first self-attention map in the i-th layer, and the finally aggregated first self-attention map If you say that, represents the number of layers in the encoder.

[0048] The self-feedback device (200) is a first guidance map ( ) and the number of encoder layers( The first self-attention map finally aggregated according to ) = A first objective function can be obtained by performing calculations based on ). The self-feedback device (200) obtains the first objective function ( based on the following mathematical formula 5) You can obtain ).

[0049]

[0050] Referring to mathematical formula 5, represents Kullback-Leibler divergence. That is, the self-feedback device (200) obtains an aggregated self-attention map by performing consecutive matrix multiplications on one or more first self-attention maps according to the number of layers of the encoder, and can apply a first objective function, which is a Kullback-Leibler divergence, to the encoder based on the aggregated self-attention map and the first guidance map.

[0051] Referring to FIG. 5, the self-feedback device (200) may extract one or more second guidance maps through one or more layers within the decoder to apply a second objective function to the decoder within the self-feedback device (200). The decoder within the self-feedback device (200) includes one or more self-attention layers and one or more cross-attention layers, and since each layer shares a representation, the representation level of each layer may be the same. Therefore, the self-feedback device (200) may not need to aggregate the second self-attention maps for multiple layers within the decoder. Instead, the self-feedback device (200) may obtain a second objective function by extracting one or more second guidance maps and second self-attention maps.

[0052] The self-feedback device (200) is a decoder within the self-feedback device (200). Second self-attention map in the nth self-attention layer ( ), of the decoder Second guidance map in the nth cross-attention layer ( A second objective function can be obtained by performing calculations based on ). The self-feedback device (200) obtains the second objective function ( based on the following mathematical formula 6) You can obtain ).

[0053]

[0054] Referring to mathematical formula 6, represents the number of layers of the decoder in the self-feedback device (200), and represents Kullback-Leibler divergence. That is, the self-feedback device (200) can apply a second objective function, which is Kullback-Leibler divergence calculated according to the number of layers of the decoder based on one or more second self-attention maps and one or more second guidance maps, to the decoder.

[0055] FIG. 6 is a flowchart illustrating a self-feedback method for temporal behavior detection according to an embodiment of the present invention.

[0056] Referring to FIG. 6, a self-feedback device based on an artificial neural network comprising an encoder, a decoder, and one or more queries for said decoder receives a video image and can extract one or more video features based on said video image (S410). The self-feedback device (e.g., the self-feedback device (200) of FIG. 2) may receive said video image from a video image acquisition device (e.g., the video image acquisition device (100) of FIG. 2).

[0057] The self-feedback device can extract one or more first self-attention maps and one or more encoder features through the encoder based on one or more image features, and can extract one or more cross-attention maps and one or more second self-attention maps through the decoder based on one or more encoder features and one or more queries (S430). The self-feedback device can extract one or more first guidance maps for the encoder and one or more second guidance maps for the decoder based on one or more cross-attention maps and the transpose of one or more cross-attention maps (S450). The self-feedback device can apply a first objective function including one or more first guidance maps and one or more first self-attention maps to the encoder (S470). The self-feedback device may apply a second objective function comprising one or more second guidance maps and one or more second self-attention maps to the decoder (S490). At this time, the artificial neural network may include the encoder including a self-attention sublayer and the decoder including a cross-attention layer and a self-attention sublayer.

[0058] In one embodiment, the self-feedback device can reduce information about spatial dimensions and extract one or more image features that are information about temporal dimensions by performing global average pooling through a 3D CNN (3-dimensional convolutional neural network) in the artificial neural network based on the image.

[0059] In one embodiment, the self-feedback device can extract the first guidance map that considers all of the one or more image features input to the encoder and the second guidance map that considers all of the one or more encoder features input to the decoder by performing an element-wise square root operation based on the one or more cross-attention maps and the transpose of the one or more cross-attention maps.

[0060] In one embodiment, the self-feedback device may obtain an aggregated self-attention map by performing consecutive matrix multiplications on one or more first self-attention maps according to the number of layers of the encoder, and apply the first objective function, which is a Kullback-Leibler divergence calculated based on the aggregated self-attention map and the first guidance map, to the encoder.

[0061] In one embodiment, the self-feedback device may apply the second objective function, which is a Kullback-Leibler divergence calculated according to the number of layers of the decoder based on the one or more second self-attention maps and the one or more second guidance maps, to the decoder.

[0062] FIG. 7 is a block diagram illustrating a self-feedback device for temporal behavior detection according to another embodiment of the present invention.

[0063] The self-feedback device (300) of FIG. 7 may be identical to the self-feedback device (200) of FIG. 2. The self-feedback device (300) may include at least one processor (310), a memory (320), and a transceiver device (330) that is connected to a network to perform communication. Additionally, the self-feedback device (300) may further include an input interface device (340), an output interface device (350), a storage device (360), etc. Each component included in the self-feedback device (300) may be connected by a bus (370) to perform communication with each other. However, each component included in the self-feedback device (300) may be connected through individual interfaces or individual buses centered around the processor (310), rather than through a common bus (370). For example, the processor (310) may be connected to at least one of a memory (320), a transmission / reception device (330), an input interface device (340), an output interface device (350), and a storage device (360) through a dedicated interface.

[0064] The processor (310) can execute a program command stored in at least one of the memory (320) and the storage device (360). The processor (310) may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Each of the memory (320) and the storage device (360) may be composed of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory (320) may be composed of at least one of read-only memory (ROM) and random access memory (RAM).

[0065] According to the embodiments of the self-feedback device of the present specification described above, a self-feedback method and model are proposed to generate a guidance map by utilizing a cross-attention map between the encoder and decoder of a model in order to overcome the temporal collapse phenomenon occurring in DETR models and DETR-based models. Through this, the problem of information loss caused by the model's attention mechanism focusing only on a few important elements during the process of input image sequences to DETR models and DETR-based models, thereby ignoring temporal features or elements of the input image and focusing only on specific sections, can be overcome. Furthermore, the problem of hindering the learning of DETR models and DETR-based models and degrading model performance due to the temporal collapse phenomenon can be overcome.

[0066] In addition, the self-feedback device of this specification demonstrated the performance of the proposed Self-DETR model against benchmarks of THUMOS14 and ActivityNet-v1.3. The metric used for performance comparison was mean Average Precision (mAP), which is the most widely used metric in object detection and temporal behavior detection, and the results are shown in Table 1 below.

[0067]

[0068] Table 1 compares the performance of the Self-Detr with the latest techniques and models. Referring to Table 1, it shows that on the THUMOS14 dataset, the Self-Detr achieved the highest performance among all techniques and models based on mAP (Avg. in Table 1). Additionally, it shows that on the ActivityNet-v1.3 benchmark, the Self-Detr achieved the highest performance compared to other conventional DETR-based models.

[0069] In addition, the self-feedback device of the present specification has different performance depending on whether the first objective function for the encoder and the second objective function for the decoder are applied, and the results regarding the performance are as shown in Table 2 below.

[0070]

[0071] Table 2 is the first objective function for the encoder in the self-feedback device ( ) and the second objective function for the decoder within the self-feedback device ( The table shows the performance for ). It can be seen that the performance is numerically improved when the first objective function is applied to the encoder or the second objective function is applied to the decoder compared to when neither the first nor the second objective function is applied to the self-feedback device. In addition, it can be seen that the performance of the self-feedback device is improved complementarily when both the first and second objective functions are used.

[0072] Embodiments according to the present specification may be implemented by various means, e.g., hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, an embodiment of the present specification may be implemented by one or more ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, etc. In the case of implementation by firmware or software, an embodiment of the present specification may be implemented in the form of modules, procedures, functions, etc., that perform the capabilities or operations described above. Software code may be stored in memory and executed by a processor. The memory may be located inside or outside the processor and may exchange data with the processor by various means already known.

[0073] Meanwhile, the embodiments of this specification may be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer-readable recording medium may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner. Furthermore, functional programs, code, and code segments for implementing the embodiments can be easily inferred by programmers in the art to which this specification belongs.

[0074] The foregoing has focused on the various embodiments of this specification. Those skilled in the art will understand that various embodiments may be implemented in modified forms without departing from the essential nature of this specification. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of this specification is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in this specification.

[0075] Most terms used in this invention are selected from those commonly used in the field, but some terms are chosen at the applicant's discretion, and their meanings are described in detail in the following description as necessary. Accordingly, this invention should be understood based on the intended meaning of the terms, rather than their mere names or meanings.

[0076] It is obvious to those skilled in the art that the present invention may be embodied in other specific forms without departing from the essential features of the invention. Accordingly, the foregoing detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

Claim 1 A self-feedback device based on an artificial neural network comprising an encoder, a decoder, and one or more queries for the decoder receives a video image and extracts one or more video features based on the video image; the self-feedback device extracts one or more first self-attention maps and one or more encoder features through the encoder based on the one or more video features, and extracts one or more cross-attention maps and one or more second self-attention maps through the decoder based on the one or more encoder features and the one or more queries; the self-feedback device extracts one or more first guidance maps for the encoder and one or more second guidance maps for the decoder based on the one or more cross-attention maps and the transpose of the one or more cross-attention maps; the self-feedback device extracts the one or more first guidance maps and the one or more first self-attention maps A step of applying a first objective function including an attention map to the encoder; and a step of the self-feedback device applying a second objective function including one or more second guidance maps and one or more second self-attention maps to the decoder;A self-feedback method for an artificial neural network comprising: a step of applying the first objective function to the encoder, wherein the first objective function is a Kullback-Leibler divergence computed based on the aggregated self-attention map and the first guidance map, and the step of applying the second objective function to the decoder, wherein the second objective function is a Kullback-Leibler divergence computed based on the number of layers of the decoder, and the second objective function is a Kullback-Leibler divergence computed based on the number of layers of the decoder. Claim 2 A self-feedback method of an artificial neural network according to claim 1, wherein the artificial neural network comprises an encoder including a self-attention sublayer and a decoder including a cross-attention layer and a self-attention sublayer. Claim 3 A self-feedback method of an artificial neural network according to claim 1, wherein the step of extracting one or more image features is characterized by performing global average pooling through a 3D CNN (3-dimensional convolutional neural network) within the artificial neural network based on the image, thereby reducing information regarding spatial dimensions and extracting one or more image features which are information regarding temporal dimensions. Claim 4 A self-feedback method of an artificial neural network according to claim 1, wherein the step of extracting one or more first guidance maps for the encoder and one or more second guidance maps for the decoder is characterized by extracting the first guidance map that considers all one or more image features input to the encoder and the second guidance map that considers all one or more encoder features input to the decoder by performing an element-wise square root operation based on the one or more cross-attention maps and the transpose of the one or more cross-attention maps. Claim 5 delete Claim 6 delete Claim 7 A computer-readable recording medium having a program for executing the method of any one of paragraphs 1 to 4 on a computer. Claim 8 A self-feedback device based on an artificial neural network comprising an encoder, a decoder, and one or more queries for said decoder, comprising: a processor; The method comprises a memory in which one or more instructions are stored by the processor, wherein the one or more instructions include: receiving a video image as input and extracting one or more video features based on the video image; extracting one or more first self-attention maps and one or more encoder features through the encoder based on the one or more video features, and extracting one or more cross-attention maps and one or more second self-attention maps through the decoder based on the one or more encoder features and one or more queries; extracting one or more first guidance maps for the encoder and one or more second guidance maps for the decoder based on the one or more cross-attention maps and the transpose of the one or more cross-attention maps; and applying a first objective function including the one or more first guidance maps and the one or more first self-attention maps to the encoder. and a step of applying a second objective function comprising one or more second guidance maps and one or more second self-attention maps to the decoder;A self-feedback device for an artificial neural network comprising: a step of applying the first objective function to the encoder, wherein the first objective function is a Kullback-Leibler divergence computed based on the aggregated self-attention map and the first guidance map, and the step of applying the second objective function to the decoder, wherein the second objective function is a Kullback-Leibler divergence computed based on the one or more second self-attention maps and the one or more second guidance maps, and the second objective function is a Kullback-Leibler divergence computed based on the number of layers of the decoder. Claim 9 A self-feedback device of an artificial neural network according to claim 8, wherein the artificial neural network comprises an encoder including a self-attention sublayer and a decoder including a cross-attention layer and a self-attention sublayer. Claim 10 A self-feedback device of an artificial neural network, wherein the step of extracting one or more image features is characterized by performing global average pooling through a 3D CNN (3-dimensional convolutional neural network) within the artificial neural network based on the image, thereby reducing information regarding spatial dimensions and extracting one or more image features which are information regarding temporal dimensions. Claim 11 A self-feedback device for an artificial neural network according to claim 8, wherein the step of extracting one or more first guidance maps for the encoder and one or more second guidance maps for the decoder is characterized by extracting the first guidance map that considers all one or more image features input to the encoder and the second guidance map that considers all one or more encoder features input to the decoder by performing an element-wise square root operation based on the one or more cross-attention maps and the transpose of the one or more cross-attention maps. Claim 12 delete Claim 13 delete Claim 14 Image acquisition device for acquiring image images; A self-feedback device based on the artificial neural network, which receives the image from the image acquisition device, applies a first objective function to the encoder of the artificial neural network, and applies a second objective function to the decoder of the artificial neural network; wherein the self-feedback device receives the image, extracts one or more video features based on the image, extracts one or more first self-attention maps and one or more encoder features through the encoder based on the one or more video features, extracts one or more cross-attention maps and one or more second self-attention maps through the decoder based on the one or more encoder features and one or more queries, extracts one or more first guidance maps for the encoder and one or more second guidance maps for the decoder based on the one or more cross-attention maps and the transpose of the one or more cross-attention maps, and the one or more first guidance maps and A first objective function comprising one or more first self-attention maps is applied to the encoder, and a second objective function comprising one or more second guidance maps and one or more second self-attention maps is applied to the decoder; when the first objective function is applied to the encoder, an aggregated self-attention map is obtained by performing consecutive matrix multiplication of the one or more first self-attention maps according to the number of layers of the encoder, and the first objective function, which is a Kullback-Leibler divergence computed based on the aggregated self-attention map and the first guidance map, is applied to the encoder, and when the second objective function is applied to the decoder,A self-feedback system of an artificial neural network characterized by applying the second objective function, which is a Kullback-Leibler divergence calculated according to the number of layers of the decoder based on the one or more second self-attention maps and the one or more second guidance maps, to the decoder. Claim 15 A self-feedback system of an artificial neural network according to claim 14, wherein the artificial neural network comprises an encoder including a self-attention sublayer and a decoder including a cross-attention layer and a self-attention sublayer. Claim 16 A self-feedback system of an artificial neural network according to claim 14, characterized in that when extracting one or more image features, global average pooling is performed through a 3D CNN (3-dimensional convolutional neural network) within the artificial neural network based on the image, thereby reducing information regarding spatial dimensions and extracting one or more image features which are information regarding temporal dimensions. Claim 17 A self-feedback system of an artificial neural network according to claim 14, characterized in that, when extracting one or more first guidance maps for the encoder and one or more second guidance maps for the decoder, the first guidance map is extracted to consider all one or more image features input to the encoder and the second guidance map is extracted to consider all one or more encoder features input to the decoder by performing an elementwise square root operation based on one or more cross-attention maps and the transpose of one or more cross-attention maps. Claim 18 delete Claim 19 delete

Citation Information

Patent Citations

  • Image target detection method based on deep supervision self-distillation

    CN114863248A