Method and apparatus for detecting object in video by using single image

Abstract

A method and an apparatus for detecting an object in a video by using a single image are disclosed. According to one aspect of the present disclosure, a computer-implemented method for detecting an object in a video is provided, the method comprising the steps of: encoding an image characteristic map extracted from the current frame and a pre-stored memory characteristic map, the memory characteristic map including a predefined number of memory characteristics for each of one or more predictable classes; generating a sampled memory characteristic map on the basis of the memory characteristic map and classification scores calculated from the encoded memory characteristic map; predicting a bounding box and a class of an object in the current frame on the basis of the encoded image characteristic map and the sampled memory characteristic map; and updating the memory characteristic map on the basis of the encoded image characteristic map.

Description

Method and device for detecting objects in video using a single image

[0001] The present disclosure relates to a method and apparatus for detecting an object in a video using a single image.

[0002] The content described below merely provides background information related to the present embodiment and does not constitute prior art.

[0003] Object detection is a fundamental and essential task in computer vision, widely used in various applications. In particular, object detection in video data is utilized in various applications such as CCTV, autonomous driving, and robot navigation. While object detection models for single images have shown high success rates, applying these models directly to video data can result in performance degradation due to object deformation caused by motion and occlusion.

[0004] To address these issues, video object detection (VOD) models specialized for processing video data have been proposed. These models utilize optical flow, Long Short-Term Memory (LSTM), or attention mechanisms to process image sequences. Methods utilizing optical flow or LSTM primarily focus on short-term frames close to the current frame, limiting their ability to capture broader feature representations. Meanwhile, attention-based methods struggle to integrate the overall information of video data because they obtain overall contextual information from randomly sampled images. Furthermore, video object detection models often utilize additional reference frames or accumulate and store information from the entire preceding frame to leverage the characteristics of adjacent frames. This, however, can lead to unnecessary references, high computational costs, and unnecessary memory usage.

[0005] The present disclosure aims to provide a single image-based video object detection method and device capable of effectively integrating contextual information across given data.

[0006] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0007] According to one aspect of the present disclosure, a computer-implemented method for detecting an object in a video is provided, the method comprising: encoding a pre-stored memory feature map, the memory feature map including a predefined number of memory features for each of predictable classes, and an image feature map extracted from a current frame; generating a sampled memory feature map based on classification scores derived from the encoded memory feature map and the memory feature map; predicting a bounding box and a class of an object in the current frame based on the encoded image feature map and the sampled memory feature map; and updating the memory feature map based on the encoded image feature map.

[0008] According to another aspect of the present disclosure, there is provided a device comprising: a memory storing instructions; and at least one processor, wherein the at least one processor executes the instructions to encode a pre-stored memory feature map, the memory feature map including a predefined number of memory features for each of predictable classes, and an image feature map extracted from a current frame, and generate a sampled memory feature map based on classification scores calculated from the encoded memory feature map and the memory feature map, and predict a bounding box and a class of an object in the current frame based on the encoded image feature map and the sampled memory feature map, and update the memory feature map based on the encoded image feature map.

[0009] According to another aspect of the present disclosure, a computer program stored in a computer-readable recording medium is provided to execute each process included in the above-described method.

[0010] According to embodiments of the present disclosure, contextual information can be effectively integrated across the entire observed data by utilizing a context memory module. This enables robust object detection based on an understanding of the environment in which the camera is installed.

[0011] According to embodiments of the present disclosure, temporal context from previous frames can be utilized without inputting additional reference frames or using auxiliary networks. This prevents processing speed reduction due to reference frames or auxiliary networks, effectively meeting real-time requirements essential for practical applications.

[0012] According to embodiments of the present disclosure, only necessary information extracted and processed from the current frame can be selectively stored, thereby maintaining the context memory module at a fixed size. Furthermore, score-based sampling allows the context memory module to effectively capture information related to the current frame.

[0013] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0014] FIG. 1 is a block diagram schematically illustrating a video object detection device according to one embodiment of the present disclosure.

[0015] FIG. 2 is a drawing referenced to explain the structure of a single image object detector according to one embodiment of the present disclosure.

[0016] FIG. 3 is a drawing referenced to explain the operation of an update module according to one embodiment of the present disclosure.

[0017] FIG. 4 is a drawing referenced to explain the operation of a sampling module according to one embodiment of the present disclosure.

[0018] FIG. 5A and FIG. 5B are drawings for reference to explain the structure of a decoder block according to various embodiments of the present disclosure.

[0019] FIG. 6 is a flowchart illustrating an object detection method according to one embodiment of the present disclosure.

[0020] FIG. 7 is a block diagram schematically illustrating an exemplary computing device that can be used to implement the devices and methods described in the present disclosure.

[0021] Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.

[0022] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be included, unless explicitly stated otherwise.

[0023] The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

[0024] FIG. 1 is a block diagram schematically illustrating a video object detection device according to one embodiment of the present disclosure. FIG. 2 is a diagram illustrating memory characteristics stored in a context memory module according to one embodiment of the present disclosure.

[0025] As illustrated in FIG. 1, a video object detection device (1) according to an embodiment of the present disclosure may include all or part of a single image object detector (10) and a context memory module (12). Not all blocks illustrated in FIG. 1 are essential components, and some blocks may be added, changed, or deleted in other embodiments. Meanwhile, the components illustrated in FIG. 1 represent functionally distinct elements, and at least one of the components may be implemented in a form in which they are integrated with each other in an actual physical environment.

[0026] A single image object detector (10) detects one or more objects from a current frame. Detecting an object may include predicting information about the class of the object and information about a bounding box surrounding the object. The information about the class may include, for example, a probability that the object belongs to each of a plurality of predefined classes and / or an identifier of the class with the highest probability. The information about the bounding box may include, for example, a combination of two or more of the upper-left coordinate of the bounding box, the lower-right coordinate of the bounding box, the center coordinate of the bounding box, and / or the size (width and height) of the bounding box.

[0027] The single-image object detector (10) can be implemented as a neural network-based model. In describing the present disclosure, it is assumed that the single-image object detector (10) follows the architecture of DETR (DEtection TRansformer), a Transformer-based detector, at least in some of its components. However, it should be noted that the technical concepts of the present disclosure can also be applied to single-image object detectors having other types of architectures, as long as they can operate in conjunction with the context memory module (12).

[0028] The context memory module (12) stores the temporal and / or spatial context of preceding frames in a fixed size. The context memory module (12) can store a fixed number of class-wise feature representations for each predictable class. The feature representations may be referred to as memory features or prototypes, and a set of these may be referred to as a class-wise memory feature map or memory feature map. For example, the context memory module (12) can store a multi-prototype class-wise memory memory feature map having multiple prototypes for each class. When the number of predictable classes is C and the number of prototypes is K, the memory feature map can be expressed as in Mathematical Expression 1.

[0029]

[0030] Here, c is an identifier representing a class corresponding to each memory characteristic, and can have a value between 1 and C. k is an identifier for distinguishing memory characteristics corresponding to the same class, and can have a value between 1 and K. The context memory module (12) expresses each class as a set of prototypes, thereby enabling various properties to be included within a class (intra-classes).

[0031] The single-image object detector (10) selectively stores only the information necessary for processing each frame in the context memory module (12), and utilizes only the information useful for the current frame. Accordingly, the single-image object detector (10) can efficiently utilize context information for the entire dataset observed to date for each frame.

[0032] Hereinafter, with reference to FIGS. 2 to 5b, the operation of a single image object detector (10) according to one embodiment of the present disclosure will be described.

[0033] FIG. 2 is a drawing referenced to explain the structure of a single image object detector according to one embodiment of the present disclosure.

[0034] A single image object detector (10) may include all or part of a backbone network (20), an encoder (21), an update module (22), a sampling module (23), a memory-guided decoder (25), a classification head (26), and a regression head (27).

[0035] The backbone network (20) extracts an image feature map (200) from the current frame. A pre-trained convolutional neural network (CNN) may be used as the backbone network (20). To provide relative spatial positional information of each feature to the encoder (21), positional encoding may be added to the image feature map (200). The positional encoding may be generated through a fixed function, such as a sine function and / or a cosine function, or may be generated by an additional learnable embedding layer.

[0036] The encoder (21) receives the image feature map (200) extracted by the backbone network (20) and the memory feature map (220) stored in the context memory module (12), respectively, and generates an encoded image feature map (202) and an encoded memory feature map (222). The encoder (21) may be implemented as a transformer encoder. The encoder (21) may include a plurality of encoder blocks, and each encoder block may include a multi-head self-attention layer. The encoder (21) may concatenate the image feature map (200) and the memory feature map (220), and sequentially apply a plurality of encoder blocks to the concatenated feature maps, thereby generating an encoded image feature map (202) and an encoded memory feature map (222).

[0037] Most single-image-based object detectors use a transformer encoder to gather spatial information from image features extracted from the current frame. The self-attention structure of the transformer encoder allows each image feature to include spatial context information of the current frame. However, if the backbone network (20) extracts ambiguous features for the current frame (e.g., when the image quality is low or some areas are occluded), it may be difficult to effectively improve the image features. To alleviate this limitation, the encoder (21) according to the present disclosure improves the image features of each single frame using fixed-size memory features obtained from all preceding frames. This allows the image features of the current frame to be improved without directly using data from other frames.

[0038] The image feature map (200), the memory feature map (220), the encoded image feature map (202), and the encoded memory feature map (222) are respectively referred to as F, M, , and When this happens, the encoder (21) can be expressed as in mathematical expression 2.

[0039]

[0040] Here, [·,·] represents the connection between two feature maps, H and W represent the height and width of the image feature map (200), C represents the number of classes to be predicted by the single image object detector (10), K represents the number of features (i.e., prototypes) stored in the context memory module (12) for each class, and d represents the dimension of the features. As shown in mathematical expression 1, the spatial dimensions of the image feature map (200) can be flattened into one dimension so that the image feature map (200) can be processed in the encoder (21) that receives a sequence as input. Similarly, in the memory feature map (220), the class dimension and the prototype dimension can be flattened into one dimension.

[0041] In the encoder (21), information about the current frame embedded in the image feature map (200) and spatio-temporal contextual information from previous frames embedded in the memory feature map (220) are aggregated. Accordingly, the encoded image feature map (202) obtains rich contextual information from the memory feature map (220), and at the same time, the encoded memory feature map (222) obtains class information suitable for the current frame.

[0042] The encoded memory feature map (222) is passed to the sampling module (23) to obtain a classification score for the current frame. The encoded image feature map (202) is passed to the memory-guided decoder (25) for object detection and is also passed to the update module (22) for memory update.

[0043] The update module (22) updates at least some of the memory features stored in the context memory module (12) based on the encoded image feature map (202). As a method for updating the memory features, a momentum update, which is a non-parametric method, can be applied. An exemplary embodiment of the update module (22) will be described below with reference to FIG. 3.

[0044] The sampling module (23) samples information necessary for processing the current frame from the context memory module (12). The sampling module (23) can extract information related to the current image from among the memory characteristics stored in the context memory module (12) and construct a sampled memory characteristic map (224). Carefully selecting relevant information from the memory is as important as creating a high-quality memory. Randomly sampling information from the memory or using only information from frames adjacent to the current frame cannot guarantee optimal memory sampling. To solve this problem, the sampling module (23) extracts information related to the current frame from the context memory module (12) based on a score calculated from the encoded memory characteristic map (222). An exemplary embodiment of the sampling module (23) will be described below with reference to FIG. 4.

[0045] The encoded memory feature map (222) and the sampled memory feature map (224) are passed to a memory-guided decoder (25) to predict the bounding box and class of the object in the current frame.

[0046] The memory-guided decoder (25) includes one or more decoder blocks (250-1 to 250-L). Each decoder block (250-1 to 250-L) is passed a fixed number of learnable (or learned) positional embeddings, object queries (240), an encoded image feature map (202), and a sampled memory feature map (224). The decoder blocks (250-1 to 250-L) can utilize the sampled spatiotemporal memory information to enhance the semantic information of the input object queries. Exemplary embodiments of the decoder blocks (250-1 to 250-L) will be described below with reference to FIGS. 5A and 5B.

[0047] The classification head (26) and the regression head (27) can predict a predefined number of classes and a set of bounding boxes using the final object queries (242) output by the last decoder block (240-L). The classification head (26) and the regression head (27) can be configured as, for example, a feed-forward network.

[0048] For the inference process, sets of predicted classes and boxes can be output as object detection results for the current frame. For the training process, sets of bounding boxes and classes predicted by the heads (26 and 27) can be matched with a ground truth (GT) set according to a predetermined matching algorithm, and learnable parameters of the single-image object detector (10) can be updated based on a loss calculated based on the matched pair. For example, Hungarian matching and Hungarian loss used in DETR can be used as the matching algorithm and loss, but the present invention is not limited to these examples.

[0049] FIG. 3 is a drawing referenced to explain the operation of an update module according to one embodiment of the present disclosure.

[0050] The update module (22) extracts one or more instance features (300) from the encoded image feature map (202). During the training process, the instance features (300) may be extracted based on the GT bounding box, and during the inference process, the instance features (300) may be extracted based on the predicted bounding box. The update module (22) may adjust the GT bounding box or the predicted bounding box to the size of the reduced feature map through Region of Interest Alignment (RoI) and extract the instance features (300) therefrom. The update module (22) may identify a class corresponding to each instance feature based on the GT class or the class predicted by the single image object detector (10). Assuming that N instance features (300) are extracted from the encoded image feature map (202), the instance features (300) may be expressed as in Mathematical Expression 3.

[0051]

[0052] Here, c represents the identifier of the class corresponding to each instance property, and can have a value between 1 and C, for example.

[0053] The update module (22) can select a memory characteristic to be updated based on the instance characteristic among K memory characteristics corresponding to the same class as each instance characteristic. For example, the update module (22) can select the memory characteristic with the highest correlation with the instance characteristic as the target memory characteristic to be updated. The update module (22) can update the selected memory characteristic based on momentum update or linear interpolation. For example, the update process of the context memory module (12) can be expressed as in mathematical equation (4).

[0054]

[0055] Here, <·,·> is defined as the correlation between two characteristics, and α is a momentum coefficient that can be set to a value greater than or equal to 0 and less than or equal to 1.

[0056] The encoded image feature map (202) includes rich contextual information aggregated from the memory feature map (220). Therefore, the memory feature map (220), which is repeatedly updated by the encoded image feature map (202) during the sequential processing of multiple frames of a video, includes contextual information about the entire previously observed dataset. Furthermore, since the memory feature map (220) has multiple prototypes for each class, it can accommodate diverse distributions of instance features appearing in the entire dataset.

[0057] Meanwhile, the context memory module (12) stores features within the embedding space on the output side of the encoder (21). In order to transfer these to the same embedding space as the image feature map (200), memory embedding may be applied before the memory feature map (220) is input to the encoder (21). Memory embedding may be performed, for example, by a shallow multi-layer perceptron (MLP).

[0058] FIG. 4 is a drawing referenced to explain the operation of a sampling module according to one embodiment of the present disclosure.

[0059] The sampling module (23) can generate a sampled memory characteristic map (224) based on the classification scores and the memory characteristic map (220) derived from the encoded memory characteristic map (222). The operation of the sampling module (23) can include a classification process and a multi-threshold sampling process.

[0060] In the classification process, the sampling module (23) obtains a classification score corresponding to each encoded memory characteristic from the encoded memory characteristic map (222) containing class information of the current frame. In the present disclosure, the classification score may also be referred to as confidence.

[0061] The sampling module (23) can obtain a classification score by applying each encoded memory feature to a classification head (400) independently configured for the corresponding encoded memory feature. The classification head (400) can be configured as, for example, a feed-forward network. In order to express the classification score as a value between 0 and 1, a sigmoid function (410) can follow the classification head (400). At this time, the encoded memory feature map (222) can be expressed as a set of encoded memory features as in mathematical expression 5, and at this time, the classification score for the k-th encoded memory feature of the c-th class can be calculated as in mathematical expression 6.

[0062]

[0063]

[0064] Here, FFN c,k represents a classification head (400) corresponding to the kth encoded memory characteristic of the cth class, and Sigmoid(·) represents a sigmoid function.

[0065] In the multi-threshold sampling process, the sampling module (23) obtains a combination of thresholded memory features by multiple thresholds (440) having different values ​​and classification scores corresponding to the same class-prototype position for each of the memory features constituting the memory feature map (220). For example, the k-th memory feature m of the c-th class c,k The tth threshold τ corresponding to t The memory characteristics thresholded by can be calculated as in mathematical expression 7.

[0066]

[0067] Here, Φ represents a no-class embedding, which is a learnable embedding (or a pre-trained embedding for the inference process). δ(·) outputs 1 when the condition is true, and 0 otherwise. That is, the sampling index is the classification score p for the kth encoded memory feature of the cth class. c,k The tth threshold τ t It is a binary value.

[0068] The sampling module (23) obtains sampled memory characteristics corresponding to each memory characteristic by embedding a combination of thresholded memory characteristics using the projection layer (460). For example, the sampled memory characteristic map (224) can be expressed as a set of sampled memory characteristics corresponding to each memory characteristic as in mathematical expression 8, and the sampled memory characteristic corresponding to the kth memory characteristic of the cth class can be calculated as in mathematical expression 9.

[0069]

[0070]

[0071] Here, T represents the number of thresholds and / or sampling indices, and Proj(·) represents a projection layer (460). The projection layer (460) generates sampled memory features by combining multi-thresholded memory features with various confidence levels.

[0072] In the training process, an asymmetric loss function (ASL) may be additionally used to train the classification head (400) and improve the class discrimination ability of the encoder (21).

[0073] FIG. 5A is a drawing referenced to explain the structure of a decoder block according to one embodiment of the present disclosure.

[0074] As illustrated in FIG. 5A, the decoder block (250) may include a self-attention layer (500), a cross-attention layer (520), and a memory cross-attention layer (540). Each attention layer may be configured as a multi-head attention layer including multiple attention heads. Meanwhile, although not illustrated in FIG. 5A, at least some of the attention layers (500, 520, and 540) may be followed by residual connection and normalization layers.

[0075] The decoder block (250) can sequentially apply object queries (502) output by a preceding decoder block (not shown) to a self-attention layer (500) and a cross-attention layer (520) to obtain an object query including semantic information about the current frame. The decoder block (250) can apply the object query output by the cross-attention layer (520) to a memory cross-attention layer (540) to strengthen class information associated with the current frame.

[0076] The self-attention layer (500) calculates self-attention for object queries (502). The object query output by the self-attention layer (500) of the lth decoder block can be expressed as in mathematical expression 10.

[0077]

[0078] Here, represents the object query (or initial object query) output by the l-1th decoder block.

[0079] The cross-attention layer (520) calculates cross-attention between the object queries output by the self-attention layer (500) and the encoded image feature map (202). The object query output by the cross-attention layer (520) of the lth decoder block can be expressed as in mathematical expression 11.

[0080]

[0081] The memory cross-attention layer (540) calculates cross-attention between the object queries output by the cross-attention layer (520) and the sampled memory feature map (224). The object query (504) output by the memory cross-attention layer (540) of the lth decoder block can be expressed as in mathematical expression 12.

[0082]

[0083] Positional information may be added to input tokens for computing query embeddings and key embeddings in the attention layers (500, 520, and 540). The positional information provided to each attention layer (500, 520, and 540) may include, for example, object queries (506), which are learnable (or learned) positional embeddings shared among all decoder blocks (250-1 to 250-L), positional encodings of the image feature map (200), and / or memory encodings. To distinguish from the object queries (506), the object queries (502) output from the preceding decoder block or the preceding attention layer may be referred to as decoder embeddings.

[0084] For example, in the self-attention layer (500), query embeddings and key embeddings may be calculated based on the sum of decoder embeddings (502) output by the preceding decoder block and object queries (506). As another example, in the cross-attention layer (520), query embeddings may be calculated based on the sum of decoder embeddings (502) output by the self-attention layer (500) and object queries (506), and key embeddings may be calculated based on the sum of the encoded image feature map (202) and positional encoding. As another example, in the memory attention layer (540), query embeddings may be calculated based on the sum of decoder embeddings (506) output by the cross-attention layer (520) and object queries, and key embeddings may be calculated based on the sum of the sampled memory feature map (224) and memory encoding. Here, the memory encoding can be generated in a manner identical or similar to the positional encoding of the image feature map (200), and can serve the function of indicating that the input tokens are features extracted from the context memory module (12).

[0085] FIG. 5b is a drawing referenced to explain the structure of a decoder block according to another embodiment of the present disclosure.

[0086] Referring to FIG. 5B, a decoder block (250) according to another embodiment of the present disclosure may provide positional information extracted from learnable (or learned) anchor boxes (anchor boxes) 512 to each of the attention layers (500, 530, and 540). In the present embodiment, in order to adjust the size of the cross-attention map to fit the size of the anchor boxes (512), the cross-attention layer (520) may be replaced with a width-and-height-modulated cross-attention layer (width&height-modulated cross-attention layer) 530. In the width-and-height-modulated cross-attention layer (530) and the memory cross-attention layer (540), positional information may be concatenated with input tokens for computing query embeddings and key embeddings.

[0087] In this embodiment, anchor boxes (512) can be updated in units of decoder blocks (250). For example, in each decoder block (250), the position and size changes (514) of anchor boxes can be predicted based on the output of the width-and-height-modulated cross-attention layer (530). New anchor boxes (516) updated based on the predicted changes (514) can be passed to a subsequent decoder block (not shown).

[0088] FIG. 6 is a flowchart illustrating an object detection method according to one embodiment of the present disclosure.

[0089] The method illustrated in FIG. 6 can be implemented by executing the functions of one or more components of the aforementioned video object detection device (1) by at least one computing device. Therefore, the following description is described in terms of operations performed by the computing device.

[0090] The computing device extracts an image feature map from the current frame (S600). The computing device can obtain the image feature map by applying the current frame to a pre-trained backbone network. The computing device can apply positional encoding to the image feature map and adjust its dimensionality.

[0091] A computing device encodes an image feature map and a pre-stored memory feature map (S620). For example, the computing device may read the memory feature map from a context memory module. The memory feature map may include a predefined number of memory features for each of one or more predictable classes. The computing device may concatenate the image feature map and the memory feature map, and obtain an encoded image feature map and an encoded memory feature map based on self-attention to the concatenated feature map. The encoded image feature map may include spatiotemporal contexts of all frames observed prior to the current frame. The encoded memory feature map may include information about the class of an object within the current frame. The encoded memory feature map may include encoded memory features corresponding to each memory feature. An encoded memory feature corresponding to a specific memory feature may have the same class-prototype location as the corresponding memory feature.

[0092] The computing device generates a sampled memory characteristic map based on the classification scores and the memory characteristic map derived from the encoded memory characteristic map (S640). The sampled memory characteristic map may include sampled memory characteristics corresponding to each memory characteristic. The sampled memory characteristic corresponding to a specific memory characteristic may have the same class-prototype location as the corresponding memory characteristic. The computing device may generate a classification score for each encoded memory characteristic. The computing device may threshold the memory characteristic corresponding to each encoded memory characteristic based on one or more predefined thresholds. For example, the computing device may binarize the classification score corresponding to a specific class-prototype location based on one or more predefined thresholds to obtain one or more sampling indices, and may select one of the memory characteristic of the corresponding class-prototype location and the classless embedding as the thresholded memory characteristic based on the value of each sampling index. The computing device may embed one or more thresholded memory characteristics selected for each sampling index to generate a sampled memory characteristic corresponding to each memory characteristic.

[0093] The computing device predicts the bounding box and class of an object within the current frame based on the encoded image feature map and the sampled memory feature map (S660). The computing device applies the object query to the memory-guided decoder to obtain an output object query with enhanced semantic information about the current frame and information about the class associated with the current frame, and can predict the bounding box and class of the object from the output object query.

[0094] A memory-guided decoder may include one or more decoder blocks. Each decoder block may include a first attention layer that generates a first object query based on self-attention to an input object query, a second attention layer that generates a second object query based on cross-attention between the first object query and an encoded image feature map, and a third attention layer that generates a third object query based on cross-attention between the second object query and a sampled memory feature map. The computing device may combine memory embeddings indicating locations of memory features (e.g., class-prototype locations) corresponding to each sampled memory feature with the sampled memory feature map and provide the memory embeddings to the third attention layer.

[0095] The computing device updates the memory feature map based on the encoded image feature map (S680). The computing device extracts one or more instance features from the encoded image feature map based on a predicted bounding box or a pre-specified ground truth (GT) bounding box, and updates one or more target memory features corresponding to the instance features extracted from the memory feature map. Here, each target memory feature may be selected from among memory features corresponding to the same class as each instance feature. For example, among the memory features corresponding to the same class as a specific instance feature, the memory feature having the greatest correlation with the instance feature may be selected as the target memory feature corresponding to the instance feature. The target memory feature corresponding to the specific instance feature may be updated based on linear interpolation between the instance feature and the target memory feature.

[0096] FIG. 7 is a block diagram schematically illustrating an exemplary computing device that can be used to implement the devices and methods described in the present disclosure.

[0097] The computing device (70) may include some or all of a memory (700), a processor (720), storage (740), an input / output interface (760), and a communication interface (780). The computing device (70) may structurally and / or functionally include at least a part of the video object detection device (1). The computing device (70) may be a stationary computing device such as a desktop computer, a server, etc., as well as a mobile computing device such as a laptop computer, a smart phone, etc. The computing device (70) may be implemented as any specialized hardware accelerator capable of efficiently processing operations for an artificial intelligence model. For example, the computing device (70) may be implemented as a graphic processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPU), etc.

[0098] The memory (700) may store a program that causes the processor (720) to perform a method or operation according to various embodiments of the present disclosure. For example, the program may include a plurality of instructions executable by the processor (720), and the method illustrated in FIG. 6 may be performed by executing the plurality of instructions by the processor (720). The memory (700) may be a single memory or a plurality of memories. In this case, information required to perform the method or operation according to various embodiments of the present disclosure may be stored in a single memory or may be divided and stored in a plurality of memories. When the memory (700) is composed of a plurality of memories, the plurality of memories may be physically separated. The memory (700) may include at least one of a volatile memory and a nonvolatile memory. The volatile memory includes a static random access memory (SRAM) or a dynamic random access memory (DRAM), and the nonvolatile memory includes a flash memory.

[0099] The processor (720) may include at least one core capable of executing at least one instruction. The processor (720) may execute instructions stored in the memory (700). The processor (720) may be a single processor or multiple processors.

[0100] Storage (740) maintains stored data even when power supplied to the computing device (70) is cut off. For example, storage (740) may include non-volatile memory, or may include storage media such as magnetic tape, optical disk, or magnetic disk.

[0101] A program stored in storage (740) may be loaded into memory (700) before being executed by processor (720). Storage (740) may store a file written in a programming language, and a program generated from the file by a compiler or the like may be loaded into memory (700). Storage (740) may store data to be processed by processor (720) and / or data processed by processor (720).

[0102] The input / output interface (760) may include input devices such as a touch interface, a keyboard, a mouse, etc., and may include output devices such as a display device or a speaker. A user may trigger the execution of a program by the processor (720) and / or check the processing results of the processor (720) through the input / output interface (760).

[0103] The communication interface (780) may provide access to an external network. For example, the computing device (70) may communicate with other devices (e.g., a camera) via the communication interface (780).

[0104] Each component of the device or method according to the present invention may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0105] Various implementations of the systems and techniques described herein may be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0106] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0107] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0108] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0109]

[0110] CROSS-REFERENCE TO RELATED APPLICATION

[0111] This patent application claims priority to Korean Patent Application No. 10-2023-0145995, filed on October 27, 2023, which is incorporated herein by reference in its entirety.

Claims

1. A computer-implemented method for detecting objects in a video, A process of encoding an image feature map extracted from the current frame and a pre-stored memory feature map, wherein the memory feature map includes a predefined number of memory features for each of one or more predictable classes; A process of generating a sampled memory characteristic map based on classification scores derived from the encoded memory characteristic map and the memory characteristic map; A process of predicting a bounding box and a class of an object within the current frame based on the encoded image feature map and the sampled memory feature map; and A computer-implemented method, comprising a process of updating the memory feature map based on the encoded image feature map.

2. In paragraph 1, The above updating process is: A process of extracting one or more instance features from the encoded image feature map based on the predicted bounding box or the pre-specified ground truth (GT) bounding box; and A computer-implemented method further comprising: a process of updating one or more target memory characteristics corresponding to the extracted instance characteristics in the memory characteristic map, wherein each target memory characteristic is selected from among the memory characteristics corresponding to the same class as each instance characteristic.

3. In paragraph 2, The process of updating the above target memory characteristics is: A computer-implemented method for updating a target memory characteristic corresponding to each instance characteristic based on linear interpolation between each instance characteristic and a target memory characteristic corresponding to each instance characteristic.

4. In paragraph 2, The above updating process is: A computer-implemented method further comprising a process of selecting, as the target memory characteristic, a memory characteristic having the greatest correlation with each instance characteristic among the memory characteristics corresponding to the same class as each instance characteristic.

5. In paragraph 1, The above encoding process is, A process of concatenating the image feature map and the memory feature map; and A computer-implemented method comprising the process of obtaining an encoded image feature map containing spatiotemporal contexts of entire frames observed prior to the current frame and an encoded memory feature map containing information about classes of objects within the current frame, based on self-attention to the connected feature map.

6. In paragraph 1, The encoded memory characteristic map and the sampled memory characteristic map each include encoded memory characteristics and sampled memory characteristics corresponding to each memory characteristic, The above generating process is, The process of calculating a classification score for each encoded memory feature; A process of thresholding a memory characteristic corresponding to each encoded memory characteristic based on one or more predefined thresholds; and A computer-implemented method comprising the step of embedding one or more thresholded memory characteristics to generate sampled memory characteristics corresponding to each memory characteristic.

7. In paragraph 6, The above thresholding process is, A process of obtaining one or more sampling indices by binarizing the classification score based on one or more predefined thresholds; and A computer-implemented method comprising a process of selecting one of the memory characteristics and class-less embeddings corresponding to each encoded memory characteristic as a thresholded memory characteristic based on the value of each sampling index.

8. In paragraph 1, The above prediction process is, A process of applying an object query to a memory-guided decoder to obtain an output object query having improved semantic information about the current frame and information about classes associated with the current frame; and A computer-implemented method comprising a process of predicting a bounding box and a class of an object from the output object query.

9. In paragraph 8, The above memory-guided decoder comprises one or more decoder blocks, Each decoder block, A first attention layer that generates a first object query based on self-attention for the input object query; A second attention layer that generates a second object query based on cross-attention between the first object query and the encoded image feature map; and A computer-implemented method, comprising a third attention layer that generates a third object query based on cross-attention between the second object query and the sampled memory feature map.

10. In paragraph 9, The sampled memory characteristic map above includes sampled memory characteristics corresponding to each memory characteristic, The process of obtaining the above output object query is: A process of combining memory embeddings indicating the locations of memory features corresponding to each sampled memory feature with the sampled memory feature map and providing them to the third attention layer. A computer-implemented method comprising:

11. A memory for storing instructions; and at least one processor, At least one processor executes the instructions, Encode an image feature map extracted from the current frame and a pre-stored memory feature map, wherein the memory feature map includes a predefined number of memory features for each of the predictable classes. Generate a sampled memory characteristic map based on the classification scores derived from the encoded memory characteristic map and the memory characteristic map, Based on the encoded image feature map and the sampled memory feature map, the bounding box and class of the object in the current frame are predicted, A device for updating the memory feature map based on the encoded image feature map.

12. In paragraph 11, The at least one processor, in updating the memory characteristic map, Extracting one or more instance features from the encoded image feature map based on the predicted bounding box or the pre-specified ground truth (GT) bounding box, and updating one or more target memory features corresponding to the extracted instance features in the memory feature map. A device wherein each target memory characteristic is selected from among the memory characteristics corresponding to the same class as each instance characteristic.

13. In paragraph 12, One or more of the above target memory characteristics are: A device that is updated based on linear interpolation between each instance characteristic and a target memory characteristic corresponding to each instance characteristic.

14. In paragraph 12, The above target memory characteristics are: A device, wherein among the memory characteristics corresponding to the same class as each instance characteristic, the memory characteristic having the greatest correlation with each instance characteristic.

15. In paragraph 11, The at least one processor, in encoding the image feature map and the memory feature map, A device for concatenating the image feature map and the memory feature map; and obtaining the encoded image feature map containing the spatiotemporal context of all frames observed prior to the current frame and the encoded memory feature map containing information about the class of an object in the current frame based on self-attention to the concatenated feature map.

16. In paragraph 11, The encoded memory characteristic map and the sampled memory characteristic map each include encoded memory characteristics and sampled memory characteristics corresponding to each memory characteristic, The at least one processor, in generating the sampled memory characteristic map, For each encoded memory feature, a classification score is calculated, Based on one or more predefined thresholds, thresholding a memory characteristic corresponding to each encoded memory characteristic, A device that embeds one or more thresholded memory characteristics and generates sampled memory characteristics corresponding to each memory characteristic.

17. In paragraph 16, At least one processor, in thresholding the memory characteristics, By binarizing the classification score based on one or more predefined thresholds, one or more sampling indices are obtained, A device that selects one of the memory characteristics and class-less embeddings corresponding to each encoded memory characteristic as a thresholded memory characteristic based on the value of each sampling index.

18. In paragraph 11, The at least one processor, in predicting the bounding box and class of the object, Applying an object query to a memory-guided decoder, an output object query is obtained that has enhanced semantic information about the current frame and information about the class associated with the current frame, A device for predicting the bounding box and class of the object from the above output object query.

19. In paragraph 18, The above memory-guided decoder comprises one or more decoder blocks, Each decoder block, A first attention layer that generates a first object query based on self-attention for the input object query; A second attention layer that generates a second object query based on cross-attention between the first object query and the encoded image feature map; and A device comprising a third attention layer that generates a third object query based on cross-attention between the second object query and the sampled memory feature map.

20. A non-transitory computer-readable storage medium having stored thereon a command, wherein the command, when executed by the computer, causes the computer to: A process of encoding an image feature map extracted from the current frame and a pre-stored memory feature map, wherein the memory feature map includes a predefined number of memory features for each of one or more predictable classes; A process of generating a sampled memory characteristic map based on classification scores derived from the encoded memory characteristic map and the memory characteristic map; A process of predicting a bounding box and a class of an object within the current frame based on the encoded image feature map and the sampled memory feature map; and A process of updating the memory feature map based on the encoded image feature map. A non-transitory computer-readable storage medium that enables execution.

Images