Device and method for expanding class decisions boundary for video classification
The class decision boundary expansion device addresses data imbalance in video classification by extracting and enhancing features using prototypes and mathematical operations, enhancing classification accuracy and reducing incorrect labeling.
Patent Information
- Application Number
- KR1020230164173
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-11-23
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2043-11-23
AI Technical Summary
Conventional video classification technologies face challenges in effectively learning minority classes due to data imbalance, where existing data augmentation methods are not specialized for the target video dataset and can lead to incorrect labeling.
A class decision boundary expansion device and method that extracts features using a head prototype and multiple tail prototypes, performs extrapolation and interpolation to calculate final class features, and utilizes self-attention and codebook-based learning to remove noise frames and enhance feature extraction.
The method effectively reduces data imbalance by expanding the decision boundary for minority classes, improving classification accuracy and reducing incorrect labeling.
Smart Images

Figure 112023130881730-PAT00013_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a class decision boundary expansion device and method, and more specifically, to a class decision boundary expansion device and method capable of augmenting minority class data for training a video classification model. Background Technology
[0002] Video classification technology is a technique that extracts features to classify classes from various information present within a video and performs class classification using these features. Conventional video classification technologies include, for example, methods that import a pre-trained model, extract features from every frame of the video, and conduct further training.
[0003] One of the major challenges in video classification technology is the issue of data imbalance. Data imbalance refers to a situation where the proportions of data to be learned are uneven and skewed; such imbalance can lead to problems where minority classes are not learned effectively compared to majority classes.
[0004] Existing methods to address this include developing techniques to augment data primarily with minority classes by mixing frames within the video or mixing them with external videos. However, these methods are not specialized for extracting these features because they are not trained on the target video dataset, and there are still problems such as the possibility of incorrect labeling for each frame. Prior art literature
[0005] Korean Published Patent Application No. 10-2023-0032317 ("Method and System for Detecting Video Semantic Sections", Date of Publication: March 7, 2023) The problem to be solved
[0006] The present invention aims to provide a class determination boundary expansion device and method that extracts features for distinguishing classes within a target video and reduces data imbalance through outer and inner addition of the features. means of solving the problem
[0007] According to one embodiment of the present invention, a class determination boundary expansion device may include a feature extraction module for class classification within a video, which extracts the average of all frames within the video as a head prototype to extract a head class feature based on the head prototype and all frames, extracts the average of a combination of some frames within the video as a first tail prototype to extract a first tail class feature based on the first tail prototype and said combination of some frames, extracts the average of a combination of some frames within the video as a second tail prototype to extract a second tail class feature based on the second tail prototype and said combination of some frames, extracts the average of a combination of some frames within the video and a combination of other frames as a second tail prototype to extract a second tail class feature based on the second tail prototype and a combination of other frames; an extrapolation module that performs extrapolation based on the first tail class feature and the second tail class feature to calculate a final tail class feature; and an interpolation module that performs interpolation based on the head class feature and the final tail class feature.
[0008] According to one embodiment, the feature extraction module may include a first aggregator that extracts features using a self-attention mechanism and a second aggregator that extracts features using codebook-based learning, and may be characterized by removing noise frames through the first aggregator and the second aggregator, extracting features respectively, and concatenating each extracted feature to finally extract a head class feature, a first tail class feature, and a second tail class feature.
[0009] According to one embodiment, the first aggregator may be characterized by calculating the similarity between a prototype and a frame in a video and extracting features based on the similarity.
[0010] According to one embodiment, the second aggregator may be characterized by utilizing a VLAD.
[0011] According to one embodiment, the extrapolation module is, (Here, w is a value between 1 and 2 inclusive, and Z p1 It is the 1st Tail Class Feature, Z p2 It can be characterized by performing extrapolation using the second tail class feature) expression.
[0012] According to one embodiment, the interpolation module is, (Here, λ is a value between 0 and 1, and Z head silver head class feature, Z tail It can be characterized by performing interpolation using the final tail class feature expression.
[0013] According to another embodiment of the present invention, a class determination boundary extension method may include a feature extraction step for class classification within a video, wherein the average of all frames within the video is extracted as a head prototype to extract a head class feature based on the head prototype and all frames, the average of a combination of some frames within the video is extracted as a first tail prototype to extract a first tail class feature based on the first tail prototype and the combination of some frames, and the average of a combination of some frames within the video and a combination of other frames is extracted as a second tail prototype to extract a second tail class feature based on the second tail prototype and the combination of other frames; an extrapolation step for calculating a final tail class feature by performing extrapolation based on the first tail class feature and the second tail class feature; and an interpolation step for performing interpolation based on the head class feature and the final tail class feature.
[0014] According to one embodiment, the feature extraction step may be characterized by utilizing a first aggregator that extracts features using a self-attention mechanism and a second aggregator that extracts features using codebook-based learning, removing noise frames through the first aggregator and the second aggregator, extracting each of the features, and concatenating the extracted features to finally extract a head class feature, a first tail class feature, and a second tail class feature.
[0015] According to one embodiment, the first aggregator may be characterized by calculating the similarity between a prototype and a frame in a video and extracting features based on the similarity.
[0016] According to one embodiment, the second aggregator may be characterized by utilizing a VLAD.
[0017] According to one embodiment, the extrapolation step is, (Here, w is a value between 1 and 2 inclusive, and Z p1 It is the 1st Tail Class Feature, Z p2 It can be characterized by performing extrapolation using the second tail class feature) expression.
[0018] According to one embodiment, the interpolation step is, (Here, λ is a value between 0 and 1, and Z head silver head class feature, Z tail It can be characterized by performing interpolation using the final tail class feature expression. Effects of the invention
[0019] The imbalance between majority class and minority class data can be reduced through the class decision boundary expansion device and method described above. Brief explanation of the drawing
[0020] FIG. 1 is a block diagram of a class determination boundary expansion device according to one embodiment. FIGS. 2 and FIGS. 3 are drawings illustrating a feature extraction process according to one embodiment. FIG. 4 is a block diagram of a first aggregator according to one embodiment. FIG. 5 is a block diagram of a second aggregator according to one embodiment. FIG. 6 is a diagram illustrating an extrapolation process according to one embodiment. FIG. 7 is a diagram illustrating an interpolation process according to one embodiment. FIG. 8 is a flowchart of a class determination boundary expansion method according to one embodiment. Specific details for implementing the invention
[0021] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.
[0022] The terms used in this specification will be briefly explained, and the invention will be described in detail.
[0023] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.
[0024] Throughout the specification, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "part," "module," and "unit" used in the specification refer to a unit that processes at least one function or operation and may be implemented as software, hardware components such as FPGAs or ASICs, or a combination of software and hardware. However, the terms "part," "module," and "unit" are not limited to software or hardware. "Part," "module," and "unit" may be configured to reside in an addressable storage medium or configured to run one or more processors. Accordingly, as an example, terms such as "part," "module," and "unit" include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.
[0025] Below, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the invention. Additionally, parts of the drawings that are irrelevant to the description are omitted to clearly explain the invention.
[0026] Terms including ordinal numbers, such as "first," "second," etc., may be used to describe various components, but the components are not limited by the terms. The terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of multiple related items or any one of the multiple related items.
[0028] Hereinafter, the device and method for class determination boundary expansion for video classification according to the present invention will be described with reference to the drawings.
[0029] The class boundary expansion device and method for video classification according to the present invention are intended to resolve the problem of imbalance between classes included in a video, and may be one of the preprocessing steps for augmenting data of minority classes that have relatively insufficient data. The class determination boundary expansion device of the present invention can bring about the effect of augmenting data of minority classes by expanding the boundary determining minority classes in the embedding space.
[0030] The class determination boundary expansion device and method of the present invention, for class classification within a video, can extract the average of all frames within the video as a head prototype to extract a head class feature based on the head prototype and all frames, extract the average of a combination of some frames among all frames as a first tail prototype to extract a first tail class feature based on the first tail prototype and the combination of some frames, and extract the average of a combination of some frames different from the said combination of some frames among all frames as a second tail prototype to extract a second tail class feature based on the second tail prototype and the combination of other some frames. Additionally, the class determination boundary expansion device and method can calculate a final tail class feature by performing extrapolation based on the first tail class feature and the second tail class feature. Additionally, the class determination boundary expansion device and method can perform interpolation based on the head class feature and the final tail class feature.
[0031] FIG. 1 is a block diagram of the class determination boundary expansion device (1) of the present invention.
[0032] The class determination boundary expansion device (1) of the present invention may include a collection unit (10), a storage unit (20), an output unit (30), and a processor (40).
[0033] The collection unit (10) can collect various data required for the class determination boundary expansion device (1). For example, the collection unit (10) can collect data related to the video to be classified.
[0034] The collection unit (10) can collect data according to user input, or it can collect information from the outside through communication.
[0035] The storage unit (20) can store information collected or generated from the collection unit (10) and the processor (40), etc. Specifically, the storage unit (20) can temporarily or non-temporarily store data collected from the collection unit (10), data required during the processing of the processor (40), data generated during the processing of the processor (40), or data resulting from the processing of the processor (40). For example, the storage unit (20) can store video data subject to classification collected from the collection unit (10), and can store head class features and tail class features generated by the processor (40), which will be described later.
[0036] The storage unit (20) may include at least one of a main memory and an auxiliary memory. The main memory may be implemented using a semiconductor storage medium such as ROM and / or RAM, for example, and the auxiliary memory may be implemented based on a device capable of storing data permanently or semi-permanently, such as a flash memory device (Solid State Drive (SSD), etc.), an SD (Secure Digital) card, a hard disk drive (HDD), a compact disc, a DVD, or a laser disc.
[0037] The output unit (30) can output the contents of the output according to the operation of the prime class boundary expansion device. The output unit (30) can receive information and signals from the storage unit (20) and / or the processor (40) and transmit them to other internal components and / or external components.
[0038] The output unit (30) can visually or audibly output the output of the processor (40) described later (e.g., image frames, class determination boundary expansion results, class classification results, etc.) to the outside. The output unit (30) may include, for example, a display, a printer device, a speaker device, a video output terminal, a data input / output terminal, but is not limited thereto. If necessary, the output unit (30) may be provided as an integral part of the input unit.
[0039] The processor (40) can exchange information with the collection unit (10), the storage unit (20), and the output unit (30). For example, it can receive information such as a video to be classified and extracted features (head class features, tail class features) from the collection unit (10) and / or the storage unit (20), and can also transmit information regarding the extended class determination boundary to the output unit (30).
[0040] The processor (40) may include one or more electronic devices capable of performing various computational and control processing, such as a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller unit (MCU), an application processor (AP), an electronic control unit (ECU), a microprocessor (Micom), or other such devices. These processing or control devices may be implemented, for example, by using one or more semiconductor chips, circuits, or related components alone or in combination. However, they are not limited thereto and may be implemented by various devices capable of processing information.
[0041] For class classification within a video, the processor (40) may extract the average of all frames within the video as a head prototype to extract a head class feature based on the head prototype and all frames, extract the average of a combination of some frames within the video as a first tail prototype to extract a first tail class feature based on the first tail prototype and the combination of some frames, and extract the average of a combination of some frames different from the above combination of some frames within the video as a second tail prototype to extract a second tail class feature based on the second tail prototype and the combination of some frames. Additionally, the processor (40) may perform extrapolation based on the first tail class feature and the second tail class feature to produce a final tail class feature. Additionally, the processor (40) may perform interpolation based on the head class feature and the final tail class feature.
[0042] The processor (40) may include a feature extraction module, an extrapolation module (430), and an interpolation module (450).
[0043] Referring to Figures 2 and 3, the feature extraction module (410) will be described.
[0044] For class classification within a video, the feature extraction module (410) can extract the average of all frames within the video as a head prototype, extract a head class feature based on the head prototype and all frames, extract the average of a combination of some frames within the video as a first tail prototype, extract a first tail class feature based on the first tail prototype and the combination of some frames, and extract the average of a combination of some frames different from the combination of some frames within the video as a second tail prototype, extract a second tail class feature based on the second tail prototype and the combination of some frames different frames.
[0045] The feature extraction module (410) can extract the average of frames within the video as a prototype for video class classification. That is, the feature extraction module (410) can calculate the mean value of all or some frames within the video and extract it as a prototype. The feature extraction module (410) can extract features of the corresponding video using the extracted prototype.
[0046] The feature extraction module (410) can extract the average of all frames in the video as a head prototype. Additionally, the feature extraction module (410) can extract the average of some frame combinations among all frames as a first tail prototype. Additionally, the feature extraction module (410) can extract the average of some frame combinations different from the aforementioned some frame combinations as a second tail prototype.
[0047] According to one embodiment, the feature extraction module (410) can extract features by using an attention mechanism to overlay information of all frame features on the extracted prototype. This may be identical or similar to the feature extraction method of the first aggregator described later, and may become clearer by referring to the description of the first aggregator described later.
[0048] The feature extraction module (410) can extract head class features based on the head prototype and all frames within the video. Here, the head class may refer to a class in which data exists relatively abundantly as a result of the class classification by the feature extraction module (410). Additionally, the head class feature may refer to a feature for classifying such a head class. The feature extraction module (410) can extract a feature for extracting the head class as a head class feature based on both the head prototype and all frames existing within the video. To explain using the example of FIG. 2, if there are 20 frames within the video, the feature extraction module (410) can extract a head class feature based on both the head prototype, which is the average of these 20 frames, and all 20 frames.
[0049] Additionally, the feature extraction module (410) can extract a first tail class feature based on a first tail prototype and a combination of some frames that formed the basis of the first tail prototype. Here, the tail class refers to a class in which data exists in relatively small numbers following the class classification process of the feature extraction module (410), and may be a concept opposite to the aforementioned head class. Additionally, the tail class feature may refer to a feature for classifying such a tail class.
[0050] The feature extraction module (410) can extract a first tail prototype, which is the average of some frames among all frames present in the video, and a feature for extracting a tail class based on said some frames as a first tail class feature.
[0051] According to one embodiment, the feature extraction module (410) can extract a first tail class feature by selecting only an arbitrary number of frames among some frames containing a tail class. Specifically, the feature extraction module (410) can calculate some frames related to the tail class among the frames in the video, select an arbitrary number of frames among these some frames to extract a first tail prototype, and extract a first tail class feature based on the first tail prototype and the arbitrary number of frames that formed the basis thereof. To explain with reference to the example of FIG. 2, as described above, when there are 20 frames in the video, the feature extraction module (410) can select a combination composed of an arbitrary number of frames among some frames related to the tail class among these, and extract a first tail class feature based on the first tail prototype which is the average of these frames and the combination composed of the arbitrary number of frames.
[0052] Additionally, the feature extraction module (410) can extract a second tail class feature based on a combination of frames different from the combination of frames that formed the basis of the second tail prototype and the second tail prototype. The feature extraction module (410) can extract a feature for extracting a tail class as a second tail class feature based on a combination of frames that formed the basis of the first tail class extraction among the frames existing in the video, which is different from the combination of frames that formed the basis of the first tail class extraction. Therefore, even though the first tail class feature and the second tail class feature are features for classifying the same tail class, they can be extracted differently from each other because the frame combinations that form the basis are different.
[0053] According to one embodiment, similar to the case of extracting the first tail class, the feature extraction module (410) may extract a second tail prototype by selecting only an arbitrary number of frames among some frames containing the tail class, and may extract a second tail class feature based on the extracted second tail prototype and any multiple frames that formed the basis thereof. To explain with reference to the example of FIG. 2, as described above, when there are 20 frames in the video, the feature extraction module (410) may select an arbitrary multiple frame combination different from the combination of some frames for extracting the first tail class among some frames related to the tail class, and extract a second tail class feature based on a combination composed of the second tail prototype which is the average of these frames and the other arbitrary multiple frames.
[0054] According to one embodiment, the feature extraction module (410) can remove noise frames for class classification within a video. The video may consist of multiple frames, and the feature extraction module (410) can remove noise frames that are less related to the class among the multiple frames of the video to be classified in order to extract features necessary for class classification. The feature extraction module (410) can improve class classification performance by removing noise frames that are less related to the class. For example, the feature extraction module (410) can remove noise frames using an attention mechanism.
[0055] According to one embodiment, the process of removing noise frames and extracting each class feature in the feature extraction module (410) can be performed simultaneously. That is, while the feature extraction module (410) extracts features for class classification, it can simultaneously remove noise frames that are not related to each class and are not helpful for classification. When removing noise frames, the feature extraction module (410) can perform feature extraction for each class using only the remaining frames from which the noise frames have been removed.
[0056] Referring to FIG. 3, the first aggregator and the second aggregator will be described.
[0057] According to one embodiment, the feature extraction module (410) includes a first aggregator utilizing a self-attention mechanism and a second aggregator utilizing codebook-based learning, thereby removing noise frames through the first aggregator and the second aggregator, extracting features respectively, and concatenating the extracted features to finally extract a head class feature, a first tail class feature, and a second tail class feature. That is, the feature extraction module (410) can finally extract a head class feature by concatenating the head class feature extracted based on a self-attention mechanism through the first aggregator and the head class feature extracted based on codebook-based learning through the second aggregator. In addition, similarly, the feature extraction module (410) can finally extract the first tail class feature by concatenating the first tail class feature extracted based on the self-attention mechanism through the first aggregator and the first tail class feature extracted based on codebook-based learning through the second aggregator. In addition, similarly, the feature extraction module (410) can finally extract the second tail class feature by concatenating the second tail class feature extracted based on the self-attention mechanism through the first aggregator and the second tail class feature extracted based on codebook-based learning through the second aggregator.
[0058] FIG. 3 is a diagram illustrating the removal of noise frames and the extraction of features in each aggregator according to one embodiment. Referring to the example in FIG. 3, it can be seen that the first aggregator extracts features for class classification within a video based on a self-attention mechanism, and the second aggregator extracts features based on codebook-based learning. In the first aggregator, it can be seen that high activation is applied to frames related to the class, and in the second aggregator, it can be seen that multiple codebooks listed on each vertical axis apply activation only to similar frames related to the class for frames listed horizontally.
[0059] FIG. 4 is a block diagram of a first aggregator according to one embodiment.
[0060] According to one embodiment, the first aggregator can extract features by inputting a prototype into the query and frame input features into the key and value. For example, a head prototype can be input into the query and the entire frame can be input into the key and value to extract a head class feature. Additionally, a first tail prototype can be input into the query and a combination of some frames used to extract the first tail prototype can be input into the key and value to extract a first tail class feature. Additionally, a second tail prototype can be input into the query and a combination of some frames used to extract the second tail prototype can be input into the key and value to extract a second tail class feature.
[0061] According to one embodiment, the first aggregator can calculate the similarity between a prototype and a frame in the video based on a self-attention mechanism and extract features based on the similarity. For example, after calculating the similarity between the prototype and each frame in the video, the first aggregator can extract features by making frames with high similarity reflect more information. On the other hand, frames with low similarity can extract features by making frames with low similarity reflect less information.
[0062] According to one embodiment, the first aggregator may be characterized by removing noise frames based on the similarity between a prototype and a plurality of frames. The first aggregator receives all frames in the video as input, extracts the average of all frames as a context vector, and then compares the similarity between the frame and the context vector to remove frames with a similarity below a preset standard as noise frames. FIG. 5 is a block diagram of a second aggregator according to one embodiment.
[0063] According to one embodiment, the second aggregator may be characterized by utilizing VLAD. However, the second aggregator is not limited to VLAD and may perform noise frame removal and feature extraction using other prototype generation methods and feature compression methods.
[0064] Referring to FIG. 6, the extrapolation module (430) will be described.
[0065] The extrapolation module (430) can calculate the final tail class feature by performing extrapolation based on the first tail class feature and the second tail class feature. Here, extrapolation refers to an extrapolation method, which can mean a method of predicting and obtaining data that was not actually obtained using existing acquired data. Extrapolation can be used to find possible values located outside of two specific values. The extrapolation module (430) can expand the surrounding embedding space of the tail class through extrapolation.
[0066] According to one embodiment, the extrapolation module (430) can produce a final tail class feature by performing extrapolation according to the following mathematical formula 1.
[0067]
[0068] Here, w is a value between 1 and 2 inclusive, and Z p1 It is the 1st Tail Class Feature, Z p2 can mean the second tail class feature.
[0069] FIG. 6 is a diagram visualizing the performance of extrapolation in the extrapolation module (430) according to one embodiment. According to the embodiment of FIG. 6, as extrapolation is performed, it can be seen that the periphery of the tail class is expanded.
[0070] Referring to FIG. 7, the interpolation module (450) will be described.
[0071] The interpolation module (450) can perform interpolation based on the head class feature and the final tail class feature. Here, interpolation refers to interpolation, and like extrapolation, it can refer to a method of predicting and obtaining data that was not actually obtained using existing acquired data. Unlike extrapolation, interpolation can be used to find possible values located inside two specific values.
[0072] The interpolation module (450) can perform interpolation on the extrapolation module (430) after performing extrapolation to adjust the embedding space with other classes. This can serve to create class decision boundaries in empty spaces on the embedding space by linearly mixing features and labels with prototype features of other videos. At this time, to prevent the decision boundary of the tail class from being formed narrowly due to the small amount of data in the tail class, this process can be performed by mixing labels with a larger weight for the labels of the tail class when mixing labels.
[0073] According to one embodiment, the interpolation module (450) can perform interpolation according to the following mathematical formula 2.
[0074]
[0075] Here, λ is a value between 0 and 1, and Z head silver head class feature, Z tail can mean the final tail class feature.
[0076] FIG. 7 is a diagram visualizing the interpolation being performed in an interpolation module (450) according to one embodiment. According to the embodiment of FIG. 7, as interpolation is performed, it can be seen that the decision boundary of the tail class is expanded.
[0078] The class determination boundary expansion method of the present invention will be explained below with reference to FIG. 8.
[0079] FIG. 8 is a flowchart of a class determination boundary expansion method according to one embodiment of the present invention.
[0080] According to FIG. 8, the class determination boundary extension method of the present invention may include a class feature extraction step (S810) for class classification within a video, wherein the average of all frames within the video is extracted as a head prototype to extract a head class feature based on the head prototype and all frames, the average of a combination of some frames within the video is extracted as a first tail prototype to extract a first tail class feature based on the first tail prototype and the combination of some frames, and the average of a combination of some frames different from the combination of some frames within the video is extracted as a second prototype to extract a second tail class feature based on the second tail prototype and the combination of some frames. Additionally, the class determination boundary extension method may include an extrapolation step (S820) for calculating a final tail class feature by performing extrapolation based on the first tail class feature and the second tail class feature. Additionally, the class determination boundary extension method may include an interpolation step (S830) that performs interpolation based on a head class feature and a final tail class feature.
[0081] According to one embodiment, the extrapolation step is, (Here, w is a value between 1 and 2 inclusive, and Z p1 It is the 1st Tail Class Feature, Z p2 It can be characterized by performing extrapolation using the second tail class feature) expression.
[0082] According to one embodiment, the interpolation step is, (Here, λ is a value between 0 and 1, and Z head silver head class feature, Z tail It can be characterized by performing interpolation using the final tail class feature expression.
[0083] According to one embodiment, the feature extraction step may be characterized by utilizing a first aggregator that extracts features using a self-attention mechanism and a second aggregator that extracts features using codebook-based learning, removing noise frames through the first aggregator and the second aggregator, extracting each of the features, and concatenating the extracted features to finally extract a head class feature, a first tail class feature, and a second tail class feature.
[0084] According to one embodiment, the first aggregator may be characterized by calculating the similarity between a prototype and a frame in a video and extracting features based on the similarity.
[0085] According to one embodiment, the second aggregator may be characterized by utilizing a VLAD.
[0087] Those skilled in the art related to the embodiments of the present invention will understand that they may be implemented in modified forms without departing from the essential nature of the description. Therefore, the disclosed methods should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the claims, not by the detailed description of the invention, and all variations within the scope of the claims should be interpreted as being included within the scope of the present invention. Explanation of the symbols
[0089] 1: Class Decision Boundary Expansion Device 10: Collection Department 20: Storage section 30: Output section 40: Processor 410: Feature Extraction Module 430: Extrapolation Module 450: Interpolation module
Claims
Claim 1 For class classification within a video, a feature extraction module that extracts the average of all frames within the video as a head prototype to extract a head class feature based on the head prototype and the all frames, extracts the average of a combination of some frames among the all frames as a first tail prototype to extract a first tail class feature based on the first tail prototype and the combination of some frames, extracts the average of a combination of some frames different from the combination of some frames among the all frames as a second tail prototype to extract a second tail class feature based on the second tail prototype and the combination of other some frames; an extrapolation module that performs extrapolation based on the first tail class feature and the second tail class feature to produce a final tail class feature; and an interpolation module that performs interpolation based on the head class feature and the final tail class feature; wherein the extrapolation module (Here, w is a value between 1 and 2 inclusive, and Z p1 It is the 1st Tail Class Feature, Z p2 A class determination boundary expansion device characterized by performing extrapolation using a second tail class feature expression. Claim 2 A class determination boundary expansion device according to claim 1, wherein the feature extraction module comprises: a first aggregator that extracts features using a self-attention mechanism; and a second aggregator that extracts features using codebook-based learning; wherein the noise frame is removed through the first aggregator and the second aggregator, the features are each extracted, and the extracted features are concatenated to finally extract the head class feature, the first tail class feature, and the second tail class feature. Claim 3 A class determination boundary expansion device according to claim 2, wherein the first aggregator calculates the similarity between the prototype and the frame in the video and extracts the feature based on the similarity. Claim 4 In paragraph 2, the class determination boundary expansion device is characterized in that the second aggregator utilizes VLAD. Claim 5 delete Claim 6 In paragraph 1, the interpolation module is, (Here, λ is a value between 0 and 1, and Z head silver head class feature, Z tail A class determination boundary expansion device characterized by performing interpolation using a final tail class feature expression. Claim 7 The present invention relates to a method for expanding class determination boundaries performed by a processor, comprising: a feature extraction step for class classification within a video, wherein the average of all frames within the video is extracted as a head prototype to extract a head class feature based on the head prototype and the all frames, the average of a combination of some frames among the all frames is extracted as a first tail prototype to extract a first tail class feature based on the first tail prototype and the combination of some frames, and the average of a combination of some frames different from the combination of some frames among the all frames is extracted as a second tail prototype to extract a second tail class feature based on the second tail prototype and the combination of other some frames; an extrapolation step for calculating a final tail class feature by performing extrapolation based on the first tail class feature and the second tail class feature; and an interpolation step for performing interpolation based on the head class feature and the final tail class feature; wherein the extrapolation step comprises (Here, w is a value between 1 and 2 inclusive, and Z p1 It is the 1st Tail Class Feature, Z p2 A class determination boundary expansion method characterized by performing extrapolation using a second tail class feature expression. Claim 8 A method for expanding class determination boundaries according to claim 7, wherein the feature extraction step utilizes a first aggregator that extracts features using a self-attention mechanism and a second aggregator that extracts features using codebook-based learning, removes noise frames through the first aggregator and the second aggregator, extracts the features respectively, and connects the extracted features to finally extract the head class feature, the first tail class feature, and the second tail class feature. Claim 9 A method for expanding class determination boundaries according to claim 8, wherein the first aggregator calculates similarity between the prototype and the frame in the video and extracts the feature based on the similarity. Claim 10 In claim 8, the second aggregator is characterized by utilizing VLAD in a class determination boundary expansion method. Claim 11 delete Claim 12 In paragraph 7, the above interpolation step is, (Here, λ is a value between 0 and 1, and Z head silver head class feature, Z tail A class determination boundary expansion method characterized by performing interpolation using a final tail class feature expression.
