Video continual learning method and apparatus

Abstract

This video continual learning method by which a computing device sequentially learns a plurality of video sessions is performed. The computing device uses a plurality of weights including weights selected in the previous video session from among the plurality of video sessions in a forward path of a neural network for learning the current video session from among the plurality of video sessions, freezes the selected weights in a backward path of the neural network for learning the current video session, updates the weights that exclude the selected weights from among the plurality of weights, acquires a binary mask for selecting some weights from among the plurality of weights on the basis of a weight score of each weight, and searches for a sub-network of the neural network for the current video session with the binary mask. Here, the neural network includes an NIR model and a Fourier sub-network operator layer, and the Fourier sub-network operator layer includes a real tensor and an imaginary tensor of a Fourier transform of a periodic function.

Description

Video continuous learning method and device

[0001] The disclosure relates to a video continuous learning method and device.

[0002] Neural implicit representation (NIR, or implicit neural representation, INR, hereafter simply "NIR") aims to represent complex data, such as videos or three-dimensional (3D) objects, using continuous functions learned by neural networks. Instead of explicitly representing data points, NIR models compress high-dimensional data within a low-dimensional embedding space. This process enables efficient data storage, compression, and synthesis. However, when compressing multiple data sets, each high-dimensional data point must be encoded by the neural network, which increases linear memory requirements. To address this issue, methods have been proposed that combine different videos into a single video format and then reduce the model size using techniques such as weight pruning and quantization post-training. While effective for current data compression, these methods have limitations, limiting the model's ability to adapt to new videos as they are added.

[0003] Continual learning is a learning paradigm in which a model learns through multiple sequential sessions. This mimics the human cognitive ability to gradually learn new concepts over a lifetime without degrading previously acquired skills. However, incremental training of NIR is challenging due to a phenomenon known as catastrophic forgetting, in which the model loses implicit representations learned from past sessions while encoding newly arrived videos. This problem is particularly critical because the neural representation of video encodes and reconstructs the target data stream based on frame indices. This is because distributional disparities between holistic videos and their individual frames can easily degrade the model's generative ability while training to continuously encode new videos. Furthermore, the compression step of the neural representation makes it difficult for the model to transfer to future tasks. Various approaches have been proposed to address catastrophic forgetting during continuous learning, but existing approaches are unsuitable for video data in continuous learning due to the significant memory and computational overhead of storing and revisiting high-dimensional samples. Although existing approaches provide solutions to prevent forgetting, they are not suitable for sequential complex video processing because they reuse some or all of the adaptive parameters without fine-grained discretization operations.

[0004] Certain embodiments may provide a video continuous learning method and device capable of preventing the threat of catastrophic forgetting.

[0005] According to some embodiments, a video continuous learning method for sequentially learning a plurality of video sessions, performed by a computing device, may be provided. The video continuous learning method may include: using a plurality of weights including weights selected from a previous video session among the plurality of video sessions in a forward pass of a neural network for learning a current video session among the plurality of video sessions; freezing the selected weights in a backward pass of the neural network for learning the current video session and updating weights excluding the selected weights among the plurality of weights; obtaining a binary mask for selecting some weights among the plurality of weights based on a weight score of each weight; and finding a subnetwork of the neural network for the current video session using the binary mask. The neural network may include a NIR model and a Fourier subnetwork operator layer, and the Fourier subnetwork operator layer may include a real tensor and an imaginary tensor of a Fourier transform of a periodic function.

[0006] According to some embodiments, a video continuous learning device may include a memory storing one or more instructions, and a processor. The processor may execute the instructions to use a plurality of weights including weights selected from a previous video session among the plurality of video sessions in a forward pass of a neural network for learning a current video session among the plurality of video sessions, freeze the selected weights in a backward pass of the neural network for learning the current video session, update weights excluding the selected weights among the plurality of weights, obtain a binary mask for selecting some of the weights among the plurality of weights based on the weight score of each weight, and find a subnetwork of the neural network for the current video session using the binary mask. The neural network may include an NIR model and a Fourier subnetwork operator layer, and the Fourier subnetwork operator layer may include a real tensor and an imaginary tensor of a Fourier transform of a periodic function.

[0007] According to some embodiments, a computer program may be provided that is executed by a computing device and stored in a recording medium. The computer program may cause the computing device to perform the steps of: using a plurality of weights including weights selected from a previous video session among the plurality of video sessions in a forward pass of a neural network for learning a current video session among a plurality of video sessions; freezing the selected weights in a backward pass of the neural network for learning the current video session and updating weights among the plurality of weights excluding the selected weights; obtaining a binary mask for selecting some weights among the plurality of weights based on a weight score of each weight; and finding a subnetwork of the neural network for the current video session using the binary mask. The neural network may include a NIR model and a Fourier subnetwork operator layer, and the Fourier subnetwork operator layer may include a real tensor and an imaginary tensor of a Fourier transform of a periodic function.

[0008] FIG. 1 is a diagram illustrating the concept of a video continuous learning method according to some embodiments.

[0009] FIG. 2 is a drawing illustrating a video continuous learning device according to some embodiments.

[0010] FIG. 3 is a diagram illustrating a process of a video continuous learning method according to some embodiments.

[0011] Figure 4 is a flowchart of a video continuous learning method according to some embodiments.

[0012] FIG. 5 is a block diagram illustrating a computing device according to some embodiments.

[0013] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement the invention. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description, and similar parts are designated with similar reference numerals throughout the specification.

[0014] In the description below, expressions written in the singular may be interpreted as singular or plural, unless explicit expressions such as “one” or “single” are used.

[0015] In the description below, terms including ordinal numbers, such as "first" and "second," may be used to describe various components, but these components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component."

[0016] In the flowchart described with reference to the drawing, the order of operations may be changed, several operations may be merged, some operations may be split, and certain operations may not be performed.

[0017] FIG. 1 is a diagram illustrating the concept of a video continuous learning method according to some embodiments.

[0018] Referring to FIG. 1, a video continuous learning method according to some embodiments can train a sub-network for a current session (session s) while maintaining the weights selected in a previous video session (session (s-1)). In FIG. 1, the edges (weights) represented by the bold dotted lines and the bold solid lines are the weights selected in the previous session (s-1), and the edges (weights) represented by the bold solid lines are the weights (reused weights) that are frozen while being reused in the current session s among the weights selected in the previous session (s-1).

[0019] To progressively improve neural representations for complex sequential videos, the video continuous learning method according to some embodiments can provide a novel NIR method, called progressive Fourier neural representation (PFNR), which identifies and utilizes lottery tickets (i.e., weights) in frequency space. To this end, the video continuous learning method can decompose the NIR into sine and cosine components (i.e., real and imaginary parts), and then define a Fourier subnetwork operator (FSO) that selectively identifies the most effective lottery tickets (weights) for representing complex periodic signals. In fact, given a backbone and FSO architecture, the method can continuously learn to identify input adaptive subnetwork modules and encode each new video into those modules during sequential training sessions.

[0020] In some embodiments, PFNR can utilize a parametric score function, or weight score. This function learns to generate binary masks for real and imaginary components, and by selecting the weights corresponding to the top percentage of rank scores in each training session, it can identify an adaptive substructure for video encoding. This allows PFNR to discover the optimal subnetwork during training by jointly training the weights and the structure (i.e., the weight scores), thereby avoiding the tedious process of repeated retraining, pruning, and rewinding. PFNR can allow overlap with subnetworks (i.e., weights) from previous sessions during training. This overlap allows the weights from previous sessions to remain fixed while transferring representations learned from previous videos when relevant. As a result, the model can continuously expand its representational space across successive video sessions, maintaining the encoding and generation quality of previous videos without degradation (i.e., without forgetting). This can be achieved without using a replay buffer to store multiple high-dimensional frames.

[0021] Thus, in some embodiments, the video continuous learning method can provide a practical training scenario for NIR, where a video continuous learning model continuously encodes multiple videos over successive training sessions. In contrast, existing NIR methods suffer from poor transferability to new videos due to variations in the overall video and frame distribution.

[0022] In some embodiments, video continuous learning methods can provide a method called progressive Fourier neural representation for complex sequential video editing. This method can continuously train compact subnetworks for each video session given a supernet backbone while perfectly preserving the generation quality of previous videos in Fourier space.

[0023] In some embodiments, the video continuous learning method can be effective across multiple sequential video sessions by achieving better performance than existing baselines in average peak signal-to-noise ratio (PSNR) and multi-scale structural similarity index measure (MS-SSIM) without quantitative or qualitative degradation when reconstructing previously encoded video during sequential video compilation.

[0024] In some embodiments, a video continuous learning method can separate information about learning parameters and network structures into two separate learnable parameters, called weights and weight scores, to find an optimal subnetwork. The weight scores can be binary masks with the same shape as the weights. The video continuous learning device can find a subnetwork by selecting the weights with the top c percent (c%) of weight scores. To this end, in some embodiments, the video continuous learning method can learn both the weights and the optimal subnetwork structure together. By separating the weights and the network structure in this way, the optimal subnetwork can be found without repeated retraining, pruning, and rewinding, thereby reducing the amount of computational effort required by a processor, such as a central processing unit (CPU), of a computing device, thereby improving computational efficiency (or CPU utilization). Furthermore, by using an optimal subnetwork instead of a dense network for each task, the memory efficiency of the computing device can be improved (e.g., memory capacity can be prevented from increasing).

[0025] FIG. 2 is a drawing illustrating a video continuous learning device according to an embodiment, and FIG. 3 is a drawing illustrating a process of a video continuous learning method according to an embodiment.

[0026] Referring to FIG. 2, the video continuous learning device (200) may be a computing device that performs continuous learning for a neural network.

[0027] A plurality of video sessions (e.g., N video sessions) are sequentially input into a video continuous learning device (200), and the video continuous learning device (200) can sequentially learn the N sessions. The video continuous learning device (200) trains a function f (or f) with a trainable parameter θ for a video in session s (e.g., the s-th session) among the plurality of video sessions during video continuous learning. θ ) can be expressed as. Among multiple video sessions, the video in the s-th session can be expressed as in mathematical expression 1, and the function f θ can be expressed as in mathematical formula 2.

[0028] [Mathematical Formula 1]

[0029]

[0030] T in mathematical expression 1 s represents the number of frames in session s, where s ∈ {1 ... |S|}, and |S| is N.

[0031] [Equation 2]

[0032]

[0033] In mathematical expression 2, H and W are the height and width, respectively.

[0034] The video continuous learning device (200) can receive a data set for learning of each session. Among N sessions, the data set of session s (D s ) can be expressed as in mathematical expression 3. In some embodiments, due to limited hardware memory and privacy concerns, the data set (D) for session s s) can only be accessed when learning session s. In some embodiments, a session identifier may be provided to each session for session identification. The video continuous learning device (200) can find a sub-network of the neural network for each session after learning the session.

[0035] [Equation 3]

[0036]

[0037] In mathematical expression 3, e s,t is the raw embedding of frame t in session s, and v s,t is e s,t The label of (i.e. e s,t represents the corresponding frame). Therefore, the data set (D t ) is a pair of raw embeddings and frames for NIR. s It can be made of a dog. The raw embedding e s,t For example, it can be given as in mathematical expression 4.

[0038] [Equation 4]

[0039]

[0040] In mathematical expression 4, e s is the embedding of session s, and e t is the embedding of frame t, and 160 is the sum of the degree of the embedding function of session s and the degree of the embedding function of frame t as an example value.

[0041] In some embodiments, the embedding function of session s and the embedding function of frame t may be, for example, positional encoding as defined in "Nerv: Neural representations for videos," Advances in Neural Information Processing Systems, 34:21557-21568, 2021 by Hao Chen et al. In this case, the embedding function of frame t in session s may be given, for example, as in Equation 5.

[0042] [Equation 5]

[0043]

[0044] In Equation 5, b and l are hyperparameters. If l is, for example, 80, the raw embedding e s,t The dimension of is 1×160. Also, b can be, for example, 1.25.

[0045] The video continuous learning scenario of the video continuous learning device (200) may aim to learn a series of video sessions by solving an optimization procedure in each session (i.e., session s). The optimization procedure may be a procedure to minimize loss. The model weights (θ) optimized by the optimization procedure * ) can be given as, for example, mathematical expression 6.

[0046] [Equation 6]

[0047]

[0048] In some embodiments, the loss function L(·,·) in Equation 6 may be composed of an L1 loss and a structural similarity index measure (SSIM) loss. The L1 loss may evenly minimize the pixel-wise RGB (red, green, blue) gap with respect to the original input frame, and the SSIM loss may maximize the similarity between the two entire frames based on luminance, contrast, and structure. In some embodiments, the SSIM loss may be given as in Equation 7.

[0049] [Equation 7]

[0050]

[0051] In mathematical expression 7, ^v t s is the output generated by model f (i.e., frame t in session s inferred by the video continuous learning device (200)), and α is a hyperparameter, which can be, for example, 0.7.

[0052] In some embodiments, the video continuous learning device (200) can find a sub-network that achieves performance equivalent to or better than a neural network (e.g., a deep neural network). For example, since continuous learning employs an over-parameterized deep neural network to secure resources for future tasks, it is possible to find a sub-network that achieves performance equivalent to or better than a deep neural network.

[0053] The video continuous learning device (200) can numerically determine the importance of a weight by associating each weight of the neural network with a learnable parameter called a weight score. That is, the higher the weight score, the more important the corresponding weight can be considered. The video continuous learning device (200) can find a subnetwork of the neural network and assign it as the solver for the current session s. The video continuous learning device (200) can find the subnetwork by selecting the c% weights with the highest weight score, where c is the target layer-by-layer capacity ratio (%). Frankle and Carbin showed the existence of a subnetwork that performs as well as the entire network in the lottery ticket hypothesis ("The lottery ticket hypothesis: Finding sparse, trainable neural networks," In Proceedings of the International Conference on Learning Representations (ICLR), 2019). Since the subnetwork has a smaller capacity than the dense network, it can reduce the expansion size of the solver, and thus the subnetwork can be used as the solver for the current session s instead of the entire original network.

[0054] Below, a method for obtaining a subnetwork of the current session s in a video continuous learning device is described with reference to Fig. 3. For convenience of explanation, Fig. 3 depicts a neural network as including an input block, an FSO block, and an output block; however, the number of blocks and the number of layers contained in each block are not limited thereto. In Fig. 3, edges connecting nodes in one layer to nodes in another layer may correspond to weights.

[0055] Referring to FIG. 3, the video continuous learning device may include a neural network (e.g., a video continuous learning model) (300) for video continuous learning.

[0056] The video continuous learning model (300) may be a model in which an FSO layer (322) is added to the NIR model. The video continuous learning model (300) is a neural network for video continuous learning and may include an input block (or stem block) (310), an FSO block (320), and an output block (330). In some embodiments, each block (310, 320, 330) may include one or more layers.

[0057] An embedding (311) determined by a session index s and a frame index t can be input to an input block (310). In some embodiments, the input block (310) can include one or more fully-connected layers (312, 313). A hidden representation in the input block (310) (e.g., the output of the input block (310)) is can be expressed as

[0058] The FSO block (320) outputs the input block (310). ) can perform FSO and may include an intermediate layer (321) and an FSO layer (322). In some embodiments, the intermediate layer (321) may be an output of an input block (310). ~ v t s ) may include one or more convolution layers (321a) that perform convolution operations. The intermediate layer (321) may further include an upscale layer (321b) that upscales the output of the convolution layer (321a). The FSO layer (322) may upscale the output of the input block (310) ( ~ v t s ) can perform FSO on the input block (310). The FSO layer (322) outputs the input block (310). ~ v t s) is Fourier transformed (F), and the real tensor (322a) and imaginary tensor (322b) of the Fourier transformed periodic function are respectively operated on, and then the inverse Fourier transform (F -1 ) can be done. The FSO block (320) may further include an adder (323) that adds the output of the intermediate layer (321) and the output of the FSO layer (322).

[0059] The output block (330) can further perform a convolution operation on the output of the FSO block (320). In some embodiments, the output block (330) can include one or more convolution layers (331) that perform a convolution operation on the output of the FSO block (320). The output block (330) can further include an upscale layer (332) that upscales the output of the convolution layer (331).

[0060] Existing video continuous learning devices use only a few learnable parameters in convolution operations to represent complex sequential image streams. To capture more parameter-efficient and forgettable NIR, the NIR model may require finer discretization and video-specific sub-parameters. Accordingly, a video continuous learning model (300) according to some embodiments may use a novel sub-network operator, FSO, in Fourier space that provides various bandwidths. The video continuous learning model (300) can adapt and redefine the existing definition of the Fourier convolutional operator proposed by Li et al. in "Fourier neural operator for parametric partial differential equations", arXiv preprint arXiv:2010.08895 to better fit the requirements of the NIR framework. The symbol F is used to represent the Fourier transform of a function f, and the Fourier transform F has dimension d. e = d in the embedding space of 1×160 vcan be mapped to the frame size indicated by F. The inverse of the Fourier transform is F -1 In some embodiments, the video continuous learning model (300) may use an FSO, denoted by K, which is customized to enhance the capabilities of the NIR system. The FSO may be expressed as in Equation 8.

[0061] [Equation 8]

[0062]

[0063] In mathematical equation 8, R φ is the real parameter of the subnetwork (θ real ⊙m s real ) and imaginary parameter (θ imag ⊙m s imag ) is the Fourier transform of a periodic function (i.e., a periodic subnetwork function) parameterized by .

[0064] Therefore, the operator R corresponding to the FSO layer (322) in the FSO block (320) φ is a complex-valued tensor of real (322a) and imaginary (322b) (i.e., real tensor and imaginary tensor) φ FSO ∈{θ real ,θ imag} can be parameterized separately as a real tensor θ. real and the imaginary tensor θ real are each session-dependent binary masks m s The real component m of s real and imaginary component m s imag A sparse real tensor (θ) masked by real ⊙m s real ) and the imaginary tensor (θ imag ⊙m s imag ) can be converted to.

[0065] Next, we explain FSO defined in mathematical expression 8.

[0066] D j = {x1, ..., x n} ⊂ Let D be an n-point discretization of the domain D. Given this setting, an observation a is associated with a finite set of input-output pairs indexed by j. j|Dj ∈{R n×da , u j|Dj ∈{R n×dv The goal of achieving discretization-invariance with neural operators is to discretize the subset D j This means that the operator can generate a response u(x) for every point x in the region D even if it is not an element of . This property can ensure that the neural operator maintains prediction and functional integrity over the continuous region D despite the specific discretization points that represent D.

[0067] The neural operator described by Li et al. in "Neural operator: Graph kernel network for partial differential equations", arXiv preprint arXiv:2003.03485 is v0→v1→,...,→v T can be formalized as an iterative architecture represented by , where v j (j = 0, 1, ..., T-1) represents a sequence of functions. Each function of this sequence is in space R dv can produce values ​​from . When the process is repeated, one state v t Next state v in t+1 The transformation can be defined by the interaction of two different types of operations: a non-local integral operator K and a local, nonlinear activation function σ. Specifically, at each iteration vt in v t+1 The update of is expressed as a composition of these two operations and can be mathematically expressed as in Equation 9.

[0068] [Equation 9]

[0069]

[0070] In mathematical expression 9, function operator K: A×Θ K → L(U(D;R dv ),U(D;R dv )) is a function space U(D;R dv ) is mapped to a boundary linear operator on φ∈Θ K is parameterized by . Here, A and U are separate spaces, and Θ K is a parameter space. Also, the function W : R dv → R dv is a linear transformation, and σ : R → R is a nonlinear activation function whose action is defined component-wise.

[0071] To enhance integration when defining complex and flexible functional mappings, Li et al., in "Fourier neural operator for parametric partial differential equations", arXiv preprint arXiv:2010.08895, choose K(a;φ) as a kernel integral transform parameterized by a neural network. In this case, the kernel integral operator mapping in Equation 9 can be defined as Equation 10.

[0072] [Equation 10]

[0073]

[0074] In Equation 10, k φ : R 2(d+da) → R dv×dv is φ∈Θ K is a neural network parameterized by k . φIt acts as a kernel function that learns from data. Combining Equations 9 and 10 allows neural networks to be generalized to infinite-dimensional spaces.

[0075] Remove dependency on function a and kernel function k φ (x,y)=k φ By applying the shift-invariance property to (xy), the kernel integral operator can be simplified into a convolution operator as in Equation 11. This transformation aligns the kernel integral operator with the principles of the basic solution and utilizes the unique properties of the convolution operator.

[0076] [Equation 11]

[0077]

[0078] Li et al. propose to replace the kernel integral operator in Equation 10 with the convolution operator (Equation 11) defined in Fourier space. Let F be a function f : D → R dv Fourier transform of, F -1 If we call them the inverse transforms of the Fourier transform, they can be expressed as in mathematical equations 12 and 13.

[0079] [Equation 12]

[0080]

[0081] [Equation 13]

[0082]

[0083] k in equation 11 φ (x,y,a(x),a(y))=k φ If we set (xy) and apply the convolution theorem, the convolution operation in Fourier space can be given as in mathematical equation 14.

[0084] [Equation 14]

[0085]

[0086] Also, the Fourier integral operator is k in Fourier space as in Equation 15. φ can be defined directly by parameterizing it.

[0087] [Equation 15]

[0088]

[0089] In Equation 15, R φ is φ∈Θ K Periodic functions parameterized by k (i.e., sine and cosine functions in the time domain): - D → R dv×dv is the Fourier transform of .

[0090] For a given frequency mode k ∈ D, v t The Fourier transform of (Fv t )(k)∈C dv is expressed as . Similarly, the complex-valued matrix associated with each frequency mode k is R φ (k)∈C dv×dv is expressed as. These settings may include the following key aspects:

[0091] 1. (Periodicity and Fourier series extension): Given the periodic nature of k, it can be expressed as a Fourier series. This allows discrete modes k∈Z. d Analysis and calculations can be performed from this perspective.

[0092] 2. (Truncation and finite dimension parameterization): To manage complexity and ensure computability, the Fourier series is finite with a maximum number of modes k. max can be truncated. The truncation is performed at |k for each dimension j = 1, ..., d. j | ≤ k max,j All modes k∈Z that satisfy the condition d Z including kmax It can be quantified as a set of operators R φ is a complex-valued tensor form containing a collection of truncated Fourier modes (k max ×dv ×d v ) can be parameterized.

[0093] 3. (Conjugate symmetry and real value of k): Due to the real-valued nature of k, conjugate symmetry can be imposed on the Fourier coefficients. This is a fundamental property of the Fourier transform of a real-valued function, and it can ensure that the inverse Fourier transform yields a real-valued function.

[0094] 4. (Z kmax Selection and efficiency considerations): The canonical choice for low-frequency modes is usually k∈Z d It includes the upper bound of the l1-norm of Z, but for efficient implementation it uses other criteria. kmax You can select a set.

[0095] There are differences between previous physical modeling for neural and Fourier operators and NIR for FSO. According to the previous definition proposed by Li et al., v t is a temporal length, but in some embodiments the function is a time-specific continuous hidden output (implicit representation) given a discrete session and time index (session s, time t). ~ v t s ) can be learned.

[0096] R in FSO described in mathematical formula 8 φ is the periodic function (i.e., periodic subnetwork function) k specified in Equation 14. φ As a Fourier transform of k φ is the real parameter of the subnetwork (θ real ⊙m s real ) and imaginary parameter (θ imag ⊙m s imag ) is parameterized by . Thus, R is the real and imaginary φ of the complex-valued tensor. FSO ∈{θ real ,θ imag} can be parameterized separately.

[0097] One important aspect of FSO is the operator R, as specified in Equation 15. φ Because it is parameterized as a complex-valued tensor, its parameters can increase with layer depth and input / output sizes. However, through careful layer-by-layer inspection and sparsity adjustment, a balance can be found that allows FSO to efficiently describe NIR.

[0098] Referring to Figure 3, the video continuous learning device has a data set (D) for the current session s. s ) can be trained by repeating the learning process using each batch.

[0099] The video continuous learning device generates raw instances (e) of each batch of the current session s. s,t ) can be input into the layer of the neural network (300) for the current session s to perform the inference process through the forward pass. The video continuous learning device can reuse the selected weights of the previous session (s-1) in the forward pass of the neural network for learning the current session s. For example, as illustrated in FIG. 3, if there is a weight selected in the previous session (s-1) among the weights (i.e., edges) of the forward pass of the current session s, the video continuous learning device can reuse the weights selected in the previous session (s-1).

[0100] The video continuous learning device can update the weights through the backward pass of the neural network (300) for learning the current session s based on the results inferred in the forward pass. In this case, the video continuous learning device can update other weights without updating the reused weights (i.e., the weights selected in the previous session (s-1)). In some embodiments, the video continuous learning device can update the raw instance e input to the forward pass based on the results inferred in the forward pass.s,t Label v assigned to s,t We can calculate the loss between them and update the weights based on the loss.

[0101] Video continuous learning devices are datasets (D t ) can be repeated for each batch to learn the current session s of the neural network. Meanwhile, the weights θ used in the video continuous learning device * ={θ,φ FSO} is the weight θ of the NIR model and the weight φ of the FSO layer (322). FSO It can be made of.

[0102] The video continuous learning device has a weighted score ρ * Select the subnetwork with the highest c% weight ^ θ s can be obtained. Here, c is the target capacity ratio (%) for each layer. Weight score ρ * is a learnable parameter for numerically determining the importance of weights, and the weight score ρ for the weight θ of the NIR model and the weight φ of the FSO layer (322) FSO Weighted score ρ for FSO {ρ,ρ consisting of FSO} can be expressed as. Weight selection is performed using a session-dependent binary mask m s It is expressed as , and if the value is 1, it indicates that the weight was selected during the forward path, and if the value is 0, it indicates that the weight was not selected. Binary mask m s is the weighted score ρ * Indicator function 1 c It can be obtained by applying the weighted score ρ * If you are in the top c% of scores1 c (ρ * ) is 1, and the weight score ρ * If you are not in the top c% of scores1 c (ρ* ) can be 0. Therefore, the subnetwork for session s ^ θ s can be given as in mathematical expression 16.

[0103] [Equation 16]

[0104]

[0105] In some embodiments, given a loss L(·), a video continuous learning device jointly learns the weights and binary masks of the sub-networks associated with each session, the weights θ * and weighted score ρ * can be updated to minimize the loss. However, this optimization procedure requires (1) all weights θ when training for a new session. * Updating interferes with the weights assigned to the previous session, and (2) the indicator function always has a slope value of 0, so the weight score ρ is used as the loss slope. * There may be a problem that it cannot be updated.

[0106] In some embodiments, the video continuous learning device can selectively update weights by allowing only weights that were not selected in the previous session to be updated to solve the first problem. To this end, the video continuous learning device accumulates a binary mask (M) when learning a session s. s-1 ) using weights θ * can be updated. In some embodiments, the cumulative binary mask (M s-1 ) can be given as in mathematical expression 17. In some embodiments, the video continuous learning device accumulates the mask (M) from 1 to the previous session (s-1). s-1 ) using a mask with weights θ * can be updated. For example, a video continuous learning device uses an optimization technique with a learning rate η to update the weights θ *can be updated as in mathematical expression 18. Accordingly, the video continuous learning device can effectively freeze the weights of the sub-network selected in the previous session (s-1).

[0107] [Equation 17]

[0108]

[0109] [Equation 18]

[0110]

[0111] The video continuous learning device is used to solve the second problem, mask m s Since it is obtained with the top c% score, a straight-through estimator can be used in the backward path. The indicator function (1 c ) always has a slope value of 0, so the video continuous learning device uses the indicator function (1 c ) and ignore the derivative of the weighted score ρ * can be updated. For example, the straight-through estimator can refer to Hinton's "Neural networks for machine learning," 2012 paper, Bengio et al.'s "Estimating or propagating gradients through stochastic neurons for conditional computation," CoRR, 2013 paper, Ramanujan et al.'s "What's hidden in a randomly weighted neural network" In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020, etc. For example, the video continuous learning device has a weighted score ρ * can be updated as in mathematical expression 19.

[0112] [Equation 19]

[0113]

[0114] In Equations 18 and 19, the loss L can be given as in Equation 20.

[0115] [Equation 20]

[0116]

[0117] b in equation 20 t is the label of the input batch (i.e. v s,t )am.

[0118] In this way, a separate weight score ρ is used as a criterion for selecting the weights of the subnetwork. * Using transfer learning, some of the weights selected in the previous session can be reused to solve the current session. This can be viewed as transfer learning. That is, the weights selected in the current session may include some of the weights selected in the previous session. Similarly, weights selected in the previous session that are irrelevant to the current session are not selected. Instead, weights are selected from the set of weights not yet selected to match the target network capacity of the current session. This can be viewed as fine-tuning from session {1,...,|S|} to session s.

[0119] The video continuous learning method described above can be expressed as the following algorithm.

[0120] Algorithm 1

[0121]

[0122] In some embodiments, the subnetwork (i.e., binary mask) of the current session can be selected during the training of the current session. Accordingly, the video continuous learning device may generate a binary attention mask (m) that means the optimal subnetwork (i.e., optimal binary mask) for the current session. s) can be determined. In some embodiments, the video continuous learning device |m s |A binary attention mask m that describes the optimal subnetwork for session s such that the model capacity is less than c. t can be determined as in mathematical formula 21.

[0123] [Equation 21]

[0124]

[0125] In Equation 21, J represents the session loss and can be given as Equation 22, and c satisfies c ≪ |θ|.

[0126] [Equation 22]

[0127]

[0128] Figure 4 is a flowchart of a video continuous learning method according to some embodiments.

[0129] In some embodiments, a video continuous learning device may receive a data set and a target capacity ratio for each of the multiple sessions to train the device. The target capacity ratio may be provided as a value for selecting weights for each layer of the neural network. Furthermore, the video continuous learning device may randomly initialize the weights and weight scores of the video continuous learning model before training the device for multiple sessions.

[0130] Referring to FIG. 4, in learning for session s among multiple sessions, the video continuous learning device can reuse weights selected in session (s-1) (previous session or second session) among multiple sessions in the forward pass of session s (current session or first session) of the neural network (S410). That is, the multiple weights used for learning for session s may include weights selected in the previous session. The video continuous learning device can Fourier transform the hidden representation in the forward pass, operate the Fourier-transformed hidden representation with FSO, and then inversely Fourier transform it. In some embodiments, the FSO may be given as a real tensor and an imaginary tensor of a Fourier-transformed periodic function.

[0131] The video continuous learning device can update weights, excluding weights selected in the previous session, among multiple weights in the reverse path of session s of the neural network, and freeze the weights selected in the previous session (S440). The video continuous learning device can learn a binary mask together with learning the weights (S450). The video continuous learning device can learn the binary mask by learning (i.e., updating) the weight scores.

[0132] In some embodiments, the video continuous learning device may obtain a binary mask for selecting weights whose weight scores are in the top c% among a plurality of weights (S420). Here, c is a target capacity ratio. In this case, the weights may include weights of the video continuous learning model and weights of the FSO. In some embodiments, the video continuous learning device may obtain a binary mask for selecting weights whose weight scores are in the top c% for each layer. The video continuous learning device may update the weights based on an accumulated binary mask that accumulates binary masks from the first session (session 1) to session (s-1) (S440). In some embodiments, the video continuous learning device may calculate a loss based on the weights (i.e., sub-networks) selected by the binary mask and the input batch of the data set (S430), and update the weights based on the accumulated binary mask and the loss (S440). The reused weights may not be updated and may be frozen by the accumulated binary mask. Additionally, the video continuous learning device can update the weight score based on the loss calculated in step S430 (S450). In this case, the weight score may include the weight score of the video continuous learning model and the weight score of the FSO. In some embodiments, the continuous function device can update the weight score while ignoring the differentiation of the indicator function used in the binary mask (S450).

[0133] In some embodiments, the video continuous learning device can learn the session s, weights, and weight scores (binary masks) in the neural network by repeating the processes of S410 to S450 for each batch of the data set of the current session.

[0134] In some embodiments, when learning for session s is completed, the video continuous learning device can determine a mask (binary attention mask) representing an optimal sub-network among the binary masks acquired during learning (S460). Accordingly, the video continuous learning device can find a sub-network for the current session s with the optimal binary mask (binary attention mask). In some embodiments, when learning for session s is completed, the video continuous learning device can accumulate the binary mask (e.g., binary attention mask) of session s to the accumulated binary mask up to session (s-1) to acquire an accumulated binary mask up to session s (S470).

[0135] In some embodiments, the video continuous learning device can convert binary masks (e.g., binary attention masks) of multiple sessions into a single accumulated mask and compress the accumulated mask into an M-bit binary map (S480). Here, M is a natural number.

[0136] As described above, in some embodiments of the video continuous learning method, the neural network can find session-adaptive winning tickets (i.e., subnetworks) and update only the weights that were not learned in previous sessions. After training for each session, the video continuous learning model can fix the subnetwork parameters, preventing the video continuous learning method from being affected by catastrophic forgetting. Furthermore, the video continuous learning method can selectively transfer previously learned knowledge to future sessions (forward propagation), significantly reducing the training time required to converge during sequential learning. This advantage can be even more important in large-scale continuous learning problems where the continuous learning device sequentially learns multiple sessions.

[0137] Meanwhile, for experimental results explaining the effectiveness of a video sequential learning method according to certain embodiments and specific examples of the video sequential learning method, please refer to the paper "Progressive Fourier Neural Representation for Sequential Video Compilation" by the inventors of the present invention, ICLR, 2024. This paper is incorporated herein by reference.

[0138] Next, an exemplary computing device capable of implementing a video continuous learning device according to some embodiments is described with reference to FIG. 5.

[0139] FIG. 5 is a block diagram illustrating a computing device according to some embodiments.

[0140] Referring to FIG. 5, the computing device includes a processor (510), a memory (520), a storage device (530), a communication interface (540), and a bus (550). The computing device (500) may further include other general-purpose components.

[0141] The processor (510) controls the overall operation of each component of the computing device (500). The processor (510) may be implemented as at least one of various processing units, such as a central processing unit (CPU), a microprocessor unit (MPU), a microcontroller unit (MCU), and a graphic processing unit (GPU), and may also be implemented as a parallel processing unit. In addition, the processor (510) may perform operations on a program for executing the video continuous learning method described above.

[0142] The memory (520) stores various data, commands, and / or information. The memory (520) can load a computer program from the storage device (530) to execute the video continuous learning method described above. The storage device (530) can store the program non-temporarily. The storage device (530) can be implemented as non-volatile memory.

[0143] The communication interface (540) supports wired and wireless Internet communication of the computing device (500). In addition, the communication interface (540) may support various communication methods other than Internet communication.

[0144] The bus (550) provides communication functions between components of the computing device (500). The bus (550) may be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0145] The computer program may include instructions that cause the processor (510) to perform a video continuous learning method when loaded into the memory (520). That is, the processor (510) may perform operations for the video continuous learning method by executing the instructions.

[0146] In some embodiments, the computer program may include instructions for using a plurality of weights including weights selected from previous video sessions among the plurality of video sessions in a forward pass of a neural network for learning a current video session among the plurality of video sessions, freezing the selected weights in a backward pass of the neural network for learning the current video session, updating weights among the plurality of weights except for the selected weights, obtaining a binary mask for selecting some of the plurality of weights based on a weight score of each weight, and finding a subnetwork of the neural network for the current video session using the binary mask. In this case, the neural network may include an NIR model and an FSO layer, and the FSO layer may include a real tensor and an imaginary tensor of a Fourier transform of a periodic function.

[0147] The entropy-weighted adversarial learning method or device according to one embodiment of the present invention described above may be implemented as a computer-readable computer program on a computer-readable medium. In one embodiment, the computer-readable medium may be a removable storage medium or a fixed storage medium. In another embodiment, the computer program recorded on the computer-readable medium may be transmitted to another computing device via a network such as the Internet, and installed and executed on the other computing device.

[0148] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims also fall within the scope of the present invention.

Claims

1. A video continuous learning method for sequentially learning multiple video sessions performed by a computing device, A step of using a plurality of weights including weights selected from a previous video session among the plurality of video sessions in the forward pass of the neural network for learning the current video session among the plurality of video sessions, A step of freezing the selected weights in the reverse path of the neural network for learning the current video session and updating weights excluding the selected weights among the plurality of weights; A step of obtaining a binary mask for selecting some of the weights among the plurality of weights based on the weight score of each weight, and A step of finding a sub-network of the neural network for the current video session with the binary mask, The above neural network includes a neural implicit representation (NIR) model and a Fourier sub-network operator layer, wherein the Fourier sub-network operator layer includes a real tensor and an imaginary tensor of a Fourier transform of a periodic function. Video continuous learning method.

2. In paragraph 1, A video continuous learning method in which the Fourier sub-network operator layer Fourier transforms the hidden representation of the NIR model, computes the real tensor and the imaginary tensor respectively on the Fourier-transformed hidden representation, and then inversely Fourier transforms the hidden representation.

3. In paragraph 1, Each weight includes the weight of the NIR model and the weight of the Fourier sub-network operator layer, The above weight score includes the weight score of the NIR model and the weight score of the Fourier sub-network operator layer. Video continuous learning method.

4. In paragraph 1, The above binary mask selects a weight whose weight score is in the top c% among the plurality of weights as the partial weight, The above c is the target capacity ratio Video continuous learning method.

5. In paragraph 4, The above binary mask is a video continuous learning method in which the weights whose weight scores are in the top c% for each layer of the neural network are selected as the partial weights.

6. In paragraph 1, A video continuous learning method, wherein the step of updating the weight includes the step of freezing the selected weight and updating the weights excluding the selected weights among the plurality of weights based on an accumulated binary mask obtained from the first video session to the previous video session among the plurality of video sessions.

7. In paragraph 6, Further comprising a step of calculating a loss based on the weights selected by the binary mask, The step of updating the weight includes a step of freezing the selected weight and updating the weights excluding the selected weight among the plurality of weights based on the accumulated binary mask and the loss. Video continuous learning method.

8. In paragraph 1, A step of calculating a loss based on the weights selected by the above binary mask, and A step of updating the weight score based on the above loss. A video continuous learning method including more.

9. In paragraph 1, A video continuous learning method further comprising a step of obtaining an accumulated binary mask by accumulating the binary masks obtained from the first video session to the current video session among the plurality of video sessions.

10. Memory that stores one or more instructions, and Includes a processor, The above processor executes the above instruction, In the forward pass of the neural network for learning the current video session among multiple video sessions, multiple weights including weights selected from previous video sessions among the multiple video sessions are used, Freeze the selected weights in the reverse path of the neural network for learning the current video session, and update the weights excluding the selected weights among the plurality of weights. Obtain a binary mask for selecting some of the weights among the plurality of weights based on the weight score of each weight, Finding a subnetwork of the neural network for the current video session with the binary mask, The above neural network includes a neural implicit representation (NIR) model and a Fourier sub-network operator layer, wherein the Fourier sub-network operator layer includes a real tensor and an imaginary tensor of a Fourier transform of a periodic function. Video continuous learning device.

11. In paragraph 10, A video continuous learning device in which the Fourier sub-network operator layer Fourier transforms the hidden representation of the NIR model, computes the real tensor and the imaginary tensor respectively on the Fourier-transformed hidden representation, and then inversely Fourier transforms the hidden representation.

12. In paragraph 10, Each weight includes the weight of the NIR model and the weight of the Fourier sub-network operator layer, The above weight score includes the weight score of the NIR model and the weight score of the Fourier sub-network operator layer. Video continuous learning device.

13. In paragraph 10, The above binary mask selects a weight whose weight score is in the top c% among the plurality of weights as the partial weight, The above c is the target capacity ratio Video continuous learning device.

14. In paragraph 13, A video continuous learning device, wherein the binary mask selects weights whose weight scores are in the top c% for each layer of the neural network as the partial weights.

15. In paragraph 10, A video continuous learning device, wherein the processor freezes the selected weight and updates weights excluding the selected weight among the plurality of weights based on an accumulated binary mask obtained from the first video session to the previous video session among the plurality of video sessions.

16. In paragraph 15, The above processor Compute the loss based on the weights selected by the above binary mask, Based on the above accumulated binary mask and the above loss, the selected weight is frozen and the weights excluding the selected weight among the plurality of weights are updated. Video continuous learning device.

17. In paragraph 10, The above processor Compute the loss based on the weights selected by the above binary mask, Updating the weighted score based on the above loss Video continuous learning device.

18. In paragraph 10, A video continuous learning device, wherein the processor obtains an accumulated binary mask by accumulating the binary masks obtained from the first video session to the current video session among the plurality of video sessions.

19. A computer program that is executed by a computing device and stored in a recording medium, The above computer program is configured so that the computing device, A step of using a plurality of weights including weights selected from a previous video session among the plurality of video sessions in the forward pass of the neural network for learning the current video session among the plurality of video sessions, A step of freezing the selected weights in the reverse path of the neural network for learning the current video session and updating weights excluding the selected weights among the plurality of weights; A step of obtaining a binary mask for selecting some of the weights among the plurality of weights based on the weight score of each weight, and Execute the step of finding a subnetwork of the neural network for the current video session with the binary mask, The above neural network includes a neural implicit representation (NIR) model and a Fourier sub-network operator layer, wherein the Fourier sub-network operator layer includes a real tensor and an imaginary tensor of a Fourier transform of a periodic function. Computer program.

Images