A video stream completion method combining static information and dynamic information
Patent Information
- Application Number
- CN202310712902.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-15
- Publication Date
- 2026-09-22
- Estimated Expiration
- 2043-06-15
AI Technical Summary
[0004]前两类的低秩张量补全模型，一方面在采样率较低(5％-15％之间)或者处理细节信息(比如人的眼睛信息，微表情信息)时，所恢复的图像或视频流效果并不理想；另一方面，这两类的静态模型没有刻画不同时刻之间张量的相关性信息，因此无法实时对新视频流进行补全；而第三类的深度学习模型需要大量的训练样本进行学习，然而这在采样率较低时，无法获取到足够多的样本，因此会导致图像或视频流的恢复效果降低
[0040]本发明提供了一种结合静态信息与动态信息的视频流补全方法。与现有技术相比具备以下有益效果：
Smart Images
Figure CN116781845B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video image processing technology, specifically to a video stream completion method that combines static and dynamic information. Background Technology
[0002] Due to factors such as human error, cost, or equipment, high-dimensional dynamic data (such as video data) may be lost during transmission, capture, or generation. Low-rank tensor completion is a method to recover lost data based on known data information and by utilizing global low-rank properties.
[0003] Current low-rank tensor completion models can be broadly categorized into three types: 1) models based on prior information from tensor decomposition; 2) models based on rank minimization; and 3) models based on deep learning. The first two types mostly analyze incomplete tensors holistically, thus belonging to static models. Furthermore, the constructed static models only consider the structural information of the tensor (low-rank or sparsity), such as models using CANDECOMP / PARAFAC decomposition (CP decomposition) of tensors and characterizing the low-rank nature of the factor matrix using the nuclear norm; models using singular value decomposition of tensors and then minimizing the corresponding pipeline rank; models establishing the average rank model of the minimized tensor; and models using tensor ring decomposition (TR decomposition) to obtain third-order factor tensors, then imposing low-rank constraints based on the nuclear norm on the factor tensors. The third type, deep learning models, learn from the acquired observation samples as a training set and then predict missing sample data.
[0004] The first two types of low-rank tensor completion models, on the one hand, do not produce ideal image or video stream recovery results when the sampling rate is low (between 5% and 15%) or when processing detailed information (such as human eye information and micro-expression information); on the other hand, these two types of static models do not characterize the correlation information of tensors at different times, so they cannot complete new video streams in real time. The third type of deep learning model requires a large number of training samples for learning, but when the sampling rate is low, it is impossible to obtain enough samples, which will lead to a decrease in the recovery effect of image or video stream. Summary of the Invention
[0005] (a) Technical problems to be solved
[0006] To address the shortcomings of existing technologies, this invention provides a video stream completion method that combines static and dynamic information. It uses a method that combines dynamic and static structural information to describe video data. The model established by this method not only greatly improves the recovery effect when the sampling rate is low or when processing detailed information, but also can repair video data in real time.
[0007] (II) Technical Solution
[0008] To achieve the above objectives, the present invention provides the following technical solution: a video stream completion method combining static and dynamic information, specifically comprising the following steps:
[0009] S1. First, use tensors Let X represent the original data, and let X = {X} (1) ,X (2) ,...,X (T) ,...} represents a sequence that increases along the last dimension, where
[0010] S2, then the tensor X at time t (t) FCTN decomposition yields N factor tensors Next, these factor tensors are analyzed from two aspects: static information and dynamic information.
[0011] S3. Finally, the static and dynamic information of the factor tensor are combined to establish a video stream completion model, and its augmented Lagrangian function is constructed. The variables in the model are iteratively calculated using the alternating direction multiplier method, and the tensor X that satisfies the error condition is output. (t) By continuously executing this process, it is possible to complete the video at different times.
[0012] Preferably, the static information analysis of the factor tensor in step S2 specifically involves: ... The matrix obtained after expansion Perform low-rank matrix decomposition to obtain U n W n Therefore, characterizing tensor X (t) The structural information only requires measuring U. n Sparsity in.
[0013] Preferably, the dynamic information analysis of the factor tensor in step S2 specifically involves: [analyzing the factor tensor]. During the analysis, all factor tensors before time t... The update task has been completed, therefore... and Selecting from the similarity scores between them Then measure and The correlation between them helps to complete the analysis at time t. Update.
[0014] Preferably, in step S1, X (t) The input contains known pixel values and a missing pixel value of 0. The missing value of 0 needs to be solved using the known pixel values.
[0015] Preferably, in step S2, the tensor X at time t is... (t) Perform FCTN decomposition, i.e.
[0016]
[0017] Among the factor tensors (R 1,2 ,...,R 1,N ,R 2,3 ,...,R N-1,N ) represents X (t) The rank of X under this decomposition (t) (i1,i2,...,i N ) represents X (t) In index (i1, i2, ..., i N The value at ) is used Briefly represent the above formula, and then measure the static and dynamic information of these factor tensors.
[0018] Preferably, the X (t) Strong internal correlation (higher-order tensors have strong internal correlation), therefore use Characterizing the sparsity of tensors, where:
[0019]
[0020] Due to X (t) FCTN decomposition was performed, therefore the factor tensor G( n t) will inherit X (t) If we obtain sparse information, then we can get
[0021]
[0022] in This indicates that tensor M n Remodel into m 1:N-1 ×n 1:N-1 For a matrix of size, the above equation is derived from Theorem 1:
[0023] Theorem 1: Given an Nth-order tensor X, its FCTN decomposition is X = FCTN(G1, G2, ..., G...). N ).let This indicates removing G from the factor tensor. t The tensor constructed afterward has
[0024]
[0025] in
[0026] on the other hand
[0027]
[0028] Then there is
[0029]
[0030] As can be seen from the above, It can not only characterize factor tensors It retains the sparsity of the original tensor X, while also preserving its original sparsity. (t) The low-rank nature of , therefore, for Perform low-rank matrix decomposition to obtain
[0031]
[0032] in It is orthogonal. because The orthogonality property of the columns makes It still has the original matrix The sparsity of [the property] can therefore be minimized. Metric tensor The sparsity of the data is used to obtain static information.
[0033] Preferably, the last dimension of the tensor X increases with time, for tensor X at time t. (t) When performing completion, this indicates that the tensors before time t have already completed the completion task. At the same time, there is a certain correlation between these tensors (video data) that increase over time. Therefore, by using the tensor information that has been completed before time t, we can assist in completing the tensor completion task that has been added at time t.
[0034] Preferably, for video data X (t) (t=1,2,...,T), then after FCTN decomposition, the factor tensor The X values at each moment will also be inherited. (t) Information on the correlation between them, therefore updated During the process, the information from the factor tensor at previous time steps will definitely be used. This represents the factor tensor before time t, at which point... The update has been completed, so it is now available.
[0035]
[0036] express Figure 1 The aforementioned similarity is determined by setting a similarity threshold s, and using the condition that the sim value is greater than s to determine the similarity. Make a selection Then use
[0037]
[0038] measure and The correlation, i.e. Figure 1 The dynamic information, where μ represents the forgetting parameter, indicates that when μ = 0, it means... There is no memory property, meaning there is no correlation between the factor tensors. When μ = 1, it indicates that... Complete memory, that is, memory up to time t. It can preserve complete information, so μ takes the value (0,1).
[0039] (III) Beneficial Effects
[0040] This invention provides a video stream completion method that combines static and dynamic information. Compared with existing technologies, it has the following advantages:
[0041] (1) The video stream completion method that combines static and dynamic information has a significantly improved recovery effect compared to existing technologies when the sampling rate is low, and can also recover detailed information in video data very well.
[0042] (2) This video stream completion method that combines static and dynamic information can use the information that has already been completed to complete the newly added video in real time. Attached Figure Description
[0043] Figure 1 This is a flowchart of the video stream completion method.
[0044] Figure 2 This is a schematic diagram illustrating the algorithm process of the video stream completion model of the present invention;
[0045] Figure 3 This is a schematic diagram illustrating the restoration effect of the gray video of the present invention at frames 5, 27, 63, 97, and 136.
[0046] Figure 4 This is a schematic diagram illustrating the restoration effect of the video at frames 101 and 105 of this invention. Detailed Implementation
[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0048] Please see Figure 1-4 This invention provides a technical solution: a video stream completion method combining static and dynamic information, specifically including the following steps:
[0049] S1. First, use tensors Let X represent the original data, and let X = {X} (1) ,X (2) ,...,X (T) ,...} represents a sequence that increases along the last dimension, where
[0050] S2, then the tensor X at time t (t) FCTN decomposition yields N factor tensors Next, these factor tensors are analyzed from two aspects: static information and dynamic information.
[0051] S3. Finally, the static and dynamic information of the factor tensor are combined to establish a video stream completion model, and its augmented Lagrangian function is constructed. The variables in the model are iteratively calculated using the alternating direction multiplier method, and the tensor X that satisfies the error condition is output. (t) By continuously executing this process, it is possible to complete the video at different times.
[0052] exist Figure 1 The tensor data obtained is Represent X as an increasing sequence of tensors, i.e., X = {X...} (1) ,X (2) ,...,X (T)},in And X (t) The input contains known pixel values and a missing pixel value of 0. The missing value of 0 needs to be solved using the known pixel values.
[0053] First, consider the tensor X at time t. (t) Perform FCTN decomposition, i.e.
[0054]
[0055] Among the factor tensors (R 1,2 ,...,R 1,N ,R 2,3 ,...,R N-1,N ) represents X (t) The rank of X under this decomposition (t) (i1,i2,...,i N ) represents X (t) In index (i1, i2, ..., i N The value at ) is used The formula (1) is briefly represented, and then the static and dynamic information of these factor tensors is measured.
[0056] (1) Static structural information of the metric factor tensor. and X (t) Performing mode-n expansions respectively yields and On the one hand X (t) Strong internal correlation (higher-order tensors have strong internal correlation), therefore use Characterizing the sparsity of tensors, where
[0057]
[0058] Due to X (t) FCTN decomposition was performed, therefore the factor tensor after decomposition is Will inherit X (t) If we obtain sparse information, then we can get
[0059]
[0060] in This indicates that tensor M n Remodel into m 1:N-1 ×n 1:N-1 For a matrix of size, equation (2) can be obtained from the following Theorem 1.
[0061] Theorem 1: Given an Nth-order tensor X, its FCTN decomposition is X = FCTN(G1, G2, ..., G...). N ).let This indicates removing G from the factor tensor. t The tensor constructed afterward has
[0062]
[0063] in
[0064] on the other hand
[0065]
[0066] Then there is
[0067]
[0068] As can be seen from the above, It can not only characterize factor tensors It retains the sparsity of the original tensor X, while also preserving its original sparsity. (t) The low-rank nature of . Therefore, for Perform low-rank matrix decomposition to obtain
[0069]
[0070] in It is orthogonal. because The orthogonality property of the columns makes It still has the original matrix The sparsity of [the property] can therefore be minimized. Metric tensor The sparsity of the property is used to obtain Figure 1 The aforementioned static information.
[0071] (2) Measuring the dynamic correlation information between the factor tensors. The last dimension of tensor X increases with time; for tensor X at time t... (t) When completion is performed, this indicates that the tensors before time t have already been completed. Furthermore, there is a certain correlation between these tensors (video data) that increase over time; therefore, the information from the tensors completed before time t can be used to assist in completing the tensors added at time t.
[0072] For video data X (t) (t=1,2,...,T), then after FCTN decomposition, the factor tensor The X values at each moment will also be inherited. (t) Information on the correlation between them, therefore updated The process will inevitably utilize information from the factor tensor at previous moments. This represents the factor tensor before time t, at which point... The update has been completed, so it is now available.
[0073]
[0074] express Figure 1 The aforementioned similarity is determined by setting a similarity threshold s, and using the condition that the sim value is greater than s to determine the similarity. Make a selection Then use
[0075]
[0076] measure and The correlation, i.e. Figure 1 The dynamic information, where μ represents the forgetting parameter, indicates that when μ = 0, it means... There is no memory property, meaning there is no correlation between the factor tensors. When μ = 1, it indicates that... Complete memory, that is, memory up to time t. It can preserve complete information, so μ takes the value (0,1).
[0077] By further combining static structural information with dynamic information, the following video stream completion model is established:
[0078]
[0079] Where Ω represents the set of observations, i.e.
[0080]
[0081] The above optimization model is solved using the alternating direction multiplier method, with auxiliary variables introduced. make Construct the augmented Lagrangian function, i.e.:
[0082]
[0083] in Let be the Lagrange multiplier. Therefore, according to equation (4), the update of the optimization variable can be expressed as the following equation:
[0084]
[0085]
[0086]
[0087]
[0088]
[0089] Lagrange multipliers are updated to:
[0090]
[0091]
[0092] (1) Update Its optimization objective function can be written as
[0093]
[0094] This subproblem can then be solved using the soft thresholding operator, i.e.
[0095]
[0096] Where H τ (x) = sign(x)⊙max(x-τ,0), where ⊙ represents the Hardmard product. (2) Update about The optimization objective function can be written as
[0097]
[0098] Therefore, we can obtain
[0099]
[0100]
[0101] This problem can be solved using Theorem 2.
[0102] Theorem 2: Given a matrix M∈R m×n The following optimization issues
[0103]
[0104] stW T W = I
[0105] The solution is W = PS T Where A = PΣS T This is a simplified singular value decomposition.
[0106] So The solution is
[0107]
[0108] (3) Update about The objective function is:
[0109]
[0110] We can directly differentiate the objective function, that is...
[0111]
[0112] fold n (·) indicates a folding operation along the nth dimension, and pinv(·) indicates an inverse or pseudo-inverse operation.
[0113] (4) Update variables Its objective function is:
[0114]
[0115] According to Theorem 1, this can be transformed into:
[0116]
[0117] Similarly, taking the derivative of the objective function directly yields:
[0118]
[0119] (5) Update variable X (t) The corresponding objective function is:
[0120]
[0121] The answer can be obtained according to Theorem 1.
[0122]
[0123] Where Ω c (The set of indices corresponding to missing values) represents the complement of set Ω.
[0124] Calculate the (k+1)th iteration step Determine if the error condition is met; if so, output X. (t) This completes the tensor X at time t. (t) The solution is completed; if not satisfied, the iteration continues until X is found. (t) The error condition is met.
[0125] Finally, the process is repeated continuously. Figure 1 The flowchart outlines the operation steps, allowing for real-time completion of image or video data.
[0126] The pseudocode for the above process can be found here. Figure 2 .
[0127] Case Analysis
[0128] The above method was validated using grayscale video and video data. Each video data point was subjected to random missing data processing (missing rate of 90%) and block processing.
[0129] Random missing data handling: Set the missing rate (missrate) and define a vector s of size dim × 1, where s(i) = i. Then, randomly select (1-missrate)×dim (round to the nearest integer) values from s as the index set Ω of the observation values.
[0130] Block processing: The last dimension I3 or I4 of the video data is divided into T blocks, and the tensor size of each block is I1×I2×d. t or I1×I2×3×d t ,satisfy
[0131] 1) Select data named "suize" from the grayscale video. Its data size is X∈R. 144×176×150N=3, then random missing values are processed with a missrate of 90%, and T=10, d1=…=d T =15, then X = {X (1) ,X (2) ,...,X (10)}, with X (t) ∈R 144×176×15 As input, perform the repair; the result is shown below. Figure 3 .
[0132] 2) Select data named "hall" from the video, with a data size of X∈R. 144×176×3×300 N=4, then random missing values are processed with a missrate of 90%, and T=20, d1=…=d T =15, then X = {X (1) ,X (2) ,...,X (20)}, with X (t) ∈R 144×176×3×15 As input, perform the repair; the result is shown below. Figure 4 .
[0133] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.
[0134] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0135] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A video stream completion method combining static and dynamic information, characterized in that: Specifically, the following steps are included: S1. First, use tensors Represent the original data, and let This represents a sequence that increases along the last dimension, where , ; S2, then tensor of time Perform fully connected tensor networks Decomposition Factor tensors Then, these factor tensors are analyzed from two aspects: static information and dynamic information. S3. Finally, the static and dynamic information of the factor tensor are combined to establish a video stream completion model, its augmented Lagrangian function is constructed, and the variables in the model are iteratively calculated using the alternating direction multiplier method to output a tensor that satisfies the error condition. By continuously executing this process, it is possible to complete the video at different times; The static information analysis of the factor tensor in step S2 specifically involves: […]. The matrix obtained after expansion Low-rank matrix decomposition yields Therefore, characterizing the tensor The structural information only needs to be measured. sparsity in; The dynamic information analysis of the factor tensor in step S2 specifically involves: [analyzing the factor tensor...] While conducting the analysis, All factor tensors before time step The update task has been completed, therefore... and Selecting from the similarity scores between them Then measure and The correlation between them, thus helping to complete the task. time Update.
2. The video stream completion method combining static and dynamic information according to claim 1, characterized in that: In step S1 The input contains known pixel values and a missing pixel value of 0. The missing value of 0 needs to be solved using the known pixel values.
3. The video stream completion method combining static and dynamic information according to claim 1, characterized in that: In step S2, the... Time tensor Perform fully connected tensor networks Decomposition, i.e. ; Among the factor tensors , express Rank under this decomposition, express In the index The value at that location, using Briefly represent the above formula, and then measure the static and dynamic information of these factor tensors.
4. The video stream completion method combining static and dynamic information according to claim 1, characterized in that: The The internal correlation is strong, therefore it is used Characterizing the sparsity of tensors, where: Due to Performed Decomposition, therefore the factor tensor after decomposition Will inherit If we obtain sparse information, then we can get ; in Indicates tensor Remodeling The size of the matrix is obtained by the following Theorem 1: Theorem 1: Given order tensor Its fully connected tensor network Decomposed into .let This indicates removing from the factor tensor The tensor constructed afterward then has ; in , on the other hand ， Then there is ； As can be seen from the above, It can not only characterize factor tensors It retains the sparsity of the original tensor while preserving its original sparsity. The low-rank nature of , therefore, for Perform low-rank matrix decomposition to obtain ; in , It is orthogonal ( ),because The orthogonality property of the columns makes It still has the original matrix The sparsity of the property can be minimized. Metric tensor The sparsity of the data is used to obtain static information.
5. A video stream completion method combining static and dynamic information according to claim 1, characterized in that: The tensor The last dimension increases with time. Time tensor When performing completion, this indicates that in The tensors from the previous time step have already been completed. Furthermore, there is a certain correlation among these tensors that have increased over time. Therefore, by leveraging… The tensor information is completed before the time step, thereby assisting in the completion of the task. The task of completing tensors that are constantly being added.
6. A video stream completion method combining static and dynamic information according to claim 5, characterized in that: For video data So after After decomposition, the factor tensor The time intervals will also be inherited. Information on the correlation between them, therefore updated During the process, the information from the factor tensor at previous time steps will definitely be used. express The factor tensor before time step, at this point The update has been completed, so it is now available. To represent similarity, a similarity threshold is set. ,use Is the value greater than This condition, for Make a selection Then use measure and The correlation, i.e., dynamic information, in which Indicates the forgotten parameter, when When, it means There is no memory property, meaning there is no correlation between the factor tensors. When, it means Complete memory, that is, complete memory Before the moment It can retain complete information, therefore Values .