Video processing method and system
By using convolutional neural networks to perform feature mapping and weighting processing in video processing, the problem of insufficient utilization of common information in dynamic scenarios is solved, the performance of video target segmentation and multi-objective separation accuracy are improved, and the processing capability of dynamic scenarios is enhanced.
Patent Information
- Application Number
- CN202410274522.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-11
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2044-03-11
AI Technical Summary
The existing supervised video target separation model is insufficiently utilized in dynamic scenarios, resulting in a degradation of performance, and the multi-objective separation accuracy is low, making it difficult to effectively improve the model's discrimination ability of different goals.
By acquiring the high-dimensional features of the video frame image, using a convolutional neural network for feature mapping and weighting, differential features are obtained and aggregated, combined with channel stitching, and inputting them to the decoder to improve the performance of video target segmentation.
The performance of supervised video target segmentation and multi-objective separation accuracy are improved, the processing ability of dynamic scenes is enhanced, and the discriminatory ability of video processing is improved.
Smart Images
Figure CN118115739B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video processing, and in particular to a video processing method and system. Background Art
[0002] In the process of video special effects processing, it is sometimes necessary to segment the background and foreground moving targets in the video sequence, because video scenes often have many interferences such as dynamic background, hard shadows, lighting changes, camera shake, etc.
[0003] Deep convolutional neural networks have powerful feature extraction capabilities and can learn semantic representations from data. In particular, the performance of fully convolutional neural networks based on transfer learning surpasses traditional algorithms in foreground video separation.
[0004] However, existing supervised video target separation models lack the use of common information in dynamic scenes, or only use part of the common information, resulting in their performance being even worse than that of unsupervised video target separation. On the other hand, most models simply concatenate high-dimensional features internally, which makes it difficult to effectively improve the model's ability to discriminate different targets, resulting in a decrease in the accuracy of multi-target separation in videos. Summary of the invention
[0005] The existing object separation models for dynamic videos make use of too little common information of dynamic scenes, resulting in poor performance and low accuracy in multi-object separation.
[0006] In view of the above problems, a video processing method and system are proposed to solve them.
[0007] In a first aspect, a video processing method includes:
[0008] Step 100: Obtain a first frame image and a current frame image of a video, and input the first frame image and the current frame image into a convolutional neural network respectively;
[0009] Step 200: Perform feature mapping on the first frame image and the current frame image to obtain a first feature vector and a second feature vector respectively, and use the first feature vector and the second feature vector to obtain a weighting factor of the first feature vector to the second feature vector;
[0010] Step 300: weighting the second high-dimensional feature using the weighting factor to obtain a third high-dimensional feature, obtaining a differential feature of the first frame image relative to the current frame image using the first high-dimensional feature and the second high-dimensional feature, and aggregating the differential feature with the second high-dimensional feature to obtain a fourth high-dimensional feature;
[0011] Step 400: splice the third high-dimensional feature and the fourth high-dimensional feature according to channels to obtain a target feature of the current frame image, and input the target feature into the decoder of the convolutional neural network for decoding to obtain a current frame image mask.
[0012] In combination with the video processing method described in the first aspect of the present invention, in a first possible implementation manner, step 200 includes:
[0013] Step 210, respectively obtaining a first high-dimensional feature corresponding to the first frame image and a second high-dimensional feature corresponding to the current frame image;
[0014] Step 220, calculating the first feature vector corresponding to all feature points of the first high-dimensional feature, and calculating the second feature vector corresponding to all feature points of the second high-dimensional feature;
[0015] Step 230: Obtain weighting factors of feature points in the first feature vector with respect to corresponding feature points in the second feature vector, and use the weighting factors to obtain weighting matrices corresponding to all feature points.
[0016] In combination with the first possible implementation manner of the first aspect of the present invention, in a second possible implementation manner, step 300 includes:
[0017] Step 310: multiply the weighted matrix by the second high-dimensional feature, and perform weighting using a learning parameter to obtain a compensation feature;
[0018] Step 320: sum the compensation feature and the second high-dimensional feature to obtain a third high-dimensional feature.
[0019] In combination with the video processing method described in the first aspect of the present invention, in a third possible implementation manner, step 300 further includes:
[0020] Step 330: perform differential calculation on the first high-dimensional feature and the second high-dimensional feature to obtain differential features.
[0021] In a second aspect, a video processing system adopts the processing method described in the first aspect, wherein the video processing system is constructed based on a convolutional neural network and includes an image acquisition module, a first calculation module, a second calculation module, a feature extraction module, and a splicing module;
[0022] The image acquisition module is used to input the acquired first frame image and current frame image into the convolutional neural network;
[0023] The feature extraction module is used to perform feature mapping on the first frame image and the current frame image to obtain a first feature vector and a second feature vector respectively;
[0024] The first calculation module is used to obtain a weighting factor of the first feature vector relative to the second feature vector, and use the weighting factor to weight the second high-dimensional feature to obtain a third high-dimensional feature;
[0025] The second calculation module is used to obtain a differential feature of the first frame image relative to the current frame image and aggregate the differential feature with the second high-dimensional feature to obtain a fourth high-dimensional feature;
[0026] The splicing module is used to splice the third high-dimensional feature with the fourth high-dimensional feature according to channels to obtain the target feature of the current frame image, and input the target feature into the decoder of the convolutional neural network for decoding to obtain the current image mask.
[0027] In conjunction with the video processing system described in the second aspect of the present invention, in a first possible implementation manner, the feature extraction module is further used to:
[0028] The first high-dimensional features of the first frame image and the second high-dimensional features of the current frame image are respectively acquired through feature mapping.
[0029] In conjunction with the first possible implementation manner of the second aspect of the present invention, in a second possible implementation manner, the feature extraction module is further used to:
[0030] A first feature vector corresponding to all feature points of the first high-dimensional feature and a second feature vector corresponding to all feature points of the second high-dimensional feature are obtained.
[0031] In conjunction with the second possible implementation manner of the second aspect of the present invention, in a third possible implementation manner, the first calculation module is further used to:
[0032] The weighting factors of the feature points in the first feature vector with respect to the corresponding feature points in the second feature vector are calculated, and the weighting matrices corresponding to all the feature points are obtained using the weighting factors.
[0033] In conjunction with the first possible implementation manner of the second aspect of the present invention, in a fourth possible implementation manner, the first calculation module is further used to:
[0034] Multiplying the weighted matrix by the first high-dimensional feature and weighting it using a learning parameter to obtain a compensation feature;
[0035] The compensation feature is summed with the second high-dimensional feature to obtain a third high-dimensional feature.
[0036] In combination with the video processing system according to the second aspect of the present invention, in a fifth possible implementation manner, the second computing module is further used to:
[0037] A differential operation is performed on the first high-dimensional feature and the second high-dimensional feature to obtain a differential feature.
[0038] A video processing method and system described in the present invention is implemented. By using a convolutional neural network to perform feature mapping on the first frame image and the current frame image, and then obtaining the differential feature of the first high-dimensional feature to the second high-dimensional feature, the utilization of the common information of the previous and next frame images is increased, and the performance of supervised video target segmentation is improved. The second high-dimensional feature is compensated according to the weighting factor of the first feature vector to the second feature vector, which further improves the performance of video target segmentation processing and improves the separation accuracy of multiple targets. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0040] Figure 1 This is a first schematic diagram of steps of a video processing method of the present invention;
[0041] Figure 2 A second schematic diagram of steps of a video processing method according to the present invention;
[0042] Figure 3 A third schematic diagram of steps of a video processing method according to the present invention;
[0043] Figure 4 A schematic diagram of a video processing system of the present invention;
[0044] The names of the parts indicated by the numbers in the accompanying drawings are: 10 - image acquisition module, 20 - feature extraction module, 30 - first calculation module, 40 - second calculation module, 50 - splicing module. DETAILED DESCRIPTION
[0045] The following will be combined with the accompanying drawings in the invention to clearly and completely describe the technical solutions in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.
[0046] The existing object separation models for dynamic videos make use of too little common information of dynamic scenes, resulting in poor performance and low accuracy in multi-object separation.
[0047] In view of the above problems, a video processing method and system are proposed to solve them.
[0048] Example 1
[0049] In a first aspect, a video processing method is provided. Figure 1 , Figure 1 The first schematic diagram of a video processing method of the present invention comprises:
[0050] Step 100: Obtain the first frame image and the current frame image of the video, and input the first frame image and the current frame image into the convolutional neural network respectively. The temporal variation commonality information can provide important prior information for the foreground / background category judgment.
[0051] The convolutional neural network model may include an encoder and a decoder; the encoder may include an input layer, a convolution layer, a maximum pooling layer, and a feature extraction layer in sequence. The decoder may include a deconvolution layer and a fully connected layer. On the encoder side, the convolution layer uses the Relu activation function, and on the decoder side, the deconvolution layer uses the Relu activation function or the sigmoid activation function.
[0052] Step 200 , feature mapping is performed on the first frame image and the current frame image respectively to obtain a first feature vector and a second feature vector, and a weighting factor of the first feature vector to the second feature vector is obtained by using the first feature vector and the second feature vector.
[0053] Preferably, if Figure 2 , Figure 2 The second schematic diagram of the steps of a video processing method of the present invention; step 200 includes: step 210, respectively obtaining the first high-dimensional feature corresponding to the first frame image and the second high-dimensional feature corresponding to the current frame image; step 220, calculating the first feature vector corresponding to all feature points of the first high-dimensional feature, calculating the second feature vector corresponding to all feature points of the second high-dimensional feature; step 230, obtaining the weighting factor of the feature point in the first feature vector for the corresponding feature point in the second feature vector, and using the weighting factor to obtain the weighting matrix corresponding to all feature points. High-dimensional advanced features are conducive to characterizing the nature of foreground / background targets, are not easily disturbed by noise, and have more stable performance.
[0054] The first frame image and the current frame image are interdependent in spatial position, and their features also have certain similarities. The first frame image has a compensatory effect on the subsequent current frame image. Each feature point in the first high-dimensional feature and the second high-dimensional feature is regarded as a feature vector with a size of 1X1Xc, where c is the feature image depth.
[0055] Let T1 j 、T t iThey respectively represent the first eigenvector of the j-th feature point in the first high-dimensional feature and the second eigenvector of the i-th feature point in the second high-dimensional feature.
[0056] The weighting factor
[0057] The weighting matrix M of the feature vectors of all feature points in the first high-dimensional feature of the first frame image relative to the second high-dimensional feature of the current frame image is:
[0058] M=(w ij ) hw×hw ∈[0,1] hw×hw (2);
[0059] Among them, h and w are the rows and columns of the feature matrices of the first high-dimensional feature and the second high-dimensional feature, that is, the size of the high-dimensional feature.
[0060] Step 300: Use the weighting factor to weight the second high-dimensional feature to obtain a third high-dimensional feature, use the first high-dimensional feature and the second high-dimensional feature to obtain a differential feature of the first frame image relative to the current frame image, and aggregate the differential feature with the second high-dimensional feature to obtain a fourth high-dimensional feature.
[0061] Preferably, if Figure 3 , Figure 3 This is a third schematic diagram of the steps of a video processing method of the present invention; step 300 includes step 310, multiplying the weighted matrix with the second high-dimensional feature, and weighting it using the learning parameter to obtain a compensation feature; step 320, summing the compensation feature with the second high-dimensional feature to obtain a third high-dimensional feature.
[0062] By performing weighted compensation, the misjudged target features can be repaired.
[0063] Then the third high-dimensional feature F W =μ(M·T t )+T t (3);
[0064] Among them, μ is the learning parameter, T t is the second eigenvector.
[0065] Preferably, step 300 also includes:
[0066] Step 330: perform differential calculation on the first high-dimensional feature and the second high-dimensional feature to obtain a differential feature.
[0067] By obtaining differential features, it is easier to obtain discriminative feature points and improve segmentation performance.
[0068] Then the differential feature T D =T1-Tt (4);
[0069] Among them, is the first high-dimensional feature.
[0070] Then the fourth high-dimensional feature
[0071] Among them, the representation features are aggregated according to the channel dimension. After aggregation, the decoder can better distinguish dynamic features and static features. Dynamic features are generally target features, and static features are generally background features, thereby improving the segmentation performance.
[0072] Step 400: splice the third high-dimensional feature and the fourth high-dimensional feature according to channels to obtain the target feature of the current frame image, and input the target feature into the decoder of the convolutional neural network for decoding to obtain the current frame image mask.
[0073] Example 2
[0074] Second, as Figure 4 , Figure 4 A schematic diagram of a video processing system of the present invention; a video processing system, adopting the processing method of the first aspect, the video processing system is constructed based on a convolutional neural network, including an image acquisition module 10, a first calculation module 30, a second calculation module 40, a feature extraction module 20 and a splicing module 50;
[0075] The image acquisition module 10 is used to input the acquired first frame image and current frame image into the convolutional neural network;
[0076] The feature extraction module 20 is used to perform feature mapping on the first frame image and the current frame image to obtain a first feature vector and a second feature vector respectively;
[0077] The first calculation module 30 is used to obtain a weighting factor of the first feature vector relative to the second feature vector, and use the weighting factor to weight the second high-dimensional feature to obtain a third high-dimensional feature;
[0078] The second calculation module 40 is used to obtain a differential feature of the first frame image relative to the current frame image and aggregate the differential feature with the second high-dimensional feature to obtain a fourth high-dimensional feature;
[0079] The splicing module 50 is used to splice the third high-dimensional feature and the fourth high-dimensional feature according to channels to obtain the target feature of the current frame image, and input the target feature into the decoder of the convolutional neural network for decoding to obtain the current image mask.
[0080] Furthermore, the feature extraction module 20 is also used to obtain the first high-dimensional features of the first frame image and the second high-dimensional features of the current frame image respectively through feature mapping.
[0081] Further, the feature extraction module 20 is further used to obtain a first feature vector corresponding to all feature points of the first high-dimensional feature and a second feature vector corresponding to all feature points of the second high-dimensional feature;
[0082] Furthermore, the first calculation module 30 is further configured to calculate weighting factors of feature points in the first feature vector with respect to corresponding feature points in the second feature vector, and to obtain weighting matrices corresponding to all feature points using the weighting factors.
[0083] Furthermore, the first calculation module 30 is also used to multiply the weighted matrix with the first high-dimensional feature, and perform weighting using the learning parameter to obtain a compensation feature; and sum the compensation feature with the second high-dimensional feature to obtain a third high-dimensional feature.
[0084] Furthermore, the second calculation module 40 is also used to perform a differential operation on the first high-dimensional feature and the second high-dimensional feature to obtain a differential feature.
[0085] A video processing method and system implementing the present invention utilizes a convolutional neural network to perform feature mapping on a first frame image and a current frame image to thereby obtain differential features of a first high-dimensional feature to a second high-dimensional feature, thereby increasing the utilization of common information of previous and subsequent frame images and improving the performance of supervised video target segmentation. The second high-dimensional feature is compensated based on a weighting factor of the first feature vector to the second feature vector, thereby further improving the performance of video target segmentation processing and improving the accuracy of separation of multiple targets.
[0086] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A video processing method, characterized in that: include: Step 100: Obtain a first frame image and a current frame image of a video, and input the first frame image and the current frame image into a convolutional neural network respectively; Step 200: Perform feature mapping on the first frame image and the current frame image to obtain a first feature vector and a second feature vector respectively, and use the first feature vector and the second feature vector to obtain a weighting factor of the first feature vector to the second feature vector; Step 300: weighting the second high-dimensional feature using the weighting factor to obtain a third high-dimensional feature, obtaining a differential feature of the first frame image relative to the current frame image using the first high-dimensional feature and the second high-dimensional feature, and aggregating the differential feature with the second high-dimensional feature to obtain a fourth high-dimensional feature; Step 400: splicing the third high-dimensional feature and the fourth high-dimensional feature according to channels to obtain a target feature of the current frame image, and inputting the target feature into the decoder of the convolutional neural network for decoding to obtain a mask of the current frame image; The step 200 comprises: Step 210, respectively obtaining a first high-dimensional feature corresponding to the first frame image and a second high-dimensional feature corresponding to the current frame image; Step 220, calculating the first feature vector corresponding to all feature points of the first high-dimensional feature, and calculating the second feature vector corresponding to all feature points of the second high-dimensional feature; Step 230: Obtain weighting factors of feature points in the first feature vector for corresponding feature points in the second feature vector, and use the weighting factors to obtain weighting matrices corresponding to all feature points; Consider each feature point in the first high-dimensional feature and the second high-dimensional feature as a feature vector, , Respectively represent the first feature vector of the j-th feature point in the first high-dimensional feature and the second feature vector of the i-th feature point in the second high-dimensional feature; The weighting factor (1); The step 300 includes: Step 310: multiply the weighted matrix by the second high-dimensional feature, and perform weighting using a learning parameter to obtain a compensation feature; Step 320: sum the compensation feature and the second high-dimensional feature to obtain a third high-dimensional feature; Step 330: perform differential calculation on the first high-dimensional feature and the second high-dimensional feature to obtain differential features.
2. A video processing system, using the processing method according to claim 1, characterized in that: The video processing system is constructed based on a convolutional neural network, and includes an image acquisition module, a first calculation module, a second calculation module, a feature extraction module and a splicing module; The image acquisition module is used to input the acquired first frame image and current frame image into the convolutional neural network; the feature extraction module is used to perform feature mapping on the first frame image and the current frame image to obtain a first feature vector and a second feature vector respectively; the first calculation module is used to obtain a weighting factor of the first feature vector relative to the second feature vector, and use the weighting factor to weight the second high-dimensional feature to obtain a third high-dimensional feature; the second calculation module is used to obtain a differential feature of the first frame image relative to the current frame image and aggregate the differential feature with the second high-dimensional feature to obtain a fourth high-dimensional feature; The splicing module is used to splice the third high-dimensional feature with the fourth high-dimensional feature according to channels to obtain the target feature of the current frame image, and input the target feature into the decoder of the convolutional neural network for decoding to obtain the current image mask.
3. The video processing system according to claim 2, characterized in that: The feature extraction module is also used for: The first high-dimensional features of the first frame image and the second high-dimensional features of the current frame image are respectively acquired through feature mapping.
4. The video processing system according to claim 3, characterized in that: The feature extraction module is also used for: Obtain a first feature vector corresponding to all feature points of the first high-dimensional feature and a second feature vector corresponding to all feature points of the second high-dimensional feature.
5. The video processing system according to claim 4, characterized in that: The second calculation module is also used for: A differential operation is performed on the first high-dimensional feature and the second high-dimensional feature to obtain a differential feature.