Video snow removal method and device, equipment and medium
Through the multi-layer perceptron network and Gaussian hybrid model of the video snow removal system, the video frame feature map is extracted and the super positive sample synthesis snow layer is constructed, which solves the problem of difficulty in removing snow patterns in videos in the existing technology, and realizes the video frame Improve the clarity of snow removal and snow scenes.
Patent Information
- Application Number
- CN202411216748.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-30
AI Technical Summary
The prior art is difficult to effectively remove snow patterns in videos, resulting in a decrease in visibility of video frames. Especially when dealing with real snowfall with unpredictable shapes and motions, it is impossible to effectively improve the clarity of the snow scene.
Using a video snow removal system, through the combination of encoder, decoupling module and decoder, the video frame feature map is extracted, a multi-layer perceptron network is constructed, semi-supervised training is performed, super positive sample synthesis snow layer is generated, combined with Gaussian hybrid model, Remove snow patterns and improve video clarity.
Effectively remove snow patterns in the video, improve the clarity of snow scenes, enhance perception fidelity, and is suitable for video processing in various real scenes.
Smart Images

Figure CN119007086A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a video snow removal method, device, equipment and medium. Background Art
[0002] Snowfall, as a kind of severe weather, often appears in outdoor videos. The degradation effect caused by snow particles and streaks seriously reduces the visibility of video frames, which in turn hinders the advanced performance of video processing algorithms in the supporting system. In the prior art, due to the domain distribution difference between synthetic data and real-world data, existing rudimentary methods are inevitably hindered by unrealistic training data and cannot handle real snowfall with unpredictable shapes and movements, resulting in blurry snow scene content in the video and low perceived fidelity. Summary of the invention
[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a video snow removal method, device, equipment and medium, which can remove snow patterns in the video and improve the clarity of the snow scene in the video.
[0004] In a first aspect, an embodiment of the present invention provides a video snow removal method, which is applied to a video snow removal system. The video snow removal system includes an encoder, a first decoupling module and a decoder. The first decoupling module is connected to the encoder and the decoder. The video snow removal method includes:
[0005] Obtaining a snow video sequence, inputting the snow video sequence into the encoder, and extracting a video frame feature map;
[0006] Inputting the video frame feature map into the first decoupling module, obtaining physical features constituting the snow video sequence, and removing useless degraded layers in the physical features to obtain a spatial feature map;
[0007] Constructing a first network and a second network, obtaining label data of the spatial feature map, and inputting the label data into the first network and / or the second network according to a preset supervision stage to perform the snow removal task of the video frame feature map, and obtaining a first snow removal result and a second snow removal result;
[0008] constructing a positive sample, an anchor sample, and a negative sample according to the first snow removal result and the second snow removal result, and replacing the video frame feature map according to the positive sample, the anchor sample, and the negative sample to obtain a super positive sample;
[0009] The super positive sample synthetic snow layer is obtained, and the super positive sample synthetic snow layer is input into the decoder to obtain a prediction result image.
[0010] In some embodiments of the present invention, the video snow removal system is further provided with a second transformer, the second transformer is provided with a multi-layer perceptron block, the decoupling module is provided with a physical transformer block and a first processing unit, and the obtaining of physical features constituting the snow video sequence includes:
[0011] Performing overlapping block embedding on the video frame feature map to obtain a linear block of the video frame feature map;
[0012] Obtaining the number of markers and the number of marker channels of the video frame feature map, and embedding a first marker in the linear block according to the number of markers and the number of marker channels to obtain a marked linear block;
[0013] The labeled linear block is input into the physical transformer block for physical information separation, so as to integrate the first decoupling module into the second transformer, replace the multi-layer perceptron block, and obtain the second decoupling module.
[0014] In some embodiments of the present invention, the video snow removal system is further provided with a time sequence decomposition router, and after the first decoupling module replaces the multi-layer perceptron block, the method further includes:
[0015] Acquire multiple input tags of the video frame feature map, wherein each of the input tags is provided with a corresponding parameter vector;
[0016] Obtaining a timing dimension of the timing decomposition router, and obtaining an adaptive weight of the video frame feature map according to the input tag and the timing dimension;
[0017] Obtaining a first convex combination of the video frame feature map according to the adaptive weight and the input tag, wherein the first convex combination includes the adaptive timing information of the timing dimension;
[0018] Obtaining component labels of the video frame feature graph according to the first function preset in the second decoupling module and the adaptive timing information;
[0019] The temporal decomposition router dynamically decodes the physical features according to the component labels and the adaptive weights to obtain a first output label, and uses the first output label as a second convex combination of the video frame feature map.
[0020] In some embodiments of the present invention, inputting the tag data into the first network and / or the second network according to a preset supervision stage includes:
[0021] Performing semi-supervised processing on the spatial feature map according to the first network and the second network to obtain labeled data and first unlabeled data of the first network, and second unlabeled data of the second network;
[0022] Acquire a first background feature and a first snow feature of the labeled data, a second background feature and a second snow feature of the first unlabeled data, and a third background feature and a third snow feature of the second unlabeled data;
[0023] Combining the second background feature with the first snow feature as a positive sample, combining the third background feature with the second snow feature as an anchor sample, and combining the first background feature with the third snow feature as a negative sample;
[0024] A contrast loss is calculated based on the positive sample, the anchor sample, and the negative sample.
[0025] In some embodiments of the present invention, the performing semi-supervised processing on the spatial feature map according to the first network and the second network includes:
[0026] Acquire complementary information according to the labeled data and the first unlabeled data;
[0027] Acquire a first supervised loss and a second supervised loss of the first network and an average moving index of the second network according to the complementary information;
[0028] The first network is updated according to the first supervised loss and the second supervised loss, and the second network is updated according to the average moving index.
[0029] In some embodiments of the present invention, obtaining the synthetic snow layer of the super positive sample includes:
[0030] Inputting the super positive samples into a preset Gaussian mixture model, quantizing the super positive samples, and obtaining a real snow layer distribution and a synthetic snow layer distribution;
[0031] Calculating the divergence of the real snow layer distribution and the synthetic snow layer distribution to obtain a super positive sample synthetic snow layer;
[0032] The contrast loss of the spatial feature map is calculated based on the super positive sample synthetic snow layer, the positive sample and the negative sample.
[0033] In some embodiments of the present invention, after calculating the contrast loss according to the positive sample, the anchor sample and the negative sample, the method further includes:
[0034] When the supervision stage is full supervision training, the label data is input into the first network to obtain a restored frame and a clean frame of the spatial feature map, and a pixel-by-pixel supervision loss of the spatial feature map is calculated according to the restored frame and the clean frame;
[0035] When the supervision stage is semi-supervised training, the first unlabeled data is input into the first network and the second network respectively, and the pixel-by-pixel consistency loss, perceptual contrast loss and prior loss of the spatial feature map are calculated.
[0036] In a second aspect, an embodiment of the present invention provides a video snow removal device, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the video snow removal method described in the first aspect above.
[0037] In a third aspect, an embodiment of the present invention provides an electronic device, comprising the video snow removal device as described in the second aspect above.
[0038] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the video snow removal method as described in the first aspect above.
[0039] The video snow removal method according to the embodiment of the present invention has at least the following beneficial effects:
[0040] Obtain a snow video sequence, input the snow video sequence into the encoder, and extract a video frame feature map; input the video frame feature map into the first decoupling module, obtain the physical features constituting the snow video sequence, and remove the useless degenerate layers in the physical features to obtain a spatial feature map; construct a first network and a second network, obtain the label data of the prediction result, and input the label data into the first network and / or the second network according to the preset supervision stage to perform the snow removal task of the video frame feature map to obtain the first snow removal result and the second snow removal result; construct positive samples, anchor samples and negative samples according to the first snow removal result and the second snow removal result, replace the video frame feature map according to the positive samples, anchor samples and negative samples to obtain a super positive sample; obtain a super positive sample synthesized snow layer, input the super positive sample synthesized snow layer into the decoder, and obtain a prediction result image. According to the technical solution of this embodiment, the snow pattern in the video can be removed and the clarity of the snow scene in the video can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 is a flow chart of a video snow removal method provided by an embodiment of the present invention;
[0042] Figure 2 is a flow chart of obtaining physical features constituting a snow video sequence provided by an embodiment of the present invention;
[0043] Figure 3 is a flow chart after replacing the multi-layer perceptron block provided by an embodiment of the present invention;
[0044] Figure 4 is a flow chart of inputting label data into a first network and / or a second network according to a preset supervision stage provided by an embodiment of the present invention;
[0045] Figure 5 is a flowchart of performing semi-supervised processing on a spatial feature map according to a first network and a second network provided by an embodiment of the present invention;
[0046] Figure 6 is a flow chart of obtaining a synthetic snow layer of a super positive sample provided by an embodiment of the present invention;
[0047] Figure 7 is a flowchart after calculating the contrast loss according to the positive samples, anchor samples and negative samples provided by an embodiment of the present invention;
[0048] Figure 8 It is a structural diagram of a video snow removal device provided by another embodiment of the present invention. DETAILED DESCRIPTION
[0049] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0050] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
[0051] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0052] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0053] An embodiment of the present invention provides a video snow removal method, which is applied to a video snow removal system. The video snow removal system includes an encoder, a first decoupling module and a decoder. The first decoupling module is connected to the encoder and the decoder. The video snow removal method includes obtaining a snow video sequence, inputting the snow video sequence into the encoder, and extracting a video frame feature map; inputting the video frame feature map into the first decoupling module, obtaining physical features constituting the snow video sequence, and removing useless degraded layers in the physical features to obtain a spatial feature map; constructing a first network and a second network, obtaining label data of the spatial feature map, and inputting the label data into the first network and / or the second network according to a preset supervision stage to perform the snow removal task of the video frame feature map to obtain a first snow removal result and a second snow removal result; constructing positive samples, anchor samples and negative samples according to the first snow removal result and the second snow removal result, replacing the video frame feature map according to the positive samples, the anchor samples and the negative samples to obtain a super positive sample; obtaining a super positive sample synthesized snow layer, and inputting the super positive sample synthesized snow layer into the decoder to obtain a predicted result image. According to the technical solution of this embodiment, the snow pattern in the video can be effectively removed, and the clarity of the snow scene in the video can be improved. By developing a video snow removal system based on the Mean-Teacher structure, the video snow removal system consists of an encoder, a first decoupling module and an encoder. By using a mixer composed of synthetic data and real data, based on the Mean-Teacher architecture, unlabeled real-world data is introduced in a semi-supervised manner during the training phase to enhance the generalization ability of the system in various real scenes. By introducing a distribution-driven contrast regularization method, the deep model is prevented from being disturbed by the shape and motion changes of different snow in synthetic data and real data; further, by using the likelihood line of the Gaussian mixture model to capture the synthetic snow layer that is most similar to real snow, the acquisition of super-positive samples is achieved by approximating the distribution of real snow components. By replacing the snow layer in the positive sample with the super-positive sample snow layer and replacing the background in the negative sample, the snow-invariant information learning is maintained and highlighted. In turn, the restoration reflected in the snow video sequence is improved, and clearer content and enhanced perceptual fidelity can be provided in both synthetic and real snow scenes.
[0054] The control method of the embodiment of the present invention is further described below based on the accompanying drawings.
[0055] Reference Figure 1 , Figure 1 A flowchart of a video snow removal method provided by an embodiment of the present invention, the video snow removal method includes but is not limited to the following steps:
[0056] Step S11, obtaining a snow video sequence, inputting the snow video sequence into an encoder, and extracting a video frame feature map;
[0057] It should be noted that when the video snow removal system performs snow removal operation on the video, the snow video sequence is first input into the encoder, and the encoder uses a general ConvNeXt backbone network as an encoder to extract the video frame feature map. In this embodiment, the video snow removal system based on the ConvNeXt encoder can accurately identify the snow area in the video and effectively remove the snow interference area in the video. Through precise feature extraction and efficient computational processing, the system can restore clearer and more realistic video content and improve the visual quality of the video.
[0058] Step S12, inputting the video frame feature map into the first decoupling module, obtaining the physical features constituting the snow video sequence, and removing the useless degraded layers in the physical features to obtain a spatial feature map;
[0059] It should be noted that in order to enhance the decoupling capability of the first decoupling module, a physical converter block in the video system is first defined. The physical converter block is provided with multiple processing units. The snow video sequence is decomposed into multiple physical features through multiple processing units and input into the first decoupling module. The first decoupling module can identify and separate the physical features in the video frame feature map, such as the dynamic features (falling speed and direction), static features (snow thickness and snow distribution) and background features (such as buildings, trees, etc.) of snowflakes, and remove the useless degraded layers in the physical features to obtain the spatial feature map of the snow video.
[0060] It should be noted that after obtaining the spatial feature map, each component feature of the second decoupling module is concatenated with the merged corresponding features and then input into the respective decoders to enhance the component features. The three decoders are composed of three convolutional layers with upsampling. Finally, these enhanced features are used for the final restoration in the subsequent prior-guided recovery module. We use the first formula to remove snow from the frame at the same time, thereby decomposing the frame into three different components S, A and T in the feature space.
[0061] The first formula is:
[0062] I snow (x)=J(x)T(x)+A(x)(1-T(x))+S(x);
[0063] Among them, I snow represents the video corrupted by snowfall, J is the corresponding clear video, T is the transmission map, A is the atmospheric light, and S is the snowfall map.
[0064] According to the first formula, the prior-based recovery process can be expressed as:
[0065]
[0066] Among them, S is the first composition feature, A is the second composition feature, and T is the third composition feature.
[0067] Step S13, constructing a first network and a second network, obtaining label data of the prediction result, and inputting the label data into the first network and / or the second network according to a preset supervision stage to perform the snow removal task of the video frame feature map, and obtaining a first snow removal result and a second snow removal result;
[0068] It should be noted that the first network and the second network are constructed, and the first snow removal result and the second snow removal result of the background features of the video frame after snow removal are obtained through different paths or processing strategies of the first network and the second network. At the same time, the supervised learning of the label data ensures the effectiveness and accuracy of the snow removal task.
[0069] Step S14, constructing positive samples, anchor samples and negative samples according to the first snow removal result and the second snow removal result, replacing the video frame feature map according to the positive samples, anchor samples and negative samples to obtain super positive samples;
[0070] It should be noted that positive samples, anchor samples and negative samples are constructed through the first snow removal results and the second snow removal results, and the video frame feature map is replaced based on multiple samples to obtain super positive samples, which further improves the generalization ability and snow removal effect of the video snow removal system, and generates super positive samples to improve the processing quality of video snow removal.
[0071] Step S15, obtaining a super positive sample synthetic snow layer, inputting the super positive sample synthetic snow layer into a decoder, and obtaining a predicted result image.
[0072] It should be noted that by synthesizing the snow layer and inputting it into the decoder, the snowflake layer in the real environment is simulated and the snowflake layer is removed to make the predicted result image closer to the actual scene, thereby achieving effective snow removal of the snow video sequence and significantly improving the quality and clarity of the predicted result image.
[0073] It should be noted that this embodiment maintains and highlights the snow-invariant information learning by replacing the snow layer with the super-positive sample snow layer in the positive sample and replacing the background in the negative sample conversely, thereby improving the recovery reflected in the snow video sequence, and providing clearer content and enhanced perceptual fidelity in both synthetic and real snow scenes.
[0074] In addition, in one embodiment, the video snow removal system is further provided with a second transformer, the second transformer is provided with a multi-layer perceptron block, and the decoupling module is provided with a physical transformer block and a first processing unit, referring to Figure 2 ,exist Figure 1 Step S12 of the illustrated embodiment also includes but is not limited to the following steps:
[0075] Step S21, performing overlapping block embedding on the video frame feature map to obtain a linear block of the video frame feature map;
[0076] Step S22, obtaining the number of markers and the number of marker channels of the video frame feature map, and embedding the first marker in the linear block according to the number of markers and the number of marker channels to obtain a marked linear block;
[0077] Step S23, inputting the labeled linear block into the physical transformer block for physical information separation, so as to integrate the first decoupling module into the second transformer, replace the multi-layer perceptron block, and obtain the second decoupling module.
[0078] It should be noted that the encoder obtains the video frame feature map and performs overlapping block embedding on each video frame feature map to obtain the linear block of the video frame feature map, and embeds all the linear blocks into the first label, where the first label is Y∈R(Nf□m)×d, m is the number of labels in a frame, and d is the label channel. After that, Y is input into the transformer block for separation of physical dependency information. In the physical transformer block, the feedforward network of the first transformer is first improved by fusing the feedforward network to enhance feature fusion. The first decoupling module is integrated into the second transformer of the physical transformer block, replacing the multilayer perceptron block. The proposed module contains specific units corresponding to different physical components, namely snow units, transmission map units, and atmospheric light units.
[0079] In addition, in one embodiment, the video snow removal system is further provided with a time sequence decomposition router, referring to Figure 3 ,exist Figure 2 After step 23 of the illustrated embodiment, the following steps are also included but not limited to:
[0080] Step S31, obtaining a plurality of input tags of a video frame feature map, wherein each input tag is provided with a corresponding parameter vector;
[0081] Step S32, obtaining the timing dimension of the timing decomposition router, and obtaining the adaptive weight of the video frame feature map according to the input mark and the timing dimension;
[0082] Step S33, obtaining a first convex combination of the video frame feature map according to the adaptive weight and the input mark, wherein the first convex combination includes adaptive timing information of the timing dimension;
[0083] Step S34, obtaining component labels of the video frame feature graph according to the first function and adaptive timing information preset in the second decoupling module;
[0084] Step S35, the time decomposition router dynamically decodes the physical features according to the component labels and the adaptive weights to obtain a first output label, and uses the first output label as a second convex combination of the video frame feature map.
[0085] It should be noted that the input tag of a sequence is represented as Z∈R(Nf□m)×d. The first function is specifically expressed as {fj:Rd→Rd}j=1n. The snow unit, transmission map unit and atmospheric light unit each process a timing adaptive tag, and each tag has a corresponding d-dimensional parameter vector, which is represented as Γ∈Rd×n. The timing adaptive weight Qi j is obtained according to the timing dimension Nf□m in the timing decomposition router. The adaptive weight can be expressed by the following formula:
[0086]
[0087] Therefore, the weighted label is obtained according to the input label, the first convex combination and the adaptive timing information, and the weighted label can be expressed by the following formula:
[0088]
[0089] in, is the weighted label, Q T is the first convex combination, and Z is the input mark.
[0090] Then, the first function is applied to each input token to obtain the output component token, which can be expressed by the following formula:
[0091]
[0092] Based on the time decomposition router, the physical features are dynamically decoded according to the component labels and decomposition weights to obtain the output labels. The output label C is calculated as the second convex combination of all output component labels. The second convex combination can be expressed by the following formula:
[0093]
[0094] Where D is the decomposition weight, i.e., the softmax result of Z·Γ on the expert dimension. The sparse mixed expert algorithm in the prior art is usually discrete and therefore non-differentiable. However, the second decoupling module in this embodiment adopts continuous and differentiable operations. The physical feature information obtained by all time tags and units is effectively utilized to extract physics-specific features.
[0095] In addition, in one embodiment, referring to Figure 4 ,exist Figure 1 Step S13 of the illustrated embodiment also includes but is not limited to the following steps:
[0096] Step S41, performing semi-supervised processing on the spatial feature map according to the first network and the second network to obtain labeled data and first unlabeled data of the first network, and second unlabeled data of the second network;
[0097] Step S42, obtaining a first background feature and a first snow feature of the labeled data, a second background feature and a second snow feature of the first unlabeled data, and a third background feature and a third snow feature of the second unlabeled data;
[0098] Step S43, combining the second background feature and the first snow feature as a positive sample, combining the third background feature and the second snow feature as an anchor sample, and combining the first background feature and the third snow feature as a negative sample;
[0099] Step S44, calculating the contrast loss based on the positive samples, anchor samples and negative samples.
[0100] It should be noted that the first background feature is obtained from the labeled data in the first network and first snow features And obtaining a second background feature from the first unlabeled data in the first network and second snow feature In addition, we also obtain the third background features from the second unlabeled data in the second network and the third snow feature According to the synthesis formula of snow I S (x) = J(x) + S(x), which recombines the background features and snow layer features from the labeled data, the first unlabeled data, and the second unlabeled data. In order to keep and highlight the information unrelated to snow, we replace the corresponding parts with snow features in the positive samples, and replace them with background features in the negative samples. Specifically, we replace and The combination of is set as a positive sample, and The combination of is set as anchor sample, and enhanced The combination of is set as a negative sample, and the contrast loss is calculated based on the recombined positive samples, anchor samples and negative samples.
[0101] In addition, in one embodiment, referring to Figure 5 ,exist Figure 4 Step S41 of the illustrated embodiment also includes but is not limited to the following steps:
[0102] Step S51, acquiring complementary information according to the marked data and the first unmarked data;
[0103] Step S52, obtaining a first supervised loss and a second supervised loss of the first network and an average moving index of the second network according to the complementary information;
[0104] In step S53, the first network is updated according to the first supervisory loss and the second supervisory loss, and the second network is just updated according to the average moving index.
[0105] It should be noted that in order to enhance the generalization ability of the system on real data, we introduced semi-supervised learning in the video snow removal task. Semi-supervised learning enables the learning system to obtain complementary information from labeled synthetic data and unlabeled real data. Our semi-supervised learning framework follows the typical setting. During training, the first network is updated by minimizing the supervised loss and the unsupervised loss, while the second network updates the supervised loss by exponential moving average, which can be expressed by the following formula:
[0106] θ' teacher =ηθ teacher +(1-η)θ student ;
[0107] In order to constrain the output of the first network, Charbonnier loss and perceptual loss are used to improve the visual quality of the restored results. Features are extracted from the 3rd, 8th and 15th layers of the pre-trained VGG-16 to calculate the perceptual loss. FocalFrequency loss is introduced to focus on the different artifacts of the model's spectral response to different regions in the image, which can be expressed by the following formula:
[0108]
[0109] The unsupervised loss uses the pixel-level Charbonnier loss as the consistency loss of the unsupervised first network and the second network to ensure that the two networks generate consistent results, which can be expressed by the following formula:
[0110]
[0111] In addition, in one embodiment, referring to Figure 6 ,exist Figure 1 Step S15 of the illustrated embodiment also includes but is not limited to the following steps:
[0112] Step S61, inputting the super positive sample into a preset Gaussian mixture model, quantizing the super positive sample, and obtaining a real snow layer distribution and a synthetic snow layer distribution;
[0113] Step S62, calculating the divergence of the real snow layer distribution and the synthetic snow layer distribution to obtain a super positive sample synthetic snow layer;
[0114] Step S63, calculating the contrast loss of the spatial feature map based on the synthesized snow layer of the super positive sample, the positive sample and the negative sample.
[0115] It should be noted that due to the distribution difference between synthetic snow and real snow, the snow layers generated by our network often show obvious differences. Therefore, our goal is to obtain a super positive sample that represents a synthetic snow layer whose characteristics are very close to the real snow layer to be used as a positive sample. Since real snow has inherently diverse structures due to its different generation states and observation angles, they can be represented by a Gaussian mixture model (GMM). Using GMM can accurately approximate the distribution of real snow layers and effectively capture multiple patterns in the data. The distribution of snow layers can be expressed as:
[0116]
[0117] The contrast loss can be expressed by the following formula:
[0118]
[0119] In addition, in one embodiment, referring to Figure 7 ,exist Figure 1 Step S11 of the illustrated embodiment also includes but is not limited to the following steps:
[0120] Step S71, when the supervision stage is full supervision training, the label data is input into the first network to obtain the restored frame and the clean frame of the spatial feature map, and the pixel-by-pixel supervision loss of the spatial feature map is calculated according to the restored frame and the clean frame;
[0121] Step S72, when the supervision stage is semi-supervised training, the first unlabeled data is input into the first network and the second network respectively, and the pixel-by-pixel consistency loss, perceptual contrast loss and prior loss of the spatial feature map are calculated.
[0122] It should be noted that since the comparison is directly based on pixel values, the pixel-by-pixel supervision loss can ensure that the restored frame is as close to the real frame as possible visually, which is more effective for detail preservation and color accuracy. By comparing the outputs of the first network and the second network for the same input, it simulates the supervision effect of the label data to a certain extent, thereby increasing the robustness and generalization ability of the model.
[0123] Furthermore, in the supervised training phase, the labeled data is fed into the student network and a pixel-wise supervised loss is calculated between the restored and clean frames. In the semi-supervised training phase, the unlabeled data is fed into the first and second networks and pixel-wise consistency loss, perceptual contrast loss, and prior loss are calculated to regularize the student network. In addition, we also utilize a distribution-driven contrastive regularization loss to prevent the model from being negatively affected by the distribution difference between synthetic and real data.
[0124] like Figure 8 As shown, Figure 8: is a structural diagram of a video snow removal device provided by an embodiment of the present invention. The present invention also provides a video snow removal device, comprising:
[0125] The processor 801 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0126] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the video snow removal method of the embodiment of this application;
[0127] Input / output interface 803, used to implement information input and output;
[0128] The communication interface 804 is used to realize the communication interaction between the present apparatus and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);
[0129] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);
[0130] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .
[0131] An embodiment of the present application also provides an electronic device, including the video snow removal device as described above.
[0132] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned video snow removal method is implemented.
[0133] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are implemented to be located in one place, or may also be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.
[0134] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0135] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.
Claims
1. A video snow removal method, characterized in that: Applied in a video snow removal system, the video snow removal system includes an encoder, a first decoupling module and a decoder, the first decoupling module is connected to the encoder and the decoder, and the video snow removal method includes: Obtaining a snow video sequence, inputting the snow video sequence into the encoder, and extracting a video frame feature map; Inputting the video frame feature map into the first decoupling module, obtaining physical features constituting the snow video sequence, and removing useless degraded layers in the physical features to obtain a spatial feature map; Constructing a first network and a second network, obtaining label data of the prediction result, and inputting the label data into the first network and / or the second network according to a preset supervision stage to perform the snow removal task of the video frame feature map, and obtaining a first snow removal result and a second snow removal result; constructing a positive sample, an anchor sample, and a negative sample according to the first snow removal result and the second snow removal result, and replacing the video frame feature map according to the positive sample, the anchor sample, and the negative sample to obtain a super positive sample; The super positive sample synthetic snow layer is obtained, and the super positive sample synthetic snow layer is input into the decoder to obtain a prediction result image.
2. The video snow removal method according to claim 1, characterized in that: The video snow removal system is further provided with a second transformer, the second transformer is provided with a multi-layer perceptron block, the decoupling module is provided with a physical transformer block and a first processing unit, and the obtaining of physical features constituting the snow video sequence includes: Performing overlapping block embedding on the video frame feature map to obtain a linear block of the video frame feature map; Obtaining the number of markers and the number of marker channels of the video frame feature map, and embedding a first marker in the linear block according to the number of markers and the number of marker channels to obtain a marked linear block; The labeled linear block is input into the physical transformer block for physical information separation, so as to integrate the first decoupling module into the second transformer, replace the multi-layer perceptron block, and obtain the second decoupling module.
3. The video snow removal method according to claim 2, characterized in that: The video snow removal system is also provided with a time sequence decomposition router. After the first decoupling module replaces the multi-layer perceptron block, the method further includes: Acquire multiple input tags of the video frame feature map, wherein each of the input tags is provided with a corresponding parameter vector; Obtaining a timing dimension of the timing decomposition router, and obtaining an adaptive weight of the video frame feature map according to the input tag and the timing dimension; Obtaining adaptive timing information of the video frame feature map according to the adaptive weight and the input mark; Acquire component labels of the video frame feature graph according to the first function preset in the second decoupling module and the adaptive timing information; The temporal decomposition router dynamically decodes the physical features according to the component labels and the adaptive weights to obtain a first output label, and uses the first output label as a convex combination of the video frame feature map.
4. The video snow removal method according to claim 1, characterized in that: The step of inputting the tag data into the first network and / or the second network according to a preset supervision stage comprises: Performing semi-supervised processing on the spatial feature map according to the first network and the second network to obtain labeled data and first unlabeled data of the first network, and second unlabeled data of the second network; Acquire a first background feature and a first snow feature of the labeled data, a second background feature and a second snow feature of the first unlabeled data, and a third background feature and a third snow feature of the second unlabeled data; Combining the second background feature with the first snow feature as a positive sample, combining the third background feature with the second snow feature as an anchor sample, and combining the first background feature with the third snow feature as a negative sample; A contrast loss is calculated based on the positive sample, the anchor sample, and the negative sample.
5. The video snow removal method according to claim 4, characterized in that: The performing semi-supervised processing on the spatial feature map according to the first network and the second network includes: Acquire the first unlabeled data to acquire complementary information according to the labeled data and the first unlabeled data; Acquire a first supervised loss and a second supervised loss of the first network and an average moving index of the second network according to the complementary information; The first network is updated according to the first supervisory loss and the second supervisory loss, and the second network is updated according to the average moving index.
6. The video snow removal method according to claim 1, characterized in that: The step of obtaining the synthetic snow layer of the super positive sample comprises: Inputting the super positive samples into a preset Gaussian mixture model, quantizing the super positive samples, and obtaining a real snow layer distribution and a synthetic snow layer distribution; Calculating the divergence of the real snow layer distribution and the synthetic snow layer distribution to obtain a super positive sample synthetic snow layer; The contrast loss of the spatial feature map is calculated based on the super positive sample synthetic snow layer, the positive sample and the negative sample.
7. The video snow removal method according to claim 4, characterized in that: After calculating the contrast loss according to the positive sample, the anchor sample and the negative sample, the method further includes: When the supervision stage is full supervision training, the label data is input into the first network to obtain a restored frame and a clean frame of the spatial feature map, and a pixel-by-pixel supervision loss of the spatial feature map is calculated according to the restored frame and the clean frame; When the supervision stage is semi-supervised training, the first unlabeled data is input into the first network and the second network respectively, and the pixel-by-pixel consistency loss, perceptual contrast loss and prior loss of the spatial feature map are calculated.
8. A video snow removal device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the video snow removal method as described in any one of claims 1 to 7.
9. An electronic device, characterized in that: Including the video snow removal device as described in claim 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the video snow removal method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Method and system for unmanned vehicle to identify traffic sign in extreme weather
CN116597411A
System and method for generating derained image using self-supervised learning model
US20230385994A1
