Weakly Supervised Learning Video Freeze Semantic Segmentation Method, Device and Storage Device

Abstract

In view of the problems such as difficult data acquisition and data annotation in the video freeze-frame image segmentation problem, the present invention provides a weakly supervised learning-based video freeze-frame semantic segmentation method, device, and storage device. The method includes: preparing a data set, initializing a multi-expert self-distillation network, training the network using the data set to obtain a feature extraction module, initializing a semantic segmentation model, and training the semantic segmentation model using the data set to obtain a freeze-frame image segmentation result; the device and the storage device are used to implement the method. The beneficial effect of the present invention is that it can solve the segmentation problem of video freeze-frame images to a certain extent.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and machine learning, and in particular to a method, device, and storage device for semantic segmentation of video flower screens using weakly supervised learning. Background Art

[0002] The networking and application of public security video surveillance systems is of great significance for improving urban and rural management. In recent years, various regions have vigorously promoted the construction of video surveillance systems, which have played a positive role in combating crime, preventing public security, promoting social management, and serving the people's livelihood. However, due to factors such as network transmission performance limitations, codec device defects, or improper human operation, video frames may be corrupted during transmission and storage, manifesting as abnormal images (distorted screens) after decoding. These image anomalies affect the efficiency of public security video surveillance systems and hinder the development of social security prevention and control systems.

[0003] Weakly supervised learning is a branch of machine learning that, compared to traditional supervised learning, uses limited data to train model parameters. In the practice of image smudge recognition, fully supervised machine learning requires a large amount of labeled image smudge data. Acquiring such data often requires high labor costs, making fully supervised image smudge recognition training difficult to implement. We hope to train an accurate model using a small amount of sample data at a low cost, and weakly supervised learning meets this need. Summary of the Invention

[0004] The present invention addresses the problems of video flower screen detection, such as difficulty in data labeling, multiple data types, and complex data labeling, and provides a weakly supervised learning video flower screen semantic segmentation method, device, and storage device. The technical solution adopted by the method is as follows:

[0005] S1. Prepare dataset.

[0006] S2, initialize the multi-expert self-distillation network;

[0007] S3. Use the dataset to train a multi-expert self-distillation network to obtain a feature extraction module.

[0008] S4, initialize the semantic segmentation model;

[0009] S5. Use the dataset to train a semantic segmentation model to obtain the flower screen image segmentation result.

[0010] Furthermore, the data set includes a first data set and a second data set, the first data set is an unlabeled image set, and the second data set is a labeled image set.

[0011] Furthermore, the multi-expert self-distillation network is divided into an expert1 module, an expert2 module, and a member module.

[0012] Furthermore, training the multi-expert self-distillation network with the first dataset is characterized by the following specific steps:

[0013] S31. Initialize the multi-expert self-distillation network, including the expert1 module, the expert2 module, and the member module, with the same initial weights for the three modules;

[0014] S32. Crop the first dataset to obtain image data with different sizes;

[0015] S33. Put the image data with different sizes into the multi-expert self-distillation network and substitute the result into the loss function;

[0016] S34. Update the multi-expert self-distillation network; the member module updates the parameters through backpropagation according to the loss function, and the parameters of the expert1 module and the expert2 module are updated using the EMA method.

[0017] S35. Use the trained expert1 module as the feature extraction module.

[0018] Furthermore, the semantic segmentation model is divided into a feature extraction module and a segmentation module. The specific initialization method is that the feature extraction network is obtained from the training in S3, and the segmentation part consists of a horizontal attention module, a feature fusion module, and a segmentation head.

[0019] Furthermore, the steps for training the semantic segmentation model are as follows:

[0020] S51. The feature extraction module extracts and outputs feature maps from the second dataset;

[0021] S52. The horizontal extraction module extracts horizontal attention from the feature maps;

[0022] S53. The feature fusion module fuses the horizontal attention into the feature maps;

[0023] S54. The segmentation head maps the fused feature maps to the target space to obtain the semantic segmentation result of the mosaic screen.

[0024] A storage device stores instructions and data for implementing a method for weakly supervised learning of video mosaic screen semantic segmentation.

[0025] A weakly supervised video screen freeze semantic segmentation device, comprising: a processor and the storage device; the processor loads and executes instructions and data in the storage device to implement a weakly supervised learning video screen freeze semantic segmentation method. The beneficial technical effects of the present invention are as follows: The present invention uses a weakly supervised method, significantly reducing the annotation operation cost of the training set in the traditional supervised learning method, fully improving the overall utilization rate of a small amount of labeled and a large amount of unlabeled data, and reducing the threshold of deep learning in practical applications. At the same time, aiming at the visual features of image screen freeze, a horizontal spatial attention module is added, which can help the model focus on the important parts in the sequence. From the above two aspects, the present invention effectively promotes the industrial implementation of the weakly supervised training method, providing new feasibility for solving the problem of semantic segmentation of surveillance video image screen freeze. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic flowchart of an embodiment of the method of the present invention;

[0027] Figure 2 is a schematic diagram of the training process of the multi-expert self-distillation network in an embodiment of the method of the present invention;

[0028] Figure 3 is a schematic diagram of the segmentation network structure in an embodiment of the method of the present invention;

[0029] Figure 4 is a schematic diagram of the working of the hardware device in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] To further understand the present invention, the following describes the preferred implementation embodiments of the present invention in conjunction with the embodiments. However, it should be understood that these descriptions are only for further explaining the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0031] Refer to the attached Figure 1 , the first illustrated embodiment of the present invention is a weakly supervised learning video screen freeze semantic segmentation method, specifically including the following steps:

[0032] The first step is to prepare the data set.

[0033] The data set includes a first data set and a second data set.

[0034] The first data set is a large data set of video screen freeze images, and this data set does not require annotation. In this embodiment, the first data set contains 10,000 screen freeze image data;

[0035] The second data set is a small data set of video screen freeze images, and the semantic annotation of the screen freeze is performed on this data set. In this embodiment, the second data set contains 400 screen freeze image data.

[0036] In the second step, initialize the multi-expert self-distillation network and train the network using the first dataset.

[0037] In the third step, train the multi-expert self-distillation network with the dataset to obtain the feature extraction module.

[0038] In the fourth step, initialize the semantic segmentation model.

[0039] The semantic segmentation model includes a feature extraction module and a segmentation module. The feature extraction module is obtained from the training in the second step, and the segmentation module is composed of a horizontal spatial attention module and a feature fusion module.

[0040] In the fifth step, train the segmentation model using the second dataset to obtain the semantic segmentation result of the screen freeze.

[0041] Refer to Appendix Figure 2 , the training process of the multi-expert self-distillation network is as follows:

[0042] In the first step, initialize three feature extraction modules, namely the expert1 module, the expert2 module, and the member module. The initial weights of the three modules are the same;

[0043] In the second step, crop the image data of the first dataset to obtain I g ={I g1 , I g2 ...}, I l ={I l1 , I l2 ...}, where

[0044] Size g =w g ×Size ori , w g ∈[0.4, 1.0]

[0045] Size l =w l ×Size ori , w l ∈[0.05, 0.4]

[0046] Among them, Size g represents the size of I g , Size l represents the size of I l , and Size ori represents the size of the original image;

[0047] In the third step, put the cropped image data into the multi-expert self-distillation network, and substitute the result into the loss function. The specific formula is as follows:

[0048] Out mon1 = f mon1 (I g )

[0049] Out mon2 = f mon2 (I g )

[0050] Out mem = f mem (I l )

[0051] loss = loss_func(Out mon1 , Out mem ) × w1 + loss_func(Out mon2 , Out mem ) × w2

[0052] Among them, f mon1 represents the expert1 network, f mon2 represents the expert2 network, f mem represents the member network,

[0053] In this embodiment, w1 = 0.5 and w2 = 0.5;

[0054] In the fourth step, the member module performs backpropagation according to the loss function to update the parameters, and the parameters of the expert1 module and the expert2 module are updated by the EMA method. The specific formula is as follows:

[0055] θ mon1 = m1 × θ mon1 + (1 - m1) × θ mem

[0056] θ mon2 = m2 × θ mon2 + (1 - m2) × θ mem

[0057] Among them, θ mon1 represents the network parameters of the expert1 module, θ mon2 represents the network parameters of the expert2 module, θ mem represents the network parameters of the member module, and m1 and m2 are the update coefficients of the expert1 module and the expert2 module respectively

[0058] In this embodiment, m1 = 0.996 and m2 = 0.99;

[0059] Step 5: Use the trained expert1 module as the final feature extraction module.

[0060] Refer to Appendix Figure 3 , the semantic segmentation model is specifically composed of a feature extraction module, a horizontal spatial attention module, a feature fusion module, and a segmentation head. Its training specifically includes the following steps:

[0061] Step 1: The feature extraction module extracts and outputs a feature map from the second dataset;

[0062] Step 2: The horizontal spatial attention module obtains the horizontal spatial attention from it.

[0063] The specific calculation process is as follows:

[0064] HSA = δ(w avg ×Mean w (M feat )); w max ×Max w (M feat ))

[0065] Among them, M feat represents the feature map, Mean w represents the function of taking the mean in the horizontal direction, Max w represents the function of taking the maximum value in the horizontal direction, w avg represents the weight of the horizontal mean feature, w max represents the weight of the horizontal maximum feature, and δ represents the sigmoid function;

[0066] Step 3: The feature fusion module integrates the horizontal direction attention into the feature map;

[0067] Step 4: The segmentation head maps the fused feature map to the target space to obtain the semantic segmentation result of the screen freeze.

[0068] In this embodiment, the final semantic segmentation model is tested on 400 test sets, and the miou index is 93.1%.

[0069] Refer to Appendix Figure 4 , Figure 4 is the schematic diagram of the hardware device working in the embodiment of the present invention. The hardware device specifically includes: a weakly supervised video screen freeze semantic segmentation device 401, a processor 402, and a storage device 403;

[0070] A weakly-supervised video freeze-frame semantic segmentation device 401: The weakly-supervised video freeze-frame semantic segmentation device 401 implements the weakly-supervised learning video freeze-frame semantic segmentation method.

[0071] Processor 402: The processor 402 loads and executes the instructions and data in the storage device 403 to implement the weakly-supervised learning video freeze-frame semantic segmentation method.

[0072] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the weakly-supervised learning video freeze-frame semantic segmentation method.

[0073] Finally, it should be noted that: The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: They can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0074] The beneficial effects of the present invention are as follows: The present invention uses a weakly-supervised method, which significantly reduces the annotation operation cost of the training set in the traditional supervised learning method, fully improves the overall utilization rate of a small amount of labeled and a large amount of unlabeled data, and reduces the threshold of deep learning in practical applications. At the same time, aiming at the visual features of image freeze-frame, a horizontal spatial attention module is added, which can help the model focus on the important parts in the sequence. From the above two aspects, the present invention effectively promotes the industrial implementation of the weakly-supervised training method, and provides new feasibility for solving the problem of semantic segmentation of freeze-frame in surveillance video images.

[0075] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A semantic segmentation method for video screen freeze in weakly supervised learning, characterized in that, It includes the following steps: S1. Prepare a dataset; the dataset includes a first dataset and a second dataset. The first dataset is an unannotated image set, and the second dataset is an annotated image set; S2. Initialize the multi-expert self-distillation network; the multi-expert self-distillation network is divided into an expert1 module, an expert2 module, and a member module; S3. Train the multi-expert self-distillation network with the first dataset to obtain a feature extraction network. The specific steps are as follows: S31. Initialize the multi-expert self-distillation network, including the expert1 module, the expert2 module, and the member module. The initial weights of the three modules are the same; S32. Crop the dataset to obtain image data of different sizes; S33. Put the image data into the multi-expert self-distillation network and substitute the result into the loss function; S34. Update the multi-expert self-distillation network. The specific method is as follows: the member module updates the parameters through backpropagation according to the loss data, and the parameters of the expert1 module and the expert2 module are updated using the EMA method; S35. Use the trained expert1 module as the feature extraction network; S4. Initialize the semantic segmentation model. The semantic segmentation model is divided into a feature extraction module and a segmentation module. The feature extraction module is trained by S3; S5. Use the second dataset to train the semantic segmentation model to obtain the segmentation result of the screen-frozen image.

2. A method for semantic segmentation of video screen freezing in weak supervision learning according to claim 1, characterized in that: The segmentation module of the semantic segmentation model consists of a horizontal spatial attention module, a feature fusion module, and a segmentation head.

3. A method for semantic segmentation of video screen freezing in weak supervision learning according to claim 2, characterized in that: The step of using the second dataset to train the semantic segmentation model to obtain the segmentation result of the screen-frozen image is specifically as follows: S51. The feature extraction module extracts and outputs a feature map from the second dataset; S52. The horizontal spatial attention module extracts the horizontal spatial attention from the feature map; S53. The feature fusion module fuses the horizontal spatial attention into the feature map; S54. The segmentation head maps the fused feature map to the target space to obtain the semantic segmentation result of the screen freezing.

4. A storage device that stores instructions and data for implementing a method for semantic segmentation of video screen freezing in weak supervision learning according to any one of claims 1 to 3.

5. A video freeze-frame semantic segmentation device based on weak supervision learning, comprising: A processor and a storage device; the processor loads and executes the instructions and data in the storage device to implement a method for semantic segmentation of video screen freezing in weak supervision learning according to any one of claims 1 to 3.

Images