Multi-scale pseudo-labeling method, system and device

Abstract

The application discloses a kind of multiscale pseudo-labeling method, system and device, comprising: S1, generating multiscale pseudo-labels;S2, multiscale pseudo-labels are fused, and get fusion pseudo-labels;S3, multiscale pseudo-labels and fusion pseudo-labels are smoothed, and get smoothed pseudo-labels;S4, establish edge detection sub, according to edge detection sub, edge prediction is obtained, and the edge of predicted;S5, establish target optimization function;S6, predicted edge and smoothed pseudo-labels are input into target optimization function, and error result is calculated;S7, according to error result, edge detection sub is updated;S8, the performance evaluation of updated edge detection sub is carried out.The application can realize the multiscale pseudo-labeling of edge.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a multi-scale pseudo-labeling method, system and device. BACKGROUND

[0002] Edge detection is a fundamental and hot research field in computer vision. It can capture the salient object edges in an image and be used for high-level vision tasks, such as object proposal, object detection and semantic segmentation, etc. In recent years, influenced by deep learning, edge detection has also entered the era of data-driven, and based on deep neural networks, more and more deep edge detectors have been proposed. Deep edge detectors are usually trained in an end-to-end manner, and have higher accuracy than traditional methods (such as Canny, SE, etc.), and their performance even exceeds human level.

[0003] However, the existing deep edge detectors also have significant drawbacks: the effect of the detector is severely dependent on large-scale, high-reliability artificial edge-labeled datasets, such as BSDS500, NYUDv2 and Multicue, etc. However, artificial edge labeling is a very time-consuming task, and it may take several minutes to label a high-quality image. Due to this, the most famous BSDS500 dataset only has 500 pairs of image data annotations. In addition, different annotators may have different opinions on whether a pixel should be labeled as an edge. For example, the Multicue dataset provides annotations from six different annotators for each image. This inconsistency makes it more difficult to train deep networks.

[0004] On the other hand, it is generally believed in the academic community that edge features appear in early vision, which means that edge detection should be a low-level vision task. Early methods have shown that with low-level visual cues, unsupervised methods can completely achieve satisfactory results. Biologically inspired methods have also shown that models constructed in a hierarchical and parallel manner can not only perceive edge contours in an unsupervised manner, but also have biological interpretability. This has inspired scholars to explore unsupervised learning of deep edge detectors, and obviously the biggest advantage is that it does not require artificial edge labeling. In addition, due to the strong learning representation ability and efficient unlabeled data utilization ability of the unsupervised learning paradigm, the unsupervised learning paradigm has received more and more attention in recent years. Based on carefully designed tasks, such models can learn useful features and generalize well to downstream tasks.

[0005] However, few methods explore the potential of unsupervised learning in deep edge detection. A pioneering work proposes to learn motion edges from semi-dense matches between video frames and simple image gradients. While the performance of this method is similar to supervised methods, it employs a multi-stage approach that makes it very difficult in practice. Similarly, there are studies that learn edges from videos by methods that estimate edges and geometric shapes (depth and normals) simultaneously, although with impressive performance, their results are largely dependent on the video dataset. The work most relevant to the present invention is the weakly supervised edge detector. This method employs weak annotations (bounding box annotations) to train an edge detector, and its results have better performance than supervised counterparts. Nevertheless, the cost of obtaining bounding box annotations is still high. In general, there is still a lack of research on training deep edge detectors in a purely unsupervised manner. SUMMARY

[0006] The present application aims to provide a multi-scale pseudo-labeling method, system and device, which aims to solve the multi-scale pseudo-labeling of edges.

[0007] The present application provides a multi-scale pseudo-labeling method, comprising:

[0008] S1, generating multi-scale pseudo-labels;

[0009] S2, fusing the multi-scale pseudo-labels to obtain fused pseudo-labels;

[0010] S3, smoothing the multi-scale pseudo-labels and the fused pseudo-labels to obtain smoothed pseudo-labels;

[0011] S4, establishing an edge detection sub, and performing edge prediction according to the edge detection sub to obtain predicted edges;

[0012] S5, establishing a target optimization function;

[0013] S6, inputting the predicted edges and the smoothed pseudo-labels into the target optimization function to calculate error results;

[0014] S7, updating the edge detection sub according to the error results;

[0015] S8, evaluating the performance of the updated edge detection sub.

[0016] The present application also provides a multi-scale pseudo-labeling system, comprising:

[0017] The multi-scale pseudo-labeling system, characterized by comprising:

[0018] A generating module for generating multi-scale pseudo-labels;

[0019] A fusion module is configured to fuse the multi-scale pseudo labels to obtain fused pseudo labels.

[0020] A smoothing module is configured to smooth the multi-scale pseudo labels and the fused pseudo labels to obtain smoothed pseudo labels.

[0021] A building module is configured to build an edge detection sub, perform edge prediction according to the edge detection sub, and obtain predicted edges.

[0022] A target module is configured to build a target optimization function.

[0023] A calculation module is configured to input the predicted edges and the smoothed pseudo labels into the target optimization function, and calculate error results.

[0024] An updating module is configured to update the edge detection sub according to the error results.

[0025] A performance evaluation module is configured to evaluate the performance of the updated edge detection sub.

[0026] The embodiment of the present application also provides a multi-scale pseudo labeling device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is used to implement the steps of the above method when executed by the processor.

[0027] The embodiment of the present application also provides a computer readable storage medium, which stores an implementation program of information transmission, and the program is used to implement the steps of the above method when executed by a processor.

[0028] The embodiment of the present application can realize multi-scale pseudo labeling of edges.

[0029] The above description is only a summary of the technical scheme of the present application, in order to more clearly understand the technical means of the present application, implement the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the specific embodiments of the present application or the technical scheme in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creating any creative labor.

[0031] Figure 1 It is a multi-scale pseudo labeling method flow chart of the embodiment of the present application;

[0032] Figure 2is a schematic diagram of a neighborhood approximation discrete form of a multi-scale pseudo-labeling method of an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of a multi-scale pseudo-labeling system of an embodiment of the present application;

[0034] Figure 4 is a schematic diagram of a multi-scale pseudo-labeling device of an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solutions of the present application will be described clearly and completely below in conjunction with embodiments. Obviously, the described embodiments are only part of, rather than all of, the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0036] Method embodiments

[0037] According to an embodiment of the present application, a multi-scale pseudo-labeling method is provided, Figure 1 is a flowchart of a multi-scale pseudo-labeling method of an embodiment of the present application, as Figure 1 shown, specifically comprising:

[0038] S1, generating multi-scale pseudo-labels;

[0039] S1 specifically comprises:

[0040] Generating multi-scale pseudo-labels, the formula is as follows:

[0041]

[0042] wherein G(X;σ s ) is an edge detector with a scale parameter σ s ; X is an input image, and s is an index of a pseudo-label;

[0043] S2, fusing the multi-scale pseudo-labels to obtain fused pseudo-labels;

[0044] S2 specifically comprises: fusing the multi-scale pseudo-labels to obtain fused pseudo-labels, the formula is as follows:

[0045]

[0046] wherein F{} is a fusion function.

[0047] S3, smoothing the multi-scale pseudo-labels and the fused pseudo-labels to obtain smoothed pseudo-labels;

[0048] S3 specifically comprises: distance-aware label smoothing is performed on the multi-scale pseudo label and the fused pseudo label to obtain a smoothed pseudo label; the distance-aware label smoothing expression is as follows:

[0049]

[0050] represents an edge after the smoothing operation at a pixel position p; and respectively represent a window pixel neighborhood, a cross pixel neighborhood and a diagonal pixel neighborhood at the position p.

[0051] S4, an edge detection sub is established, and edge prediction is performed according to the edge detection sub to obtain a predicted edge;

[0052] S5, a target optimization function is established;

[0053] S5 specifically comprises:

[0054] a target optimization function is established; the target function is as follows:

[0055]

[0056] wherein s takes values 0,...,S

[0057] wherein s takes values 1,...,S

[0058]

[0059] wherein E[] represents taking expectation, l() is a loss function, and a s is a model hyperparameter, f(X; W) is an edge detection sub, W is a to-be-learned parameter of the edge detection sub, and s is an index of a pseudo label.

[0060] S6, the predicted edge and the smoothed pseudo label are input into the target optimization function, and an error result is calculated;

[0061] S7, the edge detection sub is updated according to the error result;

[0062] S8, performance evaluation is performed on the updated edge detection sub.

[0063] The specific implementation method is as follows:

[0064] Stage 1: Pseudo label generation

[0065] In this stage, multi-scale pseudo labels are first generated in an unsupervised manner, then the generated pseudo labels are fused using a multi-scale average method, and finally all the pseudo labels are smoothed to suppress noise.

[0066] Step 1.1 Generate multi-scale pseudo-labels

[0067] This invention directly generates scale-dependent edges in an unsupervised manner, replacing manual edge annotation at mixed scales. All ground truth edges (i.e., supervised labels) are generated. Replace with the following generated pseudo-tags:

[0068]

[0069] Wherein, G(X; σ s ) is a scale parameter σ s An edge detector; X is the input image, and s is the index of the pseudo-label;

[0070] Step 1.2 Multi-scale pseudo-label fusion

[0071] Since objects of interest may appear at different scales, it is necessary to fuse pseudo-labels from different scales to obtain the final label map. This invention addresses this by adding a fusion mechanism that takes the aforementioned multi-scale pseudo-labels as input and generates fused pseudo-labels.

[0072]

[0073] In the formula, F{} is a fusion function.

[0074] Step 1.3 Perform distance-aware label smoothing

[0075] While pseudo-labels are simple to generate, they contain a large amount of unknown noise, which is detrimental to model training and generalization. Label smoothing, by introducing uniform perturbation to smooth one-hot labels, can make the model more robust. Uniform perturbation is reasonable for classification problems because there is no concept of distance between different categories. However, this is not suitable for the current situation. Therefore, this invention proposes a novel distance-aware label smoothing method. The intuition behind this method is that pixels closer to the real edge have a higher similarity to the edge, and further, the closer the predicted edge is to the real edge, the smaller the corresponding loss. For one-hot labels and uniformly perturbated smoothed labels, the model can only obtain a smaller loss feedback when the predicted edge perfectly matches the real edge, while for the proposed distance-aware smoothed labels, the model can obtain a progressive loss feedback when the predicted edge is close to the real edge.

[0076] Phase 2 Edge Prediction and Network Update

[0077] A deep edge detector is established to predict the edges of the image and learn the parameters of the detector. Then, the performance of the learned edge detector is evaluated.

[0078] Step 2.1 Establish a deep edge detection sub-system

[0079] A deep edge detector, denoted as f(X; W), is established, where W is the parameter to be learned. This detector receives an image as input and predicts its edges to obtain the predicted edges. In specific experiments, this patent uses two well-known deep edge detectors: RCF and PiDiNet.

[0080] Step 2.2 Design the objective optimization function

[0081] The objective optimization function for unsupervised edge detection designed in this invention is as follows:

[0082]

[0083] In the formula, s takes values ​​of 0,...,S

[0084] In the formula, s takes values ​​from 1 to S.

[0085]

[0086] This is a deep convolutional neural network model trained with pseudo-labels. Here, E[] represents the expectation, l() is the loss function, and α... s These are model hyperparameters.

[0087] Step 2.3 Learning Network Parameters

[0088] The loss error is calculated based on the objective function in step 2.2, and the network parameters W are learned using stochastic gradient descent. Where η is the learning rate. It is the stochastic gradient with respect to parameter W, and L is the calculated average loss error.

[0089] Step 2.4: Evaluate the model

[0090] The final learned deep edge detector f(X; W) is evaluated by comparing the difference between the predicted edges and the true edges to determine the performance of the learned model. The smaller the difference, the better the performance; the larger the difference, the worse the performance.

[0091] The preceding sections have provided a framework for the complete stages and steps. The following sections will provide more specific supplementary explanations of the details involved.

[0092] Pseudo-label generation. In this invention, the well-known unsupervised edge detector Canny is used as the multi-scale pseudo-label generator G(X; σ). s It is a gradient-based edge detector that uses the first-order gradient to calculate the edge intensity and the second-order gradient to determine the edge location. This is achieved by changing the scale parameter σ. sThis refers to the standard deviation of the Gaussian filter. The Canny detector can directly generate multi-scale binary pseudo-labels. In the experiments corresponding to this invention, a total of 5 scale parameters were selected: {σ1=2, σ2=2.5, σ3=3, σ4=3.5, σ5=4}.

[0093] Fusion Function. Since objects of interest often appear at different scales, the edges of interest are inherently of mixed scales. Therefore, pseudo-labels of mixed scales are essential for controlling the learning process of the edge detector. This invention utilizes the fusion function F(·) to obtain pseudo-labels of mixed scales, i.e., fused pseudo-labels, which is a simple and effective cross-scale average.

[0094]

[0095] It is similar to the operation in the labeling robust loss function, which averages several edge maps. However, the fusion function in this invention is based only on pseudo-labels, while another must handle multiple labels from human annotators.

[0096] Distance-aware smoothing. To suppress noise in pseudo-tags, we select a radial basis function to smooth the pseudo-tags, i.e.

[0097]

[0098] Where p and p □ These represent the predicted edge location and the corresponding ground truth edge location, respectively; η is a parameter; the neighborhood N(p □ The area of ​​effect is indicated by ). For simplicity, we use an approximate discrete form of the neighborhood, with a discrete window size of 3×3. Figure 2 This is a schematic diagram of the neighborhood approximate discrete form of an embodiment of the present invention.

[0099] remember Indicates the edge that has not been smoothed. This represents the edge after smoothing. For a pixel location p, its window neighborhood (a 3×3 window centered on p), its intersecting neighborhood (the top / left / bottom / right edges of p), and its diagonal neighborhood are denoted as follows: and The distance-aware label smoothing method can be represented as follows:

[0100]

[0101] Loss Function. To further prevent weak label effects in false labels, this invention employs a label-robust loss function to train the depth detector. This loss ignores weak edges that are more likely to be noise. The predicted value at the i-th pixel position at scale s is denoted as... The corresponding loss values ​​are as follows:

[0102]

[0103] Here, λ and ξ are hyperparameters, and α and β are the percentages of edge pixels and non-edge pixels, respectively, so α + β = 1. All hyperparameters take their default values.

[0104] Learning process. The learning process of the proposed multi-scale pseudo-annotation method is shown in the table below:

[0105]

[0106]

[0107] The multi-scale pseudo-labeling method proposed in this invention enables unsupervised learning of deep edge detectors, effectively reducing the cost of manual annotation. This invention considers the multi-scale properties of edges and matches the hierarchical structure of deep edge detectors. Furthermore, to suppress noise in unsupervised pseudo-labels, the proposed distance-aware label smoothing method weakens the negative impact of pseudo-labels, improving model performance.

[0108] System Implementation Example 1

[0109] According to embodiments of the present invention, a multi-scale pseudo-labeling system is provided. Figure 3 This is a schematic diagram of a multi-scale pseudo-labeling system according to an embodiment of the present invention, such as... Figure 3 As shown, it specifically includes:

[0110] Generation module: Used to generate multi-scale pseudo-labels;

[0111] Fusion module: Used to fuse multi-scale pseudo-labels to obtain fused pseudo-labels;

[0112] Smoothing module: Used to smooth multi-scale pseudo-labels and fused pseudo-labels to obtain smoothed pseudo-labels;

[0113] The edge detection module is used to create an edge detector, perform edge prediction based on the edge detector, and obtain the predicted edge.

[0114] Target module: Used to establish the target optimization function;

[0115] The calculation module is used to input the predicted edges and smoothed pseudo-labels into the objective optimization function and calculate the error results.

[0116] Update module: Used to update the edge detector based on the error results;

[0117] Performance evaluation module: Evaluates the performance of the updated edge detection module.

[0118] The generation module is specifically used for:

[0119] The formula for generating multi-scale pseudo-labels is as follows:

[0120]

[0121] Wherein, G(X; σ s ) is a scale parameter σ s An edge detector; X is the input image, and s is the index of the pseudo-label;

[0122] The fusion module is specifically used to fuse multi-scale pseudo-labels to obtain fused pseudo-labels, as shown in the following formula:

[0123]

[0124] In the formula, F{} is a fusion function.

[0125] The smoothing module is specifically used to: smooth the multi-scale pseudo-labels and fused pseudo-labels using distance-aware labeling to obtain smoothed pseudo-labels; the distance-aware label smoothing expression is as follows:

[0126]

[0127] This represents the edge at pixel location p after smoothing. and These represent the window pixel neighborhood, the cross pixel neighborhood, and the diagonal pixel neighborhood at position p, respectively.

[0128] The target module is specifically used for:

[0129] Establish the objective optimization function; the objective function is as follows:

[0130]

[0131] In the formula, s takes values ​​of 0,...,S

[0132] In the formula, s takes values ​​from 1 to S.

[0133]

[0134] Where E[] represents the expected value, l() is the loss function, and α s is the model hyperparameter, f(X; W) is the edge detector, W is the edge detector's learning parameter, and s is the pseudo-label index.

[0135] The embodiments of the present invention are system embodiments corresponding to the above method embodiments. The specific operation of each module can be understood by referring to the description of the method embodiments, and will not be repeated here.

[0136] Device Example 1

[0137] This invention provides a device for multi-scale pseudo-labeling, such as... Figure 4 As shown, it includes: a memory 40, a processor 42, and a computer program stored on the memory 40 and executable on the processor 42. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0138] Device Example 2

[0139] This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor 42, implements the steps described in the above method embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions to the technical solutions of the embodiments of the present invention do not cause the essence of the corresponding technical solutions to deviate from the scope of the present solution.

Claims

1. A multi-scale pseudo-labeling method, characterized in that, Comprise: S1, generating multi-scale pseudo labels; S2, fuse multi-scale pseudo labels to get fused pseudo labels; S3, smoothing multi-scale pseudo labels and fused pseudo labels to get smoothed pseudo labels; S4, establish edge detection sub, according to edge detection sub, edge prediction is obtained; S5, establish target optimization function; S6, the predicted edge and the smoothed pseudo label are input into the target optimization function, and the error result is calculated; S7, according to the error result, the edge detection sub is updated; S8, the performance of the updated edge detection sub is evaluated; The S3 specifically comprises: the multi-scale pseudo labels and the fused pseudo labels are distance perception label smoothing, and the smoothed pseudo labels are obtained; the distance perception label smoothing expression is as follows: The S5 specifically comprises: Establishing a target optimization function; the target function is as follows: where, E [ ] represents the expectation, l ( ) is a loss function, is a model hyperparameter, is an edge detection sub, W is a to-be-learned parameter of the edge detection sub, s is an index of the pseudo label.

2. The method of claim 1, wherein, The S1 specifically comprises: Generating multi-scale pseudo labels, the formula is as follows: In the formula, F {} is a fusion function.

3. A system for multi-scale pseudo-labeling, the system comprising: Comprise: Generating module: for generating multi-scale pseudo labels; Fusion module: for fusing multi-scale pseudo labels to get fused pseudo labels; Smoothing module: for smoothing multi-scale pseudo labels and fused pseudo labels to get smoothed pseudo labels; Establishment module: for establishing edge detection sub, according to edge detection sub, edge prediction is obtained; Target module: for establishing target optimization function; Calculation module: for inputting the predicted edge and the smoothed pseudo label into the target optimization function, and calculating the error result; Updating module: for updating the edge detection sub according to the error result; Performance evaluation module: the performance of the updated edge detection sub is evaluated; The smoothing module is specifically used for: the multi-scale pseudo labels and the fused pseudo labels are distance perception label smoothing, and the smoothed pseudo labels are obtained; the distance perception label smoothing expression is as follows: The target module specifically comprises: Establishing a target optimization function; the target function is as follows:

4. The system of claim 3, wherein, The generating module is specifically used for: Generating multi-scale pseudo labels, the formula is as follows: In the formula, F{} is a fusion function.

5. A multi-scale pseudo-labeling apparatus, comprising: Comprise: Memory, processor and computer program stored on the memory and executable on the processor, the computer program is executed by the processor to realize the steps of the multi-scale pseudo labeling method in any one of claims 1 to 2.

6. A computer readable storage medium characterized by, The computer readable storage medium stores the implementation program of information transmission, and the program is executed by the processor to realize the steps of the multi-scale pseudo labeling method in any one of claims 1 to 2.

Images