A training method for salient object detection based on self-boosting learning

By using progressive enhancement module and multi-level consistency regularization in significance object detection, the problem of label inconsistency and interference factors not modeled in the significance object detection method in the prior art is solved, and the training smoothness of the model is improved and generalization capabilities enable more accurate detection of significant objects.

CN115131549BActive Publication Date: 2024-11-08ZHEJIANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210677715.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-15
Publication Date
2024-11-08
Estimated Expiration
2042-06-15

AI Technical Summary

Technical Problem

The existing significance object detection method based on deep learning ignores the inconsistency and interference factors of significance binary discrete labels, resulting in high difficulty in model training and weak generalization ability.

Method used

The progressive improvement module is used to generate consistent continuous labels, and the generalization ability of the model is improved through multi-level consistency regularization, dynamically adjust model parameters and loss functions, and explicitly model interference factors in the scenario.

Benefits of technology

It improves the training smoothness and generalization ability of the model, enhances the detection accuracy of significant objects in the image, and does not require modification of the network structure, making it easier to generalize to different models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115131549B_ABST
    Figure CN115131549B_ABST
Patent Text Reader

Abstract

The present invention discloses a salient target detection training method based on self-boosting learning. The method comprises the following steps: using a salient target detection model to predict salient objects in a training image; constructing a progressive boosting module to predict consistent and continuous salient labels, guide the learning of the salient model, and update the network parameters of the progressive boosting module in a momentum update manner; using a sample adaptive module to evaluate the quality of the generated continuous labels, thereby dynamically adjusting the impact of the model's continuous labels on the salient model optimization process; and enhancing the generalization ability of the salient model through data-enhanced augmentation transformation of the input image and multi-level consistency regularization operations. The training method proposed by the present invention makes full use of the self-boosting learning and consistency learning paradigms, can better help the training of the salient detection model, and enhance the accuracy and generalization of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a salient object detection training method based on self-boosting learning. Background Art

[0002] Salient object detection is a technique for detecting the most salient objects in terms of visual features from an image. It is often used as a preprocessing task for other visual research, such as object tracking, image description generation, image retrieval, etc. The goal of salient object detection is to distinguish whether each pixel in an image is visually salient given an image. Most of the existing deep learning-based methods model the salient object detection problem as an end-to-end dense pixel prediction problem. The entire training process directly uses the binary discrete labels provided by the dataset as supervision signals to train the model to identify salient objects. There are two main problems with this training method: First, this training method ignores the impact of the inconsistent labeling and salient homogeneity of the salient binary discrete labels themselves on the training process. Second, this training method does not model the interference factors in the scene, and the generalization ability of the model is weak.

[0003] To address the first problem, the present invention proposes a progressive improvement module to generate consistent continuous labels, guide the model to learn progressively, and reduce the difficulty of the training process. To address the second problem, the present invention explicitly models the interference factors in the scene and improves the generalization ability of the model through a multi-level consistency regularization method. Summary of the invention

[0004] In view of the above problems, the present invention provides a salient object detection training method based on self-boosting learning.

[0005] The technical solutions specifically adopted in the present invention are as follows:

[0006] A salient object detection training method based on self-boosting learning comprises the following steps:

[0007] S1. Obtain a training dataset with dataset labels for salient object detection;

[0008] S2. using a salient object detection model to predict salient objects in each training image in the training data set;

[0009] S3, constructing a progressive boosting module, using the progressive boosting module to predict consistent and continuous saliency labels to guide the learning of the saliency model, and using the parameters of the saliency model to dynamically update the network parameters of the progressive boosting module;

[0010] S4. Use the sample adaptation module to evaluate the quality of the generated continuous labels, so as to dynamically adjust the impact of the model continuous labels on the saliency model optimization process;

[0011] S5. Perform augmentation transformation on the input image through data augmentation, and improve the generalization ability of the saliency model through multi-level consistency regularization operation;

[0012] S6. During the training process of the salient object detection model, the self-boosting training framework established by S1 to S5 is used to guide the model learning process and help the model better predict the salient objects in the image.

[0013] Based on the above scheme, each step can be implemented as follows:

[0014] Furthermore, in step S1, the training data set includes a number of training images I train and the corresponding salient object binary label Y train .

[0015] Furthermore, in step S2, the salient object detection model is used to predict the salient objects in the training image, which specifically includes the following sub-steps:

[0016] S21. Obtain a salient object detection model The network parameters of the model are θ SOD ; The target saliency model includes an encoder module ε SOD and decoder module

[0017] S22, for each training image I in the training data set train , which inputs the salient object detection model After that, first pass the encoder module ε SOD Get the corresponding image feature map f, and then use the decoder module Get the salient object prediction result P:

[0018] f=ε SoD (I train ,θ SOD )

[0019]

[0020] Furthermore, in step S3, a progressive boosting module is constructed, and the progressive boosting module is used to predict consistent and continuous saliency labels to guide the saliency model to learn more smoothly and stably, and the parameters of the saliency model are used to dynamically update the network parameters of the progressive boosting module, which specifically includes the following sub-steps:

[0021] S31, initialization of progressive improvement module The network structure of this module is similar to the salient object detection model in S1 Stay consistent and improve modules gradually The network parameter is θ PUM ;

[0022] S32, using progressive improvement modules Generate consistent and continuous saliency labels, which are then used to guide the saliency model to learn more smoothly and stably:

[0023]

[0024] Where: Z represents the consistent and continuous saliency labels generated by the progressive boosting module;

[0025] S33, using momentum update to integrate the parameters of the saliency model at different stages of the training process into the progressive improvement module In the network, the dynamic update of the network parameters of the progressive improvement module is realized:

[0026]

[0027] in: and Respectively represent the progressive improvement modules during the t-th and t-1-th training iterations The network parameters θ PUM , Represents the salient object detection model during the tth training iteration The network parameters θ SOD , η is a hyperparameter that controls the momentum update speed of the model.

[0028] Furthermore, the hyperparameter η is set to 0.99.

[0029] Furthermore, in step S4, the quality of the generated continuous labels is evaluated by using the sample adaptive module, so as to dynamically adjust the influence of the model continuous labels on the saliency model optimization process, which specifically includes the following sub-steps:

[0030] S41, the sample adaptation module obtains the sample adaptation weight λ by evaluating the quality of the generated continuous labels:

[0031] λ=exp(-ζ·∑(Y·log(Z)+(1-Y)·log(1-Z)))

[0032] Where: Z represents the consistent and continuous salient labels generated by the progressive boosting module in S32, Y represents the real salient object binary labels corresponding to Z provided by the training dataset in S1, and ζ is a hyperparameter to adjust the value range of λ;

[0033] S42, sample adaptive weight λ is used to dynamically adjust the impact of binary labels and continuous labels on the model optimization process, and finally obtain the loss function of the self-boosting training paradigm

[0034]

[0035]

[0036]

[0037] Where: H, W represent the length and width of the input training image, Y i,j , Z i,j , P i,j are the binary label Y, the salient label Z, and the element value of the (i, j) position in the salient object prediction result P, respectively. The loss function calculated for the prediction results and consistent continuous labels, The loss function for predicting results and binary discrete calculations.

[0038] Furthermore, the hyperparameter ζ is set to 70.

[0039] Furthermore, in step S5, the input image is augmented by data augmentation, and the generalization ability of the saliency model is improved by multi-level consistency regularization operation, which specifically includes the following sub-steps:

[0040] S51, by the original sample I of the original input train Perform data augmentation by adding Gaussian noise and random cropping to construct extended samples Used to simulate samples affected by interference factors:

[0041] S52, expand the sample Input to the salient object detection model , obtain the corresponding significant prediction results

[0042]

[0043] S53. Regularization constraints are imposed on the prediction consistency between the original sample and the expanded sample:

[0044]

[0045] in: Indicates significant prediction results The element value at position (i, j) in ;

[0046] S54, encoder module ε using salient object detection model SOD The intermediate image features are extracted from the original sample and the expanded sample respectively, and the regularization constraint of multi-scale feature consistency is imposed on them:

[0047]

[0048] in: Represents the F-norm (Frobenius Norm) of the matrix.

[0049] Furthermore, in step S6, the self-boosting training framework established by S1 to S5 is used in the training process of the salient object detection model to guide the model learning process and help the model better predict the salient objects in the image, which specifically includes the following sub-steps:

[0050] S61. Setting up the salient object detection model The total loss function is:

[0051]

[0052] S62. Use SGD optimization method and back propagation algorithm through loss function To train a salient object detection model until the loss function converges.

[0053] On the other hand, the present invention provides a salient target detection method, which uses the salient target detection training method based on self-boosting learning described in any of the above schemes to obtain a trained salient target detection model. Using the trained salient object detection model Detect the target image and obtain the salient object area in the target image.

[0054] Compared with the existing salient object detection training methods, the salient object detection training method based on self-boosting learning of the present invention has the following beneficial effects:

[0055] First, the progressive improvement module designed in the present invention can generate consistent continuous labels to guide the model training process, making the model optimization process smoother and more stable, thereby improving the accuracy of the model.

[0056] Secondly, the multi-level consistency regularization constraints adopted in the present invention improve the generalization of the model by explicitly modeling the interference factors in the scene during the training process.

[0057] Finally, the proposed training methods can be easily integrated into different saliency models without modifying the network structure, and have achieved stable performance improvements on mainstream evaluation datasets. It is a plug-in method that is easy to promote. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of the process of the salient object detection training method based on self-boosting learning in the present invention;

[0059] Figure 2 Schematic diagram of the training method.

[0060] Figure 3 This is the implementation effect of the method of the present invention on the DUTS dataset. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the following detailed description of the present invention. Those skilled in the art can fully understand the present invention without the description of these details.

[0063] refer to Figure 1 , which is a flow chart of a salient target detection training method based on self-boosting learning in the present invention, wherein the training method schematic diagram is as follows Figure 2 The specific implementation steps of the method are described in detail below. The method specifically includes the following steps:

[0064] S1. Obtain a training dataset with dataset labels for salient object detection.

[0065] In this embodiment, the training data set in the above step S1 includes a plurality of training images I train and the corresponding salient object binary label Y train .

[0066] The algorithm goal in the salient object detection model is defined as: predicting the salient object region P in the image so that it is as close to the label information as possible. It should be noted that the present invention only provides a training method for a salient object detection model, but the salient object detection model therein can be any salient object detection model, such as PoolNet, F3Net and GateNet, etc., and is not limited to a specific model form.

[0067] S2. Using a salient object detection model, predict the salient objects in each training image in the training data set.

[0068] In this embodiment, in the above step S2, a salient object detection model is used to predict salient objects in the training image, which specifically includes the following sub-steps:

[0069] S21. Obtain a salient object detection model The network parameters of the model are θ SOD ; The target saliency model includes an encoder module ε SOD and decoder module

[0070] S22, for each training image I in the training data set train , which inputs the salient object detection model After that, first pass the encoder module ε SOD Get the corresponding image feature map f, and then use the decoder module Get the salient object prediction result P:

[0071] f=ε SOD (I train ,θ SOD )

[0072]

[0073] S3. Construct a progressive boosting module, use the progressive boosting module to predict consistent and continuous saliency labels to guide the learning of the saliency model, and use the parameters of the saliency model to dynamically update the network parameters of the progressive boosting module.

[0074] In this embodiment, in the above step S3, a progressive boosting module is constructed, and the progressive boosting module is used to predict consistent and continuous saliency labels to guide the saliency model to learn more smoothly and stably, and the parameters of the saliency model are used to dynamically update the network parameters of the progressive boosting module, which specifically includes the following sub-steps:

[0075] S31, initialization of progressive improvement module The network structure of this module is similar to the salient object detection model in S1 Stay consistent and improve modules gradually The network parameter is θ PUM ;

[0076] S32, using progressive improvement modules Generate consistent and continuous saliency labels, which are then used to guide the saliency model to learn more smoothly and stably:

[0077]

[0078] Where: Z represents the consistent and continuous saliency labels generated by the progressive boosting module;

[0079] S33, using momentum update to integrate the parameters of the saliency model at different stages of the training process into the progressive improvement module In the network, the dynamic update of the network parameters of the progressive improvement module is realized:

[0080]

[0081] in: and Respectively represent the progressive improvement modules during the t-th and t-1-th training iterations The network parameters θ PUM , Represents the salient object detection model during the tth training iteration The network parameters θ SOD , η is a hyperparameter that controls the model momentum update speed. In this embodiment, the above hyperparameter η is preferably set to 0.99.

[0082] S4. Use the sample adaptation module to evaluate the quality of the generated continuous labels, so as to dynamically adjust the impact of the model continuous labels on the saliency model optimization process.

[0083] In this embodiment, in the above step S4, the sample adaptive module is used to evaluate the quality of the generated continuous labels, so as to dynamically adjust the impact of the model continuous labels on the saliency model optimization process, which specifically includes the following sub-steps:

[0084] S41, the sample adaptation module obtains the sample adaptation weight λ by evaluating the quality of the generated continuous labels:

[0085] λ=exp(-ζ·∑(Y·log(Z)+(1-Y)·log(1-Z)))

[0086] Where: Z represents the consistent and continuous salient label generated by the progressive boosting module in S32, and Y represents the real salient object binary label corresponding to Z provided by the training dataset in S1 (i.e., Y train ), ζ is a hyperparameter for adjusting the value range of λ; in this embodiment, the hyperparameter ζ can be set to 70.

[0087] S42, sample adaptive weight λ is used to dynamically adjust the impact of binary labels and continuous labels on the model optimization process, and finally obtain the loss function of the self-boosting training paradigm

[0088]

[0089]

[0090]

[0091] Where: H, W represent the length and width of the input training image, Y i,j , Z i,j , P i,j are the binary label Y, the salient label Z, and the element value of the (i, j) position in the salient object prediction result P, respectively. The loss function calculated for the prediction results and consistent continuous labels, The loss function for predicting results and binary discrete calculations.

[0092] S5. The input image is augmented through data augmentation, and the generalization ability of the saliency model is improved through multi-level consistency regularization operations.

[0093] In this embodiment, in the above step S5, the input image is augmented by data augmentation, and the generalization ability of the saliency model is improved by multi-level consistency regularization operation, which specifically includes the following sub-steps:

[0094] S51, by the original sample I of the original input train Perform data augmentation by adding Gaussian noise and random cropping to construct extended samples Used to simulate samples affected by interference factors:

[0095] S52, expand the sample Input to the salient object detection model , obtain the corresponding significant prediction results

[0096]

[0097] S53. Regularization constraints are imposed on the prediction consistency between the original sample and the expanded sample:

[0098]

[0099] in: Indicates significant prediction results The element value at position (i, j) in ;

[0100] S54, encoder module ε using salient object detection model SOD The intermediate image features are extracted from the original sample and the expanded sample respectively, and the regularization constraint of multi-scale feature consistency is imposed on them:

[0101]

[0102] in: Represents the F-norm (Frobenius Norm) of the matrix.

[0103] The above steps S1 to S5 constitute a self-boosting training framework, and the salient object detection model can be trained based on the self-boosting training framework.

[0104] S6. During the training process of the salient object detection model, the self-boosting training framework established by S1 to S5 is used to guide the model learning process and help the model better predict the salient objects in the image.

[0105] In this embodiment, in the above step S6, the self-boosting training framework established by S1 to S5 is used in the training process of the salient object detection model to guide the model learning process and help the model better predict the salient objects in the image, which specifically includes the following sub-steps:

[0106] S61. Setting up the salient object detection model The total loss function is:

[0107]

[0108] in, is the loss function of the self-boosting training paradigm obtained in sub-step S42, is the prediction consistency loss function obtained in sub-step S53, is the feature consistency loss function obtained in sub-step S54;

[0109] S62. Use SGD optimization method and back propagation algorithm through loss function To train a salient object detection model until the loss function converges.

[0110] Finally, when performing specific salient object detection tasks, the trained salient object detection model is used By detecting the target image, the salient object area in the target image can be obtained.

[0111] The following is an application of the above-mentioned salient object detection training method based on self-boosting learning to a specific dataset example to demonstrate the technical effects it can achieve.

[0112] Example

[0113] The implementation method of this embodiment is as described in S1 to S6 above, and the specific steps will not be elaborated in detail. The following only shows its effect based on case data. The present invention is implemented on a data set with true value annotations, which are:

[0114] DUTS dataset: This dataset contains 10,533 training images and 5,019 test images.

[0115] This example contains 1000 training images and 500 test images in the dataset.

[0116] The training set of the data set is trained and the test set is tested. The specific process is as follows Figure 1 As shown in the figure, PUM is the progressive improvement module, SAM is the sample adaptation module, ObjDet is the target detector, label is the discrete binary label, and prediction is the saliency prediction result.

[0117] In order to comprehensively compare the effectiveness of this method, we integrated the training method proposed in this invention into three currently better saliency models, including PoolNet, F3Net and GateNet, and then verified the effectiveness by comparing the performance of the models trained by the original training method and the method proposed in this invention.

[0118] The detection accuracy of the detection results of this embodiment is shown in the following table, mainly using the mean absolute error (MAE), F-measure and S-measure three indicators. MAE measures the absolute pixel difference between the prediction and the label, the F-measure indicator can more comprehensively evaluate the accuracy and recall rate of the model, and the S-measure indicator considers the regional perception similarity and object perception similarity of the model detection results. The smaller the MAE indicator value, the better the performance, and the larger the other two indicators, the better the performance.

[0119] method MAE F-measure S-measure PoolNet 4.00 80.90 83.60 PoolNet+OursTrain 3.90(-0.10) 81.40(+0.50) 84.10(+0.50) F3Net 3.50 84.00 88.80 F3Net+OursTrain 3.40(-0.10) 84.60(+0.60) 89.10(+0.30) GateNet 3.70 82.80 88.40 GateNet+OursTrain 3.60(-0.10) 83.40(+0.60) 88.80(+0.40)

[0120] As shown in the above three tables, three different salient object detection models PoolNet, F3Net and GateNet are tested in this example. The results with +OursTrain are the final results after combining the training method proposed in the present invention. It can be seen that after all salient detection models use the training method proposed in the present invention, the performance of all indicators can be significantly improved.

[0121] Through the above technical solutions, the embodiment of the present invention develops a salient object detection training method based on self-boosting learning based on deep learning technology. The training method proposed in the present invention makes full use of the self-boosting learning and consistency learning paradigms, which can better help the training of the salient detection model and enhance the accuracy and generalization of the model.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A salient object detection training method based on self-boosting learning, characterized in that: The following steps are involved: S1. Obtain a training dataset with dataset labels for salient object detection; S2. using a salient object detection model to predict salient objects in each training image in the training data set; S3, constructing a progressive boosting module, using the progressive boosting module to predict consistent and continuous saliency labels to guide the learning of the saliency model, and using the parameters of the saliency model to dynamically update the network parameters of the progressive boosting module; S4. Use the sample adaptation module to evaluate the quality of the generated continuous labels, so as to dynamically adjust the impact of the model continuous labels on the saliency model optimization process; S5. Perform augmentation transformation on the input image through data augmentation, and improve the generalization ability of the saliency model through multi-level consistency regularization operation; S6. In the process of training the salient object detection model, the self-boosting training framework established by S1 to S5 is used to guide the model learning process and help the model better predict the salient objects in the image. In step S1, the training data set includes a number of training images I train and the corresponding salient object binary label Y train ; In step S2, the salient object detection model is used to predict the salient objects in the training image, which specifically includes the following sub-steps: S21. Obtain a salient object detection model The network parameters of the model are ; The target saliency model includes an encoder module and decoder module S22, for each training image I in the training data set train , which inputs the salient object detection model After that, first pass the encoder module Get the corresponding image feature map f, and then use the decoder module Get the salient object prediction result P: In step S3, a progressive boosting module is constructed, and the progressive boosting module is used to predict consistent and continuous saliency labels to guide the saliency model to learn more smoothly and stably, and the parameters of the saliency model are used to dynamically update the network parameters of the progressive boosting module, which specifically includes the following sub-steps: S31, initialization of progressive improvement module The network structure of this module is similar to the salient object detection model in S1 Stay consistent and improve modules gradually The network parameter is θ PUM ; S32, using progressive improvement modules Generate consistent and continuous saliency labels, which are then used to guide the saliency model to learn more smoothly and stably: Where: Z represents the consistent and continuous saliency labels generated by the progressive boosting module; S33, using momentum update to integrate the parameters of the saliency model at different stages of the training process into the progressive improvement module In the network, the dynamic update of the network parameters of the progressive improvement module is realized: in: and Respectively represent the progressive improvement modules during the t-th and t-1-th training iterations The network parameters θ PUM , Represents the salient object detection model during the tth training iteration The network parameters θ SOD , η is a hyperparameter that controls the momentum update speed of the model; In step S4, the sample adaptation module is used to evaluate the quality of the generated continuous labels, so as to dynamically adjust the impact of the model continuous labels on the saliency model optimization process, which specifically includes the following sub-steps: S41, the sample adaptation module obtains the sample adaptation weight λ by evaluating the quality of the generated continuous labels: Where: Z represents the consistent and continuous salient labels generated by the progressive boosting module in S32, Y represents the real salient object binary labels corresponding to Z provided by the training dataset in S1, and ζ is a hyperparameter to adjust the value range of λ; S42, sample adaptive weight λ is used to dynamically adjust the impact of binary labels and continuous labels on the model optimization process, and finally obtain the loss function of the self-boosting training paradigm Where: H, W represent the length and width of the input training image, Y i,j , Z i,j , P i,j are the binary label Y, the salient label Z, and the element value of the (i, j) position in the salient object prediction result P, respectively. The loss function calculated for the prediction results and consistent continuous labels, The loss function for predicting the result and calculating the binary discrete value; In step S5, the input image is augmented by data augmentation, and the generalization ability of the saliency model is improved by multi-level consistency regularization operation, which specifically includes the following sub-steps: S51, by the original sample I of the original input train Perform data augmentation by adding Gaussian noise and random cropping to construct extended samples Used to simulate samples affected by interference factors: S52, expand the sample Input to the salient object detection model , obtain the corresponding significant prediction results S53. Regularization constraints are imposed on the prediction consistency between the original sample and the expanded sample: in: Indicates significant prediction results The element value at position (i, j) in ; S54. Encoder module using salient object detection model The intermediate image features are extracted from the original sample and the expanded sample respectively, and the regularization constraint of multi-scale feature consistency is imposed on them: in: Represents the F-norm (Frobenius Norm) of the matrix.

2. According to claim 1, a salient object detection training method based on self-boosting learning is characterized in that: The hyperparameter η is set to 0.

99.

3. According to claim 1, a salient object detection training method based on self-boosting learning is characterized in that: The hyperparameter ζ is set to 70.

4. A salient object detection training method based on self-boosting learning according to claim 1, characterized in that: In step S6, the self-boosting training framework established by S1 to S5 is used in the training process of the salient object detection model to guide the model learning process and help the model better predict the salient objects in the image, which specifically includes the following sub-steps: S61. Setting up the salient object detection model The total loss function is: S62. Use SGD optimization method and back propagation algorithm through loss function To train a salient object detection model until the loss function converges.