A deep unsupervised salient object detection method and system based on dynamic fusion

Abstract

The application belongs to the technical field of image processing, and particularly relates to a deep unsupervised salient object detection method and system based on dynamic fusion, which comprises the following steps: inputting images to be trained into different traditional manual methods and full convolution networks respectively to obtain initial pseudo labels; fusing the different initial pseudo labels in an average weight manner to output intermediate results; substituting the different initial pseudo labels and the intermediate results into a distance function and then into a softmax logistic regression model to obtain different new weights; fusing the different new weights and the corresponding different initial pseudo labels to obtain new intermediate results, and then performing multiple iterations until the obtained intermediate results tend to be stable, and taking the intermediate result obtained in the last iteration as a final pseudo label. The application can improve the quality of the final pseudo label, and has better completeness and consistency for salient object detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a deep unsupervised salient object detection method and system based on dynamic fusion. BACKGROUND

[0002] Salient object detection can be used as a preprocessing step to improve the processing speed of various vision tasks such as segmentation, image fusion, image retrieval, image recognition, etc. The task of salient object detection aims to simulate human visual recognition to identify the most attractive area or object in the scene.

[0003] Unsupervised deep learning salient object detection is completely independent of labels. However, the existing unsupervised deep learning salient object detection method cannot well mine the consistency and complementarity between the pseudo-labels of various traditional manual methods, and the foreground and background segmentation quality of the generated pseudo-labels is low. SUMMARY

[0004] The technical problem to be solved by the application is that the foreground and background segmentation quality of the pseudo-labels generated by the existing unsupervised deep learning salient object detection method is low.

[0005] The technical scheme adopted by the application to solve the technical problem is: a deep unsupervised salient object detection method based on dynamic fusion, comprising:

[0006] S1, inputting the image to be trained into different traditional manual methods to obtain different saliency cues of the training set;

[0007] S2, inputting the image to be trained into a fully convolutional network, and inputting different saliency cues into the fully convolutional network for supervision to obtain different saliency predictions;

[0008] S3, inputting different saliency predictions as new different saliency cues into step S2 for multiple iterations to obtain different final saliency prediction results as different initial pseudo-labels;

[0009] S4, fusing the different initial pseudo-labels by using an average weight to output an intermediate result;

[0010] S5, substituting the different initial pseudo-labels and the intermediate result into a distance function to output multiple results;

[0011] S6, substituting the output multiple results into a softmax logistic regression model to obtain different new weights;

[0012] S7, the different new weights are fused with the corresponding different initial pseudo labels as new intermediate results, input to step S5 and step S6 for multiple iterations until the intermediate results obtained tend to be stable, and the intermediate results obtained by the last iteration are taken as the final pseudo labels.

[0013] Further, in S1, the different traditional manual methods include MC, HS, DSR, and RBD algorithms.

[0014] Further, in S2, the fully convolutional network is used to extract significant features in the image, and the last fully connected layer is replaced with an upsampling layer to generate a significant prediction.

[0015] Further, in S2, when the fully convolutional network is trained, the error function L used is:

[0016] L = 1-f β (p,pl)

[0017] Wherein, L is the loss function value, f β (p,pl) is a measure of the similarity of p and pl.

[0018] Further, the f β (p,pl) is a measure of the similarity of p and pl, and the specific formula is as follows:

[0019]

[0020] Wherein, p is the predicted significant prediction, pl is the significant clue, and β is a preset parameter.

[0021] Further, in S5, the distance function has the following specific formula:

[0022] dis(x,y) = -ln(1-f β (x,y))

[0023] Wherein, dis(x,y) is the distance, f β (x,y) is a measure of the similarity of x and y.

[0024] Further, the f β (x,y) is a measure of the similarity of x and y, and the specific formula is as follows:

[0025]

[0026] Wherein, x is different initial pseudo label, and y is intermediate result.

[0027] A deep unsupervised significant target detection system based on dynamic fusion, comprising:

[0028] A storage unit is configured to store different traditional manual methods, a full convolution network model, and a softmax logistic regression model.

[0029] A first processing unit is configured to input an image to be trained into different traditional manual methods to obtain different training sets of saliency cues.

[0030] A second processing unit is configured to input the image to be trained into a full convolution network, and input different saliency cues into the full convolution network for supervision to obtain different saliency predictions.

[0031] A first output unit is configured to input different saliency predictions as new different saliency cues into step S2 for multiple iterations to obtain different final saliency prediction results as different initial pseudo labels.

[0032] A fusion unit is configured to fuse different initial pseudo labels in an average weight manner to output an intermediate result.

[0033] A distance calculation unit is configured to input different initial pseudo labels and the intermediate result into a distance function to output multiple results.

[0034] A weight calculation unit is configured to input the multiple results into a softmax logistic regression model to obtain different new weights.

[0035] A second output unit is configured to fuse the different new weights and corresponding different initial pseudo labels as new intermediate results, and input the new intermediate results into the distance calculation unit and the weight calculation unit for multiple iterations until the intermediate results tend to be stable, and take the intermediate result obtained in the last iteration as a final pseudo label.

[0036] A network-side server, comprising:

[0037] at least one processor; and

[0038] a memory connected in communication with the at least one processor; wherein

[0039] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the above-described deep unsupervised salient object detection methods based on dynamic fusion.

[0040] A computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above-described deep unsupervised salient object detection method based on dynamic fusion.

[0041] The beneficial effects of the present application are: the present application adopts a dynamic fusion method to explore the consistency and complementarity between different traditional manual methods to generate pseudo labels, compared with the previous simple common use, the weight of different pseudo label fusion is adjusted according to the specific situation of the image to generate the corresponding pseudo label, which is helpful for detecting the salient target in complex scenes with different sizes, and the application has better robustness for different size salient targets when detecting salient targets in complex scenes, the distance function is based on the commonly used saliency measurement method, and a logarithmic function is used to amplify the difference to obtain a suitable fusion weight, which can improve the quality of the pseudo label. BRIEF DESCRIPTION OF DRAWINGS

[0042] The present application will be further described below in combination with the drawings and embodiments.

[0043] Figure 1 is a deep unsupervised salient object detection method flow chart based on dynamic fusion according to the first embodiment of the present application;

[0044] Figure 2 is a structure schematic diagram of a deep unsupervised salient object detection system based on dynamic fusion according to the second embodiment of the present application;

[0045] Figure 3 is a structure schematic diagram of a network side server according to the third embodiment of the present application;

[0046] Figure 4 is a simulation experiment diagram according to the first embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the present application will be described clearly and completely in combination with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0048] The first embodiment of the present application relates to a deep unsupervised salient object detection method based on dynamic fusion. In the embodiment, the image to be trained is input into different conventional manual methods to obtain different saliency clues of the training set; the image to be trained is input into a full convolution network, and different saliency clues are input into the full convolution network for supervision to obtain different saliency predictions; the different saliency predictions are taken as new different saliency clues and input into step S2 for multiple iterations to obtain different final saliency prediction results as different initial pseudo labels; the different initial pseudo labels are fused by using an average weight mode to output an intermediate result; the different initial pseudo labels and the intermediate result are substituted into a distance function to output multiple results; the multiple output results are substituted into a softmax logistic regression model to obtain different new weights; after the different new weights are fused with the corresponding different initial pseudo labels, the new intermediate result is obtained and input into steps S5 and S6 for multiple iterations until the intermediate result obtained tends to be stable, and the intermediate result obtained in the last iteration is taken as a final pseudo label. The present application can explore the consistency and complementarity between the pseudo labels generated by different conventional manual methods, adjust the weight of the fusion of different pseudo labels, improve the quality of the final pseudo label, and has better completeness and consistency for salient object detection.

[0049] The implementation details of the deep unsupervised salient object detection method based on dynamic fusion of the present embodiment will be described in detail below. The following content is only provided for the implementation details for easy understanding, and is not necessary for implementing the present solution. The specific process of the present embodiment is as shown in Figure 1 The present embodiment is applied to a server on the network side.

[0050] In step S1, the image to be trained is input into different conventional manual methods to obtain different saliency clues of the training set.

[0051] Specifically, the conventional manual method adopts different priors, such as intermediate priors and global contrast priors, and the different saliency clues of the training set obtained by inputting the image to be trained into multiple different conventional manual methods can enhance the robustness of the final result.

[0052] In some examples, the different conventional manual methods include MC, HS, DSR, and RBD algorithms.

[0053] In step S2, the image to be trained is input into a full convolution network, and different saliency clues are input into the full convolution network for supervision to obtain different saliency predictions.

[0054] Specifically, the full convolutional network is used to extract salient features in the image, and the last fully connected layer is removed and replaced with an up-sampling layer to generate a saliency prediction.

[0055] During training of the full convolutional network, an error function L used is:

[0056] L = 1 - f β (p,pl)

[0057] Wherein, L is a loss function value, f β (p,pl) is a measure of the similarity between p and pl.

[0058] The f β (p,pl) is a measure of the similarity between p and pl, and the specific formula is as follows:

[0059]

[0060] Wherein, p is a predicted saliency prediction, pl is a saliency clue, and β is a preset parameter.

[0061] Step S3, different saliency predictions are input as new different saliency clues into step S2 for multiple iterations to obtain different final saliency prediction results as different initial pseudo labels.

[0062] Specifically, the saliency clue is used to supervise a full convolutional network which removes the last fully connected layer and replaces it with an up-sampling layer, and the historical average saliency prediction in the cumulative training process is accumulated, these saliency predictions are used to supervise the full convolutional network again by replacing the previous clues, and this replacement and retraining process is iterated several times, and the final saliency prediction is the desired pseudo label. The historical average saliency prediction comes from the prediction generated during training of the full convolutional network, and these predictions are continuously fused into the historical average saliency prediction with a certain weight. Step S4, different initial pseudo labels are fused in an average weight manner to output an intermediate result.

[0063] Step S5, different initial pseudo labels and intermediate results are respectively substituted into a distance function to output multiple results.

[0064] Specifically, the distance function has the following specific formula:

[0065] dis(x,y) = -ln(1-f β (x,y))

[0066] Wherein, dis(x,y) is a distance, f β (x,y) is a measure of the similarity between x and y.

[0067] The f β(x, y) is a measure of the similarity of x and y, and the specific formula is as follows:

[0068]

[0069] Wherein, x is different initial pseudo label, y is intermediate result.

[0070] Step S6, the output multiple results are substituted into a softmax logistic regression model to obtain different new weights.

[0071] Step S7, the different new weights are fused with the corresponding different initial pseudo labels as new intermediate results, and are input into steps S5 and S6 for multiple iterations until the intermediate results tend to be stable, and the intermediate result obtained by the last iteration is taken as the final pseudo label.

[0072] In some examples, as shown in Figure 4 The original image Img represents an image for experimental input in a database, the ground truth GT represents a binary image artificially labeled, and the number of times indicates the intermediate result of the number of iterations; as the number of iterations increases, the dynamic fusion method further mines the consistency between images by finely adjusting the weights of different methods, and removes the influence of background noise generated by some methods on the fusion result.

[0073] In some other examples, the present application and the existing unsupervised salient object detection method are used for salient object detection experiment on a public image database ECSSD, and the obtained results are objectively evaluated by using a recognized evaluation index, i.e., a weighted average F-measure value and an MAE value, and the evaluation simulation results are shown in Table 1.

[0074] Table 1

[0075]

[0076]

[0077] The present application has a higher weighted average F-measure value and a lower MAE value, thereby indicating that the present application has better integrity and consistency for salient object detection, and fully shows the effectiveness and superiority of the method of the present application.

[0078] The step division of the above various methods is only for the purpose of clear description, and in the implementation, one step can be combined or some steps can be split and decomposed into multiple steps, as long as the same logical relationship is included, and all are within the protection scope of the present patent; adding irrelevant modifications or introducing irrelevant designs in the process, but not changing the core design of the algorithm and process are within the protection scope of the present patent.

[0079] AsFigure 2 The second embodiment of the present application shown in the figure relates to a deep unsupervised salient object detection system based on dynamic fusion, comprising: a storage unit 201, a first processing unit 202, a second processing unit 203, a first output unit 204, a fusion unit 205, a distance calculation unit 206, a weight calculation unit 207, and a second output unit 208.

[0080] Specifically, the storage unit 201 is configured to store different traditional manual methods, a full convolution network model, and a softmax logistic regression model; the first processing unit 202 is configured to input a to-be-trained image into different traditional manual methods to calculate different training set saliency cues; the second processing unit 203 is configured to input the to-be-trained image into a full convolution network, and input different saliency cues into the full convolution network for supervision to obtain different saliency predictions, respectively; the first output unit 204 is configured to input the different saliency predictions as new different saliency cues into the second processing unit 203 for multiple iterations to obtain different final saliency prediction results as different initial pseudo labels; the fusion unit 205 is configured to fuse the different initial pseudo labels in an average weight manner to output an intermediate result; the distance calculation unit 206 is configured to input the different initial pseudo labels and the intermediate result into a distance function, respectively, to output multiple results; the weight calculation unit 207 is configured to input the multiple output results into a softmax logistic regression model to obtain different new weights; and the second output unit 208 is configured to fuse the different new weights with the corresponding different initial pseudo labels as new intermediate results, and input the new intermediate results into the distance calculation unit 206 and the weight calculation unit 207 for multiple iterations until the intermediate results tend to be stable, and take the intermediate result obtained in the last iteration as a final pseudo label.

[0081] It can be found that the present embodiment is a system embodiment corresponding to the first embodiment, and the present embodiment can be implemented in cooperation with the first embodiment. The related technical details mentioned in the first embodiment are still valid in the present embodiment. In order to reduce repetition, they will not be described here. Correspondingly, the related technical details mentioned in the present embodiment can also be applied in the first embodiment.

[0082] It is worth mentioning that each module involved in the present embodiment is a logical module. In actual application, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units not closely related to solving the technical problems proposed in the present application are not introduced in the present embodiment, but this does not mean that there are no other units in the present embodiment.

[0083] The third embodiment of the present application relates to a network side server, such as Figure 3 As shown, the network side server comprises at least one processor 301; and a memory 302 connected with the at least one processor 301; wherein the memory 302 stores instructions executable by the at least one processor 301, and the instructions are executed by the at least one processor 301 to enable the at least one processor 301 to perform the data processing method described above.

[0084] The memory 302 and the processor 301 are connected in a bus manner, the bus can include any number of interconnected buses and bridges, and the bus connects one or more processors 301 and various circuits of the memory 302 together. The bus can also connect various other circuits such as peripheral devices, voltage stabilizers and power management circuits together, which are well known in the art, and therefore, further description is not given herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements such as multiple receivers and transmitters, which provide a unit for communicating with various other devices on the transmission medium. The data processed by the processor 301 is transmitted on the wireless medium through the antenna, and further, the antenna also receives data and transmits the data to the processor 301.

[0085] The processor 301 is responsible for managing the bus and general processing, and can also provide various functions including timing, peripheral interface, voltage regulation, power management and other control functions. And the memory 302 can be used to store the data used by the processor 301 in the execution of the operation.

[0086] The fourth embodiment of the present application relates to a computer readable storage medium, which stores a computer program. The computer program is executed by the processor to realize the deep unsupervised salient object detection system based on dynamic fusion in the first embodiment.

[0087] That is, those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing related hardware, the programs are stored in a storage medium, and the storage medium includes a plurality of instructions for enabling a device (which can be a single chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage medium capable of storing program codes.

[0088] The above is the ideal embodiment of the present application, and the above description can be changed and modified without departing from the scope of the present application. The technical scope of the present application is not limited to the above description, and should be determined by the scope of the claims.

Claims

1. A deep unsupervised salient object detection method based on dynamic fusion, characterized in that, The method comprises the following steps: S1, inputting the image to be trained into different traditional manual methods to obtain different training set saliency cues; S2, inputting the image to be trained into a full convolutional network, inputting different saliency cues into the full convolutional network for supervision, and obtaining different saliency predictions; S3, inputting the different saliency predictions as new different saliency cues into step S2 for multiple iterations to obtain different final saliency prediction results as different initial pseudo labels; S4, fusing the different initial pseudo labels by using an average weight to output an intermediate result; S5, inputting the different initial pseudo labels and the intermediate result into a distance function to output multiple results; S6, inputting the multiple output results into a softmax logistic regression model to obtain different new weights; S7, fusing the different new weights and the corresponding different initial pseudo labels to obtain a new intermediate result, and inputting the new intermediate result into steps S5 and S6 for multiple iterations until the intermediate result tends to be stable, and taking the intermediate result obtained in the last iteration as a final pseudo label.

2. The method according to claim 1, wherein, In S1, the different traditional manual methods include MC, HS, DSR and RBD algorithms. 3.The deep unsupervised salient object detection method based on dynamic fusion according to claim 1, wherein, In S2, the full convolutional network is used to extract saliency features in the image, and the last fully connected layer is replaced by an up-sampling layer to generate a saliency prediction. 4.The deep unsupervised salient object detection method based on dynamic fusion according to claim 1, wherein, In S2, when the full convolutional network is trained, an error function L used is: L = 1 - f β (p, pl) where L is a loss function value, f β (p, pl) is a measure of similarity of p and pl.

5. The method according to claim 4, wherein, The f β (p, pl) is a measure of the similarity of p and pl, and is specifically formulated as follows: Wherein, p is a predicted saliency prediction, pl is a saliency cue, and β is a preset parameter.

6. The method according to claim 1, wherein, In S5, the distance function is specifically as follows: dis(x,y) = -ln(l-f β (x,y)) where dis(x, y) is the distance, f β (x, y) is a measure of similarity of x and y.

7. The dynamic fusion based deep unsupervised salient object detection method according to claim 6, characterized in that, The f β (x,y) is a measure of similarity of x and y, and is given by the following formula: Wherein, x is a different initial pseudo label, and y is an intermediate result.

8. A dynamic fusion based deep unsupervised salient object detection system, characterized in that, The method comprises the following steps: A storage unit for storing different traditional manual methods, a full convolutional network model and a softmax logistic regression model; A first processing unit for inputting an image to be trained into different traditional manual methods to calculate different training set saliency cues; A second processing unit for inputting the image to be trained into a full convolutional network, inputting different saliency cues into the full convolutional network for supervision, and obtaining different saliency predictions; A first output unit for inputting different saliency predictions as new different saliency cues into the second processing unit for multiple iterations to obtain different final saliency prediction results as different initial pseudo labels; A fusion unit for fusing different initial pseudo labels by using an average weight to output an intermediate result; A distance calculation unit for inputting different initial pseudo labels and an intermediate result into a distance function to output multiple results; A weight calculation unit for inputting the multiple output results into a softmax logistic regression model to obtain different new weights; A second output unit for fusing different new weights and corresponding different initial pseudo labels to obtain a new intermediate result, and inputting the new intermediate result into the distance calculation unit and the weight calculation unit for multiple iterations until the intermediate result tends to be stable, and taking the intermediate result obtained in the last iteration as a final pseudo label.

9. A network side server, characterized by, The method comprises the following steps: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for deep unsupervised salient object detection based on dynamic fusion according to any one of claims 1 to 7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method for deep unsupervised salient object detection based on dynamic fusion according to any one of claims 1 to 7.

Images