A visual saliency detection method and system based on multi-layer non-local network

Abstract

The present invention discloses a method and system for visual saliency detection based on a multi-layer non-local network, which obtains an image data set to be detected; inputs the image data set to be detected into a pre-trained saliency detection model to obtain a saliency image set; the pre-trained saliency detection model is a multi-layer non-local network model trained with the minimum value of a loss function as the goal. Advantages: The multi-layer non-local network of the present method expands the receptive field of the model. In contrast to the repetitiveness of the loop operation, the non-local module can directly capture long-distance dependencies by calculating the interaction between any two pixels, and is no longer constrained by the position distance of the two pixels, thereby improving efficiency and obtaining better results. In the process of training the model, a loss function composed of relative entropy, standardized scanning path saliency and structural similarity is used, so that the optimal model obtained by training has better comprehensiveness and universality.

Description

Technical Field

[0001] The invention relates to a visual saliency detection method and system based on a multi-layer non-local network, belonging to the technical field of saliency detection in image processing. Background Art

[0002] The human visual system can quickly discover and focus on the content of interest in an image. Therefore, people have begun to study the visual selective attention mechanism that simulates the human visual system. Image saliency detection technology has also become a research hotspot in recent years. It has extremely important applications in target tracking, navigation, image quality evaluation, face recognition, video optimization coding, etc.

[0003] Saliency detection aims to identify the most eye-catching objects from the input image. From 1998 to the present, salient object detection can be roughly divided into two eras: traditional methods and deep learning methods.

[0004] In the past two decades, traditional salient object detection methods can be divided into two different categories according to the type of visual subsets used or the features used: ① using block-based visual subsets or region-based visual subsets; ② using only internal clues provided by the image itself or introducing external clues such as user annotations. In general, traditional methods use a large amount of salient prior information for image saliency detection, mainly relying on hand-crafted features, which may not be able to describe complex image scenes and object structures, and cannot adapt to new scenes and objects, and have poor generalization ability, which leads to a bottleneck in saliency detection based on traditional methods. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a visual saliency detection method and system based on a multi-layer non-local network.

[0006] In order to solve the above technical problems, the present invention provides a visual saliency detection method based on a multi-layer non-local network.

[0007] Obtain the image dataset to be detected;

[0008] The image data set to be detected is input into a pre-trained saliency detection model to obtain a saliency image set; the pre-trained saliency detection model is a multi-layer non-local network model trained with the minimum value of a loss function as the target.

[0009] Furthermore, the multi-layer non-local network model is a multi-layer non-local neural network based on the architecture of VGG16.

[0010] Furthermore, the multi-layer non-local neural network based on the VGG16 architecture includes an input layer, an encoder module, a multi-scale feature module, a decoder module and an output layer connected in sequence;

[0011] The encoder module includes a first convolution unit, a second convolution unit, a third convolution unit, a fourth convolution unit and a fifth convolution unit connected in sequence; wherein the first convolution unit and the second convolution unit are each composed of two convolution layers with convolution kernels of 3×3 connected in sequence and a maximum pooling layer, and the third convolution unit, the fourth convolution unit and the fifth convolution unit are each composed of three convolution layers with convolution kernels of 3×3 connected in sequence, a maximum pooling layer and a non-local module layer;

[0012] The multi-scale feature module includes a first convolution layer, a second expansion convolution layer, a third expansion convolution layer, a fourth expansion convolution layer, a mean layer and a fifth convolution layer connected in sequence; wherein the first convolution layer is a convolution layer with a convolution kernel of 1×1, the second expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 4, the third expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 8, the fourth expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 12, the fifth convolution layer is a convolution layer with a convolution kernel of 1×1, the input of the mean layer is the output of the fourth expansion convolution layer, and the mean of the input is obtained under the condition that the input size is guaranteed to be unchanged, and it is used as the input of the fifth convolution layer;

[0013] The decoding module includes a first deconvolution unit, a second deconvolution unit and a third deconvolution unit which are connected in sequence; wherein the first deconvolution unit, the second deconvolution unit and the third deconvolution unit are each composed of an upsampling layer and a convolution layer with a convolution kernel of 3×3.

[0014] Furthermore, the loss function is:

[0015]

[0016] Where N is the total number of images in the dataset, i represents the subscript of the i-th image, and KL i 、NSS i 、SSIM i are the relative entropy, normalized scan path saliency, and structural similarity of the ith image, respectively.

[0017] A visual saliency detection system based on a multi-layer non-local network, comprising:

[0018] An acquisition module, used to acquire an image data set to be detected;

[0019] The processing module is used to input the image data set to be detected into a pre-trained saliency detection model to obtain a saliency image set; the pre-trained saliency detection model is a multi-layer non-local network model trained with the minimum value of the loss function as the goal.

[0020] Furthermore, the processing module includes a model building module for building a multi-layer non-local neural network based on the VGG16 architecture.

[0021] Furthermore, the model building module is used to construct an input layer, an encoder module, a multi-scale feature module, a decoder module and an output layer connected in sequence;

[0022] The encoder module comprises a first convolution unit, a second convolution unit, a third convolution unit, a fourth convolution unit and a fifth convolution unit connected in sequence; wherein the first convolution unit and the second convolution unit are each composed of two convolution layers with convolution kernels of 3×3 connected in sequence and a maximum pooling layer, and the third convolution unit, the fourth convolution unit and the fifth convolution unit are each composed of three convolution layers with convolution kernels of 3×3 connected in sequence, a maximum pooling layer and a non-local module layer;

[0023] The multi-scale feature module includes a first convolution layer, a second expansion convolution layer, a third expansion convolution layer, a fourth expansion convolution layer, a mean layer and a fifth convolution layer connected in sequence; wherein the first convolution layer is a convolution layer with a convolution kernel of 1×1, the second expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 4, the third expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 8, the fourth expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 12, the fifth convolution layer is a convolution layer with a convolution kernel of 1×1, the input of the mean layer is the output of the fourth expansion convolution layer, and the mean of the input is obtained under the condition that the input size is guaranteed to be unchanged, and it is used as the input of the fifth convolution layer;

[0024] The decoding module includes a first deconvolution unit, a second deconvolution unit and a third deconvolution unit which are connected in sequence; wherein the first deconvolution unit, the second deconvolution unit and the third deconvolution unit are each composed of an upsampling layer and a convolution layer with a convolution kernel of 3×3.

[0025] Furthermore, the processing module includes a loss function determination module, which is used to determine the following loss function:

[0026]

[0027] Where N is the total number of images in the dataset, i represents the subscript of the i-th image, and KL i 、NSS i 、SSIM iare the relative entropy, normalized scan path saliency, and structural similarity of the ith image, respectively.

[0028] The beneficial effects achieved by the present invention are:

[0029] 1. The multi-layer non-local network of this method expands the receptive field of the model. In contrast to the repetitiveness of the loop operation, the non-local module can directly capture long-distance dependencies by calculating the interaction between any two pixels, without being constrained by the position distance of the two pixels, which improves efficiency and obtains better results.

[0030] 2. In the process of training the model, a loss function composed of relative entropy, standardized scanning path significance and structural similarity was used to make the optimal model obtained through training have better comprehensiveness and universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of a visual saliency detection method based on a multi-layer non-local network provided by the present invention;

[0032] Figure 2 It is a schematic diagram of the structure of a multi-layer non-local network;

[0033] Figure 3 Schematic diagram of the comparison of saliency maps obtained by various algorithms on the test set in the SALICON dataset and the MIT1003 test set provided for the implementation of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0035] A visual saliency detection method based on multi-layer non-local network, such as Figure 1 As shown, it includes the following four steps:

[0036] Step 1) Build a multi-layer non-local network based on VGG16.

[0037] In the implementation of the present invention, the proposed multi-layer non-local neural network is improved based on the architecture of VGG16. The multi-layer non-local network mainly consists of three parts: an encoder module, a multi-scale feature module and a decoder module connected in sequence. The encoder module uses convolutional layers of different levels to retain fine features, and adds non-local modules at the end of the third, fourth and fifth convolutional units to process global information; the multi-scale feature module uses five convolutional layers with different expansion factors to capture multi-scale image information; the decoder module uses a combination of three upsampling layers and convolutional layers to restore the feature map to the resolution of the original image.

[0038] like Figure 2 As shown, the multi-layer non-local network includes an input layer, a first convolution unit, a second convolution unit, a third convolution unit, a fourth convolution unit, a fifth convolution unit, a multi-scale feature module, a decoder module, and an output layer connected in sequence.

[0039] The first convolution unit consists of two 3×3×64 convolutional layers and a maximum pooling layer connected in sequence.

[0040] The second convolutional unit consists of two 3×3×128 convolutional layers and a maximum pooling layer connected in sequence.

[0041] The third convolutional unit consists of three 3×3×256 convolutional layers, a maximum pooling layer, and a non-local module layer connected sequentially.

[0042] The fourth convolutional unit consists of three 3×3×512 convolutional layers, a maximum pooling layer, and a non-local module layer connected in sequence.

[0043] The fifth convolutional unit consists of three 3×3×512 convolutional layers, a maximum pooling layer, and a non-local module layer connected in sequence.

[0044] The multi-scale feature module of the multi-layer non-local network samples the given input in parallel with dilated convolutions at different sampling rates, which is equivalent to capturing the information of the image at multiple scales. The multi-scale feature module consists of the first convolution layer, the second dilated convolution layer, the third dilated convolution layer, the fourth dilated convolution layer, the mean layer and the fifth convolution layer connected in sequence. Among them, the first convolution layer is a convolution layer with 256 convolution kernels of 1×1, the second dilated convolution layer is a convolution layer with 256 convolution kernels of 3×3 and a dilation rate of 4, the third dilated convolution layer is a convolution layer with 256 convolution kernels of 3×3 and a dilation rate of 8, the fourth dilated convolution layer is a convolution layer with 256 convolution kernels of 3×3 and a dilation rate of 12, and the fifth convolution layer is a convolution layer with 256 convolution kernels of 1×1. The input of the mean layer is the output of the fourth dilated convolution layer. Under the condition of ensuring that the input size remains unchanged, the mean of the input is obtained and used as the input of the fifth convolution layer.

[0045] The decoder module includes a first deconvolution unit, a second deconvolution unit, and a third deconvolution unit connected in sequence. The first deconvolution unit consists of an upsampling layer and a convolution layer consisting of 128 3×3 convolution kernels, the second deconvolution unit consists of an upsampling layer and a convolution layer consisting of 64 3×3 convolution kernels, and the third deconvolution unit consists of an upsampling layer and a convolution layer consisting of 32 3×3 convolution kernels.

[0046] In the implementation of the present invention, the improvements of the multi-layer non-local network over the VGG16 network are as follows:

[0047] (1) A non-local module is embedded on the basis of the VGG16 network, which is conducive to combining the local information and global information of the image to capture the salient features of the image.

[0048] (2) The three embedded non-local modules are not directly embedded into the network as a whole, but are located after the third, fourth, and fifth convolutional units of the encoding module, respectively, which expands the receptive field of the model.

[0049] Step 2) Train the model with the minimum value of the loss function as the target.

[0050] In the implementation of the present invention, the training set in the contextual saliency dataset (SALICON) is used as the training dataset to input the multi-layer non-local network, and the minimum value of the loss function is set as the target. The learning rate of the training is set to 0.00001, the number of iterations is 1000, and the proposed multi-layer non-local network is trained to obtain a saliency detection model. The loss function defined is:

[0051]

[0052] Where N is the total number of images in the dataset, i represents the subscript of the i-th image, and KL i 、NSS i 、SSIM i are the relative entropy, normalized scan path saliency, and structural similarity of the ith image, respectively.

[0053] Relative entropy is usually used to measure the distance between two distributions. For saliency maps, it is the distance between the distribution of saliency map values ​​and the distribution of predicted map values. Normalized scan path saliency is defined by the average saliency value corresponding to the position of the human eye gaze point in the prediction model. Image structural similarity is an indicator to measure the similarity between two images. It is mostly used for image quality evaluation. The reason why it is used for image saliency evaluation here is that it is composed of three parts: brightness contrast, contrast contrast, and structure contrast. The structural similarity index within the window is calculated through a local window, and the window is moved in pixels until the local structural similarity index at each position is calculated, and then the average is taken as the global structural similarity index.

[0054] The loss function composed of these three indicators provides a more comprehensive feedback on the quality of the model.

[0055] The improvements in the implementation of the present invention are as follows:

[0056] The loss function proposed in the present invention is composed of three indicators for measuring the structural similarity of two images, namely relative entropy, normalized scanning path saliency and image structure similarity. The weight ratio of these three indicators is 1:1:1, which enhances the applicability of the model. The loss function composed of these three indicators can more comprehensively feedback the advantages and disadvantages of the model.

[0057] Step 3) Input the image data set to be detected into the saliency detection model to obtain a saliency image set.

[0058] After obtaining the optimal model through the operation in step 2, use the test set in the SALICON dataset and the test set in the MIT saliency dataset (MIT1003) as the test dataset, and make predictions on the trained model to obtain the saliency image set. Figure 3 , a schematic diagram of the comparison of saliency maps obtained by various algorithms on the test set in the data set SALICON and the MIT1003 test set provided by the implementation of the present invention is shown in FIG., wherein, from left to right are the original image, the saliency map of the salient attention model (SAM-VGG) based on the backbone network of the University of Oxford Visual Geometry Group (VGG), the saliency map of the salient attention model (SAM-ResNet) based on the residual network (ResNet), the saliency map of the present invention and the true heat map (GT), the first three of the original images are selected from the test set in the data set SALICON, and the last original image is selected from the MIT1003 test set.

[0059] Step 4) The obtained saliency image set is evaluated by indicators including relative entropy (KLD), standardized scan path significance (NSS), correlation coefficient (CC) and area under the ROC curve (AUC-J).

[0060] Table 1 is a comparison of AUC-J, KLD and CC between the saliency detection method based on multi-layer non-local network proposed in the present invention and other methods.

[0061] Table 1

[0062]

[0063]

[0064] It can be seen from Table 1 that the results of the method of the present invention are better than other methods in terms of the three indicators of AUC-J, KLD and CC. This is because the multi-layer non-local network combines global information and local information to capture significant features, and combines a composite loss function to train the model, which has good comprehensiveness and universality.

[0065] Correspondingly, the present invention also provides a visual saliency detection system based on a multi-layer non-local network, comprising:

[0066] An acquisition module, used to acquire an image data set to be detected;

[0067] The processing module is used to input the image data set to be detected into a pre-trained saliency detection model to obtain a saliency image set; the pre-trained saliency detection model is a multi-layer non-local network model trained with the minimum value of the loss function as the goal.

[0068] The processing module includes a model building module for building a multi-layer non-local neural network based on the architecture of VGG16.

[0069] The model building module is used to construct an input layer, an encoder module, a multi-scale feature module, a decoder module and an output layer connected in sequence;

[0070] The encoder module comprises a first convolution unit, a second convolution unit, a third convolution unit, a fourth convolution unit and a fifth convolution unit connected in sequence; wherein the first convolution unit and the second convolution unit are each composed of two convolution layers with convolution kernels of 3×3 connected in sequence and a maximum pooling layer, and the third convolution unit, the fourth convolution unit and the fifth convolution unit are each composed of three convolution layers with convolution kernels of 3×3 connected in sequence, a maximum pooling layer and a non-local module layer;

[0071] The multi-scale feature module includes a first convolution layer, a second expansion convolution layer, a third expansion convolution layer, a fourth expansion convolution layer, a mean layer and a fifth convolution layer connected in sequence; wherein the first convolution layer is a convolution layer with a convolution kernel of 1×1, the second expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 4, the third expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 8, the fourth expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 12, the fifth convolution layer is a convolution layer with a convolution kernel of 1×1, the input of the mean layer is the output of the fourth expansion convolution layer, and the mean of the input is obtained under the condition that the input size is guaranteed to be unchanged, and it is used as the input of the fifth convolution layer;

[0072] The decoding module includes a first deconvolution unit, a second deconvolution unit and a third deconvolution unit which are connected in sequence; wherein the first deconvolution unit, the second deconvolution unit and the third deconvolution unit are each composed of an upsampling layer and a convolution layer with a convolution kernel of 3×3.

[0073] The processing module includes a loss function determination module, which is used to determine the following loss function:

[0074]

[0075] Where N is the total number of images in the dataset, i represents the subscript of the i-th image, and KL i 、NSSi 、SSIM i are the relative entropy, normalized scan path saliency, and structural similarity of the ith image, respectively.

[0076] This method builds a multi-layer non-local network based on the VGG16 network to expand the receptive field of the model. In contrast to the repetitiveness of the loop operation, the non-local module can directly capture long-distance dependencies by calculating the interaction between any two pixels, without being constrained by the position distance of the two pixels, which improves efficiency and obtains better results; in the process of training the model, a loss function composed of relative entropy, standardized scan path significance and structural similarity is used, so that the optimal model obtained by training has better comprehensiveness and universality.

[0077] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0078] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0081] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A visual saliency detection method based on a multi-layer non-local network, characterized in that: Obtain the image data set to be detected; Inputting the image data set to be detected into a pre-trained saliency detection model to obtain a saliency image set; the pre-trained saliency detection model is a multi-layer non-local network model trained with the minimum value of the loss function as the goal; The multi-layer non-local network model is a multi-layer non-local neural network based on the architecture of VGG16; The multi-layer non-local neural network based on the VGG16 architecture includes an input layer, an encoder module, a multi-scale feature module, a decoder module and an output layer connected in sequence; The encoder module includes a first convolution unit, a second convolution unit, a third convolution unit, a fourth convolution unit and a fifth convolution unit connected in sequence; wherein the first convolution unit and the second convolution unit are each composed of two convolution layers with convolution kernels of 3×3 connected in sequence and a maximum pooling layer, and the third convolution unit, the fourth convolution unit and the fifth convolution unit are each composed of three convolution layers with convolution kernels of 3×3 connected in sequence, a maximum pooling layer and a non-local module layer; The multi-scale feature module includes a first convolution layer, a second expansion convolution layer, a third expansion convolution layer, a fourth expansion convolution layer, a mean layer and a fifth convolution layer connected in sequence; wherein the first convolution layer is a convolution layer with a convolution kernel of 1×1, the second expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 4, the third expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 8, the fourth expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 12, the fifth convolution layer is a convolution layer with a convolution kernel of 1×1, the input of the mean layer is the output of the fourth expansion convolution layer, and the mean of the input is obtained under the condition that the input size is guaranteed to be unchanged, and it is used as the input of the fifth convolution layer; The decoding module includes a first deconvolution unit, a second deconvolution unit and a third deconvolution unit which are connected in sequence; wherein the first deconvolution unit, the second deconvolution unit and the third deconvolution unit are each composed of an upsampling layer and a convolution layer with a convolution kernel of 3×3.

2. The method for visual saliency detection based on a multi-layer non-local network according to claim 1, characterized in that: The loss function is: ； in, N is the total number of images in the dataset, i Indicates i The subscript of the image, KL i , NSS i , SSIM i Respectively i Relative entropy, normalized scan path saliency, and structural similarity of images.

3. A visual saliency detection system based on a multi-layer non-local network, characterized in that: include: An acquisition module, used to acquire an image data set to be detected; A processing module, used for inputting the image data set to be detected into a pre-trained saliency detection model to obtain a saliency image set; the pre-trained saliency detection model is a multi-layer non-local network model trained with the minimum value of the loss function as the target; The processing module includes a model building module for building a multi-layer non-local neural network based on the architecture of VGG16; The model building module is used to construct an input layer, an encoder module, a multi-scale feature module, a decoder module and an output layer connected in sequence; The encoder module comprises a first convolution unit, a second convolution unit, a third convolution unit, a fourth convolution unit and a fifth convolution unit connected in sequence; wherein the first convolution unit and the second convolution unit are each composed of two convolution layers with convolution kernels of 3×3 connected in sequence and a maximum pooling layer, and the third convolution unit, the fourth convolution unit and the fifth convolution unit are each composed of three convolution layers with convolution kernels of 3×3 connected in sequence, a maximum pooling layer and a non-local module layer; The multi-scale feature module includes a first convolution layer, a second expansion convolution layer, a third expansion convolution layer, a fourth expansion convolution layer, a mean layer and a fifth convolution layer connected in sequence; wherein the first convolution layer is a convolution layer with a convolution kernel of 1×1, the second expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 4, the third expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 8, the fourth expansion convolution layer is a convolution layer with a convolution kernel of 3×3 and a dilation rate of 12, the fifth convolution layer is a convolution layer with a convolution kernel of 1×1, the input of the mean layer is the output of the fourth expansion convolution layer, and the mean of the input is obtained under the condition that the input size is guaranteed to be unchanged, and it is used as the input of the fifth convolution layer; The decoding module includes a first deconvolution unit, a second deconvolution unit and a third deconvolution unit which are connected in sequence; wherein the first deconvolution unit, the second deconvolution unit and the third deconvolution unit are each composed of an upsampling layer and a convolution layer with a convolution kernel of 3×3.

4. The multi-layer non-local network-based visual saliency detection system according to claim 3, characterized in that: The processing module includes a loss function determination module, which is used to determine the following loss function: ； in, N is the total number of images in the dataset, i Indicates i The subscript of the image, KL i , NSS i , SSIM i Respectively i Relative entropy, normalized scan path saliency, and structural similarity of images.

Images