Cooperative saliency detection method and system based on sparse attention and edge guidance

Abstract

The invention discloses a collaborative saliency detection method and system based on sparse attention and edge guidance, and the method comprises the steps: generating a plurality of vivid virtual samples through a generative adversarial network augmentation module, and carrying out the alternate training of a generator and a discriminator, and achieving the augmentation of the samples; constructing a sparse attention mechanism based on the deep convolution features, constructing a target function, and solving the target function to obtain the region of interest and features of the sensitive object; based on a collaborative filtering mechanism, the sensitive object focusing result is purified, and the purified sensitive object features are obtained; realizing accurate positioning of a bounding box by utilizing the characteristic of complementation between edge characteristics and sensitive object characteristics, and enhancing bottom-layer characteristics and high-layer characteristics in the collaborative saliency detection network; and training the collaborative saliency detection network through the loss function to obtain a target detection network, and detecting and identifying the sensitive object through the target detection network. According to the invention, accurate detection and identification of the sensitive object are realized.

Description

Technical Field

[0001] This invention belongs to the field of target detection and recognition, and relates to the detection and recognition of sensitive objects. Specifically, it relates to a collaborative saliency detection method and system based on sparse attention and edge guidance. Background Technology

[0002] Sensitive objects can be considered a special type of target, and their detection falls within the research scope of general target detection. In security monitoring, the detection of sensitive objects is crucial, effectively improving the perception and response capabilities to potential security threats. However, because sensitive objects are infrequent in daily life, appear less frequently in images or videos, and often have low resolution, feature selection and extraction are challenging. Therefore, it is usually necessary to combine feature enhancement methods such as generative adversarial networks with salient target detection methods.

[0003] Currently, deep learning-based methods are the mainstream approach in object detection. Ross et al. [Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 580-587.] proposed the R-CNN framework, which uses deep convolutional networks to extract features of candidate boxes representing regions and trains an SVM classifier for object detection, achieving a significant breakthrough in performance. To reduce the computational cost of R-CNN, He et al. [He K, Zhang X, Ren S, et al. Spatialpyramid pooling in deep convolutional networks for visual recognition[J].IEEE transactions on pattern analysis and machine intelligence, 2015, 37(9):1904-1916.] proposed SPP-NET, which extracts candidate box features on the convolutional feature map, transforming the multiple convolutions in R-CNN into a single convolution, greatly reducing the computational cost. Subsequently, Ross [Girshick R. Fast r-CNN[C] / / Proceedings of the IEEE international conference on computer vision. 2015: 1440-1448.] proposed Fast R-CNN. Its main innovation is the introduction of a Region of Interest (RoI) pooling layer, which samples the convolutional feature maps of candidate boxes of different sizes into a fixed-size feature map. This layer can be directly differentiated, and the gradient is directly propagated to the backbone network for optimization during training, greatly reducing the computational cost. It should be noted that both SPP-Net and Fast R-CNN require independent candidate region generation modules, resulting in still high computational costs and difficulty in GPU acceleration.To address this, Ren et al. [Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(6): 1137-1149.] proposed Faster R-CNN. A Region Proposal Network (RPN) was added to the backbone network. By setting anchors of different scales according to certain rules, candidate boxes were extracted in the convolutional feature layers of the RPN to replace the traditional candidate box generation method, thus achieving end-to-end training of the network.

[0004] Unlike the two-step strategy of the R-CNN series, the YOLO series adopts a single-step strategy: removing the candidate box extraction branch and directly completing feature extraction, candidate box regression, and classification in a single convolutional network, simplifying the network structure and improving detection speed by nearly 10 times compared to Faster R-CNN. However, the detection accuracy of this method is not as good as the two-step R-CNN series. To address this, Liu et al. [Liu W, Anguelov D, Erhan D, et al. Ssd: Single shot multiboxdetector[C] / / European conference on computer vision. Cham: Springer International Publishing, 2016: 21-37.] proposed SSD, which achieves comparable accuracy to the two-stage method while maintaining a faster speed through grid partitioning and multi-scale feature map detection.

[0005] However, the above methods are mainly designed for general object detection tasks, and it is difficult to achieve excellent performance when directly applied to the detection of sensitive objects. For the problem of co-saliency object detection in multiple images, the mainstream methods can be mainly divided into three types: bottom-up methods, fusion-based methods, and learning-based methods. Among the bottom-up methods, Li and Ngan [Li H, Ngan K N. A co-saliency model of image pairs[J]. IEEE Transactions on Image Processing, 2011, 20(12): 3365-3375.] proposed to detect paired saliency by exploring single-image saliency and multi-image saliency cues. Among fusion-based methods, Cao et al. [Cao X, Cheng Y, Tao Z, et al. Co-saliency detection via base reconstruction[C] / / Proceedings of the 22nd ACM international conference on Multimedia. 2014:997-1000.] proposed a reconstruction-based fusion method that combines knowledge mined from several existing saliency detection methods based on reconstruction error; Huang et al. [Huang R, Feng W, Sun J, et al. Image saliency and co-saliency detection by low-rank multiscale fusion[J]. International Journal of High Performance Systems Architecture, 2019, 8(4): 225-237.] fused the obtained multiscale saliency maps by using low-rank analysis and introducing GMM-based co-saliency priors. In learning-based methods, Cheng et al. [Cheng MM, Mitra NJ, Huang X, et al. Salientshape: groupsaliency in image collections[J]. The visual computer, 2014, 30(4): 443-453.] first construct a coarse appearance model for the target image region and its background, and then use the appearance model to improve saliency detection and image segmentation, thereby taking advantage of the correlation between Internet images within the same category.

[0006] While these collaborative saliency target detection methods have improved upon the collaborative saliency target detection problem, they still have certain problems with sensitive object detection: the scarcity of sensitive objects leads to insufficient training data for ordinary collaborative saliency target detection algorithms; the sparse distribution of sensitive objects in sample images can cause related methods to erroneously detect other targets, thus leading to a decline in training performance. Summary of the Invention

[0007] Purpose of the invention: In order to overcome the shortcomings of the existing technology, a collaborative saliency detection method and system based on sparse attention and edge guidance is provided.

[0008] Technical Solution: To achieve the above objectives, this invention provides a collaborative saliency detection method based on sparse attention and edge guidance, comprising the following steps:

[0009] S1: Generate multiple realistic virtual samples through the generative adversarial network augmentation module, input them into the deep network along with the original data, and train the generator and discriminator alternately to achieve sample augmentation;

[0010] S2: For the augmented data, a sparse attention mechanism is constructed based on deep convolution features, and an objective function is constructed in combination with the sparse attention mechanism. The region of interest and features of the sensitive object are obtained by solving the objective function.

[0011] S3: Based on the designed collaborative filtering mechanism, the focusing results of the sensitive substance are purified to obtain the characteristics of the purified sensitive substance;

[0012] S4: Based on the purified sensitive object features, the complementary properties between edge features and sensitive object features are utilized to achieve accurate localization of the bounding box, thereby enhancing the low-level and high-level features in the co-saliency detection network.

[0013] S5: Train the co-saliency detection network using a well-designed loss function to obtain the target detection network, and use the target detection network to detect and identify sensitive objects.

[0014] Further, step S1 specifically includes:

[0015] A1: Input based on specific sensitive objects Bootstrap generator Generated realistic virtual samples Approximate the conditional samples, and input the generated realistic virtual samples along with the real samples into the discriminator. To make a judgment, in order to distinguish between the authenticity of the input samples;

[0016] A2: Let the discriminator... The system performs discrimination detection on the input data, identifies whether the input image is genuine or fake, and returns the result to the generator. ;

[0017] A3: Generator The parameters are updated and adjusted based on the returned data: when the generated image is identified as a fake sample by the discriminator, the generator adjusts the generation strategy based on the back feedback of the discriminator to generate more realistic samples until the generated samples are indistinguishable from real samples.

[0018] A4: Repeated Execution Generator With discriminator The training process involves repeating steps A1 to A3 in a loop. In each round of training, the generator generates sample images and inputs them together with real images into the discriminator for discrimination. The discriminator updates its own parameters based on the discrimination results, while the generator further optimizes its generation strategy based on the feedback results. This process continues iterating until the quality of the generated images meets the expected requirements or the set number of iterations is reached.

[0019] Furthermore, the optimization objective function adopted in the training process of step A4 is:

[0020]

[0021] in, Represents a generator. Indicates the discriminator, Discriminator The output is the true probability of the sample.

[0022] Further, step S2 specifically includes:

[0023] B1: Convolutional features for each frame For all convolutional features, use convolution. Convolutional features The response space projected onto the sensitive object;

[0024] B2: Considering the characteristic that sensitive objects appear repeatedly in multiple frames of video, the following average filtering method is used to highlight the features of sensitive objects for the extracted convolutional features of sensitive objects:

[0025]

[0026] in, Dimensions representing features Represents the cluster centers of the sensitive object response space; each of these clusters... The projection of the convolutional features into the corresponding response space is represented as:

[0027]

[0028] B3: with Using cluster centers and sparse response priors, we construct an optimization objective function:

[0029]

[0030] in, Represents the clustering space. Represents the sensitive object response space obtained by sparse convolution; and The parameters to be solved are used to adjust the weights of different constraint terms, where, Used to control convolution weight constraints The penalty strength is adjusted to prevent model overfitting. Used to adjust sparse response constraints The weights are determined to ensure that the model suppresses redundant noise while highlighting salient features; the third term in the objective function The corresponding sparse response prior constraint is used to suppress redundant responses in convolutional features;

[0031] B4: Optimizing the objective function To leverage the properties of convex functions, the alternating direction multiplier method is employed to quickly solve the objective function, thereby obtaining the region of interest and characteristics of the sensitive object.

[0032] Furthermore, in step S3, the region correlation matrix of the region of interest is calculated, and the region correlation matrix is ​​decomposed based on the characteristic that the distribution of common sensitive objects after focusing exhibits sparse characteristics, to obtain the purified sensitive object features.

[0033] Furthermore, step S3 specifically includes:

[0034] C1: For a given set of multiple samples Using the features extracted by the sparse attention mechanism in step S2, regions of interest are constructed and focused on multiple samples; by calculating the correlation between regions of interest, a correlation matrix is ​​obtained. ; where, matrix elements Indicates sample pairs The correlation between them is defined as:

[0035]

[0036] in, This represents the feature vector extracted from sample x under the sparse attention mechanism; the correlation between regions of interest is measured by calculating the cosine similarity between sample features, thus obtaining the overall correlation matrix. ;

[0037] C2: Because shared sensitive objects exhibit a sparse distribution after sparse attention focusing, the region correlation matrix... Decompose the matrix to highlight the salient structure; Decompose into low-rank terms With sparse terms ,Right now Among them, sparse terms Represents the salient region obtained from the detection, low-rank term Corresponding to the negligible region;

[0038] The optimized objective function is constructed as follows:

[0039]

[0040]

[0041] in, Indicates the lower rank term To construct the atomic norm, express Norm;

[0042] C3: Based on the obtained sample correlation matrix Set threshold According to matrix elements The magnitude of the salience determines the significance of the sensitizer; when When determining the sample With sample There are shared sensitive substances; filter them out. The sample pairs were processed, and the remaining purification results were input into a co-significance test network for the detection and identification of sensitive substances.

[0043] Furthermore, in step S4, the purification result output by the collaborative filtering module... These are respectively related to the low-level features of the object detection network. Characteristics of high-level personnel The accuracy of the co-saliency detection network is enhanced through fusion, as shown in the following formula:

[0044]

[0045] in, It is the element-wise multiplication symbol; and These represent the feature refinement masks output by the collaborative filtering module for the low-level and high-level features of the object detection network, respectively. Used to enhance low-level features Edge and detail information related to sensitive objects Used to enhance high-level features Semantic-related discriminative information.

[0046] Furthermore, the loss function in step S5 is:

[0047]

[0048] in, It is a loss function related to the category of sensitive substances, used to identify the types of sensitive substances; It is a regression loss function related to the bounding box of the sensitive object, used to improve the localization accuracy of the detection results.

[0049] Furthermore, in the loss function of step S5:

[0050]

[0051]

[0052] in, and Representing the predicted class and the true class respectively. and These represent the coordinates of the bounding box, and the R function is the Smooth L1 function, used to measure the accuracy of the detection.

[0053] This invention also provides a collaborative saliency detection system based on sparse attention and edge guidance, comprising:

[0054] A generative adversarial network augmentation module is used to generate multiple realistic virtual samples, which are input into the deep network along with the original data, and the generator and discriminator are trained alternately to achieve sample augmentation.

[0055] The feature extraction module constructs a sparse attention mechanism based on deep convolutional features, and combines the sparse attention mechanism to construct an objective function. By solving the objective function, the region of interest and features of the sensitive object are obtained.

[0056] The collaborative filtering module, based on a pre-designed collaborative filtering mechanism, purifies the focusing results of sensitive substances to obtain the purified characteristics of the sensitive substances.

[0057] The network enhancement module, based on the purified sensitive object features, utilizes the complementary nature between edge features and sensitive object features to achieve accurate localization of the bounding box, thereby enhancing the low-level and high-level features in the co-saliency detection network.

[0058] The network training module trains the co-saliency detection network using a pre-designed loss function to obtain the object detection network;

[0059] The detection output module uses a target detection network to detect and identify sensitive objects.

[0060] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0061] 1. In terms of target sample augmentation, by introducing common sensitive objects as condition y, the generator is guided to generate virtual samples to approximate the condition samples, which effectively solves the problems of training difficulties and model collapse. In addition, in the training of the discriminator, real samples and generated samples are also input into the discriminator for adversarial learning. This can guide the model to converge quickly and make the generated samples more realistic.

[0062] 2. In terms of collaborative filtering, the sparse distribution of common sensitive objects is utilized to decompose the correlation matrix of the region of interest, thereby extracting significant and negligible regions and purifying the region of interest.

[0063] 3. In terms of sensitive object detection and identification, the construction of a saliency detection network enables efficient construction of object information and its edge information, which allows for better boundary differentiation. Furthermore, the complementary nature of edge features and sensitive object features obtained through collaborative filtering is utilized to improve localization accuracy. Attached Figure Description

[0064] Figure 1 This is a flowchart of the method of the present invention;

[0065] Figure 2 This is a flowchart of the process for generating adversarial network sample augmentation in this invention;

[0066] Figure 3 This is a schematic diagram of the sparse attention mechanism in this invention;

[0067] Figure 4 This is a framework diagram of the collaborative saliency detection network in this invention. Detailed Implementation

[0068] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0069] Example 1:

[0070] like Figure 1 As shown, this embodiment provides a collaborative saliency detection method based on sparse attention and edge guidance, including the following steps:

[0071] S1: Generate multiple realistic virtual samples through the generative adversarial network augmentation module, input them into the deep network along with the original data, and train the generator and discriminator alternately to achieve sample augmentation;

[0072] In this embodiment, step S1 specifically includes:

[0073] A1: Based on the original generative adversarial network, specific sensitive objects (such as weapons, flags, slogans, etc.) are used as conditional samples. Bootstrap generator Generated realistic virtual samples Approximation condition sample The generated realistic virtual samples are then input into the discriminator along with the real samples. To make a judgment, in order to distinguish between the authenticity of the input samples;

[0074] A2: Let the discriminator... The discriminator performs discrimination detection on the input data in step A1 to distinguish between genuine and fake input images; The goal is to learn the differences between real samples and generated virtual samples, determine the authenticity of a sample using a probability value, and feed the determination result back to the generator. ;

[0075] A3: Generator The parameters are updated and adjusted based on the returned data: when a generated image is identified as a fake sample by the discriminator, the generator adjusts its generation strategy based on the discriminator's feedback to generate more realistic samples; the training objective of the generator is to improve upon the discriminator's performance. To obtain higher discrimination scores, until the generated samples are indistinguishable from real samples;

[0076] A4: Repeated Execution Generator With discriminator The training process involves repeating steps A1 to A3 in a loop. In each round of training, the generator generates sample images and inputs them together with real images into the discriminator for discrimination. The discriminator updates its own parameters based on the discrimination results, while the generator further optimizes its generation strategy based on the feedback results. This process continues iterating until the quality of the generated images meets the expected requirements or the set number of iterations is reached.

[0077] The optimization objective function used in training is:

[0078]

[0079] in, Represents a generator. Indicates the discriminator, Discriminator The output sample's true probability. By continuously and iteratively training the generator and discriminator, the realism of the generated samples is improved, thereby achieving the goal of sample augmentation. The specific process is as follows: Figure 2 As shown.

[0080] S2: For the data after sample augmentation, considering that sensitive objects are sparsely distributed in images and videos and that the deep convolution features correspond one-to-one with the spatial distribution of the original data, a sparse attention mechanism is constructed based on the deep convolution features. Then, combined with the characteristic that sensitive objects reappear in multiple frames of video, the features of sensitive objects are highlighted by average filtering. The optimization objective is established by using the sparse response prior, and the objective function is constructed. The alternating multiplier method is used to solve the objective function to obtain the region of interest and features of sensitive objects.

[0081] In this embodiment, step S2 specifically includes:

[0082] B1: Convolutional features for each frame For all convolutional features, use convolution Convolutional features The response space projected onto the sensitive object, specifically as follows: Figure 3 As shown;

[0083] B2: Considering the characteristic that sensitive objects appear repeatedly in multiple frames of video, the following average filtering method is used to highlight the features of sensitive objects for the extracted convolutional features of sensitive objects:

[0084]

[0085] in, Dimensions representing features Represents the cluster centers of the sensitive object response space; each of these clusters... The projection of the convolutional features into the corresponding response space is represented as:

[0086]

[0087] B3: with Using cluster centers and sparse response priors, we construct an optimization objective function:

[0088]

[0089] in, Represents the clustering space. This represents the sensitive object response features obtained from sparse convolution; and These are the parameters to be solved, used to adjust the weights of different constraint terms. Used to control convolution weight constraints The penalty strength is adjusted to prevent model overfitting. Used to adjust sparse response constraints The weights are determined to ensure that the model suppresses redundant noise while highlighting salient features. The third term in the objective function... The corresponding sparse response prior constraint is used to suppress redundant responses in convolutional features, ensuring that the model produces a large response only in regions related to the sensitive object, thereby achieving sparse focusing on the sensitive object region. By solving this optimization objective, the salient feature regions of the sensitive object can be effectively highlighted under the sparsity constraint.

[0090] B4: Optimizing the objective function To leverage the properties of convex functions, the Alternating Direction Multiplier Method (ADMM) is employed to quickly solve the objective function, thereby obtaining the region of interest and characteristics of the sensitive object.

[0091] S3: To further improve the detection accuracy of sensitive objects and make full use of multi-frame video, image samples and augmented sample information, a collaborative filtering mechanism is designed to purify the focusing results of sensitive objects and obtain the purified features of sensitive objects.

[0092] In step S3, the region correlation matrix of the region of interest is calculated, and the region correlation matrix is ​​decomposed based on the characteristic that the distribution of common sensitive objects after focusing exhibits sparse characteristics, so as to obtain the purified sensitive object features.

[0093] Based on the above, step S3 in this embodiment specifically includes:

[0094] C1: For a given set of multiple samples Using the features extracted by the sparse attention mechanism in step S2, regions of interest are constructed and focused on multiple samples; by calculating the correlation between regions of interest, a correlation matrix is ​​obtained. Among them, matrix elements Indicates sample pairs The correlation between them is defined as:

[0095]

[0096] in, This represents the feature vector extracted from sample x under the sparse attention mechanism. The formula measures the correlation between regions of interest by calculating the cosine similarity between sample features, thus obtaining the overall correlation matrix. .

[0097] C2: Because the sensitive objects exhibit a sparse distribution after being focused by the sparse attention mechanism, the region correlation matrix... Decomposition is performed to highlight salient structures, specifically by: dividing the matrix Decompose into low-rank terms With sparse terms ,Right now Among them, sparse terms Represents the salient region obtained from the detection, low-rank term Corresponding to the negligible region;

[0098] The optimized objective function is constructed as follows:

[0099]

[0100]

[0101] in, Indicates the lower rank term To construct the atomic norm, express Norm;

[0102] By optimizing the above equation, the sparse term can be obtained. , among which, element Indicates sample With sample The degree of relevance; when The higher the value, the lower the likelihood that the sample contains a common sensitive substance;

[0103] C3: Based on the obtained sample correlation matrix Set threshold According to matrix elements The magnitude of the salience determines the significance of the sensitizer; when When determining the sample With sample There are shared sensitive substances; filter them out. The sample pairs were processed, and the remaining purification results were input into a co-significance test network for the detection and identification of sensitive substances.

[0104] S4: Based on the purified sensitive object features, the complementary properties between edge features and sensitive object features are utilized to achieve accurate localization of the bounding box, thereby enhancing the low-level and high-level features in the co-saliency detection network.

[0105] like Figure 4 As shown, the collaborative saliency detection network includes a collaborative filtering module;

[0106] In step S4, the purification results output by the collaborative filtering module These are respectively related to the low-level features of the object detection network. Characteristics of high-level personnel The accuracy of the co-saliency detection network is enhanced through fusion, as shown in the following formula:

[0107]

[0108] in, It is the symbol for element-wise multiplication. and These represent the feature refinement masks output by the collaborative filtering module for the low-level and high-level features of the object detection network, respectively. Used to enhance low-level features Edge and detail information related to sensitive objects Used to enhance high-level features Semantic-related discriminative information.

[0109] The above formula combines the edge information contained in the low-level features with the semantic information of the sensitive objects in the high-level features to complement each other, giving full play to the complementary advantages between the two, thereby effectively improving the detection and recognition performance of sensitive objects.

[0110] S5: Train the co-saliency detection network using a well-designed loss function to obtain the target detection network, and use the target detection network to detect and identify sensitive objects.

[0111] The loss function in this embodiment is:

[0112]

[0113] in, It is a loss function related to the category of sensitive substances, used to identify the specific types of sensitive substances; It is a regression loss function related to the bounding box of the sensitive object, used to improve the localization accuracy of the detection results;

[0114] The two loss functions mentioned above are expressed as follows:

[0115]

[0116]

[0117] in, and Representing the predicted class and the true class respectively; and These represent the coordinates of the predicted bounding box and the ground truth bounding box, respectively, in the format (x, y, w, h), where x represents the x-coordinate of the top-left corner of the bounding box, y represents the y-coordinate of the top-left corner of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box; the R function is the Smooth L1 function, used to measure the accuracy of the detection.

[0118] By jointly optimizing the two loss functions, the accuracy of sensitive object localization can be improved while maintaining classification accuracy, thereby achieving accurate determination of the sensitive object category and boundary location.

[0119] like Figure 4 As shown, the category of the sensitive object and its corresponding mask are finally output through the fully connected layer, so as to achieve accurate detection and identification of the sensitive object.

[0120] Based on the above, the method of this invention can be summarized as follows: First, a diverse set of virtual samples is generated using a generative adversarial network (GAN) augmentation module. These virtual samples are then used to train the generator and discriminator together with the original data to achieve sample augmentation. Addressing the sparse distribution of sensitive objects in images, a sparse attention mechanism is constructed and combined with multi-frame averaging filtering to highlight target features. An optimization objective is established using sparse response priors, and the objective function is quickly solved using the alternating multiplier method to obtain the regions of interest (ROIs) and features of the sensitive objects. Subsequently, the correlation matrix of the ROIs is calculated and decomposed based on a collaborative filtering mechanism, thereby refining the features of the sensitive objects. In the saliency detection network, an edge guidance mechanism is introduced to fuse low-level and high-level features, achieving accurate segmentation and recognition of the sensitive objects. This invention effectively alleviates the problems of model collapse and convergence difficulties by combining conditional sample generation with adversarial training. Furthermore, by utilizing the sparse distribution of common sensitive objects and achieving feature refinement through a collaborative filtering mechanism, it significantly improves the accuracy of target localization and boundary discrimination capabilities.

[0121] Example 2:

[0122] Based on the method provided in Embodiment 1, this embodiment provides a collaborative saliency detection system based on sparse attention and edge guidance, including:

[0123] A generative adversarial network augmentation module is used to generate multiple realistic virtual samples, which are input into the deep network along with the original data, and the generator and discriminator are trained alternately to achieve sample augmentation.

[0124] The feature extraction module constructs a sparse attention mechanism based on deep convolutional features, and combines the sparse attention mechanism to construct an objective function. By solving the objective function, the region of interest and features of the sensitive object are obtained.

[0125] The collaborative filtering module, based on a pre-designed collaborative filtering mechanism, purifies the focusing results of sensitive substances to obtain the purified characteristics of the sensitive substances.

[0126] The network enhancement module, based on the purified sensitive object features, utilizes the complementary nature between edge features and sensitive object features to achieve accurate localization of the bounding box, thereby enhancing the low-level and high-level features in the co-saliency detection network.

[0127] The network training module trains the co-saliency detection network using a pre-designed loss function to obtain the object detection network;

[0128] The detection output module uses a target detection network to detect and identify sensitive objects.

Claims

1. A collaborative saliency detection method based on sparse attention and edge guidance, characterized in that, Includes the following steps: S1: Generate multiple realistic virtual samples through the generative adversarial network augmentation module, input them into the deep network along with the original data, and train the generator and discriminator alternately to achieve sample augmentation; S2: For the augmented data, a sparse attention mechanism is constructed based on deep convolution features, and an objective function is constructed in combination with the sparse attention mechanism. The region of interest and features of the sensitive object are obtained by solving the objective function. S3: Based on the designed collaborative filtering mechanism, the focusing results of the sensitive substance are purified to obtain the characteristics of the purified sensitive substance; S4: Based on the purified sensitive object features, the complementary properties between edge features and sensitive object features are utilized to achieve accurate localization of the bounding box, thereby enhancing the low-level and high-level features in the co-saliency detection network. S5: Train the co-saliency detection network using a well-designed loss function to obtain the target detection network, and use the target detection network to detect and identify sensitive objects.

2. The collaborative saliency detection method based on sparse attention and edge guidance according to claim 1, characterized in that, Step S1 specifically includes: A1: Input based on specific sensitive objects Bootstrap generator Generated realistic virtual samples Approximate the conditional samples, and input the generated realistic virtual samples along with the real samples into the discriminator. To make a judgment, in order to distinguish between the authenticity of the input samples; A2: Let the discriminator... The system performs discrimination detection on the input data, identifies whether the input image is genuine or fake, and returns the result to the generator. ; A3: Generator The parameters are updated and adjusted based on the returned data: when the generated image is identified as a fake sample by the discriminator, the generator adjusts the generation strategy based on the back feedback of the discriminator to generate more realistic samples until the generated samples are indistinguishable from real samples. A4: Repeated Execution Generator With discriminator The training process involves repeating steps A1 to A3 in a loop. In each round of training, the generator generates sample images and inputs them together with real images into the discriminator for discrimination. The discriminator updates its own parameters based on the discrimination results, while the generator further optimizes its generation strategy based on the feedback results. This process continues iterating until the quality of the generated images meets the expected requirements or the set number of iterations is reached.

3. The collaborative saliency detection method based on sparse attention and edge guidance according to claim 2, characterized in that, The objective function used in the training process in step A4 is: ； in, Represents a generator. Indicates the discriminator, Discriminator The output is the true probability of the sample.

4. The collaborative saliency detection method based on sparse attention and edge guidance according to claim 1, characterized in that, Step S2 specifically includes: B1: Convolutional features for each frame For all convolutional features, use convolution. Convolutional features The response space projected onto the sensitive object; B2: Considering the characteristic that sensitive objects appear repeatedly in multiple frames of video, the following average filtering method is used to highlight the features of sensitive objects for the extracted convolutional features of sensitive objects: ； in, Dimensions representing features Represents the cluster centers of the sensitive object response space; each of these clusters... The projection of the convolutional features into the corresponding response space is represented as: ； B3: with Using cluster centers and sparse response priors, we construct an optimization objective function: ； in, Represents the clustering space. Represents the sensitive object response space obtained by sparse convolution; and The parameters to be solved are used to adjust the weights of different constraint terms, where, Used to control convolution weight constraints The penalty strength is adjusted to prevent model overfitting. Used to adjust sparse response constraints The weights are determined to ensure that the model suppresses redundant noise while highlighting salient features; the third term in the objective function The corresponding sparse response prior constraint is used to suppress redundant responses in convolutional features; B4: Optimizing the objective function To leverage the properties of convex functions, the alternating direction multiplier method is employed to quickly solve the objective function, thereby obtaining the region of interest and characteristics of the sensitive object.

5. The collaborative saliency detection method based on sparse attention and edge guidance according to claim 1, characterized in that, In step S3, the region correlation matrix of the region of interest is calculated, and the region correlation matrix is ​​decomposed according to the characteristic that the distribution of common sensitive objects after focusing exhibits sparse characteristics, so as to obtain the purified sensitive object features.

6. The collaborative saliency detection method based on sparse attention and edge guidance according to claim 5, characterized in that, Step S3 specifically includes: C1: For a given set of multiple samples Using the features extracted by the sparse attention mechanism in step S2, regions of interest are constructed and focused on multiple samples; by calculating the correlation between regions of interest, a correlation matrix is ​​obtained. ; where, matrix elements Indicates sample pairs The correlation between them is defined as: ； in, This represents the feature vector extracted from sample x under the sparse attention mechanism; the correlation between regions of interest is measured by calculating the cosine similarity between sample features, thus obtaining the overall correlation matrix. ; C2: Because shared sensitive objects exhibit a sparse distribution after sparse attention focusing, the region correlation matrix... Decompose the matrix to highlight the salient structure; Decompose into low-rank terms With sparse terms ,Right now Among them, sparse terms Represents the salient region obtained from the detection, low-rank term Corresponding to the negligible region; The optimized objective function is constructed as follows: ； ； in, Indicates the lower rank term To construct the atomic norm, express Norm; C3: Based on the obtained sample correlation matrix Set threshold According to matrix elements The magnitude of the salience determines the significance of the sensitizer; when When determining the sample With sample There are shared sensitive substances; filter them out. The sample pairs were processed, and the remaining purification results were input into a co-significance test network for the detection and identification of sensitive substances.

7. The collaborative saliency detection method based on sparse attention and edge guidance according to claim 1, characterized in that, In step S4, the purification result output by the collaborative filtering module These are respectively related to the low-level features of the object detection network. Characteristics of high-level personnel The accuracy of the co-saliency detection network is enhanced through fusion, as shown in the following formula: ； in, It is the element-wise multiplication symbol; and These represent the feature refinement masks output by the collaborative filtering module for the low-level and high-level features of the object detection network, respectively. Used to enhance low-level features Edge and detail information related to sensitive objects Used to enhance high-level features Semantic-related discriminative information.

8. The collaborative saliency detection method based on sparse attention and edge guidance according to claim 1, characterized in that, The loss function in step S5 is: ； in, It is a loss function related to the category of sensitive substances, used to identify the types of sensitive substances; It is a regression loss function related to the bounding box of the sensitive object, used to improve the localization accuracy of the detection results.

9. The collaborative saliency detection method based on sparse attention and edge guidance according to claim 8, characterized in that, In the loss function of step S5: ； ； in, and Representing the predicted class and the true class respectively. and These represent the coordinates of the bounding box, and the R function is the Smooth L1 function, used to measure the accuracy of the detection.

10. A collaborative saliency detection system based on sparse attention and edge guidance, characterized in that, For implementing the method of claim 1, the system comprises: A generative adversarial network augmentation module is used to generate multiple realistic virtual samples, which are input into the deep network along with the original data, and the generator and discriminator are trained alternately to achieve sample augmentation. The feature extraction module constructs a sparse attention mechanism based on deep convolutional features, and combines the sparse attention mechanism to construct an objective function. By solving the objective function, the region of interest and features of the sensitive object are obtained. The collaborative filtering module, based on a pre-designed collaborative filtering mechanism, purifies the focusing results of sensitive substances to obtain the purified characteristics of the sensitive substances. The network enhancement module, based on the purified sensitive object features, utilizes the complementary nature between edge features and sensitive object features to achieve accurate localization of the bounding box, thereby enhancing the low-level and high-level features in the co-saliency detection network. The network training module trains the co-saliency detection network using a pre-designed loss function to obtain the object detection network; The detection output module uses a target detection network to detect and identify sensitive objects.

Images