A collaborative salient object detection method and system based on a double-layer recurrent network

Abstract

The application relates to the field of cooperative saliency object detection, and provides a cooperative saliency object detection method and system based on a double-layer recurrent network. The method comprises the following steps: acquiring an image group, constructing a double-layer recurrent network, wherein the double-layer recurrent network comprises a feature extraction module, a plurality of IERR modules in series connection and a plurality of DRD modules; extracting intra-image features of the image group by using the feature extraction module; based on the intra-image features, gradually extracting inter-image features by using the plurality of IERR modules in series connection to obtain enhanced cooperative semantic features; and based on the intra-image features and the enhanced cooperative semantic features, fusing the inter-image features and the intra-image features by using the plurality of DRD modules to obtain cooperative saliency features which need to be decoded.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of co-salient object detection, and particularly relates to a co-salient object detection method and system based on a two-tier recursion network. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] For the Co-Salient Object Detection (CoSOD) task, from the perspective of learning inter-image relationships, they can be roughly divided into two categories: (i) direct learning structure, in order to obtain group semantic information, all intra-image features are directly fed into a carefully designed fusion module; (ii) recursive learning structure, the typical recursive model sequentially inputs multiple intra-image features into a recursive unit, thereby generating group semantics. Direct learning structure usually requires many carefully designed modules to solve these problems, which in turn can increase model complexity and reduce inference speed. Existing recursive learning structures, firstly, from the perspective of inter-image modeling, only utilize global group semantics, ignoring the local subgroup relationships generated in recursive learning. For example, the forgetting mechanism used in Gated Recurrent Neural Network (GRU) is not friendly to the CoSOD task, which destroys the learning of inter-image relationships. Secondly, from the perspective of inter-image distribution, the differences between different images are ignored, and the use of simple cascading distribution methods is not enough, which leads to unsatisfactory prediction results, making it impossible to completely suppress non-cooperative interference. SUMMARY

[0004] In order to solve the technical problems existing in the background art, the present application provides a co-salient object detection method and system based on a two-tier recursion network, which proposes a two-tier recursion network (TRNet). The two-tier recursion structure is reflected in the inter-image extraction stage and the inter-image distribution stage. In the inter-image extraction stage, an inter-image exploration module with recursive enhancement is designed to learn the local and global inter-image correspondence, enhancing the effectiveness and discriminability of information in the propagation process. In the inter-image distribution stage, a dynamic recursive distribution module is designed, which adaptively distributes the co-attribute for each image through an improved semi-dynamic convolution, and obtains the co-salient feature.

[0005] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0006] The first aspect of the present application provides a co-salient object detection method based on a two-tier recursion network.

[0007] A collaborative salient object detection method based on a double-layer recurrent network, comprising:

[0008] An image group is acquired, and a double-layer recurrent network is constructed, the double-layer recurrent network comprising a feature extraction module, a plurality of recurrently connected IERR modules based on a convLSTM, and a plurality of DRD modules;

[0009] The feature extraction module is used to extract intra-image features of the image group;

[0010] Based on the intra-image features, the plurality of recurrently connected IERR modules are used to extract inter-image features step by step, to obtain enhanced collaborative semantic features;

[0011] Based on the intra-image features and the enhanced collaborative semantic features, the plurality of DRD modules are used to obtain inter-image features, and the inter-image features are fused with the intra-image features to obtain collaborative salient features to be decoded.

[0012] Further, the feature extraction module uses an improved VGG16 network with one max-pooling layer and three full connection layers removed; the improved VGG16 network is used to extract multi-level features of each image in the image group, and high-level convolutional features containing rich semantic information are selected as the intra-image features of each image.

[0013] Further, the process of obtaining enhanced collaborative semantic features comprises: based on the intra-image features of the first image, the first IERR module is used to extract inter-image features of the first image, to obtain first local similarity constraint features and first enhanced collaborative semantic features; based on the intra-image features of the nth image, the (n-1)th local similarity constraint features and the (n-1)th enhanced collaborative semantic features, the nth IERR module is used to extract inter-image features of the nth image, to obtain the nth local similarity constraint features and the nth enhanced collaborative semantic features; a convolutional layer is used to perform spatial conversion of the nth local similarity constraint features and the (n-1)th local similarity constraint features and enhanced collaborative semantic features, to obtain a similarity matrix; the similarity matrix and the nth local similarity constraint features are combined by using a residual connection to obtain enhanced collaborative semantic features.

[0014] Further, the IERR module uses a convLSTM network with a forgetting gate removed and an enhanced gate added.

[0015] Further, the process of obtaining the inter-graph feature comprises: based on the intra-graph feature of each image and the n enhanced collaborative semantic features, using a DRD module to dynamically convolve each enhanced collaborative semantic feature with the intra-graph feature to obtain n convolution features; based on the n convolution features, using n convLSTM units respectively, taking the feature output by the nth convLSTM unit as the inter-graph feature.

[0016] Further, the process of dynamically convolving each enhanced collaborative semantic feature with the intra-graph feature uses an improved semi-dynamic convolution unit; in the process of calculating attention, global pooling and max pooling are used in combination, complement each other and improve performance, and the formula is as follows:

[0017] att n =φ(θ(δ(θ(avg(r n ))))+(θ(δ(θ(max(r n )))

[0018] Wherein, φ, θ and δ represent softmax layer, full connection layer and ReLU activation layer respectively; r n represents an enhanced collaborative semantic feature;

[0019] The attention weight att n is summed by elements to obtain a processed dynamic convolution kernel, and then the intra-graph feature is convolved.

[0020] Further, the double-layer recurrent network uses a binary cross-entropy loss function for training in the training stage.

[0021] The second aspect of the application provides a collaborative saliency target detection system based on a double-layer recurrent network.

[0022] A collaborative saliency target detection system based on a double-layer recurrent network comprises:

[0023] A data acquisition and model establishment module configured to acquire an image group and construct a double-layer recurrent network, wherein the double-layer recurrent network comprises a feature extraction module, a plurality of recursive serial IERR modules and a plurality of DRD modules;

[0024] The feature extraction module is configured to extract intra-graph features of the image group;

[0025] The inter-graph extraction module is configured to extract inter-graph features based on the intra-graph features using a plurality of recursive serial IERR modules to obtain enhanced collaborative semantic features;

[0026] An inter-graph distribution module is configured to: based on the intra-graph features and the enhanced collaborative semantic features, adopt a plurality of DRD modules to obtain inter-graph features, and fuse the inter-graph features with the intra-graph features to obtain collaborative saliency features to be decoded.

[0027] A third aspect of the present application provides a computer readable storage medium.

[0028] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the collaborative saliency object detection method based on a double-layer recurrent network according to the first aspect.

[0029] A fourth aspect of the present application provides a computer device.

[0030] A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the collaborative saliency object detection method based on a double-layer recurrent network according to the first aspect when executing the program.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] The present application proposes a recursive enhancement (IERR) module based on convLSTM to model more comprehensive inter-graph relationships, including global group semantics and local subgroup semantics. On the one hand, the forget gate in convLSTM is deleted, making it more suitable for the CoSOD task. On the other hand, an enhancement gate is added to the convLSTM architecture to enhance the relevance and discriminability of inter-graph relationships. The improvements of these two points not only guarantee the effectiveness of information in the step-by-step propagation, but also provide additional local subgroup semantics for the distribution stage.

[0033] The present application designs a dynamic recursive distribution (DRD) module to fully utilize local subgroup semantics and global group semantics in a dynamic manner. Using multiple group / subgroup semantic features to generate convolution kernels and performing semi-dynamic convolution on intra-graph features to obtain collaborative saliency features can to some extent alleviate the problem of sensitivity to input order of the recurrent structure. BRIEF DESCRIPTION OF DRAWINGS

[0034] The drawings constituting a part of this application are used to provide a further understanding of the present application, and the illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0035] Figure 1 is a double-layer recurrent learning network structure diagram shown by the present application;

[0036] Figure 2 is a visualization result diagram of the method proposed by the present application. DETAILED DESCRIPTION

[0037] The application will be further described with reference to the drawings and examples.

[0038] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0039] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, as the scope of the exemplary embodiments of this application will be limited in the claims. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. Unless otherwise required by context, singular terms shall include pluralities and vice versa. Plural components described as a single component shall include multiple instances of that component unless otherwise required by context.

[0040] It should be noted that the flow diagrams and block diagrams in the drawings are representative of the architectural, functional, and operational aspects of possible implementations of methods and systems according to various embodiments of the present disclosure. It should be noted that each block in the flow diagrams and / or block diagrams can represent a module, a piece of code, and / or a portion of code that comprises one or more executable instructions for implementing the specified logical functions (s). It should also be noted that in some alternative implementations, the functions noted in the box can occur out of the order noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or can sometimes be executed in reverse order, depending upon the functionality involved. It will also be noted that each block of the flow diagrams and / or block diagrams and combinations of blocks in the flow diagrams and / or block diagrams can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and

[0041] Embodiment One

[0042] As Figure 1As shown, the embodiment provides a collaborative salient object detection method based on a two-tier recursion network. The method can be applied to a server, and can also be applied to a terminal, and can also be applied to a system including a terminal and a server and is realized through interaction of the terminal and the server. The server can be a stand-alone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, security services CDN, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application. In the embodiment, the method includes the following steps:

[0043] An image group is acquired, and a two-tier recursion network is constructed, the two-tier recursion network including a feature extraction module, a plurality of recursively serial IERR modules, and a plurality of DRD modules;

[0044] The feature extraction module is used to extract intra-image features of the image group;

[0045] Based on the intra-image features, the plurality of recursively serial IERR modules are used to gradually extract inter-image features to obtain enhanced collaborative semantic features;

[0046] Based on the intra-image features and the enhanced collaborative semantic features, the plurality of DRD modules are used to obtain inter-image features, and the inter-image features and the intra-image features are fused to obtain collaborative salient features to be decoded.

[0047] The embodiment will be described in detail below with reference to the accompanying drawings:

[0048] As shown in the Figure 1 The two-tier recursion network (TRNet) proposed in the embodiment follows an end-to-end encoder-decoder architecture. Given a set of related images The goal of the CoSOD task is to discover collaborative salient objects and predict the corresponding collaborative saliency maps where N is the number of images in the group, G n is the collaborative saliency map of image I n First, a VGG16 network with the last max-pooling layer and three full connection layers discarded is used as the backbone to extract multi-level features of each image in the image group. Considering that the semantics of the collaborative salient objects in the group are consistent, high-level convolutional features containing rich semantic information are used as intra-image features of each image where C, H and W denote the channel, height and width of the feature map, respectively. Then, two layers of recurrent structures are designed to extract inter-image relationships and distribute them effectively to each image, corresponding to an inter-image extraction stage composed of multiple IERR modules and an inter-image distribution stage with DRD modules. Finally, the learned collaborative saliency features are embedded into the corresponding decoder to predict the collaborative saliency map. For the sake of clarity, two points need to be explained: (1) the multiple IERRs in the inter-image extraction stage are recurrent serial structures, each with an output; (2) each DRD in the inter-image distribution stage is a recurrent structure internally, but the structures of different DRDs are independent.

[0049] Inter-image extraction stage

[0050] Considering the unpredictable changes in the appearance and location of collaborative targets among images, directly concatenating single intra-image features into group semantic features inevitably contains a large amount of noise information from non-salient or non-collaborative regions. Moreover, processing the features of all images in a group at once will undoubtedly increase the difficulty for the network to learn inter-image features. In contrast, a recurrent architecture can learn more comprehensive inter-image relationships step by step and can obtain global group semantics and multiple local subgroup semantics, thereby enriching the expression of inter-image relationships, suppressing irrelevant interference, and reducing the learning difficulty of the network. As a typical recurrent structure, the Long Short Term Mermory network (LSTM) can eliminate irrelevant information in a sequence by updating the storage unit while retaining important information that repeats. This is consistent with the CoSOD task of finding salient and repeatedly appearing targets. Moreover, as an evolutionary version of LSTM, the convolutional long short-term memory network (convLSTM) uses convolution operations instead of dot products in traditional LSTM to represent spatial features, which has also been proven to be more suitable for image feature extraction. Inspired by these, the embodiment designs a recurrent enhancement module based on convLSTM to capture multiple local subgroup semantics and global group semantics for inter-image exploration. The main design idea can answer the following two questions:

[0051] (1) Does the CoSOD task need a forgetting mechanism? LSTM and its variants (e.g., convLSTM) are mainly used to process sequence data, and time continuity is a very important attribute. Data closer to the current time should play a greater role, so a forgetting gate is applied in convLSTM. However, the inter-image relationship of the CoSOD task needs to read the entire image group data completely, and if it is forgotten, it will inevitably weaken the needed collaborative features. Therefore, only the input gate and the output gate are retained to model the inter-image relationship under the convLSTM structure.

[0052] (2) How to sufficiently enhance the discriminability of the synergistic attributes? In the recursive transmission process, the information flow not only transmits the representation of the useful synergistic attributes, but also contains some redundant information, which is obviously not needed. Therefore, in order to make the network more focused on the valuable synergistic features, an enhancement gate is added to enhance the inter-graph correlation between the current step and the previous step in the recursive structure.

[0053] Intra-graph features F of each image n The enhanced group semantic features r will be obtained in the IERR module in turn n . Specifically, for the nth step, the IERR module needs three inputs: the intra-graph features F of the current image n , the local similarity constraint features l of the (n-1)th step n-1 , and the enhanced synergistic semantic features r of the (n-1)th step n-1 . The IERR module is a variation of convLSTM, except that the forget gate is removed and an enhancement gate is added. Similar to convLSTM, the input gate i n and the output gate o n are defined as:

[0054] i n = σ(conv3(F n )+conv3(l n-1 )) (1)

[0055] o n = σ(conv3(F n )+conv3(l n-1 )) (2)

[0056] where σ is the sigmoid activation function, conv3(·) represents a 2D convolution layer with a convolution kernel size of 3x3, l n-1 is the local similarity constraint features of the (n-1)th step. The current state is determined by the output of the previous time and the acceptance degree of the current input, which is defined as:

[0057]

[0058]

[0059] where c n-1 represents the previous state, is to retain the valuable information in the current input, tanh(·) is the tanh activation function, and ⊙ represents the Hadamard product.

[0060] While removing the forget gate can ensure no information loss from the data source perspective, the synergistic properties that need to be emphasized and the irrelevant interference that need to be suppressed have not been fully modeled. Therefore, an enhanced gate is added to improve the discriminability of inter-graph relationship extraction through relevant constraints. As can be seen from the above formula, the introduction of the local similarity constraint of the previous step can strengthen the useful synergistic information retention in the recursive process. The current local similarity constraint l n The similarity relationship between the current output o n and the state c n is described, which is defined as:

[0061] l n = o n ⊙ tanh(c n ) (5)

[0062] Then, the current local similarity constraint l n and the enhanced synergistic semantic features in the previous step are converted into new feature subspaces through 1 × 1 convolutional layers, respectively. After that, matrix multiplication operations are performed to calculate the similarity matrix S

[0063]

[0064] where conv1(·) is a 2D convolutional layer with a 1 × 1 convolution kernel, the features are reshaped into (·) T denotes matrix transposition, and softmax(·) is a softmax layer for feature normalization. The similarity matrix S n further associates the local semantic similarity of the current step with the final synergistic semantic features in the previous step, which can be combined with l n to generate a more comprehensive semantic relationship representation. Therefore, they are combined in the form of a residual connection, thereby obtaining the enhanced synergistic semantic features r n related to the features l n :

[0065]

[0066] The enhanced synergistic semantic features r n associate and strengthen the inter-graph relationship between adjacent steps in the recursive propagation, thereby ensuring a more comprehensive and distinctive synergistic expression. Note that in the first step, l0 and r0 are initialized by F1 and l1, respectively.

[0067] Inter-graph distribution stage

[0068] Multiple IERR modules in the recursive structure can learn multiple enhanced synergistic semantic features r n , where are multiple local subgroup semantic features, r N is a global group semantic feature. Therefore, how to more fully and effectively distribute these local and global semantic features to each image is the next most important problem. Most previous methods use convolution fusion to distribute, only using global group semantic features to fuse with each intra-graph feature, which is more rough and has no explicit physical meaning. At the same time, only using global group semantic features also has the problem of insufficient inter-graph relationship utilization, especially for recursive structures. In order to solve the above problems, the following scheme is disclosed in the embodiment:

[0069] First, the selection of inter-graph semantic features for distribution is crucial. As mentioned earlier, in the inter-graph extraction stage, not only global group semantic features can be obtained, but also local subgroup semantic features can be obtained step by step. Obviously, using global group semantic features alone will cause the waste of local information and cannot fully play the complementary nature of local and global. Therefore, the joint use of group and subgroup semantics can provide comprehensive inter-graph semantic relationships for the distribution stage.

[0070] Second, adaptive, targeted and efficient distribution is also crucial. The most intuitive way of inter-graph relationship distribution is to use it as a template to match each intra-graph feature and determine the collaborative saliency target. However, it should be noted that the inter-graph relationship required to determine the collaborative saliency target is different for each image, and should be dynamically selected as needed. Dynamic convolution allows the network to distribute inter-graph semantic features to each image feature in an adaptive manner.

[0071] The embodiment designs a dynamic recursive distribution module DRD, which enables the network to recursively utilize multiple local subgroup semantic information based on global group semantics. It completes the distribution through an improved semi-dynamic convolution unit. In the inter-graph relationship learning process, local subgroup semantics and global group semantics are obtained in a recursive manner. The operation in the DRD module is similar, which distributes inter-graph semantic information to each intra-graph feature in a recursive and dynamic manner. The input of the DRD module is the current intra-graph feature F n and N enhanced collaborative semantic features Each enhanced collaborative semantic feature r i is dynamically convolved with the intra-graph feature F n , and the obtained feature d n,i is input into the convLSTM unit to obtain the inter-graph feature dl n,i . The above flow definition is as follows:

[0072] d n,i = Dy(r i , F n ) (8)

[0073] dl n,i = Π(d n,i ,dl n,i-1 ) (9)

[0074] where Π(·,·) denotes a convLSTM unit and Dy(·) is an improved semi-dynamic convolution unit. The purpose of using the convLSTM unit here is to alleviate the sensitivity of the image to the input sequence. The inter-graph feature is finally defined as the output of the last convLSTM unit, denoted as E n = dl n,N . After that, it is combined with the intra-graph feature to obtain the final collaborative saliency feature that needs to be decoded:

[0075] Co n = Conv1([F n ,E n ]) (10)

[0076] where [·,·] denotes concatenation along the channel dimension.

[0077] The existing design of dynamic convolution is based on the attention dynamic aggregation of multiple parallel convolution kernels, which complicates the model without increasing the network depth or width. For attention calculation, it is realized by two fully connected layers and average pooling in the traditional semi-dynamic convolution. The operation of average pooling is actually a balancing and equalizing idea, which averages the information in the receptive field, which may weaken the important key content in the receptive field. In contrast, the max-pooling operation can preserve as much as possible the high response part of the receptive field, thereby achieving effective preservation of saliency information, but it may also preserve some false high response information. Considering these, the present embodiment uses a combination of global pooling and max-pooling, which complement each other and improve performance, thereby forming an improved semi-dynamic convolution unit. The formula of the attention module is as follows:

[0078] att n = φ(θ(δ(θ(avg(r n ))))+(θ(δ(θ(max(r n ))))) (11)

[0079] where φ, θ and δ denote softmax layer, fully connected layer and ReLU activation layer, respectively. The max-pooling branch can further emphasize important features in the receptive field based on the average pool branch, which is more suitable for the CoSOD task. After that, the K convolution kernels are summed element by element according to the attention weight att n , and the processed dynamic convolution kernel is obtained, and then the intra-graph feature F n is convolved. In order to predict the final collaborative saliency map G n , it is necessary to convolve Co nUp-sampling to the same resolution as the input image, for which a multi-layer deconvolution operation is adopted.

[0080] To obtain accurate co-saliency maps, the proposed network is trained by using binary cross-entropy loss, which is defined as:

[0081]

[0082] Where N is the number of image groups in the group, T n represents the true label, G n is the predicted co-saliency map. The entire training process is carried out in an end-to-end manner.

[0083] Figure 2 Some challenging examples in four different image groups are shown, including bananas, camels, coffee cups and lemons. The visualization results show that the present invention not only locates the co-saliency target more accurately and completely, but also can cope with various challenges such as strong semantic interference, complex background and multiple targets.

[0084] Embodiment two

[0085] The embodiment provides a co-saliency target detection system based on a double-layer recurrent network.

[0086] A co-saliency target detection system based on a double-layer recurrent network comprises:

[0087] A data acquisition and model establishment module configured to acquire an image group, and construct a double-layer recurrent network, wherein the double-layer recurrent network comprises a feature extraction module, a plurality of recurrently connected IERR modules and a plurality of DRD modules;

[0088] The feature extraction module is configured to extract intra-image features of the image group.

[0089] An inter-image extraction module configured to extract inter-image features step by step based on the intra-image features by using a plurality of recurrently connected IERR modules to obtain enhanced co-saliency semantic features.

[0090] An inter-image distribution module configured to obtain inter-image features based on the intra-image features and the enhanced co-saliency semantic features by using a plurality of DRD modules, fuse the inter-image features with the intra-image features, and obtain co-saliency features to be decoded.

[0091] It should be noted that the above data acquisition and model establishment module, feature extraction module, inter-image extraction module and inter-image distribution module have the same examples and application scenarios as the steps in embodiment one, but are not limited to the content disclosed in the above embodiment one. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a group of computer executable instructions.

[0092] Embodiment Three

[0093] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement steps in the collaborative saliency object detection method based on a double-layer recurrent network according to the above embodiment one.

[0094] Embodiment Four

[0095] The embodiment provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements steps in the collaborative saliency object detection method based on a double-layer recurrent network according to the above embodiment one when executing the program.

[0096] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage, etc.) containing computer-usable program code.

[0097] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device implemented in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s).

[0098] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction apparatus, which implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s).

[0099] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operational steps are performed on the computer or other programmable data processing device to generate a computer-implemented process, thus the instructions executed on the computer or other programmable data processing device provide the function of implementing the processes specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one block or multiple blocks.

[0100] Those of ordinary skill in the art can understand that all or part of the flow of the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the flow of the above-mentioned embodiment of each method. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), and the like.

[0101] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for co-salient object detection based on a two-layer recurrent network, characterized in that, The method comprises the following steps: acquiring an image group, constructing a double-layer recurrent network, wherein the double-layer recurrent network comprises a feature extraction module, a plurality of IERR modules in series and a plurality of DRD modules; the IERR module adopts a convLSTM network with a forgetting gate removed and an enhancement gate added; the enhancement gate improves the discrimination ability of inter-graph relationship extraction through correlation constraints, and introduces local similarity constraints of the previous step to strengthen the retention of useful collaborative information in the recursive process; based on the intra-graph features, a plurality of IERR modules in series are used to extract inter-graph features step by step to obtain enhanced collaborative semantic features; based on the intra-graph features and the enhanced collaborative semantic features, a plurality of DRD modules are used to obtain inter-graph features, and the inter-graph features are fused with the intra-graph features to obtain collaborative saliency features to be decoded; The process of obtaining the inter-graph features comprises: based on the intra-graph features of each image and n adopting a DRD module, dynamically convolving each enhanced collaborative semantic feature with the intra-graph features to obtain n convolution features. Based on n convolutional features, respectively using n convLSTM units, the features output by the first n convLSTM unit are taken as the inter-graph features.

2. The method according to claim 1, wherein, the feature extraction module adopts an improved VGG16 network with the last max-pooling layer and three full connection layers removed; the improved VGG16 network is used to extract multi-level features of each image in the image group, and high-level convolutional features containing rich semantic information are selected as the intra-graph features of each image. 3.The method of claim 1, wherein, The process of obtaining the enhanced collaborative semantic feature comprises: based on the intra-graph feature of the first image, using the first IERR module to extract the inter-graph feature of the first image, obtaining the first local similarity constraint feature and the first enhanced collaborative semantic feature; based on the intra-graph feature of the second image, the first local similarity constraint feature and the first enhanced collaborative semantic feature, using the second IERR module to extract the inter-graph feature of the second image, obtaining the second local similarity constraint feature and the second enhanced collaborative semantic feature; using the convolution layer to perform spatial conversion on the first local similarity constraint feature and the second local similarity constraint feature, obtaining the similarity matrix; using the residual connection to combine the similarity matrix and the second local similarity constraint feature, obtaining the enhanced collaborative semantic feature. n n 1 n–1 n n n n n n–1 n ​​​​​​​​​​ 4. The method of claim 1, wherein the method further comprises: the process of dynamically convolving each enhanced collaborative semantic feature with the intra-graph features adopts an improved semi-dynamic convolution unit; in the attention calculation process, global pooling and max pooling are used in combination, and the formula is as follows: wherein, , and denote a softmax layer, a fully connected layer and a ReLU activation layer, respectively; denotes enhancing the collaborative semantic features; Attention weights Element-wise K Summing up the individual convolution kernels, obtaining a processed dynamic convolution kernel, and then performing convolution on the features in the graph.

5. The method of claim 1, wherein the method further comprises: the double-layer recurrent network is trained by using a binary cross-entropy loss function in the training stage. 6.A system for co-salient object detection based on a two-layer recurrent network, characterized in that, The method comprises the following steps: a data acquisition and model establishment module configured to acquire an image group, construct a double-layer recurrent network, wherein the double-layer recurrent network comprises a feature extraction module, a plurality of IERR modules in series and a plurality of DRD modules; the IERR module adopts a convLSTM network with a forgetting gate removed and an enhancement gate added; the enhancement gate improves the discrimination ability of inter-graph relationship extraction through correlation constraints, and introduces local similarity constraints of the previous step to strengthen the retention of useful collaborative information in the recursive process; a feature extraction module configured to extract intra-graph features of the image group; an inter-graph extraction module configured to, based on the intra-graph features, use a plurality of IERR modules in series to extract inter-graph features step by step to obtain enhanced collaborative semantic features; an inter-graph distribution module configured to, based on the intra-graph features and the enhanced collaborative semantic features, use a plurality of DRD modules to obtain inter-graph features, and fuse the inter-graph features with the intra-graph features to obtain collaborative saliency features to be decoded. The process of obtaining the inter-graph features comprises: based on the intra-graph features of each image and n one enhanced collaborative semantic feature, using a DRD module, dynamically convolving each enhanced collaborative semantic feature with the intra-graph features to obtain n one convolution feature; Based on n convolutional features, respectively using n convLSTM units, the features output by the first n convLSTM unit are taken as the inter-graph features.

7. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the steps in the collaborative saliency target detection method based on the double-layer recurrent network according to any one of claims 1-5.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps in the collaborative saliency target detection method based on the double-layer recurrent network according to any one of claims 1-5.

Images