A salient object detection device and method based on multi-graph neural network collaborative learning architecture

Abstract

The present invention is a salient object detection device and method based on a multi-graph neural network collaborative learning architecture. The present invention relates to the field of computer vision technology, determines salient edge features and salient regional features, and converts the salient edge features and salient regional features into edge graph nodes and regional graph nodes respectively through graph projection operations; performs initial graph interaction to realize the transmission of information between graphs; sends the interaction to a dynamic information enhancement graph convolution layer to complete the full mining of graph node information; sends the edge graph nodes and regional graph nodes processed by the dynamic information enhancement graph convolution layer to an attention perception fusion module to complete the complementary fusion of the two features, and provides complementary clues for the learning process of salient edge features and salient regional features. The present invention improves the performance of salient object detection and makes up for the problem of insufficient relational learning ability of traditional convolutional neural networks.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and is a salient object detection device and method based on a multi-graph neural network collaborative learning architecture. Background Art

[0002] The human visual system analyzes and integrates local information in the visual scene to establish an understanding of the entire visual scene, and then quickly and accurately locates the most discriminative and eye-catching areas in the visual scene. Salient object detection is different from the traditional high-cost window traversal search to detect objects. By simulating the human visual system, it efficiently locates the most visually salient areas in the image and evenly separates them from the background, so that subsequent visual algorithms can reasonably allocate computing resources according to the saliency of the area, thereby reducing processing costs and alleviating the contradiction between massive image information and limited computing resources. It is suitable for some resource-constrained applications, such as satellites, drones, etc. At the same time, as a preprocessing step, it assists various advanced visual tasks to further obtain high-level semantic information in the visual scene, thereby establishing a local to overall understanding of the visual scene, such as target recognition, target tracking, image / video subtitles, etc.

[0003] In recent years, salient object detection methods based on deep convolutional neural networks have been widely studied, which effectively improves the detection performance and makes continuous progress in salient object detection research. However, the current salient object detection model based on deep convolutional neural networks is not ideal in processing high-frequency visual information and capturing the global structural information of objects. This is manifested in the acquired salient map, which will cause the salient object edge blur, partial object loss, and background erroneous highlighting, which will affect the performance of subsequent high-level visual analysis tasks. This situation is mainly caused by the structural characteristics of deep convolutional neural networks. On the one hand, since the deep convolutional neural network learning method is to mine the translation invariance and decomposability of the image by designing isotropic filters, downsampling (convolution + pooling) is often used to reduce the dimension of the input data, and upsampling (deconvolution) is used to restore the original size of the input data. This downsampling + upsampling operation causes irreversible loss of high-frequency information such as object edges and structures, which will cause problems such as blurred object edges and partial object loss.

[0004] As a cutting-edge deep learning method, graph neural network gathers information from irregular graph structure data in non-Euclidean space by defining anisotropic filters, better learns and infers the implicit complex logical relationships that must exist within objects, and captures rich contextual structural information, thereby improving the problems of blurred object edges, partial missing objects, and incorrectly highlighted backgrounds in salient maps, making up for the shortcomings of traditional convolutional neural networks. Therefore, the research on salient object detection methods based on graph neural networks in non-Euclidean space has important theoretical significance and application value, and expands and enriches the current research system and technical route of salient object detection. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention effectively learns and infers the complex logical relationships between sub-regions through graph neural networks, captures rich contextual structural information, and fully utilizes the complementary clues provided by both significant edge features and significant region features to maximize information interaction, thereby improving the performance of significant object detection and making up for the insufficient relational learning ability of traditional convolutional neural networks.

[0006] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0007] The present invention provides a salient object detection device and method based on a multi-graph neural network collaborative learning architecture, and the present invention provides the following technical solutions:

[0008] A method for detecting salient objects based on a multi-graph neural network collaborative learning architecture, the method comprising the following steps:

[0009] Step 1: Determine the significant edge features and significant region features, expressed as and

[0010] Step 2: Perform graph projection operations on the salient edge features and salient region features respectively Transformed into edge graph node V e With the area graph node V r ;

[0011] Step 3: Transform the edge graph G eWith area graph G r Perform initial graph interaction to transfer information between graphs;

[0012] Step 4: Transform the edge graph G after interaction e With area graph G r Send it to the dynamic information enhanced graph convolution layer to fully mine the graph node information;

[0013] Step 5: The edge graph nodes processed by the dynamic information enhancement graph convolution layer With area graph nodes The data is sent to the attention perception fusion module to complete the complementary fusion of the two features, providing complementary clues for the learning process of salient edge features and salient region features.

[0014] Step 6: The edge graph nodes and region graph nodes obtained in step 5 are processed by three layers of dynamic information enhanced graph convolution and fused by the corresponding attention perception fusion module in the same way, and then the graph reprojection operation is performed. The final output edge graph node and area graph nodes Map back to the original coordinate space.

[0015] Preferably, the edge graph node V is determined by the following formula: e With the area graph node V r :

[0016]

[0017]

[0018] In the above way, the edge graph G e =(V e ,E e ) and the area graph G r =(V r ,E r ) is constructed, where Represents the set of graph nodes, E e (E r ) represents the set of graph edges.

[0019] Preferably, the step 3 is specifically:

[0020] First, calculate the edge graph node V e With the area graph node V r The similarity matrix A inter , the specific formula is:

[0021]

[0022] Among them, fnorm (·) indicates normalization operation;

[0023] Then the graph interaction information is passed to the edge graph nodes V respectively. e With the area graph node V r ,as follows:

[0024]

[0025]

[0026] Among them, λ∈[0,1] is a trade-off parameter for balancing the importance of graph interaction information and is set to 1 based on experience.

[0027] Preferably, the step 4 is specifically:

[0028] Use Euclidean distance to calculate the nearest k neighbor nodes of each graph node, and use edge learning function To learn the current graph node v i Its neighbor node v j , The edge weight between In order to fully explore the implicit semantic relationships between the current graph node and its neighboring nodes and enhance the information interaction between graph nodes, the specific calculation process is as follows:

[0029]

[0030] Among them, r learn represents the relationship type index, R represents the total number of relationships to be learned, Represents the current graph node v i The neighbor node set, edge learning function Multi-layer perceptron MLP can be used to fit the learning, which is defined as:

[0031]

[0032] Update the current graph node v using information from multiple learnable edges i The embedded information is:

[0033]

[0034] In order to complete the information transfer between different channels within the graph nodes, the channel-based attention normalization coefficient matrix between the graph nodes is calculated, which is expressed as:

[0035]

[0036] Among them, the softmax(·) operation has an effect on the channel-based attention coefficient matrix V TThe results of V are normalized by column, and the elements in each column represent the correlation coefficient from one channel to other channels. Therefore, the information transfer between different channels in the same graph node is completed by the following formula:

[0037]

[0038] In order to improve the stability of the internal features of the same graph node, a residual structure is used to combine the original graph node features V' and V intra ,Right now:

[0039]

[0040] in, Represents the graph node features of the final output, α controls V intra The impact on the original graph node features is empirically set to 0.2.

[0041] Preferably, the step 5 is specifically:

[0042] First, the edge graph nodes With area graph nodes According to the corresponding graph nodes, add each channel to get the initial fusion graph node The calculation method is as follows:

[0043]

[0044] Secondly, the initial fusion graph node The input is fed into an attention submodule consisting of two linear layers to fuse the global node information and generate the node-by-node attention weights, namely:

[0045]

[0046] in, and is a learnable parameter shared by the two linear layers; then, a new fused graph node is obtained by assigning different attention weights to different graph nodes The specific calculation is shown in the following formula:

[0047]

[0048] Among them, *' represents the weighted multiplication operation on a row-by-row basis.

[0049] Preferably, a linear transformation is performed on the weighted graph nodes Generate more discriminative graph node embedding information; finally, the obtained new fusion graph node Respectively with the input edge graph nodes and area graph nodes The added fusion is then sent to the next dynamic information enhanced graph convolutional layer to mine the embedded information of graph nodes.

[0050] Preferably, the step 6 is specifically:

[0051] Using image reprojection operation The final output edge graph node V e final and the region graph node V r final Mapping back to the original coordinate space, that is:

[0052]

[0053]

[0054] A salient object detection device based on a multi-graph neural network collaborative learning architecture, the device comprising:

[0055] A feature classification module, wherein the feature classification module determines significant edge features and significant region features, which are represented as and

[0056] A projection module is used to project significant edge features and significant region features respectively. Transformed into edge graph node V e With the area graph node V r ;

[0057] The interaction module converts the edge graph G e With area graph G r Perform initial graph interaction to transfer information between graphs;

[0058] The feature classification module converts the edge graph G after interaction e With area graph G r Send it to the dynamic information enhanced graph convolution layer to fully mine the graph node information;

[0059] Enhanced graph convolution module, which enhances the edge graph nodes after the dynamic information enhancement graph convolution layer processing With area graph nodes The data is sent to the attention perception fusion module to complete the complementary fusion of the two features, providing complementary clues for the learning process of salient edge features and salient region features.

[0060] A fusion processing module is used to process the edge graph nodes and the region graph nodes obtained in step 5 through three layers of dynamic information enhancement graph convolution and the corresponding attention perception fusion module, and then the graph reprojection operation is performed. The final output edge graph node V e final and the region graph node V r final Map back to the original coordinate space.

[0061] A computer-readable storage medium stores a computer program, which is executed by a processor to implement a salient object detection method based on a multi-graph neural network collaborative learning architecture.

[0062] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a salient object detection method based on a multi-graph neural network collaborative learning architecture is implemented.

[0063] The present invention has the following beneficial effects:

[0064] The present invention can achieve the synergistic enhancement of significant edge features and significant region features, effectively learn and infer the complex logical relationships between sub-regions through graph neural networks, capture rich contextual structural information, and make full use of the complementary clues provided by both significant edge features and significant region features to maximize information interaction, thereby improving the performance of significant object detection and making up for the insufficient relational learning ability of traditional convolutional neural networks.

[0065] The present invention performs performance tests on four public, representative and challenging test datasets: DUTS-TE, ECSSD, PASCAL-S, and DUT-OMRON. In order to ensure the fairness of the comparison with the existing salient object detection methods, the methods for comprehensive evaluation and comparison and the proposed method are trained with the DUTS-TR training dataset, and all competing methods are generated by published source code with default parameters or obtained from publicly available results. Among them, the 8 convolutional neural network-based salient object detection methods involved in the comparison include PoolNet, BASNet, EGNet, AFNet, DFI, LDF, PFSNet, and DCN. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 Compare the results for the evaluation indicators;

[0068] Figure 2 PR curves on four standard data sets;

[0069] Figure 3 This is a visual comparison example with existing advanced methods;

[0070] Figure 4 A sample of local magnification visualization in a complex scene;

[0071] Figure 5 It is a structural schematic diagram;

[0072] Figure 6 Schematic diagram of the structure of the salient edge feature extraction module guided by semantic information;

[0073] Figure 7 It is a schematic diagram of the structure of the multi-scale fusion salient region feature extraction module;

[0074] Figure 8 Schematic diagram of the collaborative learning architecture of multi-graph neural networks;

[0075] Fig. 9 Schematic diagram of the edge-supported graph convolution module structure. DETAILED DESCRIPTION

[0076] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0078] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0079] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0080] The present invention is described in detail below in conjunction with specific embodiments. Specific embodiment one:

[0082] according to Figures 1 to 9 As shown, the specific optimization technical solution adopted by the present invention to solve the above technical problems is: the present invention relates to a salient object detection device and method based on a multi-graph neural network collaborative learning architecture.

[0083] A method for detecting salient objects based on a multi-graph neural network collaborative learning architecture, characterized in that the method comprises the following steps:

[0084] Step 1: Determine the significant edge features and significant region features, expressed as and

[0085] Step 2: Perform graph projection operations on the salient edge features and salient region features respectively Transformed into edge graph node V e With the area graph node V r ;

[0086] Step 3: Transform the edge graph G e With area graph G r Perform initial graph interaction to transfer information between graphs;

[0087] Step 4: Transform the edge graph G after interaction e With area graph G r Send it to the dynamic information enhanced graph convolution layer to fully mine the graph node information;

[0088] Step 5: The edge graph nodes processed by the dynamic information enhancement graph convolution layer With area graph nodes The data is sent to the attention perception fusion module to complete the complementary fusion of the two features, providing complementary clues for the learning process of salient edge features and salient region features.

[0089] Step 6: The edge graph nodes and region graph nodes obtained in step 5 are processed by three layers of dynamic information enhanced graph convolution and fused by the corresponding attention perception fusion module in the same way, and then the graph reprojection operation is performed. The final output edge graph node V e final and the region graph node V r final Map back to the original coordinate space.

[0090] Figure 2 The comparison results based on the PR curve show that compared with other salient object detection methods, when the recall rate is fixed, the proposed method has a higher accuracy in most cases. For example, on the DUTS-TE test dataset, when the recall rate is 0.8, the proposed method achieves a higher accuracy of about 97%. In addition, the overall performance of the proposed method on the PASCAL-S dataset and DUT-OMRON dataset with complex scenes is better than that of other methods, which further shows that the proposed method has high performance robustness and is suitable for more complex scenes.

[0091] In addition to the above quantitative analysis experiments, we also selected some representative test images for visualization comparison experiments to more intuitively demonstrate the performance comparison between the proposed method and existing advanced methods. Figure 3 The figure shows the saliency visual comparison results of the proposed method and the existing advanced methods, including some extremely challenging and complex situations: i: there is a non-salient background area inside the salient object, ii: there are multiple salient objects in the visual scene, iii: the salient object is occluded, iv: the non-salient background is complex and the salient object is small, and v: the contrast between the foreground and the background is low. In addition, in order to further evaluate the detection capabilities of different salient object detection methods, we also enlarged some image sample examples, such as Figure 4 shown.

[0092] like Figure 3As shown in Figure 2, from the visual samples of salient maps in these complex scenes, it can be seen that compared with our proposed method, most existing advanced methods do not correctly segment salient objects, and there are problems such as blurred edges of salient objects, partial missing objects, and wrong highlighting of complex backgrounds. However, the proposed method can accurately predict the most salient objects and coherent details in challenging complex scenes, such as occlusion of salient objects (rows 5-6), reflections of salient objects in the background (row 8), and low contrast (rows 9-10). In addition, from Figure 4 In the figure, we can also clearly see that the edge preservation ability of our proposed method is better than other methods in challenging scenarios such as the presence of non-salient background areas within salient objects, occlusion of salient objects, and complex backgrounds. These visual comparison results further demonstrate the excellent salient object detection ability of the proposed method and verify the effectiveness and generalization ability of the proposed method on challenging datasets. Specific embodiment 2:

[0094] The difference between the second embodiment of the present application and the first embodiment is that:

[0095] The edge graph node V is determined by the following formula e With the area graph node V r :

[0096]

[0097]

[0098] In the above way, the edge graph G e =(V e ,E e ) and the area graph G r =(V r ,E r ) is constructed, where Represents the set of graph nodes, E e (E r ) represents the set of graph edges. Specific embodiment three:

[0100] The difference between the third embodiment of the present application and the second embodiment is that:

[0101] The step 3 is specifically as follows:

[0102] First, calculate the edge graph node V e With the area graph node V r The similarity matrix A inter , the specific formula is:

[0103]

[0104] Among them, f norm (·) indicates normalization operation;

[0105] Then the graph interaction information is passed to the edge graph nodes V respectively. e With the area graph node V r ,as follows:

[0106]

[0107]

[0108] Among them, λ∈[0,1] is a trade-off parameter for balancing the importance of graph interaction information and is set to 1 based on experience. Specific embodiment four:

[0110] The difference between the fourth embodiment of the present application and the third embodiment is that:

[0111] The step 4 is specifically as follows:

[0112] Use Euclidean distance to calculate the nearest k neighbor nodes of each graph node, and use edge learning function To learn the current graph node v i With its neighbor node vj, The edge weight between In order to fully explore the implicit semantic relationships between the current graph node and its neighboring nodes and enhance the information interaction between graph nodes, the specific calculation process is as follows:

[0113]

[0114] Among them, r learn represents the relationship type index, R represents the total number of relationships to be learned, Represents the current graph node v i The neighbor node set, edge learning function Multi-layer perceptron MLP can be used to fit the learning, which is defined as:

[0115]

[0116] Update the current graph node v using information from multiple learnable edges i The embedded information is:

[0117]

[0118] In order to complete the information transfer between different channels within the graph nodes, the channel-based attention normalization coefficient matrix between the graph nodes is calculated, which is expressed as:

[0119]

[0120] Among them, the softmax(·) operation has an effect on the channel-based attention coefficient matrix V T The results of V are normalized by column, and the elements in each column represent the correlation coefficient from one channel to other channels. Therefore, the information transfer between different channels in the same graph node is completed by the following formula:

[0121]

[0122] In order to improve the stability of the internal features of the same graph node, a residual structure is used to combine the original graph node features V' and V intra ,Right now:

[0123]

[0124] in, Represents the graph node features of the final output, α controls V intra The impact on the original graph node features is empirically set to 0.2. Specific embodiment five:

[0126] The difference between the fifth embodiment of the present application and the fourth embodiment is that:

[0127] The step 5 is specifically as follows:

[0128] First, the edge graph nodes With area graph nodes According to the corresponding graph nodes, add each channel to get the initial fusion graph node The calculation method is as follows:

[0129]

[0130] Secondly, the initial fusion graph node The input is fed into an attention submodule consisting of two linear layers to fuse the global node information and generate the node-by-node attention weights, namely:

[0131]

[0132] in, and is a learnable parameter shared by the two linear layers; then, a new fused graph node is obtained by assigning different attention weights to different graph nodes The specific calculation is shown in the following formula:

[0133]

[0134] Among them, *' represents the weighted multiplication operation on a row-by-row basis. Specific embodiment six:

[0136] The difference between the sixth embodiment of the present application and the fifth embodiment is that:

[0137] Perform a linear transformation on the weighted graph nodes Generate more discriminative graph node embedding information; finally, the obtained new fusion graph node Respectively with the input edge graph nodes and area graph nodes The added fusion is then sent to the next dynamic information enhanced graph convolutional layer to mine the embedded information of graph nodes. Specific embodiment seven:

[0139] The difference between the seventh embodiment of the present application and the sixth embodiment is that:

[0140] The step 6 is specifically as follows:

[0141] Using image reprojection operation The final output edge graph node V e final and the region graph node V r final Mapping back to the original coordinate space, that is:

[0142]

[0143]

[0144] from Figure 1 It can be clearly seen that the proposed method is significantly better than the comparison method in all performance indicators on the four public benchmark datasets, especially in S α The two indicators of S and MAE show that the output saliency map of the method of the present invention has excellent structural similarity and accuracy. For example, on the PASCAL-S test dataset, compared with the second-ranked method, the method of the present invention has excellent S α , The performance of the proposed method on the DUTS-TE test dataset is improved by about 1.7%, 2.7% and 16.4% respectively. The index value ranks second, but the overall performance is the best, especially S α This further illustrates that the method of the present invention has advantages in capturing rich contextual structure information. Specific embodiment eight:

[0146] The difference between the eighth embodiment of the present application and the seventh embodiment is only that:

[0147] The present invention provides a salient object detection device based on a multi-graph neural network collaborative learning architecture, wherein the device comprises:

[0148] A feature classification module, wherein the feature classification module determines significant edge features and significant region features, which are represented as and

[0149] A projection module is used to project significant edge features and significant region features respectively. Transformed into edge graph node V e With the area graph node V r ;

[0150] The interaction module converts the edge graph G e With area graph G r Perform initial graph interaction to transfer information between graphs;

[0151] The feature classification module converts the edge graph G after interaction e With area graph G r Send it to the dynamic information enhanced graph convolution layer to fully mine the graph node information;

[0152] Enhanced graph convolution module, which enhances the edge graph nodes after the dynamic information enhancement graph convolution layer processing With area graph nodes The data is sent to the attention perception fusion module to complete the complementary fusion of the two features, providing complementary clues for the learning process of salient edge features and salient region features.

[0153] A fusion processing module is used to process the edge graph nodes and the region graph nodes obtained in step 5 through three layers of dynamic information enhancement graph convolution and the corresponding attention perception fusion module, and then the graph reprojection operation is performed. The final output edge graph node and area graph nodes Map back to the original coordinate space. Specific embodiment nine:

[0155] The difference between the ninth embodiment of the present application and the eighth embodiment is that:

[0156] The present invention provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement a salient object detection method based on a multi-graph neural network collaborative learning architecture. Specific embodiment ten:

[0158] The difference between the tenth embodiment of the present application and the ninth embodiment is that:

[0159] The present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a method for detecting significant objects based on a multi-graph neural network collaborative learning architecture is implemented. Specific embodiment eleven:

[0161] The difference between the eleventh embodiment of the present application and the tenth embodiment is that:

[0162] The overall design of a salient object detection method based on a multi-graph neural network collaborative learning architecture is as follows: Figure 5 The detailed process is as follows:

[0163] (1) First, we use ResNet50 as the backbone network to preliminarily extract edge features and regional features from its low and high layers respectively;

[0164] (2) The semantic information-guided salient edge feature extraction module is used to extract salient region features that contain partial global semantic information and local detail information. Meanwhile, the multi-scale fusion salient region feature extraction module is used to capture robust salient region features.

[0165] (3) Performing graph projection operations on the significant edge features and significant regional features respectively, transforming them into non-Euclidean space to obtain the corresponding map structure data, namely, the edge map and the regional map;

[0166] (4) The edge graph and region graph are input into the multi-graph neural network collaborative learning architecture for processing to capture richer contextual structure information while maximizing information interaction;

[0167] (5) After the information mining process, the edge map and the region map are respectively subjected to a reprojection operation to restore them to the original Euclidean space;

[0168] (6) Using the edge-supported graph convolution module, useful information is extracted from the salient edge features to further guide the feature learning of the salient regions;

[0169] (7) The final salient region features are input into the fully connected layer to generate salient prediction results.

[0170] The functions of each module are as follows:

[0171] Semantic information-guided salient edge feature extraction module: Considering that the high-level features of the network mainly contain abstract semantic information, while the low-level features mainly contain more detailed information, in order to capture salient edge features, it is not enough to use only the low-level local edge information, but also needs to integrate the high-level semantic information of the high-level as a guide. Therefore, the consistent difference aggregation method is used to fully and effectively integrate the low-level features and the high-level features. The specific fusion process is as follows: Figure 6 shown.

[0172] The specific workflow is as follows:

[0173] (1) First, the input low-level features and high-level features are element-wise multiplied, and at the same time, they are constrained by the saliency truth map to obtain high-confidence region features consistent with the salient objects, namely the joint consistency feature F c :

[0174]

[0175] in, Represents element-wise multiplication operation, F low represents the low-level features, F high represents high-level features, and S represents the saliency truth map;

[0176] (2) Combine the consistency features F c Used to enhance the low-level features F of the input low and high-level features F high The salient clues in the fusion function are used to obtain the initial fusion features.

[0177]

[0178] in, Represents element-level addition operation, initial fusion feature Achieved consistent enhancement of features, especially salient areas;

[0179] (3) Then, calculate the low-level features F of the input low and high-level features F high The joint difference feature F d :

[0180]

[0181] in, Represents element-wise subtraction operation, combined with the difference feature F d Describes the characteristic differences of significant regions;

[0182] (4) Through initial fusion features and the joint difference feature F d Get the final fusion features

[0183] Multi-scale fusion salient region feature extraction module: In order to capture more robust salient region features, based on the widely used U-Net architecture, we added additional convolutional layers and ReLU layers to the outputs of block_3, block_4, and block_5 of the backbone network. The specific structure is as follows: Figure 7 shown.

[0184] The specific workflow is as follows:

[0185] (1) The multi-scale features output by block_3, block_4, and block_5 of the backbone network ResNet50 are enhanced through the added convolutional layers, and then the ReLU activation function is used to ensure nonlinearity;

[0186] (2) The enhanced multi-scale features are added and fused to obtain robust salient region features.

[0187] Multi-graph neural network collaborative learning architecture: The salient edge features and salient regional features in the original Euclidean space are modeled as graph structure data in non-Euclidean space, namely edge graph and region graph. While using dynamic information enhanced graph convolution layers for feature learning, the learnable attention-aware fusion module completes the complementary fusion of the two features, providing complementary clues for the two graph learning processes, thereby capturing richer contextual structural information and maximizing information interaction. The detailed workflow is as follows: Figure 8 shown.

[0188] The specific workflow is as follows:

[0189] (1) According to formula (3), the similarity matrix between edge graph nodes and region graph nodes is calculated: Then the initial graph interaction information is passed to the edge graph nodes and the region graph nodes respectively to enhance the information embedded in the graph nodes, which is calculated by formula (4) and formula (5).

[0190] (2) The edge graph and the region graph that have completed information interaction are subjected to dynamic information enhancement graph convolution operations according to formulas (6)-(11) to fully mine the embedded information of the graph nodes;

[0191] (3) A learnable attention-aware fusion module is used to effectively fuse the embedded information of edge graph nodes with the embedded information of region graph nodes, and the fusion results are used to The embedded information of edge graph nodes and region graph nodes is complementary enhanced. The specific calculation process is shown in formulas (12)-(14);

[0192] (4) Similarly, the edge map and the region map are further processed by three layers of dynamic information enhanced graph convolution and the corresponding attention perception fusion module to output the final edge map and region map.

[0193] Edge-supported graph convolution module: In order to further accurately locate the salient regions, edge-supported graph convolution operation is used to extract valuable information from the salient edge features and guide the feature learning of the salient regions. The detailed guidance process is as follows Fig. 9 shown.

[0194] The specific workflow is as follows:

[0195] (1) First, a fully connected layer is used to map the input salient edge features into a salient edge prediction map.

[0196] (2) Apply the attention mechanism to predict the salient edge and the salient region feature F′ C Multiply, that is:

[0197]

[0198] in, Represents channel-level multiplication operation;

[0199] (3) Using graph projection operations F e Transformed into edge support graph G e (E)=(V e ,E e ),Right now:

[0200] Among them, V e Represents the set of graph nodes, E e represents a graph edge set;

[0201] (4) The salient region feature vector f i ' c ∈F′ C As a central node, As its edge support node, the edge learning function is defined as:

[0202]

[0203] Among them, h φ (·) is a nonlinear function with learnable parameters φ;

[0204] (5) Feature information f of the current central node i ' c Get updated, that is:

[0205]

[0206] (6) In summary, the salient edge feature E is explicitly encoded into the salient region feature F′ C Among them, namely:

[0207] Among them, ESGConv(·) indicates that the edge supports graph convolution operations.

[0208] We use the popular PyTorch and PyTorch-Geometry tool libraries to implement the proposed model, and use the SGD optimizer based on the poly learning rate adjustment strategy for optimization. The model loads a publicly available ResNet50 pre-trained model as the backbone network and is trained on a TITAN RTX GPU using the DUTS-TR training dataset. Among them, the hyperparameters that need to be adjusted during model training are the number of graph nodes (num_clusters), the number of nearest neighbor nodes, the number of relationship types, the base learning rate (base_lr), power, weight decay (weight_decay), momentum (momentum), batch size (batch_size), and the number of iterations (epochs). Quantitative experiments and qualitative experimental analysis are conducted on four public benchmark datasets: DUTS-TE, ECSSD, PASCAL-S, and DUT-OMRON. In order to ensure the fairness of the comparison with existing salient object detection methods, the methods for comprehensive evaluation and comparison and the proposed method are trained on the DUTS-TR training dataset, and all competing methods are generated by published source code with default parameters or obtained from publicly available results.

[0209] The above is only a preferred implementation of a salient object detection device and method based on a multi-graph neural network collaborative learning architecture. The protection scope of a salient object detection device and method based on a multi-graph neural network collaborative learning architecture is not limited to the above embodiments. All technical solutions under this idea belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, several improvements and changes without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A salient object detection method based on a multi-graph neural network collaborative learning architecture, characterized by: The method comprises the following steps: Step 1: Determine the significant edge features and significant region features, expressed as and Step 2: Perform graph projection operations on the salient edge features and salient region features respectively Transformed into edge graph node V e With the area graph node V r ; Step 3: Transform the edge graph G e With area graph G r Perform initial graph interaction to transfer information between graphs; Step 4: Transform the edge graph G after interaction e With area graph G r Send it to the dynamic information enhanced graph convolution layer to fully mine the graph node information; Step 5: The edge graph nodes processed by the dynamic information enhancement graph convolution layer With area graph nodes The data is sent to the attention perception fusion module to complete the complementary fusion of the two features, providing complementary clues for the learning process of salient edge features and salient region features. Step 6: The edge graph nodes and region graph nodes obtained in step 5 are processed by three layers of dynamic information enhanced graph convolution and fused by the corresponding attention perception fusion module in the same way, and then the graph reprojection operation is performed. The final output edge graph node V e final and the region graph node V r final Map back to the original coordinate space.

2. The method for detecting salient objects based on a multi-graph neural network collaborative learning architecture according to claim 1, characterized in that: The edge graph node V is determined by the following formula e With the area graph node V r : In the above way, the edge graph G e =(V e ,E e ) and the area graph G r =(V r ,E r ) is constructed, where Represents the set of graph nodes, E e (E r ) represents the set of graph edges.

3. The method for detecting salient objects based on a multi-graph neural network collaborative learning architecture according to claim 2 is characterized by: The step 3 is specifically as follows: First, calculate the edge graph node V e With the area graph node V r The similarity matrix A inter , the specific formula is: Among them, f norm (·) indicates normalization operation; Then the graph interaction information is passed to the edge graph nodes V respectively. e With the area graph node V r ,as follows: Among them, λ∈[0,1] is a trade-off parameter for balancing the importance of graph interaction information and is empirically set to 1.

4. The method for detecting salient objects based on a multi-graph neural network collaborative learning architecture according to claim 3 is characterized by: The step 4 is specifically as follows: Use Euclidean distance to calculate the nearest k neighbor nodes of each graph node, and use edge learning function To learn the current graph node v i Its neighbor nodes The edge weight between In order to fully explore the implicit semantic relationships between the current graph node and its neighboring nodes and enhance the information interaction between graph nodes, the specific calculation process is as follows: Among them, r learn represents the relationship type index, R represents the total number of relationships to be learned, Represents the current graph node v i The neighbor node set, edge learning function Multi-layer perceptron MLP can be used to fit the learning, which is defined as: Update the current graph node v using information from multiple learnable edges i The embedded information is: In order to complete the information transfer between different channels within the graph nodes, the channel-based attention normalization coefficient matrix between the graph nodes is calculated, which is expressed as: Among them, the softmax(·) operation has an effect on the channel-based attention coefficient matrix V T The results of V are normalized by column, and the elements in each column represent the correlation coefficient from one channel to other channels. Therefore, the information transfer between different channels in the same graph node is completed by the following formula: In order to improve the stability of the internal features of the same graph node, a residual structure is used to combine the original graph node features V' and V intra ,Right now: in, Represents the graph node features of the final output, α controls V intra The impact on the original graph node features is empirically set to 0.

2.

5. The method for detecting salient objects based on a multi-graph neural network collaborative learning architecture according to claim 4 is characterized by: The step 5 is specifically as follows: First, the edge graph nodes With area graph nodes According to the corresponding graph nodes, add each channel to get the initial fusion graph node The calculation method is as follows: Secondly, the initial fusion graph node The input is fed into an attention submodule consisting of two linear layers to fuse the global node information and generate the node-by-node attention weights, namely: in, and is a learnable parameter shared by the two linear layers; then, a new fused graph node is obtained by assigning different attention weights to different graph nodes The specific calculation is shown in the following formula: Among them, *' represents the weighted multiplication operation on a row-by-row basis.

6. The method for detecting salient objects based on a multi-graph neural network collaborative learning architecture according to claim 5, characterized in that: Perform a linear transformation on the weighted graph nodes Generate more discriminative graph node embedding information; Finally, the new fusion graph node is obtained Respectively with the input edge graph nodes and area graph nodes The added fusion is then sent to the next dynamic information enhanced graph convolutional layer to mine the embedded information of graph nodes.

7. The method for detecting salient objects based on a multi-graph neural network collaborative learning architecture according to claim 6 is characterized by: The step 6 is specifically as follows: Using image reprojection operation The final output edge graph node V e final and the region graph node V r final Mapping back to the original coordinate space, that is:

8. A salient object detection device based on a multi-graph neural network collaborative learning architecture, characterized by: The device comprises: A feature classification module, wherein the feature classification module determines significant edge features and significant region features, which are represented as and A projection module is used to project significant edge features and significant region features respectively. Transformed into edge graph node V e With the area graph node V r ; The interaction module converts the edge graph G e With area graph G r Perform initial graph interaction to transfer information between graphs; The feature classification module converts the edge graph G after interaction e With area graph G r Send it to the dynamic information enhanced graph convolution layer to fully mine the graph node information; Enhanced graph convolution module, which enhances the edge graph nodes after the dynamic information enhancement graph convolution layer processing With area graph nodes The data is sent to the attention perception fusion module to complete the complementary fusion of the two features, providing complementary clues for the learning process of salient edge features and salient region features. A fusion processing module, which processes the edge graph nodes and region graph nodes obtained by the enhanced graph convolution module in the same way, and then processes them by the corresponding attention perception fusion module, and then uses a graph reprojection operation. The final output edge graph node V e final and the region graph node V r final Map back to the original coordinate space.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement a salient object detection method based on a multi-graph neural network collaborative learning architecture as described in claims 1-6.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method for detecting significant objects based on a multi-graph neural network collaborative learning architecture described in claims 1-6 is implemented.

Images