A method and apparatus for extracting and integrating feature relationships within and between images
By constructing internal and cross-relationship network models, the modeling of feature relationships within and between images is enhanced, solving the problem of limited visual reasoning performance in existing technologies and achieving more efficient image feature representation and rule reasoning capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-13
- Publication Date
- 2026-03-24
AI Technical Summary
Existing relational network modeling methods have shortcomings in image feature representation and high-order nonlinear relation modeling, which limits the inference performance of machine vision and makes it impossible to accurately extract feature relationships within and between images.
An internal relation network and a cross relation network are constructed to model the spatial and channel relationships within an image, as well as the relationships between different images. The convergence of the convolutional neural network is accelerated through compression layers, attention-based relation module layers, and recovery layers. The output results are fused using a convolutional fusion function to enhance the ability to extract image feature relationships.
It improves the accuracy of image abstract visual reasoning and the ability to model rule-based reasoning, significantly enhancing the performance of convolutional neural networks in abstract visual tasks.
Smart Images
Figure CN115272716B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to a method for extracting and integrating the feature relationship between images and within images, and also relates to a device using the method, belonging to the technical field of computer vision. BACKGROUND
[0002] In recent years, abstract visual reasoning, as a key component of human intelligence, has been widely studied in the field of computer vision technology. At present, a very effective test of human visual reasoning ability has been developed, which is called the Raven test (Raven Progressive Matrices, abbreviated as RPM). The Raven test is directly located in the center of human intelligence, and is a judgment of abstract and structural reasoning ability, and describes the defining characteristics of high-level intelligence, i.e. fluid intelligence. In order to break through the current limit of visual reasoning ability, Zhu Songchun team of UCLA also constructed the relationship and analogy visual reasoning data set RAVEN based on the Raven test.
[0003] With the emergence of various convolutional neural networks (CNN), people try to measure the lateral thinking and fluid intelligence of machines. The latest progress of convolutional neural networks shows excellent performance in abstract visual reasoning. In order to develop the abstract visual perception ability of machines, there are two basic problems to be solved: one is the recognition representation of images, and the other is the discovery of the internal rules of images. Although the existing relation network (RN) modeling method has significantly improved the performance of abstract visual reasoning, the current relation network modeling only focuses on the external cross relationship of various images. Due to the lack of internal relationship modeling, the neural network cannot accurately extract the image feature representation. At the same time, the relation network based modeling method only considers the simple linear relationship of any two images, and ignores the high-order nonlinear relationship. As a potential solution, the characteristics of graph neural network (GNN) are that it can effectively model the relationship of non-Euclidean data structure. In addition, the excellent performance of neural network in non-linear relationship modeling is also widely used in recent computer vision tasks. However, due to the lack of effective relationship modeling, the visual reasoning process has a great performance loss.
[0004] In their paper "A dual attention network with semantic embedding for few-shot learning" (published in Proc. AAAI Conf. Artif. Intell., vol. 33, pp. 9079–9086, July 2019), Shipeng Yan et al. from ShanghaiTech University proposed a novel meta-learning method for few-shot learning based on two simple attention mechanisms: spatial attention to locate relevant object regions and task attention to select similar training data for label prediction. This paper implements the method using a dual attention network and designs a semantically aware meta-learning loss to train the meta-learner network in an end-to-end manner. Summary of the Invention
[0005] The primary technical problem to be solved by this invention is to provide a method for extracting and integrating the feature relationships within and between images.
[0006] Another technical problem to be solved by the present invention is an apparatus for extracting and integrating feature relationships within and between images.
[0007] To achieve the above objectives, the present invention adopts the following technical solution:
[0008] According to a first aspect of the present invention, a method for extracting and integrating feature relationships within and between images is provided, comprising the following steps:
[0009] An internal relation network is used to represent the feature relationships within an image, and a cross relation network is used to represent the feature relationships between images; wherein, the internal relation network is divided into a spatial relation module and a channel relation module;
[0010] The image sequence is input into the spatial relationship module, the channel relationship module, and the cross relationship network, respectively. In the spatial relationship module, the channel relationship module, and the cross relationship network, the image sequence is processed by the compression layer, the attention-based relationship module layer, and the recovery layer in sequence, and then enters the batch normalization layer to accelerate the convergence speed of the convolutional neural network.
[0011] The outputs of the spatial relationship module, the channel relationship module, and the cross relationship network are combined using a convolutional fusion function to produce the final output.
[0012] Preferably, in the spatial relationship module, each image is regarded as a node for constructing a graph, and a graph neural network is constructed based on the visual similarity between any two nodes to complete the modeling of row / column pixel relationships.
[0013] Preferably, in the channel relationship module, each channel is regarded as a node for constructing the graph.
[0014] Preferably, all input images are compressed into one, and then feature maps embedded with global information are used to learn the feature similarity of nodes; using the feature similarity, nodes can be updated through a graph neural network.
[0015] Preferably, the compression layer is implemented by a channel dimension compression function.
[0016] Preferably, the recovery layer is implemented by a channel dimension recovery function.
[0017] Preferably, the attention-based relational module layer is implemented through the following function:
[0018]
[0019]
[0020]
[0021] Among them, w i It is a learnable parameter used to enhance nodes. The self-weight, N i yes The adjacent nodes, As an edge, as and The connection relationship, It is a learnable parameter matrix used to reduce computational complexity and improve efficiency. Learning ability.
[0022] Preferably, the convolution fusion function is a fusion function using 1*1 2D convolution.
[0023] Preferably, the internal and cross-relationship network model constructed by the method is embedded into a convolutional neural network to complete the visual reasoning fusion part.
[0024] According to a second aspect of the present invention, a method for extracting and integrating feature relationships within and between images is provided, comprising a processor and a memory, wherein the processor reads a computer program from the memory for performing the following operations:
[0025] An internal relation network is used to represent the feature relationships within an image, and a cross relation network is used to represent the feature relationships between images; wherein, the internal relation network is divided into a spatial relation module and a channel relation module;
[0026] The image sequence is input into the spatial relationship module, the channel relationship module, and the cross relationship network, respectively. In the spatial relationship module, the channel relationship module, and the cross relationship network, the image sequence is processed by the compression layer, the attention-based relationship module layer, and the recovery layer in sequence, and then enters the batch normalization layer to accelerate the convergence speed of the convolutional neural network.
[0027] The outputs of the spatial relationship module, the channel relationship module, and the cross relationship network are combined using a convolutional fusion function to produce the final output.
[0028] Compared with existing technologies, the method provided by this invention enhances the ability to model pixel-level object relationships within an image by constructing an ICRN model and using an internal relational network, thereby enhancing the ability to recognize symbols in the image. Simultaneously, it uses a cross-relational network to establish relationships between different images to find patterns, thus enhancing the ability to model rule-based reasoning. This ICRN model can be embedded into common convolutional neural networks to enhance their reasoning capabilities in abstract visual reasoning tasks. Attached Figure Description
[0029] Figure 1 A logical framework diagram of the method for extracting and integrating feature relationships within and between images provided by the present invention;
[0030] Figure 2 Data flow diagram for the method of extracting and integrating feature relationships within and between images provided by this invention;
[0031] Figure 3 A structural diagram of the device for extracting and integrating feature relationships within and between images provided by the present invention;
[0032] Figure 4 This is a comparison chart of the performance of the ICRN model on all categories of the RAVEN dataset, showing the effects of average fusion and convolutional fusion. Detailed Implementation
[0033] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0034] As previously mentioned, recent advances in convolutional neural networks have demonstrated excellent performance in abstract visual reasoning. However, visual reasoning suffers significant performance loss due to the lack of effective relation modeling. To address this issue, this invention first provides a method for extracting and integrating feature relationships within and between images. This method constructs an Intra and CrossRelation Network (ICRN) model, where the IntraRelation Network is used to enhance the ability to model pixel-level object relationships within images, thereby improving the ability to recognize symbols in images; simultaneously, the CrossRelation Network is used to establish relationships between different images to find patterns, thus enhancing rule-based reasoning modeling capabilities.
[0035] The following is combined Figure 1 The logical architecture diagram shown provides a detailed explanation of this method.
[0036] In one embodiment of the invention, we consider the relationships between each row / column / channel in an image, using the image as a node to establish suitable and adaptable internal relationships within the image. The content of each image and selectable panels are used as nodes to learn and infer internal relationships.
[0037] For input X (for lack of generality, X is rewritten as X), l First, we obtain the nodes of the graph. Where d, K, and l represent the dimension of each node, the number of nodes, and the l-th layer of the graph neural network, respectively. For example... Figure 1 and Figure 2 As shown, the image sequence passes through the layers of a convolutional neural network and obtains feature maps. Let X = R. N*C*H*W Here, N, C, H, and W represent the number of images, the number of channels, and the length and width of the spatial dimension, respectively. We model the relationships between non-local objects and learn inference rules through internal relationship networks and cross relationship networks, respectively.
[0038] The internal relationship network comprises a spatial relationship module and a channel relationship module. The spatial relationship module constructs a graph neural network based on the visual similarity between any two nodes, modeling the row / column pixel relationships. The channel relationship module is similar to the spatial relationship module, treating each channel as a node in the graph. Accordingly, we rewrite X as X oriented towards the spatial relationship module. IS =R N*C*H*W and the X-oriented channel relationship module IC =R N*C*H*W .
[0039] To leverage global information from different images and reduce computational complexity, we compress all input images into a single image. Then, we use a feature map embedded with global information to learn the feature similarity of nodes, reflecting the connections between them. Using this feature similarity, we can update nodes through a graph neural network.
[0040] See Figure 2 The data flow diagram shown has two types of inputs in the spatial relationship module, named rows and columns. We use X... IS To uniformly represent the two inputs, the corresponding formula is defined as follows:
[0041]
[0042]
[0043] in, As input to the spatial relations module, f IS It is a dimensionality compression function that compresses both the channel dimension and the G dimension. IS The learnable parameter matrix, As f IS The output, then we use G IS It involves spatial relationship modeling. IS =R N*C*H*W This represents the output of the spatial relationships module. It's important to note that because spatial relationship modeling considers the difference between row and column relationships, Y... IS It can be divided into two parts
[0044] For the channel relationship module, we rewrite X as X C =R N*C*H*W Y IC =R N*C*H*W This represents the output of the channel relationship module. The corresponding formula is defined as follows:
[0045]
[0046]
[0047] Among them, X IC ∈R N*C*HW and They are f IC The input and output can be compressed from N images to 1, and this can be achieved through a 1x1 convolution transformation. IC This is the corresponding extended function used to restore the number of images to N.
[0048] When humans perform abstract visual reasoning, they explore rules by considering correlations and differences between different input images. Inspired by human nature, we use graph neural networks to build a network of cross-relationships between these input images. Similar to internal relationship networks, we treat each image as a node in the graph.
[0049] In the cross-relationship network, input X C ∈R N*C*H*W (For simplicity, we will rewrite X as X) C (Ignoring the reshaping operation), we first compress channel C to 1 to reduce computational complexity and learn the parameterized edge matrix A. C Then, this matrix is used to update the graph nodes to establish relationships between different images and restore channel dimensions. The corresponding formula can be written as:
[0050]
[0051]
[0052]
[0053] Among them, f C and h C These are channel dimension compression and channel dimension recovery functions, used to reduce computation and increase network depth to enhance feature modeling capabilities. It is a channel compressed feature map, used to calculate the edge matrix A. C G C It is a function for modeling cross-relationships, and can be used with formulas. Implement it.
[0054] See Figure 1 The corresponding edge matrix A IC It can be calculated using the following formula:
[0055]
[0056] It should be noted that, in Figure 2 In the data flow diagram shown, the f function and the h function serve as the compression layer and recovery layer of the convolutional neural network, respectively, to reduce computational load and increase network depth to enhance feature modeling capabilities. Their specific implementation in convolutional neural networks is a common technique known to those skilled in the art, and will not be elaborated upon here.
[0057] To simultaneously model internal and cross relationships and achieve optimal abstract visual reasoning performance, we fuse the internal relationship network and the cross relationship network together to form... Figure 1The diagram shows the internal and cross-relationship network model (abbreviated as ICRN model). Here, Y is the fusion of the outputs of the internal and cross-relationship networks.
[0058] Typically, we can use an averaging function to weight internal and cross-relationships. However, a simple averaging function cannot reflect the mutual influence between global and local inference. Therefore, we use a convolutional fusion function θ to adaptively fuse these different inference sub-modules. In particular, we concatenate internal and cross-relationship features along the channel dimension. Thus, the corresponding calculation formula is as follows:
[0059]
[0060] Where Y is the fusion result of the outputs of the spatial relationship module, the channel relationship module, and the cross relationship network, and is also the output result of the method described in this invention; [] represents the concatenation function, and θ is the fusion function using 1*1 2D convolution. and These are the row and column components of spatial modeling. Y IS Y IC and Y C These are the outputs of the spatial relationship module, the channel relationship module, and the cross relationship network, respectively, where Y C =R N*C*H*W .
[0061] like Figure 2 As shown, the computation of each spatial relationship module, channel relationship module, and cross-relationship network requires data processing through an attention-based relationship module (ARM) layer. This ARM layer is used to establish relationships between different input images, different regions within an image, and different channels within the same image. The specific implementation process is as follows:
[0062]
[0063]
[0064]
[0065] Where w i It is a learnable parameter used to enhance nodes. The self-weight, N i yes The adjacent nodes, As an edge, as and The connection relationship, It is a learnable parameter matrix that can be used to reduce computational complexity and improve efficiency. Learning ability. G IC It is a channel relationship modeling function, which can be used with formulas. accomplish.
[0066] In fact, for our problem, nodes The number of nearby nodes is fixed (we assume a fixed value of K-1). Therefore, we can rewrite the above formula as a matrix formula and define an attention-based relation modeling function G, as follows:
[0067]
[0068]
[0069] Among them, A={a ij |i,j∈{1,2,…,K}j}, W = {w i ∈R|i∈{1,2,3,……,K}} and It consists of two learnable parameter matrices, each containing only 1 and 0 values as matrix elements.
[0070] It should be noted that, in one embodiment of the present invention, the output Y of the spatial relationship module, the channel relationship module, and the cross relationship network is obtained. IS Y IC and Y C Previously, these processes involved ReLU (Rectified Linearity) and Sigmoid functions, respectively, along with batch normalization (BN) layers to accelerate the convergence of the convolutional neural network. These are standard techniques commonly known to those skilled in the art, and will not be elaborated upon here.
[0071] Based on the above-described method for extracting and integrating features within and between images, this invention further provides an apparatus for extracting and integrating features within and between images. For example... Figure 3As shown, the device includes a processor 32 and a memory 31, and may further include communication components, sensor components, power supply components, multimedia components, and input / output interfaces as needed. The memory, communication components, sensor components, power supply components, multimedia components, and input / output interfaces are all connected to the processor 32. The memory 31 can be a static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc. The processor can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. Other communication components, sensor components, power supply components, multimedia components, etc., can be implemented using common components in existing electronic devices, and will not be specifically described here.
[0072] On the other hand, in the above-described device, the processor 32 reads the computer program in the memory 31 to perform the following operations:
[0073] An internal relation network is used to represent the feature relationships within an image, and a cross relation network is used to represent the feature relationships between images; wherein, the internal relation network is divided into a spatial relation module and a channel relation module;
[0074] The image sequence is input into the spatial relationship module, the channel relationship module, and the cross relationship network, respectively. In the spatial relationship module, the channel relationship module, and the cross relationship network, the image sequence is processed by the compression layer, the attention-based relationship module layer, and the recovery layer in sequence, and then enters the batch normalization layer to accelerate the convergence speed of the convolutional neural network.
[0075] The outputs of the spatial relationship module, the channel relationship module, and the cross relationship network are combined using a convolutional fusion function to produce the final output.
[0076] It should be noted that the method and apparatus for extracting and integrating feature relationships within and between images provided by this invention can be embedded into common convolutional neural networks to enhance reasoning capabilities in abstract visual reasoning tasks. Specifically, the obtained output Y is inserted into the remaining parts of the original convolutional neural network to complete the abstract visual reasoning fusion portion.
[0077] The superior performance achieved by this invention will be demonstrated below through specific experimental data.
[0078] We evaluated the technical solution of the present invention on three popular public datasets: Raven, I-RAVEN, and PGM.
[0079] RAVEN: Relational and Analogical Visual Evaluation (Raven) is one of Raven's main Progressive Matrix (RPM) datasets. It consists of 1,120,000 images and 70,000 RPM questions across 7 different graph configurations. We also split the dataset into three parts with a 6:2:2 split for training, validation, and testing. For our experiments, we used RAVEN-10000, with 10,000 questions per graph configuration.
[0080] I-RAVEN: This dataset corrects the bias issue in the panel of eight multiple-choice questions in RAVEN. In other words, the original RAVEN had a problem where the correct answer could be found simply by browsing the answer set without considering the context image. The I-RAVEN dataset uses a tree-structured design to generate the dataset, thus correcting this problem in RAVEN. The training, validation, and test splits are the same as the original RAVEN dataset.
[0081] The PGM (Programmed Matrix) dataset is another very important dataset for the Raven's Test. The entire dataset includes data from multiple training classes; we only use the neutral class. It contains 1.42 million samples, including 1.2 million training set questions, 20,000 validation set questions, and 200,000 test set questions.
[0082] We begin by presenting a performance comparison with state-of-the-art methods on the Raven dataset. Table 1 compares the performance of abstract visual inference on the RAVEN dataset. Here, Acc represents the average accuracy of the test, and the other columns represent the corresponding test accuracy for each category. 2_2G, 3_3G, LR, UD, O-IC, and O-IG represent 2_2 grid, 3_3 grid, Left-Right, Up-Down, Out-In Center, and Out-In Grid, respectively.
[0083]
[0084] Table 1
[0085] As shown in Table 1, our proposed ICRN model achieves improved inference accuracy compared to conventional LSTM methods, convolutional neural network-based methods (such as ResNet18), and contrastive learning methods like CoPINet. Compared to the classic relational modeling method WReN, our proposed ICRN model significantly improves performance in abstract visual reasoning.
[0086] Furthermore, as shown in the detailed comparison in Table 1, our proposed ICRN model improves the average accuracy by nearly 2% for each category except for "Left-Right," "Up-Down," and "Out-In Center," and the average accuracy by 2.64%. Its accuracy compared to the original model is very high. For "2_2 grid" and "3_3 grid," our ICRN-CoPINet (i.e., ICRN embedded in a CoPINet) model significantly improves accuracy by 3.00% and 3.55% respectively compared to the original CoPINet. Simultaneously, the results indicate that relational modeling is more effective for "2_2 grid" and "3_3 grid" structures.
[0087] Embedding LSTM and ResNet18 into our proposed ICRN model significantly improves inference performance. Specifically, for “Left-Right” and “Out-In Center”, ICRN-ResNet18 achieves performance improvements of 4.61% and 3.32% respectively compared to the original model. This further demonstrates the effectiveness of our proposed ICRN model on the balanced RAVEN dataset.
[0088]
[0089] Table 2
[0090] Table 2 presents a performance comparison of abstract visual reasoning on the PGM dataset. As shown in Table 2, our proposed ICRN-CoPINet model achieves significant performance improvements on the PGM dataset compared to other state-of-the-art methods. Specifically, our proposed ICRN-CoPINet model achieves significant improvements compared to basic neural network models (e.g., CNN+MLP, LSTM, ResNet50, and Wild-ResNet). Furthermore, Table 2 shows that embedding our SCRN model achieves the best abstract visual reasoning performance compared to the CoPINet model.
[0091]
[0092] Table 3
[0093] Furthermore, we also evaluated our proposed ICRN model on the I-RAVEN dataset, as shown in Table 3. Table 3 is a comparison table of abstract visual reasoning performance on the I-RAVEN dataset, where Acc represents the average accuracy of the test accuracy, and the other columns represent the corresponding test accuracy for each category.
[0094] To further demonstrate the effectiveness of our proposed ICRN model, we embedded it into basic neural network models (such as CNN+MLP, LSTM, ResNet50, and Wild-ResNet). As shown in Tables 1 to 3, embedding the proposed ICRN model significantly improves the performance of both LSTM and WReN models. Furthermore, leveraging its internal relation modeling capabilities, our proposed ICRN model can achieve even greater performance improvements for LSTM models that already possess some relation modeling capabilities.
[0095] like Figure 4 As shown, compared to average fusion (AvgFusion), convolutional fusion (ConvFusion) of the ICRN model achieves better performance across all categories of the RAVEN dataset. "Center", "Left-Right", "Up-Down", "Out-In Center", and "Out-In Grid" categories show significant performance improvements through convolutional fusion, while the improvements for "2_2 grid" and "3_3 grid" are relatively smaller. Notably, convolutional fusion achieves an 8.08% improvement in average accuracy compared to average fusion. In conclusion, convolutional fusion can significantly improve performance due to its adaptive feature map fusion capabilities. Therefore, we adopt convolutional fusion as the preferred fusion method.
[0096] Compared with existing technologies, the method provided by this invention enhances the ability to model pixel-level object relationships within an image by constructing an ICRN model and using an internal relational network, thereby enhancing the ability to recognize symbols in the image. Simultaneously, it uses a cross-relational network to establish relationships between different images to find patterns, thus enhancing the ability to model rule-based reasoning. This ICRN model can be embedded into common convolutional neural networks to enhance their reasoning capabilities in abstract visual reasoning tasks.
[0097] The method and apparatus for extracting and integrating feature relationships within and between images provided by this invention have been described in detail above. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.
Claims
1. A method for extracting and integrating feature relationships within and between images, characterized in that... Includes the following steps: An internal relation network is used to represent the feature relationships within an image, and a cross relation network is used to represent the feature relationships between images. The internal relation network is divided into spatial relation modules and channel relation modules. In the cross relation network, each image is considered a node in the graph construction, and channel C is compressed to 1 to reduce computational complexity and learn a parameterized edge matrix. Using the edge matrix Update graph nodes to establish relationships between different images and restore channel dimensions; The image sequence is input into the spatial relationship module, the channel relationship module, and the cross relationship network, respectively. In the spatial relationship module, the channel relationship module, and the cross relationship network, the image sequence is processed by the compression layer, the attention-based relationship module layer, and the recovery layer in sequence, and then enters the batch normalization layer to accelerate the convergence speed of the convolutional neural network. use The 2D convolution is used as a convolution fusion function to fuse the outputs of the spatial relationship module, the channel relationship module, and the cross relationship network together as the output result; the internal and cross relationship network model formed by fusing the internal relationship network and the cross relationship network is embedded into the convolutional neural network to complete the visual reasoning fusion task.
2. The method as described in claim 1, characterized in that: In the spatial relationship module, each image is regarded as a node for constructing a graph. A graph neural network is constructed based on the visual similarity between any two nodes to complete the modeling of row / column pixel relationships.
3. The method as described in claim 1, characterized in that: In the channel relationship module, each channel is regarded as a node for constructing the graph.
4. The method as described in claim 1, characterized in that: All input images are compressed into one, and then feature maps embedded with global information are used to learn the feature similarity of nodes; using the feature similarity, nodes can be updated through a graph neural network.
5. The method as described in claim 1, characterized in that: The compression layer is implemented using a channel dimension compression function.
6. The method as described in claim 1, characterized in that: The recovery layer is implemented using a channel dimension recovery function.
7. The method as described in claim 1, characterized in that: The attention-based relational module layer is implemented through the following function: in, It is a learnable parameter used to enhance nodes. Self-weight, yes The adjacent nodes, As an edge, as and Connection relationship, It is a learnable parameter matrix used to reduce computational complexity and improve efficiency. Learning ability.
8. An apparatus for extracting and integrating feature relationships within and between images, characterized in that... It includes a processor and a memory, wherein the processor reads a computer program from the memory for implementing the method according to any one of claims 1 to 7.