Image processing method and system based on hybrid graph convolution network, terminal and medium

Abstract

This application relates to the field of data processing technology, and discloses an image processing method, system, terminal, and medium based on hybrid graph convolutional networks. The method includes: acquiring raw graph data, preprocessing and mapping the raw graph data to obtain feature data and a binarized adjacency matrix; inputting the feature data into a memristor cross-switch array with programmed weights to obtain intermediate node feature vectors; and obtaining a target output feature vector based on the binarized adjacency matrix and the intermediate node feature vectors. This application combines a memristor ReRAM array with a transistor-based reconfigurable circuit, and through graph partitioning and preprocessing of the raw graph data, data binarization and compression, and combination and aggregation calculations, finally generates the output features of the nodes, thereby achieving efficient combination operations and flexible aggregation processes, improving graph data processing efficiency and resource utilization.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to image processing methods, systems, terminals and media based on hybrid graph convolutional networks. Background Technology

[0002] Graph Neural Networks (GNNs), primarily based on Graph Convolutional Networks (GCNs), have proven highly effective for various graph data analysis tasks, demonstrating significant application value in areas such as network analysis and data classification. However, the computational demands and memory access bottlenecks when processing large adjacency matrices, particularly evident in the aggregation phase, lead to high memory access latency and low computational resource utilization efficiency, hindering the implementation of GCNs on large-scale sparse graphs. While matrix operations relying on dedicated processing units (PEs) and memory cell data transfers using embedded dynamic memory (DRAM) can alleviate this problem, frequent data transfers between DRAM and processing units still result in low system energy efficiency, necessitating alternative solutions.

[0003] One feasible approach is to combine novel resistive random access memory (ReRAM) with in-memory computing (CIM) to accelerate graph convolutional network computation. In-memory computing offers the advantage of avoiding frequent data transfers, making it particularly promising for high-efficiency vector-matrix multiplication (VMM) tasks. However, this approach suffers from low durability, high latency and energy consumption during write operations, and write errors due to the randomness of ReRAM. These issues lead to reduced system computational accuracy and efficiency during the aggregation process of graph convolutional networks.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main purpose of this application is to provide an image processing method, system, terminal, and medium based on hybrid graph convolutional networks. It aims to solve the problem that in the prior art, when combining novel resistive random access memory (RRAM) to accelerate graph convolutional network computation, the computational accuracy and efficiency of graph convolutional networks are low due to the high latency and high energy consumption of write operations and the randomness of the novel RRAM.

[0006] The first aspect of this application provides an image processing method based on a hybrid graph convolutional network, the image processing method based on a hybrid graph convolutional network comprising the following steps: Obtain the original graph data, and preprocess and map the original graph data to obtain feature data and a binarized adjacency matrix; The feature data is input into a memristor cross switch array with programmed weights to obtain the feature vector of the intermediate node; The target output feature vector is obtained based on the binarized adjacency matrix and the intermediate node feature vector.

[0007] Optionally, in one embodiment of this application, the original graph data includes a graph node feature matrix and an adjacency matrix; The preprocessing and mapping of the original graph data to obtain feature data and a binary adjacency matrix specifically includes: The original graph data is partitioned to obtain graph node feature matrices and adjacency matrices corresponding to multiple subgraphs, wherein the number of nodes in the multiple subgraphs is equal. For each subgraph, a binarized adjacency matrix is ​​generated based on the adjacency matrix, and feature data is generated based on the graph node feature matrix.

[0008] Optionally, in one embodiment of this application, the feature data includes a value array and a bitmap; The step of generating a binary adjacency matrix based on the adjacency matrix and generating feature data based on the graph node feature matrix specifically includes: Iterate through all the element values ​​of the adjacency matrix. If the element value is not equal to zero, set it to one; if the element value is equal to zero, keep it as zero. Generate a binary adjacency matrix based on the one or zero values ​​corresponding to each element value. Generate a value array and a bitmap based on the graph node feature matrix.

[0009] Optionally, in one embodiment of this application, the step of inputting the feature data into a memristor crossbar switch array with programmed weights to obtain the intermediate node feature vector specifically includes: The feature data is converted into an analog voltage signal; The analog voltage signal is input to a programmed weighted memristor cross switch array, and an analog current signal is output. The analog current signal is converted into a digital signal, and the digital signal is linearly processed to obtain the intermediate node feature vector.

[0010] Optionally, in one embodiment of this application, obtaining the target output feature vector based on the binarized adjacency matrix and the intermediate node feature vector specifically includes: The transistor switch array is dynamically configured according to the binarized adjacency matrix to form a configured transistor switch array. The intermediate node feature vector is input into the configured transistor switch array to obtain the target output feature vector.

[0011] Optionally, in one embodiment of this application, the step of inputting the intermediate node feature vector into the configured transistor switch array to obtain the target output feature vector specifically includes: The intermediate node feature vector is input to the configured transistor switch array, and the total current signal is output. The total current signal is converted and processed to obtain the target output feature vector.

[0012] Optionally, in one embodiment of this application, the step of inputting the intermediate node feature vector into the configured transistor switch array to obtain the target output feature vector further includes: The feature data corresponding to the next subgraph is input into the memristor cross-switch array with programmed weights to obtain the intermediate node feature vector corresponding to the next subgraph, until the target output feature vector corresponding to all subgraphs is obtained, thus completing the forward propagation of a single layer of graph convolutional neural network.

[0013] A second aspect of this application also provides an image processing system based on a hybrid graph convolutional network, wherein the image processing system based on the hybrid graph convolutional network is used to implement the image processing method based on a hybrid graph convolutional network as described in any of the above solutions; the image processing system based on the hybrid graph convolutional network includes: The graph partitioning module is used to acquire the original graph data and preprocess and map the original graph data to obtain feature data and a binarized adjacency matrix. The combined calculation module is used to input the feature data into a memristor cross switch array with programmed weights to obtain the intermediate node feature vector; The aggregation module is used to obtain the target output feature vector based on the binarized adjacency matrix and the intermediate node feature vector.

[0014] A third aspect of this application also provides a terminal, wherein the terminal includes: a memory, a processor, and an image processing program based on a hybrid graph convolutional network stored in the memory and executable on the processor, wherein when the image processing program based on the hybrid graph convolutional network is executed by the processor, it implements the steps of the image processing method based on the hybrid graph convolutional network as described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image processing program based on a hybrid graph convolutional network, and the image processing program based on the hybrid graph convolutional network implements the steps of the image processing method based on the hybrid graph convolutional network as described above when executed by a processor.

[0016] Beneficial effects: This application provides an image processing method, system, terminal, and medium based on hybrid graph convolutional networks. This application combines memristor ReRAM arrays with transistor-based reconfigurable circuits to overcome memory bandwidth bottlenecks and the inherent irregularities of graph computation. By partitioning and preprocessing the original graph data, binarizing and compressing the data, and performing combination and aggregation calculations, the output features of the nodes are finally generated, thereby achieving efficient combination operations and flexible aggregation processes, and improving the efficiency of graph data processing and resource utilization. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the data path of the image processing system of this application; Figure 2 This is a schematic diagram of graph partitioning and aggregation in the image processing system of this application; Figure 3 This is a schematic diagram of the binarized adjacency matrix control of the aggregation module in the image processing system of this application; Figure 4 This is a flowchart illustrating a preferred embodiment of the image processing method based on hybrid graph convolutional networks according to this application; Figure 5 This is a structural diagram of a preferred embodiment of the image processing system based on hybrid graph convolutional networks of this application; Figure 6 This is a structural diagram of a preferred embodiment of the terminal of this application.

[0019] Explanation of reference numerals in the attached figures: 100. Graph Partitioning Module; 200. Combinatorial Calculation Module; 300. Aggregation Module. Detailed Implementation

[0020] To make the objectives, technical solutions, and effects of this application clearer and more explicit, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only possible technical implementations of this application and not all possible implementations. Based on the embodiments in this application, those skilled in the art can obtain other embodiments without creative effort, and these embodiments are also within the protection scope of this application.

[0021] In related technologies, existing graph convolutional networks based on resistive random access memory (RRAM) expand all graph tiles during initialization and pre-store all layer weight parameters in a ReRAM array. While this strategy improves system parallelism and data throughput, it also increases hardware complexity, further leading to issues such as power consumption and resource utilization. Furthermore, graph convolutional networks face the challenge of uneven load distribution during aggregation. The density of graph vertices typically follows a power-law distribution, resulting in significant differences in computational load between nodes. This leads to uneven distribution of computational tasks across multiple cores, causing premature computational bottlenecks. Therefore, a more efficient aggregation method is needed to avoid the low resource utilization caused by frequent writes.

[0022] This application employs a Hybrid Processing Mode Graph Convolutional Network (HyP-GCN) accelerator architecture, combining a ReRAM array with transistor-based circuitry to overcome bandwidth bottlenecks and computational irregularities, enabling efficient combinatorial operations and flexible aggregation processes, while ensuring efficient graphics processing and maximizing hardware resource utilization.

[0023] First, the system architecture of the embodiments of this application will be introduced.

[0024] This application's embodiment of the image processing system based on hybrid graph convolutional networks employs a hybrid architecture. (See [link to relevant documentation]). Figure 1 The system combines a memristor-based cross-switch array-based combinatorial computing module with a transistor array-based reconfigurable aggregation module, and optimizes the sparsity and irregularity issues in graph data processing through graph partitioning and data representation strategies. The image processing system includes: a graph partitioning module, an on-chip storage module, a combinatorial computing module, and an aggregation module.

[0025] The graph partitioning module preprocesses the large-scale input raw graph data, dividing it into smaller, more balanced subgraphs. Then, it reorders the adjacency matrices of these subgraphs to form denser subgraphs. The graph partitioning module uses the METIS algorithm to partition the original graph. See also... Figure 2 In (a) of the algorithm, the METIS algorithm partitions the large-scale original input graph into several balanced subgraphs of roughly equal size, while minimizing the number of edge cuts between the subgraphs. This graph partitioning step transforms the global, irregular graph computation task into multiple local, regular computation tasks, thereby improving the efficiency of subsequent hardware processing. See also... Figure 2In (b), after the graph partitioning step, the coefficient adjacency matrix of the original graph is reordered to form a diagonally block-based dense sub-adjacency matrix. Each diagonal block represents a relatively dense connection relationship within a subgraph, which changes the data access scope in the aggregation stage from globally dispersed non-continuous access to controlled block-level access within the subgraph, reducing the number of cross-block accesses and invalid data reads, thereby improving the execution efficiency of the aggregation process.

[0026] The on-chip storage module uses a reconfigurable mapping strategy to map the node and edge information of the subgraph to feature data and a binary adjacency matrix, and stores the node feature data and binary adjacency matrix data of each subgraph. The on-chip storage module includes embedded dynamic random access memory (eDRAM) and static random access memory (SRAM). EDRAM stores the preprocessed node feature data of each subgraph; SRAM stores the binary adjacency matrix of the subgraph. It is important to note that the use of two types of memory is based on the matching of capacity / bit width characteristics and access timing requirements: node feature data is typically multi-bit and larger in size, making it more suitable for storage in high-density eDRAM to reduce area overhead; the binary adjacency matrix is ​​1-bit control information, which needs to be randomly read frequently during the aggregation stage and directly drive transistor switches, thus making it more suitable for storage in low-latency, stable-read SRAM to ensure the timing and reliability of the control path. The elements of a binary adjacency matrix consist only of "1" or "0" to represent the presence or absence of connections between nodes, replacing the floating-point adjacency matrix that requires Laplacian normalization in conventional graph convolutional networks. The binary adjacency matrix representation significantly reduces storage footprint and computational complexity, while also aligning with subsequent transistor control logic. Understandably, the introduction of the binary adjacency matrix is ​​primarily driven by hardware feasibility: adjacency is used only as a "gating" control bit, reducing storage from floating-point to 1-bit, and allowing the aggregation array to be directly driven by SRAM, significantly simplifying control logic and bandwidth / storage overhead. At the algorithmic level, floating-point adjacency (such as normalization coefficients) can be absorbed or retained as low-bit parameters offline through degree correlation coefficients / scaling factors, allowing for uniform scale correction at the aggregation output. Furthermore, training / fine-tuning can adapt this approximation, thereby achieving higher hardware deployability at an acceptable accuracy.

[0027] The combined computing module includes one or more memristor-based crossbar switch arrays, reads and parses node feature data stored on the on-chip memory module, calculates majority node information, and generates and transmits intermediate node feature vectors. The combined computing module's process includes three steps: Data reading and decoding process: The bitmap-compressed node feature data is read from the on-chip storage module, and the decoding unit restores the complete feature vector based on the bitmap information, providing input for subsequent combination calculations. Node feature data typically has extremely high sparsity. To alleviate the bandwidth pressure caused by this, the on-chip storage module integrates a bitmap compression unit. The bitmap compression unit compresses the feature vector of each node into a value array and a bitmap. The value array stores all non-zero values, and the bitmap marks the position of all non-zero values ​​with bits. Before combination calculations, the decoding unit quickly restores the value array into a complete feature vector based on the bitmap information. Vector-matrix multiplication: The feature vector recovered by the decoding unit is converted into an analog voltage signal by a digital-to-analog converter and input to the memristor cross-switch array. The weight matrix of the graph convolutional network is pre-programmed in the memristor cross-switch array. The memristor cross-switch array completes the vector-matrix multiplication operation in a highly parallel and energy-efficient manner and outputs an analog current signal. Signal Conversion and Buffering: After the analog current output signal is processed by the analog-to-digital converter and shift-accumulator unit, an intermediate node feature vector is obtained. The intermediate node feature vector result is buffered and awaits feeding into the next module. It can be understood that the shift-accumulator unit is used to perform bit-weighted alignment and accumulation of the digital result output by the ADC to support VMM result reconstruction from bit-sliced / multi-cycle output; it outputs the intermediate node feature vector in a standard format and writes it to the buffer, waiting for the aggregation module to read it.

[0028] The aggregation module includes a transistor-based reconfigurable circuit that receives the binarized adjacency matrix and dynamically controls the transistor array using the binarized adjacency matrix to aggregate the feature vectors of intermediate nodes to generate output features.

[0029] See Figure 3 Transistor-based reconfigurable circuits are matrix switching networks composed of a large number of transistors. In a reconfigurable circuit, the gate state of each transistor is determined based on a corresponding element of a binary adjacency matrix stored in static random access memory. For example... Figure 3 As shown, the aggregation module includes: a binary adjacency matrix storage (SRAM), a transistor switch array, and an aggregation bus / accumulation node. During aggregation, the controller reads a row Ai of the binary adjacency matrix A row by row. When element Aij=1, the corresponding transistor switch is turned on, allowing the intermediate feature (from the input buffer) of the j-th node to be selected and injected into the i-th aggregation channel; when Aij=0, the switch is turned off, and the channel does not participate in the accumulation. The aggregation bus sums the current / voltage of all selected features, and outputs the aggregation result of node i after ADC and digital post-processing.

[0030] Dynamic Reconstruction and Aggregation: During aggregation calculation, a row of values ​​in the binarized adjacency matrix is ​​read out sequentially. If a value is "1", the corresponding transistor switches to the on state, allowing the intermediate characteristic current signal from the intermediate node feature vector of the combined module to be transmitted to the aggregation module bus. If a value is "0", the corresponding transistor switches to the off state, and the intermediate characteristic current signal from the intermediate node feature vector of the combined module is ignored. The aggregation bus corresponds to the common accumulation line where the outputs of the transistor switch array converge to the same summing node, used to aggregate the analog currents of all selected branches and form a total analog current signal (followed by an ADC / shift accumulation).

[0031] Subsequent processing: The total analog current signal on the aggregation module bus represents the summation of all activated critical feature values. The total analog current signal is converted into a digital signal by an analog-to-digital converter. The digital signal is then processed by a shift-accumulator unit, such as scaling or offsetting, to finally generate an output feature vector, completing the calculation of one layer of convolutional neural network.

[0032] Specifically, the graph partitioning module uses the METIS algorithm to perform graph partitioning operations, so that the partitioned subgraph achieves a balance in the number of nodes and minimizes edge cuts. All elements in the binarized adjacency matrix are either 0 or 1, representing the presence or absence of connections between nodes, replacing the normalized adjacency matrix.

[0033] The combined computation module further includes: a bitmap compression unit, used to compress sparse node feature data into a value array and a bitmap, wherein the value array stores non-zero feature values ​​and the bitmap stores the positions of non-zero feature values; and a decoding unit, used to restore the compressed data to a feature vector before computation.

[0034] The transistor-based reconfigurable circuit includes multiple transistors, each transistor's gate being connected to a statically accessible memory cell in the on-chip storage module, and its on or off state being controlled by the corresponding element value of the binary connection matrix, thereby dynamically and reconfigurably forming aggregated data.

[0035] The aggregation module further includes: an analog-to-digital conversion unit for converting the analog aggregation signal output by the transistor-based reconfigurable circuit into a digital signal; and a shift and accumulation unit for further processing the digitized aggregation signal result.

[0036] In image processing computation based on hybrid graph convolutional networks, the combination computation module and the aggregation module are executed sequentially in a pipeline manner of first combining and then aggregating. The output of the combination computation module is buffered and used as the input of the aggregation module.

[0037] The image processing system of this application employs a strict "combination-then-aggregation" pipelined approach. The combination computation module first processes all node features of a subgraph, generating intermediate feature vectors and inputting them into a buffer. Subsequently, the aggregation module reads the binarized adjacency matrix corresponding to the subgraph, dynamically configures the transistor reconfigurable circuit, and performs aggregation operations on the buffered intermediate feature vectors. This pipelined design clearly separates the two computation modes, maximizing the advantages of both the memristor array and the transistor array, and minimizing the overhead of frequent data exchange between modules.

[0038] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0039] The image processing method based on hybrid graph convolutional networks described in the preferred embodiments of this application, such as... Figure 4 As shown, the image processing method based on hybrid graph convolutional networks includes the following steps: In step S101, the original graph data is obtained, and the original graph data is preprocessed and mapped to obtain feature data and a binarized adjacency matrix.

[0040] In one possible implementation, the original graph data includes a graph node feature matrix and an adjacency matrix; the feature data includes a value array and a bitmap.

[0041] Step S101 specifically includes: partitioning the original graph data to obtain graph node feature matrices and adjacency matrices corresponding to multiple subgraphs, wherein the number of nodes in the multiple subgraphs is equal; for each subgraph, generating a binarized adjacency matrix based on the adjacency matrix, and generating feature data based on the graph node feature matrix.

[0042] Specifically, the step of generating feature data is implemented as follows: traversing all element values ​​of the adjacency matrix, setting the element value to one if it is not equal to zero, and keeping the element value as zero if it is equal to zero, generating a binary adjacency matrix based on the one or zero values ​​corresponding to each element value; and generating a value array and a bitmap based on the graph node feature matrix.

[0043] Specifically, the original graph structure refers to the original graph structure data, which includes: a node feature matrix, an N×F matrix where N is the number of nodes and F is the feature dimension of each node. This matrix is ​​usually highly sparse; and an adjacency matrix, an N×N sparse matrix that represents the connection relationships between nodes, with non-zero elements representing edges.

[0044] In the graph partitioning step, the METIS algorithm is used to divide the entire large graph into K subgraphs, ensuring that each subgraph contains approximately the same number of nodes and minimizing the number of edges crossing different subgraphs (i.e., connections between subgraphs). The partitioned subgraphs are then reordered so that their adjacency matrices are diagonally block-shaped. That is, connections within a subgraph are concentrated in the diagonal blocks of the matrix, while connections between subgraphs (edge ​​cutting) are pushed to off-diagonal regions and minimized as much as possible. This approach aims to make the computational workload (number of nodes) of each subgraph comparable, facilitating parallel processing; and to reduce random access to remote data during computation by increasing the density of connections within the subgraphs.

[0045] In the adjacency matrix binarization step, for each subgraph's adjacency matrix, all elements are traversed and transformed: if the element value ≠ 0 (indicating a connection exists), it is set to 1; if the element value = 0 (indicating no connection), it remains 0. This application completely omits the Laplace normalization (involving degree matrix calculation and floating-point operations) required in traditional GCN, and simplifies the aggregation operation to a Boolean logic of "accumulate if there is a connection, ignore if there is no connection"; the generated 0 / 1 matrix can be directly used as the control signal for a transistor switch array, where 1 indicates that the switch is open to allow data to pass, and 0 indicates that the switch is closed to cut off the path, achieving a seamless mapping from algorithm logic to hardware behavior. Moreover, each connection is stored using only 1 bit, which is tens of times more compressed than storing floating-point weights.

[0046] Then, feature data compression and on-chip storage are performed. For each node's feature vector, two structures are generated: a value array, which stores all non-zero feature values ​​sequentially; and a bitmap, a bit sequence of length equal to the feature dimension, where the corresponding bit is 1 if a feature dimension is non-zero, and 0 otherwise. The compressed node feature data (value array and bitmap) is then stored in embedded dynamic random access memory (DRAM), which has a large capacity and is suitable for storing the main data; while the binarized adjacency matrix is ​​stored in static random access memory (SRAM), which is fast and suitable for frequent reading of control signals.

[0047] In step S102, the feature data is input to a memristor cross switch array with programmed weights to obtain the intermediate node feature vector.

[0048] Step S102 specifically includes: converting the feature data into an analog voltage signal; inputting the analog voltage signal into a memristor cross switch array with programmed weights to output an analog current signal; converting the analog current signal into a digital signal and performing linear processing on the digital signal to obtain the intermediate node feature vector.

[0049] Specifically, the compressed node feature data (value array and bitmap) of the current subgraph k is read from on-chip storage. More specifically, the decoding unit restores the value array into a complete node feature vector based on the bitmap information. Then, a vector-matrix multiplication simulation is performed: the decoded digital feature vector is input and converted into an analog voltage signal. This voltage signal is input to a memristor cross-switch array pre-programmed with network weights. According to Ohm's law and Kirchhoff's laws, the array performs the analog multiplication and accumulation of the digital feature vector multiplied by the weights in parallel in one operation, and outputs an analog current signal. The analog current signal is converted into a digital signal and subjected to linear processing such as shifting and accumulation to obtain the intermediate node feature vector. The intermediate node feature vector is then temporarily stored in a buffer.

[0050] In step S103, the target output feature vector is obtained based on the binarized adjacency matrix and the intermediate node feature vector.

[0051] Step S103 specifically includes: dynamically configuring the transistor switch array according to the binarized adjacency matrix to form a configured transistor switch array; inputting the intermediate node feature vector into the configured transistor switch array to obtain the target output feature vector.

[0052] The specific implementation of the step of obtaining the target output feature vector is as follows: inputting the intermediate node feature vector into the configured transistor switch array and outputting the total current signal; performing signal conversion and processing on the total current signal to obtain the target output feature vector.

[0053] Specifically, firstly, a row of the binary adjacency matrix of the current subgraph is read from the on-chip memory module (SRAM). Then, the adjacency relationship row vector of the target node i in the binary adjacency matrix is ​​input. This row vector consists of 0 and 1, and its length is equal to the number of nodes in the subgraph. 1 indicates that node i is connected to the corresponding node, and 0 indicates that there is no connection. The goal is to obtain the neighbor connection mode of the target node i as a control signal for configuring the hardware switch.

[0054] In the step of dynamically configuring the transistor switch array, the binarized adjacency matrix row A is used. bin [i,:] Dynamically configures the transistor switch array. For each element j in the row vector, if A bin If [i,j] = 1, then the switch channel connecting node j to target node i in the transistor switch array is opened; if A bin If [i,j]=0, then the corresponding switch channel is turned off. The gate of each transistor switch is connected to the storage A. bin The on-chip storage module with the value [i,j] directly controls the conduction (on) or cutoff (off) of the transistor, thereby forming a dynamically reconfigurable data path network customized only for the target node i.

[0055] In the input intermediate feature vector and analog aggregation step, the intermediate node feature vectors stored in the buffer are input to the configured transistor switch array. The digital intermediate node feature vectors are first converted into analog voltage signals by a digital-to-analog converter; the analog voltage signals are sent to the switch array, and only the turned-on switch channels are allowed to pass through; all passing analog current signals are directly summed on a shared analog bus.

[0056] In the signal conversion and post-processing steps, the total current signal aggregated on the analog bus is converted into a digital signal through an analog-to-digital converter. The converted digital signal is then processed by a shift-accumulator unit, which may perform linear operations such as scaling and biasing to obtain the final output feature vector of the target node i. The above steps are repeated for each row of the binary adjacency matrix (i.e., each target node in the subgraph), that is, for each node in the current subgraph, to complete the aggregation calculation of the entire subgraph.

[0057] While obtaining the target output feature vector corresponding to a subgraph, the method also includes the step of: inputting the feature data corresponding to the next subgraph into a memristor cross-switch array with programmed weights to obtain the intermediate node feature vector corresponding to the next subgraph, until the target output feature vectors corresponding to all subgraphs are obtained, thus completing the forward propagation of a layer graph convolutional neural network.

[0058] Specifically, after the combination module completes the calculation of subgraph 0 and stores the intermediate feature vectors in the buffer, the controller performs concurrent scheduling. For the aggregation module: a command is sent to it to start processing subgraph 0, read the binarized adjacency matrix of subgraph 0 from SRAM, read the intermediate feature vectors of subgraph 0 from the buffer, and perform aggregation calculations. For the combination module: almost simultaneously, a new command is sent to it to start processing the next subgraph (sub-subgraph 0). Figure 1 Read from eDRAM Figure 1 Compressed feature data is used to perform combined calculations, and the result is a sub-database. Figure 1 The intermediate feature vectors will be stored in another buffer unit. This enables concurrent operation of the two core hardware modules: combination and aggregation.

[0059] Repeating the above steps, each time the aggregation of a subgraph is completed, the controller immediately releases the buffer and some storage resources occupied by that subgraph and assigns the aggregation module to process the next subgraph to be processed, forming a stable flow. It is crucial that the aggregation module waits for its required intermediate features to be ready before starting; this is the only data dependency in the pipeline, and the controller must synchronize it. When the aggregation computation of the last subgraph is complete, the controller collects all outputs and notifies the system that the computation of this layer of the GCN is finished, completing the forward propagation of one layer of the graph convolutional neural network.

[0060] The specific implementation of this application will be described below with reference to a specific application scenario.

[0061] S1. Receive the input graph data, including the node feature matrix and the adjacency matrix; S2. Using the METIS algorithm, the input graph data is divided into multiple balanced subgraphs; S3. Binarize the adjacency matrix of each subgraph by setting all non-zero elements to 1 and outputting the binarized adjacency matrix. S4. Perform bitmap compression on the node feature data of each subgraph to separate the value array and bitmap. Store the compressed node feature data of the subgraph and the binarized adjacency matrix into the embedded dynamic random access memory and static random access memory of the on-chip storage module, respectively. S5. For the current subgraph, read the compressed feature data from the embedded dynamic random access memory and decode and restore it. Input the memristor cross switch array to perform vector-matrix multiplication operation to generate and pass the intermediate node feature vector. S6. For the current subgraph, read the binarized adjacency matrix from the static random access memory, dynamically configure the switching state of the transistor-based reconfigurable circuit array using the value of the binarized adjacency matrix, input the buffered intermediate feature vector into the reconfigurable circuit, perform analog aggregation, and output the final aggregation result through the analog-to-digital converter and shift accumulator unit. S7. Repeat steps S5-S6 until all subgraphs have been processed, completing the forward propagation of a single-layer graph convolutional neural network.

[0062] Prior to the combined calculation step, a bitmap compression step is also included to compress the node feature data, so that the combined calculation only transmits and processes non-zero feature values ​​and their location information.

[0063] This invention, through a co-design of software and hardware, deeply integrates graph partitioning, binarization, bitmap compression, and a hybrid memristor-transistor architecture. By combining "soft-side improvements in locality and compression to reduce transmission" with "hard-side parallel computation and edge-gating aggregation," it achieves efficient processing of sparse and irregular graph data while reducing bandwidth pressure. This effectively solves the problems of low computational efficiency and low energy efficiency caused by the sparsity and irregularity of graph data, providing a practical solution for achieving high-performance, low-power graph convolutional neural network inference. It is understood that the software in this application employs graph partitioning and reordering (such as METIS), adjacency matrix binarization, bitmap compression encoding, subgraph scheduling, and mapping strategy generation to improve locality and reduce invalid data access. The hardware employs an on-chip memory architecture (eDRAM / SRAM), a bitmap decoding unit, a ReRAM cross-array combined computation (VMM), an SRAM-controlled transistor reconfigurable aggregation array, an ADC / DAC and shift-accumulator post-processing unit, and a pipeline controller.

[0064] Next, referring to the accompanying drawings, an image processing system based on a hybrid graph convolutional network according to an embodiment of this application is described, which is used to implement the image processing method based on a hybrid graph convolutional network as described in any of the above schemes.

[0065] Figure 5 This is a structural diagram of an image processing system based on a hybrid graph convolutional network according to an embodiment of this application.

[0066] like Figure 5 As shown, the image processing system based on hybrid graph convolutional networks includes: a graph partitioning module 100, a combination calculation module 200, and an aggregation module 300.

[0067] Specifically, the graph partitioning module 100 is used to acquire the original graph data and preprocess and map the original graph data to obtain feature data and a binarized adjacency matrix. The combined calculation module 200 is used to input the feature data into a memristor cross switch array with programmed weights to obtain the intermediate node feature vector; The aggregation module 300 is used to obtain the target output feature vector based on the binarized adjacency matrix and the intermediate node feature vector.

[0068] Figure 6 A structural diagram of a terminal provided in an embodiment of this application. The terminal may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0069] When the processor 502 executes the program, it implements the image processing method based on hybrid graph convolutional networks provided in the above embodiments.

[0070] Furthermore, the terminal also includes: Communication interface 503 is used for communication between memory 501 and processor 502.

[0071] The memory 501 is used to store computer programs that can run on the processor 502.

[0072] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0073] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EIS) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0074] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0075] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0076] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method based on a hybrid graph convolutional network as described above.

[0077] One embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the features described in this application. Figure 4The image processing method based on hybrid graph convolutional networks provided in any of the corresponding embodiments.

[0078] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0079] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0080] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0081] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable storage medium could be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0082] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0083] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0084] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0085] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

[0086] It should be understood that the application of this application is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method based on hybrid graph convolutional networks, characterized in that, The image processing method based on hybrid graph convolutional networks includes: Obtain the original graph data, and preprocess and map the original graph data to obtain feature data and a binarized adjacency matrix; The feature data is input into a memristor cross switch array with programmed weights to obtain the feature vector of the intermediate node; The target output feature vector is obtained based on the binarized adjacency matrix and the intermediate node feature vector.

2. The image processing method based on hybrid graph convolutional networks according to claim 1, characterized in that, The original graph data includes a graph node feature matrix and an adjacency matrix; The preprocessing and mapping of the original graph data to obtain feature data and a binary adjacency matrix specifically includes: The original graph data is partitioned to obtain graph node feature matrices and adjacency matrices corresponding to multiple subgraphs, wherein the number of nodes in the multiple subgraphs is equal. For each subgraph, a binarized adjacency matrix is ​​generated based on the adjacency matrix, and feature data is generated based on the graph node feature matrix.

3. The image processing method based on hybrid graph convolutional networks according to claim 2, characterized in that, The feature data includes a value array and a bitmap; The step of generating a binary adjacency matrix based on the adjacency matrix and generating feature data based on the graph node feature matrix specifically includes: Iterate through all the element values ​​of the adjacency matrix. If the element value is not equal to zero, set it to one; if the element value is equal to zero, keep it as zero. Generate a binary adjacency matrix based on the one or zero values ​​corresponding to each element value. Generate a value array and a bitmap based on the graph node feature matrix.

4. The image processing method based on hybrid graph convolutional networks according to claim 2, characterized in that, The step of inputting the feature data into a memristor crossbar switch array with programmed weights to obtain the intermediate node feature vector specifically includes: The feature data is converted into an analog voltage signal; The analog voltage signal is input to a programmed weighted memristor cross switch array, and an analog current signal is output. The analog current signal is converted into a digital signal, and the digital signal is linearly processed to obtain the intermediate node feature vector.

5. The image processing method based on hybrid graph convolutional networks according to claim 2, characterized in that, The step of obtaining the target output feature vector based on the binarized adjacency matrix and the intermediate node feature vectors specifically includes: The transistor switch array is dynamically configured according to the binarized adjacency matrix to form a configured transistor switch array. The intermediate node feature vector is input into the configured transistor switch array to obtain the target output feature vector.

6. The image processing method based on hybrid graph convolutional networks according to claim 5, characterized in that, The step of inputting the intermediate node feature vector into the configured transistor switch array to obtain the target output feature vector specifically includes: The intermediate node feature vector is input to the configured transistor switch array, and the total current signal is output. The total current signal is converted and processed to obtain the target output feature vector.

7. The image processing method based on hybrid graph convolutional networks according to claim 5, characterized in that, Before inputting the intermediate node feature vector into the configured transistor switch array to obtain the target output feature vector, the process further includes: The feature data corresponding to the next subgraph is input into a memristor cross-switch array with programmed weights to obtain the intermediate node feature vector corresponding to the next subgraph, until the target output feature vector corresponding to all subgraphs is obtained, so as to complete the forward propagation of a single-layer graph convolutional neural network.

8. An image processing system based on a hybrid graph convolutional network, characterized in that, The image processing system based on hybrid graph convolutional networks is used to implement the image processing method based on hybrid graph convolutional networks according to any one of claims 1-7, wherein the image processing system based on hybrid graph convolutional networks comprises: The graph partitioning module is used to acquire the original graph data and preprocess and map the original graph data to obtain feature data and a binarized adjacency matrix. The combined calculation module is used to input the feature data into a memristor cross switch array with programmed weights to obtain the intermediate node feature vector; The aggregation module is used to obtain the target output feature vector based on the binarized adjacency matrix and the intermediate node feature vector.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and an image processing program based on a hybrid graph convolutional network stored in the memory and executable on the processor. When the image processing program based on the hybrid graph convolutional network is executed by the processor, it implements the steps of the image processing method based on a hybrid graph convolutional network as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image processing program based on a hybrid graph convolutional network, which, when executed by a processor, implements the steps of the image processing method based on a hybrid graph convolutional network as described in any one of claims 1-7.

Images