Binocular image super-resolution method and device based on graph neural network

Abstract

The present invention discloses a binocular image super-resolution method and device based on a graph neural network, the method comprising: inputting a low-resolution binocular image into a feature extraction network to generate initial features; inputting the initial features into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution and low-resolution features; fusing the first residual feature and the second residual feature and inputting them into a reconstruction network to obtain a reconstructed feature; and outputting a super-resolution image according to the reconstructed feature. The present invention can achieve better super-resolution performance and reduce erroneous textures and blurred details in super-resolution results, and can be widely used in the field of image processing technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a binocular image super-resolution method and device based on a graph neural network. Background Art

[0002] With the rapid development of binocular cameras, the demand for high-quality binocular images is increasing, and binocular image super-resolution technology has also emerged. There are generally two methods for binocular image super-resolution: one is to perform single-frame super-resolution on two frames of images separately, and the other is to perform dual-frame super-resolution on two frames of images simultaneously.

[0003] However, the single-frame super-resolution algorithm cannot utilize the complementary information from different viewpoints in the binocular image pair, which limits the super-resolution performance of the binocular image. For the existing binocular image super-resolution methods, how to adaptively utilize the complementary information from two different viewpoints while obtaining a more reliable prior is the current challenge facing binocular image super-resolution. However, the existing methods treat the unevenly distributed information from the two viewpoints in the same way, which affects the super-resolution performance. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a binocular image super-resolution method and device based on a graph neural network to achieve better super-resolution performance and reduce erroneous textures and blurred details in the super-resolution results.

[0005] One aspect of the present invention provides a binocular image super-resolution method based on a graph neural network, comprising:

[0006] Input the low-resolution binocular image into the feature extraction network to generate initial features;

[0007] Inputting the initial features into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution features and low-resolution features;

[0008] The first residual feature and the second residual feature are fused and input into a reconstruction network to obtain a reconstruction feature;

[0009] A super-resolution image is output according to the reconstruction features.

[0010] Optionally, inputting the low-resolution binocular image into a feature extraction network to generate initial features includes:

[0011] Inputting the low-resolution binocular image into a feature extraction network based on weight sharing of four RDBs to generate local features of different viewpoints;

[0012] The local features of the different viewpoints are subjected to convolution processing to obtain the initial features.

[0013] Optionally, the binocular interaction network includes a viewpoint twisting module, a binocular graph neural network module and a residual feature fusion module;

[0014] The step of inputting the initial features into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution features and low-resolution features includes:

[0015] Inputting the initial feature into the viewpoint twisting module to perform viewpoint twisting processing to obtain a first result;

[0016] After the viewpoint is twisted, the first result is filtered and updated by a binocular image neural network module to obtain a second result;

[0017] The residual feature fusion module extracts high-frequency information from the cross-viewpoint residual features and the high- and low-resolution residual features in the second result to determine the first residual features and the second residual features.

[0018] Optionally, in the step of inputting the initial features into the viewpoint twisting module to perform viewpoint twisting processing to obtain the first result,

[0019] The expression for rotating the right viewpoint to the left viewpoint is:

[0020]

[0021] Among them, F r→l Represents the twist feature from the right viewpoint to the left viewpoint; M r→l represents the disparity attention map; represents batch matrix multiplication operation; H conv (.) represents 1×1 convolution; F r Represents the initial features of the right view obtained by the feature extraction network.

[0022] Optionally, after the viewpoint is twisted, filtering and updating the first result through a binocular image neural network module to obtain a second result, including:

[0023] Constructing a first image and a second image on a left viewpoint and a right viewpoint respectively;

[0024] In the first image and the second image, the left viewpoint is used as a reference viewpoint, and a corresponding pixel block is selected on the reference viewpoint as a query block;

[0025] Taking all pixel blocks in the window centered on the query block as candidate nodes;

[0026] Determine the correlation between each candidate node and the query block by using the Euclidean distance;

[0027] Connecting the query block to the target node with the highest correlation;

[0028] Determine a corresponding edge according to a difference between the target node and the query block, and complete the construction of the first graph and the second graph;

[0029] Alternatingly updating the first map and the second map;

[0030] Obtain reference viewpoint features and cross-viewpoint features from the last layer of the binocular image neural network;

[0031] Aggregate the features of each connected node and update the aggregation results through the graph convolutional network;

[0032] Each corresponding edge is converted into a weight to complete the propagation and node update between the first graph and the second graph.

[0033] Optionally, the residual feature fusion module consists of convolution and deconvolution;

[0034] The feature fusion module calculates the residual features between different viewpoints based on the complementary information between different viewpoints;

[0035] Among them, the expression of the first residual feature is:

[0036]

[0037] Among them, F′ in Represents the first residual feature between different viewpoints; H rb represents the residual block; Represents upsampling the reference viewpoint features with a scale factor of 2; Represents upsampling of the cross-view features with a scale factor of 2;

[0038] The expression of the second residual feature is:

[0039]

[0040] Among them, F fusion represents the second residual feature; F′ in ↓2 represents the downsampling of the first residual features between different viewpoints with a scale factor of 2; F l represents the initial features of the left view obtained by the feature extraction network; F′ in Represents the first residual feature between different viewpoints.

[0041] Optionally, the expression of the reconstruction feature is:

[0042] F SR =(α0F l +α1H rb (H FEN (F fusion )))↑2

[0043] Among them, F SR represents the reconstruction feature; α0 and α1 are adaptive learnable parameters, H FEN (·) represents the feature extraction network; F l represents the initial features of the left view obtained by the feature extraction network; H rb represents the residual block; F fusion Represents the second residual feature.

[0044] Another aspect of the embodiment of the present invention further provides a binocular image super-resolution device based on a graph neural network, comprising:

[0045] The first module is used to input the low-resolution binocular image into the feature extraction network to generate initial features;

[0046] A second module is used to input the initial features into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution features and low-resolution features;

[0047] A third module is used to fuse the first residual feature and the second residual feature and input them into a reconstruction network to obtain a reconstruction feature;

[0048] The fourth module is used to output a super-resolution image according to the reconstruction features.

[0049] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0050] The memory is used to store programs;

[0051] The processor executes the program to implement the method described above.

[0052] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0053] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0054] The embodiment of the present invention inputs a low-resolution binocular image into a feature extraction network to generate initial features; the initial features are input into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution and low-resolution features; the first residual feature and the second residual feature are fused and input into a reconstruction network to obtain a reconstructed feature; and a super-resolution image is output based on the reconstructed feature. The present invention can achieve better super-resolution performance and reduce erroneous textures and blurred details in super-resolution results. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 An overall step flow chart provided for an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the structure of a binocular image neural network module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] The present invention is mainly to solve the problem of dual-frame image super-resolution. The existing dual-frame super-resolution method simply splices the features of the left and right viewpoints, and then uses the convolution kernel to equally process the unevenly distributed information from the two viewpoints. Based on the graph neural network, the present invention captures the long-range dependency between binocular image information across viewpoints, adopts the neighborhood aggregation framework for representation learning, and realizes the extraction of consistent complementary information between different viewpoints. In order to achieve better super-resolution performance, while reducing the erroneous texture and blurred details in the super-resolution results.

[0060] Specifically, one aspect of the present invention provides a binocular image super-resolution method based on a graph neural network, comprising:

[0061] Input the low-resolution binocular image into the feature extraction network to generate initial features;

[0062] Inputting the initial features into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution features and low-resolution features;

[0063] The first residual feature and the second residual feature are fused and input into a reconstruction network to obtain a reconstruction feature;

[0064] A super-resolution image is output according to the reconstruction features.

[0065] Optionally, inputting the low-resolution binocular image into a feature extraction network to generate initial features includes:

[0066] Inputting the low-resolution binocular image into a feature extraction network based on weight sharing of four RDBs to generate local features of different viewpoints;

[0067] The local features of the different viewpoints are subjected to convolution processing to obtain the initial features.

[0068] Optionally, the binocular interaction network includes a viewpoint twisting module, a binocular graph neural network module and a residual feature fusion module;

[0069] The step of inputting the initial features into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution features and low-resolution features includes:

[0070] Inputting the initial feature into the viewpoint twisting module to perform viewpoint twisting processing to obtain a first result;

[0071] After the viewpoint is twisted, the first result is filtered and updated by a binocular image neural network module to obtain a second result;

[0072] The residual feature fusion module extracts high-frequency information from the cross-viewpoint residual features and the high- and low-resolution residual features in the second result to determine the first residual features and the second residual features.

[0073] Optionally, in the step of inputting the initial features into the viewpoint twisting module to perform viewpoint twisting processing to obtain the first result,

[0074] The expression for rotating the right viewpoint to the left viewpoint is:

[0075]

[0076] Among them, F r→l Represents the twist feature from the right viewpoint to the left viewpoint; M r→l represents the disparity attention map; represents batch matrix multiplication operation; H conv(.) represents 1×1 convolution; F r Represents the initial features of the right view obtained by the feature extraction network.

[0077] Optionally, after the viewpoint is twisted, filtering and updating the first result through a binocular image neural network module to obtain a second result, including:

[0078] Constructing a first image and a second image on a left viewpoint and a right viewpoint respectively;

[0079] In the first image and the second image, the left viewpoint is used as a reference viewpoint, and a corresponding pixel block is selected on the reference viewpoint as a query block;

[0080] Taking all pixel blocks in the window centered on the query block as candidate nodes;

[0081] Determine the correlation between each candidate node and the query block by using the Euclidean distance;

[0082] Connecting the query block to the target node with the highest correlation;

[0083] Determine a corresponding edge according to a difference between the target node and the query block, and complete the construction of the first graph and the second graph;

[0084] Alternatingly updating the first map and the second map;

[0085] Obtain reference viewpoint features and cross-viewpoint features from the last layer of the binocular image neural network;

[0086] Aggregate the features of each connected node and update the aggregation results through the graph convolutional network;

[0087] Each corresponding edge is converted into a weight to complete the propagation and node update between the first graph and the second graph.

[0088] Optionally, the residual feature fusion module consists of convolution and deconvolution;

[0089] The feature fusion module calculates the residual features between different viewpoints based on the complementary information between different viewpoints;

[0090] Among them, the expression of the first residual feature is:

[0091]

[0092] Among them, F′ in Represents the first residual feature between different viewpoints; H rb represents the residual block; Represents upsampling the reference viewpoint features with a scale factor of 2; Represents upsampling of the cross-view features with a scale factor of 2;

[0093] The expression of the second residual feature is:

[0094] F fusion =(F′ in ↓2-F l )↑2+F′ in

[0095] Among them, F fusion represents the second residual feature; F′ in ↓2 represents the downsampling of the first residual features between different viewpoints with a scale factor of 2; F l represents the initial features of the left view obtained by the feature extraction network; F′ in Represents the first residual feature between different viewpoints.

[0096] Optionally, the expression of the reconstruction feature is:

[0097] F SR =(α0F l +α1H rb (H FEN (F fusion )))↑2

[0098] Among them, F SR represents the reconstruction feature; α0 and α1 are adaptive learnable parameters, H FEN (·) represents the feature extraction network; F l represents the initial features of the left view obtained by the feature extraction network; H rb represents the residual block; F fusion Represents the second residual feature.

[0099] Another aspect of the embodiment of the present invention further provides a binocular image super-resolution device based on a graph neural network, comprising:

[0100] The first module is used to input the low-resolution binocular image into the feature extraction network to generate initial features;

[0101] A second module is used to input the initial features into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution features and low-resolution features;

[0102] A third module is used to fuse the first residual feature and the second residual feature and input them into a reconstruction network to obtain a reconstruction feature;

[0103] The fourth module is used to output a super-resolution image according to the reconstruction features.

[0104] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0105] The memory is used to store programs;

[0106] The processor executes the program to implement the method described above.

[0107] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0108] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0109] The specific implementation principle of the present invention is described in detail below in conjunction with the accompanying drawings:

[0110] The present invention proposes a technical solution for binocular image super-resolution based on graph neural network. That is, a pair of low-resolution binocular images are input. Generate super-resolution stereo images The overall technical solution is as attached. Figure 1 As shown in the figure, it is divided into three parts: feature extraction network, binocular interaction network and reconstruction network. High-resolution binocular image is regarded as the true value of the data, and the symbol s represents the scaling factor (e.g., 2, 4).

[0111] like Figure 1 As shown, the present invention first inputs a low-resolution binocular image into a feature extraction network to generate initial features. The initial features are then input into a binocular interaction network, which includes a viewpoint twisting, a binocular graph neural network, and a residual feature fusion module. After the viewpoint twisting converts different viewpoints into the same viewpoint, two k-nearest neighbor graphs are constructed and alternately iterated on the viewpoint and the cross viewpoint based on feature similarity, which is called a binocular graph neural network. Subsequently, the residual feature fusion module fuses the residual features from different viewpoints and the residual features between high-resolution and low-resolution features. Finally, the fused features are input into an adaptive reconstruction network, and then the final features are converted to the image space.

[0112] The feature extraction network of the present invention is further described below:

[0113] First, Input into the feature extraction network based on 4 RDB weight sharing to obtain local features of different viewpoints Perform convolution on the local features to obtain the fusion feature F l ,F r .

[0114] The binocular interaction network of the present invention is further described below:

[0115] The binocular interaction network extracts and fuses reliable cross-viewpoint information. The network consists of three parts: viewpoint twisting, binocular graph neural network, and residual feature fusion module. After viewpoint twisting, the binocular graph neural network guides the initial features to be filtered and updated. Finally, the residual feature fusion module extracts richer high-frequency information from cross-viewpoint residual features and high- and low-resolution residual features.

[0116] In the viewpoint twisting module, the initial features obtained by the feature extraction network need to be twisted to the same viewpoint. For example, the way to twist the right viewpoint to the left viewpoint can be expressed as the following formula:

[0117]

[0118]

[0119]

[0120]

[0121] Among them, H rb (·) represents the residual block, conv(·) represents 1×1 convolution, H rs (·) represents the matrix dimension change operation. Then, Q l ,K r ∈R H×W×C Perform batch matrix multiplication Get the disparity attention map M r→l ∈R H×w×W , so the twisting feature F from right to left r→l It can be roughly calculated using the above formula.

[0122] In the binocular graph neural network, as shown in the attached Figure 2 As shown in Figure 1, the iterative binocular graph neural network layer can adaptively exploit the uneven prior information of cross-viewpoints. In order to non-locally select and exploit the complementary cross-view prior, two graphs G are constructed on the reference viewpoint (left viewpoint) and the other viewpoint respectively. in = {V in ,E in}, G cross = {V cr ,E cr}. Take each 3×3 pixel block from the reference viewpoint as a query block, denoted as q0. In Figure G in In the example, all 3×3 pixel blocks in the window centered at the reference viewpoint q0 are candidate nodes. c The correlation between q0 and the feature domain f q0 ,f qc The Euclidean distance || f q0 -f qc || 2 The main node q0 is only connected to the most relevant K nodes, and the corresponding edge is defined as the difference between the main node and the relevant node || f q0 -f qc ||. In Figure G cross The construction of the Chinese map is similar.

[0123] During the graph propagation process, the graph G is updated alternately. in and Figure G cross , the reference viewpoint features are obtained from the last layer of the binocular image neural network and cross-view features For each connected node q c ∈{q1,…,q K}, first aggregate their features, and then perform an update process based on the graph convolutional network to transform each edge into a weight {w1,…,w K}. The update function can be defined as:

[0124]

[0125] Therefore, the binocular graph neural network of the present invention has good message adaptive propagation capability and node update capability.

[0126] In the residual feature fusion module, the feature fusion equation is as follows:

[0127]

[0128] F fusion =(F′ in ↓2-F l )↑2+F′ in #

[0129] Operations ↓2 and ↑2 represent downsampling (scaling factor is 2) and upsampling (scaling factor is 2), respectively, which consist of convolution and deconvolution operations. The above formula calculates the residual feature F′ between different viewpoints. in , to pay more attention to the complementary information between viewpoints. The following formula calculates the residual feature F between high-resolution features and low-resolution features fusion, to extract more high-frequency information. In addition, this step helps to upsample the low-resolution features (scale factor is 2).

[0130] The reconstruction network of the present invention is further described below:

[0131] The fused features first pass through the same structure as the initial feature extraction network, and then pass through a series of continuous residual blocks to obtain deeper information. This embodiment provides an adaptive skip residual structure that can better balance the contribution of prior information between the initial features and the deep features. Therefore, the reconstructed feature F SR It can be obtained by the following formula:

[0132] F SR =(α0F l +α1H rb (H FEN (F fusion )))↑2

[0133] Among them, α0, α1 are adaptive learnable parameters, H FEN (·) represents the feature extraction network. In addition, the bicubic interpolation component plays an important role in the final super-resolution result. The final output super-resolution image is shown in the following formula:

[0134]

[0135] Among them, H tran (·) represents the process of transforming from feature space to image space, The image is upsampled by bicubic interpolation.

[0136] In summary, the present invention proposes a binocular super-resolution network based on graph neural network, and proposes a binocular graph neural network module to capture the long-range dependency between binocular image information across viewpoints, and adaptively utilizes the uneven prior information from cross viewpoints to obtain consistent complementary information through the neighborhood aggregation framework to improve the super-resolution performance of stereo images. At the same time, a residual feature fusion module is also proposed to fuse the residual information of different viewpoints and the high- and low-resolution residual information to preserve more high-frequency details.

[0137] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0138] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0139] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0140] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0141] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0142] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0143] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0144] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0145] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A binocular image super-resolution method based on graph neural network, characterized in that: include: Input the low-resolution binocular image into the feature extraction network to generate initial features; Inputting the initial features into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution features and low-resolution features; The first residual feature and the second residual feature are fused and input into a reconstruction network to obtain a reconstruction feature; Outputting a super-resolution image according to the reconstruction features; The binocular interaction network includes a viewpoint twisting module, a binocular graph neural network module and a residual feature fusion module; The step of inputting the initial features into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution features and low-resolution features includes: Inputting the initial feature into the viewpoint twisting module to perform viewpoint twisting processing to obtain a first result; After the viewpoint is twisted, the first result is filtered and updated by a binocular image neural network module to obtain a second result; Extracting high-frequency information from the cross-viewpoint residual features and the high-resolution and low-resolution residual features in the second result through the residual feature fusion module to determine the first residual feature and the second residual feature; After the viewpoint is twisted, the first result is filtered and updated by a binocular image neural network module to obtain a second result, including: Constructing a first image and a second image on a left viewpoint and a right viewpoint respectively; In the first image and the second image, the left viewpoint is used as a reference viewpoint, and a corresponding pixel block is selected on the reference viewpoint as a query block; Taking all pixel blocks in the window centered on the query block as candidate nodes; Determine the correlation between each candidate node and the query block by using the Euclidean distance; Connecting the query block to the target node with the highest correlation; Determine a corresponding edge according to a difference between the target node and the query block, and complete the construction of the first graph and the second graph; Alternatingly updating the first map and the second map; Obtain reference viewpoint features and cross-viewpoint features from the last layer of the binocular image neural network; Aggregate the features of each connected node and update the aggregation results through the graph convolutional network; Each corresponding edge is converted into a weight to complete the propagation and node update between the first graph and the second graph.

2. The binocular image super-resolution method based on graph neural network according to claim 1, characterized in that: The step of inputting the low-resolution binocular image into a feature extraction network to generate initial features includes: Inputting the low-resolution binocular image into a feature extraction network based on weight sharing of four RDBs to generate local features of different viewpoints; The local features of the different viewpoints are subjected to convolution processing to obtain the initial features.

3. The binocular image super-resolution method based on graph neural network according to claim 1, characterized in that: In the step of inputting the initial features into the viewpoint twisting module to perform viewpoint twisting processing to obtain the first result, the expression for twisting the right viewpoint to the left viewpoint is: Among them, F r→l Represents the twist feature from the right viewpoint to the left viewpoint; M r→l represents the disparity attention map; represents batch matrix multiplication operation; H conv (.) represents 1×1 convolution; F r Represents the initial features of the right view obtained by the feature extraction network.

4. The binocular image super-resolution method based on graph neural network according to claim 1, characterized in that: The residual feature fusion module consists of convolution and deconvolution; The feature fusion module calculates the residual features between different viewpoints based on the complementary information between different viewpoints; Among them, the expression of the first residual feature is: Among them, F i ′ n Represents the first residual feature between different viewpoints; H rb represents the residual block; Represents upsampling the reference viewpoint features with a scale factor of 2; Represents upsampling of the cross-view features with a scale factor of 2; The expression of the second residual feature is: F fusion =(F′ in ↓2-F l )↑2+F′ in Among them, F fusion represents the second residual feature; F′ in ↓2 represents the downsampling of the first residual features between different viewpoints with a scale factor of 2; F l represents the initial features of the left view obtained by the feature extraction network; F′ in Represents the residual features between different viewpoints.

5. The binocular image super-resolution method based on graph neural network according to claim 1, characterized in that: The expression of the reconstruction feature is: F SR =(α0F l +α1H rb (H FEN (F fusion )))↑2 Among them, F SR represents the reconstruction feature; α0 and α1 are adaptive learnable parameters, H FEN (·) represents the feature extraction network; F l represents the initial features of the left view obtained by the feature extraction network; H rb represents the residual block; F fusion Represents the second residual feature.

6. A device for implementing the binocular image super-resolution method based on graph neural network as described in any one of claims 1 to 5, characterized in that: include: The first module is used to input the low-resolution binocular image into the feature extraction network to generate initial features; A second module is used to input the initial features into a binocular interaction network to determine a first residual feature between different viewpoints and a second residual feature between high-resolution features and low-resolution features; A third module is used to fuse the first residual feature and the second residual feature and input them into a reconstruction network to obtain a reconstruction feature; The fourth module is used to output a super-resolution image according to the reconstruction features.

7. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images