Image super-resolution methods, electronic devices and chips

Abstract

This application discloses an image super-resolution method, electronic device, and chip. The image super-resolution method includes: training an initial upsampling module and an initial downsampling module based on a training dataset to obtain a target upsampling module and a target downsampling module; constructing an initial model by stacking the target upsampling module and the target downsampling module; training the initial model based on the training dataset to obtain an image super-resolution model; and using the image super-resolution model to perform image super-resolution processing on the image to be processed to obtain a super-resolution image corresponding to the image to be processed.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and in particular to an image super-resolution method, electronic device, and chip. Background Technology

[0002] Image super-resolution technology, compared to traditional image interpolation algorithms, often provides greater zoom magnification and better image quality, and has been widely adopted in recent years.

[0003] Currently, techniques such as Neural Architecture Search (NAS) and Differentiable Architecture Search (DARTS) are often used to construct network structures for achieving image super-resolution.

[0004] However, due to the lack of significant differences between the weights of the architecture parameters, it is impossible to reasonably select different operators, thus failing to obtain an ideal network structure and reducing the image super-resolution effect. Summary of the Invention

[0005] This application provides an image super-resolution method, electronic device, and chip that can obtain a more ideal network structure, thereby achieving better image effects when performing image super-resolution.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide an image super-resolution method, the method comprising:

[0008] The initial upsampling module and the initial downsampling module are trained based on the training dataset to obtain the target upsampling module and the target downsampling module.

[0009] An initial model is constructed through the stacking of the target upsampling module and the target downsampling module;

[0010] The initial model is trained based on the training dataset to obtain an image super-resolution model;

[0011] The image super-resolution model is used to perform image super-resolution processing on the image to be processed, thereby obtaining the super-resolution image corresponding to the image to be processed.

[0012] Secondly, embodiments of this application provide an electronic device, the electronic device comprising: an acquisition unit, a construction unit, and...

[0013] The acquisition unit is used to train the initial upsampling module and the initial downsampling module respectively based on the training dataset to obtain the target upsampling module and the target downsampling module;

[0014] The construction unit is used to construct an initial model through the stacking process of the target upsampling module and the target downsampling module;

[0015] The acquisition unit is further configured to train the initial model based on the training dataset to obtain an image super-resolution model; and to use the image super-resolution model to perform image super-resolution processing on the image to be processed to obtain a super-resolution image corresponding to the image to be processed.

[0016] Thirdly, embodiments of this application provide an electronic device, which includes a processor and a memory storing processor-executable instructions. When the instructions are executed by the processor, the image super-resolution method described above is implemented.

[0017] Fourthly, embodiments of this application provide a chip, the chip including programmable logic circuits and / or program instructions, which, when the chip is running, implements the image super-resolution method as described above.

[0018] This application provides an image super-resolution method, electronic device, and chip. The electronic device trains an initial upsampling module and an initial downsampling module based on a training dataset to obtain a target upsampling module and a target downsampling module. An initial model is constructed by stacking the target upsampling and downsampling modules. The initial model is then trained based on the training dataset to obtain an image super-resolution model. This model is then used to perform image super-resolution processing on the image to be processed, resulting in a super-resolution image corresponding to the image to be processed. Therefore, in this application, operators in the search space can be selected based on the evolution trend of the architecture parameters, thereby achieving automatic searching of the upsampling and downsampling modules. Furthermore, stacking the upsampling and downsampling modules can yield a better network structure for image super-resolution, resulting in better image quality during image super-resolution. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating the effect of super-resolution.

[0020] Figure 2 This is a schematic diagram of the attention-residual UNET subnetwork structure based on a superkernel;

[0021] Figure 3 A diagram illustrating the search process for DARTS;

[0022] Figure 4A schematic diagram illustrating the evolution trend of architectural parameters;

[0023] Figure 5 A schematic diagram of the implementation process of the image super-resolution method. Figure 1 ;

[0024] Figure 6 A diagram illustrating the setting of initial weights;

[0025] Figure 7 This is a schematic diagram of the network structure of the upsampling (downsampling) module;

[0026] Figure 8 This is a schematic diagram of the network structure of an image super-resolution model;

[0027] Figure 9 This is a schematic diagram of the PixelShuffle operator operation process;

[0028] Figure 10 This is a schematic diagram illustrating the implementation of image super-resolution.

[0029] Figure 11 A schematic diagram of the implementation process of the image super-resolution method. Figure 2 ;

[0030] Figure 12 A schematic diagram of the implementation process of the image super-resolution method. Figure 3 ;

[0031] Figure 13 A schematic diagram of the implementation process of the image super-resolution method. Figure 4 ;

[0032] Figure 14 Schematic diagram of the composition structure of an electronic device Figure 1 ;

[0033] Figure 15 Schematic diagram of the composition structure of an electronic device Figure 2 . Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.

[0035] In recent years, with the widespread adoption of high-definition devices, the resolution of terminal display devices has generally increased to 2K (e.g., 2560x1440) or even higher. As a key technology for making blurry images clearer, image super-resolution (SR) technology has wide applications in various fields such as games, movies, cameras, and medical imaging. Through image super-resolution technology, high-definition reconstruction of multimedia content can be completed without consuming a large amount of resources. Designers only need to make simple and minor modifications to achieve results comparable to manual design, greatly simplifying the workflow and reducing costs.

[0036] On the other hand, image super-resolution technology is also widely used in camera shooting. Among these applications, the camera's zoom capability, as a crucial indicator of mobile phone shooting performance, is often highly valued by users. It can generally be divided into two parts: optical zoom and digital zoom. Specifically, optical zoom adjusts the focal length by adjusting the lens; however, due to limitations in device size, its adjustment capabilities are relatively limited. In contrast, digital zoom uses algorithms to adjust the image to simulate optical zoom. The quality of the algorithm largely determines the magnification and the quality of the digital zoom result.

[0037] Image super-resolution technology, compared to traditional image interpolation algorithms, often provides greater zoom magnification and better image quality, and has been widely adopted in recent years. Figure 1 This is a schematic diagram illustrating the effect of super-resolution, such as... Figure 1 As shown, the local area within the red frame in the image remains clear even after digital zoom.

[0038] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. It is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Its applications span all areas of artificial intelligence, primarily using induction and synthesis rather than deduction. Some common machine learning tasks include: classification, missing input classification, regression, transcription, machine translation, structured output, anomaly detection, synthesis and sampling, missing value imputation, denoising, density estimation, etc.

[0039] Neural Architecture Search (NAS) is a hot topic in the field of Automated Machine Learning (Auto-ML). By designing cost-effective search methods, it automatically acquires neural networks with strong generalization capabilities and low hardware requirements, greatly liberating researchers' creativity. Classical NAS methods mainly include three aspects: search space, search strategy, and evaluation prediction.

[0040] Figure 2 This is a schematic diagram of the attention-residual UNET subnetwork structure based on a superkernel, as shown below. Figure 2 As shown, the NAS-based image super-resolution method can employ a U-shaped network structure (UNET) and a channel attention mechanism. It searches for the optimal convolutional kernel size and number of channels using a neural network search approach. UNET is a deep learning framework that typically includes encoding and decoding processes. Specifically, Figure 2 The illustrated hyperkernel-based attention-residual UNET subnetwork is an improvement on the UNET architecture, adding an attention mechanism and a hyperkernel-based densely connected residual block (SK-DCRB). Each encoder layer has the same structure. Specifically, it consists of convolutional layers composed of ReLU activation functions and SK-DCR blocks, with a kernel size k=3. Similar to the standard UNET architecture, the output of the encoder layer is passed to the decoder layer in the same spatial layer. Each decoder layer receives the output of the skipped connections and the result of the previous layer. Unlike the UNET architecture, a channel attention block (CAB) is applied after the concatenation operation in each decoder layer. This can be downsampled using convolutions with a stride of 2 and upsampled using shuffle / subpixel layers.

[0041] However, the aforementioned techniques employ relatively complex channel attention structures, which are difficult to implement in certain applications, such as application-specific integrated circuits (ASICs). Furthermore, since the technique searches in two dimensions—kernel size and number of channels—it cannot guarantee that all operators in the preset network structure are optimal.

[0042] Differentiable Architecture Search (DARTS) solves the neural network search problem by relaxing the discrete search space, allowing the network search to be optimized using gradient updates. This method has been widely studied and used in the field due to its high efficiency. DARTS represents the network space as a directed acyclic graph. The key is that node connections and activation functions are cleverly combined into a matrix, where each element represents the weight of the connection and activation function. A softmax function is used during the search, transforming the search space into a continuous space and making the objective function differentiable. During the search, DARTS traverses all nodes, using the weighted average of all connections on each node for computation, while simultaneously optimizing the structural weights and network weights. After the search, the connection and activation function with the largest weights are selected to form the final network.

[0043] Figure 3 A diagram illustrating the search process for DARTS, as shown below. Figure 3 As shown in Figure (a), firstly, each node in the module is defined as a feature map, and the nodes are connected through algorithmic operations. Simultaneously, the number of nodes in each module needs to be determined (e.g., 4 nodes: node 0, node 1, node 2, and node 3), and the operator types in the search space are defined, such as Conv3×3 and Max pooling. Secondly, in Figure (b), based on the operators given in the search space, each operator is assigned an initial value for its connection weight parameters, the connections between nodes are determined, and a directed acyclic graph is constructed to represent the module structure. Next, in Figure (c), the weight parameters corresponding to each operator are updated using a gradient update strategy based on the architecture parameters. Finally, in Figure (d), after all parameters converge, the operator with the largest connection weight between each pair of nodes is selected as the operator in the final network. In this way, the required network structure is found. NAS-UNET uses the above DARTS search method to search for the upsampling and downsampling module structures in the UNET structure.

[0044] However, the architecture parameter update method used in NAS-UNET has a problem when applied to image super-resolution networks: the differences between the convergent architecture parameter weights are not very significant, making it difficult to select operators. For example, Figure 4 A schematic diagram illustrating the evolution trend of architectural parameters, such as Figure 4 As shown, the weight coefficients corresponding to each operator converge to 0.167, indicating that the network has difficulty in choosing between operators.

[0045] In other words, current machine learning techniques cannot achieve an ideal network structure, which in turn prevents the attainment of better image quality when using neural networks for image super-resolution.

[0046] To address the aforementioned issues, in the embodiments of this application, the terminal can select operators in the search space based on the evolution trend diagram of the architecture parameters, thereby enabling automatic searching of the upsampling and downsampling modules. Furthermore, by stacking the upsampling and downsampling modules, a better network structure for achieving image super-resolution can be obtained, resulting in better image performance during image super-resolution.

[0047] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0048] One embodiment of this application provides an image super-resolution method. Figure 5 A schematic diagram of the implementation process of the image super-resolution method. Figure 1 ,like Figure 5 As shown in the embodiments of this application, the method for an electronic device to achieve image super-resolution may include the following steps:

[0049] Step 101: Train the initial upsampling module and the initial downsampling module respectively based on the training dataset to obtain the target upsampling module and the target downsampling module.

[0050] In the embodiments of this application, the electronic device can first train an initial upsampling module based on a training dataset to obtain a corresponding target upsampling module, and at the same time, it can train an initial downsampling module based on a training dataset to obtain a corresponding target downsampling module.

[0051] It is understood that, in the embodiments of this application, the electronic device is any terminal device with communication and storage functions, such as: tablet computer, mobile phone, e-reader, remote control, personal computer (PC), laptop computer, in-vehicle device, smart TV, wearable device, personal digital assistant (PDA), portable media player (PMP), navigation device and other terminal devices.

[0052] Furthermore, in embodiments of this application, the training dataset may include high-resolution images and low-resolution images that have a one-to-one correspondence.

[0053] It should be noted that, in the embodiments of this application, the initial upsampling module can be used to perform image upsampling processing, specifically consisting of multiple different network nodes and multiple different operators. Each of the multiple different network nodes can be defined as a feature map, and every two network nodes are connected through algorithmic operations, i.e., connected through different operators.

[0054] Accordingly, in the embodiments of this application, the initial downsampling module can be used to perform image downsampling processing, specifically consisting of multiple different network nodes and multiple different operators. Each of the multiple different network nodes can be defined as a feature map, and every two network nodes are connected through algorithmic operations, i.e., connected through different operators.

[0055] Furthermore, in the embodiments of this application, for every two network nodes in the initial upsampling module, each operator connecting the two network nodes can be set with a corresponding first initial weight, wherein the first initial weight can be used to predict the processing effect of each operator connecting two different network nodes.

[0056] Accordingly, in the embodiments of this application, for every two network nodes in the initial downsampling module, each operator connecting the two network nodes can be set with a corresponding second initial weight, wherein the second initial weight can be used to predict the processing effect of each operator connecting two different network nodes.

[0057] It is understood that, in the embodiments of this application, the first initial weight and the second initial weight can be represented as natural numbers greater than 0 and less than 1.

[0058] It should be noted that, in the embodiments of this application, for multiple operators between two different network nodes, the electronic device can set the corresponding multiple first initial weights to the same value; correspondingly, for multiple operators between two different network nodes, the electronic device can set the corresponding multiple second initial weights to the same value.

[0059] For example, Figure 6 A diagram illustrating the setting of initial weights, as shown below. Figure 6 As shown, for three different operators between network node a and network node b in the upsampling module (or downsampling module), the electronic device can set the corresponding three first initial weights (or second initial weights) to p, where p is greater than 0 and less than 1. That is, the initial weights corresponding to operator 1, operator 2, and operator 3 are all p.

[0060] Furthermore, in the embodiments of this application, during the training of the initial upsampling module based on the training dataset, the electronic device can continuously correct and adjust the first initial weights, thereby determining the first adjusted weights corresponding to each first initial weight, and finally determining the target upsampling module that has completed training based on the first adjusted weights.

[0061] Accordingly, in the embodiments of this application, during the training of the initial downsampling module based on the training dataset, the electronic device can continuously correct and adjust the second initial weights, thereby determining the second adjusted weights corresponding to each second initial weight, and finally determining the target downsampling module that has completed training based on the second adjusted weights.

[0062] It should be noted that, in the embodiments of this application, the target upsampling module can be used to perform image upsampling processing, wherein each pair of network nodes in the target upsampling module can be connected by one or more operators.

[0063] Accordingly, in the embodiments of this application, the target downsampling module can be used to perform image downsampling processing, wherein each pair of network nodes in the target downsampling module can be connected by one or more operators.

[0064] In other words, in this application, the target upsampling module and target downsampling module obtained after training based on the training dataset do not limit the number of operators between every two network nodes to one, but can have at least one operator.

[0065] Furthermore, in the embodiments of this application, Figure 7 This is a schematic diagram of the network structure of the upsampling (downsampling) module, such as... Figure 7 As shown, the upsampling module (or downsampling module) can include two inputs (input 0 and input 1). For the downsampling module, one input can be the feature map data output from the first-layer downsampling module, and the other input can be the feature map data output from the previous-layer downsampling module. For the upsampling module, one input can be the feature map data output from the previous-layer upsampling module, and the other input can be the output after performing an identity mapping on the feature map data output from the corresponding downsampling module. Finally, the operation results corresponding to different operators are concatenated and then output.

[0066] Step 102: Construct the initial model by stacking the target upsampling module and the target downsampling module.

[0067] In the embodiments of this application, after the electronic device trains the initial upsampling module and the initial downsampling module based on the training dataset to obtain the target upsampling module and the target downsampling module, it can further construct the initial model by stacking the target upsampling module and the target downsampling module.

[0068] It should be noted that, in the embodiments of this application, when constructing the initial model, the electronic device may choose to use a combination of downsampling operators, a combination of upsampling operators, at least one target upsampling module, and at least one target downsampling module to determine the initial model to be trained. Specifically, the electronic device may first stack at least one target upsampling module and at least one target downsampling module to obtain a stacked network structure; then, the combination of downsampling operators and the combination of upsampling operators are added to the stacked network structure to complete the construction of the initial model.

[0069] It is understood that, in the embodiments of this application, after the electronic device completes the construction of the target upsampling module and the target downsampling module, it can rebuild a network structure based on the target upsampling module and the target downsampling module by stacking them multiple times.

[0070] Furthermore, in the embodiments of this application, when constructing the initial model, the electronic device can also select to use different combinations of operators to add to the head and tail of the network structure constructed based on the target upsampling module and the target downsampling module, respectively, to finally complete the construction of the initial model. For example, a combination of downsampling operators is used at the head of the network structure to reduce the size of the feature map, and a combination of upsampling operators is used at the tail of the network structure to increase the size of the feature map.

[0071] It should be noted that, in the embodiments of this application, the combination of downsampling operators may include the Conv operator and the Activation operator; the combination of upsampling operators may include the Conv operator, the Activation operator, and the PixelShuffle operator.

[0072] Step 103: Train the initial model based on the training dataset to obtain the image super-resolution model.

[0073] In the embodiments of this application, after the electronic device constructs an initial model based on the target upsampling module and the target downsampling module, it can train the initial model based on the training dataset and finally obtain the image super-resolution model.

[0074] It is understood that, in the embodiments of this application, the image super-resolution model can be used to perform image super-resolution processing.

[0075] Furthermore, in the embodiments of this application, after the electronic device completes the construction of the initial model by stacking the target upsampling module and the target downsampling module, and adding the combination of upsampling operators and the combination of downsampling operators, it can use the training dataset to train the initial model and obtain the corresponding image super-resolution model.

[0076] It should be noted that, in the embodiments of this application, the training dataset for training the initial model may also include a one-to-one correspondence between high-resolution images and low-resolution images. The electronic device can use the same training dataset to train the initial upsampling module, the initial downsampling module, and the initial model respectively.

[0077] Figure 8 This is a schematic diagram of the network structure of an image super-resolution model, such as... Figure 8 As shown, taking a stack of four target upsampling modules and four target downsampling modules as an example, the network structure of an image super-resolution model for image super-resolution processing mainly includes two parts: the encoder on the left and the decoder on the right. The encoder includes a downsampling module, and the decoder includes an upsampling module. The feature map data from each downsampling module undergoes identity mapping before being input into the corresponding upsampling module. In other words, each identity mapping process selectively reuses channels of the feature map data from the encoder and feeds them into the upsampling module in the decoder for upsampling, thereby improving the reconstruction effect of the super-resolution image.

[0078] Furthermore, in the embodiments of this application, such as Figure 8As shown, the network structure of the image super-resolution model for image super-resolution processing can also include a combination of upsampling operators and a combination of downsampling operators. The downsampling operator combination is located before the first downsampling module, and the upsampling operator combination is located after the first upsampling module. The downsampling operator combination is used to decrease the input feature map size by a factor of 2, and the upsampling operator combination is used to increase the input feature map size by a factor of 2, as well as upsampling by a factor of r (r > 0). For example, in the beginning and end of the UNET network, a downsampling operator combination is formed by a convolution operator with a stride of 2 and an activation function (Leaky ReLU), while an upsampling operator combination is formed by a convolution operator with a stride of 2, an activation function (Leaky ReLU), and a pixel shuffle operator (PixelShuffle), achieving a 2-fold decrease and a 2-fold increase in the input feature map size. The PixelShuffle operator can achieve an upsampling of r... 2 A feature map tensor of size ×h×w is mapped to a feature map tensor of size rw×rh, thereby achieving r-fold upsampling of the image.

[0079] Figure 9 This is a schematic diagram of the PixelShuffle operator operation process, as shown below. Figure 9 As shown, a low-resolution (H×W) input image is transformed into a high-resolution (rH×rW) image using the PixelShuffle operation. However, the implementation does not directly generate this high-resolution image through interpolation or other methods; instead, it first obtains r through convolution. 2 The feature maps of each channel (the feature map size is the same as the input low-resolution image size) are then processed by periodic shuffling to output a high-resolution image, where r is the upscaling factor, which is the magnification factor of the image.

[0080] Step 104: Use an image super-resolution model to perform image super-resolution processing on the image to be processed, and obtain the super-resolution image corresponding to the image to be processed.

[0081] In the embodiments of this application, after the electronic device trains an initial model based on the training dataset and obtains an image super-resolution model, it can use the image super-resolution model to perform image super-resolution processing. Specifically, it can perform image super-resolution processing on the image to be processed based on the image super-resolution model, and finally obtain the super-resolution image corresponding to the image to be processed.

[0082] It should be noted that, in the embodiments of this application, the image to be processed and the corresponding super-resolution image have different image sizes. For example, the image size of the image to be processed is H×W, and the image size of the corresponding super-resolution image is 2H×2W.

[0083] Figure 10 This is a schematic diagram illustrating the implementation of image super-resolution, as shown below. Figure 10 As shown, taking a stack of four target upsampling modules and four target downsampling modules as an example, the image to be processed (H×W) is reduced by a factor of two after a combination of downsampling operators. Then, it is sequentially passed through four layers of target downsampling modules in the encoding section to extract image features. The feature map data output from the last layer of target downsampling modules undergoes identity mapping and is input into the last layer of target upsampling modules. Then, it is sequentially passed through four layers of target upsampling modules in the decoding section to extract image features. Finally, after a combination of upsampling operators to increase the image size by a factor of two, the super-resolution image (2H×2W) corresponding to the image to be processed is obtained. In this case, the feature map data output from each layer of target downsampling modules undergoes identity mapping and is input into the corresponding layer of target upsampling modules. That is, each layer of identity mapping selectively multiplexes the channels of the feature map data from the encoding section and sends them to the target upsampling modules in the decoding section for upsampling, thereby improving the reconstruction effect of the super-resolution image.

[0084] In summary, the image super-resolution method proposed in steps 101 to 104 provides a novel and relatively simple search method to determine the upsampling and downsampling modules. This allows for the stacking of these modules to obtain a network structure with better image super-resolution performance. Furthermore, during the training of the upsampling and downsampling modules, operators in the search space can be selected based on the evolution trend of the architecture parameters, thus enabling automatic searching of the upsampling and downsampling modules. This approach achieves a super-resolution network model with superior performance compared to manually designed networks.

[0085] It is understood that, in the embodiments of this application, due to the automatic search of the upsampling module and the downsampling module, the UNET structure has fewer parameters compared to the manually designed UNET structure, and the automatically searched network structure has higher versatility and portability.

[0086] In other words, the image super-resolution method proposed in this application determines the upsampling and downsampling modules in the super-resolution network structure based on observing the evolution trend of architectural parameters, thereby obtaining a better super-resolution network structure. The method is simple, easy to implement, and can achieve relatively ideal image super-resolution results. At the same time, the searched network model can reduce the computational load of the network to a certain extent while ensuring high performance.

[0087] This application provides an image super-resolution method. An electronic device trains an initial upsampling module and an initial downsampling module based on a training dataset to obtain a target upsampling module and a target downsampling module. An initial model is constructed by stacking the target upsampling and downsampling modules. The initial model is then trained based on the training dataset to obtain an image super-resolution model. This model is then used to perform image super-resolution processing on the image to be processed, resulting in a super-resolution image. Therefore, in this application, operators in the search space can be selected based on the evolution trend of the architecture parameters, thereby achieving automatic searching of the upsampling and downsampling modules. Furthermore, stacking the upsampling and downsampling modules can yield a better network structure for image super-resolution, resulting in better image quality during image super-resolution.

[0088] Based on the above embodiments, in another embodiment of this application, Figure 11 A schematic diagram of the implementation process of the image super-resolution method. Figure 2 ,like Figure 11 As shown, before training the initial upsampling module and the initial downsampling module based on the training dataset to obtain the target upsampling module and the target downsampling module, i.e. before step 101, the method for achieving image super-resolution using an electronic device may further include the following steps:

[0089] Step 105: Construct an initial upsampling module and an initial downsampling module based on multiple network nodes and multiple operators respectively.

[0090] Step 106: For every two network nodes in the initial upsampling module, determine the first initial weight corresponding to each operator.

[0091] Step 107: For every two network nodes in the initial downsampling module, determine the second initial weight corresponding to each operator.

[0092] In embodiments of this application, the electronic device can first construct an initial upsampling module and an initial downsampling module using multiple network nodes and multiple operators, respectively. Then, a first initial weight corresponding to each operator between every two network nodes in the initial upsampling module can be determined, and a second initial weight corresponding to each operator between every two network nodes in the initial downsampling module can be determined.

[0093] Furthermore, in the embodiments of this application, the initial upsampling module can be composed of multiple network nodes, wherein different network nodes can be connected by various types of operators. For example, the initial upsampling module can be composed of 4 network nodes and 3 types of operators, and the different network nodes and different types of operators together constitute the search space of the initial upsampling module.

[0094] Accordingly, in the embodiments of this application, the initial downsampling module can be composed of multiple network nodes, wherein different network nodes can be connected by various types of operators. For example, the initial downsampling module can be composed of 4 network nodes and 3 types of operators, and the different network nodes and different types of operators together constitute the search space of the initial downsampling module.

[0095] It should be noted that, in the embodiments of this application, an initial weight value can be set for each operator in the initial upsampling module, that is, a first initial weight corresponding to each operator in the initial upsampling module can be preset. Correspondingly, an initial weight value can also be set for each operator in the initial downsampling module, that is, a second initial weight corresponding to each operator in the initial downsampling module can be preset.

[0096] It is understood that, in the embodiments of this application, when setting the first initial weight and the second initial weight, the electronic device can set all the different first initial weights corresponding to different operators in the initial upsampling module to the same value. Correspondingly, the electronic device can also set all the different first initial weights corresponding to different operators in the initial upsampling module to the same value.

[0097] Furthermore, in the embodiments of this application, after the electronic device completes the construction of the initial upsampling module and the initial downsampling module, the electronic device needs to first define the objective function, and then use the objective function to complete the training of the initial upsampling module and the initial downsampling module respectively, and finally obtain the target upsampling module and the target downsampling module.

[0098] It should be noted that the objective function can be defined in various ways in the embodiments of this application. For example, the optimization objective function with energy consumption constraints can be defined as argminL1, where L1 is the norm loss term, that is, the network loss is measured by the 1 norm between the network output and the given true value.

[0099] It is understood that, in the embodiments of this application, performance metrics can be predefined during the training of the initial upsampling module and the initial downsampling module. For example, objective metrics for evaluating network performance can be Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM).

[0100] Furthermore, in the embodiments of this application, Figure 12 A schematic diagram of the implementation process of the image super-resolution method. Figure 3 ,like Figure 12 As shown, the method for obtaining the target upsampling module and the target downsampling module by training the initial upsampling module and the initial downsampling module respectively based on the training dataset may include the following steps:

[0101] Step 101a: Input the training dataset into the initial upsampling module to obtain the first output value.

[0102] Step 101b: Determine the first loss parameter based on the objective function and the first output value.

[0103] Step 101c: Adjust the first initial weights according to the first loss parameters to obtain the first adjusted weights.

[0104] Step 101d: Determine the target upsampling module based on the first adjusted weight.

[0105] In the embodiments of this application, after the electronic device completes the construction of the initial upsampling module, it can train the initial upsampling module based on the training dataset and finally determine the target upsampling module after training.

[0106] It is understood that, in the embodiments of this application, during training, the electronic device can input the training dataset into the initial upsampling module and then output a first output value. After obtaining the first output value, the electronic device can further determine the corresponding first loss parameter using a predefined objective function and the first output value. Then, the electronic device can adjust the first initial weight of any operator in the initial upsampling module according to the first loss parameter to obtain the corresponding first adjusted weight, and finally determine the corresponding target upsampling module based on the first adjusted weight.

[0107] It should be noted that, in the embodiments of this application, the process of training the initial upsampling module is the process of continuously adjusting the weight value (architecture parameter) corresponding to each operator. Any optimization method can be used to update and optimize the weight value (architecture parameter) corresponding to each operator until the weight values ​​of all operators converge.

[0108] It is understood that, in the embodiments of this application, in addition to the weight values ​​(architectural parameters) corresponding to the operators in the initial upsampling module needing to be continuously adjusted during the training process, the network parameters of the initial upsampling module also need to be continuously updated and optimized until all network parameters converge.

[0109] Furthermore, in the embodiments of this application, when determining the target upsampling module based on the first adjusted weight, the electronic device can first determine the weight change trend corresponding to each operator based on the first initial weight and the first adjusted weight; then, for every two network nodes in the initial upsampling module, operators with decreasing weight change trends are removed, and finally the target upsampling module can be obtained.

[0110] It should be noted that, in the embodiments of this application, after determining that all network parameters of the initial upsampling module and the weight values ​​of all operators have converged, in order to obtain the final target upsampling module, the electronic device may select operators based on the changing trend of the weight values ​​of each operator.

[0111] Furthermore, in the embodiments of this application, for all operators in the initial upsampling module, the electronic device may choose to retain operators whose weight change trend is increasing, while removing operators whose weight change trend is decreasing, thereby determining the target upsampling module after training is completed.

[0112] In other words, in the embodiments of this application, operators with upward-trending architecture parameters can be considered as operators that are more friendly to the current task. Therefore, all operators with upward-trending architecture parameters can be selected as the final operators to be retained. At the same time, operators with non-increasing architecture parameters can be directly discarded.

[0113] It should be noted that, in the embodiments of this application, if multiple operators are retained between two network nodes in the initial upsampling module, then multiple operation results corresponding to the multiple operators are determined; the multiple operation results are spliced ​​to obtain a first spliced ​​result; and the first spliced ​​result is input to the latter of the two network nodes.

[0114] It is understood that, in the embodiments of this application, after removing operators whose weight change trend is decreasing, if multiple operators whose weight change trend is increasing are retained between any two network nodes in the initial upsampling module, the electronic device can sequentially determine the multiple operation results corresponding to the multiple operators, and then concatenate the multiple operation results to obtain the corresponding first concatenated result. Finally, the first concatenated result can be determined as the input of the next network node, that is, the first concatenated result is input to the next network node among the two network nodes.

[0115] In other words, in the embodiments of this application, if multiple operators are retained between two connected network nodes at the same time, the operation results of these operators can be output to the next network node by image stitching.

[0116] Furthermore, in the embodiments of this application, Figure 13 A schematic diagram of the implementation process of the image super-resolution method. Figure 4 ,like Figure 13 As shown, the method for obtaining the target upsampling module and the target downsampling module by training the initial upsampling module and the initial downsampling module respectively based on the training dataset may include the following steps:

[0117] Step 101e: Input the training dataset into the initial downsampling module to obtain the second output value.

[0118] Step 101f: Determine the second loss parameter based on the objective function and the second output value.

[0119] Step 101g: Adjust the second initial weights according to the second loss parameters to obtain the second adjusted weights.

[0120] Step 101h: Determine the target downsampling module based on the second adjusted weight.

[0121] In the embodiments of this application, after the electronic device completes the construction of the initial downsampling module, it can train the initial downsampling module based on the training dataset and finally determine the target downsampling module after training.

[0122] It is understood that, in the embodiments of this application, during training, the electronic device can input the training dataset into the initial downsampling module and then output a second output value. After obtaining the second output value, the electronic device can further determine the corresponding second loss parameter using a predefined objective function and the second output value. Then, the electronic device can adjust the second initial weight of any operator in the initial downsampling module according to the second loss parameter to obtain the corresponding second adjusted weight. Finally, the corresponding target downsampling module can be determined based on the second adjusted weight.

[0123] It should be noted that, in the embodiments of this application, the process of training the initial downsampling module is the process of continuously adjusting the weight value (architecture parameter) corresponding to each operator. Any optimization method can be used to update and optimize the weight value (architecture parameter) corresponding to each operator until the weight values ​​of all operators converge.

[0124] It is understood that, in the embodiments of this application, in addition to the weight values ​​(architectural parameters) corresponding to the operators in the initial downsampling module needing to be continuously adjusted during the training process, the network parameters of the initial downsampling module also need to be continuously updated and optimized until all network parameters converge.

[0125] Furthermore, in the embodiments of this application, when determining the target downsampling module according to the second adjusted weight, the electronic device can first determine the weight change trend corresponding to each operator according to the second initial weight and the second adjusted weight; then, for every two network nodes in the initial downsampling module, the operators whose weight change trend is decreasing are removed, and finally the target downsampling module can be obtained.

[0126] It should be noted that, in the embodiments of this application, after determining that all network parameters of the initial downsampling module and the weight values ​​of all operators have converged, in order to obtain the final target downsampling module, the electronic device may select operators based on the changing trend of the weight values ​​of each operator.

[0127] Furthermore, in the embodiments of this application, for all operators in the initial downsampling module, the electronic device may choose to retain operators whose weight change trend is increasing, while removing operators whose weight change trend is decreasing, thereby determining the target downsampling module after training is completed.

[0128] In other words, in the embodiments of this application, operators with upward-trending architecture parameters can be considered as operators that are more friendly to the current task. Therefore, all operators with upward-trending architecture parameters can be selected as the final operators to be retained. At the same time, operators with non-increasing architecture parameters can be directly discarded.

[0129] It should be noted that, in the embodiments of this application, if multiple operators are retained between two network nodes in the initial downsampling module, then multiple operation results corresponding to the multiple operators are determined; the multiple operation results are concatenated to obtain a second concatenated result; and the second concatenated result is input to the latter of the two network nodes.

[0130] It is understood that, in the embodiments of this application, after removing operators whose weight change trend is decreasing, if multiple operators whose weight change trend is increasing are retained between any two network nodes in the initial downsampling module, the electronic device can sequentially determine the multiple operation results corresponding to the multiple operators, and then concatenate the multiple operation results to obtain the corresponding second concatenated result. Finally, the second concatenated result can be determined as the input of the next network node, that is, the second concatenated result is input to the next network node among the two network nodes.

[0131] In other words, in the embodiments of this application, if multiple operators are retained between two connected network nodes at the same time, the operation results of these operators can be output to the next network node by image stitching.

[0132] This application provides an image super-resolution method. An electronic device trains an initial upsampling module and an initial downsampling module based on a training dataset to obtain a target upsampling module and a target downsampling module. An initial model is constructed by stacking the target upsampling and downsampling modules. The initial model is then trained based on the training dataset to obtain an image super-resolution model. This model is then used to perform image super-resolution processing on the image to be processed, resulting in a super-resolution image. Therefore, in this application, operators in the search space can be selected based on the evolution trend of the architecture parameters, thereby achieving automatic searching of the upsampling and downsampling modules. Furthermore, stacking the upsampling and downsampling modules can yield a better network structure for image super-resolution, resulting in better image quality during image super-resolution.

[0133] Based on the above embodiments, another embodiment of this application provides an image super-resolution method that can automatically search for image super-resolution networks based on the evolution trend of architecture parameters, thereby constructing a more ideal image super-resolution network to obtain better image super-resolution results.

[0134] It is understood that the image super-resolution method proposed in this application can eliminate or retain operators based on the weight change trend of the weight value corresponding to each operator when constructing an image super-resolution model for image super-resolution, thereby realizing automatic network search. Compared with common manual search methods, it can obtain image super-resolution networks with better performance.

[0135] In other words, in practical applications, DARTS search typically selects the operator with the highest weight while discarding all others. This approach can lead to significant errors in network performance when the differences in operator weights are not obvious, resulting in suboptimal network performance. In contrast, the image super-resolution method proposed in this application can obtain a better image super-resolution network through automatic network search during the construction of the network for image super-resolution processing. Specifically, during the network search process, multiple operators can be retained to obtain upsampling and downsampling modules, and the image super-resolution network is constructed by stacking the upsampling and downsampling modules multiple times.

[0136] Furthermore, in the embodiments of this application, based on the above... Figure 8 Similar to NAS-UNET, this application aims to search for a downsampling module and an upsampling module. Then, based on the structures of the downsampling and upsampling modules obtained from the network architecture search, a soft-compressed network structure is reconstructed through multiple stacking operations. Finally, the compressed network is retrained on an existing dataset to obtain a final network model capable of achieving image super-resolution for inference. The implementation details of the downsampling and upsampling modules are as described above. Figure 7 As shown, this directed acyclic graph adopts the module construction method in NAS-UNET and DARTS.

[0137] It should be noted that, in the embodiments of this application, the feature image size can be reduced by a factor of 2 at the beginning of the softly compressed network structure using a combination of Conv / Activation operators (downsampling operators), and simultaneously increased by a factor of 2 at the end of the softly compressed network structure using a combination of Conv / Activation / PixelShuffle operators (upsampling operators). The PixelShuffle operator can achieve the reduction of r... 2 A feature map tensor of size ×h×w is mapped to a feature map tensor of size rw×rh, thereby achieving r-fold upsampling of the image.

[0138] Furthermore, in the embodiments of this application, the automatic search network framework structure mainly includes a downsampling module and an upsampling module. The two modules replace the Conv / ReLU / Pooling structure in the original UNET network encoding and the Conv / ReLU / Upsampling structure in the decoding, respectively, and are used to construct the backbone network for image super-resolution.

[0139] It should be noted that, in the embodiments of this application, in terms of search space design, the downsampling module and upsampling module to be searched in this application can be composed of multiple network nodes (such as four network nodes) and multiple different types of operators (such as upsampling operator, downsampling operator, and identity mapping operator). The combination of the above multiple network nodes and multiple operators constitutes the search space of the downsampling module and the upsampling module.

[0140] It is understood that, in the embodiments of this application, in order to find a better network structure, more lightweight alternative operators (such as channel shuffle) can be introduced in the search space design, thereby further reducing the number of network parameters and making the searched model structure more lightweight.

[0141] Furthermore, during the training of the downsampling and upsampling modules, it is also necessary to define the objective function. For example, in terms of searching for the objective function, the energy-constrained optimization objective function can be defined as argmin L1, which measures the network loss by the 1-norm between the network output and the given true value.

[0142] Furthermore, during the training of the downsampling and upsampling modules, performance metrics need to be defined. For example, during the search process, objective metrics for evaluating network performance can be Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity-to-Simple (SSIM).

[0143] It should be noted that, in the embodiments of this application, the training process of the downsampling module and the upsampling module is mainly a process of continuously updating and optimizing the network parameters and architecture parameters (weight values ​​corresponding to each operator) of the downsampling module and the upsampling module. For example, for the network parameters and architecture parameters contained in the entire network, the Adam optimization method can be used simultaneously to update and optimize these two sets of parameters until all parameters converge.

[0144] Furthermore, in the embodiments of this application, after determining that the network parameters and architecture parameters of the downsampling module and the upsampling module have converged, it is necessary to further determine the evolution trend of the architecture parameters (the weight change trend corresponding to each operator), and perform operator screening based on the evolution trend of the architecture parameters.

[0145] It is understood that, in the embodiments of this application, when filtering operators based on the evolution trend of architecture parameters, operators with increasing weight changes can be retained, while operators with decreasing weight changes can be removed, thereby completing the search of the downsampling module and the upsampling module.

[0146] Furthermore, in the embodiments of this application, during the construction of the model for achieving image super-resolution, the network structure can be reconstructed based on the downsampling module and upsampling module that retain multiple operators, and then the reconstructed network structure can be retrained to finally obtain a network model that can be used for inference and achieve image super-resolution.

[0147] It is understood that, in the embodiments of this application, based on the trained network model for achieving image super-resolution, the application can conduct experimental verification using an image super-resolution dataset to determine the generality and transferability of the network model. Alternatively, the network structures of the searched downsampling and upsampling modules can be simultaneously applied to other datasets; that is, the network model can be trained on such datasets, and the generality and transferability of the network search can be observed based on the model's inference results.

[0148] The image super-resolution method proposed in this application can determine the upsampling and downsampling modules through a novel and relatively simple search method. This allows for the stacking of the upsampling and downsampling modules to obtain a network structure with better performance in achieving image super-resolution. Specifically, during the training of the upsampling and downsampling modules, operators in the search space can be selected based on the evolution trend of the architecture parameters, thus achieving automatic search for the upsampling and downsampling modules. This enables the creation of a super-resolution network model with superior performance compared to manually designed networks.

[0149] It is understood that, in the embodiments of this application, due to the automatic search of the upsampling module and the downsampling module, the UNET structure has fewer parameters compared to the manually designed UNET structure, and the automatically searched network structure has higher versatility and portability.

[0150] In other words, the image super-resolution method proposed in this application determines the upsampling and downsampling modules in the super-resolution network structure based on observing the evolution trend of architectural parameters, thereby obtaining a better super-resolution network structure. The method is simple, easy to implement, and can achieve relatively ideal image super-resolution results. At the same time, the searched network model can reduce the computational load of the network to a certain extent while ensuring high performance.

[0151] This application provides an image super-resolution method. An electronic device trains an initial upsampling module and an initial downsampling module based on a training dataset to obtain a target upsampling module and a target downsampling module. An initial model is constructed by stacking the target upsampling and downsampling modules. The initial model is then trained based on the training dataset to obtain an image super-resolution model. This model is then used to perform image super-resolution processing on the image to be processed, resulting in a super-resolution image. Therefore, in this application, operators in the search space can be selected based on the evolution trend of the architecture parameters, thereby achieving automatic searching of the upsampling and downsampling modules. Furthermore, stacking the upsampling and downsampling modules can yield a better network structure for image super-resolution, resulting in better image quality during image super-resolution.

[0152] Based on the above embodiments, in another embodiment of this application... Figure 14 Schematic diagram of the composition structure of an electronic device Figure 1 ,like Figure 14 As shown, the electronic device 10 proposed in this application embodiment may include: an acquisition unit 11, a construction unit 12,

[0153] The acquisition unit 11 is used to train the initial upsampling module and the initial downsampling module respectively based on the training dataset to obtain the target upsampling module and the target downsampling module;

[0154] The construction unit 12 is used to construct an initial model through the stacking process of the target upsampling module and the target downsampling module;

[0155] The acquisition unit 11 is further configured to train the initial model based on the training dataset to obtain an image super-resolution model; and to use the image super-resolution model to perform image super-resolution processing on the image to be processed to obtain a super-resolution image corresponding to the image to be processed.

[0156] In the embodiments of this application, further, Figure 15 Schematic diagram of the composition structure of an electronic device Figure 2 ,like Figure 15 As shown, the electronic device 10 proposed in this application embodiment may further include a processor 13, a memory 14 storing instructions executable by the processor 13, and further, the electronic device 10 may also include a communication interface 15 and a bus 16 for connecting the processor 13, the memory 14 and the communication interface 15.

[0157] In the embodiments of this application, the processor 13 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other types, and this application embodiment does not specifically limit this. The electronic device 10 may also include a memory 14, which can be connected to the processor 13. The memory 14 is used to store executable program code, which includes computer operation instructions. The memory 14 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0158] In embodiments of this application, bus 16 is used to connect communication interface 15, processor 13 and memory 14 and the mutual communication between these devices.

[0159] In embodiments of this application, memory 14 is used to store instructions and data.

[0160] Furthermore, in the embodiments of this application, the processor 13 is configured to train an initial upsampling module and an initial downsampling module based on a training dataset to obtain a target upsampling module and a target downsampling module; construct an initial model through stacking the target upsampling module and the target downsampling module; train the initial model based on the training dataset to obtain an image super-resolution model; and use the image super-resolution model to perform image super-resolution processing on the image to be processed to obtain a super-resolution image corresponding to the image to be processed.

[0161] In practical applications, the aforementioned memory 14 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 13.

[0162] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0163] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] This application provides an electronic device that trains an initial upsampling module and an initial downsampling module based on a training dataset to obtain a target upsampling module and a target downsampling module. An initial model is constructed by stacking the target upsampling and downsampling modules. An image super-resolution model is then obtained by training the initial model based on the training dataset. The image super-resolution model is then used to perform image super-resolution processing on the image to be processed, resulting in a super-resolution image corresponding to the image to be processed. Therefore, in this application's embodiments, operators in the search space can be selected based on the evolution trend of the architecture parameters, thereby achieving automatic searching of the upsampling and downsampling modules. Furthermore, stacking the upsampling and downsampling modules can yield a better network structure for achieving image super-resolution, resulting in better image quality during image super-resolution.

[0165] This application provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the image super-resolution method described above.

[0166] Specifically, the program instructions corresponding to an image super-resolution method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to an image super-resolution method in the storage media are read or executed by an electronic device, the following steps are included:

[0167] The initial upsampling module and the initial downsampling module are trained based on the training dataset to obtain the target upsampling module and the target downsampling module.

[0168] An initial model is constructed through the stacking of the target upsampling module and the target downsampling module;

[0169] The initial model is trained based on the training dataset to obtain an image super-resolution model;

[0170] The image super-resolution model is used to perform image super-resolution processing on the image to be processed, thereby obtaining the super-resolution image corresponding to the image to be processed.

[0171] This application provides a chip, which includes programmable logic circuitry and / or program instructions. When the chip runs, it implements the image super-resolution method described above, specifically including the following steps:

[0172] The initial upsampling module and the initial downsampling module are trained based on the training dataset to obtain the target upsampling module and the target downsampling module.

[0173] An initial model is constructed through the stacking of the target upsampling module and the target downsampling module;

[0174] The initial model is trained based on the training dataset to obtain an image super-resolution model;

[0175] The image super-resolution model is used to perform image super-resolution processing on the image to be processed, thereby obtaining the super-resolution image corresponding to the image to be processed.

[0176] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0177] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0178] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0180] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. An image super-resolution method, characterized by, The method comprises: training an initial up-sampling module and an initial down-sampling module based on a training data set, to obtain a target up-sampling module and a target down-sampling module; wherein the initial up-sampling module and the initial down-sampling module are respectively constructed based on a plurality of network nodes and a plurality of operators; the target up-sampling module is obtained after the elimination of operators and / or the splicing of operation results of the operators based on the weight change trend of the operators in the initial up-sampling module; the weight change trend of the operators in the initial up-sampling module is determined based on a first initial weight of the operators in the initial up-sampling module and a first adjusted weight corresponding to the first initial weight; and / or, the target down-sampling module is obtained after the elimination of operators and / or the splicing of operation results of the operators based on the weight change trend of the operators in the initial down-sampling module; the weight change trend of the operators in the initial down-sampling module is determined based on a second initial weight of the operators in the initial down-sampling module and a second adjusted weight corresponding to the second initial weight; constructing an initial model through the stacking processing of the target up-sampling module and the target down-sampling module; training the initial model based on the training data set, to obtain an image super-resolution model; performing image super-resolution processing on a to-be-processed image by using the image super-resolution model, to obtain a super-resolution image corresponding to the to-be-processed image.

2. The method of claim 1, wherein, Before the training of the initial up-sampling module and the initial down-sampling module based on the training data set, to obtain the target up-sampling module and the target down-sampling module, the method further comprises: constructing the initial up-sampling module and the initial down-sampling module based on a plurality of network nodes and a plurality of operators, respectively; for each two network nodes in the initial up-sampling module, determining a first initial weight corresponding to each operator; for each two network nodes in the initial down-sampling module, determining a second initial weight corresponding to each operator.

3. The method of claim 2, wherein, The training of the initial up-sampling module and the initial down-sampling module based on the training data set, to obtain the target up-sampling module and the target down-sampling module, comprises: inputting the training data set into the initial up-sampling module, to obtain a first output value; determining a first loss parameter based on an objective function and the first output value; adjusting the first initial weight according to the first loss parameter, to obtain a first adjusted weight; determining the target up-sampling module according to the first adjusted weight.

4. The method of claim 3, wherein, The determination of the target up-sampling module according to the first adjusted weight comprises: determining a weight change trend corresponding to each operator according to the first initial weight and the first adjusted weight; for each two network nodes in the initial up-sampling module, eliminating the operator with the decreasing weight change trend, to obtain the target up-sampling module.

5. The method of claim 4, wherein, The method further comprises: if a plurality of operators are reserved between two network nodes in the initial up-sampling module, determining a plurality of operation results corresponding to the plurality of operators; performing splicing processing on the plurality of operation results, to obtain a first spliced result; inputting the first spliced result into a latter one of the two network nodes.

6. The method of claim 2, wherein, The training of the initial up-sampling module and the initial down-sampling module based on the training data set respectively comprises: inputting the training data set into the initial down-sampling module to obtain a second output value; determining a second loss parameter based on an objective function and the second output value; adjusting the second initial weight according to the second loss parameter to obtain a second adjusted weight; determining the target down-sampling module according to the second adjusted weight.

7. The method of claim 6, wherein, The determination of the target down-sampling module according to the second adjusted weight comprises: determining a weight change trend of each operator according to the second initial weight and the second adjusted weight; for each two network nodes in the initial down-sampling module, eliminating the operator with a decreasing weight change trend to obtain the target down-sampling module.

8. The method of claim 7, wherein, The method further comprises: if multiple operators are reserved between two network nodes in the initial down-sampling module, determining multiple operation results corresponding to the multiple operators; performing splicing processing on the multiple operation results to obtain a second spliced result; inputting the second spliced result into a latter one of the two network nodes.

9. The method of claim 1, wherein, The construction of the initial model through the stacking processing of the target up-sampling module and the target down-sampling module comprises: performing stacking processing on at least one target up-sampling module and at least one target down-sampling module to obtain a stacked network structure; adding a down-sampling operator combination and an up-sampling operator combination to the stacked network structure to construct the initial model.

10. The method of claim 9, wherein: the down-sampling operator combination comprises a Conv operator and an Activation operator; the up-sampling operator combination comprises a Conv operator, an Activation operator, and a PixelShuffle operator.

11. An electronic device, comprising: The electronic device comprises an acquisition unit and a construction unit, The acquisition unit is configured to train an initial up-sampling module and an initial down-sampling module based on a training data set respectively to obtain a target up-sampling module and a target down-sampling module; wherein the initial up-sampling module and the initial down-sampling module are constructed based on multiple network nodes and multiple operators respectively; the target up-sampling module is obtained through the elimination of operators and / or the splicing of operation results of the operators based on the weight change trend of the operators in the initial up-sampling module; and the weight change trend of the operators in the initial up-sampling module is determined based on a first initial weight of the operators in the initial up-sampling module and a first adjusted weight corresponding to the first initial weight. And / or, the target down-sampling module is obtained after the operator pruning based on the weight change trend of the operator in the initial down-sampling module and / or the splicing of the operation results of the operator; the weight change trend of the operator in the initial down-sampling module is determined based on the second initial weight of the operator in the initial down-sampling module and the second adjusted weight corresponding to the second initial weight; The construction unit is configured to construct an initial model through a stack processing of the target up-sampling module and the target down-sampling module. The acquisition unit is further configured to train the initial model based on the training data set to obtain an image super-resolution model, and perform image super-resolution processing on a to-be-processed image by using the image super-resolution model to obtain a super-resolution image corresponding to the to-be-processed image.

12. An electronic device, comprising: The electronic device includes a processor, a memory storing instructions executable by the processor, and when the instructions are executed by the processor, the method of any one of claims 1-10 is implemented.

13. A chip, characterized by The chip includes programmable logic circuit and / or program instructions, and when the chip is running, the method of any one of claims 1-10 is implemented.

Images