Super-resolution model processing methods, devices, computer equipment and storage media

Abstract

This application discloses a super-resolution model processing method, apparatus, computer device, and storage medium. The method includes: determining the model parameters and channel selection parameters of the super-resolution model; performing multiple rounds of iterative alternating optimization on the model parameters and channel selection parameters to obtain target model parameters and target channel selection parameters; determining the target super-resolution model based on the target model parameters and target channel selection parameters; wherein, the target channel selection parameters are used to trim the model dictionary of the super-resolution model to obtain a lightweight target super-resolution model that retains the target model parameters. Using this method, the super-resolution model can be made lightweight.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a super-resolution model processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, super-resolution technology has emerged, which can recover high-resolution images from low-resolution images, improve image quality, and thus obtain clearer pictures. For example, super-resolution technology can be used to perform real-time inference on blurry videos to obtain high-definition videos.

[0003] Traditionally, deep learning models are deployed by selecting deployment strategies provided by hardware vendors. However, the large inference size of super-resolution models hinders the application of traditional deep learning deployment strategies on these models. Summary of the Invention

[0004] Therefore, it is necessary to provide a super-resolution model processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the above-mentioned technical problems, which can achieve lightweight super-resolution models.

[0005] Firstly, this application provides a super-resolution model processing method, including:

[0006] Determine the model parameters and channel selection parameters of the super-resolution model;

[0007] The model parameters and channel selection parameters are optimized through multiple rounds of iterative alternation to obtain the target model parameters and target channel selection parameters;

[0008] The target super-resolution model is determined based on the target model parameters and the target channel selection parameters. The target channel selection parameters are used to prune the model dictionary of the super-resolution model to obtain a lightweight target super-resolution model that has the target model parameters.

[0009] Secondly, this application also provides a super-resolution model processing apparatus, comprising:

[0010] The first determining module is used to determine the model parameters and channel selection parameters of the super-resolution model;

[0011] The optimization module is used to perform multiple rounds of iterative optimization of the model parameters and channel selection parameters to obtain the target model parameters and target channel selection parameters.

[0012] The second determining module is used to determine the target super-resolution model based on the target model parameters and the target channel selection parameters; wherein, the target channel selection parameters are used to trim the model dictionary of the super-resolution model to obtain a lightweight target super-resolution model that has the target model parameters.

[0013] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method.

[0014] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0015] Fifthly, this application also provides a computer program product comprising a computer program that, when executed by a processor, implements the steps in the method described above.

[0016] The aforementioned super-resolution model processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product obtain target channel selection parameters and target model parameters by iteratively optimizing the model parameters and channel selection parameters of the super-resolution model to be lightweighted through multiple rounds of optimization. Optimizing the channel selection parameters used for trimming the model dictionary, rather than adjusting the model dictionary itself, saves computational resources. Furthermore, by using alternating optimization, rather than directly adjusting both the model parameters and the model dictionary simultaneously, the complexity of the lightweight super-resolution model can be reduced while finding more suitable target channel selection parameters and target model parameters. Then, based on the target channel selection parameters and target model parameters, a lightweight target super-resolution model is obtained, achieving lightweighting of the super-resolution model while maintaining its performance. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a super-resolution model processing method provided in an embodiment of this application;

[0018] Figure 2 This is a schematic diagram illustrating the principle of filtering through threads provided in an embodiment of this application.

[0019] Figure 3 A schematic diagram of the architecture of the super-resolution model provided in the embodiments of this application;

[0020] Figure 4 A schematic diagram illustrating the principle of lightweight processing for the super-resolution model provided in this application embodiment;

[0021] Figure 5 A flowchart illustrating another super-resolution model processing method provided in this application embodiment;

[0022] Figure 6 A structural block diagram of a super-resolution model processing device provided in an embodiment of this application;

[0023] Figure 7 An internal structural diagram of a computer device provided in an embodiment of this application;

[0024] Figure 8 An internal structural diagram of another computer device provided in an embodiment of this application;

[0025] Figure 9 This is an internal structural diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] In some embodiments, such as Figure 1 As shown, a super-resolution model processing method is provided, illustrated by applying the method to a computer device. It is understood that this method can be applied to a terminal, a server, or a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0028] S102, Determine the model parameters and channel selection parameters of the super-resolution model.

[0029] The super-resolution model is used to generate or reconstruct high-resolution images given a low-resolution input image. Model parameters are learned parameters and are internal configuration variables of the model. Channel selection parameters are used to prune the model dictionary of the super-resolution model. Super-resolution model lightweighting refers to reducing the model's complexity and computational load as much as possible without sacrificing its realism. In other words, the target super-resolution model obtained after super-resolution model lightweighting has lower complexity and less inference computation compared to the original super-resolution model.

[0030] For example, in the lightweighting process of the super-resolution model, there are two optimization objectives: channel selection parameters and model parameters. Channel selection parameters can be binary data. For instance, β is the channel selection parameter. i Select the i-th binarized data from the channel parameters, when β i When = 0, the i-th channel in the super-resolution model's model dictionary will be pruned. The computer equipment can determine the model parameters and channel selection parameters of the super-resolution model.

[0031] In some embodiments, the computer device may include at least one of a terminal and a server. The terminal may be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, etc. Portable wearable devices may include smartwatches, smart bracelets, head-mounted devices, etc. The server may be implemented using a standalone server or a server cluster consisting of multiple servers.

[0032] S104 involves iteratively optimizing the model parameters and channel selection parameters in multiple rounds to obtain the target model parameters and target channel selection parameters.

[0033] In some embodiments, the computer device can, in a multi-round iterative optimization process, fix the initial channel selection parameters for each round, optimize the initial model parameters for each round to obtain the optimized model parameters for each round; fix the optimized model parameters for each round, optimize the initial channel selection parameters for each round to obtain the optimized channel selection parameters for each round. The optimized model parameters for the current round are used as the initial model parameters for the next round. The optimized channel selection parameters for the current round are used as the initial channel selection parameters for the next round. The computer device can determine the target channel selection parameters and the target model parameters when the iteration stops.

[0034] It should be noted that this embodiment is only used to illustrate one case of alternating optimization and is not intended to limit the optimization order of model parameters and channel selection parameters in each round of alternating optimization. It is understood that the alternating optimization order of model parameters and channel selection parameters can be flexible.

[0035] S106. Determine the target super-resolution model based on the target model parameters and the target channel selection parameters.

[0036] The target channel selection parameter is used to prune the model dictionary of the super-resolution model to obtain a lightweight target super-resolution model with target model parameters.

[0037] For example, a computer device can determine a model dictionary including a target super-resolution model with target model parameters selected by the target channel.

[0038] In some embodiments, the computer device can deploy the target super-resolution model onto the target hardware. It is understood that the target super-resolution model is more lightweight than the super-resolution model and has less complexity in deploying it onto the target hardware.

[0039] In the aforementioned super-resolution model processing method, the target channel selection parameters and target model parameters are obtained through multiple rounds of iterative, alternating optimization of the model parameters and channel selection parameters of the super-resolution model to be lightweighted. Optimizing the channel selection parameters used for pruning the model dictionary, rather than adjusting the model dictionary itself, saves computational resources. Furthermore, by using alternating optimization, rather than directly adjusting both the model parameters and the model dictionary simultaneously, the complexity of the lightweight super-resolution model can be reduced while finding more suitable target channel selection parameters and target model parameters. Then, based on the target channel selection parameters and target model parameters, the lightweight target super-resolution model is obtained, achieving lightweighting of the super-resolution model while maintaining its performance.

[0040] In some embodiments, the method further includes:

[0041] The target super-resolution model is used to determine the filter tensor data corresponding to the low-resolution image; the filter tensor data refers to the tensor data used for image filtering.

[0042] Determine the image data to be filtered corresponding to the low-resolution image;

[0043] Elements with the same index in each channel of the filtered tensor data and the image data to be filtered are assigned to the same thread, and elements with adjacent indexes in each channel are assigned to consecutive threads.

[0044] The super-resolution result is obtained by using a thread to filter the image data to be filtered based on the filter tensor data, and then obtaining the super-resolution result output by the target super-resolution model.

[0045] For example, the target model parameters are the parameters of the regression network in the target super-resolution model. The target super-resolution model includes a regression network with target model parameters and a model dictionary pruned by selecting parameters through the target channels. When the target super-resolution model is deployed on target hardware, the computer device can use a low-resolution image as input to the target super-resolution model, determine the dictionary weight coefficients corresponding to the low-resolution image output by the regression network in the target super-resolution model, and use these dictionary weight coefficients to query the filtered tensor data corresponding to the low-resolution image from the model dictionary.

[0046] Computer devices can upsample low-resolution images using a target super-resolution model to obtain the image data to be filtered. The computer devices can then cache the filtered tensor data and the image data to be filtered in the shared memory of the streaming multiprocessor in the target hardware.

[0047] Here, the target hardware refers to the hardware used to deploy the target super-resolution model. Elements in the filtered tensor data and the image data to be filtered are allocated to threads in various thread blocks within the streaming multiprocessor. It can be understood that during the filtering process using the filtered tensor data to filter the image data, the computer equipment uses the target hardware to calculate the Hadamard product and the reduced sums on each channel of the filtered tensor data and the image data to be filtered.

[0048] The computer device can assign elements with the same position index in each channel of the filtered tensor data and the image data to the same thread. That is, elements with the same position index but from different channels in the filtered tensor data and the image data to be filtered will be assigned to the same thread. The computer device can then perform a Hadamard product calculation on the filtered tensor data and the image data to be filtered using each assigned thread, obtaining the element-wise product across multiple channels corresponding to each position index. Furthermore, the computer device can calculate the reduced sum across multiple channels corresponding to each position index using each assigned thread, i.e., summing the element-wise products across multiple channels corresponding to each position index, to obtain the super-resolution data for each pixel. In essence, the reduced sum across multiple channels corresponding to each position index is the super-resolution data for each pixel. Each position index can be used to index the super-resolution data for each pixel in the super-resolution result. The super-resolution data for each pixel is actually the super-resolution data at the position indicated by each position index.

[0049] Computer devices can allocate elements with adjacent position indices in each channel to consecutive threads. That is, elements with adjacent position indices in the filtered tensor data and the image data to be filtered are allocated to adjacent threads. It can be understood that the position of a thread within a thread block matches the position of each pixel in the image. The output of each thread is the super-resolution data for each pixel. By deploying super-resolution data with adjacent position indices on adjacent threads, it is convenient to accumulate the super-resolution data output from each thread into a super-resolution result.

[0050] It should be noted that deploying the target super-resolution model to the target hardware requires multiple rounds of iterative search of the thread block size of the target hardware. The computer device can use the method provided in this embodiment to determine the model inference latency corresponding to the new sample thread block size found in each round of iterative search.

[0051] In some embodiments, such as Figure 2 The diagram shows a schematic representation of the principle of filtering through threads. Figure 2 The diagram illustrates the thread allocation for elements at positions (1,1) and (1,2) in channels 0 and 1 of the filtered tensor data and the image data to be filtered. Thread block 0 and thread block 1 belong to the same streaming multiprocessor. The filtered tensor data and the image data to be filtered are cached in the shared memory of this streaming multiprocessor.

[0052] Computer devices can compute elements in the filtered tensor data and the image data to be filtered using adjacent thread blocks in a streaming multiprocessor. Understandably, the threads in one thread block are insufficient to process all elements in the filtered tensor data and the image data to be filtered. Therefore, the computer device can assign elements that thread block 0 cannot process to thread block 1, which is adjacent to thread block 0. Specifically, the element at (1,1) in channel 0 and channel 1 is assigned to thread 0 in thread block 0, and the element at (1,2) in channel 0 and channel 1 is correspondingly assigned to thread 1 in thread block 0.

[0053] Computer devices can accumulate the super-resolution data output by each thread into a super-resolution result.

[0054] In this embodiment, elements with the same index in each channel of the filtered tensor data and the image data to be filtered are assigned to the same thread, and elements with adjacent indexes in each channel are assigned to consecutive threads. The threads perform filtering processing on the image data to be filtered based on the filtered tensor data to obtain the super-resolution result output by the target super-resolution model. Although the reduction in the size of the model dictionary may accelerate inference to some extent, the filtering operation in the super-resolution model is still the bottleneck of model inference latency. The above thread allocation method can perform parallel computation on the super-resolution data of each pixel in the super-resolution result, which can minimize the limitations of filtering operations and improve the inference efficiency of the target super-resolution model.

[0055] In some embodiments, the model parameters and channel selection parameters are iteratively optimized in multiple rounds to obtain the target model parameters and target channel selection parameters, including:

[0056] In each round of alternating optimization, the initial channel selection parameters are fixed, and the initial model parameters are optimized to obtain the optimized model parameters for this round. The optimized model parameters are then fixed, and the initial channel selection parameters are optimized to obtain the optimized channel selection parameters for this round. The optimized model parameters and optimized channel selection parameters are used as the initial model parameters and channel selection parameters for the next round to continue alternating optimization until the alternating optimization stopping condition is met, thus obtaining the target model parameters and target channel selection parameters.

[0057] For example, during each round of alternating optimization, the computer device can fix the initial channel selection parameters for that round and perform a linear transformation on the model parameters of the super-resolution model to obtain the optimized model parameters for that round. The computer device can then fix the optimized model parameters for that round and search for channel selection parameters that meet a preset sparsity requirement to obtain the optimized channel selection parameters for that round.

[0058] Understandably, computer equipment can fix one optimization objective and optimize another to obtain an optimized objective that better fits the fixed objective. That is, the optimized model parameters in this round are a better fit with the initial channel selection parameters compared to the initial model parameters in this round. The optimized channel selection parameters in this round are a better fit with the optimized model parameters compared to the initial channel selection parameters in this round.

[0059] The computer equipment can determine the optimized channel selection parameters and model parameters of the current round as the initial channel selection parameters and model parameters for the next round, and continue to optimize alternately until the target channel selection parameters and target model parameters that minimize the performance loss of the lightweight super-resolution model are obtained. The computer equipment can determine the model dictionary, which includes the model selected by the target channel selection parameters, and the target super-resolution model with the target model parameters.

[0060] It is understandable that pruning the model dictionary of the super-resolution model by using channel selection parameters will cause performance loss in the super-resolution model. The process of alternating optimization is to continuously search for the optimal target channel selection parameters and the target model parameters that best match the target channel selection parameters, so that the performance loss of the lightweight target super-resolution model is relatively minimal compared to the super-resolution model before lightweighting.

[0061] In some embodiments, the computer device can search for channel selection parameters while keeping the model parameters of the super-resolution model unchanged, to obtain initial channel selection parameters that meet the preset sparsity requirements. It can be understood that, under the condition of meeting the preset sparsity requirements and keeping the model parameters of the super-resolution model unchanged, pruning the model dictionary using the initial channel selection parameters minimizes the performance loss of the super-resolution model.

[0062] In some embodiments, the initial model parameters in the first round are the model parameters of the super-resolution model. It can be understood that the model parameters of the super-resolution model are the initial model parameters.

[0063] In one embodiment, the first objective loss value and the second objective loss value characterize the performance loss of the super-resolution model under changes in channel selection parameters and model parameters. The objective model parameter is the model parameter with the smallest corresponding first objective loss value among the model parameters generated during multiple rounds of iterative optimization. The objective channel selection parameter is the channel selection parameter with the smallest corresponding second objective loss value among the channel selection parameters generated during multiple rounds of iterative optimization.

[0064] In one embodiment, the computer device can determine the channel selection parameters and model parameters as variables in the original loss function to obtain a third objective function. The computer device can also determine the model parameters and channel selection parameters generated during each round of alternating optimization as inputs to the third objective function to obtain the third objective loss value output by the third objective function. The computer device can further determine the target model parameters and target channel selection parameters that minimize the third objective loss value. It is understood that the variables in the third objective function can also be channel selection parameters and linear transformation coefficients.

[0065] In one embodiment, the process of performing multiple rounds of iterative optimization of the model parameters and channel selection parameters to obtain the target model parameters and target channel selection parameters can be expressed as formula (1).

[0066]

[0067] Where β represents the channel selection parameter generated during the multi-round alternating optimization process, and W represents the model parameter generated during the multi-round alternating optimization process. gt The characterization is based on the super-resolution results. The super-resolution result represents the inference prediction, L represents the number of channels in the model dictionary of the super-resolution model, and β represents the super-resolution model. i In the channel selection parameters, Φ represents the sample dictionary weight coefficients, D represents the model dictionary of the super-resolution model, N represents the number of low-resolution images, and argmin represents the model parameters that minimize the third objective function. 'a' is the preset sparsity requirement. β||0 represents the L0 norm of the channel selection parameters. In LaparNet(X, W), LaparNet represents the regression network in the super-resolution model, and X represents the low-resolution images. It is understandable that... This is the third objective function.

[0068] In this embodiment, during each round of alternating optimization, the initial channel selection parameters are fixed, and the initial model parameters are optimized. Then, the optimized model parameters are fixed, and the initial channel selection parameters are optimized again. This process of alternating optimization of the optimized channel selection parameters and the optimized model parameters continues until the target channel selection parameters and target model parameters are obtained. By alternating optimization of the model parameters and channel selection parameters, the complexity of the lightweight super-resolution model is reduced, while more suitable target channel selection parameters and target model parameters are found, resulting in a lightweight target super-resolution model and achieving lightweighting of the super-resolution model.

[0069] In some embodiments, the initial model parameters for this round are optimized to obtain the optimized model parameters for this round, including:

[0070] The initial model parameters are iteratively optimized and adjusted in the direction of reducing the first objective loss value to obtain the optimized model parameters for this round.

[0071] The first target loss value is used to characterize the difference between the super-resolution result predicted by the super-resolution model and the reference super-resolution result when the initial channel selection parameters remain unchanged in this round; the super-resolution result predicted by the super-resolution model is the super-resolution image obtained by super-resolution inference prediction of the sample low-resolution image; the reference super-resolution result is the super-resolution image pre-set for the sample low-resolution image.

[0072] Among them, low-resolution images refer to low-resolution images used as samples.

[0073] For example, the computer device can determine the first objective function for this round. The first objective function is constructed based on the original loss function and the initial channel selection parameters for this round.

[0074] The computer can use each model parameter generated during this round of alternating optimization as input to the first objective function, obtaining the first objective loss value corresponding to each model parameter, which is the output of the first objective function. The computer can then determine the model parameters that minimize the output of the first objective function, thus obtaining the optimized model parameters for this round. It can be understood that among the model parameters generated during this round of alternating optimization, the optimized model parameters have the smallest corresponding first objective loss value.

[0075] It should be noted that the original loss function of the super-resolution model is used to evaluate its performance. The better the performance of the super-resolution model, the smaller the difference between the inference prediction and the reference super-resolution result. To minimize the performance loss of the super-resolution model during its lightweighting process, a first objective function can be constructed based on the original loss function. This first objective function is used to evaluate the different model performance losses caused by various model parameters, given a fixed initial channel selection parameter for this round.

[0076] In some embodiments, the computer device may determine the channel selection parameters in the original loss function as the initial channel selection parameters for this round, and use the model parameters as variables in the original loss function to obtain the first objective function.

[0077] In some embodiments, the original loss function may include at least one of the mean absolute error function and the mean squared error function.

[0078] In some embodiments, the original loss function can be the mean square error (L2 loss function). The process of optimizing the initial model parameters in this round can be expressed as formula (2).

[0079]

[0080] Where W represents the model parameters generated during this round of alternating optimization, and H gt Characterized by super-resolution results, F W，D′ B T The super-resolution result of the inference prediction is represented by D′, which represents the model dictionary after the initial channel selection parameters are pruned in this round. T F represents the low-resolution image of the upsampled sample. WD′ The transformation tensor data is represented by N, which represents the number of low-resolution images in the sample. argmin represents the model parameters that minimize the first objective function. This is understandable. Let be the first objective function.

[0081] In this embodiment, by iteratively optimizing and adjusting the initial model parameters in the current round in the direction of reducing the first target loss value, it is possible to obtain the optimized model parameters that result in the minimum loss of model performance in the current round while keeping the initial channel selection parameters fixed. This alternating optimization method reduces the computational complexity and saves computational resources while ensuring accuracy.

[0082] In some embodiments, the model parameters are optimized by optimizing the linear transformation coefficients corresponding to the model parameters; during each round of alternating optimization, the model parameters of the super-resolution model remain unchanged; the target model parameters are determined based on the target linear transformation coefficients obtained after the alternating optimization stops.

[0083] The initial model parameters are iteratively optimized and adjusted in the direction of reducing the first objective loss value, resulting in the optimized model parameters for this round, including:

[0084] The initial linear transformation coefficients of this round are optimized in the direction of reducing the loss value of the first objective, resulting in the optimized linear transformation coefficients of this round.

[0085] In this round of alternating optimization, the linear transformation coefficients generated are used to adjust the sample dictionary weight coefficients obtained by the super-resolution model through regression analysis of the low-resolution sample images; the adjusted sample dictionary weight coefficients are used to determine the transformation tensor data from the model dictionary after clipping through the initial channel selection parameters in this round; the transformation tensor data is used to transform the low-resolution sample images to obtain the super-resolution results for inference prediction.

[0086] For example, the computer device can determine the first objective function for this round. The computer device can use each linear transformation coefficient generated during this round of alternating optimization as input to the first objective function, obtaining the first objective loss value corresponding to each linear transformation coefficient as the output of the first objective function. The computer device can determine the linear transformation coefficients that minimize the output of the first objective function, obtaining the optimized linear transformation coefficients for this round. Among the linear transformation coefficients generated during this round of alternating optimization, the optimized linear transformation coefficients correspond to the smallest first objective loss value.

[0087] Understandably, super-resolution models have a large number of parameters, and directly optimizing these parameters would result in high computational complexity. However, the linear transformation coefficients reflect the impact of different model parameters on the super-resolution model. Therefore, indirectly optimizing the model parameters by optimizing the linear transformation coefficients can effectively reduce computational complexity and save computational resources.

[0088] In some embodiments, the model dictionary can be a filter kernel dictionary, and the transform tensor data can be filter tensor data. The filter tensor data includes the filter kernel parameters corresponding to each pixel in the low-resolution image, which is equivalent to the filter corresponding to the low-resolution image. The dictionary weight coefficients include the filter kernel weight coefficients corresponding to each pixel in the low-resolution image. The filter tensor data is obtained by weighted summation of each filter kernel in the filter kernel dictionary using the filter kernel weight coefficients corresponding to each pixel.

[0089] In some embodiments, the original loss function can be the mean square error (L2 loss function). The process of optimizing the initial linear transformation coefficients of this round can be expressed as formula (3).

[0090]

[0091] Where γ is the linear transformation coefficient, W represents the model parameters of the super-resolution model, and H... gt The characterization is based on the super-resolution results. Super-resolution results of representational inference prediction, γ i Let L represent the i-th value in the linear transformation coefficients, L represent the number of channels in the super-resolution model's model dictionary, D′ represent the model dictionary after the initial channel selection parameters are pruned in this round, and B represents the number of channels in the super-resolution model's model dictionary. T Characterizing the low-resolution image of the upsampled sample, γ i F W，D′ The transformation tensor data is represented by N, which represents the number of low-resolution images in the sample. argmin represents the model parameters that minimize the first objective function. This is understandable. Let be the first objective function. The model parameters, optimized in this round, are adapted to D′. γWD′ The model parameters are characterized by linear transformation of the coefficients.

[0092] In some embodiments, such as Figure 3 The diagram shows the architecture of a super-resolution model. A computer device can use a low-resolution image as input to the super-resolution model, which then upsamples the low-resolution image to obtain the upsampled low-resolution image.

[0093] The super-resolution model includes a regression network for linear regression analysis of the low-resolution image. The output of the regression network is the dictionary weight coefficients Φ. The regression network includes multiple local fusion blocks (LFBs), pixel shuffle layers, and convolutional layers. Each local fusion block includes multiple residual blocks (RBs) and convolutional layers.

[0094] The super-resolution model includes a filter kernel dictionary. The dictionary weight coefficients output by the regression network are used to look up the filter kernel dictionary to assemble the filter tensor data. The upsampled low-resolution image is then filtered using the filter tensor data to obtain the super-resolution result predicted by the super-resolution model. The filter tensor data includes the filter kernel parameters corresponding to each pixel in the low-resolution image. The dictionary weight coefficients include the filter kernel weight coefficients corresponding to each pixel. L is the number of filter kernels in the model dictionary, F i represents the filter kernel parameters corresponding to the i-th pixel. K*K is the size of the filter kernel in the model dictionary.

[0095] In some embodiments, such as Figure 4 The diagram illustrates the principle of lightweight processing for super-resolution models. The upper part of the diagram shows the inference and prediction process of the unlightened super-resolution model. The filtered tensor data F is obtained by querying the model dictionary D using the dictionary weight coefficients Φ. w，D The super-resolution results for inference prediction are obtained by filtering the tensor data.

[0096] The lower half of the figure illustrates the lightweighting of the super-resolution model. During the lightweighting process, a linear transformation of the model parameters is required. In fact, the dictionary weight coefficients corresponding to the linearly transformed model parameters are equivalent to directly applying a linear transformation to the dictionary weight coefficients corresponding to the untransformed model parameters. Therefore, during alternating optimization, the model parameters can be optimized by optimizing the linear transformation coefficients.

[0097] The computer equipment can perform regression analysis on the low-resolution image to obtain the initial dictionary weight coefficients Φ. During the lightweighting process of the super-resolution model, it is not necessary to recalculate the dictionary weight coefficients using the generated model parameters; only the initial dictionary weight coefficients need to be transformed using linear transformation coefficients.

[0098] Computer equipment can perform a linear transformation on the dictionary weight coefficients using linear transformation coefficients, i.e., Φ·γ, to obtain the linearly transformed dictionary weight coefficients Φ'. Computer equipment can also prune the model dictionary using channel linear transformation coefficients to obtain the pruned model dictionary D'. It can be understood that the i-th value in the channel selection parameter is 0, i.e., β. i =0 indicates that the i-th channel in the model dictionary is pruned.

[0099] Computer equipment can query the clipped model dictionary using the linearly transformed dictionary weight coefficients to obtain the filtered tensor data F. W’,D’ Computer equipment can filter tensor data F W’,D’ The super-resolution result of the inference prediction is obtained by performing filtering.

[0100] In some embodiments, a computer device can obtain target model parameters by performing a linear transformation on the model parameters of the super-resolution model using target linear transformation coefficients.

[0101] In some embodiments, a computer device may perform a tensor multiplication operation on the target linear transformation coefficients and the model parameters of the super-resolution model to obtain the target model parameters.

[0102] In this embodiment, the initial linear transformation coefficients of this round are optimized in the direction of reducing the first target loss value, resulting in optimized linear transformation coefficients for this round. Compared with directly adjusting the model parameters, this greatly reduces computational complexity and data volume, saving computational resources.

[0103] In some embodiments, the initial channel selection parameters for this round are optimized to obtain the optimized channel selection parameters for this round, including:

[0104] The initial channel selection parameters for this round are iteratively optimized and adjusted in the direction of reducing the loss value of the second objective, resulting in the optimized channel selection parameters for this round.

[0105] The second objective loss value is used to characterize the difference between the super-resolution result predicted by the super-resolution model and the reference super-resolution result when the model parameters remain unchanged after this round of optimization; the super-resolution result predicted by the inference changes with the channel selection parameters when the model parameters remain unchanged after this round of optimization.

[0106] For example, the computer device can determine the second objective function for this round. The second objective function is constructed based on the original loss function, the optimized model parameters for this round, and the corresponding regularization function for this round. The regularization coefficient of the regularization function is related to the number of rounds of alternating optimization; as the number of rounds of alternating optimization increases, the coefficient of the regularization function also increases. It can be understood that the regularization coefficient of the regularization function is different in each round of alternating optimization.

[0107] The computer can use each channel selection parameter generated during this round of optimization as input to the second objective function to obtain the second objective loss value corresponding to each channel selection parameter. The computer can then determine the channel selection parameters that minimize the output of the second objective function, thus obtaining the optimized channel selection parameters for this round. It can be understood that among the channel selection parameters generated during this round of alternating optimization, the optimized channel selection parameters have the smallest corresponding second loss value.

[0108] In some embodiments, the variables in the regularization function can be channel selection parameters. The regularization function includes regularizing the channel selection parameters and weighting the regularized channel selection parameters using regularization coefficients. For example, the regularization function can be an L1 regularization function.

[0109] In some embodiments, the regularization coefficient of the regularization function in the next round of alternating optimization is twice that in the current round of alternating optimization. That is, the regularization coefficient increases by 1 time in each round.

[0110] In some embodiments, the computer device can optimize the initial channel selection parameters for the current round based on a regression algorithm. For example, the regression algorithm could be a lasso regression algorithm.

[0111] In some embodiments, the original loss function can be the mean square error (L2 loss function). The process of optimizing the initial channel selection parameters for this round can be expressed as formula (4).

[0112]

[0113] Where W represents the model parameters after this round of optimization, H gt Characterized by super-resolution results, F W，β B T The super-resolution result is represented by the inference prediction, β represents the channel selection parameter generated during this round of alternating optimization, and B T F represents the low-resolution image of the upsampled sample. W，βThe transformation tensor data is represented by N, which represents the number of low-resolution images in the sample. argmin represents the model parameters that minimize the first objective function. λ is the regularization coefficient, and λ||β||1 is the regularization function. α is the preset sparsity requirement, for example, 10%. L is the number of channels in the model dictionary.

[0114] Understandable This is the second objective function.

[0115] In some embodiments, the computer device may, during each round of alternating optimization, fix the initial channel selection parameters of the current round and optimize the initial linear transformation coefficients of the current round; fix the optimized linear transformation coefficients of the current round and optimize the initial channel selection parameters of the current round.

[0116] In this embodiment, by iteratively optimizing and adjusting the initial channel selection parameters in the current round in the direction of reducing the second objective loss value, the optimized channel selection parameters of the current round are obtained. Under the condition of fixing the optimized model parameters of the current round, the optimized channel selection parameters of the current round that result in the minimum loss of model performance can be obtained. This alternating optimization method reduces the computational complexity and saves computational resources while ensuring accuracy.

[0117] In some embodiments, the second target loss value is determined based on the prediction difference value and the regularization value; the prediction difference value is the difference between the super-resolution result predicted by inference and the reference super-resolution result, assuming the model parameters remain unchanged after this round of optimization; the regularization value is determined based on the coefficient used to characterize the sparsity requirement and the channel selection parameter generated during this round of alternating optimization.

[0118] For example, the computer device can determine the model parameters in the original loss function as the optimized model parameters for this round, and use the channel selection parameters as variables in the original loss function to obtain the component loss function. The computer device can use the channel selection parameters as variables in the regularization function and determine the regularization coefficients corresponding to this round of alternating optimization to obtain the regularization function. The computer device can determine a second objective function that includes the component loss function and the regularization function.

[0119] The channel selection parameters generated during this round of alternating optimization are used as input to the second loss function, resulting in the predicted difference value output by the component loss function and the regularization value corresponding to the regularization function. The computer can then sum the predicted difference value and the regularization value to obtain the second target loss value.

[0120] It should be noted that the channel selection parameters generated in each round of alternating optimization all meet the preset sparsity requirements. In each round of alternating optimization, as the regularization coefficient increases, the channel selection parameters after each round of optimization become increasingly sparse, that is, the sparsity requirement becomes increasingly higher as the regularization coefficient increases.

[0121] In this embodiment, the second target loss value is determined based on the prediction difference value and the regularization value. The prediction difference value is the difference between the super-resolution result predicted by inference and the reference super-resolution result when the model parameters remain unchanged after this round of optimization. The regularization value is determined based on the coefficient used to characterize the sparsity requirement and the channel selection parameter generated during this round of alternating optimization. The channel selection parameter after this round of optimization can be obtained through the second loss value, and then the target channel selection parameter can be obtained to achieve the lightweighting of the super-resolution model.

[0122] In some embodiments, such as Figure 5 The diagram shows a flowchart of a super-resolution model processing method. This method includes steps S502 to S510. Wherein:

[0123] S502 deploys the target super-resolution model to the target hardware.

[0124] For example, the target hardware may be a processor, such as a central processing unit or a graphics processing unit. A computer device may deploy the target super-resolution model onto the target hardware. It should be noted that this application does not limit the manner in which the target super-resolution model is deployed on the target hardware.

[0125] In some embodiments, the target hardware may be an embedded graphics card.

[0126] S504, in each round of iterative search for the thread block size of the target hardware, uses a proxy model to fit the sample thread block dataset to obtain the fitted proxy model.

[0127] The sample thread block dataset includes the sample thread block size and the corresponding model inference latency. The thread block size corresponds to the number of threads in the thread block. The model inference latency indicates the time required for the target super-resolution model deployed on the target hardware to perform inference predictions.

[0128] For example, after the target super-resolution model is deployed to the target hardware, the computer device can search for the optimal thread block size. The computer device can search for multiple sample thread block sizes and their corresponding model inference latency to obtain a sample thread block dataset. During each iteration of the search for the thread block size of the target hardware, the computer device can use a surrogate model to fit the sample thread block dataset to obtain a fitted surrogate model.

[0129] In some embodiments, the surrogate model may be a model such as Gaussian processes or random forest regression.

[0130] S506, based on the fitted surrogate model, determines the new sample thread block size.

[0131] For example, a computer device can determine a new sample thread block size that minimizes the inference latency of the predictive model using a fitted surrogate model.

[0132] S508: Based on the new sample thread block size, run the target super-resolution model on the target hardware to obtain the model inference latency corresponding to the new sample thread block size, and add the new sample thread block size and the corresponding model inference latency to the sample thread block dataset to continue iterative search.

[0133] For example, the predictive model inference latency is indicated by the surrogate model fitted in this round. The computer device can run the target super-resolution model on the target hardware with the new sample thread block size to obtain the actual model inference latency corresponding to the new sample thread block size. The computer device can determine the thread block size obtained in the previous round, where the model inference latency corresponding to the thread block size obtained in the previous round is the smallest in the sample thread block dataset. The computer device can add the new sample thread block size and its corresponding model inference latency to the sample thread block dataset, and determine the thread block size obtained in this round by comparing the new thread block size with the model inference latency corresponding to the thread block size obtained in the previous round, to continue the iterative search. It can be understood that the thread block size obtained in this round is the sample thread block size with the smallest model inference latency found up to this round.

[0134] After the iterative search is completed, the S510 runs the target super-resolution model on the target hardware based on the target thread block size.

[0135] The target thread block size is determined from multiple sample thread block sizes based on the model inference latency.

[0136] For example, the computer device can determine the size of the thread block obtained in the last round, thus obtaining the target thread block size. It can be understood that the model inference latency corresponding to the target thread block size in the sample thread block dataset is minimized. The computer device can determine the end of the iterative search after reaching a preset number of iterations. After the iterative search ends, the computer device can divide the target hardware into thread blocks according to the target thread block size, and run the target super-resolution model using the divided thread blocks.

[0137] In this embodiment, a proxy model is used to fit the sample thread dataset to search for new sample thread block sizes, enriching the sample thread block dataset. The new sample thread block size with the smallest inference latency indicated by the fitted proxy model needs to be run on the target hardware to obtain the actual corresponding model inference latency. Thus, the actual corresponding model inference latency can be compared with the sample thread block sizes in the sample thread dataset to obtain the target thread block size with the smallest actual corresponding model inference latency. By selecting an appropriate thread block size, the contradiction between parallelism and congestion caused by resource competition is balanced, so that the computing resources on the target hardware are fully utilized, and the inference prediction speed of the target super-resolution model on the target hardware is improved.

[0138] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0139] Based on the same inventive concept, this application also provides a super-resolution model processing apparatus. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of one or more super-resolution model processing apparatus embodiments provided below can be found in the limitations of the super-resolution model processing method above, and will not be repeated here.

[0140] In some embodiments, such as Figure 6 As shown, a super-resolution model processing device 600 is provided, comprising:

[0141] The first determining module 602 is used to determine the model parameters and channel selection parameters of the super-resolution model;

[0142] The optimization module 604 is used to perform multiple rounds of iterative and alternating optimization of the model parameters and channel selection parameters to obtain the target model parameters and target channel selection parameters.

[0143] The second determining module 606 is used to determine the target super-resolution model based on the target model parameters and the target channel selection parameters; wherein, the target channel selection parameters are used to trim the model dictionary of the super-resolution model to obtain a lightweight target super-resolution model that has the target model parameters.

[0144] In some embodiments, the second determining module 606 is further configured to determine the filtered tensor data corresponding to the low-resolution image through the target super-resolution model; the filtered tensor data refers to the tensor data used for image filtering; determine the image data to be filtered corresponding to the low-resolution image; assign elements with the same position index in each channel of the filtered tensor data and the image data to be filtered to the same thread, and assign elements with adjacent position indices in each channel to consecutive threads; and perform filtering processing on the image data to be filtered based on the filtered tensor data through the threads to obtain the super-resolution result output by the target super-resolution model.

[0145] In some embodiments, in the process of iteratively optimizing the model parameters and channel selection parameters in multiple rounds to obtain the target model parameters and target channel selection parameters, the optimization module 604 is specifically used for:

[0146] In each round of alternating optimization, the initial channel selection parameters of this round are fixed, and the initial model parameters of this round are optimized to obtain the optimized model parameters of this round; the optimized model parameters of this round are fixed, and the initial channel selection parameters of this round are optimized to obtain the optimized channel selection parameters of this round.

[0147] The optimized model parameters and channel selection parameters of this round are used as the initial model parameters and channel selection parameters for the next round to continue alternating optimization until the alternating optimization stopping condition is met, thus obtaining the target model parameters and target channel selection parameters.

[0148] In some embodiments, in optimizing the initial model parameters of this round to obtain the optimized model parameters of this round, the optimization module 604 is specifically used for:

[0149] The initial model parameters are iteratively optimized and adjusted in the direction of reducing the first objective loss value to obtain the optimized model parameters for this round.

[0150] The first target loss value is used to characterize the difference between the super-resolution result predicted by the super-resolution model and the reference super-resolution result when the initial channel selection parameters remain unchanged in this round; the super-resolution result predicted by the super-resolution model is the super-resolution image obtained by super-resolution inference prediction of the sample low-resolution image; the reference super-resolution result is the super-resolution image pre-set for the sample low-resolution image.

[0151] In some embodiments, the model parameters are optimized by optimizing the linear transformation coefficients corresponding to the model parameters; during each round of alternating optimization, the model parameters of the super-resolution model remain unchanged; the target model parameters are determined based on the target linear transformation coefficients obtained after the alternating optimization stops.

[0152] In terms of iteratively optimizing and adjusting the initial model parameters in this round in order to reduce the first objective loss value, and obtaining the optimized model parameters in this round, the optimization module 604 is specifically used for:

[0153] The initial linear transformation coefficients of this round are optimized in the direction of reducing the loss value of the first objective, resulting in the optimized linear transformation coefficients of this round.

[0154] In this round of alternating optimization, the linear transformation coefficients generated are used to adjust the sample dictionary weight coefficients obtained by the super-resolution model through regression analysis of the low-resolution sample images; the adjusted sample dictionary weight coefficients are used to determine the transformation tensor data from the model dictionary after clipping through the initial channel selection parameters in this round; the transformation tensor data is used to transform the low-resolution sample images to obtain the super-resolution results for inference prediction.

[0155] In some embodiments, in optimizing the initial channel selection parameters of this round to obtain the optimized channel selection parameters of this round, the optimization module 604 is specifically used for:

[0156] The initial channel selection parameters for this round are iteratively optimized and adjusted in the direction of reducing the loss value of the second objective, resulting in the optimized channel selection parameters for this round.

[0157] The second objective loss value is used to characterize the difference between the super-resolution result predicted by the super-resolution model and the reference super-resolution result when the model parameters remain unchanged after this round of optimization; the super-resolution result predicted by the inference changes with the channel selection parameters when the model parameters remain unchanged after this round of optimization.

[0158] In some embodiments, the second determining module 606 is further configured to: deploy the target super-resolution model to the target hardware; during each round of iterative search of the thread block size of the target hardware, use a surrogate model to fit the sample thread block dataset to obtain a fitted surrogate model; the sample thread block dataset includes the sample thread block size and the model inference latency corresponding to the sample thread block size; determine a new sample thread block size based on the fitted surrogate model; run the target super-resolution model on the target hardware based on the new sample thread block size to obtain the model inference latency corresponding to the new sample thread block size, and add the new sample thread block size and the corresponding model inference latency to the sample thread block dataset to continue iterative search; after the iterative search is completed, run the target super-resolution model on the target hardware based on the target thread block size; the target thread block size is determined from multiple sample thread block sizes based on the model inference latency.

[0159] Each module in the aforementioned super-resolution model processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0160] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores the model dictionary and model parameters. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external computer devices via a network connection. When the computer program is executed by the processor, it implements the steps in the super-resolution model processing method described above.

[0161] In some embodiments, a computer device is provided, which may be a computer device, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external computer devices; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements the steps in the super-resolution model processing method described above. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0162] Those skilled in the art will understand that Figure 7 or Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0163] In some embodiments, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0164] In some embodiments, such as Figure 9 The diagram shows the internal structure of a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps in the above-described method embodiments.

[0165] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0167] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0168] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A super-resolution model processing method, characterized in that, include: Determine the model parameters and channel selection parameters of the super-resolution model; The model parameters and channel selection parameters are optimized through multiple rounds of iterative alternation to obtain the target model parameters and target channel selection parameters; A target super-resolution model is determined based on the target model parameters and the target channel selection parameters; wherein, the target channel selection parameters are used to trim the model dictionary of the super-resolution model to obtain a lightweight target super-resolution model that has the target model parameters. The step of performing multiple rounds of iterative optimization on the model parameters and the channel selection parameters to obtain the target model parameters and the target channel selection parameters includes: In each round of alternating optimization, the initial channel selection parameters of this round are fixed, and the initial model parameters of this round are linearly transformed to obtain the optimized model parameters of this round; the optimized model parameters of this round are fixed, and channel selection parameters that meet the preset sparsity requirements are searched among the initial channel selection parameters of this round to obtain the optimized channel selection parameters of this round. The optimized model parameters and channel selection parameters of this round are used as the initial model parameters and channel selection parameters for the next round to continue alternating optimization until the alternating optimization stopping condition is met, thereby obtaining the target model parameters and target channel selection parameters.

2. The method according to claim 1, characterized in that, The method further includes: The target super-resolution model is used to determine the filter tensor data corresponding to the low-resolution image; the filter tensor data refers to the tensor data used for image filtering. Determine the image data to be filtered corresponding to the low-resolution image; Elements with the same index in each channel of the filtered tensor data and the image data to be filtered are assigned to the same thread, and elements with adjacent indexes in each channel are assigned to consecutive threads. The thread performs filtering processing on the image data to be filtered based on the filtering tensor data to obtain the super-resolution result output by the target super-resolution model.

3. The method according to claim 1, characterized in that, The linear transformation of the initial model parameters to obtain the optimized model parameters includes: The initial model parameters are iteratively optimized and adjusted in the direction of reducing the first objective loss value to obtain the optimized model parameters for this round. Wherein, the first target loss value is used to characterize the difference between the super-resolution result predicted by the super-resolution model and the reference super-resolution result when the initial channel selection parameters of the current round remain unchanged; the super-resolution result predicted by the super-resolution model is the super-resolution image obtained by super-resolution inference prediction of the sample low-resolution image; the reference super-resolution result is the super-resolution image pre-set for the sample low-resolution image.

4. The method according to claim 3, characterized in that, The model parameters are optimized by optimizing the linear transformation coefficients corresponding to the model parameters; the model parameters of the super-resolution model remain unchanged during each round of alternating optimization. The target model parameters are determined based on the target linear transformation coefficients obtained after alternating optimization stops; The process involves iteratively optimizing and adjusting the initial model parameters in this round in the direction of reducing the first objective loss value, resulting in the optimized model parameters for this round, including: The initial linear transformation coefficients of this round are optimized in the direction of reducing the loss value of the first objective, resulting in the optimized linear transformation coefficients of this round. In this round of alternating optimization, the linear transformation coefficients generated are used to adjust the sample dictionary weight coefficients obtained by the super-resolution model through regression analysis of the sample low-resolution images; the adjusted sample dictionary weight coefficients are used to determine the transformation tensor data from the model dictionary after being clipped by the initial channel selection parameters of this round; the transformation tensor data is used to transform the sample low-resolution images to obtain the super-resolution result of the inference prediction.

5. The method according to claim 1, characterized in that, The process of searching for channel selection parameters that meet the preset sparsity requirements from the initial channel selection parameters of this round to obtain the optimized channel selection parameters for this round includes: The initial channel selection parameters for this round are iteratively optimized and adjusted in the direction of reducing the loss value of the second objective, so as to obtain the optimized channel selection parameters for this round. Wherein, the second target loss value is used to characterize the difference between the super-resolution result predicted by the super-resolution model inference and the reference super-resolution result when the model parameters after this round of optimization remain unchanged; when the model parameters after this round of optimization remain unchanged, the super-resolution result predicted by inference changes with the channel selection parameters.

6. The method according to claim 5, characterized in that, The second target loss value is determined based on the prediction difference value and the regularization value; the prediction difference value is the difference between the super-resolution result predicted by inference and the reference super-resolution result when the model parameters remain unchanged after this round of optimization; the regularization value is determined based on the coefficient used to characterize the sparsity requirement and the channel selection parameter generated during this round of alternating optimization.

7. The method according to claim 1, characterized in that, The method further includes: Deploy the target super-resolution model to the target hardware; During each round of iterative search for the thread block size of the target hardware, a proxy model is used to fit the sample thread block dataset to obtain the fitted proxy model; the sample thread block dataset includes the sample thread block size and the model inference latency corresponding to the sample thread block size; Based on the fitted proxy model, determine the new sample thread block size; Based on the new sample thread block size, the target super-resolution model is run on the target hardware to obtain the model inference latency corresponding to the new sample thread block size. The new sample thread block size and the corresponding model inference latency are then added to the sample thread block dataset to continue iterative searching. After the iterative search is completed, the target super-resolution model is run on the target hardware based on the target thread block size; the target thread block size is determined from multiple sample thread block sizes based on the model inference latency.

8. A super-resolution model processing device, characterized in that, include: The first determining module is used to determine the model parameters and channel selection parameters of the super-resolution model; An optimization module is used to perform multiple rounds of iterative and alternating optimization on the model parameters and the channel selection parameters to obtain target model parameters and target channel selection parameters; The second determining module is used to determine the target super-resolution model based on the target model parameters and the target channel selection parameters; wherein, the target channel selection parameters are used to trim the model dictionary of the super-resolution model to obtain a lightweight target super-resolution model that has the target model parameters; In terms of performing multiple rounds of iterative optimization on the model parameters and the channel selection parameters to obtain the target model parameters and target channel selection parameters, the optimization module is specifically used for: In each round of alternating optimization, the initial channel selection parameters of this round are fixed, and the initial model parameters of this round are linearly transformed to obtain the optimized model parameters of this round; the optimized model parameters of this round are fixed, and channel selection parameters that meet the preset sparsity requirements are searched among the initial channel selection parameters of this round to obtain the optimized channel selection parameters of this round. The optimized model parameters and channel selection parameters of this round are used as the initial model parameters and channel selection parameters for the next round to continue alternating optimization until the alternating optimization stopping condition is met, thereby obtaining the target model parameters and target channel selection parameters.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images