Model generation method and electronic equipment

Abstract

The invention provides a model generation method and electronic equipment. Firstly, an initial model is determined, and sub-modules in a pre-processing module and a post-processing module of the initial model are in direct-feed connection, for example, the pre-processing module comprises N convolution blocks (N is greater than or equal to 2) which are sequentially in direct-feed connection. Then, a virtual path is constructed for the initial model, the virtual path includes that outputs of part of or all sub-modules in the pre-processing module are accumulated and then connected to a virtual post-processing module, the sub-module in the pre-processing module at least comprises a non-last sub-module in the pre-processing module, and the virtual post-processing module is the same as the post-processing module; and finally, training the initial model by adopting the weighted sum of the model loss and the virtual path loss, gradually updating the weight in the model until the weighted sum of the new model loss and the virtual path loss meets a preset condition, and applying the new model in the NPU. In this way, the lightweight model with a simple structure is generated, and the accuracy of forward inference of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to a model generation method and an electronic device. Background Art

[0002] The scale of the model is crucial for terminal applications. When the model scale is too large, it will bring problems such as high memory usage, high computational complexity, and prolonged forward inference time. Especially for low- and medium-performance terminals, complex models will seriously hinder their application in real scenarios.

[0003] How to design a lightweight model is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a model generation method and electronic device. The method includes first constructing a simple direct-feedback initial model, then constructing a virtual path for the model, and training the initial model by combining the loss of the model and the loss of the virtual path to obtain a final updated simple direct-feedback initial model.

[0005] In a first aspect, the present application provides, for example, a model generation method, the method comprising: generating an initial model, the initial model comprising a pre-processing module and a post-processing module, the pre-processing module and the post-processing module both comprising a plurality of sub-modules that are directly connected in sequence; generating a virtual path of the initial model, the virtual path comprising: an accumulation module and a virtual post-processing module; the input of the accumulation module is a first input, the first input is the same as the output of one or more sub-modules in the pre-processing module, the output of the accumulation module is the input of the virtual post-processing module, and the structure of the virtual post-processing module is the same as that of the post-processing module; updating the initial model and the virtual path based on the loss of the initial model and the loss of the virtual path until the total loss of the updated model and the virtual path is less than a first threshold.

[0006] Implementing the method provided in the first aspect can not only generate a lightweight model, but also improve the model training effect and ensure the output performance of the lightweight model.

[0007] In combination with the method described in the first aspect, the accumulation module is connected to one or more sub-modules in the pre-processing module, and the first input is the output of one or more sub-modules in the pre-processing module; or, the virtual path also includes a virtual pre-processing module, which has the same structure as part or all of the pre-processing module, and the accumulation module is connected to one or more sub-modules in the virtual pre-processing module, and the first input is the output of one or more sub-modules in the virtual pre-processing module.

[0008] In this way, virtual paths can be constructed in a variety of ways, including reusing some or all of the sub-modules in the model, or independently constructing virtual paths that are identical to some or all of the sub-modules in the model, providing a variety of implementable methods.

[0009] In combination with the method described in the first aspect, the initial model is updated based on the loss of the initial model and the loss of the virtual path, specifically including: updating the weights of the sub-modules in the initial model based on the loss of the initial model and the loss of the virtual path, specifically including updating the weights of each sub-module in the pre-processing module and the post-processing module.

[0010] In this way, by updating the weights of the sub-modules in the model during the training process, the output performance of the sub-modules in each intermediate state of the model can be optimized, thereby improving the overall output performance of the model.

[0011] In combination with the method described in the first aspect, the initial model is updated based on the loss of the initial model and the loss of the virtual path, and the virtual path is updated until the total loss of the updated model and the virtual path is less than a first threshold, specifically including: updating the initial model and updating the virtual path based on the loss of the initial model and the loss of the virtual path until the weighted sum of the loss of the updated model and the loss of the virtual path is less than the first threshold.

[0012] In this way, the model is updated by combining the weighted sum of the model loss and the virtual path loss, and the weights corresponding to the two losses can be set according to actual needs.

[0013] In combination with the method described in the first aspect, the weight of the loss of the updated model is greater than the weight of the loss of the virtual path.

[0014] In this way, the model loss can be focused on, and the virtual path loss can be considered to a lesser extent to update the model, thereby providing additional supervision of the pre-processing module in the model to a certain extent.

[0015] In combination with the method described in the first aspect, the method is applied to the first device, and the method also includes: storing the updated model in the second device.

[0016] In this way, after the model is generated in the first device at the local end, the model can also be applied in more second devices to expand the coverage of the model.

[0017] In combination with the method described in the first aspect, the method further includes: inputting a first image into the updated model, and outputting a second image after processing by the updated model, wherein a resolution of the second image is higher than a resolution of the first image.

[0018] In this way, the existing model in the field of image processing can be optimized. In addition, the model generation method provided by this application can also be used in other fields.

[0019] In combination with the method described in the first aspect, the submodule in the pre-processing module includes at least two convolution blocks; the first convolution block of the two convolution blocks is used to extract features from the image; and the second convolution block of the two convolution blocks is used to extract features from the output of the first convolution block.

[0020] In a second aspect, the present application provides an electronic device, comprising: a model, a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method as described in any one of the first aspects.

[0021] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects.

[0022] In a fourth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the method described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the structure of a complex model provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a lightweight model provided in an embodiment of the present application; Figure 3 A schematic diagram of the effect of using the loss function equivalent optimization model structure provided in the embodiment of the present application; Figure 4 A schematic diagram of the principle of training a lightweight model provided in an embodiment of the present application; Figure 5 A schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present application; Figure 6 A schematic diagram of the software architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. Mentioning "embodiment" in the present application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application may be combined with other embodiments.

[0025] The complexity of the model greatly affects its application effect on the terminal. A complex model will increase the time consumption and power consumption of the forward inference on the terminal, resulting in fewer terminal types that can be covered by the model, which seriously hinders the application of the model in more scenarios. Among them, the forward inference of the terminal refers to the process in which the terminal uses the model to process data, including the entire process from receiving input data to generating output data. The time consumption and power consumption of this process directly affect the terminal response speed and user experience. Therefore, the complexity of the model is crucial for real-time applications on the terminal.

[0026] In order to design a lightweight model, in a feasible way, the model can be simplified by reducing the number of parameters and calculations in the model, improving its forward inference performance on the terminal, so that it can be applied to more types of terminals and cover more application scenarios. However, there are limits to the optimization of the number of parameters and calculations in the model, and how to further lightweight the model is an urgent problem to be solved.

[0027] By comparing and analyzing models with similar parameter amounts and calculation amounts and different model structures, the present application can find that the model structure will also seriously affect the time and power consumption of the forward inference of the model at the mid-end. Therefore, by designing the model structure, the limitations of optimizing lightweight models can be further broken through. Based on this, in order to further lightweight the model, the present application provides a model generation method and an electronic device, the method comprising: first, determining the initial model, each submodule in the pre-processing module and the post-processing module of the initial module adopts a direct feed connection, for example, the pre-processing module includes N convolution blocks (N≥2) that are directly fed back in sequence. Then, a virtual path is constructed for the initial model, the virtual path includes the output of some or all submodules in the pre-processing module after accumulation and then connected to the virtual post-processing module, some submodules in the pre-processing module at least include the non-last submodule in the pre-processing module, and the virtual post-processing module is the same as the post-processing module. Finally, the initial model is trained using the weighted sum of the model loss and the virtual path loss, and the weights in the model are gradually updated until the weighted sum of the new model loss and the virtual path loss meets the preset conditions, and the new model is applied to the NPU. In this way, a lightweight model with a simple structure is generated and the accuracy of forward inference is improved.

[0028] The type of task performed by the model in the embodiment of the present application is specifically to output an image after super-resolution processing.

[0029] The pre-processing module and post-processing module of the model described in this application do not refer to specific entity sub-modules, but are general terms for dividing the entity sub-modules in the model from a functional perspective. Among them, the pre-processing module is mainly responsible for performing necessary pre-processing on the original input data, such as feature extraction and other operations, to ensure that the data is suitable for subsequent processing of the model. Among them, the post-processing module is mainly used to further process the output results of the model to obtain a high-resolution image. In an embodiment of the present application, the image input to the lightweight super-resolution model can also be referred to as the first image, and the image output after the lightweight super-resolution model performs super-resolution processing on the first image can also be referred to as the second image.

[0030] The model generation method provided in this application can bring the following beneficial effects: (1) Providing a lightweight model with a simple structure. Specifically, the determined initial model and the final model obtained after training the initial model only include multiple submodules using a direct feed connection method. Compared with the existing complex structure model, additional residual connections, merging (contact) operations, etc. are removed.

[0031] (2) Improve the training effect of lightweight models. Specifically, in the process of training the model, avoid the drop in model performance caused by removing complex connection structures. By constructing a virtual path, the model is trained together with the loss of the virtual path and the loss of the model. Considering that the loss of the virtual path can supervise multiple sub-modules in the model that are input to the virtual path, avoid the intermediate state offset of the model, thereby playing an additional supervision role in the model training process and improving the model training effect.

[0032] (3) Expand the application scenarios of the model and improve the application effect of the model. Specifically, by simplifying the structure of the model, the time and power consumption of the model's forward inference on the terminal can be reduced, covering more terminal types and expanding the application scenarios of the model.

[0033] Next, the structure of the complex model before optimization, the structure of the optimized lightweight model provided by this application, the effect of achieving the equivalent optimization model structure by designing the loss function, and the principle of training the lightweight model will be introduced in detail.

[0034] Figure 1 The structure of a complex model is shown as an example.

[0035] Figure 1 Specifically, we take the model for implementing image super-resolution processing (referred to as super-resolution model) as an example to illustrate a complex model. Figure 1As shown, the pre-processing module of the complex model includes convolution block 1, convolution block 2 and convolution block 3, and the post-processing module of the complex model includes convolution, truncated activation function and upsampling module (such as PixelShuffle). When designing a model, it is usually necessary to design the basic component modules of the model, including the number of input channels, the number of output channels and the size of the convolution kernel of the convolution block, and the number of modules needs to be determined, which will determine the number of parameters and the amount of calculation of the model. After designing the basic component modules of the model, it is also necessary to design the connection structure of these modules. Introducing connection structures other than direct feed connections can enhance the representation ability of the model, but it will also bring additional computing operations, such as introducing residual connections, concat operations, etc. This application does not impose specific restrictions on the number of component modules in the model, and only focuses on the connection structure between each module.

[0036] Among them, the pre-processing module is located at the input end of the model. The pre-processing module refers to the module that performs pre-processing operations on the input data, which usually includes the following pre-processing operations: data expansion, which improves the generalization ability of the model by increasing the number and diversity of data sets; feature extraction, which selects features that are useful for model training from the original features and removes irrelevant or redundant features.

[0037] The post-processing module is located at the output end of the model. The post-processing module is to optimize the output data of the model so that it can be better applied to actual scenarios. It usually includes the following optimization operations: feature extraction, extracting useful information from the output of the model for further analysis; data post-processing, constraining the data within a certain range to reduce the model quantization error.

[0038] In practical applications, the specific implementation of pre-processing and post-processing modules will vary depending on the specific tasks and data characteristics of the model. For example, in the image super-resolution task, pre-processing may include operations such as image scaling and cropping.

[0039] Continue to see Figure 1 ,Convolutional blocks 1, 2, and 3 in the pre-processing module adopt residual connections.

[0040] Among them, residual connection (also known as skip connection) means that the output of the previous layer or layers skips to the middle layer or layers, and is accumulated to the output of the next layer to form a residual path. In deep models, the residual connection method is more complicated and increases the amount of calculation of the model, but the residual connection can be used to solve the problem of gradient disappearance and gradient explosion. Specifically, in Figure 1In the model structure shown, the residual connection can be used to directly output the output of convolution block 1 across the middle convolution blocks 2 and 3 to the output of convolution block 3 for accumulation, and also to directly output the output of convolution block 2 across the middle convolution block 3 to the output of convolution block 3 for accumulation, which helps to solve the problem of gradient vanishing and gradient exploding in deep convolution. Optionally, there are various derivative schemes of residual connection in model design. Figure 1 In the example, convolution block 1 spans multiple convolution blocks and convolution block 2 spans one convolution block. In other residual connection methods, there may be different connection methods, such as connecting additional modules on the residual path. Although these connection methods usually do not generate additional model parameters, they will bring additional calculations and will inevitably increase the forward inference delay of the model when it is deployed on the mobile terminal. In addition, Figure 1 The concat operation, not shown in the figure, requires structural changes to specific convolutional layers to make it work. It will directly modify the number of input channels of the convolutional layer connected to it, thus introducing additional computational overhead and inevitably increasing the forward inference latency of the model when deployed on mobile devices.

[0041] Among them, each convolution block in convolution block 1-convolution block 3 contains one or more convolution layers and one or more activation function modules, and each convolution layer is connected to an activation function. Among them, the convolution layer is the core component of the convolution block, which is responsible for extracting local features from the input data. Specifically, the convolution layer processes the input data by applying a filter (or called a convolution kernel) to extract local features. Each convolution layer can contain one or more convolution kernels, and the size (such as 3x3, etc.), step size and padding method of these convolution kernels can be set according to specific needs. The result of the convolution operation of the convolution layer is a feature map. Each pixel value of the feature map is the weight of the filter multiplied by the pixel value of the corresponding position of the input image, and then all the product results are added to obtain a new pixel value. Among them, the role of the activation function is to introduce nonlinear transformation. Common activation functions include ReLU, Sigmoid and Tanh, which enable the model to learn more complex feature representations. In short, the convolution block is a structural unit composed of multiple convolution layers and activation functions. Through the combination of multi-layer convolution and nonlinear transformation, the complex features of the input data are further extracted. Taking the super-resolution model as an example, convolution block 1, convolution block 2 and convolution block 3 can be used to extract image features.

[0042] Continue to see Figure 1 ,The convolution, truncated activation function and upsampling module in the post-processing module adopt direct feed connection in sequence.

[0043] Among them, direct feed connection means that the output of the previous layer is directly used as the input of the next layer, and the output of each layer is calculated based on the output of the previous layer without the intervention of the middle layer. In deep models, direct feed connection is simple and direct, and it is the basis for building multi-layer models. However, direct feed connection may cause gradient vanishing or gradient exploding problems, thus affecting the inference effect of the model.

[0044] Among them, the convolution in the post-processing module is used to perform channel transformation on the feature map output by the pre-processing module, and transform it into a number of channels that is convenient for subsequent upsampling processing. The truncation activation function in the post-processing module is used to directly truncate the output of the model to between 0 and 1 in the process of quantizing the original 32-bit floating-point weights and responses into 8-bit integer weights and responses, thereby reducing the error during model quantization. The upsampling module in the post-processing module is used to upsample the image to achieve the final high-resolution effect.

[0045] Understandably, Figure 1 The structure of the complex model shown is only an example. The complex model before optimization may include other connection methods such as concat connection in addition to direct feed connection and residual connection. Figure 1 For the time being, only a complex model including direct feed connection and residual connection is used as an example to illustrate. In addition, in the structure of other optional complex models, the number of convolution blocks can be more or less, and the back end can also have other functional modules. The embodiment of the present application does not limit the connection structure, number of modules, etc. in the model.

[0046] Figure 2 The structure of a lightweight model is shown as an example.

[0047] Figure 2 A lightweight model is also specifically described by taking a model for implementing image super-resolution processing (referred to as super-resolution model) as an example. Figure 2 As shown, the pre-processing module of the lightweight model includes convolution block 1, convolution block 2 and convolution block 3, and the post-processing module of the lightweight model includes convolution, truncated activation function and upsampling module. Figure 2 The basic building blocks of the lightweight model shown are Figure 1 The basic building blocks shown are the same. Figure 1 The structure is simplified based on the complex model shown in the figure. For details about the structural simplification method, please refer to the following article. Figure 3-Figure 4 Detailed introduction.

[0048] Continue to see Figure 2 , the convolution blocks 1, 2 and 3 in the pre-processing module are connected in direct feed-forward mode, and the convolution, truncated activation function and up-sampling modules in the post-processing module are also connected in direct feed-forward mode. Figure 2 In the model structure shown, direct feed connection means that the output of the previous layer is directly used as the input of the next layer, and the output of each layer is calculated based on the output of the previous layer without the intervention of the intermediate layer. In deep models, direct feed connection is simple and direct, and is the basis for building multi-layer models. However, direct feed connection may cause gradient vanishing or gradient exploding problems, thus affecting the inference effect of the model.

[0049] Among them, the convolution in the post-processing module is used to perform channel transformation on the feature map output by the pre-processing module, and transform it into a number of channels that is convenient for subsequent upsampling processing. The truncation activation function in the post-processing module is used to directly truncate the output of the model to between 0 and 1 in the process of the model quantizing the original 32-bit floating-point weights to 8-bit integer weights, thereby reducing the error during model quantization. The upsampling module in the post-processing module is used to upsample the image to achieve the final high-resolution effect. Among them, the core idea of ​​model quantization is to reduce the accuracy of model parameters from a higher bit width (such as FP32) to a lower bit width (such as int8). The main purpose of doing so is to reduce the size of the model and increase the inference speed. In the quantization process, data type conversion is usually involved, that is, the original floating-point tensor is converted into an integer tensor for processing. Specifically, 8-bit quantization compresses the weights and activation values ​​that originally require 32 bits of storage space to only 8 bits for storage, thereby significantly reducing the size of the model and memory usage. This quantization method can not only reduce the model size and improve the inference speed, but also adapt to edge computing devices, because the microprocessors on some terminals usually have low power requirements and the hardware accelerators only support int8.

[0050] Understandably, Figure 2 The structure of the lightweight model shown is only an example. In other optional lightweight model structures, the number of convolution blocks can be more or less, and the back end can also have other functional modules. The embodiments of the present application do not limit the connection structure, number of modules, etc. in the model.

[0051] The previous article only combines Figure 1 and Figure 2 This paper introduces the structure of the complex model before optimization and the structure of the lightweight model after optimization. Next, it introduces in detail how to achieve the effect of equivalent optimization model structure by designing loss function and the principle of training lightweight models.

[0052] Figure 3 The effect of using the loss function to equivalently optimize the model structure is shown by example.

[0053] Figure 3The two effects brought about by model optimization are specifically shown. The lightweight model provided in this application can have any one or both of the following effects: compared with the lightweight model trained with the original loss function, the lightweight model optimized with the new loss function can improve the output performance of the model; compared with the complex model, the lightweight model optimized with the new loss function has a simpler structure and its output performance is close to that of the complex model.

[0054] by Figure 3 Taking effect 1 as an example, when a virtual path is constructed and a lightweight model (i.e. Figure 2 The output performance of the model shown in the figure) is better than the output performance of the lightweight model obtained by training without constructing a virtual path and using the original loss function L, which shows that constructing a virtual path and using a new loss function can improve the training effect of the lightweight model. Therefore, a virtual path is constructed and a new loss function is used to train the lightweight model. Among them, the new loss function includes the loss of the lightweight model and the loss of the virtual path, which can be expressed as: Ls=1*L(model output 0, target)+w*L(virtual output 0, target). Among them, the original loss function of the lightweight model only includes the loss of the lightweight model, which can be expressed as: L(model output 1, target). Regarding the specific implementation method of using the new loss function to train the lightweight model, please refer to the following description Figure 4 The description of is not repeated here.

[0055] by Figure 3 As an example, when a virtual path is constructed and a lightweight model (i.e. Figure 2 The output performance of the model shown in the figure is compared with the output performance of the complex model trained using the original loss function L (i.e. Figure 1 If the difference between the model shown in the figure is less than the threshold, it means that the performance of the lightweight model after the simplified structure is close to that of the complex model before the simplified structure. Therefore, the structure of the complex model is simplified and the simplified lightweight model is applied. Among them, the original loss function of the complex model only contains the loss of the complex model, which can be expressed as: L(model output 2, target).

[0056] It can be seen that satisfying the above two model optimization effects is equivalent to simplifying the structure of the complex model and using the new loss function to train the simplified lightweight model. This can achieve the purpose of simplifying the model structure while ensuring the model performance, thereby meeting the needs of model simplification.

[0057] Understandably, Figure 3The two effects shown can be obtained after the lightweight model is trained with the new loss function, and are compared with other models. They do not constitute a necessary stage in the model generation method described in this application.

[0058] Figure 4 The principle of training a lightweight model is illustrated by way of example.

[0059] Based on the previous Figure 1-Figure 2 From the introduction of the complex model before optimization and the lightweight model before optimization shown in , we can see that the lightweight model has a simpler structure than the complex model. Therefore, before training the lightweight model, we need to design a lightweight model with a simple structure as the initial model to be trained. Since the model with a simple structure may have performance problems, based on this, when training based on the initial model, we also construct a virtual path and combine the loss of the virtual path to provide additional supervision on the initial model.

[0060] like Figure 4 As shown, the architecture for training a lightweight model includes an initial model (also known as the main path) and a virtual path.

[0061] The initial model is the same as the one introduced above. Figure 2 The basic components of the lightweight model shown in the figure are the same, and the connection method is also the same. The difference is the weight of each submodule in the model. After the initial model is trained and the weight is updated, Figure 2 The final lightweight model shown. Optionally, the initial model can also be other models with simple structures, which only need to meet the condition of a direct-feedback network composed of multiple sub-modules. For example, the pre-processing module in the initial model includes N convolution blocks (N≥2) that are directly fed back in sequence, and the last convolution block is connected to the post-processing module of the initial model. The post-processing module in the initial model includes M sub-modules (M≥1) that are directly fed back in sequence. The number and functions of the M sub-modules are related to the specific tasks of the model. The embodiments of the present application do not limit the number of convolution blocks in the pre-processing module and the number and type of functional modules in the post-processing module.

[0062] Among them, the virtual path is a path constructed in order to avoid introducing other connection structures and only maintain the main path as the structure of the model, which is used to play a role similar to other connection structures. The virtual path includes: an accumulation module and a virtual post-processing module. The accumulation module is used to add the outputs of convolution block 1, convolution block 2 and convolution block 3 in the pre-processing module of the initial model and output them to the virtual post-processing module. The virtual post-processing module is the same as the post-processing module in the initial module. Optionally, the virtual path can also be a path of other structures. For example, the accumulation module in the virtual path is used to add the outputs of all or part of the convolution blocks in the pre-processing module and then connect them to the virtual post-processing module. The embodiment of the present application does not limit the number of outputs of the convolution blocks that need to be accumulated. Optionally, in addition to multiplexing, the input of the accumulation module in the virtual path Figure 4 The outputs of multiple submodules in the pre-processing module in the initial model shown, and the input of the accumulation module in the virtual path can also construct a virtual pre-processing module separately, and the virtual pre-processing module has the same partial or complete results as the pre-processing module in the model, where the same part specifically depends on what the input of the accumulation module contains. In the embodiment of the present application, the input of the virtual path can also be referred to as the first input.

[0063] Continue to see Figure 4 After establishing the initial model (i.e., the main path) and constructing a virtual path for the initial model, a new loss function is required to train the initial model to update the weights in the model until the new loss is minimized or certain conditions are met, and the training is completed.

[0064] Specifically, in the use of Figure 4In the process of training the model using the training architecture shown, not only the loss function is used to calculate the loss of the main path, but also the loss function is used to calculate the loss of the virtual path. The model is trained together by combining the loss of the main path and the loss of the virtual path. Taking a specific example, when the loss function of the model is L, the output of the model (the output of the main path) is recorded as "model output", the output of the virtual path is "virtual output", and the output in the ideal case is "target (or target)", then the loss of the main path can be expressed as L (model output, target), the loss of the virtual path can be expressed as L (virtual output, target), and the loss of the entire training architecture can be expressed as Ls = 1 * L (model output, target) + w * L (virtual output, target). Among them, the target is the label corresponding to the input data in the training data pair, and w is the weight of the loss of the virtual path, which is used to balance the importance of the virtual path in the entire training architecture. When w is equal to 0, the virtual path is not effective. At this time, it is equivalent to having only the original main path in the training architecture; when w is equal to 1, the virtual path and the main path are effective at the same time and have the same effect. After determining w and obtaining Ls, Ls can be used to directly start the training phase to obtain the weights in the updated model, usually w ≥ 0. In this way, in addition to considering the hierarchical output of convolution block 1 through convolution block 2 and then through convolution block 3 in the main path to a large extent during the training process, the outputs of convolution block 1, convolution block 2 and convolution block 3 are also considered in the virtual path to a certain extent.

[0065] In this way, since the loss used in the training process takes into account the outputs of each of the convolution blocks 1-3 contained in the virtual path, this is similar to the direct jump-type transmission output of the convolution block 1-convolution block 3 under the residual connection mode. It is equivalent to supervising the intermediate states such as the convolution block 1-convolution block 3 to avoid the intermediate state offset. After the training is completed, the virtual path is removed, and only the main path is used as the trained model. It can be seen that the training method of the lightweight model provided in the present application is equivalent to achieving the purpose of simplifying the model structure by means of the design of the loss function.

[0066] Next, the product form, software and hardware architecture, etc. of the terminal deployed by the lightweight model described in this application are introduced.

[0067] The terminals on which the lightweight model can be deployed are also called electronic devices, such as those deployed on mobile phones.

[0068] Figure 5 The hardware architecture of the electronic device provided by the present application is exemplified.

[0069] like Figure 5As shown, the electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a sensor module 180, a camera 193, a display screen 194, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180K, etc.

[0070] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0071] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a memory and a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, since the computing power of the NPU is relatively low, applying a lightweight model to the NPU can greatly speed up the output efficiency of the model. Therefore, the lightweight model provided in this application can be specifically applied to the NPU of the terminal.

[0072] In the embodiment of the present application, the electronic device can execute the following Figure 2 The lightweight model shown in the figure can realize super-resolution processing of the input image. The lightweight model of the electronic device can be generated by the electronic device and then stored, or generated by other electronic devices and then pre-set in the electronic device. The embodiment of the present application does not impose specific restrictions on the electronic device that generates the lightweight model and applies the lightweight model. Whether it is generated by the electronic device at the application end or by other electronic devices, the generation method is the same as described above. Figure 3-Figure 4 The methods introduced will not be described in detail here.

[0073] In the embodiment of the present application, the electronic device can display the original input image and the high-resolution image output after being processed by the lightweight super-resolution model through the display screen 194, and the embodiment of the present application does not limit this.

[0074] In the embodiment of the present application, the electronic device can capture an image through the camera 193. When the resolution of the captured image is low, the electronic device can use a lightweight super-resolution model to process the image to output a high-resolution image. In addition, in addition to super-resolution processing of images captured by the camera, the lightweight super-resolution model can also perform super-resolution processing on images downloaded from a browser and images from an application, which is not limited in the embodiment of the present application.

[0075] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The external memory interface 120 may be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0076] In the embodiment of the present application, the electronic device can store the above-mentioned Figure 2 The lightweight model shown, and the data output by using the lightweight model are stored.

[0077] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it.

[0078] In an embodiment of the present application, the electronic device may receive an operation for capturing an image through the pressure sensor 180A or the touch sensor 180K, or receive an operation for inputting a low-resolution image into a lightweight super-resolution model, or receive an operation for storing a high-resolution image output after being processed by a lightweight super-resolution model, etc. The embodiment of the present application does not limit this and does not elaborate.

[0079] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device.

[0080] Figure 6 The software architecture of the electronic device provided by the embodiment of the present application is exemplified.

[0081] like Figure 6As shown, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.

[0082] The application layer can include a series of application packages. Figure 6 As shown, the application package may include image processing applications, camera, gallery, calendar, call, map, navigation, Bluetooth, video, short message and other applications. Among them, the image processing application can integrate the aforementioned lightweight super-resolution model to provide image super-resolution processing for users, and this application does not limit the name of the image processing application.

[0083] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 6 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0084] Android runtime includes core libraries and virtual machines. System libraries can include multiple functional modules, such as surface manager, 3D graphics processing library, 2D graphics engine, media library, etc.

[0085] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0086] It should be understood that each step in the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiment of the present application can be directly reflected in the NPU performing the super-resolution operation of the lightweight model and cooperating with the CPU to complete the entire super-resolution process.

[0087] The present application also provides a chip system, which includes at least one processor, for implementing the method executed by the electronic device in any of the above embodiments. In a possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0088] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method executed by the electronic device in any one of the above embodiments is implemented.

[0089] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the method executed by the electronic device in any of the above embodiments.

[0090] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0091] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0092] The terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0093] In short, the above is only an embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of this application shall be included in the protection scope of this application.

Claims

1. A model generation method, characterized in that: The method comprises: Generate an initial model, the initial model includes a pre-processing module and a post-processing module, the pre-processing module and the post-processing module each include a plurality of sub-modules that are directly fed back in sequence; Generate a virtual path of the initial model, the virtual path comprising: an accumulation module and a virtual post-processing module; the input of the accumulation module is a first input, the first input is the same as the output of one or more sub-modules in the pre-processing module, the output of the accumulation module is the input of the virtual post-processing module, and the structure of the virtual post-processing module is the same as that of the post-processing module; The initial model is updated based on the loss of the initial model and the loss of the virtual path until the total loss of the updated model and the virtual path is less than a first threshold.

2. The method according to claim 1, characterized in that: The accumulation module is connected to one or more submodules in the pre-processing module, and the first input is the output of one or more submodules in the pre-processing module; Alternatively, the virtual path also includes a virtual pre-processing module, which has a partial or complete structure identical to that of the pre-processing module, the accumulation module is connected to one or more sub-modules in the virtual pre-processing module, and the first input is the output of one or more sub-modules in the virtual pre-processing module.

3. The method according to claim 1 or 2, characterized in that: Updating the initial model based on the loss of the initial model and the loss of the virtual path specifically includes: updating the weights of submodules in the initial model based on the loss of the initial model and the loss of the virtual path.

4. The method according to claim 1, characterized in that: Updating the initial model based on the loss of the initial model and the loss of the virtual path until the total loss of the updated model and the virtual path is less than a first threshold, specifically comprising: updating the initial model based on the loss of the initial model and the loss of the virtual path until the weighted sum of the loss of the updated model and the loss of the virtual path is less than a first threshold.

5. The method according to claim 4, characterized in that The weight of the loss of the updated model is greater than the weight of the loss of the virtual path.

6. The method according to claim 1, characterized in that The method is applied to a first device, and further comprises: storing the updated model in a second device.

7. The method according to claim 1, characterized in that The method further comprises: A first image is input to the updated model, and a second image is output after being processed by the updated model, wherein the resolution of the second image is higher than the resolution of the first image.

8. The method according to claim 1, characterized in that: The submodule in the pre-processing module includes at least two convolution blocks; The first convolution block of the two convolution blocks is used to extract features from the image; The latter convolution block of the two convolution blocks is used to extract features from the output of the previous convolution block.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images