Picture processing model training method and picture processing method

Abstract

The embodiment of the specification provides a picture processing model training method and a picture processing method. The picture processing model training method is applied to a model training system. The system comprises a terminal and a server. The method deploys a processing module of a picture processing model on at least two hardware of the terminal. In the case that a target running time of initial picture processing satisfies a preset time condition, it is determined that the picture processing performance of the picture processing model is relatively optimal when the picture processing model is deployed on the terminal in this structure. The picture processing model is sent to the server for training and deployment. When the picture processing model is used for picture processing in subsequent terminals, the performance of the picture processing model is relatively optimal while the algorithm effect is not reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of machine learning, in particular to a picture processing model training method. BACKGROUND

[0002] Deep neural networks have become a new technology for classification and processing of high-dimensional data such as images and videos, and are widely used in the technical fields of image processing and video processing.

[0003] Applying a deep neural network model to the super-resolution (converting low-resolution videos or images into high-resolution videos or images through algorithms) scenario is very difficult to optimize the performance of the deep neural network model for implementing super-resolution due to the characteristics of high real-time, large computation, and large features.

[0004] Currently, it is a great difficulty to optimize the performance of the deep neural network model for implementing super-resolution on the terminal while maintaining the algorithm effect. SUMMARY

[0005] Therefore, the embodiments of the present specification provide a picture processing model training method. One or more embodiments of the present specification also relate to a picture processing model training system, a picture processing model, two picture processing methods, two picture processing devices, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects in the prior art.

[0006] According to a first aspect of the embodiments of the present specification, a picture processing model training method is provided, applied to a model training system, the system comprising a terminal and a server, the method comprising:

[0007] The terminal, deploying at least two hardware processing modules of a picture processing model respectively, processing an initial picture according to the processing modules to obtain a target running time of a target picture output by the processing modules, wherein the resolution of the target picture is greater than that of the initial picture,

[0008] In a case where it is determined that the target running time meets a preset time condition, the processing module is sent to the server;

[0009] The server trains the received processing module to obtain the picture processing model, and sends the picture processing model to the terminal.

[0010] According to a second aspect of the embodiments of the present specification, a picture processing model training system is provided, comprising a terminal and a server, wherein,

[0011] The terminal is configured to process an initial picture according to the processing modules to obtain a target running time of a target picture output by the processing modules, wherein a resolution of the target picture is greater than that of the initial picture,

[0012] In a case where it is determined that the target running time meets a preset time condition, the processing modules are sent to the server;

[0013] The server is configured to train the received processing modules to obtain the picture processing model, and send the picture processing model to the terminal.

[0014] According to a third aspect of an embodiment of the present specification, a picture processing model is provided, comprising a first processing module, a second processing module, and a third processing module, wherein,

[0015] The first processing module comprises at least one first convolution module and at least one second convolution module, and the first processing module and the third processing module are deployed on a first hardware of a terminal;

[0016] The second processing module comprises a down-sampling module, an up-sampling module, and a second processing submodule, and the second processing module is deployed on a second hardware of the terminal, the first hardware being different from the second hardware, wherein the picture processing model is a model obtained by the above picture processing model training method.

[0017] According to a fourth aspect of an embodiment of the present specification, a picture processing method applied to a terminal is provided, comprising:

[0018] An initial picture is input into a first processing module and a second processing module of a picture processing model, wherein the first processing module and the second processing module run on different hardware respectively;

[0019] Picture features of the initial picture output by the first processing module and feature weights of the picture features output by the second processing module are obtained;

[0020] The initial picture, the picture features, and the feature weights of the picture features are input into a third processing module of the picture processing model to obtain a target picture corresponding to the initial picture,

[0021] wherein a resolution of the target picture is greater than that of the initial picture, and the picture processing model is a model obtained by the above picture processing model training method.

[0022] According to a fifth aspect of an embodiment of the present specification, a picture processing device applied to a terminal is provided, comprising:

[0023] The first input module is configured to input an initial picture into a first processing module and a second processing module of a picture processing model respectively, wherein the first processing module and the second processing module run on different hardware respectively;

[0024] The first obtaining module is configured to obtain picture features of the initial picture output by the first processing module and feature weights of the picture features output by the second processing module;

[0025] The second input module is configured to input the initial picture, the picture features and the feature weights of the picture features into a third processing module of the picture processing model to obtain a target picture corresponding to the initial picture,

[0026] wherein the target picture has a resolution greater than that of the initial picture, and the picture processing model is a model obtained through the training method of the picture processing model.

[0027] According to a sixth aspect of an embodiment of the present specification, a picture processing method is provided, comprising:

[0028] receiving an initial picture input by a user through a picture input interface;

[0029] inputting the initial picture into a first processing module and a second processing module of a picture processing model respectively, wherein the first processing module and the second processing module run on different hardware respectively;

[0030] obtaining picture features of the initial picture output by the first processing module and feature weights of the picture features output by the second processing module;

[0031] inputting the initial picture, the picture features and the feature weights of the picture features into a third processing module of the picture processing model to obtain a target picture corresponding to the initial picture, wherein the target picture has a resolution greater than that of the initial picture;

[0032] displaying the target picture to the user through a picture display interface, wherein the picture processing model is a model obtained through the training method of the picture processing model.

[0033] According to a seventh aspect of an embodiment of the present specification, a picture processing apparatus is provided, comprising:

[0034] a picture receiving module configured to receive an initial picture input by a user through a picture input interface;

[0035] a third input module configured to input the initial picture into a first processing module and a second processing module of the picture processing model respectively, wherein the first processing module and the second processing module are respectively run on different hardware;

[0036] a second obtaining module configured to obtain picture features of the initial picture output by the first processing module and feature weights of the picture features output by the second processing module;

[0037] a fourth input module configured to input the initial picture, the picture features and the feature weights of the picture features into a third processing module of the picture processing model to obtain a target picture corresponding to the initial picture, wherein a resolution of the target picture is greater than that of the initial picture;

[0038] a display module configured to display the target picture to the user through a picture display interface, wherein the picture processing model is a model obtained through the training method of the picture processing model.

[0039] According to an eighth aspect of an embodiment of the present specification, a computing device is provided, comprising:

[0040] a memory and a processor;

[0041] the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the training method of the picture processing model or the picture processing method.

[0042] According to a ninth aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, and the instructions, when executed by a processor, implement the steps of the training method of the picture processing model or the picture processing method.

[0043] According to a tenth aspect of an embodiment of the present specification, a computer program is provided, and when the computer program is executed in a computer, the computer is caused to execute the steps of the training method of the picture processing model or the picture processing method.

[0044] An embodiment of the present specification provides a picture processing model training method, applied to a model training system, the system comprising a terminal and a server, the method comprising: the terminal, deploying a processing module of a picture processing model on at least two hardware respectively, processing an initial picture according to the processing module to obtain a target running time of a target picture output by the processing module, wherein the resolution of the target picture is greater than that of the initial picture, and in a case where the target running time meets a preset time condition, sending the processing module to the server; the server, training the received processing module to obtain the picture processing model, and sending the picture processing model to the terminal.

[0045] Specifically, the picture processing model training method deploys a processing module of a picture processing model on at least two hardware of a terminal, and in a case where a target running time of processing an initial picture by the processing module deployed on the at least two hardware meets a preset time condition, it is determined that the picture processing model has a structure that is relatively optimal for picture processing performance when deployed on the terminal; and after the picture processing model is sent to a server for training and deployment, the picture processing model has relatively optimal performance when used for picture processing by a subsequent terminal while maintaining the algorithm effect. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a specific scene processing schematic diagram of a picture processing method provided by an embodiment of the present specification;

[0047] Figure 2 is a flowchart of a picture processing model training method provided by an embodiment of the present specification;

[0048] Figure 3 is a structure schematic diagram of a picture processing model provided by an embodiment of the present specification;

[0049] Figure 4 is a flowchart of a picture processing method applied to a terminal provided by an embodiment of the present specification;

[0050] Figure 5 is a flowchart of another picture processing method provided by an embodiment of the present specification;

[0051] Figure 6 is a picture processing model training system provided by an embodiment of the present specification;

[0052] Figure 7 is a structure schematic diagram of a picture processing device applied to a terminal provided by an embodiment of the present specification;

[0053] Figure 8is a structural schematic diagram of another picture processing apparatus provided by an embodiment of the present specification.

[0054] Figure 9 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0055] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples described herein. Those of ordinary skill in the art, and others, can readily ascertain combinations and sub-combinations of the elements of the present specification without departing from the scope of the present specification. Thus, the scope of the present specification is not to be limited by the specific details in the description.

[0056] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0057] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only to distinguish one from another. For example, without departing from the scope of one or more embodiments of the present specification, first can be termed second, and similarly, second can be termed first. The term "if' as used herein, can be interpreted as meaning "when" or "in response to determining" depending on the context.

[0058] First, the noun terms related to one or more embodiments of the present specification are explained.

[0059] End: refers to a mobile terminal device, such as a mobile phone or other embedded device installed with a mobile operating system, etc.

[0060] Super-resolution: a short name for super-resolution, which is to convert low-resolution video or image into high-resolution video or image through algorithm.

[0061] Heterogeneous: refers to the presence of two or more different hardware computing units on a device, which can perform parallel operations, such as a device with both cpu (central processing unit) and gpu (graphics processing unit).

[0062] In the specification, a training method of a picture processing model is provided. One or more embodiments of the specification simultaneously relate to a training system of a picture processing model, a picture processing model, two picture processing methods, two picture processing apparatuses, a computing device, a computer-readable storage medium, and a computer program, which are described in detail in the following embodiments.

[0063] Referring to Figure 1 , Figure 1 A specific scene processing diagram of a picture processing method according to one embodiment of the specification is shown.

[0064] Taking the application of the picture processing method provided by the embodiments of the specification in the super-resolution scene, i.e., the scene of converting a low-resolution image into a high-resolution image, as an example, a detailed introduction is made.

[0065] Figure 1 The terminal 102 and the pre-trained picture processing model 104 deployed on two different hardware (such as cpu and gpu) of the terminal 102 are included in the embodiment. For example, the cpu branch module and the cpu tail module in the picture processing model are deployed in the cpu of the terminal 102, the gpu branch module in the picture processing model is deployed in the gpu of the terminal 102, and the cpu branch module and the gpu branch module are heterogeneously parallel running when the subsequent picture processing model processes a low-resolution image.

[0066] The picture processing model can be trained in the server and deployed in the terminal 102, or trained in other edge nodes and deployed in the terminal 102, or distributedly trained in the edge nodes and the server and deployed in the terminal 102, etc.

[0067] In specific implementation, the terminal 102 determines a low-resolution image to be converted. The low-resolution image can be a low-resolution image uploaded by a user, a low-resolution image obtained from other application programs by the terminal 102, or a low-resolution image parsed from a playing video obtained from a video playing application program, etc.

[0068] The terminal 102 inputs the low-resolution image into the picture processing model deployed on two different hardware, realizes heterogeneously parallel processing of the low-resolution image in each processing module of the picture processing model, and obtains a high-resolution image corresponding to the low-resolution image output by the picture processing model, and displays the high-resolution image to the user.

[0069] The picture processing method provided by the embodiment of the present specification splits different modules of a picture processing model and respectively deploys them on different hardware of a terminal. Subsequently, when the terminal converts a low-resolution image into a high-resolution image according to the picture processing model, the heterogeneous parallel operation of the picture processing model can be realized, the running time of one module can be used to cover the running time of another module, so as to compress the overall processing time of the image processing module, but the algorithm effect can be guaranteed unchanged.

[0070] Referring to Figure 2 , Figure 2 A flowchart of a training method of a picture processing model according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0071] The training method of the picture processing model is applied to a model training system, which includes a terminal and a server.

[0072] Step 202: The terminal respectively deploys processing modules of a picture processing model on at least two hardware, processes an initial picture according to the processing modules, obtains a target running time of a target picture output by the processing modules, wherein the resolution of the target picture is greater than that of the initial picture, and in the case where the target running time meets a preset time condition, the processing modules are sent to the server.

[0073] Among them, the terminal can be understood as the terminal in the above term explanation; the at least two hardware includes but is not limited to cpu, gpu, dsp (digital signal processor), npu (network processor) and the like in the terminal; the initial picture includes but is not limited to any type and any size of low-resolution picture.

[0074] In specific implementation, the at least two hardware includes cpu and non-cpu (such as gpu, dsp, npu, etc.), the terminal respectively deploys the processing modules of the picture processing model on the cpu and the non-cpu hardware, and subsequently the heterogeneous parallel operation of the picture processing model on the cpu and the non-cpu can be realized, so as to improve the running efficiency of the picture processing model without reducing the algorithm effect.

[0075] In actual application, the non-cpu can be one or more, and the selection of the non-cpu can be set according to specific application conditions. In order to facilitate understanding, the following embodiments are introduced with the at least two hardware including cpu and one non-cpu, and the non-cpu being gpu; but it is not limited to the implementation mode of the at least two hardware including cpu and more than one non-cpu.

[0076] In the case that the at least two hardware includes a CPU and a non-CPU, i.e., a GPU, the processing modules of the picture processing model deployed on the at least two hardware also include at least two. Specifically, the processing modules of the picture processing model are split and deployed on the corresponding hardware to realize the subsequent heterogeneous parallel running of the picture processing model in the terminal. The number and manner of splitting the processing modules of the picture processing model can also be set according to specific applications, such as splitting into two processing modules or three processing modules. For ease of understanding, in the following embodiments, the processing modules of the picture processing model are split into three, i.e., a first processing module, a second processing module, and a third processing module.

[0077] In the case that the at least two hardware includes a CPU and a non-CPU, i.e., a GPU, the processing modules of the picture processing model deployed on the at least two hardware also include at least two. Specifically, the processing modules of the picture processing model are split and deployed on the corresponding hardware to realize the subsequent heterogeneous parallel running of the picture processing model in the terminal. The number and manner of splitting the processing modules of the picture processing model can also be set according to specific applications, such as splitting into two processing modules or three processing modules. For ease of understanding, in the following embodiments, the processing modules of the picture processing model are split into three, i.e., a first processing module, a second processing module, and a third processing module.

[0078] The at least two hardware respectively deploying the processing modules of the picture processing model includes:

[0079] The terminal deploys the first processing module and the third processing module of the picture processing model on the first hardware, and deploys the second processing module of the picture processing model on the second hardware, where the first hardware is different from the second hardware.

[0080] In the case that the first hardware is a CPU, the second hardware can be understood as a non-CPU, such as a GPU.

[0081] Taking the case that the first hardware is a CPU and the second hardware is a GPU as an example, deploying the first processing module and the third processing module of the picture processing model on the first hardware, and deploying the second processing module of the picture processing model on the second hardware can be understood as deploying the first processing module and the third processing module of the picture processing model on the CPU, and deploying the second processing module of the picture processing model on the GPU.

[0082] The first processing module can be understood as the CPU branch module in the above embodiments, the third processing module can be understood as the CPU tail module in the above embodiments, and the second processing module can be understood as the GPU branch module in the above embodiments.

[0083] The training method of the picture processing model provided in the embodiments of the present specification deploys different processing modules of the picture processing model on two different hardware of the terminal, so that the subsequent picture processing model can realize a heterogeneous parallel running mode when processing a picture, and realize fast processing of the picture.

[0084] Therefore, in the case that the picture processing model includes three processing modules and each processing module is deployed on different terminal hardware, the picture processing model can achieve heterogeneous parallel fast processing of the picture by different processing modules when processing the picture, thereby saving the model running time. The specific implementation is as follows:

[0085] The target running time of the picture processing model in processing the initial picture includes:

[0086] The terminal obtains the initial picture.

[0087] The target running time of the picture processing model in processing the initial picture includes:

[0088] In specific implementation, the target picture corresponding to the initial picture and the target running time of the picture processing model in processing the initial picture are obtained by processing the initial picture according to the first processing module, the second processing module, and the third processing module, respectively. The resolution of the target picture is higher than that of the initial picture, that is, the target picture is a high-resolution picture with a higher resolution than the initial picture in the case that the initial picture is a low-resolution picture.

[0089] In actual application, since the first processing module, the second processing module, and the third processing module all process the initial picture, the running time of each processing module in processing the initial picture can be determined, and the internal structure of each processing module can be adjusted according to the running time, thereby optimizing the model structure of the picture processing model. Specifically, the running time of each processing module in processing the initial picture is determined as follows:

[0090] The target running time of the picture processing model in processing the initial picture includes:

[0091] The terminal inputs the initial picture into the first processing module to obtain the picture feature of the initial picture and the first running time of the first processing module in outputting the picture feature.

[0092] inputting the initial picture into the second processing module to obtain the feature weight of the picture feature and a second running time of the second processing module outputting the feature weight;

[0093] inputting the initial picture, the picture feature and the feature weight into the third processing module to obtain a target picture and a third running time of the third processing module outputting the target picture,

[0094] The target running time includes the first running time, the second running time and the third running time.

[0095] Specifically, the first processing module is configured to extract the picture feature of the initial picture, the second processing module is configured to determine the feature weight of the picture feature of the initial picture, and the third processing module is configured to determine the target picture with high resolution of the initial picture according to the initial picture, the picture feature of the initial picture and the feature weight of the picture feature.

[0096] Then, the specific process of processing the initial picture according to the first processing module, the second processing module and the third processing module to obtain the target picture is as follows:

[0097] First, the initial picture is input into the first processing module and the second processing module respectively, the picture feature of the initial picture is obtained through the first processing module, and the first running time of the picture feature is obtained through the first processing module, and at the same time, the feature weight of the picture feature of the initial picture is obtained through the second processing module, and the second running time of the feature weight is obtained through the second processing module.

[0098] Then, the initial picture, the picture feature and the feature weight are input into the third processing module to obtain the target picture with high resolution corresponding to the initial picture, and the third processing module obtains the third running time of the target picture.

[0099] The training method of the picture processing model provided by the embodiments of the present specification respectively obtains the running time of the first processing module, the second processing module and the third processing module in processing the initial picture to obtain the target picture, and subsequently adjusts the module structure according to the running time of each processing module in processing the picture to optimize the picture processing model.

[0100] Then, after obtaining the running time of each processing module in processing the picture, it can be judged according to the running time whether the structure of the processing module in the current picture processing model is reasonable and whether it needs to be adjusted. The specific implementation is as follows:

[0101] The method further includes: in a case where the running time meets a preset time condition, sending the processing module to the server, including:

[0102] determining the first running time of the first processing module, the second running time of the second processing module and the third running time of the third processing module, and determining whether the first running time is less than the second running time, and the sum of the second running time and the third running time is less than the test running time;

[0103] In the case where it is determined that the first running time is less than the second running time, and the sum of the second running time and the third running time is less than the test running time, the first processing module, the second processing module and the third processing module are sent to the service end.

[0104] The preset picture processing model can be understood as a baseline model, that is, a picture processing model different from the picture processing model described above but also capable of converting the initial picture to the target picture.

[0105] In addition, the preset time condition can be understood as: the first running time is less than the second running time, and the sum of the second running time and the third running time is less than the test running time.

[0106] Specifically, while the initial picture is processed by the first processing module, the second processing module and the third processing module of the picture processing model, the initial picture is also processed by the baseline model to obtain the target picture corresponding to the initial picture, and the test running time of the baseline model processing the initial picture to obtain the target picture.

[0107] When the first running time of the first processing module is less than the second running time of the second processing module, and the sum of the second running time of the second processing module and the third running time of the third processing module is less than the test running time, it can be determined that the module structure of the first processing module, the second processing module and the third processing module of the picture processing model is relatively reasonable, which can make the performance of the picture processing model better when the picture processing model is used for subsequent picture processing. At this time, the first processing module, the second processing module and the third processing module are sent to the service end for model training of the picture processing model.

[0108] On the contrary, in the case where the target running time of the target picture obtained by the processing module processing the initial picture does not satisfy the preset time condition, it can be determined that the structure of the processing module in the current picture processing model is not reasonable, and needs to be adjusted to satisfy the performance optimization of the picture processing model. The specific implementation is as follows:

[0109] After obtaining the target running time of the target picture output by the processing module, the method further comprises:

[0110] adjust the number of convolution modules in the processing module when it is determined that the target running time does not satisfy the preset time condition;

[0111] continue to perform the step of inputting the initial picture into the processing module for processing to obtain the target running time of the target picture output by the processing module until the target running time satisfies the preset time condition.

[0112] The processing module includes a convolution module, and the number of convolution modules affects the picture processing performance of the picture processing model.

[0113] In the above example, if the target running time includes a first running time, a second running time, and a third running time, and the preset time condition is that the first running time is less than the second running time, and the sum of the second running time and the third running time is less than the test running time; then in the case that the first running time is greater than or equal to the second running time, or the sum of the second running time and the third running time is greater than or equal to the test running time, it can be determined that the structure of the processing module in the current picture processing model is unreasonable, and the picture processing model cannot achieve better performance when processing pictures; therefore, the number of convolution modules in each processing module needs to be adjusted according to the target running time;

[0114] Then, continue to input the initial picture into each adjusted processing module for processing to obtain a target picture and a target running time; match the target running time with the above-mentioned condition again until the target running time satisfies the preset time condition, and then end; at the same time, send the final first processing module, second processing module, and third processing module to the server for model training of the picture processing model.

[0115] Then, specifically, the terminal adjusts the number of convolution modules in the processing module when it is determined that the target running time does not satisfy the preset time condition;

[0116] continue to perform the step of inputting the initial picture into the processing module for processing to obtain the target running time of the target picture output by the processing module until the target running time satisfies the preset time condition.

[0117] It can be understood that: the terminal adjusts the number of convolution modules in the first processing module, the second processing module, and / or the third processing module when it is determined that the first running time is greater than or equal to the second running time, or the sum of the second running time and the third running time is greater than or equal to the test running time;

[0118] The step of continuously executing the input of the initial picture into the first processing module, the second processing module and the third processing module for processing to obtain the first running time, the second running time and the third running time of the target picture output by the first processing module, the second processing module and the third processing module is performed until the first running time, the second running time and the third running time satisfy the preset time condition, that is, the condition that the first running time is less than the second running time, and the sum of the second running time and the third running time is less than the test running time.

[0119] In step 204, the server trains the received processing module to obtain the picture processing model and sends the picture processing model to the terminal.

[0120] After the terminal sends the adjusted first processing module, the second processing module and the third processing module of the picture processing model to the server, the server can train the received first processing module, the second processing module and the third processing module from the terminal to obtain the trained picture processing model, and then send the trained picture processing model to the terminal.

[0121] Specifically, the training steps of the picture processing model are as follows:

[0122] The training of the received processing module to obtain the picture processing model and the sending of the picture processing model to the terminal include:

[0123] The server obtains a training sample set, wherein the training sample set includes at least one initial sample picture and a target sample picture corresponding to the at least one initial sample picture, and the resolution of the target sample picture is greater than that of the initial sample picture.

[0124] The at least one initial sample picture is used as a training sample, and the target sample picture corresponding to the at least one initial sample picture is used as a sample label.

[0125] The first processing module, the second processing module and the third processing module received are trained according to the training sample and the sample label, and the picture processing model is obtained.

[0126] The picture processing model is sent to the terminal.

[0127] In actual applications, the picture processing model can be trained according to the training sample set on the server side, or the first processing module, the second processing module and the third processing module of the picture processing model can be respectively placed on different edge nodes for training, and then distributed model training is implemented, so as to improve the training efficiency of the model and save the computing resources of the server side. The specific training mode can be set according to actual applications, and the embodiments of the present specification do not make any limitation in this regard.

[0128] Then, after the server sends the trained picture processing model to the terminal, the terminal can continue to deploy the picture processing model in a heterogeneous parallel manner, so that the picture processing model can improve the processing performance in a heterogeneous parallel running manner without reducing the algorithm effect when performing picture processing subsequently. The specific implementation manner is as follows:

[0129] The method further includes the following steps after the picture processing model is sent to the terminal:

[0130] The terminal deploys the first processing module and the third processing module of the picture processing model to the first hardware.

[0131] The second processing module of the picture processing model is deployed to the second hardware.

[0132] The specific explanations of the first processing module, the second processing module, the third processing module, the first hardware and the second hardware can be referred to the above embodiments, and will not be repeated here.

[0133] The training method of the picture processing model provided by the embodiments of the present specification deploys the processing modules of the picture processing model on at least two hardware of the terminal, and determines that the picture processing performance of the picture processing model is relatively optimal when the picture processing model is deployed in this structure on the terminal, in the case that the target running time of the initial picture processing satisfies the preset time condition. Then, the picture processing model is sent to the server for training and deployment, so that the performance of the picture processing model is also relatively optimal when the terminal uses the picture processing model to perform picture processing subsequently, while the algorithm effect is not reduced.

[0134] In addition, the structure of each processing module in the picture processing model is determined on the terminal first, and then the corresponding picture processing model on each terminal is sent to the server for model training, so that each terminal can have a better picture processing effect when applying the trained picture processing model. For example, if the structure of the picture processing model used by a low-end mobile device is unreasonable, it may cause incompatibility or poor picture processing effect. Therefore, by determining the structure of the picture processing model on the low-end mobile device first and then training the picture processing model, the trained picture processing model can be deployed on the low-end mobile device to avoid the above problems. In addition, to avoid determining the structure of the picture processing model on each terminal and wasting computing resources, a median mobile device can be found in each type of mobile device. After determining the structure of the picture processing model on the mobile device, it is sent to the server for model training. Therefore, the trained picture processing model can be adapted to most terminals.

[0135] Therefore, the picture processing model obtained by the training method of the picture processing model provided by the embodiments of the present specification can also be deployed in a low-end mobile device and can perform real-time super-resolution task processing on a low-end mobile device.

[0136] Referring to Figure 3 , Figure 3 FIG. 1 shows a structure diagram of a picture processing model according to an embodiment of the present specification.

[0137] Specifically, the picture processing model includes a first processing module 302, a second processing module 304, and a third processing module 306. The first processing module 302 includes at least one first convolution module 3022 and at least one second convolution module 3024, and the first processing module and the third processing module are deployed on a first hardware of a terminal.

[0138] The second processing module includes a down-sampling module 3042, an up-sampling module 3044, and a second processing sub-module 3046, and the second processing module is deployed on a second hardware of the terminal, which is different from the first hardware. The picture processing model is a model obtained by the above-mentioned picture processing model training method.

[0139] Wherein, the first processing module 302, the second processing module 304, the third processing module 306; and the detailed explanation of the first hardware, the second hardware can be referred to the introduction of the above-mentioned embodiments, which will not be repeated here.

[0140] And the first convolution module 3022 can be understood as an ecb module, and can also be other convolution modules; and in the case of the first convolution module 3022 being an ecb module, the second convolution module can be understood as an ecb-linear, that is, a module after removing activation in the ecb module.

[0141] In addition, the picture processing module further comprises:

[0142] The down-sampling module 3042 comprises at least one first convolution module and at least one third convolution module with a preset stride,

[0143] The up-sampling module 3044 comprises a pixel recombination module and at least one first convolution module,

[0144] The second processing sub-module 3046 comprises at least two first convolution modules.

[0145] Among them, the first convolution module can be understood as the first convolution module 3022 described above; the third convolution module can be understood as a convolution module with a stride (step length) of 2 or 3.

[0146] In actual application, in the case of the first processing module 302 being a cpu_branch module (that is, a cpu branch module), it can be composed of cpu_m+1 ecb modules (first convolution modules 3022), wherein the ecb-linear module (second convolution module 3024) is a mandatory module, and the remaining cpu_m ecb modules can be adjusted according to the actual application environment and algorithm effect, and cpu_m is greater than or equal to 0 and is a positive integer.

[0147] The second processing module 304 is a gpu_branch module, which is composed of a downsample block 3042, an up sample block 3044, and a Gpu_Block 3046. The downsample block 3042 is composed of gpu_dm+1 ecb modules and a stride=2 convolution module, where gpu_dm is greater than or equal to 0 and is a positive integer, which can be adjusted according to the actual application environment and algorithm effect. The up sample block 3044 is composed of a PixelShuffle (PS) operation and um ecb modules, where um is greater than or equal to 0 and is a positive integer, which can be adjusted according to the actual application environment and algorithm effect. The Gpu_Block 3046 is a backbone of a GPU, which is composed of gpu_m Gpu_Block modules, and each Gpu_Block module includes a residual network composed of two ecb modules.

[0148] In specific implementation, when the picture processing model includes a cpu_branch module, a gpu_branch module, and a cpu_tail module, the cpu_branch module and the cpu_tail module run on the cpu, the gpu_branch module runs on the gpu, and the cpu_branch module and the gpu_branch module are in heterogeneous parallel relationship. When the picture processing model runs on the edge, the execution time of the gpu_branch module masks the execution time of the cpu_branch module, and the total running time of the two modules is determined by the running time of the gpu_branch module. That is, when the overall running time of the picture processing model running on the edge satisfies the following two conditions, the performance of the picture processing model is better, and the algorithm effect remains unchanged:

[0149] Tcpu_branch<Tgpu_branch

[0150] Tgpu_branch+Tcpu_tail<Tgpu_m4c4

[0151] Wherein, Tcpu_branch, Tcpu_tail represent the time of cpu_branch module, cpu_tail module running independently in cpu respectively, Tgpu_branch represents the time of gpu_branch module running independently in gpu, Tgpu_m4c4 represents the baseline model, that is, the running time of the whole model before optimization.

[0152] Through the above conditions, the cpu_branch module and the gpu_branch module in the picture processing model run heterogeneously and in parallel on the terminal, which can achieve the purpose of compressing the running time.

[0153] When performing super-resolution operation through the picture processing model, the LR (low resolution) picture is input into the picture processing model, and after being processed by the cpu_branch module and the gpu_branch module in heterogeneous and parallel manner, the cpu_tail module performs operation through the following formula, so as to obtain the HR (high resolution) picture:

[0154] y = x + query * key

[0155] Wherein, x represents the y channel of the input low resolution image / video frame, query represents the output of the cpu_branch module, and key represents the output of the gpu_branch module.

[0156] The operation is realized through the efficient implementation of the simplified attention mechanism on the terminal, and has good algorithm effect retention; and in actual application, the performance of the cpu_tail module can be further optimized by performing op merging, neon (multi-data acceleration instruction), parallel and other optimizations on the three ops (combining multiple calculation units into one calculation unit to speed up calculation) in the cpu_tail module, that is, '+', '*', and 'PS (pixel recombination, an upsampling operator)'.

[0157] The picture processing module can realize the conversion from low resolution image to high resolution image through the cpu_tail module.

[0158] The picture processing model provided by the embodiment of the present specification splits one picture processing model into multiple modules, respectively deploys the cpu_branch module and the gpu_branch module in the cpu and the non-cpu on the terminal for heterogeneous parallelism, hides the running time of the cpu_branch module in the cpu, so as to compress the overall model inference time, and the algorithm effect can remain unchanged; and when the picture processing model is used for super-resolution processing on the terminal subsequently, the heterogeneous parallelism and the down-sampling structure are used to reduce the memory / display memory bandwidth overhead between the gpu and the cpu, and solve the problem of large display memory bandwidth overhead caused by the better super-resolution performance on the terminal.

[0159] When the actual effect experiment is performed on the picture processing model, the inference speed ratio of the picture processing model is 2.2 (6.67 ms vs 14.847 ms) under the condition that the algorithm effect is unchanged compared with the baseline algorithm-m4c4 algorithm, the performance speed is more than 2 times compared with the original model, and the model can perform real-time inference at a speed of 6.67 ms.

[0160] Referring to Figure 4 , Figure 4 A flowchart of a picture processing method applied to a terminal is shown, which specifically includes the following steps.

[0161] Step 402: input an initial picture into a first processing module and a second processing module of a picture processing model respectively, wherein the first processing module and the second processing module are respectively run on different hardware.

[0162] The picture processing model provided by the embodiment of the present specification can be understood as a model obtained by the training method of the picture processing model, and the model structure of the picture processing model is the same as that of the picture processing model provided by the above embodiment.

[0163] In addition, the detailed description of the initial picture, the first processing module, the second processing module, the third processing module, and the first hardware and the second hardware can also be referred to the specific introduction of the above embodiment, which will not be repeated here.

[0164] Specifically, before the initial picture is processed by the picture processing model, the terminal needs to obtain the pre-trained picture processing model from the server, and respectively deploy different modules of the picture processing model on different hardware to realize subsequent heterogeneous parallel processing. The specific implementation manner is as follows:

[0165] Before the initial picture is input into the first processing module and the second processing module of the picture processing model respectively, the method further includes:

[0166] receive a picture processing model sent by a server, deploy a first processing module and a third processing module of the picture processing model on a first hardware;

[0167] deploy a second processing module of the picture processing model on a second hardware, wherein the first hardware is different from the second hardware.

[0168] The picture processing model sent by the server can be obtained by pre-training on the server after determining the model structure on the terminal according to the training method of the picture processing model.

[0169] In actual application, the terminal sends a picture processing model obtaining instruction to the server, and the server sends a pre-trained picture processing model to the terminal in response to the instruction. After receiving the picture processing model, the terminal deploys the cpu_branch module and the cpu_tail module of the picture processing model on the cpu according to the previous deployment mode of model structure determination, and deploys the gpu_branch module on the gpu, so that the picture processing model realizes heterogeneous parallel processing when processing the initial picture subsequently.

[0170] The picture processing method provided by the embodiments of the present specification enables the terminal to obtain a pre-trained picture processing model from the server and split the picture processing model into modules for deployment in different hardware, so that the picture processing model can improve its running performance without reducing the algorithm effect in subsequent running.

[0171] Step 404: obtaining picture features of the initial picture output by the first processing module and feature weights of the picture features output by the second processing module.

[0172] Specifically, after inputting the initial picture to be processed into the first processing module and the second processing module, the first processing module outputs the picture features of the initial picture, and the second processing module outputs the feature weights of the picture features.

[0173] Step 406: inputting the initial picture, the picture features and the feature weights of the picture features into the third processing module of the picture processing model to obtain a target picture corresponding to the initial picture.

[0174] The resolution of the target picture is greater than that of the initial picture, and the picture processing model is a model obtained by the training method of the picture processing model.

[0175] After obtaining the picture features and the feature weights of the initial picture, the initial picture, the picture features and the feature weights of the picture features are input into the third processing module of the picture processing model, and the target picture corresponding to the initial picture can be obtained.

[0176] Specifically, the third processing module of the picture processing model is inputted with the initial picture, the picture feature, and the feature weight of the picture feature, to obtain a target picture corresponding to the initial picture.

[0177] The picture feature and the feature weight of the picture feature are inputted into the third processing module of the picture processing model for processing, to obtain a pixel difference value.

[0178] The pixel value of the initial picture and the pixel difference value are inputted into the third processing module of the picture processing model for processing, to obtain a target picture corresponding to the initial picture.

[0179] In a specific implementation, the picture feature and the feature weight of the picture feature are inputted into the third processing module of the picture processing model. The third processing module calculates a pixel difference value according to the picture feature and the feature weight of the picture feature, and then processes the pixel value of the initial picture and the pixel difference value, to obtain a target picture corresponding to the initial picture.

[0180] In actual application, the picture processing model also learns the pixel difference value between a low-resolution picture and a high-resolution picture during model training. When a low-resolution picture is inputted into the picture processing model for processing, the picture processing model determines a target picture corresponding to the initial picture according to the learned pixel difference value and the current pixel value of the initial picture.

[0181] In the picture processing method provided by the embodiments of the present disclosure, the picture processing model sent by the server and received by the terminal is split into multiple modules and deployed on different hardware, so that these modules can be heterogeneously and in parallel during implementation, to hide the running time of the modules in a single hardware device, thereby compressing the overall model inference time, improving the picture processing efficiency, and not affecting the picture processing accuracy.

[0182] In actual application, the time consumption of a brand a mobile phone using a preset inference framework to super-resolution a 480x720 picture is taken as an example for specific experiment.

[0183] The six models are respectively deployed on the above mobile phones, and the time consumption of super-resolution of low-resolution pictures with a resolution of 480x720 is taken as an example for experiment, wherein the six models include two m4c4 models (i.e. baseline model one), two m4c8 models (i.e. baseline model two) and two different configurations of heterogeneous models (such as ecbSR_TR), wherein the first m4c4 model uses a normal convolution module and does not use an ecb module; the second m4c4 model uses an ecb module; the first m4c8 model uses a normal convolution module and does not use an ecb module; the second m4c8 model uses an ecb module; the parameter configuration of the first ecbSR_TR is: cpu_m=1, cpu_c=4, gpu_dm=2, gpu_dc=4, gpu_m=1, gpu_c=4, gpu_um=0, gpu_uc=1; the parameter configuration of the second ecbSR_TR is: cpu_m=1, cpu_c=4, gpu_dm=0, gpu_dc=4, gpu_m=1, gpu_c=4, gpu_um=0, gpu_uc=1.

[0184] Furthermore, the scale (super-resolution magnification) of the six models is all magnified by 2 times, and the same training set (Movie_4K) is used; the data set (Movie_4K_test_191105) used for testing of the first m4c4 model is 38.5013db (decibel); the data set used for testing of the second m4c4 model is 38.6462; the data set used for testing of the first m4c8 model is 38.7615; the data set used for testing of the second m4c8 model is 39.0231; the data set used for testing of the first ecbSR_TR is 38.7229; the data set used for testing of the second ecbSR_TR is 38.5256; the BS (i.e. BatchSize) size used for training of the six models is all 32, and the number of gpus (ngpu) used for training is all 2; the total number of iterations (iter) for training is all 100000 times; the cpu branch running time consumption (cpu_branch) of the first ecbSR_TR model and the second ecbSR_TR model is all 2.632 milliseconds; the cpu tail running time consumption (cpu_tail) of the first ecbSR_TR model and the second ecbSR_TR model is all 0.652 milliseconds.

[0185] The GPU branch execution time (gpu_branch) for the first and second m4c4 models was 14.847 milliseconds; the GPU branch execution time for the first and second m4c8 models was 18.298 milliseconds; the GPU branch execution time for the first ecbSR_TR model was 9.278 milliseconds; and the GPU branch execution time for the second ecbSR_TR model was 6.018 milliseconds.

[0186] In summary, the total execution time of the two m4c4 models is 14.847 milliseconds; the total execution time of the two m4c8 models is 18.298 milliseconds; the total execution time of the first ecbSR_TR model is 9.93 milliseconds; and the total execution time of the second ecbSR_TR model is 6.67 milliseconds.

[0187] The experimental results above show that when performing super-resolution processing on images of the same resolution on the same mobile phone, the overall running time is shorter when using a heterogeneous model.

[0188] See Figure 5 , Figure 5 This specification illustrates a flowchart of another image processing method provided in one embodiment, which specifically includes the following steps.

[0189] Step 502: Receive the initial image input by the user through the image input interface;

[0190] Step 504: Input the initial images into the first processing module and the second processing module of the image processing model respectively, wherein the first processing module and the second processing module run on different hardware;

[0191] Step 506: Obtain the image features of the initial image output by the first processing module, and the feature weights of the image features output by the second processing module;

[0192] Step 508: Input the initial image, the image features, and the feature weights of the image features into the third processing module of the image processing model to obtain the target image corresponding to the initial image, wherein the resolution of the target image is greater than that of the initial image;

[0193] Step 510: Display the target image to the user through the image display interface, wherein the image processing model is a model obtained through the above-described image processing model training method.

[0194] Specifically, the execution subject of the image processing method provided in the embodiments of this specification can be a terminal, that is, the terminal receives an initial image input by the user through its image input interface; after performing various image processing on the image processing model, a target image with a resolution greater than the initial image is obtained. The detailed steps of image processing using the image processing model can be found in the above embodiments, and will not be repeated here.

[0195] In another possible implementation, the user interaction of this image processing method can be achieved by the user clicking the resolution conversion control for high definition or ultra-high definition in the video viewing software on the terminal while watching a video, which triggers the terminal to convert each video frame of the currently playing video from low resolution to high resolution. The specific resolution conversion algorithm can be implemented using the image processing method of the embodiments in this specification.

[0196] In the image processing method provided in the embodiments of this specification, after receiving a user's image resolution conversion request, the terminal can trigger the image processing method to convert a low-resolution initial image into a high-resolution target image. That is, the terminal splits the image processing model sent by the server into multiple modules and deploys them on different hardware, so that these modules can run heterogeneously in parallel during specific implementation, thereby hiding the running time of the modules in a single hardware device, thus compressing the overall model inference time, improving the efficiency of image resolution conversion, and not affecting the accuracy of the image content after the image resolution conversion.

[0197] See Figure 6 , Figure 6 This specification illustrates a training system for an image processing model according to an embodiment, comprising a terminal 602 and a server 604, wherein...

[0198] The terminal 602 is configured to deploy image processing model processing modules on at least two hardware devices, process the initial image according to the processing modules, and obtain the target running time of the target image output by the processing modules, wherein the resolution of the target image is greater than that of the initial image.

[0199] If the target running time meets the preset time conditions, the processing module is sent to the server 604;

[0200] The server 604 is configured to train the received processing module to obtain the image processing model, and send the image processing model to the terminal 602.

[0201] Optionally, the terminal 602 deploys a first processing module of a picture processing model on a first hardware, and deploys a second processing module of the picture processing model on a second hardware, wherein the first hardware is different from the second hardware.

[0202] Optionally, the terminal 602 acquires an initial picture.

[0203] The terminal 602 processes the initial picture according to the first processing module, the second processing module, and the third processing module to obtain a target picture, and a target running time of the terminal 602 processing the initial picture according to the first processing module, the second processing module, and the third processing module to obtain the target picture.

[0204] Optionally, the terminal 602 inputs the initial picture into the first processing module to obtain a picture feature of the initial picture, and a first running time of the first processing module outputting the picture feature.

[0205] The terminal 602 inputs the initial picture into the second processing module to obtain a feature weight of the picture feature, and a second running time of the second processing module outputting the feature weight.

[0206] The terminal 602 inputs the initial picture, the picture feature, and the feature weight into the third processing module to obtain a target picture, and a third running time of the third processing module outputting the target picture.

[0207] The target running time includes the first running time, the second running time, and the third running time.

[0208] Optionally, the terminal 602 inputs the initial picture into a preset picture processing model to obtain the target picture, and a test running time of the preset picture processing model outputting the target picture.

[0209] In a case where it is determined that the first running time is less than the second running time, and a sum of the second running time and the third running time is less than the test running time, the terminal 602 sends the first processing module, the second processing module, and the third processing module to the server 604.

[0210] Optionally, the terminal 602 adjusts a number of convolution modules in the processing module in a case where it is determined that the target running time does not satisfy the preset time condition.

[0211] The terminal 602 continues to perform the step of inputting the initial picture into the processing module to obtain a target running time of the processing module outputting a target picture until the target running time satisfies the preset time condition.

[0212] Optionally, the server 604 obtains a training sample set, wherein the training sample set comprises at least one initial sample picture and a target sample picture corresponding to the at least one initial sample picture, and the target sample picture has a resolution greater than that of the initial sample picture.

[0213] The at least one initial sample picture is taken as a training sample, and the target sample picture corresponding to the at least one initial sample picture is taken as a sample label.

[0214] The first processing module, the second processing module and the third processing module received are trained according to the training sample and the sample label, and the picture processing model is obtained.

[0215] The picture processing model is sent to the terminal 602.

[0216] Optionally, the terminal 602 deploys the first processing module and the third processing module of the picture processing model to the first hardware.

[0217] The second processing module of the picture processing model is deployed to the second hardware.

[0218] The training device of the picture processing model provided by the embodiment of the present specification deploys the processing modules of the picture processing model to at least two hardware of the terminal, and determines that the picture processing performance of the picture processing model is better when the picture processing model is deployed in the terminal in the case that the target running time of the initial picture processing satisfies the preset time condition. The picture processing model is sent to the server for training and deployment, so that the picture processing performance of the picture processing model is better when the subsequent terminal uses the picture processing model to process pictures while maintaining the algorithm effect.

[0219] The above is a schematic scheme of the training device of the picture processing model of the embodiment. It should be noted that the technical scheme of the training device of the picture processing model and the technical scheme of the training method of the picture processing model belong to the same concept, and the details of the technical scheme of the training device of the picture processing model which are not described in detail can be referred to the description of the technical scheme of the training method of the picture processing model.

[0220] Referring to Figure 7 , Figure 7 FIG. 1 shows a structural schematic diagram of a picture processing device applied to a terminal according to an embodiment of the present specification.

[0221] Specifically, the picture processing device comprises:

[0222] The first input module 702 is configured to input an initial picture into a first processing module and a second processing module of a picture processing model respectively, wherein the first processing module and the second processing module run on different hardware respectively.

[0223] The first obtaining module 704 is configured to obtain picture features of the initial picture output by the first processing module and feature weights of the picture features output by the second processing module.

[0224] The second input module 706 is configured to input the initial picture, the picture features and the feature weights of the picture features into a third processing module of the picture processing model to obtain a target picture corresponding to the initial picture.

[0225] The target picture has a resolution greater than that of the initial picture, and the picture processing model is obtained by the training method of the picture processing model.

[0226] Optionally, the apparatus further includes:

[0227] The model deployment module is configured to:

[0228] receive the picture processing model sent by the server, and deploy a first processing module and a third processing module of the picture processing model on a first hardware.

[0229] deploy a second processing module of the picture processing model on a second hardware, wherein the first hardware is different from the second hardware.

[0230] Optionally, the second input module 706 is further configured to:

[0231] input the picture features and the feature weights of the picture features into the third processing module of the picture processing model for processing to obtain a pixel difference value.

[0232] input pixel values of the initial picture and the pixel difference value into the third processing module of the picture processing model for processing to obtain the target picture corresponding to the initial picture.

[0233] The picture processing apparatus provided by the embodiments of the present disclosure splits the picture processing model sent by the server and received by the terminal into multiple modules deployed on different hardware, so that these modules can run in heterogeneous parallel when implemented, so as to hide the running time of the modules in a single hardware device, thereby compressing the overall model inference time, improving the picture processing efficiency, and not affecting the picture processing accuracy.

[0234] The above is a schematic scheme of the picture processing device of the embodiment. It should be noted that the technical scheme of the picture processing device and the technical scheme of the picture processing method described above belong to the same concept. The details of the technical scheme of the picture processing device that are not described in detail can be referred to the description of the technical scheme of the picture processing method.

[0235] Referring to Figure 8 , Figure 8 A structural schematic diagram of another picture processing device provided by an embodiment of the present specification is shown.

[0236] Specifically, the picture processing device comprises:

[0237] The picture receiving module 802 is configured to receive an initial picture input by a user through a picture input interface.

[0238] The third input module 804 is configured to input the initial picture into a first processing module and a second processing module of a picture processing model respectively, wherein the first processing module and the second processing module run on different hardware respectively.

[0239] The second obtaining module 806 is configured to obtain picture features of the initial picture output by the first processing module and feature weights of the picture features output by the second processing module.

[0240] The fourth input module 808 is configured to input the initial picture, the picture features and the feature weights of the picture features into a third processing module of the picture processing model to obtain a target picture corresponding to the initial picture, wherein the resolution of the target picture is greater than that of the initial picture.

[0241] The display module 810 is configured to display the target picture to the user through a picture display interface, wherein the picture processing model is obtained by the training method of the picture processing model described above.

[0242] The picture processing device provided by the embodiment of the present specification can trigger the picture processing method to realize the conversion from the initial picture with low resolution to the target picture with high resolution after the terminal receives the picture resolution conversion request of the user, that is, the terminal splits the picture processing model sent by the server into multiple modules and deploys them on different hardware, so that these modules can run in heterogeneous parallel mode when they are specifically implemented, so as to hide the running time of the modules in a single hardware device, thereby compressing the overall model inference time, improving the efficiency of picture resolution conversion, and not affecting the accuracy of the picture content after the picture resolution conversion.

[0243] The above is a schematic solution of another picture processing apparatus of the embodiment. It should be noted that the technical solution of the another picture processing apparatus and the technical solution of the above another picture processing method belong to the same concept, and the details of the technical solution of the another picture processing apparatus which are not described in detail can be referred to the description of the technical solution of the another picture processing method.

[0244] Figure 9 A structural block diagram of a computing device 900 according to one embodiment of the present specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected with the memory 910 through a bus 930, and a database 950 is used to save data.

[0245] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 can include one or more of any type of network interface (e.g., a network interface card (NIC)), wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a near-field communication (NFC) interface, and the like.

[0246] In one embodiment of the present specification, the above-mentioned components of the computing device 900 and other components not shown in the Figure 9 may be connected with each other, for example, through a bus. It should be understood that Figure 9 The structural block diagram of the computing device shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced as needed by those skilled in the art.

[0247] The computing device 900 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 900 can also be a mobile or stationary server.

[0248] The processor 920 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the above picture processing model training method or picture processing method.

[0249] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the picture processing model training method or the picture processing method belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the picture processing model training method or the picture processing method.

[0250] An embodiment of the present specification further provides a computer readable storage medium storing computer executable instructions, which, when executed by a processor, implement the steps of the picture processing model training method or the picture processing method.

[0251] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the picture processing model training method or the picture processing method belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the picture processing model training method or the picture processing method.

[0252] An embodiment of the present specification further provides a computer program, which, when executed in a computer, causes the computer to perform the steps of the picture processing model training method or the picture processing method.

[0253] The above is a schematic scheme of the computer program of the embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the picture processing model training method or the picture processing method belong to the same concept, and the details of the technical scheme of the computer program that are not described in detail can be referred to the description of the technical scheme of the picture processing model training method or the picture processing method.

[0254] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0255] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or subtractions according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0256] It should be noted that for the foregoing method embodiments, the descriptions are expressed as a series of action combinations for the sake of simplicity and brevity, but those skilled in the art should know that the present application is not limited by the order of the actions described, because some steps can be performed in other orders or at the same time according to the present application. In addition, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0257] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0258] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, according to the content of the present application, many modifications and changes can be made. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and use the present application. The present application is limited only by the claims and their full scope and equivalents.

Claims

1. A training method for an image processing model, applied to a model training system, the system comprising a terminal and a server, the method comprising: The terminal deploys image processing model processing modules on at least two hardware devices. The initial image is processed by these processing modules to obtain a target execution time for the target image output by the processing modules. The target image has a higher resolution than the initial image. The deployment of image processing model processing modules on at least two hardware devices includes: the terminal deploying a first processing module and a third processing module of the image processing model on a first hardware device, and a second processing module of the image processing model on a second hardware device. The first hardware device and the second hardware device are different. The first processing module is used to extract image features from the initial image. The second processing module is used to determine the feature weights of the image features from the initial image. The third processing module is used to determine the target image based on the initial image, the image features, and the feature weights. The at least two hardware devices process the initial image in parallel. If the target running time meets the preset time condition, the processing module is sent to the server. The preset time condition is that the first running time of the first processing module is less than the second running time of the second processing module, and the sum of the second running time and the third running time of the third processing module is less than the test running time. The server trains the received processing module to obtain the image processing model, and then sends the image processing model to the terminal.

2. The training method for the image processing model according to claim 1, wherein the step of processing the initial image by the processing module to obtain the target running time of the target image output by the processing module includes: The terminal acquires the initial image; The initial image is processed by the first processing module, the second processing module, and the third processing module to obtain the target image, and the target running time of the target image is obtained by processing the initial image by the first processing module, the second processing module, and the third processing module.

3. The training method for the image processing model according to claim 2, wherein the step of processing the initial image according to the first processing module, the second processing module, and the third processing module to obtain a target image, and the step of processing the initial image according to the first processing module, the second processing module, and the third processing module to obtain a target running time for the target image, comprises: The terminal inputs the initial image into the first processing module to obtain the image features of the initial image and the first processing module outputs the first running time of the image features; The initial image is input into the second processing module to obtain the feature weights of the image features, and the second processing module outputs the second running time of the feature weights; The initial image, the image features, and the feature weights are input into the third processing module to obtain the target image, and the third processing module outputs the third running time of the target image. The target running time includes the first running time, the second running time, and the third running time.

4. The training method for the image processing model according to claim 3, wherein sending the processing module to the server when the running time meets a preset time condition includes: The terminal inputs the initial image into a preset image processing model to obtain the target image and the test running time of the target image output by the preset image processing model. If it is determined that the first running time is less than the second running time, and the sum of the second running time and the third running time is less than the test running time, the first processing module, the second processing module, and the third processing module are sent to the server.

5. The training method for the image processing model according to claim 1, after obtaining the target running time of the target image output by the processing module, further includes: If the terminal determines that the target running time does not meet the preset time condition, it adjusts the number of convolution modules in the processing module. Continue executing the step of inputting the initial image into the processing module for processing and obtaining the target running time of the target image output by the processing module, until the target running time meets the preset time condition.

6. The training method for the image processing model according to claim 2, wherein training the received processing module to obtain the image processing model and sending the image processing model to the terminal comprises: The server obtains a training sample set, wherein the training sample set includes at least one initial sample image and a target sample image corresponding to the at least one initial sample image, and the resolution of the target sample image is greater than that of the initial sample image. Use the at least one initial sample image as a training sample, and use the target sample image corresponding to the at least one initial sample image as a sample label; The first processing module, the second processing module, and the third processing module are trained based on the training samples and the sample labels to obtain the image processing model; The image processing model is sent to the terminal.

7. The training method for the image processing model according to claim 6, further comprising, after sending the image processing model to the terminal: The terminal deploys the first processing module and the third processing module of the image processing model onto the first hardware. The second processing module of the image processing model is deployed to the second hardware.

8. An image processing model, comprising a first processing module, a second processing module, and a third processing module, wherein, The first processing module includes at least one first convolutional module and at least one second convolutional module, and the first processing module and the third processing module are deployed on the first hardware of the terminal; The second processing module includes a downsampling module, an upsampling module, and a second processing submodule. The second processing module is deployed on the second hardware of the terminal. The first hardware is different from the second hardware. The image processing model is a model obtained by any one of the training methods of claims 1-7.

9. The image processing model according to claim 8, further comprising: The downsampling module includes at least one first convolutional module and at least one third convolutional module with a preset stride. The upsampling module includes a pixel recombination module and at least one first convolution module. The second processing submodule includes at least two first convolutional modules.

10. An image processing method, applied to a terminal, comprising: The initial images are input into the first processing module and the second processing module of the image processing model, wherein the first processing module and the second processing module run on different hardware. The image features of the initial image output by the first processing module and the feature weights of the image features output by the second processing module are obtained. The initial image, the image features, and the feature weights of the image features are input into the third processing module of the image processing model to obtain the target image corresponding to the initial image. Wherein, the resolution of the target image is greater than that of the initial image, and the image processing model is a model obtained by any one of the training methods of claims 1-7.

11. The image processing method according to claim 10, further comprising, before inputting the initial images into the first processing module and the second processing module of the image processing model: Receive the image processing model sent by the server, and deploy the first processing module and the third processing module of the image processing model on the first hardware. The second processing module of the image processing model is deployed in the second hardware, wherein the first hardware is different from the second hardware.

12. The image processing method according to claim 10 or 11, wherein inputting the initial image, the image features, and the feature weights of the image features into the third processing module of the image processing model to obtain the target image corresponding to the initial image includes: The image features and their feature weights are input into the third processing module of the image processing model for processing to obtain pixel difference values. The pixel values ​​and pixel difference values ​​of the initial image are input into the third processing module of the image processing model for processing to obtain the target image corresponding to the initial image.

13. An image processing method, comprising: Receive the initial image entered by the user through the image input interface; The initial images are input into the first processing module and the second processing module of the image processing model, wherein the first processing module and the second processing module run on different hardware. The image features of the initial image output by the first processing module and the feature weights of the image features output by the second processing module are obtained. The initial image, the image features, and the feature weights of the image features are input into the third processing module of the image processing model to obtain the target image corresponding to the initial image, wherein the resolution of the target image is greater than that of the initial image; The target image is displayed to the user through an image display interface, wherein the image processing model is a model obtained by any one of the training methods of claims 1-7.

Images