Model training method, image enhancement method, equipment and storage medium

Abstract

The invention provides a model training method, an image enhancement method and related equipment, and relates to the technical field of image processing. The training method is applied to the cloud equipment and comprises the following steps: acquiring a training sample set comprising a plurality of sample images; performing enhancement processing on the sample image based on a first image enhancement model to obtain a first enhanced image; performing coding processing on the sample image and the first enhanced image based on an image coding model to obtain enhanced intensity; and training the image coding model according to the sample image, the first enhanced image and the enhanced intensity to obtain a trained image coding model. The image coding model obtained by the method can accurately reflect the degree of image enhancement, so that the terminal equipment can perform enhancement processing according to the enhancement strength when the model is deployed for enhancement processing, and the enhancement effect is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a model training method, an image enhancement method, a model training device, an image enhancement device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of deep learning and large-model technology in the field of image processing, the demand for deploying large models in the cloud for image and video enhancement is increasing, which in turn exacerbates the demand for computing resources. Therefore, deploying lightweight models to terminal devices for enhancement has become a trend.

[0003] However, the computing power of terminal devices is limited, and the model complexity they can support is far lower than that of large models in the cloud. This limitation affects the enhancement effect, specifically: insufficient enhancement intensity makes it difficult to significantly improve the visual experience, resulting in negligible enhancement effects; or excessive enhancement intensity leads to negative effects such as overly sharp images. Summary of the Invention

[0004] This disclosure provides a model training method, an image enhancement method, a model training apparatus, an image enhancement apparatus, an electronic device, a computer-readable storage medium, and a computer program product to overcome or at least partially solve the above-mentioned problems.

[0005] According to one aspect of the present disclosure, a model training method is provided, the method being applied to a cloud device, comprising: acquiring a training sample set; the training sample set including multiple sample images; performing enhancement processing on the sample images based on a first image enhancement model to obtain a first enhanced image of the sample images; the resolution of the first enhanced image being higher than the resolution of the sample images; performing encoding processing on the sample images and the first enhanced image based on an image coding model to obtain an enhancement intensity of the first enhanced image relative to the sample images; and training the image coding model based on the sample images, the first enhanced image, and the enhancement intensity to obtain a trained image coding model.

[0006] In some embodiments of this disclosure, training the image coding model based on the sample image, the first enhanced image, and the enhancement intensity to obtain a trained image coding model includes: performing enhancement processing on the sample image based on the enhancement intensity and the first enhanced image to obtain a second enhanced image of the sample image; and training the image coding model based on the first enhanced image and the second enhanced image to obtain a trained image coding model.

[0007] In some embodiments of this disclosure, the step of enhancing the sample image based on the enhancement intensity and the first enhanced image to obtain a second enhanced image of the sample image includes: calculating the image residual between the first enhanced image and the sample image; enhancing the image residual according to the enhancement intensity to obtain image enhancement information of the sample image; and obtaining the second enhanced image according to the sample image and the image enhancement information of the sample image.

[0008] In some embodiments of this disclosure, the image coding model includes a multimodal pre-trained neural network model and a linear layer; wherein, the step of encoding the sample image and the first enhanced image based on the image coding model to obtain the enhancement intensity of the first enhanced image relative to the sample image includes: encoding the sample image and the first enhanced image respectively based on the multimodal pre-trained neural network model to obtain image features of the sample image and image features of the first enhanced image; and performing a linear transformation on the image features of the sample image and the image features of the first enhanced image based on the linear layer to obtain the enhancement intensity.

[0009] In some embodiments of this disclosure, training the image coding model based on the first enhanced image and the second enhanced image to obtain the trained image coding model includes: calculating the image loss between the first enhanced image and the second enhanced image; adjusting the parameters of the linear layer based on the image loss to obtain the adjusted linear layer, thereby obtaining the trained image coding model.

[0010] In some embodiments of this disclosure, the first image enhancement model is a pre-trained enhancement model.

[0011] According to another aspect of the embodiments of this disclosure, an image enhancement method is provided, the method being applied to a cloud device, comprising: acquiring a target image; performing image enhancement processing on the target image based on a first image enhancement model to obtain a first target enhanced image of the target image; performing encoding processing on the target image and the first target enhanced image based on a trained image encoding model to obtain a target enhancement intensity of the first target enhanced image relative to the target image; wherein the trained image encoding model is obtained according to the model training method described in the above embodiments; and sending the target enhancement intensity to a terminal device; wherein the target enhancement intensity is used to perform image enhancement processing on the target image.

[0012] According to another aspect of the present disclosure, an image enhancement method is provided, the method being applied to a terminal device, comprising: acquiring a target image; receiving a target enhancement intensity of the target image sent by a cloud device; the target enhancement intensity being obtained by encoding the target image and a first target enhancement image based on a trained image encoding model; performing image enhancement processing on the target image based on a second image enhancement model to obtain a second target enhancement image of the target image; and performing enhancement processing on the target image based on the target enhancement intensity and the second target enhancement image to obtain a third target enhancement image of the target image.

[0013] According to another aspect of the present disclosure, a model training apparatus is provided, the apparatus being applied to a cloud device, comprising: a sample acquisition module configured to acquire a training sample set; the training sample set including a plurality of sample images; a sample enhancement module configured to enhance the sample images based on a first image enhancement model to obtain a first enhanced image of the sample images; the resolution of the first enhanced image being higher than the resolution of the sample images; a sample encoding module configured to encode the sample images and the first enhanced image based on an image encoding model to obtain an enhancement intensity of the first enhanced image relative to the sample images; and a model training module configured to train the image encoding model based on the sample images, the first enhanced image, and the enhancement intensity to obtain a trained image encoding model.

[0014] In some embodiments of this disclosure, the model training module is further configured to: perform enhancement processing on the sample image based on the enhancement intensity and the first enhancement image to obtain a second enhancement image of the sample image; and train the image coding model according to the first enhancement image and the second enhancement image to obtain a trained image coding model.

[0015] In some embodiments of this disclosure, the model training module is further configured to: calculate the image residual between the first enhanced image and the sample image; perform enhancement processing on the image residual according to the enhancement intensity to obtain image enhancement information of the sample image; and obtain the second enhanced image according to the sample image and the image enhancement information of the sample image.

[0016] In some embodiments of this disclosure, the image encoding model includes a multimodal pre-trained neural network model and a linear layer; wherein, the model training module is further configured to: encode the sample image and the first enhanced image respectively based on the multimodal pre-trained neural network model to obtain the image features of the sample image and the image features of the first enhanced image; and perform a linear transformation on the image features of the sample image and the image features of the first enhanced image based on the linear layer to obtain the enhancement intensity.

[0017] In some embodiments of this disclosure, the model training module is further configured to: calculate the image loss between the first enhanced image and the second enhanced image; adjust the parameters of the linear layer based on the image loss to obtain the adjusted linear layer, thereby obtaining the trained image coding model.

[0018] In some embodiments of this disclosure, the first image enhancement model is a pre-trained enhancement model.

[0019] According to another aspect of the embodiments of this disclosure, an image enhancement apparatus is provided, the apparatus being applied to a cloud device, comprising: a first image acquisition module configured to acquire a target image; a first image enhancement module configured to perform image enhancement processing on the target image based on a first image enhancement model to obtain a first target enhanced image of the target image; an image encoding module configured to perform encoding processing on the target image and the first target enhanced image based on a trained image encoding model to obtain a target enhancement intensity of the first target enhanced image relative to the target image; the trained image encoding model is obtained according to the model training method described in the above embodiments; and an intensity sending module configured to send the target enhancement intensity to a terminal device; the target enhancement intensity is used to perform image enhancement processing on the target image.

[0020] According to another aspect of the present disclosure, an image enhancement apparatus is provided, the apparatus being applied to a terminal device, comprising: a second image acquisition module configured to acquire a target image; an intensity receiving module configured to receive a target enhancement intensity of the target image sent by a cloud device; the target enhancement intensity being obtained by encoding the target image and a first target enhancement image based on a trained image encoding model; a second image enhancement module configured to perform image enhancement processing on the target image based on a second image enhancement model to obtain a second target enhancement image of the target image; and a third image enhancement module configured to perform enhancement processing on the target image based on the target enhancement intensity and the second target enhancement image to obtain a third target enhancement image of the target image.

[0021] According to another aspect of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the above-described model training method or the above-described image enhancement method.

[0022] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-described model training method or the above-described image enhancement method.

[0023] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the model training method described above, or implements the image enhancement method described above.

[0024] In the model training method provided in this embodiment, the sample image is first enhanced based on a first image enhancement model to obtain an enhanced image with a higher resolution than the sample image. Then, the sample image and the enhanced image are encoded based on an image coding model to obtain the enhancement intensity of the first enhanced image relative to the sample image. The image coding model is then trained based on the output enhancement intensity to obtain a trained image coding model. The trained image coding model obtained by this method accurately reflects the degree of image enhancement and effectively guides the image enhancement process of the terminal device. Specifically, when the terminal device performs image enhancement processing on a target image, the cloud device uses the trained image coding model to encode the target image and the first enhanced image of the target image to obtain the target enhancement intensity, and transmits the target enhancement intensity to the terminal device. This allows the terminal device to enhance the target image according to the received target enhancement intensity, avoiding the problem of weak enhancement effect due to insufficient enhancement intensity and preventing negative effects such as over-sharpening caused by excessive enhancement intensity, thus improving the enhancement effect of the terminal device.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0027] Figure 1A schematic diagram of an exemplary system architecture for which the model training method or image enhancement method of the present disclosure embodiments can be applied is shown;

[0028] Figure 2 A flowchart of a model training method according to an embodiment of this disclosure is shown;

[0029] Figure 3 A schematic diagram illustrating model training performed by a cloud-based device according to an embodiment of this disclosure is shown;

[0030] Figure 4 A flowchart of an image enhancement method according to an embodiment of this disclosure is shown;

[0031] Figure 5 A flowchart of another image enhancement method according to an embodiment of this disclosure is shown;

[0032] Figure 6 A schematic diagram illustrating the joint implementation of image enhancement by a cloud device and a terminal device according to an embodiment of this disclosure is shown;

[0033] Figure 7 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown;

[0034] Figure 8 A block diagram of an image enhancement apparatus according to an embodiment of the present disclosure is shown;

[0035] Figure 9 A block diagram of another image enhancement apparatus according to an embodiment of the present disclosure is shown;

[0036] Figure 10 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0037] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0038] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0039] The collection, updating, analysis, processing, use, transmission, and storage of user personal information disclosed herein comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0040] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different network and / or processor devices and / or microcontroller devices.

[0041] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0042] In this specification, the terms “a,” “one,” “the,” “the,” and “at least one” are used to indicate the presence of at least one element / component / etc.; the term “multiple” refers to two or more; the terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion meaning and that other elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms “first,” “second,” and “third,” etc., are used only as markings and are not a limitation on the number of objects.

[0043] Figure 1 A schematic diagram of an exemplary system architecture to which the model training method or image enhancement method of the present disclosure can be applied is shown. Figure 1 As shown, the system architecture may include a cloud device 101, a network 102, and a terminal device 103. The network 102 serves as a medium for providing a communication link between the terminal device 103 and the cloud device 101. The network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0044] Cloud device 101 is a device with functions such as data processing, data storage, and data transmission and reception, and is not limited in this embodiment. Optionally, cloud device 101 is a server, which can be a single server, a server cluster composed of multiple servers, or any one of a cloud computing platform and a virtualization center, and is not limited in this embodiment. That is to say, cloud device 101 can be a server or a server cluster.

[0045] The terminal device 103 that transmits data with the cloud device 101 may include, but is not limited to, mobile devices such as smartphones, tablets, and laptops, as well as terminal devices with specific functions or forms such as smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, and smart wearable devices. Alternatively, the terminal device 103 may also be a personal computer, such as a laptop computer or a desktop computer. Optionally, the operating system running on the electronic device may include, but is not limited to, Android, iOS, Linux, and Windows.

[0046] In this embodiment of the disclosure, the process of cloud device 101 for model training may include: acquiring a training sample set, which includes multiple sample images; performing enhancement processing on the sample images based on a first image enhancement model to obtain a first enhanced image of the sample images, wherein the resolution of the first enhanced image is higher than that of the sample images; performing encoding processing on the sample images and the first enhanced image based on an image coding model to obtain the enhancement intensity of the first enhanced image relative to the sample images; and training the image coding model according to the sample images, the first enhanced image, and the enhancement intensity to obtain a trained image coding model.

[0047] In this embodiment of the disclosure, the process of image enhancement by the cloud device 101 may be as follows: acquiring a target image; performing image enhancement processing on the target image based on a first image enhancement model to obtain a first target enhanced image of the target image; performing encoding processing on the target image and the first target enhanced image based on a trained image encoding model to obtain a target enhancement intensity of the first target enhanced image relative to the target image, wherein the trained image encoding model is obtained according to a model training method; sending the target enhancement intensity to the terminal device 103, and then the terminal device performs image enhancement processing on the target image based on the target enhancement intensity.

[0048] In this embodiment of the disclosure, the process of image enhancement by the terminal device 103 may be as follows: acquiring a target image; receiving the target enhancement intensity of the target image sent by the cloud device; performing image enhancement processing on the target image based on a second image enhancement model to obtain a second target enhanced image of the target image; and performing enhancement processing on the target image based on the target enhancement intensity and the second target enhanced image to obtain a third target enhanced image of the target image.

[0049] In addition, it should be noted that, Figure 1 The example shown is merely one application environment of the model training method or image enhancement method provided in this disclosure. Figure 1 The number of cloud devices 101, network 102 and terminal devices 103 is merely illustrative. Depending on actual needs, there can be any number of cloud devices, network and terminal devices.

[0050] Figure 2 A flowchart of a model training method according to an embodiment of this disclosure is shown. Figure 2 The execution subject of the method provided in the embodiments can be any electronic device, such as... Figure 1 The cloud device 101 in this embodiment is not limited thereto. See also Figure 2 The model training method provided in this disclosure includes the following steps.

[0051] Step S210: Obtain the training sample set; the training sample set includes multiple sample images.

[0052] In this embodiment of the disclosure, a training sample set is obtained, which includes multiple sample images.

[0053] In an exemplary embodiment, the sample image may be an image with a resolution lower than a preset resolution threshold, that is, the sample image may also be referred to as a low-resolution image sample.

[0054] Step S220: Enhance the sample image based on the first image enhancement model to obtain a first enhanced image of the sample image; the resolution of the first enhanced image is higher than that of the sample image.

[0055] In this embodiment of the disclosure, the first image enhancement model is capable of enhancing the input image to output an image with higher resolution and richer details. In step S220, the sample image is input into the first image enhancement model to enhance the sample image and output a first enhanced image with a higher resolution than the sample image.

[0056] In some embodiments of this disclosure, the first image enhancement model is a pre-trained enhancement model.

[0057] The first image enhancement model can be constructed based on deep learning methods using convolutional neural networks. For example, EDSR (Enhanced Deep Residual Networks) is a typical deep learning-based image super-resolution reconstruction model that can improve reconstruction performance by increasing network depth and residual connections.

[0058] Besides deep learning methods, first-order image enhancement models can also be built using machine learning methods, such as RAISR (Rapid and Accurate Image Super Resolution). RAISR uses local statistical information in the image and machine learning algorithms to predict pixel values ​​in high-resolution images, so as to achieve high-quality image enhancement without increasing computational complexity.

[0059] When selecting the first image enhancement model, a trade-off can be made based on the specific application scenario and requirements. For example, when high reconstruction quality is required, a deep learning-based model, such as EDSR, can be selected; when computational resources are limited, a machine learning-based model, such as RAISR, can be considered. Of course, other pre-trained enhancement models can also be selected as the first image enhancement model in this embodiment of the disclosure, and there is no limitation on this.

[0060] In this embodiment of the disclosure, the first image enhancement model is a pre-trained enhancement model that can quickly and efficiently generate a first enhanced image by utilizing learned image enhancement knowledge and experience; and when the image coding model is subsequently trained using the first enhanced image output by the first image enhancement model, there is no need to spend a lot of time on image enhancement preprocessing, thus accelerating the training process of the image coding model.

[0061] Step S230: Encode the sample image and the first enhanced image based on the image coding model to obtain the enhancement intensity of the first enhanced image relative to the sample image.

[0062] In this embodiment of the disclosure, the image coding model is a machine learning or deep learning model used to analyze and quantify image features. In this embodiment of the disclosure, the image coding model can be used to measure the difference between the original image (i.e., the sample image) and the enhanced image (i.e., the first enhanced image), thereby determining the degree or intensity of the enhancement.

[0063] In this embodiment of the disclosure, the image coding model serves to evaluate the enhancement effect of the first image enhancement model on the sample image. Specifically, the image coding model encodes the sample image and the first enhanced image, calculates the difference between the two, and then obtains the enhancement strength of the first enhanced image relative to the sample image.

[0064] Step S240: Train the image coding model based on the sample image, the first enhanced image, and the enhancement intensity to obtain the trained image coding model.

[0065] In this embodiment of the disclosure, after obtaining the first enhanced image through the first image enhancement model and the enhancement intensity through the image coding model, the image coding model can be trained based on the sample image, the first enhanced image, and the enhancement intensity, and finally the trained image coding model is obtained.

[0066] In the model training method provided in this embodiment, the sample image is first enhanced based on a first image enhancement model to obtain an enhanced image with a higher resolution than the sample image. Then, the sample image and the enhanced image are encoded based on an image coding model to obtain the enhancement intensity of the first enhanced image relative to the sample image. The image coding model is then trained based on the output enhancement intensity to obtain a trained image coding model. The trained image coding model obtained by this method accurately reflects the degree of image enhancement and effectively guides the image enhancement process of the terminal device. Specifically, when the terminal device performs image enhancement processing on a target image, the cloud device uses the trained image coding model to encode the target image and the first enhanced image of the target image to obtain the target enhancement intensity, and transmits the target enhancement intensity to the terminal device. This allows the terminal device to enhance the target image according to the received target enhancement intensity, avoiding the problem of weak enhancement effect due to insufficient enhancement intensity and preventing negative effects such as over-sharpening caused by excessive enhancement intensity, thus improving the enhancement effect of the terminal device.

[0067] In some embodiments of this disclosure, training an image coding model based on a sample image, a first enhanced image, and an enhancement intensity to obtain a trained image coding model includes: enhancing a sample image based on the enhancement intensity and the first enhanced image to obtain a second enhanced image of the sample image; and training the image coding model based on the first enhanced image and the second enhanced image of the sample image to obtain a trained image coding model.

[0068] In this embodiment, after obtaining the enhancement intensity based on the image coding model, the enhancement intensity is transmitted to the intensity control unit. The intensity control unit then performs enhancement processing on the sample image based on the enhancement intensity and the first enhanced image to obtain a second enhanced image of the sample image. The image quality of the second enhanced image is superior to that of the first enhanced image. Next, the image coding model is trained using the first and second enhanced images to finally obtain the trained image coding model.

[0069] The intensity control unit is a specially designed processing unit used to precisely calculate and regulate the intensity of image enhancement. Through a series of algorithms and processes, this unit achieves fine-grained control over image enhancement to ensure that the enhanced image achieves the desired visual effect. The enhancement intensity obtained based on the image coding model provides the intensity control unit with precise control information, guiding it on how to further enhance the first enhanced image. This precise control allows the intensity control unit to avoid over-enhancement or under-enhancement, ensuring that the second enhanced image is superior in image quality to the first.

[0070] In the model training method provided in this embodiment, after obtaining the enhancement strength provided by the image coding model, the enhancement effect is adjusted according to the enhancement strength to obtain a second enhanced image with better image quality than the first enhanced image. When training the image coding model, the first enhanced image and the second enhanced image after further enhancement are used as comparison data, so that the image coding model learns more about the relationship between image features and enhancement strength.

[0071] In some embodiments of this disclosure, enhancing a sample image based on enhancement intensity and a first enhanced image to obtain a second enhanced image of the sample image includes: calculating the image residual between the first enhanced image and the sample image; enhancing the image residual according to the enhancement intensity to obtain image enhancement information of the sample image; and obtaining the second enhanced image based on the sample image and the image enhancement information of the sample image.

[0072] In this embodiment of the disclosure, the intensity control unit controls the enhancement effect by separating the residual between the low-resolution image and the enhanced image of the low-resolution image, and combining the enhancement intensity. The specific formula is as follows:

[0073] I 输出 =(I 增强 -I 低质 )*α+I 低质 (1)

[0074] In formula (1), I 低质 For low-resolution images, i.e., images with a resolution lower than a preset resolution threshold, I 增强 This is an enhanced version of a low-resolution image, where α is the enhancement intensity and I is the enhancement intensity. 输出 Enhanced image output by the intensity control unit.

[0075] In an exemplary embodiment, during the model training phase, the sample image, the first enhanced image, and the enhancement intensity are substituted into formula (1). Specifically, the sample image is used as the low-resolution image I. 低质 The first enhanced image is used as the enhanced image I of the low-resolution image. 增强The enhancement intensity α is taken as the enhancement intensity of the first enhanced image compared to the sample image, and the final output image I is obtained. 输出 This is the second enhanced image of the sample image.

[0076] In the model training method provided in this embodiment, the image residual between the enhanced image and the sample image is calculated, and then the image enhancement effect is controlled by using the calculated image residual and enhancement intensity. This can extract richer and more detailed image enhancement information, which is used to generate a second enhanced image, so that the image quality of the second enhanced image is better than that of the first enhanced image.

[0077] In some embodiments of this disclosure, the image coding model includes a multimodal pre-trained neural network model and a linear layer. Specifically, encoding a sample image and a first enhanced image based on the image coding model to obtain the enhancement strength of the first enhanced image relative to the sample image includes: encoding the sample image and the first enhanced image separately based on the multimodal pre-trained neural network model to obtain image features of the sample image and image features of the first enhanced image; and performing a linear transformation on the image features of the sample image and the image features of the first enhanced image based on the linear layer to obtain the enhancement strength.

[0078] In this embodiment, the image coding model consists of a multimodal pre-trained neural network model (Contrastive Language–Image Pre-training, CLIP) and linear layers. In other words, the image coding model is obtained by adjusting the Linear Probe on top of the open-source CLIP model. The purpose of Linear Probe adjustment training is to enable the model to more accurately predict enhancement intensity by fine-tuning the parameters of the linear layers based on the CLIP model.

[0079] In this embodiment of the disclosure, a visual encoder using the CLIP model encodes the input sample image and the first enhanced image respectively to obtain image features of the sample image and image features of the first enhanced image. A linear layer receives the image features of the sample image and the image features of the first enhanced image from the visual encoder, and processes the image features through linear transformations (such as matrix multiplication) to calculate the enhancement intensity of the first enhanced image relative to the sample image.

[0080] In some embodiments of this disclosure, an image coding model is trained based on a first enhanced image and a second enhanced image of a sample image to obtain a trained image coding model, including: calculating the image loss between the first enhanced image and the second enhanced image; adjusting the parameters of the linear layer based on the image loss to obtain the adjusted linear layer, thereby obtaining the trained image coding model.

[0081] In this embodiment, the image coding model consists of a CLIP model and linear layers. The CLIP model, as an open-source and extensively trained multimodal pre-trained model, is responsible for extracting deep features from the image, while the linear layers are responsible for converting the extracted features into enhancement strength. Since the CLIP model is a large open-source model, the core of training the image coding model is to adjust the parameters of the linear layers to ensure that the model can accurately predict the enhancement strength from the image features.

[0082] After obtaining the enhancement strength output by the image coding model, the sample image is enhanced based on the enhancement strength and the first enhanced image to obtain the second enhanced image. The image loss between the first and second enhanced images is calculated, for example, using loss functions such as mean square error (MSE) or structural similarity (SSIM) to compare the differences between the first and second enhanced images at the pixel or feature level. Based on the calculated image loss, the parameters of the linear layer are adjusted using the backpropagation algorithm, aiming to enable the linear layer to more accurately predict the enhancement strength based on the image features extracted by the CLIP model. This process is repeated until a preset number of training epochs is reached or the calculated image loss is less than the preset loss, resulting in a parameter-adjusted linear layer and a fully trained image coding model.

[0083] In the model training method provided in this disclosure, the image coding model includes a multimodal pre-trained neural network model and a linear layer. The multimodal pre-trained neural network model can capture the complex feature relationship between the sample image and the first enhanced image, while the linear layer is responsible for converting these features into specific enhancement intensities. During training, by calculating the image loss between the first and second enhanced images and adjusting the parameters of the linear layer accordingly, fine optimization of the image coding model is achieved, enabling the trained image coding model to predict enhancement intensities more accurately.

[0084] Figure 3 A schematic diagram illustrating model training using a cloud-based device according to an embodiment of this disclosure is shown. Figure 3 As shown, the sample image is input into the first image enhancement model 310, and the sample image is enhanced based on the first image enhancement model 310 to obtain the first enhanced image. Then, the sample image and the first enhanced image are input into the image coding model 320, and the sample image and the first enhanced image are encoded based on the image coding model 320 to obtain the enhancement intensity. Next, the enhancement intensity, the first enhanced image, and the sample image are input into the intensity control unit 330, so that the intensity control unit 330 performs enhancement processing based on the above formula (1) and outputs a second enhanced image with image quality superior to the first enhanced image.

[0085] The second enhanced image has better image quality than the first enhanced image. The image coding model 320 consists of a CLIP model and linear layers. After obtaining the second enhanced image, the parameters of the linear layers in the image coding model 320 can be adjusted using the first and second enhanced images to obtain the trained image coding model.

[0086] It should be noted that during the training of the image coding model 320, the first image enhancement model 310 can also be adjusted so that the two models can learn from each other during training and jointly improve the model performance.

[0087] from Figure 3 It can be seen that during the model training process of the cloud device, an image coding model 320 and an intensity control unit 330 are introduced. The image coding model 320 predicts the enhancement intensity. By utilizing the predicted enhancement intensity, the intensity control unit 330 can perform more refined image enhancement processing and generate enhanced images with better image quality. That is, the image quality of the enhanced image output by the intensity control unit 330 is better than the image quality of the enhanced image output by the pre-trained enhancement model.

[0088] Figure 4 A flowchart of an image enhancement method according to an embodiment of this disclosure is shown. Figure 4 The execution subject of the method provided in the embodiments can be any electronic device, such as... Figure 1 The cloud device 101 in the embodiment, for example Figure 1 In this embodiment, the cloud device 101 and the terminal device 103 jointly implement the image enhancement method, but this disclosure is not limited thereto. (See also...) Figure 4 The image enhancement method provided in this disclosure includes the following steps.

[0089] Step S410: Obtain the target image.

[0090] In this embodiment of the disclosure, the target image refers to the image that needs to be enhanced, and may also be called the image to be processed, the image to be enhanced, the low-quality image, etc.

[0091] Step S420: Perform image enhancement processing on the target image based on the first image enhancement model to obtain the first target enhanced image of the target image.

[0092] In this embodiment of the disclosure, the first image enhancement model is capable of enhancing an input image to output an image with higher resolution and richer details. The first image enhancement model is a pre-trained enhancement model. It can be constructed based on deep learning methods using convolutional neural networks, such as EDSR, or using machine learning methods, such as RAISR.

[0093] When selecting the first image enhancement model, a trade-off can be made based on the specific application scenario and requirements. For example, when high reconstruction quality is required, a deep learning-based model, such as EDSR, can be selected; when computational resources are limited, a machine learning-based model, such as RAISR, can be considered. Of course, other pre-trained enhancement models can also be selected as the first image enhancement model in this embodiment of the disclosure, and there is no limitation on this.

[0094] In some embodiments of this disclosure, during the training of the image coding model, the pre-trained enhancement model can also be adjusted. In this case, the first image enhancement model is the model after adjusting the pre-trained enhancement model.

[0095] In step S420, the target image is input into the first image enhancement model to enhance the target image and output a first target enhanced image with a resolution higher than that of the target image.

[0096] Step S430: Encode the target image and the first target enhancement image based on the trained image coding model to obtain the target enhancement intensity of the first target enhancement image relative to the target image; wherein, the trained image coding model is obtained according to the model training method of the above embodiment.

[0097] In this embodiment, the target image and the first target enhancement image are input into a trained image coding model to encode the target image and the first target enhancement image, thereby obtaining the target enhancement intensity of the first target enhancement image relative to the target image. The image coding model consists of a CLIP model and linear layers. The training process of the image coding model has been described in the above embodiments and will not be repeated here.

[0098] Step S440: Send the target enhancement intensity to the terminal device; the target enhancement intensity is used to perform image enhancement processing on the target image.

[0099] In this embodiment of the disclosure, after obtaining the target enhancement intensity, the target enhancement intensity is sent to the terminal device, so that the terminal device can perform enhancement processing on the target image based on the target enhancement intensity.

[0100] In the image enhancement method provided in this embodiment, a first image enhancement model and a trained image encoding model are deployed on a cloud device. When enhancement processing of a target image is required, the target image is first enhanced based on the first image enhancement model to obtain a first target enhanced image with a higher resolution than the target image. Then, the target image and the first target enhanced image are encoded based on the trained image encoding model to obtain the target enhancement intensity of the first target enhanced image relative to the target image. This target enhancement intensity is then sent to the terminal device, allowing the terminal device to enhance the target image according to this target enhancement intensity. Therefore, this method utilizes the first image enhancement model deployed on the cloud and the trained image encoding model to determine the target enhancement intensity when the terminal device performs enhancement processing. This allows the terminal device to enhance the target image according to the target enhancement intensity, avoiding the problem of weak enhancement effect due to insufficient enhancement intensity and preventing negative effects such as over-sharpening caused by excessive enhancement intensity, thus improving the enhancement effect of the terminal device.

[0101] Figure 5 A flowchart of another image enhancement method according to an embodiment of this disclosure is shown. Figure 5 The execution subject of the method provided in the embodiments can be any electronic device, such as... Figure 1 The terminal device 103 in the embodiment, for example Figure 1 In this embodiment, the cloud device 101 and the terminal device 103 jointly implement the image enhancement method, but this disclosure is not limited thereto. (See also...) Figure 5 The image enhancement method provided in this disclosure includes the following steps.

[0102] Step S510: Obtain the target image.

[0103] In this embodiment of the disclosure, the target image refers to the image that needs image enhancement processing, and may also be referred to as the image to be processed, the image to be enhanced, the low-quality image, etc. For example, the cloud device sends the target image to the terminal device, and the terminal device receives the target image sent by the cloud device.

[0104] Step S520: Receive the target enhancement intensity of the target image sent by the cloud device.

[0105] In this embodiment of the disclosure, the terminal device receives the target enhancement intensity of the target image sent by the cloud device. The cloud device inputs the target image and the first target enhancement image into a trained image coding model to encode the target image and the first target enhancement image through the trained image coding model, thereby obtaining the target enhancement intensity of the first target enhancement image relative to the target image, and then transmits the target enhancement intensity to the terminal device.

[0106] The first target augmented image is obtained by the cloud device using the first image augmentation model to enhance the target image, and the resolution of the first target augmented image is higher than that of the target image. Furthermore, the image coding model consists of a CLIP model and linear layers. The training process of the image coding model has been described in the above embodiments and will not be repeated here.

[0107] Step S530: Perform image enhancement processing on the target image based on the second image enhancement model to obtain the second target enhanced image of the target image.

[0108] In this embodiment, the second image enhancement model is capable of enhancing the input image to output an image with higher resolution and richer details. The second image enhancement model is a pre-trained enhancement model. Furthermore, due to the limited computing resources of the terminal device, the second image enhancement model can be a lightweight model corresponding to the first image enhancement model.

[0109] Lightweighting is a method for optimizing models by employing various techniques, including but not limited to model pruning, quantization, knowledge distillation, and compact model design. The goal of lightweighting is to reduce the size of the model, decrease computational complexity and resource consumption, while ensuring that the model's performance (e.g., accuracy, image quality) in tasks such as image enhancement remains at or as close as possible to the level of the original model (i.e., the unlightweighted model).

[0110] In this embodiment, a second image enhancement model is obtained by lightweighting the first image enhancement model. The performance of the second image enhancement model in image enhancement tasks is close to that of the first image enhancement model, but it has a smaller model size, lower computational complexity, and lower resource consumption, making it more suitable for running on terminal devices with limited computing power.

[0111] In step S530, the target image is input into the second image enhancement model to enhance the target image and output a second target enhanced image with a resolution higher than that of the target image.

[0112] Step S540: Enhance the target image based on the target enhancement intensity and the second target enhancement image to obtain the third target enhancement image of the target image.

[0113] In this embodiment of the disclosure, the image residual between the second target enhancement image and the target image is calculated; the image residual is enhanced according to the target enhancement intensity to obtain the image enhancement information of the target image; and the third target enhancement image is obtained according to the target image and the image enhancement information of the target image.

[0114] In an exemplary embodiment, during the model inference stage, the target image, the second target enhancement image, and the target enhancement intensity are substituted into the above formula (1). Specifically, the target image is used as the low-resolution image I. 低质 The enhanced image of the second target is used as the enhanced image I of the low-resolution image. 增强 Using the target enhancement intensity as the enhancement intensity α, the final output image I is obtained. 输出 This is the third enhanced image of the target image.

[0115] The image enhancement method provided in this disclosure receives a target enhancement intensity sent by a cloud device, and performs enhancement processing on the target image based on a second image enhancement model to obtain a second target enhanced image with a higher resolution than the target image. Then, the target image is further enhanced based on the target enhancement intensity and the second target enhanced image to finally obtain a third target enhanced image of the target image. It is evident that this method only requires deploying a lightweight second image enhancement model on the terminal device. Enhancement processing is performed based on this lightweight model and the target enhancement intensity sent by the cloud device, avoiding the problem of weak enhancement effects caused by insufficient enhancement intensity on the terminal device, and also preventing negative effects such as over-sharpening of the image caused by excessive enhancement intensity on the terminal device, thus improving the enhancement effect of the enhancement processing performed by the terminal device.

[0116] Figure 6 A schematic diagram illustrating the joint implementation of image enhancement by a cloud device and a terminal device according to an embodiment of this disclosure is shown. Figure 6 As shown, a first image enhancement model 610 and a trained image coding model 620 are deployed on a cloud device, and a second image enhancement model 630 and an intensity control unit 640 are deployed on a terminal device. The second image enhancement model 630 is a lightweight model corresponding to the first image enhancement model 610.

[0117] When the cloud device performs image processing, it inputs the target image into a first image enhancement model 610, and enhances the target image based on the first image enhancement model 610 to obtain a first target enhanced image. Then, it inputs the target image and the first target enhanced image into a trained image encoding model 620, and encodes the target image and the first target enhanced image based on the trained image encoding model 620 to obtain the target enhancement intensity. The target enhancement intensity is then sent to the terminal device. Finally, the cloud device sends the target image to the terminal device.

[0118] When the terminal device performs image processing, it receives a target image sent by the cloud device, inputs the target image into the second image enhancement model 630, and enhances the target image based on the second image enhancement model 630 to obtain a second target enhanced image. The target image, the first target enhanced image, and the target enhancement intensity from the cloud device are transmitted to the intensity control unit 640 for enhancement processing, and a third target enhanced image is output.

[0119] from Figure 6 It can be seen that cloud devices and terminal devices work together to perform image enhancement processing. The target enhancement intensity is obtained through the cloud device, and the target image and target enhancement intensity are transmitted to the terminal device. After being processed by the enhancement model and intensity control unit of the terminal device, an enhanced image with better image quality is obtained.

[0120] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0121] Figure 7 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown. This model training apparatus can be applied to any electronic device, such as… Figure 1 The cloud device 101 in the embodiment is not limited thereto.

[0122] like Figure 7 As shown, the device 700 includes a sample acquisition module 710, a sample enhancement module 720, a sample encoding module 730, and a model training module 740. The sample acquisition module 710 is configured to acquire a training sample set, which includes multiple sample images. The sample enhancement module 720 is configured to enhance the sample images based on a first image enhancement model to obtain a first enhanced image of the sample images; the resolution of the first enhanced image is higher than that of the sample images. The sample encoding module 730 is configured to encode the sample images and the first enhanced image based on an image encoding model to obtain the enhancement intensity of the first enhanced image relative to the sample images. The model training module 740 is configured to train the image encoding model based on the sample images, the first enhanced image, and the enhancement intensity to obtain a trained image encoding model.

[0123] In some embodiments of this disclosure, the model training module 740 is further configured to: perform enhancement processing on the sample image based on the enhancement intensity and the first enhancement image to obtain a second enhancement image of the sample image; and train the image coding model based on the first enhancement image and the second enhancement image to obtain a trained image coding model.

[0124] In some embodiments of this disclosure, the model training module 740 is further configured to: calculate the image residual between the first enhanced image and the sample image; perform enhancement processing on the image residual according to the enhancement intensity to obtain image enhancement information of the sample image; and obtain a second enhanced image based on the sample image and the image enhancement information of the sample image.

[0125] In some embodiments of this disclosure, the image encoding model includes a multimodal pre-trained neural network model and a linear layer; wherein, the model training module 740 is further configured to: encode the sample image and the first enhanced image respectively based on the multimodal pre-trained neural network model to obtain the image features of the sample image and the image features of the first enhanced image; and perform a linear transformation on the image features of the sample image and the image features of the first enhanced image based on the linear layer to obtain the enhancement intensity.

[0126] In some embodiments of this disclosure, the model training module 740 is further configured to: calculate the image loss between the first enhanced image and the second enhanced image; adjust the parameters of the linear layer based on the image loss to obtain the adjusted linear layer, thereby obtaining the trained image coding model.

[0127] In some embodiments of this disclosure, the first image enhancement model is a pre-trained enhancement model.

[0128] Figure 8 A block diagram of an image enhancement apparatus according to an embodiment of the present disclosure is shown. This image enhancement apparatus can be applied to any electronic device, such as… Figure 1 The cloud device 101 in the embodiment is not limited thereto.

[0129] like Figure 8 As shown, the device 800 includes a first image acquisition module 810, a first image enhancement module 820, an image encoding module 830, and an intensity transmission module 840. The first image acquisition module 810 is configured to acquire a target image. The first image enhancement module 820 is configured to perform image enhancement processing on the target image based on a first image enhancement model to obtain a first target enhanced image of the target image. The image encoding module 830 is configured to perform encoding processing on the target image and the first target enhanced image based on a trained image encoding model to obtain a target enhancement intensity of the first target enhanced image relative to the target image; the trained image encoding model is obtained according to the model training method of the above embodiment. The intensity transmission module 840 is configured to send the target enhancement intensity to a terminal device; the target enhancement intensity is used to perform image enhancement processing on the target image.

[0130] Figure 9 A block diagram of another image enhancement apparatus according to an embodiment of the present disclosure is shown. This image enhancement apparatus is applicable to any electronic device, such as… Figure 1 The terminal device 103 in the embodiment is not limited thereto.

[0131] like Figure 9 As shown, the device 900 includes a second image acquisition module 910, an intensity receiving module 920, a second image enhancement module 930, and a third image enhancement module 940. The second image acquisition module 910 is configured to acquire a target image. The intensity receiving module 920 is configured to receive the target enhancement intensity of the target image sent by a cloud device; the target enhancement intensity is obtained by encoding the target image and a first target enhancement image based on a trained image encoding model. The second image enhancement module 930 is configured to perform image enhancement processing on the target image based on the second image enhancement model to obtain a second target enhancement image of the target image. The third image enhancement module 940 is configured to perform enhancement processing on the target image based on the target enhancement intensity and the second target enhancement image to obtain a third target enhancement image of the target image.

[0132] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0133] Figure 10 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. It should be noted that... Figure 10 The illustrated electronic device 1000 is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein. Figure 10 As shown, the electronic device 1000 is manifested in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, and a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010).

[0134] The storage unit stores program code that can be executed by the processing unit 1010, causing the processing unit 1010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1010 can perform, as follows: Figure 2 The steps are shown in the figure.

[0135] Storage unit 1020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 10201 and / or cache memory 10202, and may further include read-only memory (ROM) 10203. Storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0136] Bus 1030 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0137] Electronic device 1000 can also communicate with one or more external devices 1100 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1040. As shown, network adapter 1040 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0139] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0140] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0142] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0143] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0144] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0145] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0146] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0147] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered illustrative only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0148] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A model training method, characterized in that: The method is applied to a cloud device, comprising: Acquire a training sample set; the training sample set includes a plurality of sample images; Performing enhancement processing on the sample image based on a first image enhancement model to obtain a first enhanced image of the sample image; the resolution of the first enhanced image is higher than the resolution of the sample image; Performing encoding processing on the sample image and the first enhanced image based on an image encoding model to obtain an enhancement strength of the first enhanced image relative to the sample image; The image coding model is trained according to the sample image, the first enhanced image and the enhancement strength to obtain a trained image coding model.

2. The method according to claim 1, characterized in that: The step of training the image coding model according to the sample image, the first enhanced image and the enhancement strength to obtain the trained image coding model comprises: Performing enhancement processing on the sample image based on the enhancement intensity and the first enhanced image to obtain a second enhanced image of the sample image; The image coding model is trained according to the first enhanced image and the second enhanced image to obtain a trained image coding model.

3. The method according to claim 2, characterized in that The step of performing enhancement processing on the sample image based on the enhancement intensity and the first enhanced image to obtain a second enhanced image of the sample image includes: Calculating an image residual between the first enhanced image and the sample image; Performing enhancement processing on the image residual according to the enhancement strength to obtain image enhancement information of the sample image; The second enhanced image is obtained according to the sample image and the image enhancement information of the sample image.

4. The method according to claim 2, characterized in that: The image coding model includes a multimodal pre-trained neural network model and a linear layer; The step of encoding the sample image and the first enhanced image based on the image coding model to obtain the enhancement strength of the first enhanced image relative to the sample image includes: Based on the multimodal pre-trained neural network model, encoding processing is performed on the sample image and the first enhanced image respectively to obtain image features of the sample image and image features of the first enhanced image; The image features of the sample image and the image features of the first enhanced image are linearly transformed based on the linear layer to obtain the enhancement strength.

5. The method according to claim 4, characterized in that The step of training the image coding model according to the first enhanced image and the second enhanced image to obtain the trained image coding model comprises: calculating an image loss between the first enhanced image and the second enhanced image; The parameters of the linear layer are adjusted based on the image loss to obtain an adjusted linear layer, so as to obtain the trained image coding model.

6. The method according to any one of claims 1 to 5, characterized in that: The first image enhancement model is a pre-trained enhancement model.

7. An image enhancement method, characterized in that: The method is applied to a cloud device, comprising: Get the target image; Performing image enhancement processing on the target image based on a first image enhancement model to obtain a first target enhanced image of the target image; Based on the trained image coding model, the target image and the first target enhanced image are encoded to obtain a target enhancement strength of the first target enhanced image relative to the target image; the trained image coding model is obtained by the model training method according to any one of claims 1 to 6; The target enhancement strength is sent to a terminal device; the target enhancement strength is used to perform image enhancement processing on the target image.

8. An image enhancement method, characterized in that: The method is applied to a terminal device, comprising: Get the target image; Receiving a target enhancement strength of the target image sent by a cloud device; the target enhancement strength is obtained by encoding the target image and the first target enhanced image based on a trained image encoding model; Performing image enhancement processing on the target image based on a second image enhancement model to obtain a second target enhanced image of the target image; The target image is enhanced based on the target enhancement strength and the second target enhanced image to obtain a third target enhanced image of the target image.

9. A model training device, characterized in that: The device is applied to a cloud device, including: A sample acquisition module is configured to acquire a training sample set; the training sample set includes a plurality of sample images; A sample enhancement module is configured to perform enhancement processing on the sample image based on a first image enhancement model to obtain a first enhanced image of the sample image; the resolution of the first enhanced image is higher than the resolution of the sample image; a sample encoding module, configured to perform encoding processing on the sample image and the first enhanced image based on an image encoding model to obtain an enhancement strength of the first enhanced image relative to the sample image; The model training module is configured to train the image coding model according to the sample image, the first enhanced image and the enhancement strength to obtain a trained image coding model.

10. An image enhancement device, characterized in that: The device is applied to a cloud device, including: A first image acquisition module is configured to acquire a target image; A first image enhancement module is configured to perform image enhancement processing on the target image based on a first image enhancement model to obtain a first target enhanced image of the target image; an image encoding module, configured to perform encoding processing on the target image and the first target enhanced image based on a trained image encoding model to obtain a target enhancement strength of the first target enhanced image relative to the target image; the trained image encoding model is obtained by the model training method according to any one of claims 1 to 6; The intensity sending module is configured to send the target enhancement intensity to the terminal device; the target enhancement intensity is used to perform image enhancement processing on the target image.

11. An image enhancement device, characterized in that: The device is applied to a terminal device, comprising: A second image acquisition module is configured to acquire a target image; An intensity receiving module is configured to receive a target enhancement intensity of the target image sent by a cloud device; the target enhancement intensity is obtained by encoding the target image and the first target enhanced image based on a trained image encoding model; A second image enhancement module is configured to perform image enhancement processing on the target image based on a second image enhancement model to obtain a second target enhanced image of the target image; The third image enhancement module is configured to perform enhancement processing on the target image based on the target enhancement strength and the second target enhanced image to obtain a third target enhanced image of the target image.

12. An electronic device, characterized in that: include: processor; A memory for storing the processor executable instructions; wherein the processor is configured to execute the executable instructions to implement the model training method as described in any one of claims 1 to 6, or to implement the image enhancement method as described in claim 7, or to implement the image enhancement method as described in claim 8.

13. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the model training method as described in any one of claims 1 to 6, or execute the image enhancement method as described in claim 7, or execute the image enhancement method as described in claim 8.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the model training method as described in any one of claims 1 to 6 is implemented, or the image enhancement method as described in claim 7 is implemented, or the image enhancement method as described in claim 8 is implemented.

Images