Image processing method, device, equipment and readable storage medium

Through multi-particle distillation training technology, the image enhancement benchmark model with wider width and deeper depth is improved, and the performance and accuracy of the image enhancement model is solved, which is the problem of low image quality when the bandwidth is limited, and the higher Quality image output.

CN116797466BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210250095.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-14
Publication Date
2024-11-26
Estimated Expiration
2042-03-14

AI Technical Summary

Technical Problem

In the prior art, when playing multimedia data, bandwidth limitations lead to low media quality, and the quality of multimedia data uploaded by creators is uncontrollable, resulting in low image clarity and unclear blur.

Method used

By acquiring sample images and inputting them into the image enhancement model, the first image enhancement benchmark model and the second image enhancement benchmark model, the multi-particle distillation training technology combines the benchmark model with wider width and deeper depth, transferring knowledge to improve the Image enhancement model performance and accuracy.

Benefits of technology

The training efficiency of the image enhancement model and the quality of the output image are significantly improved, making the processed image quality higher, and solving the problem of low media quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116797466B_ABST
    Figure CN116797466B_ABST
Patent Text Reader

Abstract

The present application discloses an image processing method, apparatus, device and readable storage medium, the method comprising: obtaining a sample image, inputting the sample image into an image enhancement model, a first image enhancement benchmark model and a second image enhancement benchmark model respectively; outputting an enhanced image corresponding to the sample image through the image enhancement model, outputting a first pseudo-label image corresponding to the sample image through the first image enhancement benchmark model, and outputting a second pseudo-label image corresponding to the sample image through the second image enhancement benchmark model; performing distillation training on the image enhancement model through the enhanced image, the first pseudo-label image and the second pseudo-label image to obtain a target image enhancement model, and performing image enhancement processing on the target image based on the target image enhancement model. By adopting the present application, the image quality of the output image can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, device and readable storage medium. Background Art

[0002] With the rapid development of mobile Internet and multimedia technology, watching multimedia data (such as videos, etc.) has gradually become a daily entertainment method. However, when playing multimedia data, there are often many problems. For example, when the bandwidth of the playback end is limited, only the low bit rate mode can be selected for viewing, and the low bit rate mode brings the problem of low media quality and blurriness; for example, the quality of multimedia data uploaded by the creator is uncontrollable, which will also cause the objects in the multimedia data to be blurred. In other words, when playing multimedia data, the multimedia data played may have the problem of low clarity and low media quality. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, apparatus, device, and readable storage medium, which can improve the image quality of output images.

[0004] An embodiment of the present application provides an image processing method, including:

[0005] Obtain a sample image, and input the sample image into an image enhancement model, a first image enhancement benchmark model, and a second image enhancement benchmark model respectively; the width of the network structure of the first image enhancement benchmark model is greater than the width of the network structure of the image enhancement model; the depth of the network structure of the second image enhancement benchmark model is greater than the depth of the network structure of the image enhancement model;

[0006] Output an enhanced image corresponding to the sample image through the image enhancement model, output a first pseudo-label image corresponding to the sample image through the first image enhancement benchmark model, and output a second pseudo-label image corresponding to the sample image through the second image enhancement benchmark model;

[0007] The image enhancement model is distilled and trained by using the enhanced image, the first pseudo-label image and the second pseudo-label image to obtain a target image enhancement model, and image enhancement processing is performed on the target image based on the target image enhancement model.

[0008] An embodiment of the present application provides an image processing device, including:

[0009] A sample acquisition module, used for acquiring a sample image;

[0010] A sample input module, used to input sample images into the image enhancement model, the first image enhancement benchmark model and the second image enhancement benchmark model respectively; the width of the network structure of the first image enhancement benchmark model is greater than the width of the network structure of the image enhancement model; the depth of the network structure of the second image enhancement benchmark model is greater than the depth of the network structure of the image enhancement model;

[0011] An image output module, used to output an enhanced image corresponding to the sample image through an image enhancement model, output a first pseudo-label image corresponding to the sample image through a first image enhancement benchmark model, and output a second pseudo-label image corresponding to the sample image through a second image enhancement benchmark model;

[0012] A model training module, used to perform distillation training on the image enhancement model through the enhanced image, the first pseudo-label image and the second pseudo-label image to obtain a target image enhancement model;

[0013] The model application module is used to perform image enhancement processing on the target image based on the target image enhancement model.

[0014] In one embodiment, the sample acquisition module includes:

[0015] An input image acquisition unit is used to acquire an original input image; the original input image includes a target object;

[0016] A region extraction unit is used to extract the region where the key part of the target object is located in the original input image to obtain a part region image;

[0017] An angle resetting unit, used for performing angle resetting processing on key parts in the part region image to obtain an original label image;

[0018] The image degradation unit is used to degrade the original label image to obtain a sample image; the image quality of the sample image is lower than the image quality of the original label image.

[0019] In one embodiment, the model training module includes:

[0020] A loss value determining unit, configured to determine a width loss value for the image enhancement model according to the enhanced image and the first pseudo-label image;

[0021] The loss value determining unit is further used to determine a depth loss value for the image enhancement model according to the enhanced image and the second pseudo-label image;

[0022] The loss value determining unit is further used to determine a target loss value for the image enhancement model according to the width loss value and the depth loss value;

[0023] The parameter adjustment unit is used to adjust the model parameters of the image enhancement model according to the target loss value to obtain the target image enhancement model.

[0024] In one embodiment, the loss value determination unit includes:

[0025] A loss value determination subunit, used to determine an image perception loss value for the image enhancement model according to the enhanced image and the first pseudo-label image;

[0026] The loss value determination subunit is further used to determine an image style loss value for the image enhancement model according to the enhanced image and the first pseudo-label image;

[0027] The loss value determination subunit is also used to obtain the image smoothing loss value corresponding to the sample image;

[0028] The loss value determination subunit is also used to obtain the feature difference loss value corresponding to the sample image;

[0029] The width loss value determination subunit is further used to determine the width loss value according to the image perception loss value, the image style loss value, the image smoothness loss value and the feature difference loss value.

[0030] In one embodiment, the loss value determination subunit is further specifically used to perform convolution processing on the enhanced image to obtain a first image convolution feature corresponding to the enhanced image;

[0031] The loss value determination subunit is further specifically used to perform convolution processing on the first pseudo-label image to obtain a second image convolution feature corresponding to the first pseudo-label image;

[0032] The loss value determination subunit is also specifically used to obtain an image perception loss function, and determine the image perception loss value according to the first image convolution feature, the second image convolution feature and the image perception loss function.

[0033] In one embodiment, the loss value determination subunit is further specifically used to extract enhanced image style features of the enhanced image and label image style features of the first pseudo label image;

[0034] The loss value determination subunit is also specifically used to obtain the image style loss function, and determine the image style loss value according to the enhanced image style features, the label image style features and the image style loss function.

[0035] In one embodiment, the loss value determination subunit is further specifically used to obtain a target pixel pair in the enhanced image; the target pixel pair is composed of a first pixel and a second pixel, and the first pixel and the second pixel are adjacent pixels in the enhanced image;

[0036] The loss value determination subunit is further specifically used to determine a pixel difference between a first pixel value corresponding to the first pixel and a second pixel value corresponding to the second pixel;

[0037] The loss value determination subunit is also specifically used to obtain an image smoothing loss function, and determine the image smoothing loss value based on the pixel difference and the image smoothing loss function.

[0038] In one embodiment, the loss value determination subunit is further specifically used to extract the image convolution feature corresponding to the sample image through the feature extraction layer of the image enhancement model to obtain the first sample convolution feature;

[0039] The loss value determination subunit is further specifically used to extract the image convolution feature corresponding to the sample image through the feature extraction layer of the first image enhancement benchmark model to obtain the second sample convolution feature;

[0040] The loss value determination subunit is also specifically used to obtain a feature difference loss function, and determine a feature difference loss value for the image enhancement model according to the first sample convolution feature, the second sample convolution feature and the feature difference loss function.

[0041] In one embodiment, the model training module includes:

[0042] An intermediate model determination unit, used to perform distillation training on the image enhancement model through the enhanced image, the first pseudo-label image, and the second pseudo-label image to obtain an intermediate image enhancement model;

[0043] An iteration number obtaining unit, used to obtain the model iteration number corresponding to the intermediate image enhancement model;

[0044] A model determination unit, configured to determine the intermediate image enhancement model as the image enhancement model if the number of model iterations satisfies a model convergence condition;

[0045] A model adjustment unit, used for obtaining a true label image corresponding to the sample image if the number of model iterations does not meet the model convergence condition;

[0046] The model adjustment unit is further used to adjust the first image enhancement benchmark model according to the real label image and the first pseudo label image to obtain a first adjusted enhanced benchmark model;

[0047] The model adjustment unit is further used to adjust the second image enhancement benchmark model according to the real label image and the second pseudo label image to obtain a second adjusted enhanced benchmark model;

[0048] The model adjustment unit is also used to perform distillation training on the intermediate image enhancement model based on the sample image, the first updated pseudo-label image re-output by the first adjusted enhanced baseline model, and the second updated pseudo-label image re-output by the second adjusted enhanced baseline model to obtain the target image enhancement model.

[0049] In one embodiment, the model adjustment unit includes:

[0050] The label image acquisition subunit is used to acquire the original label image corresponding to the sample image; the sample image is obtained by degrading the original label image;

[0051] The real label determination subunit is used to determine the original label image as the real label image.

[0052] In one embodiment, the model adjustment unit includes:

[0053] The original image acquisition subunit is used to acquire the original label image corresponding to the sample image; the sample image is obtained by degrading the original label image;

[0054] An initial model training subunit, used for inputting the sample image and the original label image into the label generation initial model;

[0055] The initial model training subunit is further used to train the label generation initial model through the sample image and the original label image to obtain the label generation model;

[0056] The real label determination subunit is used to input the sample image into the label generation model, output the label output result image corresponding to the sample image through the label generation model, and determine the label output result image as the real label image.

[0057] In one embodiment, the initial model training subunit is further specifically used to obtain a model adversarial loss function, and determine a model adversarial loss value for generating an initial model for the label according to the model adversarial loss function, the sample image, and the original label image;

[0058] The initial model training subunit is further specifically used to obtain a perceptual loss function, and determine a perceptual loss value of an initial model for label generation according to the perceptual loss function, the sample image, and the original label image;

[0059] The initial model training subunit is further specifically used to obtain a reconstruction loss function, and determine a reconstruction loss value of the initial model for label generation according to the reconstruction loss function, the sample image and the original label image;

[0060] The initial model training subunit is also specifically used to generate a target model loss value based on the model adversarial loss value, the perception loss value, and the reconstruction loss value, and to train the initial label generation model based on the target model loss value to obtain a label generation model.

[0061] An embodiment of the present application provides a computer device, including: a processor and a memory;

[0062] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method in the embodiment of the present application.

[0063] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the method in the embodiment of the present application is executed.

[0064] In one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in one aspect of the embodiments of the present application.

[0065] In an embodiment of the present application, after obtaining a sample image, the sample image can be respectively input into an image enhancement model, a first image enhancement benchmark model, and a second image enhancement benchmark model. Through the first pseudo-label image output by the first image enhancement benchmark model, the second pseudo-label image output by the second image enhancement benchmark model, and the enhanced image output by the image enhancement model, the image enhancement model can be distilled and trained to obtain an image enhancement model. Among them, the network structure of the first image enhancement benchmark model is wider than the image enhancement model, and the network structure of the second image enhancement benchmark model is deeper than the image enhancement model. Through distillation training, the knowledge of the first image enhancement benchmark model and the second image enhancement benchmark model can be continuously transferred to the image enhancement model. In other words, the present application can use a first image enhancement benchmark model with a wider width and a second image enhancement benchmark model with a deeper depth to simultaneously perform supervised training to achieve the purpose of distilling the image enhancement model, and finally obtain a target image enhancement model with higher performance and accuracy. Through multi-granularity distillation (i.e., a first image enhancement benchmark model with a wider width and a second image enhancement benchmark model with a deeper depth), the fitting ability of the image enhancement model can be greatly improved, so that the image enhancement model can be distilled and learned more quickly to obtain a target image enhancement model with higher performance and accuracy. The quality of the output image processed by the trained target image enhancement model will also be higher. In summary, the present application can improve the training efficiency of the image enhancement model and improve the image quality of the output image. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 This is a network architecture diagram provided by an embodiment of the present application;

[0068] Figure 2 This is a schematic diagram of a scenario provided by an embodiment of the present application;

[0069] Figure 3 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0070] Figure 4 It is a schematic diagram of a process of performing distillation training on an image enhancement model provided in an embodiment of the present application;

[0071] Figure 5 is a system structure diagram of a training label generation model provided in an embodiment of the present application;

[0072] Figure 6 is a system structure diagram of a multi-granularity distillation training provided in an embodiment of the present application;

[0073] Figure 7 It is a system flow chart provided by an embodiment of the present application;

[0074] Figure 8 It is a structural schematic diagram of a method provided in an embodiment of the present application;

[0075] Fig. 9 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0078] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0079] The solution provided in the embodiment of the present application belongs to the computer vision technology (Computer Vision, CV) and machine learning (Machine Learning, ML) under the field of artificial intelligence.

[0080] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to identify and measure targets, and further perform graphics processing to make computer processing into images that are more suitable for observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies.

[0081] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0082] See also Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a service server 1000 and a terminal device cluster (i.e., a terminal device cluster). The terminal device cluster may include one or more terminal devices, and the number of terminal devices is not limited here. Figure 1 As shown, the multiple terminal devices may specifically include terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n. Figure 1 As shown, the terminal device 100a, the terminal device 100b, the terminal device 100c, ..., the terminal device 100n can be respectively connected to the above-mentioned business server 1000 through a network, so that each terminal device can exchange data with the business server 1000 through the network connection. The network connection here does not limit the connection method, and can be directly or indirectly connected through a wired communication method, directly or indirectly connected through a wireless communication method, or through other methods, which are not limited in this application.

[0083] like Figure 1Each terminal device shown in the figure can be integrated with a target application. When the target application is running in each terminal device, the background server corresponding to each terminal device can store the business data in the application and Figure 1 Data interaction is performed between the business servers 1000 shown. Among them, the target application may include an application with data information functions such as displaying text, images, audio and video. For example, the application can be a multimedia application (such as a video application), which can be used for users to upload pictures or videos, and can also be used for users to play and watch images or videos uploaded by others; the application can also be an entertainment application, which can be used for users to upload pictures or videos, and obtain special effects pictures or special effects videos with special effects (for example, funny effects, two-dimensional effects, magnification effects, etc.); the application can also be a picture beautification application, which can be used for users to upload pictures or videos, and obtain beautified pictures or beautified videos with beautification effects (for example, enlarged eyes, smaller lips, etc.). The application can also be other applications with data information processing functions, such as educational applications, social applications, etc., and the applications will not be illustrated one by one here. The business server 1000 in the present application can collect business data from the background of these applications (such as the background server cluster corresponding to the above-mentioned terminal device cluster). For example, the business data can be pictures uploaded by users, etc. The business server 1000 can identify the target object from these business data and perform image enhancement processing on the key parts of the target object, so that the key parts of the target object can have higher clarity and higher image quality; then, the business server 1000 can perform subsequent processing on the target object after image enhancement (for example, special effects processing, beautification processing, etc.).

[0084] In the embodiment of the present application, a terminal device can be selected from multiple terminal devices as the target terminal device, and the terminal device can include: smart phones, tablet computers, laptops, desktop computers, smart TVs, smart speakers, desktop computers, smart watches, smart cars and other smart terminals with data processing functions (such as image data processing functions), but are not limited to these. Figure 1 The terminal device 100a shown is used as the target terminal device. The target application mentioned above can be integrated into the target terminal device. At this time, the target terminal device can exchange data with the business server 1000.

[0085] For example, when a user uses a target application (such as a video application) in a terminal device, the business server 1000 can detect and collect, through the terminal device, that the user has uploaded a target image containing a target object. The business server 1000 can identify the area to which the target object belongs in the target image, and extract the area from the target image to obtain an image containing only the target object but not the background (which can be referred to as a target area image). Subsequently, the business server 1000 can perform image enhancement processing on the target area image containing only the target object to obtain a target area image with higher image quality (such as higher clarity, more obvious line boundaries, etc.). Subsequently, the business server 1000 can put the target area image with higher image quality back into the area to which the target object belongs in the target image to obtain a target enhanced image with higher image quality corresponding to the target image. Subsequently, the business server 1000 can return the target enhanced image to the terminal device, and the user can view the target enhanced image (view the target object with higher image quality) on the display page of the terminal device.

[0086] Among them, the specific method for the business server 1000 to perform image enhancement processing on the target area image can be processed according to the target image enhancement model. In order to improve the display effect after image enhancement processing, the image enhancement model can be trained and adjusted so that the target image enhancement model obtained after training and adjustment reaches the optimal value. Based on the trained target image enhancement model, the target image can be enhanced (such as performing image enhancement processing on the target area image in the target image). For the specific process of training and adjusting the image enhancement model to obtain the target image enhancement model, please refer to the subsequent Figure 3 The description of steps S101 to S103 in the corresponding embodiment.

[0087] Optionally, it is understandable that the background server corresponding to the terminal device can detect and collect pictures or videos uploaded by the user, and the background server can identify the area to which the target object (such as animals, buildings, etc.) belongs in the target image, and extract the area from the target image to obtain a target area image that only contains the target object but not the background. Subsequently, the background server can perform image enhancement processing on the target area image that only contains the target object, or on the background image that does not contain the target object, to obtain a target area image with higher quality. After placing the target area image back into the area to which the target object belongs in the target image, a complete target image containing the target object with higher image quality can be obtained. For the target image with higher image quality, the background server can perform subsequent processing (such as special effects processing, beauty processing, etc.), so as to obtain a target object or background image with higher image quality and special effects or beauty effects.

[0088] It is understandable that the method provided in the embodiments of the present application can be executed by a computer device, including but not limited to a terminal device or a business server. Among them, the business server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0089] The terminal device and the service server may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0090] Optionally, it can be understood that the above-mentioned computer device (such as the above-mentioned business server 1000, terminal device 100a, terminal device 100b, etc.) can be a node in a distributed system, wherein the distributed system can be a blockchain system, and the blockchain system can be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes can form a peer-to-peer (P2P, Peer To Peer) network, and the P2P protocol is an application layer protocol running on the transmission control protocol (TCP, Transmission Control Protocol) protocol. In a distributed system, any form of computer equipment, such as business servers, terminal devices and other electronic devices, can become a node in the blockchain system by joining the peer-to-peer network. For ease of understanding, the concept of blockchain will be explained below: Blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism and encryption algorithm, which is mainly used to organize data in chronological order and encrypt it into an account book so that it cannot be tampered with or forged, and can verify, store and update data at the same time. When the computer device is a blockchain node, due to the tamper-proof and anti-counterfeiting characteristics of the blockchain, the data in this application (such as uploaded image data, video data, target image data after image enhancement processing, etc.) can be authentic and secure, so that the results obtained after related data processing based on these data can be more reliable.

[0091] It should be noted that in the specific implementation of this application, user information, user data (such as uploaded images, videos, etc.) and other related data need to be obtained with the authorization of the user. In other words, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0092] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. For ease of understanding, please refer to Figure 2 , Figure 2 is a schematic diagram of a scenario provided by an embodiment of the present application. Figure 2 The service server 200 shown may be the above Figure 1 The business server 1000 shown in FIG. Figure 2 The terminal device 100a shown in the figure can be Figure 1 Any terminal device selected from the terminal device cluster of the corresponding embodiment, for example, the terminal device may be the above-mentioned terminal device 100b; and Figure 2 The terminal device 100b shown in the figure can be Figure 1 Any terminal device selected from the terminal device cluster of the corresponding embodiment, for example, the terminal device may be the above-mentioned terminal device 100a.

[0093] like Figure 2 As shown, user a can be the target user. User a uploads an image 20a through the target application (such as a short video application) in the terminal device 100a, wherein the image 20a includes the target object B. The image 20a uploaded by user a can be used as the target image, and the business server 200 can receive the target image 20a through the terminal device 100a. Subsequently, the business server 200 can input the target image 20a into the target image recognition model, and the target image recognition model can identify the area where the target object B is located in the target image 20a. Figure 2 , the target image recognition model can recognize that the area where the target object B is located in the target image 20a is area P (i.e., the area included in the boundary of the target object B), and the target image recognition model can extract the image content included in area P to obtain the extracted target area image 20b. Subsequently, the service server can no longer consider other areas in the target image 20a except area P, and only perform image enhancement processing on the target area image 20b.

[0094] like Figure 2 As shown, the target area image 20b can be input into the target image enhancement model, and the target object B in the target area image 20b can be enhanced by the target image enhancement model. Figure 2As shown, the target image enhancement model can improve the clarity of the target object B, making the lines of the image clearer and more obvious. At the same time, some defects in the target object B are weakened to enhance the clarity of the image. In other words, the target area image 20b after image enhancement processing by the target image enhancement model has higher clarity. Furthermore, the business server 200 can put the target area image with higher clarity back into the area P of the target image 20a, thereby obtaining a target image 20a with higher clarity. Figure 2 As shown, the service server 200 can send the target image 20a with a higher definition to the terminal device 100b, so when user b uses the target application through the terminal device 100b and browses the target image 20a, what is viewed is the target image 20a with a higher definition. Similarly, the service server 200 can also return the target image 20a with a higher definition to the terminal device 100a, and user a can view the target image 20a with a higher definition on the display interface of the terminal device 100a.

[0095] Optionally, it is understandable that the above-mentioned target image recognition model can also be integrated into the target image enhancement model, so that the target image enhancement model can also have the function of image recognition. Then, when the target image is input into the target image enhancement model, the target object in the target image can be identified by the image recognition function in the target image enhancement model, and then the target object is subjected to image enhancement processing. Any object in the target image in the present application can be used as a target object. For example, the remaining objects (such as the object of the store) contained in the above-mentioned target image 20a except the target object B can also be used as target objects, and the target image enhancement model can also perform image enhancement processing on other target objects except the target object B at the same time.

[0096] It is understandable that in order to improve the accuracy of the area of ​​the target object (such as animals, buildings, etc.) in the target image recognized by the target image recognition model, the image recognition model can be trained and adjusted so that the target image recognition model obtained after the training and adjustment is optimal, and image recognition processing can be performed based on the target image recognition model. For example, the image recognition model can be trained by training samples with real object area labels. The image recognition model can output the predicted object area in the training sample, and the image recognition model can be iteratively trained according to the error value between the predicted object area and the real object area label, and finally a target image recognition model with higher precision and higher recognition accuracy is obtained. Similarly, in order to improve the display effect after image enhancement processing, the image enhancement initial model can be trained and adjusted so that the target image enhancement model obtained after the training and adjustment is optimal, and image enhancement processing can be performed based on the trained target image enhancement model. For the specific process of training and adjusting the image enhancement initial model to obtain the target image enhancement model, please refer to the subsequent Figure 3 The description in the corresponding embodiment.

[0097] For further understanding, please see Figure 3 , Figure 3 is a flowchart of an image processing method provided by an embodiment of the present application. The method can be performed by a terminal device (for example, the above Figure 1 or Figure 2 The terminal device shown in the figure) can also be executed by the terminal device and the service server (as shown in the figure above) Figure 1 or Figure 2 The image processing method is executed by the service server 1000 in the corresponding embodiment. For ease of understanding, this embodiment takes the method executed by the above terminal device as an example. Among them, the image processing method may at least include the following steps S101-S103:

[0098] Step S101, obtain a sample image, and input the sample image into an image enhancement model, a first image enhancement baseline model, and a second image enhancement baseline model respectively; the width of the network structure of the first image enhancement baseline model is greater than the width of the network structure of the image enhancement model; the depth of the network structure of the second image enhancement baseline model is greater than the depth of the network structure of the image enhancement model.

[0099] In this application, the image enhancement model may refer to an image enhancement initial model, and the model parameters in the image enhancement initial model are initial parameters. The image enhancement initial model here may refer to a pixel2pixel model. For the specific structure of the pixel2pixel model, this application is designed as a U-Net structure, that is, a codec module is obtained by stacking multiple convolutional layers (conv layer), that is, the image enhancement initial model may be composed of multiple codec modules and convolutional layers. Sample images can be used to train image enhancement models. Through the training of sample images, the image enhancement model can be optimized so that the enhancement effect output by the image enhancement model can be better and better (such as image clarity is getting higher and higher, that is, image quality is getting higher and higher).

[0100] It is understandable that the present application can adopt the training method of multi-granularity distillation to train the image enhancement model. Wherein, the training method of multi-granularity distillation here can refer to the large model using multiple granularities to train the image enhancement model. The large model of multiple granularities in the present application can refer to the wide teacher large model (Wider Teacher) and the deep teacher large model (Deeper Teacher). And the wide teacher large model can refer to a model whose network structure is wider than the image enhancement model. For example, assuming that the network structure for the image enhancement model has a channel number of C, the channel multiple can be taken as 4 (of course, it can also be other values ​​greater than or equal to 1, and the channel multiple 4 is taken here only for example), and the network structure with the channel number of C*4 is obtained, and the network structure can be used as the network structure of the width teacher large model. In other words, the network structure of the width teacher large model is wider than the network structure of the image enhancement model. The deep teacher model can refer to a model whose network structure is deeper than that of the image enhancement model. For example, the network structure of the image enhancement model may include multiple codec modules (each codec module may be composed of multiple convolutional layers). The present application may insert a residual module (Res Block) in different positions of these codec modules to obtain a deeper network structure, which can be used as the network structure of the deep teacher model. In other words, the network structure of the deep teacher model is deeper than that of the image enhancement model.

[0101] The first image enhancement benchmark model in the present application may refer to a large model of width teacher (i.e., the first image enhancement benchmark model may refer to a large model whose network structure is wider than that of the image enhancement model), and the second image enhancement benchmark model may refer to a large model of depth teacher (i.e., the second image enhancement benchmark model may refer to a large model whose network structure is deeper than that of the image enhancement model); both the first image enhancement benchmark model and the second image enhancement benchmark model may refer to models used as training benchmarks for image enhancement models. The training of multi-granularity distillation in the present application may refer to a method of using dual large models (i.e., large model of width teacher + large model of depth teacher) and image enhancement models to perform step-by-step iterative training at the same time. During the training process, the learning of the image enhancement model can be guided simultaneously based on two networks with different structures (i.e., the first image enhancement benchmark model and the second image enhancement benchmark model), which can greatly enhance the fitting ability of the small model (i.e., the image enhancement model), and can quickly learn the knowledge transferred by the two large models, thereby improving the training efficiency and greatly improving the enhancement effect of the image output by the small model.

[0102] From the above, it can be seen that the first image enhancement benchmark model and the second image enhancement benchmark model are mainly used as benchmarks for the image enhancement model to guide the learning of the image enhancement model. The purpose of training is mainly to make the image output by the image enhancement model closer and closer to the image output by the two large models (the first image enhancement benchmark model can also be called the first image enhancement guidance model, and the second image enhancement benchmark model can be called the second image enhancement guidance model). Then, after obtaining the sample image, the sample image needs to be input into the first image enhancement benchmark model, the second image enhancement benchmark model and the image enhancement model respectively, and the two large models and the small model are iteratively trained step by step through the sample image.

[0103] Among them, the sample image in the present application may refer to an image with low image quality (such as low clarity, blurred image, unclear line boundaries, etc.). A specific implementation method for obtaining the sample image may be: the original input image may be obtained; wherein the original input image includes the target object; then, the area where the key parts of the target object are located may be extracted from the original input image to obtain a part area image; then, the key parts in the part area image may be angle reset to obtain the original label image; the original label image may be degraded to obtain the sample image; wherein, due to the degradation, the image quality of the sample image is lower than the image quality of the original label image.

[0104] It is understandable that the original input image here may refer to an image input by a user containing a target object (such as an animal, plant, building, etc.). After the original input image is obtained, the original input image may be subjected to image recognition. For example, image recognition may be performed using a trained target image recognition model to identify the region where the key parts of the target object are located in the target image, thereby obtaining a part region image (also referred to as a sample region image). The key parts here may be defined according to the object type of the target object. For example, when the object type of the target object is an animal type, the key parts may refer to the head, neck, hand, leg, foot, etc.; and when the object type of the target object is a plant type, the key parts may refer to the stem and leaf parts, roots, etc. In other words, the key parts may be flexibly defined according to the different target objects, and the key parts will not be illustrated one by one here.

[0105] Furthermore, the key part in the part area image can be subjected to angle reset processing, so as to obtain the original label image, that is, the angle of the key part in the original label image is the standard angle (preset angle). The angle here can refer to the angle between the calibration auxiliary line corresponding to the key part and the coordinate axis in the image coordinate system in the image coordinate system; the standard angle can be understood as the angle between the calibration auxiliary line corresponding to the key part and the coordinate axis in the image coordinate system is the preset angle. The image coordinate system here can refer to a coordinate system with a vertex of the image as the origin and two image edges on the vertex (the vertex is the intersection of the two image edges) as the two coordinate axes, and the calibration auxiliary line can refer to a straight line composed of a first calibration position and a second calibration position in the key part. The first calibration position and the second calibration position can be flexibly defined according to different key parts. For example, when the key part is the head, the first calibration position can refer to the center position of the eyebrows in the head, and the second calibration position can refer to the center position of the lips in the head. In this application, the standard angle can be the angle indicated by the parallel relationship (such as 0 degrees), that is, the calibration auxiliary line and the coordinate axis (the side of a certain image) are parallel. In other words, the key parts in the original label image are angle reset, which can actually be understood as rotating the key parts based on the image and the calibration auxiliary line so that the angle formed by the calibration auxiliary line and the side of the image is the standard angle.

[0106] Furthermore, the original label image can be degraded, wherein the degrading process here can refer to converting a high-quality image (such as a key part of the target object without blur and clear lines, which can be referred to as HQ) into a low-quality image (such as blur and unclear line boundaries, which can be referred to as LQ). By analyzing the factors affecting image quality, the present application can express the degradation from HQ to LQ as shown in formula (1):

[0107]

[0108] Among them, K in formula (1) can be used to characterize blur kernel processing (blur kernel);↓ s Can be used to characterize downsampling processing; n σ Can be used to characterize noise processing, JPEG q Can be used to characterize image compression processing; I can be used to characterize high-quality images (i.e., original label images); I d It can be used to characterize low-quality images. In practical applications, for the original label image, one or two or three or four operations can be randomly selected from the above four operations (including blur kernel processing operation, downsampling processing operation, noise processing operation and image compression processing operation) as the degradation strategy of the original label image. Through the processing of the above 1 to 4 operations, a low-quality image corresponding to the original label image can be obtained, and the low-quality image can be used as a sample image.

[0109] It should be noted that in a special case, when selecting from the above four operations, if 0 operations are selected, it can be indicated that there is no need to degrade the high-quality image, and the sample image is a high-quality image. By training with high-quality images and original labeled images, the model can also be trained to perform identity mapping on high-quality images.

[0110] Step S102: output an enhanced image corresponding to the sample image through the image enhancement model, output a first pseudo-label image corresponding to the sample image through the first image enhancement benchmark model, and output a second pseudo-label image corresponding to the sample image through the second image enhancement benchmark model.

[0111] In the present application, an enhanced image corresponding to a sample image can be outputted through an image enhancement model, and an enhanced image corresponding to the sample image can be outputted through a first image enhancement benchmark model. The enhanced image outputted by the first image enhancement benchmark model is mainly used for learning the image enhancement model (as a label image of the image enhancement model), and the image outputted by the first image enhancement benchmark model can be referred to as a pseudo-label image (for ease of distinction, referred to as a first pseudo-label image); similarly, an enhanced image corresponding to the sample image can be outputted through a second image enhancement benchmark model, and the enhanced image outputted by the second image enhancement benchmark model is mainly used for learning the image enhancement model (as a label image of the image enhancement model), and the image outputted by the second image enhancement benchmark model can be referred to as a pseudo-label image (for ease of distinction, referred to as a second pseudo-label image).

[0112] Step S103, performing distillation training on the image enhancement model by using the enhanced image, the first pseudo-label image, and the second pseudo-label image to obtain a target image enhancement model, and performing image enhancement processing on the target image based on the target image enhancement model.

[0113] In the present application, the image enhancement model is distilled and trained by using an enhanced image, a first pseudo-label image, and a second pseudo-label image to obtain a target image enhancement model. The specific implementation method is as follows: based on the enhanced image and the first pseudo-label image, a width loss value for the image enhancement model can be determined; based on the enhanced image and the second pseudo-label image, a depth loss value for the image enhancement model can be determined; based on the width loss value and the depth loss value, a target loss value for the image enhancement model can be determined, and according to the target loss value, the model parameters of the image enhancement model can be adjusted to obtain the target image enhancement model.

[0114] Among them, it can be understood that both the first pseudo-label image and the second pseudo-label image can be used as label images of the enhanced image, a loss value can be determined based on the first pseudo-label image and the enhanced image, and a loss value can also be determined based on the second pseudo-label image and the enhanced image, then the loss value between the first pseudo-label image and the enhanced image can be called a width loss value; the loss value between the second pseudo-label image and the enhanced image can be called a depth loss value. The specific implementation method for determining the width loss value for the image enhancement model based on the enhanced image and the first pseudo-label image can be: based on the enhanced image and the first pseudo-label image, the image perception loss value for the image enhancement model can be determined; based on the enhanced image and the first pseudo-label image, the image style loss value for the image enhancement model can be determined; then, the image smoothing loss value corresponding to the sample image can be obtained, and the feature difference loss value corresponding to the sample image can also be obtained; the width loss value can be determined based on the image perception loss value, the image style loss value, the image smoothing loss value and the feature difference loss value.

[0115] That is, the width loss value may include an image perception loss value, an image style loss value, an image smoothing loss value, and a feature difference loss value. A specific method for determining the width loss value based on the image perception loss value, the image style loss value, the image smoothing loss value, and the feature difference loss value may be as follows: the image perception loss value, the image style loss value, the image smoothing loss value, and the feature difference loss value may be added together to obtain the width loss value. For ease of understanding, please refer to formula (2), which may be a specific implementation method for determining the width loss value:

[0116] KD loss1=w1L CD +w2L p +w3L s +w4L tv Formula (2)

[0117] Among them, KD loss in formula (2) can be used to represent the width loss value; L CD It can be used to characterize the feature difference loss value, which can refer to Channel-wise Distill Loss; L p It can be used to represent the image perceptual loss value, which can refer to perceptual loss; L s Can be used to characterize the image style loss value; L tv It can be used to characterize the image smoothing loss value, which can refer to the Total variation loss; w1, w2, w3, and w4 can be used to characterize L CD , L p , L s , L tv Corresponding weight coefficients: In a feasible embodiment, w1, w2, w3, and w4 may all be 1.

[0118] Among them, the specific implementation method of determining the image perception loss value for the image enhancement model based on the enhanced image and the first pseudo-label image can be as follows: the enhanced image can be convolved to obtain the first image convolution feature corresponding to the enhanced image; then, the first pseudo-label image can be convolved to obtain the second image convolution feature corresponding to the first pseudo-label image; then, the image perception loss function can be obtained, and the image perception loss value can be determined according to the first image convolution feature, the second image convolution feature and the image perception loss function.

[0119] It can be understood that the specific implementation method for determining the image perception loss value can be shown as formula (3):

[0120]

[0121] Among them, as shown in formula (3), L p It can be used to characterize the image perception loss value; x can be used to characterize the sample image, and G(x) can be used to characterize the enhanced image. Can be used to characterize the first pseudo-label image; Can be used to characterize the first image convolution feature; It can be used to characterize the convolution features of the second image.

[0122] Furthermore, a specific implementation method for determining the image style loss value for the image enhancement model based on the enhanced image and the first pseudo-label image may be as follows: enhanced image style features of the enhanced image and label image style features of the first pseudo-label image may be extracted; then, an image style loss function may be obtained, and the image style loss value may be determined based on the enhanced image style features, the label image style features and the image style loss function.

[0123] It can be understood that the specific implementation method for determining the image style loss value can be shown as formula (4):

[0124]

[0125] Among them, as shown in formula (4), L s can be used to characterize the image style loss value; x can be used to characterize the sample image, and G(x) can be used to characterize the enhanced image. Can be used to characterize the first pseudo-label image; Can be used to characterize the style features of label images; It can be used to characterize and enhance image style features. Among them, Gram(·) can be used to represent the Gram matrix, which is defined as any k (k <= n) vectors α1, α2, …, α in the n-dimensional Euclidean space. k The matrix composed of the inner products of .

[0126] Furthermore, a specific implementation method for obtaining an image smoothing loss value corresponding to a sample image may be as follows: a target pixel pair in an enhanced image may be obtained; wherein the target pixel pair is composed of a first pixel and a second pixel, and the first pixel and the second pixel are adjacent pixels in the enhanced image; then, a pixel difference between a first pixel value corresponding to the first pixel and a second pixel value corresponding to the second pixel may be determined; then, an image smoothing loss function may be obtained, and the image smoothing loss value may be determined based on the pixel difference and the image smoothing loss function.

[0127] It can be understood that the specific implementation method for determining the image smoothing loss value can be shown as formula (5):

[0128]

[0129] Where, as in formula (5), L tv Can be used to characterize the image smoothing loss value; (x i,j-1 -x i,j ) and (x i+1,j -x i,j ) can be used to characterize the difference between adjacent pixels in an image; it is understandable that the process of image enhancement is also accompanied by an image restoration process, and in the image restoration process, the noise on the image may have a very large impact on the restoration result, because many restoration algorithms will amplify the noise. At this time, in the model of the optimization problem, some regularization terms can be added to maintain the smoothness of the image, and TV loss (image smoothing loss value) is a commonly used regularization term (which can be used in conjunction with other losses to constrain noise). Usually, the difference in adjacent pixel values ​​in an image can be solved to a certain extent by reducing TV loss. In other words, TV loss can be used to characterize the final result of the difference in adjacent pixel values ​​in an image. By continuously reducing TV loss, the difference between adjacent pixels can be continuously reduced, making the image gradually smoother.

[0130] Furthermore, the specific implementation method for obtaining the feature difference loss value corresponding to the sample image can be as follows: through the feature extraction layer of the image enhancement model, the image convolution features corresponding to the sample image can be extracted to obtain the first sample convolution features; through the feature extraction layer of the first image enhancement baseline model, the image convolution features corresponding to the sample image can also be extracted to obtain the second sample convolution features; the feature difference loss function can be obtained, and according to the first sample convolution features, the second sample convolution features and the feature difference loss function, the feature difference loss value for the image enhancement model can be determined.

[0131] It can be understood that the specific implementation method for determining the feature difference loss value can be shown as formula (6):

[0132]

[0133] Where, as in formula (6), L CD It can be used to characterize the feature difference loss value (Channel-wise DistillLoss); x can be used to characterize the sample image, and G(x) can be used to characterize the enhanced image. Can be used to characterize the first pseudo-label image; can be used to characterize the extracted second sample convolution features; F(G(x)) can be used to characterize the extracted first sample convolution features. It can be understood that L CDIt can be used to constrain the difference between the image enhancement model and the first image enhancement baseline model in the intermediate layer feature map (different numbers of channels are mapped through additional learning modules to ensure the same). Where F(·) represents the structure used to transform the number of channels of the two intermediate layer feature maps.

[0134] It can be understood that the specific implementation process of determining the depth loss value for the image enhancement model based on the enhanced image and the second pseudo-label image is roughly consistent with the implementation process of determining the width loss value. The specific implementation method thereof may be as follows: an image perception loss value for an image enhancement model may be determined according to the enhanced image and the second pseudo-label image (for the sake of distinction, the image perception loss value determined between the enhanced image and the first pseudo-label image may be referred to as the first image perception loss value; the image perception loss value determined between the enhanced image and the second pseudo-label image may be referred to as the second image perception loss value); an image style loss value for an image enhancement model may be determined according to the enhanced image and the second pseudo-label image (for the sake of distinction, the image style loss value determined between the enhanced image and the first pseudo-label image may be referred to as the first image style loss value; the image style loss value determined between the enhanced image and the second pseudo-label image may be referred to as the second image style loss value); an image smoothing loss value corresponding to the sample image may be obtained (for the sake of distinction, the image smoothing loss value determined in the process of determining the width loss value may be referred to as the first image smoothing loss value; the image smoothing loss value determined in the process of determining the depth loss value may be referred to as the second image smoothing loss value. In practical applications, since the image smoothing loss value is determined based on the same enhanced image, the first image smoothing loss value and the second image smoothing loss value may be the same). Subsequently, the depth loss value can be determined according to the second image perception loss value, the second image style loss value, and the second image smoothness loss value. That is to say, when determining the depth loss value, compared with the width loss value, it is not necessary to determine the depth loss value based on the feature difference loss value (L CD ) to determine.

[0135] It can be understood that the specific implementation method for determining the depth loss value can be as shown in formula (7):

[0136] KD loss2=e1L p +e2L s +e3L tv Formula (7)

[0137] Among them, KD loss2 as shown in formula (7) can be used to characterize the depth loss value; L p can be used to represent an image perceptual loss value (a second image perceptual loss value), where the image perceptual loss value may refer to perceptual loss; Ls Can be used to characterize the image style loss value (second image style loss value); L tv It can be used to represent the image smoothing loss value (the second image smoothing loss value), which can refer to the Total variation loss; e1, e2, and e3 can be used to represent L p , L s , L tv Corresponding weight coefficients: In a feasible embodiment, e1, e2, and e3 may all be 1.

[0138] Among them, for L p , L s , L tv The specific determination method of , its principle can be referred to the above formula (3) - formula (5), and the first pseudo-label image is replaced by the second pseudo-label image (that is, can be used to characterize the second pseudo-label image).

[0139] Furthermore, after determining the width loss value and the depth loss value, a target loss value can be determined based on the width loss value and the depth loss value. Based on the target loss value, the model parameters of the image enhancement model can be adjusted to obtain a target image enhancement model. The specific method for determining the target loss value based on the width loss value and the depth loss value can be shown as formula (8):

[0140] L Z =q1KD loss1+q2KD loss2 Formula (8)

[0141] Among them, KD loss1 in formula (8) can be used to represent the width loss value; KD loss2 can be used to represent the depth loss value; L Z It can be used to represent the target loss value; q1 and q2 can be used to represent the weight coefficients corresponding to KD loss1 and KD loss2 respectively. In a feasible embodiment, q1 and q2 can both be 1.

[0142] It can be understood that after adjusting the model parameters of the image enhancement model based on the target loss value, the current model iteration number (which can be understood as the number of model parameter adjustments) can be counted. If the model iteration number has met the model convergence condition (such as the number of iterations reaches the number threshold), then the image enhancement model can be determined as the target image enhancement model at this time; if the model iteration number does not meet the model convergence condition (such as the number of iterations does not reach the number threshold), the real label image corresponding to the sample image can be obtained, and the first image enhancement benchmark model is trained and adjusted according to the real label image and the first pseudo-label image to obtain the first adjusted enhanced benchmark model; the second image enhancement benchmark model is trained and adjusted according to the real label image and the second pseudo-label image to obtain the second adjusted enhanced benchmark model; according to the sample image, the first updated pseudo-label image re-output by the first adjusted enhanced benchmark model, and the second updated pseudo-label image re-output by the second adjusted enhanced benchmark model, a new round of distillation training is performed on the image enhancement model with the model parameters currently adjusted to obtain the target image enhancement model.

[0143] That is to say, each time the student model is trained, for the teacher model (i.e., the first image enhancement benchmark model and the second image enhancement benchmark model), this application will also use the loss value to train the two teacher models (i.e., the first image enhancement benchmark model and the second image enhancement benchmark model) respectively, and the teacher model and the student model (i.e., the model to be trained using the first image enhancement benchmark model and the second image enhancement benchmark model as the training benchmark, such as the image enhancement model in this application is the student model) are iteratively trained step by step at the same time, so that the image quality of the enhanced images output by the two teacher models will become higher and higher, which can gradually guide the student model to output images of higher and higher quality. For the specific iterative training process of the model, please refer to the subsequent Figure 4 The description in the corresponding embodiment.

[0144] In an embodiment of the present application, after obtaining a sample image, the sample image can be input into an image enhancement model, a first image enhancement benchmark model, and a second image enhancement benchmark model, respectively. The image enhancement model can be distilled and trained by using the first pseudo-label image output by the first image enhancement benchmark model, the second pseudo-label image output by the second image enhancement benchmark model, and the enhanced image output by the image enhancement model to obtain a target image enhancement model. Among them, the network structure of the first image enhancement benchmark model is wider than the image enhancement model, and the network structure of the second image enhancement benchmark model is deeper than the image enhancement model. Through distillation training, the knowledge of the first image enhancement benchmark model and the second image enhancement benchmark model can be continuously transferred to the image enhancement model. In other words, the present application can use a wider first image enhancement benchmark model (i.e., a width teacher model) and a deeper second image enhancement benchmark model (i.e., a depth teacher model) to simultaneously perform supervised training to achieve the purpose of distilling the image enhancement model, and ultimately obtain a target image enhancement model with higher performance and accuracy. Through multi-granularity distillation (i.e., a first image enhancement benchmark model with a wider width and a second image enhancement benchmark model with a deeper depth), the fitting ability of the image enhancement model can be greatly improved, so that the image enhancement model can be distilled and learned more quickly to obtain a target image enhancement model with higher performance and accuracy. The quality of the output image processed by the trained target image enhancement model will also be higher. In summary, the present application can improve the training efficiency of the image enhancement model and improve the image quality of the output image.

[0145] For further information, see Figure 4 , Figure 4 is a schematic diagram of a process for performing distillation training on an image enhancement model provided in an embodiment of the present application. The process may correspond to the above Figure 3 In the corresponding embodiment, the specific process of performing distillation training on the image enhancement model through the enhanced image, the first pseudo-label image and the second pseudo-label image to obtain the target image enhancement model is as follows: Figure 4 As shown, the process may include at least the following steps S201 to S206:

[0146] Step S201, performing distillation training on the image enhancement model by using the enhanced image, the first pseudo-label image and the second pseudo-label image to obtain an intermediate image enhancement model.

[0147] Specifically, for the definitions of the enhanced image, the first pseudo-label image, the second pseudo-label image, and the image enhancement model, please refer to the above Figure 3 The description in the corresponding embodiment is as follows: Figure 3It can be seen from the corresponding embodiment that the width loss value can be determined by the enhanced image and the first pseudo-label image; the depth loss value can be determined by the enhanced image and the second pseudo-label image; and based on the width loss value and the depth loss value, the target loss value can be determined, and the model parameters of the image enhancement model can be adjusted based on the target loss value. The image enhancement model with the model parameters adjusted can be called an intermediate image enhancement model.

[0148] Step S202, obtaining the number of model iterations corresponding to the intermediate image enhancement model.

[0149] Specifically, after the model parameters of the image enhancement model can be adjusted based on the target loss value, the current number of model iterations (which can be understood as the number of model parameter adjustments) can be counted.

[0150] Step S203: If the number of model iterations meets the model convergence condition, the intermediate image enhancement model is determined as the image enhancement model.

[0151] Specifically, if the number of model iterations meets the model convergence condition (such as the number of model iterations reaches an iteration number threshold, where the iteration number threshold may be a manually specified value), the intermediate image enhancement model may be directly determined as the target image enhancement model.

[0152] Step S204: If the number of model iterations does not meet the model convergence condition, a real label image corresponding to the sample image is obtained, and the first image enhancement benchmark model is adjusted according to the real label image and the first pseudo label image to obtain a first adjusted enhanced benchmark model.

[0153] Specifically, if the number of model iterations does not meet the model convergence condition (such as the number of model iterations does not reach the iteration number threshold), the real label image corresponding to the sample image can be obtained, and the first image enhancement benchmark model can be adjusted according to the real label image and the first pseudo label image to obtain a first adjusted enhanced benchmark model. Among them, the specific implementation method for obtaining the real label image corresponding to the sample image can be: the original label image corresponding to the sample image can be obtained; wherein the sample image is obtained by degrading the original label image; the original label image can be determined as the real label image. As described above, the sample image can refer to an image obtained by degrading the original label image (i.e., a high-quality image), and the real label image here can refer to the original label image.

[0154] Optionally, it can be understood that the specific implementation method for obtaining the true label image corresponding to the sample image can also be: the original label image corresponding to the sample image can be obtained; wherein the sample image is obtained by degrading the original label image; then, the sample image and the original label image can be input into the label generation initial model, and the label generation initial model can be trained through the sample image and the original label image to obtain the label generation model; then, the sample image can be input into the label generation model, and the label output result image corresponding to the sample image can be output through the label generation model, and the label output result image can be determined as the true label image.

[0155] Among them, for training the initial label generation model through sample images and original label images, the specific implementation method of obtaining the label generation model can be: the model adversarial loss function can be obtained, and the model adversarial loss value of the initial label generation model can be determined according to the model adversarial loss function, the sample image and the original label image; then, the perceptual loss function can be obtained, and the perceptual loss value of the initial label generation model can be determined according to the perceptual loss function, the sample image and the original label image; the reconstruction loss function can be obtained, and the reconstruction loss value of the initial label generation model can be determined according to the reconstruction loss function, the sample image and the original label image; the target model loss value can be generated according to the model adversarial loss value, the perceptual loss value and the reconstruction loss value, and the initial label generation model can be trained according to the target model loss value to obtain the label generation model.

[0156] It is understandable that after the original label image is degraded, a sample image can be obtained, and in the present application, a label generation model (the label generation model can actually be understood as a model with image enhancement processing function, for the sake of distinction, it is referred to as a label generation model) can be trained based on the sample image and the original label image to obtain a trained label generation model, and the enhanced image (i.e., the label output result image) output by the trained label generation model can be used as the real label image corresponding to the sample image, and the real label image is used as the real label image in the subsequent multi-granularity distillation training stage for iterative training of the two teacher models. It should be understood that compared to using the original label image as the real label image in the multi-granularity distillation training stage, using the label output result image can further reduce the gap (gap) of the small model (i.e., the student model) fitting, and can make it easier for the student model in the multi-granularity distillation training stage to learn the fitting ability of high-definition data (i.e., high-quality images). It can be seen that the significance of using the label output result image as the real label image in the multi-granularity distillation training stage should not be underestimated.

[0157] It can be understood that the specific implementation method for determining the adversarial loss value of the above model can be shown as formula (9):

[0158]

[0159] Among them, as shown in formula (9), L A can be used to characterize the model adversarial loss value, which can refer to adversarial loss; x can be used to characterize the sample image, and G(x) can be used to characterize the first pseudo-label image; E (x) It can be used to characterize the loss of sample images; D(G(x)) can be used to obtain the image difference features output by the discriminator after the output image of the model and the true label image are input into the discriminator.

[0160] For the specific implementation of determining the above-mentioned perceptual loss value, please refer to the above-mentioned formula (3). When determining the perceptual loss value of the real label image and the first pseudo-label image, x in formula (3) can be used to represent the sample image, and G(x) can be used to represent the first pseudo-label image; Can be used to characterize true label images.

[0161] The specific implementation method for determining the above reconstruction loss value can be shown as formula (10):

[0162]

[0163] Among them, as shown in formula (10), L r can be used to characterize a reconstruction loss value, and the reconstruction loss value can refer to reconstruction loss; x can be used to characterize a sample image, and G(x) can be used to characterize a first pseudo-label image; Can be used to characterize true label images.

[0164] Furthermore, after determining the model adversarial loss value, the perceptual loss value, and the reconstruction loss value, the target model loss value can be determined based on the model adversarial loss value, the perceptual loss value, and the reconstruction loss value. Based on the target model loss value, the label generation initial model can be trained to obtain the label generation model. The specific implementation method for determining the target model loss value can be shown as formula (11):

[0165] L=L A +αL p +βL r Formula (11)

[0166] Wherein, L as shown in formula (11) can be used to characterize the target model loss value; L A Can be used to characterize the model's adversarial loss value, L p Can be used to characterize the perceived loss value; L rIt can be used to characterize the reconstruction loss value; α and β can be used to characterize the weight coefficients corresponding to the perception loss value and the reconstruction loss value, respectively.

[0167] For easier understanding, please also refer to Figure 5 , Figure 5 is a system structure diagram of a training label generation model provided in an embodiment of the present application. Figure 5 As shown, the label generation initial model can be composed of multiple convolutional layers, multiple encoding and decoding modules and conversion modules, wherein the encoding and decoding modules and the conversion modules can be stacked by multiple convolutional layers. After the sample image is input into the label generation initial model, the application can refer to the alpla blend method in the output layer. The label generation initial model can output the Attention Alpha image (which can be called the attention image) and the Pred RGB image. The Attention Alpha image, the Pred RGB image and the sample image can be fused by the alplablend method, so as to obtain the image finally output by the model (i.e., the output result image). According to the original label image corresponding to the sample image and the output result image, the perceptual loss value and the reconstruction loss value can be generated. After the sample image and the original label image are input into the discriminator, the model adversarial loss value can be generated. Based on the perceptual loss value, the reconstruction loss value and the model adversarial loss value, the label generation initial model can be trained to obtain the label generation model. After the sample image is input into the label generation model, according to the label generation model, the label output result image can be output as the real label image.

[0168] It can be understood that after obtaining the real label image, the specific implementation method for adjusting the first image enhancement benchmark model according to the real label image and the first pseudo label image to obtain the first adjusted enhanced benchmark model can be: the first pseudo label image and the real label image can be input into the discriminator, and the discriminator can output the image difference features corresponding to the first pseudo label image and the real label image; then, the image adversarial loss function can be obtained, and the image adversarial loss value for the first image enhancement benchmark model can be determined according to the image adversarial loss function and the image difference features; the model parameters of the first image enhancement benchmark model are adjusted according to the image adversarial loss value to obtain the first adjusted enhanced benchmark model. It can be understood that for the loss value of the first adjusted enhanced benchmark model, an adversarial loss value can be used. For the specific determination method of the adversarial loss value, please refer to the above description, which will not be repeated here.

[0169] Step S205: adjusting the second image enhancement benchmark model according to the real label image and the second pseudo label image to obtain a second adjusted enhancement benchmark model.

[0170] Specifically, consistent with the first adjusted enhanced benchmark model, the loss value of the second adjusted enhanced benchmark model may also adopt an adversarial loss value. The specific implementation method of training and adjusting the second image enhancement benchmark model is the same as the principle of training and adjusting the first image enhancement benchmark model, and will not be repeated here.

[0171] Step S206, performing distillation training on the intermediate image enhancement model according to the sample image, the first updated pseudo-label image re-output by the first adjusted enhanced baseline model, and the second updated pseudo-label image re-output by the second adjusted enhanced baseline model to obtain a target image enhancement model.

[0172] Specifically, the sample image can be re-input into the first adjustment enhancement benchmark model, the second adjustment enhancement benchmark model and the intermediate image enhancement model. The first updated pseudo-label image can be re-output through the first adjustment enhancement benchmark model, the second updated pseudo-label image can be re-output through the second adjustment enhancement benchmark model, and the updated enhanced image can be re-output through the intermediate image enhancement model. Based on the first updated pseudo-label image, the second updated pseudo-label image and the intermediate image enhancement model, a new round of distillation training can be performed on the intermediate image enhancement model. For its specific implementation method, please refer to the above Figure 3 The description of the corresponding embodiment will not be repeated here. It can be understood that by iteratively training the two large models and the small model step by step at the same time, the model parameters of the two large models can be continuously adjusted, so as to guide the small model to learn until the small model meets the model convergence conditions, thereby continuously improving the accuracy of the small model, so that the small model can have better image enhancement capabilities.

[0173] In an embodiment of the present application, a first image enhancement benchmark model with a wider width (i.e., a width teacher model) and a second image enhancement benchmark model with a deeper depth (i.e., a depth teacher model) are used to simultaneously perform supervised training to achieve the purpose of distilling the image enhancement model, and ultimately obtain a target image enhancement model with higher performance and accuracy. Through multi-granularity distillation (i.e., a first image enhancement benchmark model with a wider width and two granularities of a second image enhancement benchmark model with a deeper depth), the fitting ability of the image enhancement model can be greatly improved, so that the image enhancement model can be distilled and learned more quickly, and a target image enhancement model with higher performance and accuracy is obtained. The quality of the image processed and output by the trained target image enhancement model will also be higher.

[0174] For further understanding, please refer to Figure 6 , Figure 6 is a system structure diagram of a multi-granularity distillation training provided in an embodiment of the present application. Figure 6As shown, the image enhancement model can be composed of multiple convolutional layers, multiple encoding and decoding modules and conversion modules, wherein the encoding and decoding modules and the conversion modules can be stacked by multiple convolutional layers. The width teacher model can refer to a model with a wider network structure (more channels) than the image enhancement model, and the width teacher model can also be called a width reference model, which can refer to the first image enhancement reference model mentioned above; the depth teacher model can refer to a model with a deeper network structure (adding a residual module) than the image enhancement model, and the depth teacher model can also be called a depth reference model, which can refer to the second image enhancement reference model mentioned above. After the sample image is input into the image enhancement model, the width teacher model and the depth teacher model, the width teacher model can output a first pseudo-label image, the depth teacher model can output a second pseudo-label image, and the image enhancement model can output an enhanced image. At the same time, after the sample image is input into the width teacher model, the width teacher model can extract the intermediate layer features of the sample image (which can be called convolution features, i.e. feature map). After the sample image is input into the image enhancement model, the image enhancement model can extract the intermediate layer features of the sample image (which can be called convolution features, i.e. feature map). Based on the intermediate layer features extracted by the two models, the feature difference loss value (i.e. L CD ); based on the first pseudo-label image and the enhanced image and the feature difference loss value, the width loss value can be determined; based on the second pseudo-label image and the enhanced image, the depth loss value can be determined; according to the width loss value and the depth loss value, the student model can be trained and adjusted.

[0175] At the same time, according to the first pseudo-label image and the real label image, the wide teacher model can be trained and adjusted; according to the second pseudo-label image and the real label image, the deep teacher model can be trained and adjusted. It should be understood that multi-granularity distillation training uses dual large models (Wider Teacher + Deeper Teacher) and small models (Student) to iterate training at the same time. The dual large models can be used to guide the learning of the student model at the same time, which can greatly enhance the fitting ability of the student model, making the gap between the generated results and the large model smaller, and improving the image quality of the model output image.

[0176] For further information, see Figure 7 , Figure 7 This is a system flow chart provided by the embodiment of the present application. Figure 7 As shown, the process may include at least the following steps S21 to S25:

[0177] Step S21: Obtain the original input image.

[0178] Step S22: Acquire high-quality images, and obtain low-quality-high-quality training data pairs through a quality reduction strategy.

[0179] Specifically, the high-quality image here may refer to the original label image, and the low-quality-high-quality training data pair may refer to an image pair consisting of the original label image and the sample image.

[0180] Step S23: Train the pixel2pixel model to obtain a label generation model.

[0181] Specifically, the label generation model can output the true label image corresponding to the sample image.

[0182] Step S24: adopt a multi-granularity distillation training method to perform distillation training on the image enhancement model to obtain a target image enhancement model.

[0183] Step S25: Perform image enhancement processing using the target image enhancement model.

[0184] Specifically, for the specific implementation of steps S21 to S25, please refer to the previous Figure 3-Figure 4 The description in the corresponding embodiment will not be repeated here.

[0185] In an embodiment of the present application, a first image enhancement benchmark model with a wider width (i.e., a width teacher model) and a second image enhancement benchmark model with a deeper depth (i.e., a depth teacher model) are used to simultaneously perform supervised training to achieve the purpose of distilling the image enhancement model, and ultimately obtain a target image enhancement model with higher performance and accuracy. Through multi-granularity distillation (i.e., a first image enhancement benchmark model with a wider width and two granularities of a second image enhancement benchmark model with a deeper depth), the fitting ability of the image enhancement model can be greatly improved, so that the image enhancement model can be distilled and learned more quickly, and a target image enhancement model with higher performance and accuracy is obtained. The quality of the image processed and output by the trained target image enhancement model will also be higher.

[0186] For further information, see Figure 8 , Figure 8 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may be a computer program (including program code) running in a computer device, for example, the image processing device is an application software; the image processing device may be used to execute Figure 3 As shown in the method. Figure 8 As shown, the image processing device 1 may include: a sample acquisition module 11, a sample input module 12, an image output module 13, a model training module 14 and a model application module 15.

[0187] The sample acquisition module 11 is used to acquire a sample image;

[0188] A sample input module 12 is used to input the sample image into the image enhancement model, the first image enhancement benchmark model and the second image enhancement benchmark model respectively; the width of the network structure of the first image enhancement benchmark model is greater than the width of the network structure of the image enhancement model; the depth of the network structure of the second image enhancement benchmark model is greater than the depth of the network structure of the image enhancement model;

[0189] An image output module 13 is used to output an enhanced image corresponding to the sample image through an image enhancement model, output a first pseudo-label image corresponding to the sample image through a first image enhancement benchmark model, and output a second pseudo-label image corresponding to the sample image through a second image enhancement benchmark model;

[0190] A model training module 14 is used to perform distillation training on the image enhancement model through the enhanced image, the first pseudo-label image and the second pseudo-label image to obtain a target image enhancement model;

[0191] The model application module 15 is used to perform image enhancement processing on the target image based on the target image enhancement model.

[0192] The specific implementation of the sample acquisition module 11, the sample input module 12, the image output module 13, the model training module 14 and the model application module 15 can be found in the above Figure 3 The description of step S101 to step S103 in the corresponding embodiment will not be repeated here.

[0193] In one embodiment, the sample acquisition module 11 may include: an input image acquisition unit 111 , a region extraction unit 112 , an angle resetting unit 113 , and an image degradation unit 114 .

[0194] The input image acquisition unit 111 is used to acquire an original input image; the original input image includes a target object;

[0195] The region extraction unit 112 is used to extract the region where the key part of the target object is located in the original input image to obtain a part region image;

[0196] An angle resetting unit 113, used to perform angle resetting processing on key parts in the part region image to obtain an original label image;

[0197] The image degradation unit 114 is used to degrade the original label image to obtain a sample image; the image quality of the sample image is lower than the image quality of the original label image.

[0198] The specific implementation of the input image acquisition unit 111, the region extraction unit 112, the angle reset unit 113 and the image degradation unit 114 can be found in the above Figure 3 The description of step S101 in the corresponding embodiment will not be repeated here.

[0199] In one embodiment, the model training module 14 may include: a loss value determining unit 141 and a parameter adjusting unit 142 .

[0200] A loss value determining unit 141, configured to determine a width loss value for the image enhancement model according to the enhanced image and the first pseudo-label image;

[0201] The loss value determining unit 141 is further used to determine a depth loss value for the image enhancement model according to the enhanced image and the second pseudo-label image;

[0202] The loss value determining unit 141 is further used to determine a target loss value for the image enhancement model according to the width loss value and the depth loss value;

[0203] The parameter adjustment unit 142 is used to adjust the model parameters of the image enhancement model according to the target loss value to obtain the target image enhancement model.

[0204] The specific implementation of the loss value determination unit 141 and the parameter adjustment unit 142 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0205] In one embodiment, the loss value determining unit 141 may include: a loss value determining subunit 1411 and a width loss value determining subunit 1412 .

[0206] A loss value determination subunit 1411, configured to determine an image perception loss value for the image enhancement model based on the enhanced image and the first pseudo-label image;

[0207] The loss value determination subunit 1411 is further used to determine an image style loss value for the image enhancement model according to the enhanced image and the first pseudo-label image;

[0208] The loss value determination subunit 1411 is also used to obtain the image smoothing loss value corresponding to the sample image;

[0209] The loss value determination subunit 1411 is also used to obtain the feature difference loss value corresponding to the sample image;

[0210] The width loss value determining subunit 1412 is used to determine the width loss value according to the image perception loss value, the image style loss value, the image smoothness loss value and the feature difference loss value.

[0211] The specific implementation of the loss value determination subunit 1411 and the width loss value determination subunit 1412 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0212] In one embodiment, the loss value determination subunit 1411 is further specifically used to perform convolution processing on the enhanced image to obtain a first image convolution feature corresponding to the enhanced image;

[0213] The loss value determination subunit 1411 is further specifically used to perform convolution processing on the first pseudo-label image to obtain a second image convolution feature corresponding to the first pseudo-label image;

[0214] The loss value determination subunit 1411 is further specifically used to obtain an image perception loss function, and determine the image perception loss value according to the first image convolution feature, the second image convolution feature and the image perception loss function.

[0215] In one embodiment, the loss value determination subunit 1411 is further specifically used to extract enhanced image style features of the enhanced image and label image style features of the first pseudo label image;

[0216] The loss value determination subunit 1411 is also specifically used to obtain an image style loss function, and determine an image style loss value according to the enhanced image style features, the label image style features and the image style loss function.

[0217] In one embodiment, the loss value determination subunit 1411 is further specifically used to obtain a target pixel pair in the enhanced image; the target pixel pair is composed of a first pixel and a second pixel, and the first pixel and the second pixel are adjacent pixels in the enhanced image;

[0218] The loss value determination subunit 1411 is further specifically used to determine a pixel difference between a first pixel value corresponding to the first pixel and a second pixel value corresponding to the second pixel;

[0219] The loss value determination subunit is also specifically used to obtain an image smoothing loss function, and determine the image smoothing loss value based on the pixel difference and the image smoothing loss function.

[0220] In one embodiment, the loss value determination subunit 1411 is further specifically used to extract the image convolution feature corresponding to the sample image through the feature extraction layer of the image enhancement model to obtain the first sample convolution feature;

[0221] The loss value determination subunit 1411 is further specifically used to extract the image convolution feature corresponding to the sample image through the feature extraction layer of the first image enhancement benchmark model to obtain the second sample convolution feature;

[0222] The loss value determination subunit 1411 is also specifically used to obtain a feature difference loss function, and determine a feature difference loss value for the image enhancement model according to the first sample convolution feature, the second sample convolution feature and the feature difference loss function.

[0223] In one embodiment, the model training module 14 may include: an intermediate model determination unit 143 , an iteration number acquisition unit 144 , a model determination unit 145 , and a model adjustment unit 146 .

[0224] An intermediate model determination unit 143 is used to perform distillation training on the image enhancement model through the enhanced image, the first pseudo-label image, and the second pseudo-label image to obtain an intermediate image enhancement model;

[0225] An iteration number obtaining unit 144 is used to obtain the model iteration number corresponding to the intermediate image enhancement model;

[0226] A model determination unit 145, configured to determine the intermediate image enhancement model as a target image enhancement model if the number of model iterations satisfies a model convergence condition;

[0227] A model adjustment unit 146, configured to obtain a true label image corresponding to the sample image if the number of model iterations does not satisfy a model convergence condition;

[0228] The model adjustment unit 146 is further configured to adjust the first image enhancement benchmark model according to the real label image and the first pseudo label image to obtain a first adjusted enhancement benchmark model;

[0229] The model adjustment unit 146 is further configured to adjust the second image enhancement benchmark model according to the real label image and the second pseudo label image to obtain a second adjusted enhancement benchmark model;

[0230] The model adjustment unit 146 is also used to perform distillation training on the intermediate image enhancement model according to the sample image, the first updated pseudo-label image re-output by the first adjusted enhanced baseline model, and the second updated pseudo-label image re-output by the second adjusted enhanced baseline model to obtain a target image enhancement model.

[0231] The specific implementation of the intermediate model determination unit 143, the iteration number acquisition unit 144, the model determination unit 145 and the model adjustment unit 146 can be found in the above Figure 4 The description of steps S201 to S206 in the corresponding embodiment will not be repeated here.

[0232] In one embodiment, the model adjustment unit 146 may include: a label image acquisition subunit 1461 and a real label determination subunit 1462 .

[0233] The label image acquisition subunit 1461 is used to acquire the original label image corresponding to the sample image; the sample image is obtained by degrading the original label image;

[0234] The real label determination subunit 1462 is used to determine the original label image as the real label image.

[0235] The specific implementation of the label image acquisition subunit 1461 and the real label determination subunit 1462 can be found in the above Figure 4 The description of step S204 in the corresponding embodiment will not be repeated here.

[0236] In one embodiment, the model adjustment unit 146 may include: an original image acquisition subunit 1463 , an initial model training subunit 1464 , and a true label determination subunit 1465 .

[0237] The original image acquisition subunit 1463 is used to acquire the original label image corresponding to the sample image; the sample image is obtained by degrading the original label image;

[0238] The initial model training subunit 1464 is used to input the sample image and the original label image into the label generation initial model;

[0239] The initial model training subunit 1464 is further used to train the label generation initial model through the sample image and the original label image to obtain the label generation model;

[0240] The real label determination subunit 1465 is used to input the sample image into the label generation model, output the label output result image corresponding to the sample image through the label generation model, and determine the label output result image as the real label image.

[0241] The specific implementation of the original image acquisition subunit 1463, the initial model training subunit 1464 and the true label determination subunit 1465 can be found in the above Figure 4 The description of step S204 in the corresponding embodiment will not be repeated here.

[0242] In one embodiment, the initial model training subunit 1464 is further specifically used to obtain a model adversarial loss function, and determine a model adversarial loss value for generating an initial model for the label according to the model adversarial loss function, the sample image, and the original label image;

[0243] The initial model training subunit 1464 is further specifically used to obtain a perceptual loss function, and determine a perceptual loss value for generating an initial model for the label according to the perceptual loss function, the sample image, and the original label image;

[0244] The initial model training subunit 1464 is further specifically used to obtain a reconstruction loss function, and determine a reconstruction loss value of the initial model for label generation according to the reconstruction loss function, the sample image and the original label image;

[0245] The initial model training subunit 1464 is also specifically used to generate a target model loss value based on the model adversarial loss value, the perception loss value and the reconstruction loss value, and to train the initial label generation model based on the target model loss value to obtain a label generation model.

[0246] In an embodiment of the present application, after obtaining a sample image, the sample image can be respectively input into an image enhancement model, a first image enhancement benchmark model, and a second image enhancement benchmark model. Through the first pseudo-label image output by the first image enhancement benchmark model, the second pseudo-label image output by the second image enhancement benchmark model, and the enhanced image output by the image enhancement model, the image enhancement model can be distilled and trained to obtain an image enhancement model. Among them, the network structure of the first image enhancement benchmark model is wider than the image enhancement model, and the network structure of the second image enhancement benchmark model is deeper than the image enhancement model. Through distillation training, the knowledge of the first image enhancement benchmark model and the second image enhancement benchmark model can be continuously transferred to the image enhancement model. In other words, the present application can use a first image enhancement benchmark model with a wider width and a second image enhancement benchmark model with a deeper depth to simultaneously perform supervised training to achieve the purpose of distilling the image enhancement model, and finally obtain a target image enhancement model with higher performance and accuracy. Through multi-granularity distillation (i.e., a first image enhancement benchmark model with a wider width and a second image enhancement benchmark model with a deeper depth), the fitting ability of the image enhancement model can be greatly improved, so that the image enhancement model can be distilled and learned more quickly to obtain a target image enhancement model with higher performance and accuracy. The quality of the output image processed by the trained target image enhancement model will also be higher. In summary, the present application can improve the training efficiency of the image enhancement model and improve the image quality of the output image.

[0247] For further information, see Fig. 9 , Fig. 9 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig. 9 As shown above Figure 7The device 1 in the corresponding embodiment can be applied to the above-mentioned computer device 8000, and the above-mentioned computer device 8000 may include: a processor 8001, a network interface 8004 and a memory 8005. In addition, the above-mentioned computer device 8000 also includes: a user interface 8003, and at least one communication bus 8002. Among them, the communication bus 8002 is used to realize the connection and communication between these components. Among them, the user interface 8003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 8003 may also include a standard wired interface and a wireless interface. The network interface 8004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 8005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 8005 may optionally also be at least one storage device located away from the aforementioned processor 8001. Fig. 9 As shown, the memory 8005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0248] exist Fig. 9 In the computer device 8000 shown, the network interface 8004 can provide a network communication function; the user interface 8003 is mainly used to provide an input interface for the user; and the processor 8001 can be used to call the device control application stored in the memory 8005 to achieve:

[0249] Obtain a sample image, and input the sample image into an image enhancement model, a first image enhancement benchmark model, and a second image enhancement benchmark model respectively; the width of the network structure of the first image enhancement benchmark model is greater than the width of the network structure of the image enhancement model; the depth of the network structure of the second image enhancement benchmark model is greater than the depth of the network structure of the image enhancement model;

[0250] Output an enhanced image corresponding to the sample image through the image enhancement model, output a first pseudo-label image corresponding to the sample image through the first image enhancement benchmark model, and output a second pseudo-label image corresponding to the sample image through the second image enhancement benchmark model;

[0251] The image enhancement model is distilled and trained by using the enhanced image, the first pseudo-label image and the second pseudo-label image to obtain a target image enhancement model, and image enhancement processing is performed on the target image based on the target image enhancement model.

[0252] It should be understood that the computer device 8000 described in the embodiment of the present application can execute the above Figures 3 to 7 The description of the image processing method in the corresponding embodiment can also be performed as described above. Figure 8 The description of 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0253] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the computer device 1000 for data processing mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, the computer program can execute the above-mentioned data processing. Figures 3 to 7 The description of the above-mentioned image processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0254] The computer-readable storage medium may be an internal storage unit provided by any of the aforementioned embodiments or the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0255] In one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in one aspect of the embodiments of the present application.

[0256] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0257] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0258] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0259] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a sample image, and input the sample image into an image enhancement model, a first image enhancement benchmark model, and a second image enhancement benchmark model respectively; The width of the network structure of the first image enhancement benchmark model is greater than the width of the network structure of the image enhancement model; the depth of the network structure of the second image enhancement benchmark model is greater than the depth of the network structure of the image enhancement model; Outputting an enhanced image corresponding to the sample image through the image enhancement model, outputting a first pseudo-label image corresponding to the sample image through the first image enhancement benchmark model, and outputting a second pseudo-label image corresponding to the sample image through the second image enhancement benchmark model; The image enhancement model is distilled and trained by the enhanced image, the first pseudo-label image and the second pseudo-label image to obtain a target image enhancement model; the distillation training of the image enhancement model by the enhanced image, the first pseudo-label image and the second pseudo-label image to obtain the target image enhancement model includes: determining an image perception loss value for the image enhancement model according to the enhanced image and the first pseudo-label image; determining an image style loss value for the image enhancement model according to the enhanced image and the first pseudo-label image; obtaining an image smoothing loss value corresponding to the sample image, and obtaining a feature difference loss value corresponding to the sample image; determining a width loss value for the image enhancement model according to the image perception loss value, the image style loss value, the image smoothing loss value and the feature difference loss value; determining a depth loss value for the image enhancement model according to the enhanced image and the second pseudo-label image; determining a target loss value for the image enhancement model according to the width loss value and the depth loss value, and adjusting model parameters of the image enhancement model according to the target loss value to obtain a target image enhancement model; Perform image enhancement processing on the target image based on the target image enhancement model.

2. The method according to claim 1, characterized in that The obtaining of the sample image comprises: Acquire an original input image; the original input image includes a target object; Extracting the region where the key parts of the target object are located in the original input image to obtain a part region image; Performing angle resetting processing on the key parts in the part region image to obtain an original label image; The original label image is degraded to obtain the sample image; the image quality of the sample image is lower than the image quality of the original label image.

3. The method according to claim 1, characterized in that The determining, according to the enhanced image and the first pseudo-label image, an image perception loss value for the image enhancement model comprises: Performing convolution processing on the enhanced image to obtain a first image convolution feature corresponding to the enhanced image; Performing convolution processing on the first pseudo-label image to obtain a second image convolution feature corresponding to the first pseudo-label image; An image perceptual loss function is obtained, and the image perceptual loss value is determined according to the first image convolution feature, the second image convolution feature, and the image perceptual loss function.

4. The method according to claim 1, characterized in that: The step of determining an image style loss value for the image enhancement model according to the enhanced image and the first pseudo-label image includes: Extracting enhanced image style features of the enhanced image and label image style features of the first pseudo-label image; An image style loss function is obtained, and the image style loss value is determined according to the enhanced image style feature, the label image style feature, and the image style loss function.

5. The method according to claim 1, characterized in that The obtaining the image smoothing loss value corresponding to the sample image includes: Acquire a target pixel pair in the enhanced image; the target pixel pair consists of a first pixel and a second pixel, and the first pixel and the second pixel are adjacent pixels in the enhanced image; Determine a pixel difference between a first pixel value corresponding to the first pixel and a second pixel value corresponding to the second pixel; An image smoothing loss function is obtained, and the image smoothing loss value is determined according to the pixel difference and the image smoothing loss function.

6. The method according to claim 1, characterized in that The obtaining the feature difference loss value corresponding to the sample image includes: Extracting the image convolution feature corresponding to the sample image through the feature extraction layer of the image enhancement model to obtain a first sample convolution feature; Extracting the image convolution feature corresponding to the sample image through the feature extraction layer of the first image enhancement benchmark model to obtain a second sample convolution feature; A feature difference loss function is obtained, and a feature difference loss value for the image enhancement model is determined according to the first sample convolution feature, the second sample convolution feature, and the feature difference loss function.

7. The method according to claim 1, characterized in that The step of performing distillation training on the image enhancement model by using the width loss value, the enhanced image, and the second pseudo-label image to obtain a target image enhancement model includes: Performing distillation training on the image enhancement model through the width loss value, the enhanced image, and the second pseudo-label image to obtain an intermediate image enhancement model; Obtaining the number of model iterations corresponding to the intermediate image enhancement model; If the number of model iterations satisfies the model convergence condition, determining the intermediate image enhancement model as the target image enhancement model; If the number of model iterations does not meet the model convergence condition, obtaining a true label image corresponding to the sample image, and adjusting the first image enhancement benchmark model according to the true label image and the first pseudo label image to obtain a first adjusted enhancement benchmark model; Adjusting the second image enhancement benchmark model according to the real label image and the second pseudo label image to obtain a second adjusted enhancement benchmark model; The intermediate image enhancement model is distilled and trained based on the sample image, the first updated pseudo-label image re-output by the first adjusted enhanced baseline model, and the second updated pseudo-label image re-output by the second adjusted enhanced baseline model to obtain the target image enhancement model.

8. The method according to claim 7, characterized in that The obtaining of a real label image corresponding to the sample image comprises: Acquire an original label image corresponding to the sample image; the sample image is obtained by degrading the original label image; The original label image is determined as the true label image.

9. The method according to claim 7, characterized in that: The obtaining of a real label image corresponding to the sample image comprises: Acquire an original label image corresponding to the sample image; the sample image is obtained by degrading the original label image; Inputting the sample image and the original label image into an initial label generation model, training the initial label generation model through the sample image and the original label image to obtain a label generation model; The sample image is input into the label generation model, and the label generation model outputs a label output result image corresponding to the sample image, and the label output result image is determined as the true label image.

10. The method according to claim 9, characterized in that The step of training the initial label generation model by using the sample image and the original label image to obtain the label generation model includes: Obtaining a model adversarial loss function, and determining a model adversarial loss value of an initial model generated for the label according to the model adversarial loss function, the sample image, and the original label image; Obtaining a perceptual loss function, and determining a perceptual loss value of an initial model for generating the label according to the perceptual loss function, the sample image, and the original label image; Obtaining a reconstruction loss function, and determining a reconstruction loss value of an initial model for generating the label according to the reconstruction loss function, the sample image and the original label image; A target model loss value is generated according to the model adversarial loss value, the perception loss value and the reconstruction loss value, and the label generation initial model is trained according to the target model loss value to obtain the label generation model.

11. An image processing device, characterized in that: include: A sample acquisition module, used for acquiring a sample image; A sample input module, used to input the sample image into an image enhancement model, a first image enhancement benchmark model and a second image enhancement benchmark model respectively; the width of the network structure of the first image enhancement benchmark model is greater than the width of the network structure of the image enhancement model; the depth of the network structure of the second image enhancement benchmark model is greater than the depth of the network structure of the image enhancement model; An image output module, configured to output an enhanced image corresponding to the sample image through the image enhancement model, output a first pseudo-label image corresponding to the sample image through the first image enhancement benchmark model, and output a second pseudo-label image corresponding to the sample image through the second image enhancement benchmark model; A model training module, configured to perform distillation training on the image enhancement model through the enhanced image, the first pseudo-label image, and the second pseudo-label image to obtain a target image enhancement model; the distillation training on the image enhancement model through the enhanced image, the first pseudo-label image, and the second pseudo-label image to obtain the target image enhancement model includes: determining an image perception loss value for the image enhancement model according to the enhanced image and the first pseudo-label image; determining an image style loss value for the image enhancement model according to the enhanced image and the first pseudo-label image; obtaining an image smoothing loss value corresponding to the sample image, and obtaining a feature difference loss value corresponding to the sample image; determining a width loss value for the image enhancement model according to the image perception loss value, the image style loss value, the image smoothing loss value, and the feature difference loss value; determining a depth loss value for the image enhancement model according to the enhanced image and the second pseudo-label image; determining a target loss value for the image enhancement model according to the width loss value and the depth loss value, and adjusting model parameters of the image enhancement model according to the target loss value to obtain a target image enhancement model; The model application module is used to perform image enhancement processing on the target image based on the target image enhancement model.

12. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code so that the computer device executes the method described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 10.

14. A computer program product or a computer program, characterized in that The computer program product or computer program comprises computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 10.