Model training and image processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides a model training and image processing method and device, a storage medium and a program product. In the embodiment of the invention, the size of the image processing model is compressed through a knowledge distillation mode, and the lightweight image processing model is obtained, so that the image processing model can reduce the demand of computing resources in actual use, maintain good model performance and obtain high-quality images; and particularly, the method has a good landing effect in a resource-limited environment. Particularly, in the field of facial beautification processing, a lightweight image processing model not only can efficiently operate in a resource-limited environment, but also can keep a good facial beautification effect, reduce loss of pores and skin texture details, and is relatively good in adaptability.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training and image processing method, device, storage medium and program product. Background Art

[0002] In live broadcast scenarios, adding a beauty function is very common. The host can use the beauty function to beautify the facial image, which not only improves the host's appearance, but also increases the audience's viewing interest, which is conducive to improving the audience's experience of watching live broadcasts.

[0003] In the prior art, beautification algorithms used on mobile devices are mostly based on traditional CV (Computer Vision) operators for face detection, feature point location and region segmentation, and then manually designed filter templates are used to extract features and beautify the segmented face region. This method has poor beautification effects, easily leads to the loss of pores and skin texture details, and has poor adaptability. Summary of the invention

[0004] Multiple aspects of the present application provide a model training and image processing method, device, storage medium and program product to provide a lightweight image processing model, so that the image processing model can reduce the demand for computing resources in actual use, maintain good model performance, and obtain high-quality image quality.

[0005] An embodiment of the present application provides a model training method, including: obtaining multiple training samples, the training samples including sample images and annotated images corresponding to the sample images; inputting the sample images into a student model to be trained for target processing operations to obtain a first target image, and the difference between the annotated image and the sample image represents the target processing operation; inputting the annotated image into a pre-trained teacher model for target processing operations to obtain a second target image; based on the annotated image and the second target image, generating a third target image required for supervising and guiding the student model; and adjusting the model parameters of the student model based on the loss function between the third target image and the first target image to obtain an image processing model capable of performing target processing operations on the input image.

[0006] An embodiment of the present application provides an image processing method, including: receiving an image to be processed sent by a terminal device; inputting the image to be processed into an image processing model for target processing operation to obtain a target image; sending the target image to the terminal device so that the terminal device displays the target image; wherein the image processing model is trained based on the above-mentioned model training method.

[0007] An embodiment of the present application provides an image processing method, including: responding to an image processing operation, obtaining an image to be processed, where the image to be processed is an image in a local gallery or a video frame in a live video stream; inputting the image to be processed into an image processing model for a target processing operation to obtain a target image; displaying the target image or sending the target image to a live audience; wherein the image processing model is trained based on the above-mentioned model training method.

[0008] An embodiment of the present application provides an electronic device, comprising: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program to execute the steps in the model training method or the image processing method.

[0009] An embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the model training method or the image processing method.

[0010] An embodiment of the present application provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the model training method or the image processing method.

[0011] The technical solution of the embodiment of the present application compresses the size of the image processing model by knowledge distillation to obtain a lightweight image processing model, so that the image processing model can reduce the demand for computing resources in actual use, maintain good model performance, and obtain high-quality images, especially in resource-constrained environments. In particular, in the field of beauty processing, the lightweight image processing model can not only run efficiently in a resource-constrained environment, but also maintain a good beauty effect, reduce the loss of pores and skin texture details, and has good adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0014] Figure 2 Schematic diagram of the training process of an exemplary teacher model;

[0015] Figure 3 is a schematic diagram of the training process of an exemplary image processing model;

[0016] Figure 4is a schematic diagram of the training process of an exemplary image processing model;

[0017] Figure 5 A flowchart of another model training method provided in an embodiment of the present application;

[0018] Figure 6 is a schematic diagram of the training process of an exemplary image processing model;

[0019] Figure 7 A flowchart of an image processing method provided in an embodiment of the present application;

[0020] Figure 8 A flowchart of another image processing method provided in an embodiment of the present application;

[0021] Fig. 9 is an exemplary application scenario diagram;

[0022] Fig.10 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0023] Fig.11 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0024] Fig.12 A schematic diagram of the structure of another image processing device provided in an embodiment of the present application;

[0025] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0027] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the access relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. In the text description of the present application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship. In addition, in the embodiments of the present application, "first", "second", "third", etc. are only used to distinguish the contents of different objects and have no other special meanings.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] In live broadcast scenarios, adding a beauty function is very common. The host can use the beauty function to beautify the facial image, which not only improves the host's appearance, but also increases the audience's viewing interest, which is conducive to improving the audience's experience of watching live broadcasts.

[0030] In the prior art, beautification algorithms used on mobile devices are mostly based on traditional CV (Computer Vision) operators for face detection, feature point location and region segmentation, and then manually designed filter templates are used to extract features and beautify the segmented face region. This method has poor beautification effects, easily leads to the loss of pores and skin texture details, and has poor adaptability.

[0031] Based on the above, the embodiments of the present application provide a model training and image processing method, device, storage medium and program product. In the embodiments of the present application, the size of the image processing model is compressed by knowledge distillation to obtain a lightweight image processing model, so that the image processing model can reduce the demand for computing resources in actual use, maintain good model performance, and obtain high-quality images, especially in resource-constrained environments. It has a good landing effect. In particular, in the field of beauty processing, the lightweight image processing model can not only run efficiently in a resource-constrained environment, but also maintain a good beauty effect, reduce the loss of pores and skin texture details, and has good adaptability.

[0032] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The following is a detailed description of the technical solutions provided by each embodiment of the present application in conjunction with the accompanying drawings.

[0033] Figure 1 A flowchart of a model training method provided in an embodiment of the present application. Figure 1 , the method may include the following steps:

[0034] 101. Obtain multiple training samples, where the training samples include sample images and annotated images corresponding to the sample images.

[0035] 102. Input the sample image into the student model to be trained to perform a target processing operation to obtain a first target image, and the difference between the labeled image and the sample image represents the target processing operation.

[0036] 103. Input the labeled image into the pre-trained teacher model for target processing operation to obtain a second target image.

[0037] 104. Generate a third target image required for supervising and guiding the student model based on the annotated image and the second target image.

[0038] 105. According to the loss function between the third target image and the first target image, adjust the model parameters of the student model to obtain an image processing model that can perform the target processing operation on the input image.

[0039] In this embodiment, there is no restriction on the image processing model and the target processing operation performed by it. For example, the image processing model is a skin refining model, and the target processing operation is a skin refining operation. For another example, the image processing model is a facial beautification model, and the target processing operation is various facial beautification operations such as skin refining, whitening, face shape modification, and eye shape modification. For another example, the image processing model is an image enhancement model, and the target processing operation is sharpening processing, color correction processing, exposure adjustment, etc.

[0040] In the model training phase of the image processing model, multiple training samples need to be prepared, and the training samples include sample images and annotated images corresponding to the sample images. Different image processing models require different training samples.

[0041] For example, the sample image of the skin resurfacing model is the image to be resurfaced, and the target object in the image to be resurfaced has various skin defects, such as but not limited to: acne marks, scars, large pores, etc. The target object includes, but is not limited to: face, neck, arms, etc.

[0042] Optionally, at least some of the sample images in the multiple training samples include target objects with different skin qualities. Thus, the diverse training samples can allow the dermabrasion processing model to better adapt to different skin conditions, thereby improving the generalization performance of the dermabrasion processing model.

[0043] The annotated image corresponding to the sample image of the skin resurfacing model may be a target object in the image whose skin is free of various defects, and the annotated image may be obtained through processing by professionals.

[0044] For another example, the sample image of the beautification model is the image to be beautified, and the target object in the image to be beautified has various defects, including but not limited to: acne marks, scars, large pores, spots, uneven skin color, different-sized eyes, thin face, fat face, etc. Among them, the target object includes but is not limited to: face, neck, arms, etc. Optionally, the defects of the target object included in at least some of the sample images in the multiple training samples are different. In this way, the diversified training samples can make the beautification model better adapt to different conditions and improve the generalization performance of the beautification model.

[0045] The annotated image corresponding to the sample image of the beautification model may be an image in which the target object does not have various defects. The annotated image is the annotated beautification image, and the annotated image may be obtained through processing by professionals.

[0046] For example, the sample images of the image enhancement model can be various blurred images, various dimly colored images, or various underexposed images. The annotated images of the sample images are images with clear edges, bright colors, or sufficient exposure, and the annotated images can be processed by professionals.

[0047] Optionally, at least some of the sample images in the multiple training samples have different blurriness, darkness or exposure levels. Diversified training samples can allow the image enhancement model to better adapt to different conditions, thereby improving the generalization performance of the image enhancement model.

[0048] In this embodiment, a teacher model is trained in advance using multiple training samples. Figure 2 In the training phase of the teacher model, the sample image is input into the teacher model, and the loss function of the teacher model is determined based on the processed sample image and the labeled image output by the teacher model. The model parameters of the teacher model are adjusted with the goal of minimizing the loss function. The model training operation is iterated until the model training iteration end condition is met. The model training iteration end condition may be, for example, that the number of model training times reaches a specified number or the value of the loss function reaches a specified value, which is not limited.

[0049] In this embodiment, a pre-trained teacher model (ie, a pre-trained teacher model) is obtained, and a student model is trained based on the pre-trained teacher model using a knowledge distillation method, where the trained student model is an image processing model that can perform a target processing operation on an input image.

[0050] For ease of understanding, take the training of the skin resurfacing model as an example, see Figure 3 , input the sample images corresponding to the target object's skin with various blemishes such as acne marks into the student model to be trained, and obtain the first target image output by the student model; input the annotated image corresponding to the sample image into the pre-trained teacher model, and obtain the second target image output by the teacher model; based on the annotated image and the second target image, generate the third target image required for supervising and guiding the student model. For example, the annotated image and the second target image are subjected to image fusion processing to obtain the third target image. Image fusion processing includes, but is not limited to: pixel-based image fusion method, wavelet transform-based image fusion method, or sparse representation-based image fusion method. Pixel-based image fusion method refers to weighted summing of pixels of images involved in fusion to obtain a fused image. Wavelet transform-based image fusion method refers to fusing wavelet transformed images. Sparse representation-based image fusion method refers to weighted summing of sparse representations of images involved in fusion to obtain a fused image.

[0051] Further optional, in order to ensure the edge of the image is clear and improve the image quality, see Figure 3 , according to the annotated image and the second target image, the third target image required for supervising and guiding the student model is generated by: fusing the annotated image and the second target image to obtain a fused image; sharpening the fused image to obtain a third target image. It can be understood that image sharpening is a technology that enhances image details and edges, which can improve the clarity of the image. Image sharpening is usually achieved by enhancing the high-frequency components in the image, which can make the edges in the image more obvious and the details more prominent. Optionally, image sharpening can use a second-order Laplace operator to extract image edge information.

[0052] See also Figure 3 , when training the student model, the model parameters of the student model are adjusted according to the loss function between the third target image and the first target image to obtain an image processing model that can perform the target processing operation on the input image.

[0053] For example, a generative adversarial loss function is calculated based on the third target image and the first target image, and the model parameters of the student model are adjusted with the goal of minimizing the generative adversarial loss function. The generative adversarial loss function includes, but is not limited to, a cross entropy loss function.

[0054] For another example, a pixel-level loss function is calculated based on the third target image and the first target image, and the model parameters of the student model are adjusted with the goal of minimizing the pixel-level loss function. The pixel-level loss function measures the error between the pixel value of a pixel point in the third target image and the pixel value of the corresponding pixel point in the first target image. Pixel-level loss functions include, but are not limited to, mean absolute error function, root mean square error function, or cross entropy loss function.

[0055] Further optionally, in order to improve the model performance of the image processing model, the model parameters of the student model are adjusted according to the loss function between the third target image and the first target image, so as to obtain the image processing model capable of performing the target processing operation on the input image. The implementation method is as follows: at least one loss function among the adversarial loss function and the pixel-level loss function is calculated and generated according to the third target image and the first target image; if there is a loss function in at least one loss function that does not meet the corresponding termination condition, the model parameters of the student model are adjusted until at least one loss function meets the corresponding termination condition, so as to obtain the image processing model. Among them, the termination condition corresponding to the loss function includes, but is not limited to: the value of the loss function reaches a specified minimum value, the number of model training times reaches a specified number of times, etc.

[0056] The technical solution provided in the embodiment of the present application compresses the size of the image processing model by knowledge distillation to obtain a lightweight image processing model, so that the image processing model can reduce the demand for computing resources in actual use, maintain good model performance, and obtain high-quality images, especially in resource-constrained environments. In particular, in the field of beauty processing, the lightweight image processing model can not only run efficiently in a resource-constrained environment, but also maintain a good beauty effect, reduce the loss of pores and skin texture details, and has good adaptability.

[0057] In some optional embodiments, by integrating the knowledge of multiple teacher models, the student model can learn stronger generalization ability, so as to perform better on unseen data. Based on this, the number of teacher models is multiple; the implementation method of inputting the labeled image into the pre-trained teacher model for target processing operation to obtain the second target image is: inputting the labeled image into multiple teacher models for target processing operation respectively to obtain multiple first intermediate state images; fusing the multiple first intermediate state images to obtain the second target image; wherein the network architectures of the multiple teacher models are the same or different.

[0058] See also Figure 4The multi-teacher model includes multiple teacher models such as teacher model 1, teacher model 2, ... teacher model n, where n is a natural number greater than 1. The labeled image is input into each teacher model for target processing operation to obtain a first intermediate state image. Multiple first intermediate state images are fused to obtain a second target image.

[0059] In some optional embodiments, the input data of the teacher model is perturbed based on at least one data augmentation process to help the student model better generalize to unseen data; and the output results of the perturbed input data are weighted fused, so that the student model can learn a more stable and robust output, and maintain consistent prediction results even in the face of slightly changing inputs, thereby improving the output stability of the student model.

[0060] Based on the above, the implementation method of inputting the labeled image into the pre-trained teacher model for target processing operation to obtain the second target image is as follows: performing at least one data enhancement process on the labeled image to obtain at least one second enhanced image; inputting at least one second enhanced image into the teacher model for target processing operation to obtain at least one third intermediate image; performing the inverse operation of the corresponding data enhancement process on at least one third intermediate image to obtain at least one fourth intermediate image; performing fusion processing on at least one fourth intermediate image to obtain the second target image.

[0061] In practical applications, data augmentation processing includes, but is not limited to: geometric deformation, color space perturbation, and random masking. Geometric deformation mainly refers to changing the shape and position of the image, including rotation, translation, scaling, shearing and other operations. These operations can simulate changes in perspective in practical applications and help student models learn to recognize objects from different angles and positions. Color space perturbation is mainly used to change the color properties of an image, including changes in brightness, contrast, saturation, hue, etc. These operations can simulate changes in lighting conditions and help student models learn to recognize objects under different lighting conditions. Random masking: Randomly masking parts of an image can be used to simulate occluded or damaged images.

[0062] In this embodiment, the inverse operation of the data enhancement process refers to the process of how to restore or recover the original image after the data enhancement is applied to the image. For example, the data enhancement process is rotation, and the inverse operation corresponding to the rotation is to rotate the same angle again but in the opposite direction. For another example, the data enhancement process is translation, and the inverse operation corresponding to the translation is to translate the same distance in the opposite direction.

[0063] See also Figure 4After the annotated image is subjected to at least one data enhancement process, a plurality of second enhanced images such as second enhanced image 1, second enhanced image 2, ..., second enhanced image m are obtained, where m is a natural number greater than 1. Each second enhanced image is input into the teacher model for target processing operation to obtain a third intermediate state image. Each third intermediate state image is subjected to the inverse operation of the corresponding data enhancement process to obtain a fourth intermediate state image; at least one fourth intermediate state image is subjected to fusion processing to obtain a second target image.

[0064] Figure 5 A flowchart of another model training method provided in an embodiment of the present application. Figure 5 , the method may include the following steps:

[0065] 501. Obtain multiple training samples, where the training samples include sample images and annotated images corresponding to the sample images.

[0066] The implementation of step 501 is the same as that of step 101 in the aforementioned embodiment, and will not be repeated here.

[0067] 502. Input the sample image into the student model to be trained to perform a target processing operation to obtain a first target image, and the difference between the labeled image and the sample image represents the target processing operation.

[0068] The implementation of step 502 is the same as the implementation of step 102 in the aforementioned embodiment, and will not be repeated here.

[0069] 503. Perform at least one data enhancement process on the labeled image to obtain at least one first enhanced image.

[0070] 504. For any teacher model among the multiple teacher models, input at least one first enhanced image into any teacher model for target processing operation to obtain at least one second intermediate state image corresponding to any teacher model.

[0071] 505. Perform the inverse operation of the corresponding data enhancement processing on at least one second intermediate state image corresponding to any teacher model to obtain at least one first intermediate state image.

[0072] 506. Fusing the multiple first intermediate state images corresponding to the multiple teacher models to obtain a second target image; wherein the network architectures of the multiple teacher models are the same or different.

[0073] 507. Generate a third target image required for supervising and guiding the student model based on the annotated image and the second target image.

[0074] The implementation of step 507 is the same as the implementation of step 104 in the aforementioned embodiment, and will not be repeated here.

[0075] 508. According to the loss function between the third target image and the first target image, adjust the model parameters of the student model to obtain an image processing model capable of performing a target processing operation on the input image.

[0076] The implementation of step 508 is the same as that of step 105 in the aforementioned embodiment, and will not be repeated here.

[0077] In this embodiment, a multi-teacher model and data enhancement processing are combined, see Figure 6 After the annotated image is subjected to at least one data enhancement process, a plurality of first enhanced images such as first enhanced image 1, first enhanced image 2...first enhanced image m are obtained, where m is a natural number greater than 1. The multi-teacher model includes a plurality of teacher models such as teacher model 1, teacher model 2...teacher model n, where n is a natural number greater than 1. The plurality of first enhanced images are input into each teacher model in the plurality of teacher models, and each teacher model outputs at least one second intermediate state image. Each second intermediate state image performs the inverse operation of the corresponding data enhancement process to obtain a first intermediate state image; the plurality of first intermediate state images are fused to obtain a second target image.

[0078] The technical solution provided by the embodiment of the present application compresses the size of the image processing model by knowledge distillation to obtain a lightweight image processing model, so that the image processing model can reduce the demand for computing resources in actual use, maintain good model performance, and obtain high-quality images, especially in a resource-constrained environment. It has a good landing effect. In particular, in the field of beauty processing, the lightweight image processing model can not only run efficiently in a resource-constrained environment, but also maintain a good beauty effect, reduce the loss of pores and skin texture details, and have good adaptability. Further, by integrating the knowledge of multiple teacher models, the student model can learn a more powerful generalization ability, so as to perform better on unseen data. Based on at least one data enhancement process, the input data of the teacher model is disturbed to help the student model better generalize to unseen data; and the output results of the disturbed input data are weighted fused, so that the student model can learn a more stable and robust output, and even in the face of slightly changing inputs, it can maintain consistent prediction results, thereby improving the output stability of the student model.

[0079] The trained image processing model can be deployed on the server or the terminal device without restriction. Figure 7 An image processing method is introduced.

[0080] Figure 7 This is a flowchart of an image processing method provided by an embodiment of the present application. Figure 7 , the method may include the following steps:

[0081] 701. Receive an image to be processed sent by a terminal device.

[0082] 702. Input the image to be processed into the image processing model to perform a target processing operation to obtain a target image;

[0083] 703. Send the target image to the terminal device so that the terminal device displays the target image.

[0084] The technical solution provided in this embodiment deploys an image processing model on the server side, and the terminal device requests the server side to use the image processing model to perform a target processing operation on the image to be processed to obtain a target image; the server side returns the target image to the terminal device for display. Since the size of the image processing model is compressed by knowledge distillation, a lightweight image processing model is obtained, so that the image processing model can reduce the demand for computing resources when actually used, maintain good model performance, and obtain high-quality images, especially in a resource-constrained environment, which has a good landing effect.

[0085] To facilitate understanding, several exemplary scenario embodiments are introduced below.

[0086] Scenario Example 1:

[0087] It is very common to add a beauty function to live broadcast scenes. Users can use the beauty function to beautify facial images, and e-commerce live broadcasts are no exception. During the live broadcast, the host's terminal device collects the host's live stream data, and the terminal device sends the host's live stream data to the server. The server uses a beauty model or a skin smoothing model to process the video frames in the live stream data to obtain a beauty image. The beauty image is returned to the host's terminal device, and can also be sent to the audience's terminal device. This can greatly improve the host's appearance and enhance the live broadcast experience.

[0088] Scenario Example 2:

[0089] In a video conferencing scenario, users can conduct real-time video communication through a video conferencing platform. Due to the different devices and network conditions used by users, the video quality may be affected. In order to improve the user experience, an image enhancement model is deployed on the server to process the video stream and thus improve the video quality. The user's terminal device sends the video stream to the server, and the server inputs each frame of the image into the deployed image enhancement model for processing. The enhanced image is encoded into a video stream and returned to the user's terminal device.

[0090] Figure 8 This is a flowchart of another image processing method provided by an embodiment of the present application. Figure 8 , the method may include the following steps:

[0091] 801. In response to an image processing operation, an image to be processed is obtained, where the image to be processed is an image in a local gallery or a video frame in a live video stream.

[0092] 802. Input the image to be processed into the image processing model to perform a target processing operation to obtain a target image.

[0093] 803. Display the target image or send the target image to live broadcast viewers.

[0094] In this embodiment, the image processing model can be deployed on the server side or on the terminal device side.

[0095] In actual applications, users can trigger image processing operations through the application program interface, and the terminal device responds to the image processing operation and obtains the image to be processed. The image to be processed can be an image from the local gallery or a video frame in the live video stream. The image to be processed is processed using the image processing model to obtain the target image. The target image can be sent to the host's terminal device for display, or to the terminal device of the live audience for display, without any restrictions.

[0096] The technical solution provided in this embodiment uses an image processing model to perform a target processing operation on the image to be processed to obtain a target image; and returns the target image to the terminal device for display. Since the size of the image processing model is compressed by knowledge distillation, a lightweight image processing model is obtained, so that the image processing model can reduce the demand for computing resources when actually used, maintain good model performance, and obtain high-quality images, especially in a resource-constrained environment, which has a good landing effect.

[0097] Combine the following Fig. 9 This article introduces a real-time beautification solution for mobile terminals. In live broadcast scenarios, beautification algorithms are very important for improving user experience. It not only improves the appearance of the anchor, but also increases the viewing interest of the audience. The requirements for real-time beautification in live broadcast scenarios are mainly reflected in the following aspects: 1. Real-time: Live broadcast scenarios require that the beautification algorithm can run in real time in the video stream, which means that the beautification algorithm must be efficient enough to quickly process each frame of video data under the limited computing resources of the mobile terminal. 2. Adaptability: The beautification algorithm needs to be able to automatically adapt to various skin types to ensure consistent beautification effects in any situation. 3. Detail retention: In addition to simple smoothing and skin grinding effects, advanced beautification algorithms also need to be able to retain the natural texture and details of the skin, such as pores, wrinkles, etc., to avoid making the picture look too fake.

[0098] In traditional solutions, most real-time beautification algorithms on mobile terminals are based on traditional CV (Computer Vision) operators, using manually designed filter templates to extract and beautify facial area features. This method is difficult to achieve satisfactory beautification effects on poor skin quality, has poor adaptability, and causes the loss of pores and skin texture details, and is also difficult to achieve real-time beautification. Therefore, it is necessary to study a new real-time beautification solution for mobile terminals for live broadcast scenarios.

[0099] The skin refining model trained using the above model training method can run well on mobile terminals and can adapt to the changes in facial skin quality of input pictures / video frames. Compared with traditional skin refining methods, it has better skin refining effects on any skin type and can cover most models on the market. It also takes less time on average on terminal devices of most models, and can live stream continuously for 30 minutes, achieving real-time beauty on mobile terminals.

[0100] The knowledge distillation method is used to train a lightweight skin smoothing model deployed on the mobile terminal, which effectively avoids the video jitter problem after skin smoothing caused by the unstable output of the small model. It can smooth the skin while retaining the skin texture details, and has the ability to adapt to different skin types, especially on poor skin types such as acne and freckles. The skin smoothing effect is obvious, achieving a satisfactory beauty effect for users. The second-order Laplace operator is used to extract image edge information, enhance the details of skin pores while protecting areas such as eyebrows from being over-sharpened, and can achieve better beauty effects in skin smoothing tasks.

[0101] See also Fig. 9 In the live broadcast scenario, the host sits in a room and broadcasts online. A skin smoothing model is deployed on the host's terminal device, which uses the skin smoothing model to beautify the host's facial features in real time during the live broadcast. During the live broadcast, the host can enable the beauty function to make his face appear smoother and brighter on the screen of the terminal device. The host can interact with the audience through the comment area on the screen, answer questions, accept gifts or participate in discussions. This interaction helps to increase the fun and appeal of the live broadcast. The beautified live broadcast picture is pushed to the terminal device on the audience side, presenting a high-quality live broadcast to the audience, while the host enjoys the confidence and comfort brought by real-time beauty.

[0102] Fig.10 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. Fig.10 As shown, the device may include:

[0103] An acquisition module 10 is used to acquire a plurality of training samples, wherein the training samples include a sample image and an annotated image corresponding to the sample image;

[0104] The training module 20 is used to input the sample image into the student model to be trained for target processing operation to obtain a first target image, and the difference between the annotated image and the sample image represents the target processing operation; input the annotated image into the pre-trained teacher model for target processing operation to obtain a second target image; based on the annotated image and the second target image, generate a third target image required for supervising and guiding the student model; according to the loss function between the third target image and the first target image, adjust the model parameters of the student model to obtain an image processing model capable of performing target processing operations on the input image.

[0105] Optionally, when the training module 20 generates a third target image required for supervising and guiding the student model based on the annotated image and the second target image, it is specifically used to: fuse the annotated image and the second target image to obtain a fused image; and sharpen the fused image to obtain the third target image.

[0106] Optionally, there are multiple teacher models; when the training module 20 inputs the labeled image into the pre-trained teacher model for target processing operation to obtain a second target image, it is specifically used to: input the labeled image into multiple teacher models for target processing operations respectively to obtain multiple first intermediate state images; fuse the multiple first intermediate state images to obtain the second target image; wherein the network architectures of the multiple teacher models are the same or different.

[0107] Optionally, when the training module 20 inputs the annotated image into multiple teacher models for target processing operations to obtain multiple first intermediate state images, it is specifically used to: perform at least one data enhancement process on the annotated image to obtain at least one first enhanced image; for any teacher model, input at least one first enhanced image into any teacher model for target processing operation to obtain at least one second intermediate state image corresponding to any teacher model; perform the inverse operation of the corresponding data enhancement process on at least one second intermediate state image corresponding to any teacher model to obtain at least one first intermediate state image.

[0108] Optionally, when the training module 20 inputs the annotated image into a pre-trained teacher model for target processing operation to obtain a second target image, it is specifically used to: perform at least one data enhancement process on the annotated image to obtain at least one second enhanced image; input at least one second enhanced image into the teacher model for target processing operation to obtain at least one third intermediate image; perform the inverse operation of the corresponding data enhancement process on at least one third intermediate image to obtain at least one fourth intermediate image; and perform fusion processing on at least one fourth intermediate image to obtain a second target image.

[0109] Optionally, when the training module 20 adjusts the model parameters of the student model according to the loss function between the third target image and the first target image to obtain an image processing model that can perform the target processing operation on the input image, it is specifically used to: calculate and generate at least one loss function among the adversarial loss function and the pixel-level loss function according to the third target image and the first target image; if there is a loss function in at least one loss function that does not meet the corresponding termination condition, adjust the model parameters of the student model until at least one loss function meets the corresponding termination condition to obtain the image processing model.

[0110] Optionally, the annotated image is an annotated beautified image, the target processing operation is a beautification operation, and the image processing model is a beautification model.

[0111] Optionally, the target objects included in at least some of the sample images in the multiple training samples have different skin qualities, and the beautification processing at least includes skin resurfacing processing.

[0112] Fig.10 The device shown can execute the model training method shown in the above embodiment, and its implementation principle and technical effect will not be described in detail. Fig.10 The specific manner in which each module and unit of the device shown performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0113] Fig.11 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig.11 As shown, the device may include:

[0114] The receiving module 10 is used to receive the image to be processed sent by the terminal device;

[0115] The processing module 20 is used to input the image to be processed into the image processing model to perform a target processing operation to obtain a target image;

[0116] The sending module 30 is used to send the target image to the terminal device so that the terminal device can display the target image.

[0117] Fig.11 The device shown in the figure can execute the image processing method shown in the above embodiment, and its implementation principle and technical effect will not be described in detail. Fig.11 The specific manner in which each module and unit of the device shown performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0118] Fig.12 This is a schematic diagram of the structure of another image processing device provided in an embodiment of the present application. Fig.12 As shown, the device may include:

[0119] The acquisition module 10 is used to respond to the image processing operation and acquire the image to be processed, where the image to be processed is an image in a local gallery or a video frame in a live video stream;

[0120] The processing module 20 is used to input the image to be processed into the image processing model to perform a target processing operation to obtain a target image;

[0121] The display module 20 is used to display the target image or send the target image to the live broadcast audience.

[0122] Optionally, the image processing model is deployed on the terminal device side.

[0123] Fig.12 The device shown in the figure can execute the image processing method shown in the above embodiment, and its implementation principle and technical effect will not be described in detail. Fig.12 The specific manner in which each module and unit of the device shown performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0124] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 105 can be device A; for another example, the execution subject of steps 101 and 104 can be device A, and the execution subject of step 105 can be device B; and so on.

[0125] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0126] Fig.13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.13 As shown, the electronic device includes: a memory 11 and a processor 12;

[0127] The memory 11 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0128] The memory 11 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0129] The processor 12 is coupled to the memory 11 and is used to execute the computer program in the memory 11 for: steps in the model training method or steps in the image processing method.

[0130] Optional, such as Fig.13 As shown, the electronic device also includes: a communication component 13, a display 14, a power component 15, an audio component 16 and other components. Fig.13 Only some components are shown schematically, which does not mean that the electronic device only includes Fig.13 In addition, Fig.13 The components in the dashed box are optional components, not mandatory components, and the specific components may depend on the product form of the electronic device. The electronic device of this embodiment may be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT (Internet of Things) device, or may be a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it may include Fig.13 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Fig.13 Components within the dashed box.

[0131] For the detailed implementation process of the processor executing each action, please refer to the relevant description in the aforementioned method embodiment or device embodiment, which will not be repeated here.

[0132] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0133] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement each step in the above method embodiment that can be executed by an electronic device.

[0134] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2Generation, 2 generations), 3G (3Generation, 3 generations), 4G (4Generation, 4 generations) / LTE (long Term Evolution), 5G (5Generation, 5 generations) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0135] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0136] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0137] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0138] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0139] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0142] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0143] Memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

[0144] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, Phase Change RAM (Phase Change RAM, PRAM), Static Random Access Memory (Static Random-Access Memory, SRAM), Dynamic Random Access Memory (Dynamic Random Access Memory, DRAM), other types of Random Access Memory (Random Access Memory, RAM), Read Only Memory (Read On ly Memory, ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, CD-ROM, Digital Versatile Disc (Digital Versatile Disc, DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0145] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0146] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A model training method, characterized in that: include: Acquire a plurality of training samples, wherein the training samples include sample images and annotated images corresponding to the sample images; Inputting the sample image into a student model to be trained to perform a target processing operation to obtain a first target image, wherein the difference between the labeled image and the sample image represents the target processing operation; Inputting the annotated image into a pre-trained teacher model for target processing operation to obtain a second target image; Generating a third target image required for supervising and guiding the student model according to the annotated image and the second target image; According to the loss function between the third target image and the first target image, the model parameters of the student model are adjusted to obtain an image processing model capable of performing the target processing operation on the input image.

2. The method according to claim 1, characterized in that: Generating a third target image required for supervising and guiding the student model according to the annotated image and the second target image, comprising: Fusing the annotated image and the second target image to obtain a fused image; The fused image is sharpened to obtain the third target image.

3. The method according to claim 1, characterized in that The number of the teacher models is multiple; the labeled image is input into the pre-trained teacher model for target processing operation to obtain a second target image, including: Inputting the labeled image into a plurality of teacher models to perform target processing operations respectively, so as to obtain a plurality of first intermediate state images; Performing fusion processing on the multiple first intermediate images to obtain the second target image; Among them, the network architectures of the multiple teacher models are the same or different.

4. The method according to claim 3, characterized in that Inputting the labeled image into a plurality of teacher models to perform target processing operations respectively to obtain a plurality of first intermediate state images, including: Performing at least one data enhancement process on the annotated image to obtain at least one first enhanced image; For any teacher model, inputting the at least one first enhanced image into the any teacher model for target processing operation to obtain at least one second intermediate state image corresponding to the any teacher model; The inverse operation of the corresponding data enhancement processing is performed on at least one second intermediate state image corresponding to any of the teacher models to obtain at least one first intermediate state image.

5. The method according to claim 1, characterized in that Inputting the labeled image into a pre-trained teacher model for target processing operation to obtain a second target image, including: Performing at least one data enhancement process on the annotated image to obtain at least one second enhanced image; Inputting the at least one second enhanced image into the teacher model for target processing operation to obtain at least one third intermediate state image; Performing the inverse operations of the data enhancement processing respectively corresponding to the at least one third intermediate state image to obtain at least one fourth intermediate state image; The at least one fourth intermediate state image is fused to obtain the second target image.

6. The method according to any one of claims 1 to 5, characterized in that According to the loss function between the third target image and the first target image, adjusting the model parameters of the student model to obtain an image processing model capable of performing the target processing operation on the input image, comprising: At least one loss function among a generative adversarial loss function and a pixel-level loss function is calculated based on the third target image and the first target image; If there is a loss function among the at least one loss function that does not meet the corresponding termination condition, adjust the model parameters of the student model until the at least one loss function meets the corresponding termination condition to obtain the image processing model.

7. The method according to any one of claims 1 to 5, characterized in that: The annotated image is an annotated beauty image, the target processing operation is a beauty operation, and the image processing model is a beauty model.

8. The method according to claim 7, characterized in that The target objects included in at least some of the sample images in the multiple training samples have different skin qualities, and the beautification operation at least includes skin resurfacing processing.

9. An image processing method, characterized in that: include: Receiving an image to be processed sent by a terminal device; Inputting the image to be processed into an image processing model to perform a target processing operation to obtain a target image; Sending the target image to the terminal device so that the terminal device displays the target image; Wherein, the image processing model is trained based on the method described in any one of claims 1-8.

10. An image processing method, characterized in that: include: In response to the image processing operation, an image to be processed is obtained, where the image to be processed is an image in a local gallery or a video frame in a live video stream; Inputting the image to be processed into an image processing model to perform a target processing operation to obtain a target image; Display the target image or send the target image to live viewers; wherein the image processing model is trained based on the method described in any one of claims 1-8.

11. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is coupled to the memory and configured to execute the computer program to perform the steps of the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps in the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, enables the processor to implement the steps of the method according to any one of claims 1 to 10.

Images