A method, apparatus, device and storage medium for processing media content

Abstract

Embodiments of the present disclosure relate to a method, apparatus, electronic device and storage medium for processing media content. The method comprises: obtaining first media content; and applying a target effect to the first media content using a target model to generate second media content, wherein the target model is trained based on the following process: determining a plurality of sub-effects corresponding to the target effect; determining a plurality of special effect models corresponding to the plurality of sub-effects from a set of pre-trained special effect models; processing a plurality of sample images according to a preset order using the plurality of special effect models to generate a plurality of corresponding output images; and training the target model using the plurality of sample images and the plurality of corresponding output images. According to embodiments of the present disclosure, the generation efficiency of the target model can be improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to a method, apparatus, device, and computer-readable storage medium for processing media content. Background Technology

[0002] With the development of computer technology, terminal devices such as mobile phones have the ability to process media content in real time.

[0003] However, the process of generating resources for real-time media content processing on terminal devices is quite complex, resulting in a shortage of resources available for this purpose. This will impact the user experience. Summary of the Invention

[0004] In a first aspect of this disclosure, a method for processing media content is provided. The method includes: acquiring first media content; and applying a target effect to the first media content using a target model to generate second media content, wherein the target model is trained based on the following process: determining multiple sub-effects corresponding to the target effect; determining multiple special effects models corresponding to the multiple sub-effects from a pre-trained set of special effects models; processing multiple sample images according to a preset order using the multiple special effects models to generate multiple corresponding output images; and training the target model using the multiple sample images and the multiple corresponding output images.

[0005] In a second aspect of this disclosure, an apparatus for processing media content is provided. The apparatus includes: an acquisition module configured to acquire first media content; and a generation module configured to apply a target effect to the first media content using a target model to generate second media content, wherein the target model is trained based on the following process: determining multiple sub-effects corresponding to the target effect; determining multiple special effect models corresponding to the multiple sub-effects from a pre-trained set of special effect models; processing multiple sample images according to a preset order using the multiple special effect models to generate multiple corresponding output images; and training the target model using the multiple sample images and the multiple corresponding output images.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0011] Figures 2A to 2E Example interfaces according to some embodiments of this disclosure are shown;

[0012] Figure 3 A flowchart illustrating an example process for processing media content according to some embodiments of this disclosure is shown;

[0013] Figure 4 A flowchart illustrating an example process for training a target model according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A flowchart illustrating an example process for training a target model according to some embodiments of this disclosure is shown;

[0015] Figure 6 A schematic structural block diagram of an example apparatus for processing media content according to some embodiments of the present disclosure is shown; and

[0016] Figure 7 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0021] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0022] As mentioned above, terminal devices typically employ machine learning models capable of processing media content. To meet diverse user needs, terminal devices deploy multiple machine learning models. However, the training process for each machine learning model is complex and requires significant human resources. This results in low efficiency in generating machine learning models, limiting the number of models available to users and impacting the user experience.

[0023] Embodiments of this disclosure propose a scheme for processing media content. The scheme includes: acquiring first media content; and applying a target effect to the first media content using a target model to generate second media content, wherein the target model is trained based on the following process: determining multiple sub-effects corresponding to the target effect; determining multiple special effects models corresponding to the multiple sub-effects from a pre-trained set of special effects models; processing multiple sample images according to a preset order using the multiple special effects models to generate multiple corresponding output images; and training the target model using the multiple sample images and the multiple corresponding output images.

[0024] In this way, embodiments of the present disclosure can utilize multiple pre-trained effect models to generate training samples for training the target model, thereby training the target model. This reduces the human resources required to train the target model and improves the generation efficiency of the target model to some extent.

[0025] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0026] Example Environment

[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include terminal device 110.

[0028] In this example environment 100, terminal device 110 may run an application 120 that supports processing media content. Application 120 may be any suitable type of application for processing media content, examples of which may include, but are not limited to, image processing applications, video processing applications, or other suitable applications. User 140 may interact with application 120 via terminal device 110 and / or its attached devices.

[0029] exist Figure 1 In environment 100, if application 120 is active, terminal device 110 can use application 120 to present interface 150 for supporting the processing of media content.

[0030] In some embodiments, terminal device 110 communicates with server 130 to provide services to application 120. Terminal device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 can also support any type of interface for user 140 (such as "wearable" circuitry).

[0031] Server 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 130 can provide backend services for applications 120 in terminal devices 110 that support media content processing.

[0032] A communication connection can be established between server 130 and terminal device 110. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections; the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, server 130 and terminal device 110 can achieve signaling interaction through the communication connection between them.

[0033] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0034] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0035] Example Interaction

[0036] Figures 2A to 2E Example interfaces 200A to 200E according to some embodiments of the present disclosure are shown. Interfaces 200A to 200E may, for example, be provided by... Figure 1 The terminal device 110 shown is provided.

[0037] like Figure 2A As shown, in some embodiments, when the terminal device 110 receives operation information from the user 140 to launch the application 120, the terminal device 110 may present an interface 200A. The interface 200A is used for the user 140 to input first media content.

[0038] In some embodiments, interface 200A may include controls for inputting first media content. As an example, two controls may be provided. One control is used to upload the first media content stored on terminal device 110. The other control is used to upload the first media content by taking a photo. The control for uploading the first media content may display the word "Upload". The control for uploading the first media content by taking a photo may display the word "Take Photo".

[0039] Regarding the "upload" control, when terminal device 110 receives operation information from user 140 regarding the "upload" control, terminal device 110 can display as follows: Figure 2B The interface 200B is shown. In some embodiments, interface 200B may include locally stored data such as a local photo album. Interface 200B may also be configured with a control for user 140 to select an image, so that terminal device 110 can upload the selected image. As an example, the control for user 140 to select an image may display the word "Select". When terminal device 110 receives operation information from user 140 on the "Select" control in interface 200B, terminal device 110 can upload the image selected by user 140 in interface 200B.

[0040] Regarding the "take a photo" control, when terminal device 110 receives operation information from user 140 regarding the "take a photo" control, terminal device 110 can invoke the camera function and display the corresponding interface. When terminal device 110 obtains the shooting result, terminal device 110 can transmit the shooting result through methods such as... Figure 2C The interface shown is presented in 200C. Figure 2C As shown, in some embodiments, the interface 200C may include, but is not limited to, an image preview area 210 indicating the shooting result, a control for uploading the shooting result, and a control for reshooting, so that the terminal device 110 can acquire the image obtained by shooting. As an example, the control for uploading the shooting result may display the word "Select". The control for reshooting may display the word "Retake Photo".

[0041] In some embodiments, such as Figure 2D As shown, after acquiring the image selected by user 140, terminal device 110 can display interface 200D. As an example, interface 200D can be configured with an image preview area 210 for user 140 to preview the selected image. Furthermore, interface 200D can also be configured with controls indicating the application of corresponding effects to the target model 440, and controls for returning to the image selection step. The control indicating the application of corresponding effects to the target model 440 can display the word "Generate". The control returning to the image selection step can display the word "Reselect".

[0042] In some embodiments, such as Figure 2E As shown, after the terminal device 110 generates second media content based on the first media content, the terminal device 110 can display information related to the second media content through the interface 200E to provide the second media content. As an example, the information related to the second media content may be at least one of a preview image of the second media content and a download link for the second media content.

[0043] It should be understood that Figures 2A to 2E The media content generation interface shown is merely an example; other suitable interfaces can also be used to generate and provide second media content. The various graphic elements in the interface can have different arrangements and different visual representations, one or more elements can be omitted or replaced, and one or more other elements may also be present. The embodiments of this disclosure are not limited in this respect.

[0044] Example process

[0045] Figure 3 A flowchart of an example process 300 for processing media content according to some embodiments of the present disclosure is shown. Process 300 can be implemented at terminal device 110. Reference is made below. Figure 1 To describe process 300.

[0046] like Figure 3 As shown in box 310, terminal device 110 acquires the first media content.

[0047] In some embodiments, the first media content may be media data obtained by the terminal device 110 from the user 140. The first media content may be presented in other forms such as images or videos. As an example, the first media content may be transmitted to the terminal device 110 by means of taking a picture, wired / wireless transmission, etc.

[0048] In box 320, terminal device 110 uses target model 440 to apply target effect to the first media content to generate second media content.

[0049] In some embodiments, the second media content is the media content formed after applying the target effect to the first media content. Similar to the presentation format of the first media content, the second media content can be in the form of an image, video, or other presentation formats.

[0050] In some embodiments, the target model 440 may be of various types depending on the type of media content to be processed and the type of target effect. Examples may include, but are not limited to, a model that can process portrait images, a model that can process face images, a model that can process videos, etc.

[0051] The following will be further combined Figure 4 and Figure 5 To describe the specific training process of the target pattern. Figure 4 A flowchart illustrating an example process 400 for training a target model 440 according to some embodiments of the present disclosure is shown. Figure 5 A flowchart illustrating an example process 500 for training a target model 440 according to some embodiments of the present disclosure is shown. It should be understood that process 400 and / or process 500 can be performed by suitable electronic devices, such as server 130. Process 500 will be described below using server 130 as an example.

[0052] like Figure 5 As shown in box 510, server 130 determines multiple sub-effects corresponding to the target effect.

[0053] In some embodiments, the target effect can be categorized into multiple types based on the type of media content to be processed, or based on the type of object to be processed. For example, the target effect may include various cosmetic effects applied to a facial object.

[0054] In some embodiments, multiple sub-effects may be similar to the target effect. Sub-effects can be categorized into multiple types based on the type of media content to be processed, or based on the type of object to be processed. For example, when the target effect includes multiple makeup effects applied to a facial object, correspondingly, the multiple sub-effects may include makeup effects applied to different parts of the facial object. For instance, target effect A includes makeup effect a1 applied to part 1, makeup effect a2 applied to part 2, and makeup effect a3 applied to part 3. Makeup effects a1, a2, and a3 are different sub-effects corresponding to the target effect.

[0055] In some embodiments, server 130 may determine multiple sub-effects corresponding to the target effect based on the following steps:

[0056] First, multiple preset objects to which the target effect will be applied are identified. In some embodiments, the multiple preset objects can be different parts of a person, such as eyes, skin, etc. As an example, when the target effect includes a makeup effect applied to different parts of a facial object, the multiple preset objects can be different parts of the facial object, such as eyebrows, eyes, mouth, etc. Specifically, the server 130 can break down the objects to which the target effect will be applied to determine the multiple preset objects.

[0057] Then, based on multiple preset objects, multiple sub-effects are determined. In some embodiments, each sub-effect may correspond to a subset of the multiple preset objects. In other words, each sub-effect may act on at least one preset object. At least one preset object is a subset of the multiple preset objects to which the target effect is to be applied. Taking different parts of a face as an example, the multiple preset objects to which the target effect is to be applied can be a cosmetic effect applied to the eyes, or a cosmetic effect applied to the eyes and eyebrows. As an example, when determining multiple sub-effects based on multiple preset objects, server 130 may determine multiple sub-effects based on the common combinations of the multiple preset objects to which the cosmetic effect is applied. For example, if cosmetic effect a, cosmetic effect b, and cosmetic effect c all act on the eyes and eyelashes, then server 130 may take the cosmetic effect applied to the eyes and eyelashes in the target effect as one of the sub-effects.

[0058] In box 520, server 130 determines multiple special effects models corresponding to multiple sub-effects from a pre-trained set of special effects models 410.

[0059] In some embodiments, the pre-trained effects model set 410 may include multiple pre-trained effects models. These multiple effects models have various cosmetic effects, which can be applied to the input image of the effects model to enhance the image. As an example, the effects models in the pre-trained effects model set 410 can be applied to different parts of a facial object.

[0060] In some embodiments, server 130 may determine multiple effect models corresponding to multiple sub-effects based on the following steps:

[0061] First, at least one candidate effect model corresponding to the target sub-effect among multiple sub-effects is determined from the effect model set 410. In some embodiments, when determining at least one candidate effect model, it can be selected one by one from multiple sub-effects to determine at least one candidate effect model corresponding to each selected sub-effect. The selected sub-effect is the target sub-effect.

[0062] The following section will provide a detailed explanation of how, after determining the target sub-effect, at least one candidate effect model corresponding to the target sub-effect is identified.

[0063] In some embodiments, after determining the target sub-effect, server 130 can determine at least one preset object on which the target sub-effect acts. Based on the at least one preset object, server 130 can determine all effect models in effect model set 410 that act on the at least one preset object as candidate effect models. Then, server 130 can filter multiple candidate effect models to obtain at least one candidate effect model corresponding to the target sub-effect. As an example, server 130 can filter based on the degree of deviation between the appearance effect of the candidate effect model and the target sub-effect. The degree of deviation here includes, but is not limited to, the deviation of the color tone of the appearance effect. For example, if the target sub-effect is a warm-toned appearance effect, server 130 can determine at least one candidate effect model with a warm-toned appearance effect from all candidate effect models as a candidate effect model.

[0064] Once at least one candidate effect model corresponding to the target sub-effect is determined, the server 130 may, in response to the fact that the number of at least one candidate effect model is greater than a threshold, determine the target effect model corresponding to the target sub-effect from the at least one candidate effect model based on the model evaluation information of the at least one candidate effect model.

[0065] In some embodiments, when the number of at least one candidate effect model corresponding to the target sub-effect is greater than a threshold, it indicates that there are many available candidate effect models, and the server 130 needs to determine the target effect model from them. Conversely, when the number of at least one candidate effect model corresponding to the target sub-effect is less than or equal to the threshold, it indicates that there are few available candidate effect models, and at least one candidate effect model is the target effect model. As an example, the threshold can be set to 1.

[0066] If the number of at least one candidate special effects model exceeds a threshold, server 130 may first determine the model evaluation information for each candidate special effects model, and then determine the target special effects model based on the model evaluation information of at least one candidate special effects model. In some embodiments, the model evaluation information may indicate the quality of each special effects model in the special effects model set 410. As an example, the model evaluation information may be obtained based on model information associated with the special effects model and may be presented as a score.

[0067] In some embodiments, model information may include, but is not limited to, the number of times the special effects model is used by user 140. Server 130 can determine the model evaluation information for each candidate special effects model using the model information in various ways. For example, server 130 can determine the number of times each special effects model is used by user 140. Then, server 130 determines the maximum number of times the special effects model is used by user 140. Server 130 then determines the model evaluation information for that special effects model based on the ratio of the number of times the special effects model is used by user 140 to the maximum value. Based on this, server 130 can determine the model evaluation information for each candidate special effects model. Alternatively, in some embodiments, the number of times the special effects model is used by user 140 can be directly used as the model evaluation information.

[0068] In some embodiments, the special effects model can also be used as the basis for other special effects models. For ease of explanation, the special effects model used as the basis for other special effects models will be referred to as the reference special effects model. Based on this, the number of times the reference special effects model is used by user 140 can include the number of times the reference special effects model is used by user 140 and the number of times the special effects model derived from the reference special effects model is used by user 140. Thus, server 130 can set corresponding weights to determine the model evaluation information of each special effects model. As an example, server 130 can determine the first score and the second score of each special effects model in the manner described above. The first score is the ratio of the number of times the special effects model is used by user 140 to the maximum value of the number of times the special effects model is used by user 140. The second score is the ratio of the number of times the special effects model derived from the special effects model is used by user 140 to the corresponding maximum value. When the special effects model is not the reference special effects model, the second score of the special effects model can be 0. Server 130 can determine the model evaluation information of each special effects model based on the product of the first score and the corresponding weight and the product of the second score and the corresponding weight. Based on this, server 130 can determine the model evaluation information for each candidate special effects model. In addition, in some embodiments, the model evaluation information for each special effects model can also be determined directly based on the number of times the special effects model derived from the reference special effects model is used by user 140.

[0069] In some embodiments, after determining the model evaluation information of at least one candidate special effects model, the server 130 can determine the candidate special effects model with the best model evaluation information as the target special effects model. As an example, the candidate special effects model with the best model evaluation information is the candidate special effects model with the highest score reflected in the model evaluation information.

[0070] In the above manner, server 130 can sequentially determine the target effect model corresponding to each sub-effect, thereby determining multiple effect models corresponding to multiple sub-effects.

[0071] In box 530, server 130 uses multiple special effects models to process multiple sample images 420 in a preset order to generate multiple corresponding output images.

[0072] In some embodiments, sample images 420 may be a plurality of sample images 420 determined from a set of sample images 420 that satisfy preset constraints, wherein the preset constraints may indicate that each sample image 420 includes a plurality of preset objects. In other words, sample images 420 are all sample images 420 in the set of sample images 420 that include a plurality of preset objects. Here, the plurality of preset objects are the plurality of preset objects to be applied to achieve the target effect. As an example, the set of sample images 420 may include real images (FFHQ dataset) and synthetic images (FFHQ-like dataset).

[0073] In some embodiments, server 130 may generate multiple corresponding output images based on the following steps:

[0074] First, server 130 can combine multiple special effects models into a model chain 430 according to a preset order. In some embodiments, taking two adjacent special effects models in the preset order as an example, the process of server 130 combining multiple special effects models into a model chain 430 is as follows: the output end of the first special effects model is connected to the input end of the second special effects model, wherein the first special effects model can be the one that appears earlier in the sequence among the two adjacent special effects models, and the second special effects model can be the one that appears later in the sequence among the two adjacent special effects models. In this way, server 130 can combine multiple special effects models into a model chain 430.

[0075] In some embodiments, the preset order can be determined based on the relationship between multiple preset objects affected by the effects of multiple special effects models. For example, the effect of special effects model A affects preset objects a1 and a2. The effect of special effects model B affects preset object b. The effect of special effects model C affects preset objects c1 and c2. Through multiple trials, it was found that processing preset object b in sample image 420 first, then processing preset objects a1 and a2, and finally processing preset objects c1 and c2 results in the best effect for the processed sample image 420. Therefore, the preset order can be special effects model B - special effects model A - special effects model C.

[0076] Then, server 130 can use model chain 430 to process multiple sample images 420 to generate multiple output images.

[0077] In some embodiments, the process by which the server 130 processes multiple sample images 420 using the model chain 430 can be as follows: The server 130 inputs a sample image 420 into the model chain 430, where it is processed by the first special effects model in the model chain 430 to obtain a first intermediate image. Then, the second special effects model in the model chain 430 processes the first intermediate image to obtain a second intermediate image. Next, the third special effects model in the model chain 430 processes the second intermediate image to obtain a third intermediate image. This process continues until the Nth special effects model in the model chain 430, i.e., the last special effects model, processes the (N-1)th intermediate image to obtain an output image, where N is an integer greater than 1. Through the above process, the server 130 can generate multiple output images corresponding to the multiple sample images 420.

[0078] In some embodiments, the generated multiple output images corresponding to multiple sample images 420 can indicate the dressing effect of the model chain 430. If the dressing effect of the model chain 430 differs significantly from the target effect of the target model 440, the server 130 can adjust the model chain 430. As an example, the server 130 can make adjustments by redetermining the multiple effect models corresponding to the multiple sub-effects or by adjusting the preset order.

[0079] In some embodiments, the server 130 may redetermine multiple special effect models corresponding to multiple sub-effects by first determining the special effect model that needs adjustment, and then redetermining the target special effect model from at least one candidate special effect model based on the sub-effects corresponding to the special effect model that needs adjustment. As an example, when redetermining the target special effect model, the server 130 may determine the target special effect model in descending order of the scores indicated by the model evaluation information.

[0080] In some embodiments, the server 130 may adjust the preset order by first adjusting the preset order and then recombine multiple special effects models into a model chain 430 according to the adjusted preset order. When the effect of the model chain 430 reflected in the output image still does not match the target effect, the server 130 may adjust the preset order again. As an example, the server 130 may adjust the position of only one special effects model in the preset order each time it adjusts the preset order, and each time it may only move the position of that special effects model forward or backward by one position in the preset order.

[0081] In some embodiments, to reduce the cost of adjusting the appearance effect of the model chain 430 to the target effect of the target model 440, before processing multiple sample images 420 using the model chain 430 to generate multiple output images, the server 130 may first input at least one sample image 420 into the model chain 430 to generate at least one reference image corresponding to the at least one sample image 420. Then, the server 130 determines whether the model chain 430 needs adjustment based on the at least one sample image 420 and the at least one reference image, and adjusts it as described above if adjustment is required.

[0082] In some embodiments, server 130 may adjust model chain 430 in response to the deviation between at least one reference image and at least one corresponding sample image 420. Specifically, server 130 may use image subtraction to obtain the deviation between at least one reference image and at least one corresponding sample image 420, i.e., the subtraction result. The subtraction result may indicate the dressing effect corresponding to multiple preset objects. Further, server 130 may determine at least one dressing effect in the subtraction result that does not match other dressing effects. Through at least one mismatched dressing effect, server 130 may determine the multiple preset objects corresponding to it, and thus determine the special effects model that needs to be adjusted. When adjusting model chain 430, server 130 may adjust it in the manner described above.

[0083] In some embodiments, when the server 130 determines that there are no mismatched dressing effects in the difference results, it indicates that the dressing effect of the model chain 430 matches the target effect of the target model 440 relatively well. At this time, the server 130 can use the model chain 430 to process multiple sample images 420 to generate multiple corresponding output images.

[0084] In box 540, server 130 trains target model 440 using multiple sample images 420 and corresponding multiple output images.

[0085] In some embodiments, server 130 may train target model 440 based on the following steps: First, server 130 constructs multiple training image pairs using multiple sample images 420 and corresponding multiple output images. Then, server 130 constructs a training sample set 450 based on the multiple training image pairs. Finally, server 130 trains target model 440 using training sample set 450.

[0086] In some embodiments, server 130 can determine a set of training image pairs for constructing training sample set 450 by filtering multiple training image pairs. As an example, server 130 can determine a set of training image pairs whose image evaluation information meets preset conditions from multiple training image pairs based on image evaluation information of multiple training image pairs.

[0087] In some embodiments, the image evaluation information of training image pairs can be obtained by subtracting the output image from the sample image 420 in the training image pair determined by server 130. As an example, server 130 can determine the subtraction result of each training image pair based on the output image and sample image 420 of each training image pair. Since the dressing effect of model chain 430 is affected by sample image 420, the subtraction results of different training image pairs will differ. The greater the difference, the better the dressing effect. Based on this, by classifying the subtraction results of multiple training image pairs, image evaluation information for each training image pair can be obtained. The classification result of the subtraction results of training image pairs can indicate the quality level of the training image pairs. Image evaluation information is the quality level of the training image pairs. The process of classifying the subtraction results of multiple training image pairs can be implemented by server 130 or manually.

[0088] In some embodiments, the preset condition related to the image evaluation information may be that the quality level indicated by the image evaluation information reaches a preset level. When the quality level indicated by the image evaluation information reaches the preset level, it indicates that the quality of the training sample pair is good and can be used as training samples. Conversely, when the quality level indicated by the image evaluation information does not reach the preset level, it indicates that the quality of the training sample pair is average and cannot be used as training samples. Based on this, the server 130 can obtain a set of training image pairs with good quality to construct the training sample set 450.

[0089] Based on the process described above, embodiments of this disclosure construct a training sample set 450 by utilizing a model chain 430 of multiple special effects models associated with the target effect to train the target model 440. Embodiments of this disclosure can reduce the human resources required to train the target model 440 and improve the generation efficiency of the target model 440 to a certain extent.

[0090] Example devices and equipment

[0091] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 6 A schematic structural block diagram of an example apparatus 600 for processing media content according to certain embodiments of the present disclosure is shown. Apparatus 600 may be implemented as or included in terminal device 110. Various modules / components in apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0092] like Figure 6As shown, the device 600 includes: an acquisition module 610 configured to acquire first media content; and a generation module 620 configured to apply a target effect to the first media content using a target model to generate second media content, wherein the target model is trained based on the following process: determining multiple sub-effects corresponding to the target effect; determining multiple special effect models corresponding to the multiple sub-effects from a pre-trained set of special effect models; processing multiple sample images according to a preset order using the multiple special effect models to generate multiple corresponding output images; and training the target model using the multiple sample images and the multiple corresponding output images.

[0093] In some embodiments, determining multiple sub-effects corresponding to a target effect includes: determining multiple preset objects to which the target effect is to be applied; and determining multiple sub-effects based on the multiple preset objects, wherein each sub-effect corresponds to a subset of the multiple preset objects.

[0094] In some embodiments, the plurality of sample images are determined based on the following process: determining a plurality of sample images from a set of sample images that satisfy a preset constraint, wherein the preset constraint indicates that each sample image includes a plurality of preset objects.

[0095] In some embodiments, determining multiple effect models corresponding to multiple sub-effects from a pre-trained set of effect models includes: determining at least one candidate effect model corresponding to a target sub-effect among the multiple sub-effects from the set of effect models; and, in response to the number of at least one candidate effect model being greater than a threshold, determining a target effect model corresponding to the target sub-effect from at least one candidate effect model based on model evaluation information of at least one candidate effect model.

[0096] In some embodiments, processing multiple sample images using multiple special effects models in a preset order to generate multiple corresponding output images includes: combining multiple special effects models into a model chain in a preset order, wherein the output end of a first special effects model in the model chain is connected to the input end of a second special effects model; and processing multiple sample images using the model chain to generate multiple output images.

[0097] In some embodiments, training a target model using multiple sample images and corresponding multiple output images includes: constructing multiple training image pairs using multiple sample images and corresponding multiple output images; constructing a training sample set based on the multiple training image pairs; and training the target model using the training sample set.

[0098] In some embodiments, constructing a training sample set based on multiple training image pairs includes: determining a set of training image pairs whose image evaluation information satisfies preset conditions from the multiple training image pairs based on image evaluation information of the multiple training image pairs; and constructing a training sample set based on a set of training image pairs.

[0099] In some embodiments, the target effect includes multiple cosmetic effects applied to a facial object, and the multiple sub-effects include cosmetic effects applied to different parts of the facial object.

[0100] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0101] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0102] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0103] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0104] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0105] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0106] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0108] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0110] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for processing media content, comprising: Obtain first-hand media content; as well as The target effect is applied to the first media content using a target model to generate the second media content. The target model is trained based on the following process: determining multiple sub-effects corresponding to the target effect; determining multiple special effect models corresponding to the multiple sub-effects from a pre-trained set of special effect models; and using the multiple special effect models to process multiple sample images in a preset order to generate multiple corresponding output images. The target model is trained using the plurality of sample images and the corresponding plurality of output images.

2. The method according to claim 1, wherein determining the plurality of sub-effects corresponding to the target effect includes: Identify multiple preset objects to which the target effect will be applied; as well as Based on the plurality of preset objects, the plurality of sub-effects are determined, wherein each sub-effect corresponds to a subset of the plurality of preset objects.

3. The method of claim 2, wherein the plurality of sample images are determined based on the following process: The plurality of sample images that satisfy preset constraints are determined from the sample image set, wherein the preset constraints indicate that each sample image includes the plurality of preset objects.

4. The method according to claim 1, wherein determining the plurality of special effects models corresponding to the plurality of sub-effects from the pre-trained set of special effects models comprises: Determine at least one candidate special effect model corresponding to the target sub-effect among the plurality of sub-effects from the set of special effect models; as well as In response to the fact that the number of at least one candidate effect model is greater than a threshold, a target effect model corresponding to the target sub-effect is determined from the at least one candidate effect model based on the model evaluation information of the at least one candidate effect model.

5. The method according to claim 1, wherein processing multiple sample images according to a preset order using the plurality of special effects models to generate multiple corresponding output images includes: According to the preset order, the multiple special effects models are combined into a model chain, wherein the output end of the first special effects model in the model chain is connected to the input end of the second special effects model; as well as The model chain is used to process the multiple sample images to generate the multiple output images.

6. The method according to claim 1, wherein training the target model using the plurality of sample images and the corresponding plurality of output images comprises: Multiple training image pairs are constructed using the multiple sample images and the corresponding multiple output images; Based on the multiple training image pairs, a training sample set is constructed; as well as The target model is trained using the training sample set.

7. The method of claim 6, wherein constructing a training sample set based on the plurality of training image pairs comprises: Based on the image evaluation information of the plurality of training image pairs, a set of training image pairs whose image evaluation information satisfies preset conditions is determined from the plurality of training image pairs; as well as The training sample set is constructed based on the set of training image pairs.

8. The method of claim 1, wherein the target effect includes multiple cosmetic effects applied to a facial object, and the multiple sub-effects include cosmetic effects applied to different parts of the facial object.

9. An apparatus for processing media content, comprising: The acquisition module is configured to acquire the first media content; as well as A generation module is configured to apply a target effect to the first media content using a target model to generate second media content, wherein the target model is trained based on the following process: determining multiple sub-effects corresponding to the target effect; determining multiple special effect models corresponding to the multiple sub-effects from a pre-trained set of special effect models; processing multiple sample images according to a preset order using the multiple special effect models to generate multiple corresponding output images; and training the target model using the multiple sample images and the corresponding multiple output images.

10. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8.

Images