Method and device for processing target feature in media content, equipment and medium

Abstract

The embodiment of the invention provides a method and device for processing a target feature in media content, equipment and a medium. The method comprises the steps that a candidate mask set is generated according to target pixels and line draft information of original media content, the target pixels comprise first pixels, and the first pixels are used for representing target features; fusing the candidate mask patterns in the candidate mask set to obtain a target mask pattern; and shielding a modification operation on the target feature based on the target mask pattern. According to the embodiment of the invention, in the setting operation process of the original media content, the operation on the target feature is shielded based on the target mask pattern, so that the target feature can be kept unchanged in different operation scenes of the original media content, the setting operations such as stylization on the target object of the original media content are improved, and the user experience is improved. And the problem of target feature change is solved.

Description

Technical Field

[0001] This disclosure relates to computer technology, and more particularly to a method, apparatus, device, and medium for processing target features in media content. Background Art

[0002] With the development of computer technology, the demand for editing original media content based on text descriptions is increasing. For example, text descriptions can be used to stylize faces or change hairstyles in original images to obtain target images with specific styles.

[0003] Currently, using a pre-defined diffusion model to perform stylization or hairstyle changes on target objects in original media content may alter the target features corresponding to those objects. For example, using a diffusion model to stylize a face in an original image yields a face with a specific style. Because model calculations are performed on the face in the original image, the facial features change, making the facial features in the target image different from those in the original image. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and medium for processing target features in media content, which can shield operations on target features in the original media content to keep the target features unchanged.

[0005] In a first aspect, embodiments of this disclosure provide a method for processing target features in media content, including:

[0006] A candidate mask set is generated based on the target pixels and line art information of the original media content, wherein the target pixels include a first pixel, which is used to characterize the target features;

[0007] The target mask image is obtained by fusing the candidate mask images from the candidate mask set;

[0008] Modification operations on the target features are shielded based on the target mask image.

[0009] Secondly, embodiments of this disclosure also provide a processing apparatus for target features in media content, the apparatus comprising:

[0010] A mask generation module is used to generate a candidate mask set based on the target pixels and line art information of the original media content, wherein the target pixels include a first pixel, which is used to characterize the target features;

[0011] A mask fusion module is used to fuse candidate mask images from the candidate mask set to obtain a target mask image;

[0012] An operation masking module is used to mask modification operations on the target features based on the target mask image.

[0013] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] Storage device for storing one or more programs.

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for processing target features in media content as described in any embodiment of this disclosure.

[0017] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a method for processing target features in media content as described in any embodiment of this disclosure.

[0018] This disclosure provides a method, apparatus, device, and medium for processing target features in media content. A candidate mask set is generated using target pixels and line art information from the original media content. The candidate mask images in the candidate mask set are then fused to obtain a target mask image. During the setting operations on the original media content, operations on the target features are shielded based on the target mask image. This ensures that the target features remain unchanged under different operation scenarios on the original media content, mitigating the problem of target feature changes caused by stylization or other setting operations on the target object of the original media content. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart illustrating a method for processing target features in media content, provided as an embodiment of this disclosure;

[0021] Figure 2 This is a schematic diagram of a target mask image under a human face pose provided in an embodiment of the present disclosure;

[0022] Figure 3 This is a schematic diagram of a target mask image under another face pose provided in an embodiment of the present disclosure;

[0023] Figure 4 A flowchart illustrating another method for processing target features in media content provided in this embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of a processing device for target features in media content provided in an embodiment of this disclosure;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0030] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0031] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0032] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0033] Figure 1This is a flowchart illustrating a method for processing target features in media content according to an embodiment of the present disclosure. This embodiment is applicable to situations involving stylization or hairstyle changes for faces, such as scenarios where a diffusion model is used for face stylization or hairstyle changes. The method can be executed by a device for processing target features in media content. This device can be implemented in software and / or hardware, optionally through an electronic device, such as a mobile terminal, PC, or server.

[0034] like Figure 1 As shown, the method includes:

[0035] S110. Generate a candidate mask set based on the target pixels and line art information of the original media content.

[0036] In this embodiment of the disclosure, the original media content represents the media content to be set up. The original media content includes images or videos, and includes a target object. For example, the target object may include a human face, a pet's face, a prominent object, etc. The original media content may include images or videos stored in an electronic device. Alternatively, the original media content may include images or videos downloaded from a server. Alternatively, the original media content may also include images or videos taken by the user.

[0037] The setup operations can include stylizing or reshaping the target object using an image generation model. Image generation models include diffusion models, among others. During forward diffusion, the diffusion model iteratively adds noise to the latent feature map corresponding to the original media content to obtain a noise map. During back diffusion, iteratively performs noise prediction and image generation operations on the noise map to output the target media content. The target media content represents the processing result obtained by stylizing or reshaping the target object in the original media content.

[0038] The target pixel includes a first pixel and a second pixel. The first pixel is used to represent target features. For example, the first pixel is the pixel corresponding to the target object in the original media content. The first pixel may include pixels corresponding to forehead keypoints and facial keypoints. Forehead keypoints represent the forehead area of ​​the face. Facial keypoints represent specific parts of the face, such as eyebrows, nose, mouth, and chin. Target features represent facial appearance and are visual features that distinguish one person from another. The second pixel can represent pixels corresponding to pose estimation information.

[0039] Line art is a drawing method that uses lines to depict the outlines and detailed structures of objects, aiming to achieve the integrity and accuracy of the image. Line art information can represent the line information used to describe the outlines and detailed structures of objects in the image. A line art mask corresponding to the input image can be obtained by inputting an image into a preset line art generation model. The line art generation model represents a deep learning model used to generate line art or sketches of various realistic objects. The line art mask can be represented as an image with a black background, using white lines to describe the outlines and detailed structures of objects.

[0040] A candidate mask set is a collection of candidate mask images. These candidate mask images are generated in different ways based on target pixels and line art information. For example, a first candidate mask image is generated based on a first pixel in the original media content; a second pixel is determined based on pose estimation information from the original media content, and a second candidate mask image is determined based on the second pixel; a line art mask image corresponding to the original media content is obtained, and a local line art mask image of the target object in the original media content is determined based on the line art mask image and the first pixel; a candidate mask set is constructed based on the first candidate mask image, the second candidate mask image, and the local line art mask image.

[0041] For example, to obtain an initial mask, based on the positions of the pixels corresponding to the forehead and facial keypoints in the first pixel, the forehead and facial keypoints corresponding to the target object in the original media content are drawn onto the initial mask to obtain a first candidate mask. The initial mask can be an image with the same size as the original media content but without any content drawn on it. For example, the initial mask can be a completely black image with the same size as the original media content. Therefore, the positions of the forehead and facial keypoints corresponding to the target object in the original media content can be determined in the initial mask. The forehead and facial keypoints are then drawn onto the initial mask based on the set color and position to obtain the first candidate mask.

[0042] The second pixel is used to represent key points for pose estimation, which can be determined by a human pose estimation model. For example, the second pixel includes pixels corresponding to key points of facial features, hands, and feet.

[0043] For example, an initial mask is obtained, and based on the positions of the pixels corresponding to the key points of facial features, hands, and feet in the second pixel, the key points of facial features, hands, and feet in the original media content are drawn onto the initial mask to obtain a second candidate mask.

[0044] A local line art mask can represent a mask for a specific region within the line art mask corresponding to the original media content. For example, a local line art mask can be generated by extracting the line art information corresponding to the face region from the line art mask corresponding to the original media content. Alternatively, the line art information corresponding to the face region can be obtained from the line art mask corresponding to the original media content, and a local line art mask can be generated based on the difference between the line art information corresponding to the face region and the forehead region.

[0045] For example, the original media content is input into a pre-defined line drawing generation model to obtain the line drawing mask output by the model. The original media content is then input into a pre-defined human body analysis network to obtain the components of the human body. This human body analysis network is used to identify pixel-level components of a human image, thereby segmenting the person into multiple semantically consistent regions. For example, the human body in an image can be segmented into hair, face, left ear, left arm, left leg, right ear, right arm, and right leg regions.

[0046] If the application scenario requires protection of the entire face's features, this can be achieved by configuring the parameter settings to the corresponding values. Similarly, if the application scenario requires editing the upper half of the face, the lower half's features can be protected by configuring the parameter settings to the corresponding values. For example, in a hairstyle change scenario, where bangs need to be changed to no bangs, the lower half's features need to be preserved. It should be noted that the parameter settings can be implemented in multiple ways. For instance, a parameter setting can indicate whether to protect half of the face's features. Parameter values ​​for these settings include true and false. Alternatively, a parameter value of 1 indicates protection of the entire face, while a parameter value of 0 indicates protection of only half of the face. This disclosure does not specifically limit the implementation method of the parameter settings.

[0047] S120. The candidate mask images in the candidate mask set are fused to obtain the target mask image.

[0048] The target mask is obtained by fusing candidate masks according to a preset fusion rule. The preset fusion rule may include rules for fusing candidate masks based on pixel values ​​in the candidate masks. Since pixel values ​​include values ​​from different channels, fusion rules can be set along the channel dimension. For example, for original media content in RGB color mode, each pixel includes three channels, and the pixel value is represented as the R channel pixel value, G channel pixel value, and B channel pixel value. For original media content in Lab color mode, each pixel includes three channels, and the pixel value is represented as the L channel pixel value, a channel pixel value, and b channel pixel value. This disclosure does not specifically limit the color mode of the original media content.

[0049] In some embodiments, the preset fusion rule may be to replace the pixel values ​​corresponding to the target color channels of the base image with the pixel values ​​at the corresponding pixel positions of the first candidate mask and the base image. It may also replace the pixel values ​​corresponding to the remaining color channels of the base image with the pixel values ​​at the corresponding pixel positions of the local line drawing mask and the base image.

[0050] For example, a set of candidate mask images in the candidate mask set is used as a base image; for the target color channel of the base image, the pixel values ​​of the remaining candidate masks in the candidate mask set (excluding the base image) in the corresponding color channel are used to update the pixel values ​​of the target color channel to obtain the target mask image.

[0051] For example, the candidate mask that contains the most information about the original media content is used as the base image. Since the second candidate mask is generated based on pose estimation keypoints in the original media content and contains the most information about the original media content, while the first candidate mask represents information about the face region in the original media content, and the local line drawing mask is also used to represent information about the face region in the original media content, the second candidate mask contains the most information about the original media content and is therefore used as the base image.

[0052] The target color channel includes a first target color channel and a second target color channel, wherein the first target color channel includes one or more color channels, and the second target color channel includes one or more color channels. In some embodiments, the first target color channel and the second target color channel cover all color channels of the pixel.

[0053] Specifically, the pixel value corresponding to the first target color channel of the pixel in the first candidate mask in the candidate mask set is used to update the pixel value at the corresponding pixel position in the second candidate mask; the pixel value corresponding to the second target color channel of the pixel in the local line drawing mask in the candidate mask set is used to update the pixel value at the corresponding pixel position in the second candidate mask.

[0054] Assuming the original media content is an image in RGB color mode, the pixel value corresponding to the first target color channel can be represented as the pixel value corresponding to the red channel. Similarly, the pixel value corresponding to the second target color channel can be represented as the pixel values ​​corresponding to the green and blue channels. Alternatively, the pixel value corresponding to the first target color channel can be represented as the pixel values ​​corresponding to the red and blue channels, and the pixel value corresponding to the second target color channel can be represented as the pixel value corresponding to the green channel.

[0055] Figure 2 This is a schematic diagram of a target mask image under facial pose provided in an embodiment of this disclosure. Figure 2 As shown, the target mask includes facial features and pose information, with eyebrows, eyes, nose, mouth, and facial contours marked by facial key points (represented by magenta dots). Connecting the ears, eyes, nose, and mouth with lines of different colors creates polylines. These polylines are then used to determine the facial pose. Figure 3 This is a schematic diagram of a target mask image under another face pose provided in an embodiment of this disclosure. (Combined with...) Figure 2 and 3 It can be seen that the angle of the blue line segment in the broken line is different under different postures.

[0056] S130. Based on the target mask image, the modification operation on the target features is shielded.

[0057] The "masking" function, which masks operations on target features, ensures that the target features remain consistent before and after performing a setting operation on the original media content. The setting operation represents an operation that alters facial features. For example, a setting operation could include facial stylization or changing a hairstyle. If the setting operation is changing a hairstyle, the facial features remain consistent before and after the hairstyle change. If the setting operation is facial stylization, the facial features remain consistent before and after the stylization.

[0058] Obtain the operation request to be performed on the original media content. If the requested operation is a set operation, then the operation on the target feature is masked based on the target mask. If the requested operation is not a set operation, then the requested operation is executed, and the operation masked based on the target mask is abandoned.

[0059] For example, the original media content, target prompt words, and target mask are input into a preset diffusion model, wherein the target prompt words are text information describing the expected effect of the setting operation, and the preset diffusion model is used to shield the modification operation of the target features during the setting operation on the original media content; the preset diffusion model performs the setting operation on the original media content based on the target prompt words and target mask, and outputs the target media content, which has the same target features as the original media content.

[0060] In this embodiment, the training sample set of the diffusion model includes media content samples, cue word samples, and mask image samples. The media content samples, cue word samples, and mask image samples are input into the diffusion model to be trained. The diffusion model performs a generation operation based on the media content samples, cue word samples, and mask image samples to obtain the generation result. A loss value is calculated based on the target features of the generated result and the target features of the media content samples. The parameters of the diffusion model to be trained are adjusted during backpropagation based on the loss value. This process is iteratively executed until the model training is complete. It should be noted that, in order to train a diffusion model with image inpainting capabilities, a completely white mask image sample is also required as model input, and then the model is trained using the above training method.

[0061] For example, a media content sample is determined based on the original image or video. A candidate mask set corresponding to the media content sample is generated based on the target pixels and line art information of the media content sample. The candidate mask set includes a first candidate mask image, a second candidate mask image, and a local line art mask image of the media content sample. The local line art mask image is determined based on the line art information of the entire face, including: generating a first face region of the media content sample based on facial key points. The media content sample is input into a human body analysis network to obtain the second face region of the media content sample output by the human body analysis network. The union of the first and second face regions is taken to obtain the face region of the media content sample, and a face mask is generated based on the face region. The line art mask image and the face mask corresponding to the media content sample are combined to determine the face line art mask image in the media content sample. The first candidate mask image, the second candidate mask image, and the face line art mask image in the candidate mask set corresponding to the media content sample are fused to obtain the mask image sample corresponding to the media content sample. It should be noted that the generation method of the candidate mask set and the fusion method of the candidate mask images have been described in the above embodiments, and will not be repeated here.

[0062] The technical solution of this disclosure generates a candidate mask set by using the target pixels and line drawing information of the original media content, and merges the candidate mask images in the candidate mask set to obtain a target mask image. Then, during the setting operation of the original media content, the operation on the target features is shielded based on the target mask image. This can keep the target features unchanged under different operation scenarios of the original media content, and improve the problem of target feature changes caused by setting operations such as stylization of the target object of the original media content.

[0063] Figure 4 This is a flowchart illustrating another method for processing target features in media content provided by an embodiment of the present disclosure. Based on the above embodiments, this embodiment of the present disclosure specifically defines the method for generating local line drawing mask maps.

[0064] like Figure 4 As shown, the method includes:

[0065] S401. Obtain the original media content and the parameter values ​​corresponding to the parameter configuration items.

[0066] For example, a content selection control and parameter configuration items are displayed through the target interactive interface. In response to a triggering action of the content selection control on the target interactive interface, the user selects the original media content. For instance, when a user clicks the content selection control, a list of content sources is displayed. The content source list includes options such as obtaining from local storage, capturing images, or downloading from the internet. If the user clicks the "obtain from local storage" option, the original media content is selected from locally stored images or videos. If the user clicks the "capture images" option, the original media content is captured using the camera on the electronic device. If the user clicks the "download from the internet" option, the user is redirected to the target search engine to download the original media content from the internet.

[0067] In response to parameter configuration settings, the system retrieves parameter values ​​and determines whether to protect the entire or partial area of ​​the target object based on these values. For example, it might determine whether to protect features of the entire face or only half of the face based on the parameter values.

[0068] S402. Generate a first candidate mask based on the first pixel in the original media content.

[0069] S403. Determine the second pixel based on the pose estimation information of the original media content, and determine the second candidate mask based on the second pixel.

[0070] S404. Obtain the line drawing mask corresponding to the original media content.

[0071] S405. If the parameter value represents the protection of the complete area of ​​the target object, generate a first reference area based on the first pixel, and obtain the second reference area corresponding to the target object.

[0072] The first reference region represents the region formed by defined key points in the target object. For example, the defined key points are facial key points. The second reference region represents the region corresponding to the target object generated by the human body parsing network.

[0073] S406. Take the union of the first reference region and the second reference region to obtain the third reference region.

[0074] S407. Based on the line drawing mask and the third reference area, determine the local line drawing mask of the target object in the original media content.

[0075] In some embodiments, if the entire face is to be protected, the pixels representing facial key points in the first pixel are obtained, and the pixels representing facial key points are connected to form a first reference region. That is, the first reference region represents the face region formed by the facial key points corresponding to the original media content. The second reference region represents the face region of the target object in the original media content determined by the human body analysis network. For example, the original media content is input into the human body analysis network to obtain the second reference region. The union of the first and second reference regions is taken to obtain the third reference region. Since the third reference region is determined by combining the first and second reference regions, the first reference region is corrected using the second reference region to obtain a more accurate face region. The third reference region is dilated until the dilated third reference region overlaps with the bilateral ear regions (left ear region and right ear region). For example, the dilated face region covers part of the bilateral ear region, or the dilated face region overlaps with the bilateral ear region at the boundary. The line drawing information of the face in the line drawing mask is obtained based on the third reference region to obtain the local line drawing mask of the face in the original media content.

[0076] S408. If the parameter value indicates that a local area of ​​the target object is to be protected, a fourth reference area is generated based on the first pixel, and a second reference area corresponding to the target object is obtained.

[0077] The fourth reference region represents the area formed by designated key points in the target object. For example, the designated key points are the forehead key points.

[0078] S409. Determine the fifth reference region based on the difference between the second reference region and the fourth reference region.

[0079] S410. Based on the line drawing mask and the fifth reference area, determine the local line drawing mask of the target object in the original media content.

[0080] In some embodiments, to protect half of the face, pixels corresponding to key points on the forehead in the first pixel can be connected to generate a fourth reference region. The fourth reference region is then dilated to obtain an expanded fourth reference region. This dilation can include dilation above and below the forehead to increase the coverage area of ​​the forehead region, making the expanded forehead region closer to the actual forehead coverage area. The face region of the target object in the original media content, determined using a human body analysis network, is used as the second reference region. The fourth reference region is subtracted from the second reference region to obtain the fifth reference region. Using the fourth reference region to represent the forehead region, the forehead region is subtracted from the face region to obtain the fifth reference region corresponding to the lower half of the face. The line art information of the lower half of the face in the line art mask is obtained based on the fifth reference region to obtain a local line art mask of the face in the original media content.

[0081] S411. A candidate mask set is constructed based on the first candidate mask image, the second candidate mask image, and the local line drawing mask image.

[0082] S412. The candidate mask images in the candidate mask set are fused to obtain the target mask image.

[0083] S413. Based on the target mask image, the modification operation on the target features is shielded.

[0084] The technical solution of this disclosure, by configuring the feature protection range of the target object by setting the parameter value of the parameter configuration item, and then generating local line drawing masks with different ranges according to the parameter value, the first pixel and the line drawing mask, protects the complete area of ​​the target object or protects a local area of ​​the target object through the local line drawing mask, thereby achieving precise setting of the target features to be protected, and can adapt to the image processing needs in different scenarios.

[0085] Figure 5 This is a schematic diagram of a device for processing target features in media content, provided as an embodiment of the present disclosure. The device can be implemented in software and / or hardware, and optionally, it can be implemented using an electronic device, such as a mobile terminal, a PC, or a server.

[0086] like Figure 5 As shown, the device includes: a mask generation module 510, a mask fusion module 520, and an operation shielding module 530.

[0087] The mask generation module 510 is used to generate a candidate mask set based on the target pixels and line art information of the original media content, wherein the target pixels include a first pixel, which is used to characterize the target features;

[0088] The mask fusion module 520 is used to fuse the candidate mask images in the candidate mask set to obtain the target mask image;

[0089] The operation shielding module 530 is used to shield modifications to the target features based on the target mask image.

[0090] Optionally, the mask generation module 510 is specifically used for:

[0091] A first candidate mask is generated based on the first pixel in the original media content;

[0092] The second pixel is determined based on the pose estimation information of the original media content, and the second candidate mask is determined based on the second pixel;

[0093] Obtain the line art mask corresponding to the original media content, and determine the local line art mask of the target object in the original media content based on the line art mask and the first pixel;

[0094] A candidate mask set is constructed based on the first candidate mask image, the second candidate mask image, and the local line drawing mask image.

[0095] Optionally, determining the local line art mask of the target object in the original media content based on the line art mask and the first pixel includes:

[0096] A first reference region is generated based on the first pixel, and a second reference region corresponding to the target object is obtained.

[0097] The third reference region is obtained by taking the union of the first reference region and the second reference region;

[0098] Based on the line drawing mask and the third reference area, a local line drawing mask of the target object in the original media content is determined.

[0099] Optionally, determining the local line art mask of the target object in the original media content based on the line art mask and the first pixel includes:

[0100] A fourth reference region is generated based on the first pixel, and a second reference region corresponding to the target object is obtained;

[0101] The fifth reference region is determined based on the difference between the second reference region and the fourth reference region;

[0102] Based on the line drawing mask and the fifth reference area, a local line drawing mask of the target object in the original media content is determined.

[0103] Optionally, the mask fusion module 520 is specifically used for:

[0104] Use the selected candidate mask image in the candidate mask set as the base image;

[0105] For the target color channel of the base image, the pixel values ​​of the remaining candidate masks (excluding the base image) in the corresponding color channel are used to update the pixel values ​​of the target color channel, thus obtaining the target mask image.

[0106] Optionally, the base image is the second candidate mask image in the candidate mask set, and the target color channel includes a first target color channel and a second target color channel; updating the pixel value corresponding to the target color channel using the pixel values ​​of the remaining candidate masks in the candidate mask set (excluding the base image) in the corresponding color channel includes:

[0107] The pixel value corresponding to the first target color channel of the pixel in the first candidate mask image of the candidate mask set is used to update the pixel value at the corresponding pixel position in the second candidate mask image;

[0108] The pixel value at the corresponding pixel position in the second candidate mask is updated using the pixel value corresponding to the second target color channel of the pixel in the local line drawing mask image of the candidate mask set.

[0109] Optionally, the shielding module 530 is specifically used for:

[0110] The original media content, target prompt words, and target mask are input into a preset diffusion model. The target prompt words are text information describing the expected effect of the set operation. The preset diffusion model is used to block modification operations on target features during the process of performing the set operation on the original media content.

[0111] The preset diffusion model performs a setting operation on the original media content based on the target prompt word and the target mask map, and outputs the target media content, which has the same target features as the original media content.

[0112] The target feature processing apparatus for media content provided in this disclosure can execute the target feature processing method for media content provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0113] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0114] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 6 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 6 The diagram below shows the structure of the terminal device or server 600. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0115] like Figure 6As shown, electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. An edit / output (I / O) interface 605 is also connected to bus 604.

[0116] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0117] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0118] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0119] The electronic device provided in this embodiment and the method for processing target features in media content provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0120] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the method for processing target features in media content provided in the above embodiments.

[0121] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0122] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0123] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0124] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0125] A candidate mask set is generated based on the target pixels and line art information of the original media content, wherein the target pixels include a first pixel, which is used to characterize the target features;

[0126] The target mask image is obtained by fusing the candidate mask images from the candidate mask set;

[0127] Modification operations on the target features are shielded based on the target mask image.

[0128] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0130] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0131] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0132] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0133] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0134] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0135] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for processing target features in media content, characterized in that, include: A candidate mask set is generated based on the target pixels and line art information of the original media content, wherein the target pixels include a first pixel, which is used to characterize the target features; The target mask image is obtained by fusing the candidate mask images from the candidate mask set; Modification operations on the target features are shielded based on the target mask image.

2. The method according to claim 1, characterized in that, The process of generating a candidate mask set based on the target pixels and line art information of the original media content includes: A first candidate mask is generated based on the first pixel in the original media content; The second pixel is determined based on the pose estimation information of the original media content, and the second candidate mask is determined based on the second pixel; Obtain the line art mask corresponding to the original media content, and determine the local line art mask of the target object in the original media content based on the line art mask and the first pixel; A candidate mask set is constructed based on the first candidate mask image, the second candidate mask image, and the local line drawing mask image.

3. The method according to claim 2, characterized in that, The step of determining the local line art mask of the target object in the original media content based on the line art mask and the first pixel includes: A first reference region is generated based on the first pixel, and a second reference region corresponding to the target object is obtained. The third reference region is obtained by taking the union of the first reference region and the second reference region; Based on the line drawing mask and the third reference area, a local line drawing mask of the target object in the original media content is determined.

4. The method according to claim 2, characterized in that, The step of determining the local line art mask of the target object in the original media content based on the line art mask and the first pixel includes: A fourth reference region is generated based on the first pixel, and a second reference region corresponding to the target object is obtained; The fifth reference region is determined based on the difference between the second reference region and the fourth reference region; Based on the line drawing mask and the fifth reference area, a local line drawing mask of the target object in the original media content is determined.

5. The method according to claim 1, characterized in that, The process of fusing the candidate mask images from the candidate mask set to obtain the target mask image includes: Use the selected candidate mask image in the candidate mask set as the base image; For the target color channel of the base image, the pixel values ​​of the remaining candidate masks (excluding the base image) in the corresponding color channel are used to update the pixel values ​​of the target color channel, thus obtaining the target mask image.

6. The method according to claim 5, characterized in that, The base image is the second candidate mask image in the candidate mask set, and the target color channel includes a first target color channel and a second target color channel; The step of updating the pixel value corresponding to the target color channel by using the pixel values ​​of the remaining candidate masks (excluding the base image) in the candidate mask set for the corresponding color channel includes: The pixel value corresponding to the first target color channel of the pixel in the first candidate mask image of the candidate mask set is used to update the pixel value at the corresponding pixel position in the second candidate mask image; The pixel value at the corresponding pixel position in the second candidate mask is updated using the pixel value corresponding to the second target color channel of the pixel in the local line drawing mask image of the candidate mask set.

7. The method according to claim 1, characterized in that, The modification operation of the target features based on the target mask image includes: The original media content, target prompt words, and target mask are input into a preset diffusion model. The target prompt words are text information describing the expected effect of the set operation. The preset diffusion model is used to block modification operations on target features during the process of performing the set operation on the original media content. The preset diffusion model performs a setting operation on the original media content based on the target prompt word and the target mask map, and outputs the target media content, which has the same target features as the original media content.

8. A processing apparatus for target features in media content, characterized in that, include: A mask generation module is used to generate a candidate mask set based on the target pixels and line art information of the original media content, wherein the target pixels include a first pixel, which is used to characterize the target features; A mask fusion module is used to fuse candidate mask images from the candidate mask set to obtain a target mask image; An operation masking module is used to mask modification operations on the target features based on the target mask image.

9. An electronic device, characterized in that, The electronic device includes: one or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method for processing target features in media content as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform a method for processing target features in media content as described in any one of claims 1-7.

Images