Processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining to-be-processed content; obtaining reference content corresponding to the to-be-processed content; generating target description information based on the reference content, the target description information being used for describing the style of the reference content in a text form; performing feature extraction based on the target description information and the reference content to obtain a first reference style feature; the first reference style feature is used for representing the style of the reference content; based on the first reference style feature, the to-be-processed content is stylized, first target content is obtained, and the style of the first target content is the same as that of the reference content. According to the method, the user does not need to input cue words, the reference style features used for representing the styles desired by the user can be adaptively extracted from the reference content, stylization processing of the to-be-processed content is achieved, and the quality of the generated target content and the use experience of the user are improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the development of AI (Artificial Intelligence) technology, more and more users are using AI-generated images and videos in the style they want, based on text descriptions, such as converting photos into oil painting styles. However, the solutions in these technologies rely on user text descriptions, requiring users to input prompts, which places high demands on the user. Moreover, the prompts affect the quality of the generated images or videos; deviations or errors in the input will lead to poor quality images or videos, impacting the user experience. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, this disclosure provides a processing method, apparatus, electronic device and storage medium.

[0004] According to a first aspect of the present disclosure, a processing method is provided, the method comprising:

[0005] Obtain the content to be processed, which includes images to be processed or videos containing multiple frames of images to be processed;

[0006] Obtain reference content corresponding to the content to be processed, wherein the reference content includes reference images or reference videos containing multiple frames of reference images;

[0007] Based on the reference content, target description information is generated, which is used to describe the style of the reference content in text form;

[0008] Based on the target description information and the reference content, feature extraction is performed to obtain a first reference style feature; the first reference style feature is used to characterize the style of the reference content.

[0009] Based on the first reference style feature, the content to be processed is stylized to obtain the first target content, the style of the first target content being the same as the style of the reference content.

[0010] In some embodiments, generating target description information based on the reference content includes:

[0011] The reference content is parsed to obtain text description information; the text description information is used to describe the style and content of the reference content in text form;

[0012] The target description information is obtained by preprocessing the text description information.

[0013] In some embodiments, the preprocessing based on the text description information to obtain the target description information includes:

[0014] In response to the editing operation on the text description information, the edited text description information is obtained;

[0015] From the edited text description information, words related to the style description are filtered out, and the filtered words are determined as the target description information.

[0016] In some embodiments, the step of extracting features based on the target description information and the reference content to obtain first reference style features includes:

[0017] Feature extraction is performed on the target description information to obtain the text features of the target description information;

[0018] Based on the text features and the reference content, the target encoding features are obtained;

[0019] Decoding is performed based on the text features and the target encoding features to obtain the target decoded features;

[0020] Based on the target encoding features and the target decoding features, the first reference style features are extracted.

[0021] In some embodiments, when the reference content includes a reference video, the encoding based on the text features and the reference content to obtain target encoded features includes:

[0022] Encoding is performed based on the text features and multiple reference images in the reference video to obtain the encoded features corresponding to each reference image;

[0023] The target coding features are obtained by concatenating the coding features corresponding to the multiple reference images according to the channel dimension.

[0024] In some embodiments, the stylization processing of the content to be processed based on the first reference style feature to obtain the first target content includes:

[0025] The content to be processed is encoded to obtain the processing features of the content to be processed;

[0026] Based on the first reference style feature, the feature to be processed is stylized to obtain the target feature;

[0027] The target features are decoded to obtain the first target content.

[0028] In some embodiments, the step of extracting features based on the target description information and the reference content to obtain first reference style features includes:

[0029] A visual style coding model is invoked to extract features from the target description information and the reference content to obtain the first reference style features; wherein, the visual style coding model is used to extract style features from the input content.

[0030] In some embodiments, the visual style encoding model includes a text encoder, a first encoder, a first decoder, and an attention network; the step of invoking the visual style encoding model to extract features from the target description information and the reference content to obtain the first reference style features includes:

[0031] The text encoder is invoked to extract features from the target description information to obtain the text features of the target description information.

[0032] The first encoder is invoked to encode based on the text features and the reference content to obtain the target encoded features;

[0033] The first decoder is invoked to decode based on the text features and the target encoded features to obtain the target decoded features;

[0034] The attention network is invoked to extract the first reference style features based on the target encoding features and the target decoding features.

[0035] In some embodiments, the first encoder includes N encoding layers; the step of invoking the first encoder to encode based on the text features and the reference content to obtain target encoded features includes:

[0036] The text features and the reference content are input into the first encoding layer, and the first encoded features are output.

[0037] The (K-1)th encoded feature output from the (K-1)th encoding layer, or the combined feature corresponding to the (K-1)th encoded feature, is input into the Kth encoding layer, and the Kth encoded feature is output, until the Nth encoding layer outputs the Nth encoded feature; wherein, the combined feature corresponding to the (K-1)th encoded feature includes the (K-1)th encoded feature and the text feature, K is less than or equal to N, and K and N are integers greater than 1;

[0038] The target coding features include the second coding feature to the Nth coding feature.

[0039] In some embodiments, the target encoded feature further includes a first concatenated feature obtained by concatenating the second to the Nth encoded features; the first decoder includes N decoding layers; the step of calling the first decoder to decode based on the text features and the target encoded feature to obtain the target decoded feature includes:

[0040] The text features and the first concatenation features are input into the first decoding layer, and the first decoded features are output.

[0041] The (K-1)th decoded feature output by the (K-1)th decoded layer and the text feature are input into the Kth decoded layer, and the Kth decoded feature is output, until the Nth decoded layer outputs the Nth decoded feature;

[0042] The target decoding features include the second to the Nth decoding features.

[0043] In some embodiments, the attention network includes N-1 attention blocks, and the target decoding feature further includes a second concatenated feature obtained by concatenating the second to Nth decoding features; the step of invoking the attention network to extract the first reference style feature based on the target encoding feature and the target decoding feature includes:

[0044] The second concatenation feature, the second decoding feature, and the Nth encoding feature are input into the first attention block, and the first style feature is output.

[0045] The M-1th style feature, M+1th decoding feature, and N-M+1th encoding feature output by the M-1th attention block are input into the Mth attention block, and the Mth style feature is output. This process continues until the N-1th attention block outputs the N-1th style feature. The first reference style feature is determined based on the N-1th style feature. Here, M is an integer greater than 1 and less than N.

[0046] In some embodiments, each attention block includes a cross-attention layer and a self-attention layer; the step of inputting the second concatenation feature, the second decoding feature, and the Nth encoding feature into the first attention block and outputting the first style feature includes:

[0047] The second splicing feature and the second decoding feature are input into the cross-attention layer in the first attention block, and the first initial style feature is output.

[0048] The first initial style feature and the Nth encoded feature are input into the self-attention layer in the first attention block, and the first style feature is output.

[0049] The process of inputting the (M-1)th style feature, the (M+1)th decoding feature, and the (N-M+1)th encoding feature output from the (M-1)th attention block into the Mth attention block and outputting the Mth style feature includes:

[0050] The (M-1)th style feature and the (M+1)th decoding feature are input into the Mth cross-attention layer, and the Mth initial style feature is output.

[0051] The Mth initial style feature and the N-M+1th encoded feature are input into the Mth self-attention layer, and the Mth style feature is output.

[0052] In some embodiments, the stylization processing of the content to be processed based on the first reference style feature to obtain the target content includes:

[0053] The stylization processing model is invoked, and the content to be processed is stylized based on the first reference style feature to obtain the target content; wherein, the stylization processing model is used to stylize the input video or image.

[0054] In some embodiments, the training process of the visual style encoding model and the stylization processing model includes:

[0055] Based on the first sample content, the first sample reference content, the first sample description information, and the second sample content, a visual style encoding model to be trained and a stylization processing model to be trained are trained to obtain an initial visual style encoding model and an initial stylization processing model; wherein, the second sample content and the first sample content have the same content but different styles, the second sample content and the first sample reference content have the same style, the first sample reference content includes a first sample reference video or a first sample reference image, and the first sample description information is used to describe the style of the first sample reference content in text form;

[0056] While keeping the model parameters of the target network in the initial stylization processing model unchanged, the initial visual style encoding model and the initial stylization processing model are trained based on the third sample content, the second sample reference content, the second sample description information, and the fourth sample content to obtain the visual style encoding model and the stylization processing model; wherein, the fourth sample content and the third sample content have the same content but different styles, the fourth sample content and the second sample reference content have the same style, the second sample reference content includes a second sample reference video or a second sample reference image, the second sample description information is used to describe the style of the second sample reference content in text form, the target network in the initial stylization processing model refers to any network other than the cross-attention network, and the input of the cross-attention network is the first reference style feature.

[0057] In some embodiments, the method further includes:

[0058] Select any preset style option from multiple preset style options, and determine the preset style feature corresponding to the selected preset style option as the second reference style feature; wherein, each preset style option corresponds to a preset style feature, and the preset style feature is a style feature extracted based on preset content, and the preset content includes preset images or preset videos;

[0059] Based on the second reference style feature, the content to be processed is stylized to obtain the second target content, and the style of the second target content is the same as the style represented by the second reference style feature.

[0060] According to a second aspect of the present disclosure, a processing apparatus is provided, the apparatus comprising:

[0061] The first acquisition module is configured to acquire content to be processed, which includes images to be processed or videos containing multiple frames of images to be processed.

[0062] The second acquisition module is configured to acquire reference content corresponding to the content to be processed, wherein the reference content includes a reference image or a reference video containing multiple frames of reference images;

[0063] The description information generation module is configured to generate target description information based on the reference content, wherein the target description information is used to describe the style of the reference content in text form;

[0064] The feature extraction module is configured to perform feature extraction based on the target description information and the reference content to obtain a first reference style feature; the first reference style feature is used to characterize the style of the reference content.

[0065] The stylization processing module is configured to perform stylization processing on the content to be processed based on the first reference style feature to obtain a first target content, wherein the style of the first target content is the same as the style of the reference content.

[0066] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0067] processor;

[0068] Memory used to store processor-executable instructions;

[0069] The processor is configured to perform the processing method as described in the first aspect of the embodiments of this disclosure.

[0070] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the processing method as described in the first aspect of the present disclosure.

[0071] The method described above, as disclosed in this invention, has the following beneficial effects:

[0072] The method provided in this disclosure, for the content to be processed, does not require user input of prompts. Instead, it directly obtains the reference content corresponding to the image to be processed, directly generates target description information to describe the style of the reference content based on the reference content, extracts a first reference style feature based on the target description information and the reference content, and performs stylization processing on the content to be processed based on the first reference style feature to obtain the first target content. Compared with obtaining prompts with subjective user meaning, directly obtaining the reference content can adaptively extract reference style features to represent the style desired by the user, thereby achieving stylization processing of the content to be processed. This is equivalent to transferring the style of the reference content to the content to be processed, which allows the content to be processed to better replicate the style of the reference content, improves the quality of the generated target content and the user experience, and lowers the user threshold by eliminating the need for user input of prompts, making it convenient for more users to use.

[0073] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0074] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0075] Figure 1 This is a flowchart illustrating a processing method according to an exemplary embodiment;

[0076] Figure 2 This is a flowchart illustrating a processing method according to an exemplary embodiment;

[0077] Figure 3 This is a flowchart illustrating a feature extraction process according to an exemplary embodiment;

[0078] Figure 4 This is a schematic diagram illustrating a visual style coding model according to an exemplary embodiment;

[0079] Figure 5 This is a schematic diagram illustrating a processing procedure according to an exemplary embodiment;

[0080] Figure 6 This is a flowchart illustrating a model training process according to an exemplary embodiment;

[0081] Figure 7 This is a flowchart illustrating a processing method according to an exemplary embodiment;

[0082] Figure 8 This is a block diagram illustrating a processing apparatus according to an exemplary embodiment;

[0083] Figure 9 This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0084] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0085] The solutions in related technologies rely on user text descriptions, that is, users need to input prompts. For example, when a user wants to obtain an image in the style of an oil painting, they need to use prompts to describe what the desired oil painting style is like. However, for users who are not familiar with oil painting, it is difficult to describe it clearly, which places high demands on users. Moreover, the prompts affect the quality of the generated image or video. If the input words are inaccurate or incorrect, the quality of the final generated image or video will be very poor, affecting the user experience.

[0086] This disclosure provides a processing method that involves: acquiring content to be processed, including an image to be processed or a video containing multiple frames of the image to be processed; acquiring reference content corresponding to the content to be processed, including a reference image or a video containing multiple frames of the reference image; generating target description information based on the reference content, the target description information being used to describe the style of the reference content in text form; performing feature extraction based on the target description information and the reference content to obtain a first reference style feature; the first reference style feature being used to characterize the style of the reference content; and stylizing the content to be processed based on the first reference style feature to obtain a first target content, the style of the first target content being the same as the style of the reference content. Compared to acquiring prompts with subjective user meaning, directly acquiring reference content allows for adaptive extraction of reference style features to characterize the style desired by the user, thus achieving stylization of the content to be processed. This is equivalent to transferring the style of the reference content to the content to be processed, enabling the content to be processed to better replicate the style of the reference content, improving the quality of the generated target content and the user experience. Furthermore, users do not need to input prompts, lowering the barrier to entry and making it easier for more users to use the platform.

[0087] The method provided in this disclosure can be applied to various applications. For example, it can be applied to camera or photo album applications built into electronic devices, enabling one-click style conversion of captured images or videos. Alternatively, it can be applied to third-party applications, providing corresponding APIs (Application Programming Interfaces) to enable related editing functions on the electronic device, eliminating the need for the third-party application to implement these functions on a cloud service. This method can also be applied to various application scenarios; for example, it can be used to generate anime characters, virtual avatars, etc.

[0088] The method provided in this disclosure is executed by an electronic device, which may be a mobile phone, tablet computer, laptop computer, desktop computer, wearable device, smart home device, vehicle terminal, server, or other device.

[0089] Figure 1 This is a flowchart illustrating a processing method according to an exemplary embodiment, executed by an electronic device, see [link to flowchart]. Figure 1 The method includes the following steps:

[0090] Step S101: Obtain the content to be processed, which includes images to be processed or videos containing multiple frames of images to be processed.

[0091] The image to be processed can be an image captured by the user in real time or an image already existing locally on the electronic device. Similarly, the video to be processed can be a video captured by the user in real time or a video already existing locally on the electronic device. The processing of the images to be processed contained in the images and videos to be processed is similar in subsequent processing. This disclosure does not limit the shooting method or content of the images and videos to be processed.

[0092] Retrieve content to be processed, including: in response to the upload operation of content to be processed, obtain the content to be processed.

[0093] Step S102: Obtain the reference content corresponding to the content to be processed. The reference content includes reference images or reference videos containing multiple frames of reference images.

[0094] The style of the reference content refers to the style the user wants the content to be processed to be converted into. The reference content can be a reference image or a reference video. It should be noted that when the content to be processed is an image, the reference content can be either a reference image or a reference video; similarly, when the content to be processed is a video, the reference content can be either a reference image or a reference video. This embodiment does not limit the shooting method or content of the reference image and reference video, as long as the style of the reference content is the style desired by the user.

[0095] Obtaining reference content includes: responding to the upload operation of reference content and obtaining the reference content.

[0096] Step S103: Based on the reference content, generate target description information, which is used to describe the style of the reference content in text form.

[0097] In this embodiment, the user does not need to input prompts; they can directly generate target description information based on the reference content. This target description information describes the style of the reference content in text form and may include words related to style description, such as warm colors, anime, cute, highlights, and vibrant colors. For the user, although no prompts are required, the target description information generated based on the reference content can still be used in subsequent processing without affecting the subsequent stylization process.

[0098] Step S104: Based on the target description information and reference content, feature extraction is performed to obtain the first reference style feature; the first reference style feature is used to characterize the style of the reference content.

[0099] When extracting style features, considering both the target description information and the reference content simultaneously can result in a more comprehensive and accurate first reference style feature.

[0100] Step S105: Based on the first reference style features, the content to be processed is stylized to obtain the first target content, the style of the first target content is the same as the style of the reference content.

[0101] Based on the style represented by the first reference style feature, the style of the content to be processed is transformed into the style represented by the first reference style feature, thereby obtaining the first target content with the same style as the reference content.

[0102] The method provided in this disclosure, for the content to be processed, does not require user input of prompts. Instead, it directly obtains the reference content corresponding to the image to be processed, directly generates target description information to describe the style of the reference content based on the reference content, extracts a first reference style feature based on the target description information and the reference content, and performs stylization processing on the content to be processed based on the first reference style feature to obtain the first target content. Compared with obtaining prompts with subjective user meaning, directly obtaining the reference content can adaptively extract reference style features to represent the style desired by the user, thereby achieving stylization processing of the content to be processed. This is equivalent to transferring the style of the reference content to the content to be processed, which allows the content to be processed to better replicate the style of the reference content, improves the quality of the generated target content and the user experience, and lowers the user threshold by eliminating the need for user input of prompts, making it convenient for more users to use.

[0103] Figure 2 This is a flowchart illustrating a processing method according to an exemplary embodiment, executed by an electronic device, see [link to flowchart]. Figure 2 The method includes the following steps:

[0104] Step S201: Obtain the content to be processed.

[0105] Step S202: Obtain the reference content corresponding to the content to be processed.

[0106] The implementation methods for steps S201-S202 are the same as those for steps S101-S102 above, and will not be repeated here.

[0107] Step S203: Parse the reference content to obtain text description information.

[0108] The text description information is used to describe the style and content of the reference content in text form.

[0109] In some embodiments, an LLM (Large Language Model) is invoked to parse the reference content and obtain text description information. The LLM can be any type of language model, and this disclosure does not impose any limitations on it.

[0110] In one example, the LLM questions can be preset to include phrases like "What is the main content described by the current image / video?", "What is the main style of the current image / video?", and "What is the main visual style of the current image / video?". When the LLM is invoked to parse the reference content, answers to these questions are generated, which are the text descriptions. To avoid excessive text descriptions, the number of words in the answer to each question can be limited to a preset number; for example, the answer to each question cannot exceed 30 words.

[0111] Optionally, when the reference content is a reference video, the multiple frames of images to be processed contained in the reference video can be parsed separately to obtain the text description information corresponding to each frame of images to be processed. When parsing each frame of images to be processed, the parsing can be performed based solely on this frame of images to be processed, or it can be performed based on this frame of images to be processed and the previous and next frames of images to be processed, that is, the context of each frame of images to be processed is considered during the parsing.

[0112] Step S204: Preprocess the text description information to obtain the target description information.

[0113] Since the text description information contains words that describe the style and content of the reference content, and the content of the reference content is irrelevant to the subsequent stylization process, while the presence of content-related words will affect the subsequent stylization process, preprocessing is performed based on the text description information to obtain the target description information, which contains words that describe the style of the reference content.

[0114] In some embodiments, after parsing the reference content, text description information is displayed, allowing the user to edit the text description information. The user can choose whether to edit the text description information. If the user edits the text description information, the electronic device responds to the editing operation, obtains the edited text description information, filters words related to the style description from the edited text description information, and determines the filtered words as the target description information.

[0115] It should be noted that the above implementation method is only illustrated by first editing the text description information and then filtering words related to the style description. In another embodiment, the text description information does not need to be edited, but words related to the style description can be directly filtered from the text description information.

[0116] Step S205: Extract features from the target description information to obtain the text features of the target description information.

[0117] Text features are used to characterize the style expressed in the target description information. This disclosure does not limit the method of extracting text features.

[0118] Step S206: Encode based on text features and reference content to obtain target encoded features.

[0119] In some embodiments, when the reference content is a reference image, text features and reference content are encoded to obtain the first encoded feature. For the Kth encoding, encoding is performed based on the (K-1)th encoded feature or the combined features corresponding to the (K-1)th encoded feature to obtain the Kth encoded feature, and so on until the Nth encoded feature is obtained after N encodings. The combined features corresponding to the (K-1)th encoded feature include the (K-1)th encoded feature and text features, where K is less than or equal to N, and K and N are integers greater than 1. The target encoded features include the second to Nth encoded features. Through multiple encodings, multiple levels of encoded features are obtained.

[0120] Since text features are extracted from the target description information based on the reference content, the accuracy of the target description information cannot be guaranteed. Therefore, in the above encoding process, except for the first encoding, subsequent encoding processes can either continue encoding based solely on the encoded features obtained from the previous encoding, or they can be based on both the encoded features obtained from the previous encoding and the text features. For example, if N encoding steps are required, text features can be used in odd-numbered encoding steps and not used in even-numbered encoding steps, thereby reducing the influence of text features on the encoding result.

[0121] In some embodiments, after N encoding operations to obtain the second to Nth encoding features, the second to Nth encoding features can be concatenated to obtain a first concatenated feature. The target encoding feature also includes this first concatenated feature.

[0122] In some embodiments, when the reference content includes a reference video, encoding is performed based on text features and the reference content to obtain target encoded features. This includes: encoding based on text features and multiple reference images in the reference video to obtain encoded features corresponding to each reference image; and concatenating the encoded features corresponding to the multiple reference images according to the channel dimension to obtain the target encoded features. That is, for N encoding operations, the above implementation method is used to obtain N encoded features corresponding to each reference image. Then, for multiple reference images, the encoded features obtained from the Kth encoding of the multiple reference images are concatenated according to the channel dimension to obtain the Kth encoded feature corresponding to the multiple reference images. The target encoded features include the 2nd to Kth encoded features corresponding to the multiple reference images.

[0123] Optionally, the second to the Kth coding features corresponding to the multi-frame reference images are concatenated to obtain the first concatenated features corresponding to the multi-frame reference images. The target coding features also include the first concatenated features corresponding to the multi-frame reference images.

[0124] The reference images that need to be encoded in the reference video can be all the reference images in the reference video, or only a portion of the reference images in the reference video. For example, the reference images that need to be encoded can be multiple reference images selected at preset frame intervals.

[0125] Step S207: Decode the target based on the text features and the target encoding features to obtain the target decoded features.

[0126] In some embodiments, when the target encoding feature includes a first concatenation feature, decoding is performed based on the text feature and the target encoding feature to obtain the target decoding feature, including: decoding the text feature and the first concatenation feature to obtain the first decoding feature; for the Kth decoding, decoding is performed based on the (K-1)th decoding feature and the text feature to obtain the Kth decoding feature, until the Nth decoding feature is obtained after N decodings; wherein, the target decoding feature includes the second to the Nth decoding features.

[0127] In some embodiments, after N decodings to obtain the second to Nth decoding features, the second to Nth decoding features can be concatenated to obtain a second concatenated feature, and the target decoding feature includes the second concatenated feature.

[0128] Step S208: Extract the first reference style features based on the target encoding features and the target decoding features.

[0129] In some embodiments, when the target coding features include the second to Nth coding features and the first concatenation feature, and the target decoding features include the second to Nth decoding features and the second concatenation feature, the first reference style feature is extracted based on the target coding features and the target decoding features. This includes: extracting the first style feature based on the second concatenation feature, the second decoding feature, and the Nth coding feature; for the Mth style feature extraction, extracting the Mth style feature based on the extracted (M-1)th style feature, the (M+1)th decoding feature, and the (N-M+1)th coding feature, until the (N-1)th style feature is extracted, and determining the first reference style feature based on the (N-1)th style feature; where M is an integer greater than 1 and less than N.

[0130] Optionally, the (N-1)th style feature can be determined as the first reference style feature, or the (N-1)th style feature can be dimensionally mapped to obtain the first reference style feature. The dimensional mapping includes reducing or increasing the dimensionality.

[0131] Step S209: Based on the first reference style features, the content to be processed is stylized to obtain the first target content.

[0132] Using the first reference style feature as a reference, the content to be processed is stylized to obtain the first target content. When the content to be processed is an image, the first target content is the first target image; when the content to be processed is a video, the first target content is the first target video.

[0133] In some embodiments, the content to be processed is encoded to obtain the features to be processed of the content to be processed; stylization processing is performed based on the first reference style features and the features to be processed to obtain the target features; and the target features are decoded to obtain the first target content.

[0134] The method provided in this disclosure, for the content to be processed, does not require user input of prompts. Instead, it directly obtains the reference content corresponding to the image to be processed, directly generates target description information to describe the style of the reference content based on the reference content, extracts a first reference style feature based on the target description information and the reference content, and performs stylization processing on the content to be processed based on the first reference style feature to obtain the first target content. Compared with obtaining prompts with subjective user meaning, directly obtaining the reference content can adaptively extract reference style features to represent the style desired by the user, thereby achieving stylization processing of the content to be processed. This is equivalent to transferring the style of the reference content to the content to be processed, which allows the content to be processed to better replicate the style of the reference content, improves the quality of the generated target content and the user experience, and lowers the user threshold by eliminating the need for user input of prompts, making it convenient for more users to use.

[0135] In some embodiments, a visual style coding model is invoked to extract features from the target description information and reference content to obtain first reference style features. The visual style coding model is used to extract style features from the input content. The visual style coding model includes a text encoder, a first encoder, a first decoder, and an attention network. The text encoder is connected to both the first encoder and the first decoder; the first encoder is connected to the first decoder; and the first encoder and the first decoder are connected to the attention network. The following describes... Figure 3 The illustrated embodiment explains the process of extracting first reference style features by calling a visual style coding model:

[0136] Figure 3 This is a flowchart illustrating a feature extraction process according to an exemplary embodiment, performed by an electronic device, see [link to flowchart]. Figure 3 The method includes the following steps:

[0137] Step S301: Call the text encoder to extract features from the target description information to obtain the text features of the target description information.

[0138] The text encoder can be a T5 encoder or other encoders used to extract text features. If a T5 encoder is used, its simplicity allows for integration on the electronic device side, enabling rapid mapping of target description information to text features. See the example below. Figure 4 The diagram shows a visual style coding model that uses a T5 encoder.

[0139] Step S302: Invoke the first encoder to encode based on text features and reference content to obtain target encoded features.

[0140] The first encoder is used to extract features from the input content.

[0141] In some embodiments, the first encoder includes N encoding layers. Calling the first encoder to encode based on text features and reference content to obtain target encoded features includes: inputting text features and reference content into the first encoding layer and outputting the first encoded feature; inputting the (K-1)th encoded feature output from the (K-1)th encoding layer or the combined feature corresponding to the (K-1)th encoded feature into the Kth encoding layer and outputting the Kth encoded feature, until the Nth encoding layer outputs the Nth encoded feature; wherein the combined feature corresponding to the (K-1)th encoded feature includes the (K-1)th encoded feature and the text feature, K is less than or equal to N, and K and N are integers greater than 1; wherein the target encoded feature includes the second to Nth encoded features.

[0142] See one example. Figure 4 The diagram illustrates a visual style coding model. In this model, taking the first encoder comprising five encoding layers and the video to be processed as an example, the process begins... Figure 4 It can be seen that the second to fourth encoded features of the video to be processed are obtained by concatenating the encoded features of multiple images to be processed, and then the second to fifth encoded features are concatenated to obtain the first concatenated feature (Cat-fea-En). From Figure 4 It can also be seen that text features were used as input to the 1st, 3rd, and 5th encoding layers, but not as input to the 2nd and 4th encoding layers.

[0143] Step S303: Call the first decoder to decode based on text features and target encoding features to obtain target decoded features.

[0144] In some embodiments, the first decoder includes N decoding layers; calling the first decoder to decode based on text features and target encoded features to obtain target decoded features includes: inputting text features and a first concatenation feature into the first decoding layer and outputting the first decoded feature; inputting the (K-1)th decoded feature output by the (K-1)th decoding layer and the text features into the Kth decoding layer and outputting the Kth decoded feature, until the Nth decoding layer outputs the Nth decoded feature; wherein, the target decoded feature includes the second to the Nth decoded features.

[0145] See one example. Figure 4 The diagram shows a visual style coding model. In this visual style encoder, taking the first decoder as an example, it includes five decoding layers (De-Trans). Figure 4 As can be seen, the first concatenated feature is input into the first decoding layer, the text features are input into each decoding layer, and then the second to fifth decoding features are concatenated to obtain the second concatenated feature (Cat-fea-Dn).

[0146] Step S304: Invoke the attention network to extract the first reference style features based on the target encoding features and target decoding features.

[0147] In some embodiments, the attention network includes N-1 attention blocks, and the target decoding feature further includes a second concatenated feature obtained by concatenating the second to Nth decoding features; the attention network is invoked, and a first reference style feature is extracted based on the target encoding feature and the target decoding feature, including: inputting the second concatenated feature, the second decoding feature, and the Nth encoding feature into the first attention block, and outputting the first style feature; inputting the (M-1)th style feature, the (M+1)th decoding feature, and the (N-M+1)th encoding feature output by the (M-1)th attention block into the Mth attention block, and outputting the Mth style feature, until the (N-1)th attention block outputs the (N-1)th style feature, and determining the first reference style feature based on the (N-1)th style feature; wherein, M is an integer greater than 1 and less than N.

[0148] In some embodiments, each attention block includes a cross-attention layer and a self-attention layer. Inputting the second concatenated feature, the second decoded feature, and the Nth encoded feature into the first attention block and outputting the first style feature includes: inputting the second concatenated feature and the second decoded feature into the cross-attention layer in the first attention block and outputting the first initial style feature; inputting the first initial style feature and the Nth encoded feature into the self-attention layer in the first attention block and outputting the first style feature. Inputting the (M-1)th style feature, the (M+1)th decoded feature, and the (N-M+1)th encoded feature output from the (M-1)th attention block into the Mth attention block and outputting the Mth style feature includes: inputting the (M-1)th style feature and the (M+1)th decoded feature into the Mth cross-attention layer and outputting the Mth initial style feature; inputting the Mth initial style feature and the (N-M+1)th encoded feature into the Mth self-attention layer and outputting the Mth style feature.

[0149] Optionally, the second concatenated feature is used as the query value Q of the first cross-attention layer, and the second decoded feature is used as K and V; then the output of the first cross-attention layer is used as the query value Q of the second encoded feature, and the output of the first cross-attention layer is used as K and V. Subsequent cross-attention layers and self-attention layers follow the same principle. This design promotes hierarchical knowledge transfer and yields more accurate style features.

[0150] It should be noted that the visual style coding model may include one or more attention networks. When multiple attention networks are included, the structures of the multiple attention networks are the same. The difference is that the input of the first attention network is the second to the Nth encoded features, the second to the Nth decoded features, and the second concatenation feature. Starting from the second attention network, the input is the second to the Nth encoded features, the second to the Nth decoded features, and the output of the previous attention network. The processing process inside the attention network is similar to the processing process of the attention network in the above embodiment, and will not be repeated here.

[0151] See one example. Figure 4 The diagram illustrates a visual style encoding model. Taking a visual style encoder comprising three attention networks (CS-Trans blocks), each attention network containing four attention blocks, and each attention block including a cross-attention layer and a self-attention layer, as an example, the processing of the first attention network is described in detail. The processing of the second and third attention networks is not described in detail. Figure 4 (The text does not show the use of encoded and decoded features as input). From Figure 4As can be seen, the Cat-fea-Dn and the second decoded feature are input into the first cross-attention layer. Then, the first initial style feature and the fifth encoded feature output from the first cross-attention layer are input into the first self-attention layer, which outputs the first style feature. This process is repeated until the fourth style feature output from the fourth self-attention layer is obtained. Finally, the fourth style feature is input into the next attention network.

[0152] In some embodiments, the visual style encoding model further includes a CNN (Convolutional Neural Network) layer. The CNN layer is invoked to perform dimensionality mapping on the (N-1)th style feature output by the attention network to obtain a first reference style feature. Optionally, this CNN layer can be one or more. Figure 4 Taking two CNN layers as an example, the output of the last attention network is passed through two CNN layers to obtain the first reference style feature (StyleEmbedding).

[0153] In some embodiments, after invoking the visual style encoding model, a stylization processing model is then invoked to stylize the content to be processed based on the first reference style features, thereby obtaining the target content. The stylization processing model is used to stylize the input video or image.

[0154] In some embodiments, the stylization processing model includes a second encoder, a feature processing network, and a second decoder. The first encoder is invoked to encode the content to be processed, obtaining the features to be processed; the feature processing network is invoked to stylize the features to be processed based on a first reference style feature, obtaining the target feature; the second decoder is invoked to decode the target feature, obtaining the first target content. The second encoder and the second decoder can be the encoder and decoder (VAE Encoder and VAE Decoder) in a variational autoencoder (VAE) with reduced diffusion dimensions.

[0155] Optionally, the feature processing network includes a cross-attention network, which comprises multiple cross-attention layers. The input to the cross-attention network is a first reference style feature. For example, the first reference style feature and the feature to be processed are used as the input to the first cross-attention layer. Then, for subsequent cross-attention layers, the output of the previous cross-attention layer and the first reference style feature are used as the input, until the output of the last cross-attention layer is obtained. The output of the last cross-attention layer is then used as the target feature.

[0156] See one example. Figure 5The schematic diagram of the processing procedure shows that, firstly, the reference content (image / video) is input into the LLM, which outputs text description information. Then, the text description information is preprocessed to obtain the target description information. The target description information and the reference content are used as input to the visual style encoding model, which outputs the first reference style feature. At the same time, the content to be processed is input into the second encoder in the Latent Diffusion Model. Then, the features to be processed output by the second encoder and the first reference style feature are input into the cross-attention network to output the target feature. Finally, the target feature is input into the second decoder to output the first target content.

[0157] Figure 6 This is a flowchart illustrating a model training process according to an exemplary embodiment, performed by an electronic device, see [link to flowchart]. Figure 6 This includes the following steps:

[0158] Step S601: Based on the content of the first sample, the reference content of the first sample, the descriptive information of the first sample, and the content of the second sample, train the visual style encoding model to be trained and the stylization processing model to be trained to obtain the initial visual style encoding model and the initial stylization processing model.

[0159] The second sample content is the same as the first sample content but different in style. The second sample content is the same in style as the first sample reference content. The first sample reference content includes a first sample reference video or a first sample reference image. The first sample description information is used to describe the style of the first sample reference content in text form.

[0160] Step S602: While keeping the model parameters of the target network in the initial stylization processing model unchanged, train the initial visual style encoding model and the initial stylization processing model based on the content of the third sample, the reference content of the second sample, the descriptive information of the second sample, and the content of the fourth sample to obtain the visual style encoding model and the stylization processing model.

[0161] Among them, the fourth sample content and the third sample content have the same content but different styles. The fourth sample content and the second sample reference content have the same style. The second sample reference content includes the second sample reference video or the second sample reference image. The second sample description information is used to describe the style of the second sample reference content in text form. The target network in the initial stylization processing model refers to the network other than the cross attention network. The input of the cross attention network is the first reference style feature.

[0162] It should be noted that the model processing during training is similar to the model processing when using the model as described above, and will not be repeated here.

[0163] Another point to note is that different styles can be processed using the same visual style encoding model and stylization model, or the visual style encoding model and stylization model corresponding to each style can be trained separately to achieve more accurate stylization. For example, for 3D style and ordinary style (2D style), the corresponding visual style encoding model and stylization model can be trained separately.

[0164] Another point to note is that, in another embodiment, the stylization processing model and the visual style encoding model can be trained separately.

[0165] In this embodiment of the disclosure, keeping the model parameters of a portion of the network in the initial stylization processing model unchanged can reduce the amount of training and improve training efficiency.

[0166] Furthermore, compared with the stylization models used in related technologies for image or video stylization processing, the stylization processing model and visual style coding model provided in this disclosure are more accurate, easier for users to use, and more universal. In addition, the model structure is simple and easy to deploy on the terminal side.

[0167] It should be noted that the above embodiments are based on reference content, and the content to be processed is stylized to obtain the first target content. In another embodiment, a similar implementation method to the above-described generation of the first reference style feature based on reference content can be adopted. First, a preset style feature is generated based on preset content. Then, when stylizing the content to be processed, multiple preset style options are provided to the user, allowing the user to directly select the preset style feature as the reference style feature to process the content to be processed, without the need to generate the reference style feature in real time.

[0168] Figure 7 This is a flowchart illustrating a processing method according to an exemplary embodiment, executed by an electronic device, see [link to flowchart]. Figure 7 The method includes the following steps:

[0169] Step S701: Obtain the content to be processed.

[0170] Step S702: Select any preset style option from multiple preset style options, and determine the preset style feature corresponding to the selected preset style option as the second reference style feature.

[0171] Each preset style option corresponds to a preset style feature, which is a style feature extracted based on preset content, including preset images or preset videos.

[0172] The style represented by the second reference style feature may be the same as or different from the style represented by the first reference style feature in the above embodiments. This disclosure does not impose any restrictions on this.

[0173] In some embodiments, a style selection interface is displayed, which includes multiple preset style options and descriptive information for each preset style option. The descriptive information describes the style corresponding to the preset style option. In response to the selection operation of any preset style option, the preset style feature corresponding to the selected preset style option is determined as a second reference style feature.

[0174] Step S703: Based on the second reference style features, the content to be processed is stylized to obtain the second target content.

[0175] The style of the second target content is the same as the style represented by the second reference style feature.

[0176] The implementation method for stylizing the content to be processed based on the second reference style feature is similar to the implementation method for stylizing the content to be processed based on the first reference style feature in the above embodiments, and will not be described again here.

[0177] The method provided in this disclosure can offer users selectable preset style options, allowing them to choose the style they want without requiring them to input prompts, thus improving the quality of the generated target content and the user experience.

[0178] Figure 8 This is a block diagram illustrating a processing apparatus according to an exemplary embodiment, configured in an electronic device, see [link to relevant documentation]. Figure 8 The device includes:

[0179] The first acquisition module 801 is configured to acquire content to be processed, which includes images to be processed or videos containing multiple frames of images to be processed.

[0180] The second acquisition module 802 is configured to acquire reference content corresponding to the content to be processed, the reference content including reference images or reference videos containing multiple frames of reference images;

[0181] The description information generation module 803 is configured to generate target description information based on reference content. The target description information is used to describe the style of the reference content in text form.

[0182] The feature extraction module 804 is configured to perform feature extraction based on target description information and reference content to obtain a first reference style feature; the first reference style feature is used to characterize the style of the reference content.

[0183] The stylization processing module 805 is configured to stylize the content to be processed based on the first reference style features to obtain the first target content, the style of which is the same as that of the reference content.

[0184] In some embodiments, the description information generation module 803 is configured to:

[0185] The reference content is parsed to obtain text description information; the text description information is used to describe the style and content of the reference content in text form;

[0186] Preprocessing is performed based on the text description information to obtain the target description information.

[0187] In some embodiments, the description information generation module 803 is configured to:

[0188] In response to editing operations on text description information, the edited text description information is obtained;

[0189] Filter words related to style description from the edited text description information, and determine the filtered words as the target description information.

[0190] In some embodiments, the feature extraction module 804 is configured to:

[0191] Feature extraction is performed on the target description information to obtain the text features of the target description information;

[0192] Encoding is performed based on text features and reference content to obtain target encoded features;

[0193] Decoding is performed based on text features and target encoded features to obtain target decoded features;

[0194] Based on the target encoding features and target decoding features, the first reference style features are extracted.

[0195] In some embodiments, when the reference content includes a reference video, the feature extraction module 804 is configured to:

[0196] Encoding is performed based on text features and multiple reference images from the reference video to obtain the encoded features corresponding to each reference image;

[0197] The target encoded features are obtained by concatenating the encoded features corresponding to multiple reference images according to the channel dimension.

[0198] In some embodiments, the stylization processing module 805 is configured to:

[0199] The content to be processed is encoded to obtain the processing features of the content to be processed;

[0200] Based on the first reference style features, the features to be processed are stylized to obtain the target features;

[0201] Decode the target features to obtain the first target content.

[0202] In some embodiments, the feature extraction module 804 is configured to:

[0203] The visual style coding model is invoked to extract features from the target description information and reference content to obtain the first reference style features; wherein, the visual style coding model is used to extract the style features of the input content.

[0204] In some embodiments, the visual style coding model includes a text encoder, a first encoder, a first decoder, and an attention network; the feature extraction module 804 is configured to:

[0205] The text encoder is invoked to extract features from the target description information, thereby obtaining the text features of the target description information.

[0206] The first encoder is invoked to encode based on text features and reference content to obtain the target encoded features;

[0207] The first decoder is invoked to decode based on text features and target encoded features to obtain target decoded features;

[0208] An attention network is invoked to extract the first reference style features based on the target encoding and target decoding features.

[0209] In some embodiments, the first encoder includes N encoding layers; the feature extraction module 804 is configured to:

[0210] Input the text features and reference content into the first encoding layer, and output the first encoded features;

[0211] The (K-1)th encoded feature output from the (K-1)th encoded layer or the combined feature corresponding to the (K-1)th encoded feature is input into the Kth encoded layer, and the Kth encoded feature is output, until the Nth encoded layer outputs the Nth encoded feature; wherein, the combined feature corresponding to the (K-1)th encoded feature includes the (K-1)th encoded feature and the text feature, K is less than or equal to N, and K and N are integers greater than 1;

[0212] The target coding features include the second coding feature to the Nth coding feature.

[0213] In some embodiments, the target coding feature further includes a first concatenated feature obtained by concatenating the second to Nth coding features; the first decoder includes N decoding layers; the feature extraction module 804 is configured to:

[0214] Input the text features and the first concatenated features into the first decoding layer, and output the first decoded features;

[0215] Input the (K-1)th decoded feature and text feature output from the (K-1)th decoded layer into the Kth decoded layer, and output the Kth decoded feature, until the Nth decoded layer outputs the Nth decoded feature;

[0216] The target decoding features include the second to the Nth decoding features.

[0217] In some embodiments, the attention network includes N-1 attention blocks, and the target decoding feature further includes a second concatenated feature obtained by concatenating the second to Nth decoding features; the feature extraction module 804 is configured to:

[0218] The second concatenated feature, the second decoded feature, and the Nth encoded feature are input into the first attention block, and the first style feature is output.

[0219] The M-1th style feature, M+1th decoding feature, and N-M+1th encoding feature output from the M-1th attention block are input into the Mth attention block, and the Mth style feature is output. This process continues until the N-1th attention block outputs the N-1th style feature. The first reference style feature is determined based on the N-1th style feature. Here, M is an integer greater than 1 and less than N.

[0220] In some embodiments, each attention block includes a cross-attention layer and a self-attention layer; the feature extraction module 804 is configured to:

[0221] The second concatenated feature and the second decoded feature are input into the cross-attention layer in the first attention block, and the first initial style feature is output.

[0222] The first initial style feature and the Nth encoded feature are input into the self-attention layer in the first attention block, and the first style feature is output.

[0223] Input the (M-1)th style feature and the (M+1)th decoded feature into the Mth cross-attention layer, and output the Mth initial style feature;

[0224] The Mth initial style feature and the N-M+1th encoded feature are input into the Mth self-attention layer, and the Mth style feature is output.

[0225] In some embodiments, the stylization processing module 805 is configured to:

[0226] The stylization processing model is invoked to perform stylization processing on the content to be processed based on the first reference style features, thereby obtaining the target content; wherein, the stylization processing model is used to perform stylization processing on the input video or image.

[0227] In some embodiments, the apparatus further includes a training module configured to:

[0228] Based on the first sample content, the first sample reference content, the first sample description information, and the second sample content, a visual style encoding model and a stylization processing model to be trained are trained to obtain an initial visual style encoding model and an initial stylization processing model. Among them, the second sample content and the first sample content have the same content but different styles, the second sample content and the first sample reference content have the same style, the first sample reference content includes a first sample reference video or a first sample reference image, and the first sample description information is used to describe the style of the first sample reference content in text form.

[0229] While keeping the model parameters of the target network in the initial stylization processing model unchanged, an initial visual style encoding model and an initial stylization processing model are trained based on the third sample content, the second sample reference content, the second sample description information, and the fourth sample content, resulting in the visual style encoding model and the stylization processing model. The fourth sample content and the third sample content have the same content but different styles. The fourth sample content and the second sample reference content have the same style. The second sample reference content includes a second sample reference video or a second sample reference image. The second sample description information is used to describe the style of the second sample reference content in text form. The target network in the initial stylization processing model refers to any network other than the cross-attention network, and the input to the cross-attention network is the first reference style feature.

[0230] In some embodiments, the apparatus further includes:

[0231] The style selection module is configured to select any preset style option from multiple preset style options, and determine the preset style feature corresponding to the selected preset style option as the second reference style feature; wherein, each preset style option corresponds to a preset style feature, and the preset style feature is a style feature extracted based on preset content, which includes preset images or preset videos;

[0232] The stylization processing module 805 is also configured to stylize the content to be processed based on the second reference style feature to obtain the second target content, the style of the second target content being the same as the style represented by the second reference style feature.

[0233] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0234] This disclosure also provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the processing method described in the above embodiments.

[0235] Figure 9 This is a block diagram of an electronic device 900 according to an exemplary embodiment.

[0236] Reference Figure 9 The electronic device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0237] Processing component 902 typically controls the overall operation of electronic device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0238] Memory 904 is configured to store various types of data to support the operation of electronic device 900. Examples of this data include instructions for any application or method operating on electronic device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0239] Power supply component 906 provides power to various components of electronic device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 900.

[0240] Multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0241] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when electronic device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0242] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0243] Sensor assembly 914 includes one or more sensors for providing state assessments of various aspects of electronic device 900. For example, sensor assembly 914 can detect the on / off state of electronic device 900, the relative positioning of components such as the display and keypad of electronic device 900, changes in position of electronic device 900 or a component of electronic device 900, the presence or absence of user contact with electronic device 900, orientation or acceleration / deceleration of electronic device 900, and temperature changes of electronic device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0244] Communication component 916 is configured to facilitate wired or wireless communication between electronic device 900 and other devices. Electronic device 900 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0245] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0246] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of an electronic device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0247] This disclosure also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to perform the processing methods described in the above embodiments.

[0248] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0249] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A processing method, characterized in that, The method includes: Obtain the content to be processed, which includes images to be processed or videos containing multiple frames of images to be processed; Obtain reference content corresponding to the content to be processed, wherein the reference content includes reference images or reference videos containing multiple frames of reference images; Based on the reference content, target description information is generated, which is used to describe the style of the reference content in text form; Based on the target description information and the reference content, feature extraction is performed to obtain a first reference style feature; the first reference style feature is used to characterize the style of the reference content. Based on the first reference style feature, the content to be processed is stylized to obtain the first target content, the style of the first target content being the same as the style of the reference content.

2. The processing method according to claim 1, characterized in that, The step of generating target description information based on the reference content includes: The reference content is parsed to obtain text description information; the text description information is used to describe the style and content of the reference content in text form; The target description information is obtained by preprocessing the text description information.

3. The processing method according to claim 2, characterized in that, The preprocessing based on the text description information to obtain the target description information includes: In response to the editing operation on the text description information, the edited text description information is obtained; From the edited text description information, words related to the style description are filtered out, and the filtered words are determined as the target description information.

4. The processing method according to claim 1, characterized in that, The step of extracting features based on the target description information and the reference content to obtain the first reference style features includes: Feature extraction is performed on the target description information to obtain the text features of the target description information; Based on the text features and the reference content, the target encoding features are obtained; Decoding is performed based on the text features and the target encoding features to obtain the target decoded features; Based on the target encoding features and the target decoding features, the first reference style features are extracted.

5. The processing method according to claim 4, characterized in that, When the reference content includes a reference video, the encoding based on the text features and the reference content to obtain target encoding features includes: Encoding is performed based on the text features and multiple reference images in the reference video to obtain the encoded features corresponding to each reference image; The target coding features are obtained by concatenating the coding features corresponding to the multiple reference images according to the channel dimension.

6. The processing method according to claim 1, characterized in that, The step of stylizing the content to be processed based on the first reference style feature to obtain the first target content includes: The content to be processed is encoded to obtain the processing features of the content to be processed; Based on the first reference style feature, the feature to be processed is stylized to obtain the target feature; The target features are decoded to obtain the first target content.

7. The processing method according to claim 1, characterized in that, The step of extracting features based on the target description information and the reference content to obtain the first reference style features includes: A visual style coding model is invoked to extract features from the target description information and the reference content to obtain the first reference style features; wherein, the visual style coding model is used to extract style features from the input content.

8. The processing method according to claim 7, characterized in that, The visual style encoding model includes a text encoder, a first encoder, a first decoder, and an attention network; the step of invoking the visual style encoding model to extract features from the target description information and the reference content to obtain the first reference style features includes: The text encoder is invoked to extract features from the target description information to obtain the text features of the target description information. The first encoder is invoked to encode based on the text features and the reference content to obtain the target encoded features; The first decoder is invoked to decode based on the text features and the target encoded features to obtain the target decoded features; The attention network is invoked to extract the first reference style features based on the target encoding features and the target decoding features.

9. The processing method according to claim 8, characterized in that, The first encoder includes N encoding layers; the step of calling the first encoder to encode based on the text features and the reference content to obtain target encoded features includes: The text features and the reference content are input into the first encoding layer, and the first encoded features are output. The (K-1)th encoded feature output from the (K-1)th encoding layer, or the combined feature corresponding to the (K-1)th encoded feature, is input into the Kth encoding layer, and the Kth encoded feature is output, until the Nth encoding layer outputs the Nth encoded feature; wherein, the combined feature corresponding to the (K-1)th encoded feature includes the (K-1)th encoded feature and the text feature, K is less than or equal to N, and K and N are integers greater than 1; The target coding features include the second coding feature to the Nth coding feature.

10. The processing method according to claim 9, characterized in that, The target encoded feature further includes a first concatenated feature obtained by concatenating the second to the Nth encoded features; the first decoder includes N decoding layers; the step of calling the first decoder to decode based on the text features and the target encoded feature to obtain the target decoded feature includes: The text features and the first concatenation features are input into the first decoding layer, and the first decoded features are output. The (K-1)th decoded feature output by the (K-1)th decoded layer and the text feature are input into the Kth decoded layer, and the Kth decoded feature is output, until the Nth decoded layer outputs the Nth decoded feature; The target decoding features include the second to the Nth decoding features.

11. The processing method according to claim 10, characterized in that, The attention network includes N-1 attention blocks, and the target decoding feature further includes a second concatenated feature obtained by concatenating the second to the Nth decoding features; the step of calling the attention network to extract the first reference style feature based on the target encoding feature and the target decoding feature includes: The second concatenation feature, the second decoding feature, and the Nth encoding feature are input into the first attention block, and the first style feature is output. The M-1th style feature, M+1th decoding feature, and N-M+1th encoding feature output by the M-1th attention block are input into the Mth attention block, and the Mth style feature is output. This process continues until the N-1th attention block outputs the N-1th style feature. The first reference style feature is determined based on the N-1th style feature. Here, M is an integer greater than 1 and less than N.

12. The processing method according to claim 11, characterized in that, Each attention block includes a cross-attention layer and a self-attention layer; the step of inputting the second concatenation feature, the second decoding feature, and the Nth encoding feature into the first attention block and outputting the first style feature includes: The second splicing feature and the second decoding feature are input into the cross-attention layer in the first attention block, and the first initial style feature is output. The first initial style feature and the Nth encoded feature are input into the self-attention layer in the first attention block, and the first style feature is output. The process of inputting the (M-1)th style feature, the (M+1)th decoding feature, and the (N-M+1)th encoding feature output from the (M-1)th attention block into the Mth attention block and outputting the Mth style feature includes: The (M-1)th style feature and the (M+1)th decoding feature are input into the Mth cross-attention layer, and the Mth initial style feature is output. The Mth initial style feature and the N-M+1th encoded feature are input into the Mth self-attention layer, and the Mth style feature is output.

13. The processing method according to claim 7, characterized in that, The step of stylizing the content to be processed based on the first reference style feature to obtain the target content includes: The stylization processing model is invoked, and the content to be processed is stylized based on the first reference style feature to obtain the target content; wherein, the stylization processing model is used to stylize the input video or image.

14. The processing method according to claim 13, characterized in that, The training process of the visual style encoding model and the stylization processing model includes: Based on the first sample content, the first sample reference content, the first sample description information, and the second sample content, a visual style encoding model to be trained and a stylization processing model to be trained are trained to obtain an initial visual style encoding model and an initial stylization processing model; wherein, the second sample content and the first sample content have the same content but different styles, the second sample content and the first sample reference content have the same style, the first sample reference content includes a first sample reference video or a first sample reference image, and the first sample description information is used to describe the style of the first sample reference content in text form; While keeping the model parameters of the target network in the initial stylization processing model unchanged, the initial visual style encoding model and the initial stylization processing model are trained based on the third sample content, the second sample reference content, the second sample description information, and the fourth sample content to obtain the visual style encoding model and the stylization processing model; wherein, the fourth sample content and the third sample content have the same content but different styles, the fourth sample content and the second sample reference content have the same style, the second sample reference content includes a second sample reference video or a second sample reference image, the second sample description information is used to describe the style of the second sample reference content in text form, the target network in the initial stylization processing model refers to any network other than the cross-attention network, and the input of the cross-attention network is the first reference style feature.

15. The processing method according to claim 1, characterized in that, The method further includes: Select any preset style option from multiple preset style options, and determine the preset style feature corresponding to the selected preset style option as the second reference style feature; wherein, each preset style option corresponds to a preset style feature, and the preset style feature is a style feature extracted based on preset content, and the preset content includes preset images or preset videos; Based on the second reference style feature, the content to be processed is stylized to obtain the second target content, and the style of the second target content is the same as the style represented by the second reference style feature.

16. A processing apparatus, characterized in that, The device includes: The first acquisition module is configured to acquire content to be processed, which includes images to be processed or videos containing multiple frames of images to be processed. The second acquisition module is configured to acquire reference content corresponding to the content to be processed, wherein the reference content includes a reference image or a reference video containing multiple frames of reference images; The description information generation module is configured to generate target description information based on the reference content, wherein the target description information is used to describe the style of the reference content in text form; The feature extraction module is configured to perform feature extraction based on the target description information and the reference content to obtain a first reference style feature; the first reference style feature is used to characterize the style of the reference content. The stylization processing module is configured to perform stylization processing on the content to be processed based on the first reference style feature to obtain a first target content, wherein the style of the first target content is the same as the style of the reference content.

17. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the processing method as described in any one of claims 1-15.

18. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the processing method as described in any one of claims 1-15.

Images