Image processing method, computer terminal and storage medium

Abstract

The invention discloses an image processing method, a computer terminal and a storage medium, and relates to the field of artificial intelligence, image processing and video editing. The method comprises the steps that an input instruction acting on an operation interface is responded, an original image and prompt information are displayed on the operation interface, and the prompt information is used for describing processing of the original image; a processing instruction acting on the operation interface is responded, a target image corresponding to the prompt information is displayed on the operation interface, the target image is obtained by utilizing a target mask to guide an image denoising model to conduct image editing on the original image and the prompt information, and the target mask is used for representing a mask meeting a preset mask screening condition. The technical problem that the image processing effect is poor is solved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence, image processing, and video editing; specifically, it relates to an image processing method, a computer terminal, and a storage medium. Background Technology

[0002] Currently, in the field of image processing, attention masks are commonly used for structural control to improve image processing quality. However, in related technologies, methods for structural control using attention masks typically apply the attention mask directly, leading to inaccurate processing results, especially artifacts and temporal discontinuities at shape edges, resulting in poor image processing performance.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an image processing method, a computer terminal, and a storage medium to at least solve the technical problem of poor image processing results.

[0005] According to one aspect of the embodiments of this application, an image processing method is provided. The method may include: responding to an input command applied to an interface, displaying an original image and prompt information on the interface, wherein the prompt information describes the processing performed on the original image; and responding to a processing command applied to the interface, displaying a target image corresponding to the prompt information on the interface, wherein the target image is obtained by image editing of the original image and prompt information using a target mask-guided image denoising model, and the target mask is used to characterize a mask that meets preset mask filtering conditions.

[0006] According to another aspect of the embodiments of this application, a video processing method is provided. The method may include: responding to an input command applied to an operation interface, displaying an original video and prompt information on the operation interface, wherein the prompt information describes the processing performed on the original video; and responding to a processing command applied to the operation interface, displaying a target video corresponding to the prompt information on the operation interface, wherein the target video is obtained by using a target mask to guide a video denoising model to perform video editing on the original video and prompt information, and the target mask is used to characterize a mask that meets preset mask filtering conditions.

[0007] According to another aspect of the embodiments of this application, an image processing method is provided. The method may include: acquiring an original image and prompt information by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter including the original image and the prompt information, the prompt information describing the processing performed on the original image; generating a target mask that meets preset mask filtering conditions; using the target mask to guide an image denoising model to perform image editing on the original image and the prompt information, generating a target image corresponding to the prompt information; and outputting the target image by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter including the target image.

[0008] According to another aspect of the embodiments of this application, a computer terminal is also provided. The computer terminal may include a memory and a processor: the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the method of any one of the above.

[0009] According to another aspect of the embodiments of this application, a processor is also provided, which is used to run a program, wherein the method of any one of the above is executed when the program is running.

[0010] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is running, the device where the storage medium is located executes any of the methods described above.

[0011] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method of any one of the above.

[0012] In this embodiment, in response to an input command applied to the operation interface, the original image and prompt information are displayed on the operation interface. The prompt information describes the processing performed on the original image. In response to a processing command applied to the operation interface, the target image corresponding to the prompt information is displayed on the operation interface. The target image is obtained by using a target mask to guide an image denoising model to edit the original image and prompt information. The target mask represents a mask that meets preset mask selection conditions. In other words, this embodiment generates a target mask that meets preset mask selection conditions. That is, the target mask is not an attention mask directly generated based on multiple attention features, but rather a mask selected from attention masks that meets preset mask selection conditions. Therefore, using this target mask to guide the image denoising model to edit the original image and prompt information avoids the inaccuracy of the processing results caused by directly using attention masks, thereby improving the image processing effect and solving the technical problem of poor image processing results.

[0013] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0015] Figure 1 This is a schematic diagram illustrating an application scenario of an image processing method according to an embodiment of this application;

[0016] Figure 2 This is a flowchart of an image processing method according to an embodiment of this application;

[0017] Figure 3 This is a flowchart of a video processing method according to an embodiment of this application;

[0018] Figure 4 This is a flowchart of another image processing method according to an embodiment of this application;

[0019] Figure 5 This is a schematic diagram of a FreeMask according to an embodiment of this application;

[0020] Figure 6 This is a schematic diagram of a qualitative comparison result of FreeMask according to an embodiment of this application;

[0021] Figure 7 This is a structural block diagram of a computing environment for an image processing method according to an embodiment of this application;

[0022] Figure 8 This is a schematic diagram of an image processing apparatus according to an embodiment of this application;

[0023] Figure 9 This is a schematic diagram of a video processing apparatus according to an embodiment of this application;

[0024] Figure 10 This is a schematic diagram of another image processing apparatus according to an embodiment of this application;

[0025] Figure 11 This is a structural block diagram of a computer terminal according to an embodiment of this application;

[0026] Figure 12 This is a block diagram of an electronic device according to an embodiment of the image processing method of this application;

[0027] Figure 13 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0031] Attention masks are binary masks obtained by binarying certain elements in the attention matrix, which represent the relevant semantics at the corresponding positions. In particular, the attention scores corresponding to text in the cross-attention matrix scores have significant image semantic information. After binarizing these elements, the attention mask can be called the cross-attention mechanism mask.

[0032] Zero-shot refers to samples that can be used for inference directly without requiring additional data or additional fine-tuning and training.

[0033] Video editing refers to adjusting the global or local pixels of a video to achieve purposes such as stylization, attribute editing, or shape editing.

[0034] The image processing method provided in the embodiments of this application can be applied to, for example, Figure 1 The application scenarios shown are not limited to these. Figure 1 This is a schematic diagram illustrating an application scenario of an image processing method according to an embodiment of this application. Figure 1 In the application scenario shown, the server 10 can be in the cloud. The server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 can include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. These client devices collectively constitute the client relative to the server. An interface for displaying the original image, prompts, and the target image can be deployed on the graphical user interface of the client device. The client device 20 can interact with the user through the graphical user interface to implement the image processing method provided in this embodiment.

[0035] In this embodiment, the operation interface is deployed on the client device 20. In step S102, in response to an input command applied to the operation interface, the original image and prompt information are displayed on the operation interface. The prompt information describes the processing performed on the original image. After displaying the original image and prompt information on the operation interface, since the client device 20 and server 10 can be connected via a network, the original image and prompt information displayed on the operation interface can be sent to server 10. After receiving the original image and prompt information, server 10 can perform the following operations on the original image and prompt information: generate a target mask that meets preset mask filtering conditions, and use the target mask to guide an image denoising model to perform image editing on the original image and prompt information to obtain the target image. After server 10 processes the original image and prompt information to generate the target image, the generated target image can be sent to client device 20, and client device 20 performs the following operation: In step S104, in response to a processing command applied to the operation interface, the target image corresponding to the prompt information is displayed on the operation interface.

[0036] Under the aforementioned operating environment, this application provides the following: Figure 2 The image processing method shown. Figure 2 This is a flowchart of an image processing method according to an embodiment of this application, such as... Figure 2 As shown, the method may include the following steps:

[0037] Step S202: In response to the input command applied to the operation interface, the original image and prompt information are displayed on the operation interface.

[0038] In the technical solution provided by step S202 of this application, in response to the input command applied to the operation interface, the original image and prompt information can be displayed on the operation interface. The operation interface can be the interface displayed on the client device, such as an interface for interaction between the user and the computer or software system, or a graphical user interface, command-line interface, etc. This is merely an example and does not impose specific limitations on the form of the operation interface.

[0039] It should be noted that input commands can be commands that control the display of the original image and prompts on the operation interface. For example, they can be commands entered by the user on the operation interface. The user can generate input commands by entering text on the keyboard, clicking buttons or options on the operation interface with the mouse, or dragging the original image and prompts to the specified area.

[0040] It should be noted that the original image can be the image to be edited displayed on the above-mentioned interface, which can be called the input image and can be represented by Z0. The prompt information can be used to describe the processing performed on the original image and can be represented by P_0.

[0041] In one optional embodiment, the user can open the operation interface deployed on the client device and locate the button or designated area for uploading the original image and prompt information. By clicking the button to upload the original image and prompt information or dragging and dropping the original image and prompt information to the designated area, an input command can be generated. Based on the generated input command, the corresponding original image and prompt information can be loaded and displayed on the operation interface.

[0042] Step S204: In response to the processing command applied to the operation interface, display the target image corresponding to the prompt information on the operation interface.

[0043] In the technical solution provided in step S204 of this application, after responding to the input command applied to the operation interface and displaying the original image and prompt information on the operation interface, responding to the processing command applied to the operation interface, the target image corresponding to the prompt information can be displayed on the operation interface. The processing command can be an instruction to process the original image and prompt information displayed on the operation interface; for example, it can be an image editing command. The user can generate the processing command by clicking on the original image and prompt information displayed on the operation interface. This is only an example and does not impose specific limitations on the processing command.

[0044] It should be noted that the target image can be an image obtained by editing the original image and the prompt information, which can be called an edited image. It can be obtained by using a target mask to guide an image denoising model to edit the original image and the prompt information.

[0045] It should be noted that the target mask can be a mask that meets the preset mask filtering conditions, and can be used with M. * By representing the target image, a target mask corresponding to the image denoising model can be determined from multiple initial masks. The image denoising model can be used to edit the original image and the prompt information to obtain the target image. The image denoising model can also be used to apply the target mask to achieve the fusion of different types of attention features.

[0046] It should be noted that the preset mask filtering conditions can be pre-set according to the actual situation, and these preset mask filtering conditions can be adjusted according to different needs.

[0047] In this embodiment, after the original image and prompt information are displayed on the operation interface, in response to the processing instructions applied to the operation interface, the original image and prompt information displayed on the operation interface can be processed. For example, the original image and prompt information displayed on the operation interface can be image edited to obtain the target image corresponding to the prompt information, and the target image can be displayed on the operation interface.

[0048] Optionally, after displaying the original image and prompts on the user interface, the original image and prompts can be sent to the server. Upon receiving the original image and prompts, the server can generate a target mask that meets preset mask filtering conditions, and use the target mask to guide an image denoising model to edit the original image and prompts to obtain the target image. Users can click on the original image and prompts displayed on the user interface to generate processing instructions. Based on the processing instructions, the target image generated by the server can be sent to the client device, where the user interface deployed on the client device will display the target image.

[0049] Through steps S202 to S204 of this application, in response to input commands applied to the operation interface, the original image and prompt information are displayed on the operation interface, wherein the prompt information describes the processing performed on the original image; in response to processing commands applied to the operation interface, the target image corresponding to the prompt information is displayed on the operation interface, wherein the target image is obtained by using a target mask to guide an image denoising model to perform image editing on the original image and prompt information, and the target mask is used to represent a mask that meets preset mask filtering conditions. That is to say, the embodiment of this application generates a target mask that meets preset mask filtering conditions. That is, the target mask is not an attention mask directly generated based on multiple attention features, but a mask that meets preset mask filtering conditions selected from attention masks. Therefore, using this target mask to guide the image denoising model to perform image editing on the original image and prompt information avoids the inaccuracy of the processing result caused by directly using attention masks, thereby improving the image processing effect and solving the technical problem of poor image processing effect.

[0050] The method described in this embodiment will be further described below.

[0051] This embodiment first introduces the method of filtering target masks based on the target layer number and target time step corresponding to the image denoising model.

[0052] As an optional implementation, in response to processing instructions applied to the user interface, the method further includes: during the processing of the original image and prompt information using an image inversion model, acquiring multiple attention features output by multiple cross-attention layers of the image inversion model at multiple time steps; generating multiple initial masks based on object information in the prompt information and multiple attention features, wherein different initial masks are used to characterize the position of object features corresponding to object information in the corresponding attention features; determining the target layer number and target time step corresponding to the image denoising model, wherein the target layer number is used to characterize the number of cross-attention layers corresponding to the mask that satisfies the preset mask selection conditions, and the target time step is used to characterize the time step corresponding to the mask that satisfies the preset mask selection conditions; and determining a target mask from the multiple initial masks based on the target layer number and target time step.

[0053] In this embodiment, after displaying the original image and prompt information on the user interface, in response to processing instructions applied to the user interface, during the processing of the original image and prompt information using the image inversion model, multiple attention features output by multiple cross-attention layers of the image inversion model at multiple time steps can be obtained. After obtaining multiple attention features, multiple initial masks can be generated based on the object information in the prompt information and the multiple attention features. Further, multiple initial masks corresponding to the image denoising model can be determined, and the initial masks corresponding to the same layer number and the same time step are summarized to obtain the target layer number and target time step corresponding to the image denoising model. After generating multiple initial masks and determining the target layer number and target time step corresponding to the image denoising model, the target mask can be determined from the multiple initial masks based on the determined target layer number and target time step corresponding to the image denoising model.

[0054] Optionally, the image inversion model can be used to invert the original image and prompt information to obtain the image information of the original image. This can be called a Digital Domain Image Manipulation (DDIM) inversion model. The image inversion model can also be used to generate a target mask corresponding to the image denoising model. Multiple initial masks can be generated based on multiple attention features of the image inversion model, and the target mask corresponding to the image denoising model can be determined from these initial masks. These multiple attention features are output by multiple cross-attention layers of the image inversion model at multiple time steps during the processing of the original image and prompt information using the image inversion model; this can be called a cross-attention map.

[0055] Optionally, the object information can be the object word information in the prompt information, which can be called object word information and can be represented by p_0. The initial mask can be a mask candidate, and different initial masks can be used to characterize the position of the object feature corresponding to the object information in the corresponding attention feature.

[0056] It should be noted that mask candidates can be used Let L represent the number of cross-interest layers in the diffusion network (U-Net), T represent the number of sampling steps, i represent the number of layers, and t represent the time step. Let M be referred to as the mask candidate.

[0057] It should be noted that the target layer number can be the minimum cost layer corresponding to the image denoising model, or it can be used to characterize the number of cross-attention layers corresponding to a mask that satisfies the preset mask selection conditions. This can be represented by l. * This is represented as follows. The target time step can be the time step with the minimum cost corresponding to the image denoising model, and can be used to characterize the time step of the cross-attention layer corresponding to the mask that satisfies the preset mask selection conditions. It can be represented by t. * The time step is used to determine the granularity of the model's consideration of information at different time points when generating the target mask; that is, the range of historical and future information the model can consider each time parameters are updated. A smaller time step allows the model to focus more on information at the current moment, while a larger time step masks more historical or future information. Therefore, choosing the target time step allows the model to better capture the relationships and dependencies between sequence data when generating the target mask.

[0058] Optionally, after displaying the original image and prompts on the user interface, in response to processing instructions applied to the interface, multiple mask candidates can be generated based on the object word information in the prompts and multiple cross-attention maps. Further, based on the target layer number l corresponding to the image denoising model... * and target time step t * The target mask M can be determined from multiple generated mask candidates. * .

[0059] The method for generating multiple initial masks based on object information and multiple attention features in the prompt information will be further described below.

[0060] As an optional implementation, multiple initial masks are generated based on object information in the prompt information and multiple attention features, including: extracting multiple sub-features corresponding to the object information from multiple attention features based on the position of the object information in the prompt information; performing binarization processing on the multiple sub-features based on a threshold hyperparameter to obtain processed sub-features; and performing reshaping processing on the processed sub-features according to the resolution of the original image to generate multiple initial masks.

[0061] In this embodiment, in response to processing instructions applied to the user interface, multiple sub-features corresponding to the object information can be extracted from multiple attention features based on the position of the object information in the prompt information. After extracting the multiple sub-features corresponding to the object information from the multiple attention features, the multiple sub-features can be binarized based on a threshold hyperparameter to obtain processed sub-features. Further shaping of the processed sub-features according to the resolution of the original image can generate multiple initial masks. Here, the sub-features can be sub-attention features, which can be called cross-attention subgraphs, hereinafter referred to as subgraphs. The threshold hyperparameter can be a hyperparameter set according to the actual situation; for example, the threshold hyperparameter can be 0.3. This is only an example and no specific limitation is made on the value of the threshold hyperparameter.

[0062] Optionally, in response to processing instructions applied to the user interface, based on the position of the object word information in the prompt information, multiple sub-graphs corresponding to the object word information can be extracted from multiple cross-attention graphs. That is, for each cross-attention graph, a sub-graph corresponding to the object word information p_0 can be extracted from the cross-attention graph corresponding to the prompt information P_0.

[0063] For example, the shape dimensions of Cross Attention (CA) are (N*C, H*W, S'). Here, N represents the batch size, C represents the number of channels in the feature map, H represents the height of the feature map, W represents the width of the feature map, and S' represents the length of the word embedding vector. The length of S' is the same as the length of the word embedding vector. If the position of the word embedding vector corresponding to the object word information p_0 is s, and the position of the word embedding vector corresponding to the prompt information P_0 is s, then the word embedding vector of the selected cross attention subgraph is s. It should be noted that this is only an example and does not impose specific restrictions on the method of extracting the cross attention subgraph.

[0064] Optionally, after extracting multiple sub-images corresponding to the object word information from multiple cross-attention maps, a threshold hyperparameter can be selected, and the extracted sub-images can be binarized according to the selected threshold hyperparameter to obtain processed sub-images. Further reshaping of the processed sub-images according to the resolution of the original image can generate multiple mask candidates. For example, setting the threshold hyperparameter to 0.3, after normalizing the sub-images, sub-images with a value greater than the threshold hyperparameter 0.3 are set to 1, and sub-images with a value less than the threshold hyperparameter 0.3 are set to 0. The processed sub-images are then reshaped to match the resolution of the input video frame (i.e., the original image), thus obtaining the final mask candidate M.

[0065] The method described above for determining the target mask from multiple initial masks based on the target layer number and target time step will be further introduced below.

[0066] As an optional implementation, a target mask is determined from multiple initial masks based on the target layer number and target time step, including: determining the processing type corresponding to the prompt information, wherein the processing type is used to characterize whether the shape of the object corresponding to the object information in the original image is changed; if the processing type characterizes that the shape of the object is changed, a mask corresponding to the target layer number and target time step is determined from multiple initial masks to obtain the target mask; if the processing type characterizes that the shape of the object is not changed, a mask corresponding to the target layer number is determined from multiple initial masks to obtain the target mask.

[0067] In this embodiment, after generating multiple initial masks and determining the target layer number and target time step corresponding to the image denoising model, the processing type corresponding to the prompt information can be determined. Further, if the processing type corresponding to the prompt information indicates a change in the object's shape, the mask corresponding to the target layer number and target time step can be determined from the multiple initial masks to obtain the target mask. If the processing type corresponding to the prompt information indicates no change in the object's shape, the mask corresponding to the target layer number can be determined from the multiple initial masks to obtain the target mask. The processing type can be used to characterize whether the shape of the object corresponding to the object information in the original image is changed; for example, the processing type can be used to characterize whether the object's shape is changed, or it can be used to characterize whether the object's shape is not changed.

[0068] It should be noted that the objects corresponding to the object information in the original image can be represented by p0 and p1 respectively, meaning that p0 and p1 are both objects to be edited. If p0 and p1 are the same, the object to be edited remains unchanged. If p0 and p1 are different, the object to be edited undergoes a shape change, thus determining the task requirements.

[0069] Optionally, by determining the processing type corresponding to the prompt information, since the processing type can be used to characterize whether the shape of the object is changed or not, different processing types can be used to meet the needs of different tasks. When the processing type characterizes a change in the shape of the object, the target layer number l can be determined from multiple mask candidates. * and target time step t * The corresponding target mask, which can be used This is represented. When dealing with type representations without altering the object's shape, the target layer number can be determined from multiple mask candidates. * The corresponding target mask, which can be used To express.

[0070] This embodiment utilizes the systematic semantic differences in cross-attention to control mask accuracy, which can meet the needs of different tasks.

[0071] This embodiment describes the methods for determining the structure-related mask matching cost and the corresponding target layer number, as well as the time-related mask matching cost and the corresponding target time step.

[0072] As an optional implementation, determining the target number of layers and the target time step corresponding to the image denoising model includes: acquiring a dataset, wherein the dataset contains a preset image, preset prompt information, and a preset mask; determining multiple preset attention features output by multiple cross-attention layers at multiple time steps during the processing of the preset image and preset prompt information using the image inversion model; generating multiple candidate masks based on the multiple preset attention features and preset prompt information; and determining the target number of layers and the target time step based on the multiple candidate masks and the preset mask.

[0073] In this embodiment, a dataset can be acquired, which may include a preset image, preset prompt information, and a preset mask. During the processing of the preset image and preset prompt information using an image inversion model, multiple preset attention features output by multiple cross-attention layers at multiple time steps can be determined. Further, based on the preset prompt information and multiple preset attention features, multiple candidate masks can be generated. After generating multiple candidate masks, the target layer number and target time step can be determined based on the generated candidate masks and the preset mask. The dataset can be a video dataset with dense annotation and frame-by-frame segmentation, specifically a Densely Annotated Video Segmentation (DAVIS) dataset. The preset mask can be a reference mask, which can be defined as M0∈B. F×H×WThe preset mask can be obtained by preprocessing the initial image and can be used to generate structure-dependent mask matching costs and time-dependent mask matching costs. Candidate masks can be the mask candidates mentioned above; the mask candidates at layer l and time step t can be... To express.

[0074] Optionally, after acquiring the dataset, for a given video in the dataset, a corresponding ground truth segmentation mask can be used as a reference mask. During the processing of the preset image and preset cue information using the image inversion model, multiple preset attention features output by multiple cross-attention layers at multiple time steps can be determined. Further, based on the preset cue information and the multiple preset attention features, multiple candidate masks can be generated. Finally, based on the determined reference mask and the generated multiple candidate masks, the target layer number and target time step can be determined.

[0075] The method for determining the target layer and target time step based on multiple candidate masks and a preset mask will be further described below.

[0076] As an optional implementation, determining the target number of layers and the target time step based on multiple candidate masks and a preset mask includes: obtaining performance metrics for multiple candidate masks based on the degree of overlap between the multiple candidate masks and the preset mask, wherein different candidate masks correspond to different numbers of layers and / or time steps; summarizing the performance metrics of candidate masks corresponding to the same number of layers to obtain a summary metric for multiple numbers of layers, and summarizing the performance metrics of candidate masks corresponding to the same time step to obtain a summary metric for multiple time steps; determining the target number of layers based on the summary metric for multiple numbers of layers, and obtaining the target time step based on the summary metric for multiple time steps.

[0077] In this embodiment, after generating multiple candidate masks based on preset prompts and multiple preset attention features, performance metrics for the candidate masks can be obtained based on the degree of overlap between the candidate masks and the preset mask. Since different candidate masks correspond to different layers and / or time steps, after obtaining the performance metrics for multiple candidate masks, the performance metrics for candidate masks corresponding to the same layer number can be summarized to obtain a summary metric for multiple layers, and the performance metrics for candidate masks corresponding to the same time step can be summarized to obtain a summary metric for multiple time steps. Further, based on the summary metric for multiple layers, the target layer number can be determined, and based on the summary metric for multiple time steps, the target time step can be obtained. The performance metrics for multiple candidate masks can be the joint average intersection (MIoU) between the candidate masks and the preset mask; that is, based on the degree of overlap between the candidate masks and the preset mask, the MIoU of the multiple candidate masks can be determined. MIoU can be used as a metric to measure the performance of the candidate masks. To express.

[0078] It should be noted that the aggregated metric for multiple layers can be the layout-related mask matching cost (LMMC), which can be expressed as D. LMMC This is represented. The aggregated metric for multiple time steps can be the time-step matched mask cost (TMMC), which can be expressed using D. TMMC The structure-dependent mask matching cost (LMMC) and time-dependent mask matching cost (TMMC) can be collectively referred to as the Matched Mask Cost (MMC).

[0079] Optionally, in obtaining the reference mask M0, and generating multiple candidate masks... Then, the joint average intersection point between multiple candidate masks and the reference mask can be calculated using the following formula:

[0080]

[0081] M0 can be used to represent a reference mask. It can be used to represent candidate masks. It can be used to represent the joint average intersection point MIoU. In the above formula, l can be used to represent the number of layers, and t can be used to represent the time step.

[0082] After obtaining the performance metrics of multiple candidate masks, the performance metrics of candidate masks corresponding to the same layer number can be summarized using the following formula to obtain the structure-related mask matching cost for multiple layer numbers:

[0083]

[0084] Among them, D LMMC It can be used to represent the aggregated structure-related mask matching cost LMMC, D l It can be used to represent the performance index of the candidate mask corresponding to the same layer number before aggregation, L can be used to represent the number of cross-interest layers in U-Net, T can be used to represent the number of sampling steps, l can be used to represent the layer number, and t can be used to represent the time step.

[0085] Furthermore, the performance metrics of candidate masks at the same time step can be summarized using the following formula to obtain the time-related mask matching cost for multiple time steps:

[0086]

[0087] Among them, D TMMC It can be used to represent the aggregated time-dependent mask matching cost TMMC, D t It can be used to represent the performance index of the candidate mask corresponding to the same time step before aggregation, L can be used to represent the number of cross-interest layers in U-Net, T can be used to represent the number of sampling steps, l can be used to represent the number of layers, and t can be used to represent the time step.

[0088] Further, through structure-related mask matching, the cost D is... LMMC The target number of layers can be determined, that is, the number of layers to achieve the minimum cost:

[0089]

[0090] Among them, l * It can be used to represent the target layer number, D l D can be used to represent the performance metric of candidate masks at the same level before aggregation. LMMC It can be used to represent the aggregated structure-related mask matching cost LMMC.

[0091] And through time-dependent mask matching cost D TMMC The target time step can be determined, that is, the time step to achieve the minimum cost can be determined:

[0092]

[0093] Among them, t * It can be used to represent the target time step, Dt D can be used to represent the performance metric of candidate masks at the same time step before aggregation. TMMC It can be used to represent the aggregated time-related mask matching cost TMMC.

[0094] This embodiment can quantify the diversity of cross-attention maps in model structure and denoising time steps by introducing structure-dependent mask matching cost (LMMC) and time-dependent mask matching cost (TMMC).

[0095] This embodiment describes a method for applying target masks selected based on mask matching cost (MMC) to primary attention types.

[0096] As an optional implementation, the image denoising model includes multiple types of attention layers, including at least two of the following: cross-attention, self-attention, and temporal attention. In response to processing commands applied to the user interface, the method further includes: inverting the original image and prompt information using an image inversion model to obtain image information of the original image; during image editing of the original image and prompt information using the image denoising model, acquiring the first attention feature output by each attention layer; during image editing of the image information and prompt information using the image denoising model, acquiring the second attention feature output by each attention layer; using a target mask to guide the image denoising model to fuse the first and second attention features output by the same attention layer to obtain a fused feature corresponding to the same attention layer, wherein the fused feature corresponding to the last attention layer is the target feature output by the image denoising model; and reconstructing the target feature to generate a target image.

[0097] In this embodiment, the image denoising model includes multiple types of attention layers, including at least two of the following: cross-attention, self-attention, and temporal attention. After displaying the original image and prompt information on the operation interface, in response to the processing instructions applied to the operation interface, the image inversion model can be used to invert the original image and prompt information to obtain the image information of the original image. Further, during the image editing process of the original image and prompt information using the image denoising model, the first attention feature output by each attention layer can be obtained. After obtaining the image information of the original image, during the image editing process of the image information and prompt information using the image denoising model, the second attention feature output by each attention layer can be obtained. After obtaining the first and second attention features, a target mask can be used to guide the image denoising model to fuse the first and second attention features output by the same attention layer to obtain the fused feature corresponding to the same attention layer. The fused feature corresponding to the last attention layer can be determined as the target feature output by the image denoising model. Further image reconstruction of the target feature can generate the target image.

[0098] It should be noted that the image information can be data obtained by inverting the original image and the prompt information, and can be represented by Z. T The representation is as follows: The first attention feature can be the feature obtained by passing the original image as input through the attention layer, which can be called the source attention feature. The second attention feature can be the feature obtained by editing and denoising the inverted image information through the model, which can be called the edited attention feature. The target feature can be the feature obtained by fusing the source attention feature and the edited attention feature, which can be called the hybrid attention feature.

[0099] Optionally, an image inversion model is used to invert the original image and the prompt information to obtain image information. The original image and the inverted image information are then used as inputs to pass through an attention layer to obtain source attention features and edited attention features, respectively. The source attention features and edited attention features are then fused; for example, they can be added together, or the key and query of the source attention features can be added to the key and query of the edited attention features, respectively, to obtain a mixed attention feature. Finally, the mixed attention feature is used to reconstruct the target image, generating the target image.

[0100] This embodiment applies the target mask selected via Mask Matching Cost (MMC) to the main attention types: cross attention, self attention, and temporal attention, which can alleviate the over-constraint of mixing while improving editing quality.

[0101] The following section further introduces the method of feature fusion of the first and second attention features output by the temporal attention layer using the target mask-guided image denoising model.

[0102] As an optional implementation, when the same attention layer is a temporal attention layer, the target mask guides the image denoising model to perform feature fusion on the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature corresponding to the same attention layer. This includes: expanding the target mask according to the spatial dimension to obtain the expanded mask; determining the first key vector and the first query vector corresponding to the first attention feature output by the same attention layer, and the second key vector and the second query vector corresponding to the second attention feature output by the same attention layer; and fusing the first key vector, the first query vector, the second key vector, and the second query vector using the expanded mask to obtain the fused feature.

[0103] In this embodiment, the target mask can be expanded according to the spatial dimension to obtain the expanded mask. Further, the first key vector and first query vector corresponding to the first attention feature output by the same attention layer, and the second key vector and second query vector corresponding to the second attention feature output by the same attention layer can be determined. After expanding the target mask according to the spatial dimension to obtain the expanded mask, the first key vector, first query vector, second key vector, and second query vector can be fused using the expanded mask to obtain the fused feature. Here, the first attention feature output by the same attention layer can be called the source temporal attention, and the first key vector can be the key value of the source temporal attention, which can be used... In this representation, the first query vector can be a query of the source temporal attention, which can be used... This is represented as follows. The second attention feature output by the same attention layer can be called the editing temporal attention, and the second key vector can be the key value of the editing temporal attention, which can be represented as... In this representation, the second query vector can be a query for editing time attention, which can be used... The features obtained by fusing source temporal attention and edit temporal attention can be called hybrid temporal attention.

[0104] Optionally, since temporal attention is crucial for maintaining temporal motion consistency, pixel-level motion is tracked by capturing the relative transformations between pixels across different frames. However, excessive fusion of this fine-grained motion information can result in the edited video being identical to the source video. To address the over-constraint problem, this embodiment uses a mask to decouple the edited region within the temporary attention features, where the batch size matches the number of input feature labels. At time step t of the denoising process, the target mask is applied. Convert to the following formula:

[0105]

[0106] in, It can be used to represent a target mask. This can be used to represent the expanded mask obtained after expanding the target mask. F can be used to represent the expansion function, and δ can be used to represent the processing type corresponding to the prompt information. δ = 1 indicates that the processing type represents changing the shape of the object, and δ = 0 indicates that the processing type represents not changing the shape of the object. Target Mask It is reshaped in the spatial dimension, and the target mask is flattened into a two-dimensional tensor m using the expansion function F. * ∈B h′w′×F h'w' equals the batch size of the temporary attention, i.e., the resolution of the image features. For tasks that preserve the entire structure (e.g., stylization), δ = 1 and Further The dimensions of batches applied to temporal attention:

[0107]

[0108] in, Keys that can be used to represent source temporal attention It can be used to represent queries that indicate source temporal attention. Key-value pairs that can be used to represent the time attention of editors Queries that can be used to represent the time attention of editors It can be used to represent the key-value pairs of the mixed temporal attention. This can be used to represent queries that represent mixed temporal attention; both the key-value pairs and the queries mentioned above belong to the vector set R. h′w′×F×dim ⊙ can be used to represent the Dammar product, m * It can be used to represent the expanded mask obtained after expanding the target mask, thereby realizing the fusion of time attention mechanism guided by mask structure.

[0109] The following section further introduces the method of feature fusion of the first attention feature and the second attention feature output by the cross-attention layer using the target mask-guided image denoising model.

[0110] As an optional implementation, when the same attention layer is a cross-attention layer, the target mask guides the image denoising model to perform feature fusion on the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature corresponding to the same attention layer. This includes: unfolding the target mask according to the spatial dimension to obtain the unfolded mask; and using the unfolded mask to perform feature fusion on the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature.

[0111] In this embodiment, the target mask can be expanded according to the spatial dimension to obtain the expanded mask. The expanded mask is then used to fuse the first attention feature and the second attention feature output from the same attention layer to obtain the fused feature. The first attention feature output from the cross-attention layer can be called the source cross-attention, and can be used... This is represented. The second attention feature output by the cross-attention layer can be called the edited cross-attention, which can be represented using... This is represented as follows. The features obtained by fusing the source cross-attention and the edit cross-attention can be called the hybrid cross-attention, which can be represented using... To express.

[0112] Optionally, to preserve structural details, cross-attention maps are often reweighted, refined, or replaced with the source map. However, the above methods ignore the influence of adjacent function words, which can retain some spatial information of the source object, reducing editing accuracy. For example, when prompting "a jeep is driving in the countryside," cross-attention to "a" or "driving" might display the outline of "jeep," affecting accuracy. To alleviate the above problem, this embodiment introduces orthogonal mask blending to enhance standard blending operations. The target is masked according to spatial dimensions. Expand the mask to obtain the expanded mask. Then, the first and second attention features output by the cross-attention layer can be fused using the following formula, thereby achieving the fusion of the temporal attention mechanism guided by the mask structure:

[0113]

[0114] in, It can be used to represent source cross attention. It can be used to represent cross-attention during editing. It can be used to represent the mixed cross-attention, m * It can be used to represent the expanded mask obtained after expanding the target mask.

[0115] The following section further introduces the method of feature fusion of the first attention feature and the second attention feature output by the self-attention layer using the target mask-guided image denoising model.

[0116] As an optional implementation, when the same attention layer is a self-attention layer, the target mask guides the image denoising model to perform feature fusion on the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature corresponding to the same attention layer. This includes: unfolding the target mask according to the spatial dimension to obtain the unfolded mask; and using the unfolded mask to perform feature fusion on the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature.

[0117] In this embodiment, the target mask can be expanded according to the spatial dimension to obtain the expanded mask. The expanded mask is then used to fuse the first attention feature and the second attention feature output from the same attention layer to obtain the fused feature. The first attention feature output from the self-attention layer can be a source attention feature, and the second attention feature output from the self-attention layer can be an edited self-attention feature.

[0118] Optionally, the mask-guided self-attention blending is extended from the original attribute and shape editing to stylization. For stylization, the mask is applied in reverse, causing the mask to focus on the source attention map. Since stylization can cause undesirable deformations of moving objects, this deformation can be mitigated through masked self-attention blending. That is, fusing the first and second attention features output by the self-attention layer can alleviate this deformation, thus achieving the fusion of a mask-structure-guided self-attention mechanism.

[0119] This embodiment describes the method for selecting the image denoising model to use through an interactive approach.

[0120] As an optional implementation, in response to a determination command applied to the operation interface, the method further includes: displaying identification information of at least one preset denoising model on the operation interface; and in response to a selection command applied to the identification information, displaying identification information of the image denoising model corresponding to the selection command on the operation interface, wherein the at least one preset denoising model includes an image denoising model.

[0121] In this embodiment, after displaying the original image and prompt information on the operation interface, in response to a confirmation command applied to the operation interface, the identification information of at least one preset denoising model can be displayed on the operation interface. After displaying the identification information of at least one preset denoising model on the operation interface, in response to a selection command applied to the identification information, the identification information of the image denoising model corresponding to the selection command can be displayed on the operation interface. The confirmation command can be an instruction to confirm the identification information of at least one preset denoising model; the user can generate the confirmation command by clicking a button on the operation interface or by dragging and dropping the identification information of at least one preset denoising model. At least one preset denoising model includes an image denoising model. The identification information can be used to indicate the corresponding preset denoising model; that is, the identification information of each preset denoising model can be different. The selection command can be an instruction to select the identification information of an image denoising model from at least one identification information; the user can generate the selection command by clicking one or more identification information displayed on the operation interface. At least one preset denoising model can be at least one pre-set preset denoising model displayed on the operation interface, allowing the user to select the identification information of one or more preset denoising models as the identification information of the image denoising model according to the actual situation. If the identification information of at least one preset denoising model displayed on the operation interface does not meet the user's needs, at least one preset denoising model can also be a preset denoising model uploaded by the user according to actual needs.

[0122] In one optional embodiment, a user can generate a confirmation command by clicking a button to upload the identification information of at least one preset denoising model or by dragging and dropping the identification information of at least one preset denoising model to a designated area. Based on the generated confirmation command, the corresponding preset denoising model identification information can be loaded and displayed on the user interface. Further, after displaying the preset denoising model identification information on the user interface, the user can interactively select the image denoising model to use. For example, the user can select one or more identification information from the displayed preset denoising model identification information to generate a selection command. Based on the generated selection command, the corresponding image denoising model selected by the user can be displayed on the user interface.

[0123] In this embodiment, in response to an input command applied to the operation interface, the original image and prompt information are displayed on the operation interface. The prompt information describes the processing performed on the original image. In response to a processing command applied to the operation interface, the target image corresponding to the prompt information is displayed on the operation interface. The target image is obtained by using a target mask to guide an image denoising model to edit the original image and prompt information. The target mask represents a mask that meets preset mask selection conditions. In other words, this embodiment generates a target mask that meets preset mask selection conditions. That is, the target mask is not an attention mask directly generated based on multiple attention features, but rather a mask selected from attention masks that meets preset mask selection conditions. Therefore, using this target mask to guide the image denoising model to edit the original image and prompt information avoids the inaccuracy of the processing results caused by directly using attention masks, thereby improving the image processing effect and solving the technical problem of poor image processing results.

[0124] This application also provides a video processing method. Figure 3 This is a flowchart of a video processing method according to an embodiment of this application, such as... Figure 3 As shown, the method may include the following steps:

[0125] Step S302: In response to the input command applied to the operation interface, the original video and prompt information are displayed on the operation interface.

[0126] In the technical solution provided by step S302 of this application, in response to the input command applied to the operation interface, the original video and prompt information can be displayed on the operation interface. The operation interface can be the operation interface on the aforementioned client device, such as an interface for interaction between the user and a computer or software system, or a graphical user interface, command-line interface, etc. This is merely an example and does not impose specific limitations on the form of the operation interface.

[0127] It should be noted that input commands can be commands that control the display of the original video and prompts on the operation interface. For example, they can be commands entered by the user on the operation interface. The user can generate input commands by entering text on the keyboard, clicking buttons or options on the operation interface with the mouse, or dragging the original image and prompts to the specified area.

[0128] It should be noted that the original video can be the video to be edited displayed on the above operation interface, which can be called the input video and can be represented by Z0. The prompt information can be used to describe the processing performed on the original video and can be represented by P_0.

[0129] In one optional embodiment, the user can open the operation interface deployed on the client device and locate the button or designated area for uploading the original video and prompt information. By clicking the button or dragging and dropping the original video and prompt information to the designated area, an input command can be generated. Based on the generated input command, the corresponding original video and prompt information can be loaded and displayed on the operation interface.

[0130] Step S304: In response to the processing command applied to the operation interface, display the target video corresponding to the prompt information on the operation interface.

[0131] In the technical solution provided in step S304 of this application, after responding to the input command applied to the operation interface and displaying the original video and prompt information on the operation interface, responding to the processing command applied to the operation interface, the target video corresponding to the prompt information can be displayed on the operation interface. The processing command can be an instruction to process the original video and prompt information displayed on the operation interface. For example, it can be a video editing command. The user can generate the processing command by clicking on the original image and prompt information displayed on the operation interface. This is only an example and does not impose specific limitations on the processing command.

[0132] It should be noted that the target video can be a video obtained by editing the original video and the prompt information; this can be called an edited video. It can be obtained by using a target mask to guide a video denoising model to edit the original video and the prompt information. To express.

[0133] It should be noted that the target mask can be a mask that meets the preset mask filtering conditions, and can be used with M. * By representing the target mask, the corresponding image denoising model can be determined from multiple initial masks. Video denoising models can be used to edit the original video and prompts to obtain the target video. Video denoising models can also be used to apply target masks to achieve the fusion of different types of attention features.

[0134] It should be noted that the preset mask filtering conditions can be pre-set according to the actual situation, and these preset mask filtering conditions can be adjusted according to different needs.

[0135] In this embodiment, after the original video and prompt information are displayed on the operation interface, in response to the processing instructions applied to the operation interface, the original video and prompt information displayed on the operation interface can be processed. For example, the original video and prompt information displayed on the operation interface can be edited to obtain the target video corresponding to the prompt information, and the target video can be displayed on the operation interface.

[0136] Optionally, after displaying the original video and prompt information on the user interface, the original video and prompt information can be sent to the server. Upon receiving the original video and prompt information, the server can process them using a video inversion model. This process involves obtaining multiple attention features output by multiple cross-attention layers of the video inversion model at multiple time steps. Based on these output attention features, a target mask that meets preset mask filtering conditions is generated. This target mask then guides a video denoising model to edit the original video and prompt information to obtain the target video. Users can click on the original video and prompt information displayed on the user interface to generate processing instructions. According to these instructions, the target video generated by the server can be sent to the client device, where it is displayed on the user interface.

[0137] Through steps S302 to S304 of this application, in response to input commands applied to the operation interface, the original video and prompt information are displayed on the operation interface, wherein the prompt information describes the processing performed on the original video; in response to processing commands applied to the operation interface, the target video corresponding to the prompt information is displayed on the operation interface, wherein the target video is obtained by using a target mask to guide a video denoising model to perform video editing on the original video and prompt information, and the target mask is used to represent a mask that meets preset mask filtering conditions. That is to say, the embodiment of this application generates a target mask that meets preset mask filtering conditions. That is, the target mask is not an attention mask directly generated based on multiple attention features, but a mask that meets preset mask filtering conditions selected from attention masks. Therefore, using this target mask to guide the image denoising model to perform image editing on the original image and prompt information avoids the inaccuracy of the processing results caused by directly using attention masks, thereby improving the image processing effect and solving the technical problem of poor image processing effect.

[0138] This application also provides another image processing method. Figure 4 This is a flowchart of another image processing method according to an embodiment of this application, such as... Figure 4 As shown, the method may include the following steps:

[0139] Step S402: Obtain the original image and prompt information by calling the first interface.

[0140] In the technical solution provided by step S402 of this application, the original image and prompt information can be obtained by calling the first interface. The first interface may include a first parameter, and the value of the first parameter may include the original image and prompt information.

[0141] It should be noted that the original image can be the image to be edited displayed on the above-mentioned operation interface, which can be called the input image and can be represented by Z0. The prompt information can be used to describe the processing performed on the original image and can be represented by P_0.

[0142] Step S404: Generate a target mask that meets the preset mask filtering conditions.

[0143] In the technical solution provided in step S404 of this application, multiple initial masks can be obtained. After obtaining multiple initial masks, a target mask that meets the preset mask filtering conditions can be selected from the multiple initial masks.

[0144] It should be noted that the target mask can be a mask that meets the preset mask filtering conditions, and can be used with M. * This method allows for the determination of the target mask corresponding to the image denoising model from multiple initial masks. Preset mask filtering conditions can be pre-defined based on actual conditions and can be adjusted according to different needs.

[0145] In this embodiment, after obtaining the original image and prompt information by calling the first interface, a target mask that meets the preset mask filtering conditions can be determined from multiple initial masks.

[0146] Step S406: Use the target mask to guide the image denoising model to perform image editing on the original image and the prompt information, and generate the target image corresponding to the prompt information.

[0147] In the technical solution provided in step S406 of this application, after generating a target mask that meets the preset mask filtering conditions, the target mask guides the image denoising model to perform image editing on the original image and the prompt information, thereby generating the target image corresponding to the prompt information. The image denoising model can be used to edit the original image and the prompt information to obtain the target image, and the image denoising model can be used to apply the target mask to achieve the fusion of different types of attention features.

[0148] In this embodiment, after generating a target mask that meets the preset mask filtering conditions, the target mask can be used to guide the image denoising model to perform image editing on the original image and the prompt information to obtain the target image.

[0149] Step S408: Output the target image by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0150] In the technical solution provided by step S408 of this application, after using the target mask to guide the image denoising model to edit the original image and the prompt information and generate the target image corresponding to the prompt information, the target image can be output by calling the second interface.

[0151] It should be noted that the second interface may include a second parameter, and the value of the second parameter may include the target image. The target image can be an image obtained by editing the original image and the prompt information, which can be called an edited image. It can be obtained by using a target mask to guide an image denoising model to edit the original image and the prompt information.

[0152] In this embodiment, after obtaining the original image and prompt information by calling the first interface, a target mask that meets the preset mask filtering conditions can be generated. Furthermore, the target mask can be used to guide the image denoising model to perform image editing on the original image and prompt information to obtain the target image, and the target image can be output by calling the interface.

[0153] Through steps S402 to S408 of this application, the original image and prompt information are obtained by calling a first interface, wherein the first interface includes a first parameter, the parameter value of which includes the original image and the prompt information, and the prompt information is used to describe the processing performed on the original image; a target mask that meets the preset mask filtering conditions is generated; the target mask is used to guide the image denoising model to perform image editing on the original image and the prompt information, generating the target image corresponding to the prompt information; and the target image is output by calling a second interface, wherein the second interface includes a second parameter, the parameter value of which includes the target image. In other words, the target mask generated in this embodiment meets the preset mask filtering conditions. That is, the target mask is not an attention mask directly generated based on multiple attention features, but a mask that meets the preset mask filtering conditions selected from the attention masks. Therefore, using this target mask to guide the image denoising model to perform image editing on the original image and the prompt information avoids the inaccuracy of the processing result caused by directly using the attention mask, thereby improving the image processing effect and solving the technical problem of poor image processing effect.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application, such as the data to be verified, are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0155] Currently, with significant advancements in text-to-video diffusion models and the ability of zero-shot video editing to generate temporally coherent videos, zero-shot video editing based on video diffusion models has rapidly expanded. To improve editing quality, structural control is commonly employed in video editing, and cross-attention masks are one of the more effective and efficient sources.

[0156] However, since cross-attention masks are not always clear, but rather change with the model structure and denoising time step, they may introduce artifacts such as blurring and flickering when applied directly to video editing.

[0157] In related technologies, video editing tasks can include zero-shot video editing tasks that use attention masks for shape editing (e.g., FateZero), zero-shot video editing tasks that use other structural controls (e.g., Controlvideo, TokenFlow, RerenderAVideo), video editing tasks that require fine-tuning and use attention masks for structural control (e.g., Video-P2P), and video editing tasks that require fine-tuning and use other structural controls (e.g., FlowVid).

[0158] Video editing methods that utilize attention masks for structural control, such as the zero-shot editing task Fatezero and the fine-tuning task Video-P2P, both leverage the endogeneity of cross-attention masks to achieve shape editing control without external assistance. However, these methods do not consider the structural and temporal differences in attention masks. Directly applying such masks may lead to inaccurate editing results, especially artifacts and temporal discontinuities at shape edges.

[0159] Video editing methods utilizing external structural control typically employ optical flow estimation (Rerender aVideo), depth estimation, edge estimation (ControlVideo), or semantic segmentation masks (InstructV2V) for structural control. However, the precise temporally independent masks used in these methods may not be suitable for some editing tasks, such as shape editing, where the edited shape does not conform to the source shape mask. Furthermore, these methods require externally pre-trained models, introducing additional computational overhead for inference, and directly applying image-processing-specific models to video processing often results in noticeable artifacts.

[0160] To address the aforementioned issues, this application utilizes Mask Matching Cost (MMC) to quantify the variability of cross-attention masks and proposes a method for selecting masks based on specific video editing tasks. Furthermore, by using masks selected through MMC, a comprehensive mask fusion mechanism is proposed, which performs mask feature mixing within self-attention, cross-attention, and temporal attention modules. This method can be seamlessly integrated into zero-shot video editing frameworks, exhibiting improved performance without requiring control aids or parameter fine-tuning. Moreover, it enables adaptive decoupling of unedited semantic layouts through mask precision control.

[0161] This embodiment proposes a method for zero-shot video editing using an attention mechanism mask, which can be applied to content creation and editing, game asset generation and editing, and other fields. This embodiment improves the accuracy of zero-shot video editing algorithms based on image / video diffusion models by optimizing mask selection during the editing process. Based on the finding that the masks in current zero-shot video editing algorithms are not accurate enough, two strategies are proposed to select the target mask, and the masking mechanism is integrated into the three core structures of the network.

[0162] Considering the importance of attention masks in zero-shot video editing, this embodiment proposes an adaptive mask (FreeMask), which uses strategic masking to achieve structure-controlled editing through cross-attention masks. FreeMask is based on two key observations: firstly, attention masks become clearer as denoising progresses; secondly, since attention masks have more noise in the outer layers and are less accurate in the inner layers due to low resolution, the accuracy of attention masks is highest in the middle layers.

[0163] To quantify the above observations, this embodiment introduces mask matching cost MMC, a metric based on joint average intersection MIoU, which can be used to measure the layer-by-layer and time-step attention mask accuracy, namely, structure-dependent mask matching cost LMMC and time-step-dependent mask matching cost TMMC.

[0164] By using the two metrics mentioned above, given a specific video diffusion model, we can determine the target number of layers and the target time step for generating masks using cross-attention maps. This will enable us to design specific mask usage strategies for different editing tasks, such as style translation, attribute editing, and shape editing.

[0165] Finally, considering that a common problem in attention fusion during video editing is over-constraint, due to inaccurate mask guidance and partial mask fusion on subsets of attention types, the source video excessively restricts the editing target. Therefore, this implementation integrates MMC-selected masks for structural control and performs feature fusion across a more comprehensive range of attention layer types, including Temp Attention (TA), Cross Attention (CA), and Self Attention (SA). This embodiment alleviates over-constraint and improves the editing accuracy of FreeMask through the aforementioned integrated and comprehensive fusion methods.

[0166] Figure 5 This is a schematic diagram of a FreeMask according to an embodiment of this application, as shown below. Figure 5 As shown, a target mask satisfying preset mask selection conditions is generated based on multiple attention features of the image inversion model (DDIM inversion*T). This target mask is then used to guide the image denoising model (Denoising*T) to edit the original image Input Video and the prompt information (p0 and p1), resulting in the edited image Edited Video. Furthermore, the target mask selected via mask matching cost (MMC) is applied to the main attention types: cross attention (CA), self-attention (SA), and temporal attention (TA), i.e., fusion (M-Fusion) of these attention types.

[0167] like Figure 5 The right half of the diagram shows the selection of the target mask M by the mask matching cost MMC. * And the process of fusion attention. For the target mask M selected by MMC * The process begins by determining the joint average intersection point (MIoU) between multiple mask candidates M and the reference mask M0. Further, based on the MIoU, the structure-dependent mask matching cost (LMMC) for multiple layer numbers and the time-dependent mask matching cost (TMMC) for multiple time steps are obtained. Finally, the layer number achieving the minimum cost is determined based on the LMMC, and the time step achieving the minimum cost is determined based on the TMMC. After quantifying the semantic differences between structure-dependent and time-step-dependent masks using LMMC and TMMC respectively, a target mask is selected to guide the editing process. For the attention fusion process, the selected target mask can be applied to cross-attention (M-CA), self-attention (M-SA), and temporal attention (M-TA).

[0168] In other words, the above-described editing workflow utilizes inversion followed by denoising and stores intermediate attention features for later editing. In the preprocessing step, model-related LMMC and TMMC metrics are first calculated using the DAVIS video dataset. During video editing, after performing DDIM inversion and before denoising, semantically adaptive MMC metrics are calculated to select the attention mask. During denoising, masks are applied to guide the fusion of different types of attention features.

[0169] Regarding the mask cost matching process: First, mask candidates are extracted. Mask candidates can be generated. Where L can be used to represent the number of cross-interest layers in the diffusion U-Net, and T can be used to represent the number of sampling steps. This can be used to represent mask candidates at layer l and time step t. The above candidates come from... Figure 5 The cross-attention map is obtained in the DDIM inversion step. Specifically, for each cross-attention map, a sub-map corresponding to the object word p0 is extracted from the cue words. The mapping (i.e., the above sub-map) is binarized according to the selected threshold hyperparameter, and the binarized sub-map is reshaped to match the resolution of the input video frame, thereby obtaining the final mask candidate M.

[0170] Secondly, the time-step-dependent mask matching cost (TMMC) and the structure-dependent mask matching cost (LMMC) are determined. To quantify the diversity of cross-attention maps across model structure and denoising time steps, structure-dependent mask matching cost (LMMC) and time-step-dependent mask matching cost (TMMC) are introduced. A video dataset with dense annotations and frame-by-frame segmentation, namely the DAVIS dataset, is used. For a given video in the dataset, the corresponding ground truth segmentation mask is used as a reference mask, which can be defined by M0∈B. F×H×W The following is a representation. The joint average intersection point MIoU between the mask candidate M and the reference mask M0 is further calculated. This joint average intersection point can be represented by the following formula to evaluate the matching performance:

[0171]

[0172] in, M0 can be used to represent the joint average intersection point, and M0 can be used to represent the reference mask. It can be used to represent a candidate mask, l can be used to represent the layer number, and t can be used to represent the time step.

[0173] Determining the joint average intersection point The structure-related mask matching cost can then be calculated using the following formula:

[0174]

[0175] Among them, D LMMC It can be used to represent the structure-dependent mask matching cost, D l The performance metrics can be used to represent candidate masks at the same layer number before aggregation. L can represent the number of cross-interest layers in U-Net, T can represent the number of sampling steps, l can represent the layer number, and t can represent the time step. Further, the structure-related mask matching cost D is used... LMMC The number of layers required to achieve the minimum cost can be determined:

[0176]

[0177] Among them, l * D can be used to represent the number of layers to achieve the minimum cost. l D can be used to represent the performance metric of candidate masks at the same level before aggregation. LMMC It can be used to represent the aggregated structure-related mask matching cost LMMC.

[0178] Determining the joint average intersection point The time-step related mask matching cost can then be calculated using the following formula:

[0179]

[0180] Among them, D TMMC It can be used to represent the time-step-dependent mask matching cost, D t The performance metrics for candidate masks at the same time step before aggregation can be represented by L, T, l, and t. L can represent the number of cross-interest layers in U-Net, T can represent the number of sampling steps, l can represent the number of layers, and t can represent the time step. Further, the mask matching cost D related to the time step is calculated. TMMC The time step to achieve the minimum cost can be determined:

[0181]

[0182] Among them, t * D can be used to represent the number of layers to achieve the minimum cost. t D can be used to represent the performance metric of candidate masks at the same time step before aggregation. TMMC It can be used to represent the aggregated time-related mask matching cost TMMC.

[0183] Finally, a semantically adaptive mask cost matching method is designed. After quantizing the semantic differences between structure-dependent and time-step-dependent masks using LMMC and TMMC respectively, a target mask can be selected to guide the editing process. For the target mask, where, This can be used to represent the target mask at layer l and time step t. Mask accuracy is controlled using systematic semantic differences in cross-attention to meet the needs of different tasks. Time-independent, accurate masks are crucial for tasks requiring high structural consistency. Conversely, for tasks involving shape editing and structural transformation, time-aware (coarse-to-fine) masks are needed. Based on this, a semantically adaptive MMC is designed using TMMC and LMMC, adaptively selecting time-independent and time-aware masks through a designed Kronecker delta function.

[0184]

[0185] Here, p0 and p1 correspond to the objects in the editing prompt, and δ can be used to represent the processing type corresponding to the prompt information. It can be used to represent a target mask. If p0 and p1 are the same, then δ = 1, and the object to be edited remains unchanged. If p0 and p1 are different, then δ = 0, and the shape of the object to be edited changes, thus determining the task requirements. When δ = 0, the target layer number l can be selected from multiple mask candidates. * Corresponding mask As the target mask. When δ = 1, the target layer number l can be selected from multiple mask candidates. * and target time step t * Corresponding mask As a target mask. That is, for tasks that require flexible shape changes, such as shape editing, use a time-aware mask. For tasks that require high structural coherence, use a time-independent mask.

[0186] Regarding the application of attention mask structure control in attention mechanisms, the mask selected by MMC can be applied to the main attention type to alleviate over-constraints and improve editing quality.

[0187] First, a temporal attention mechanism guided by a mask structure is used for fusion. Since temporal attention is crucial for maintaining temporal motion consistency, pixel-level motion is tracked by capturing the relative transformations between pixels in different frames. However, excessive fusion of this fine-grained motion information can result in the edited video being identical to the source video. To address this over-constraint problem, a mask is used to decouple the edited region within the temporary attention features, where the batch size matches the number of input feature labels. At time step t in the denoising process, the mask is... Convert to the following formula:

[0188]

[0189] in, It can be used to represent a target mask. This can be used to represent the expanded mask obtained by expanding the target mask. F can be used to represent the expansion function, and δ can be used to represent the processing type corresponding to the prompt message. Mask It is reshaped in the spatial dimension and flattened into a two-dimensional tensor m using the expansion function F. * ∈B h′w′×F , where h'w' equals the batch size of the temporary attention, i.e., the resolution of the image features. For tasks that preserve the entire structure (e.g., stylization), δ = 1 and Further The dimensions of batches applied to temporal attention:

[0190]

[0191] in, Keys that can be used to represent source temporal attention It can be used to represent queries that indicate source temporal attention. Key-value pairs that can be used to represent the time attention of editors Queries that can be used to represent the time attention of editors It can be used to represent the key-value pairs of the mixed temporal attention. Queries that can be used to represent mixed temporal attention belong to the vector set R. h′w′×F×dim ⊙ can be used to represent the Dammar product, m * It can be used to represent the expanded mask obtained after expanding the target mask.

[0192] Secondly, a temporal attention mechanism based on mask structure is used for fusion. To preserve structural details, cross-attention maps are often reweighted, refined, or replaced with the source map. However, the above methods ignore the influence of adjacent function words, which can retain some spatial information of the source object, reducing editing accuracy. For example, when prompting "a jeep is driving in the countryside," cross-attention to "a" or "driving" might display the outline of "jeep," affecting accuracy. To alleviate this problem, orthogonal mask blending is introduced to enhance the standard blending operation. The following formula can be used to... Decapsulation And mix it with the source cross-attention before performing the usual fusion operation:

[0193]

[0194] in, It can be used to represent source cross attention. It can be used to represent cross-attention during editing. It can be used to represent the mixed cross-attention, m* It can be used to represent the expanded mask obtained after expanding the target mask.

[0195] Finally, based on a mask-guided self-attention mechanism, following the mask-guided self-attention feature blending methods commonly used in works such as prompt-to-prompt and Fate / Zero, the mask-guided self-attention blending is extended from the original attribute and shape editing to stylization. For stylization, the mask is applied in reverse, causing the mask to focus on the source attention map. Since stylization can cause undesirable deformations of moving objects, this deformation can be mitigated through masked self-attention blending.

[0196] In this embodiment, FreeMask can be seamlessly integrated into zero-shot video editing frameworks without the need for additional control assistance or parameter fine-tuning. FreeMask offers significant advantages for zero-shot video editing in the absence of user-specific controls.

[0197] Table 1 compares FreeMask with other methods. As shown in Table 1, the mask-guided fusion method in this embodiment differs from other similar methods. M can be used to represent a mask, SA can be used to represent self-attention, CA can be used to represent cross-attention, TA can be used to represent temporal attention, L can be used to represent latent variables in the intermediate process, SM can be used to represent a mask that explains structural changes, TM can be used to represent a mask that represents changes in time step, M-SA can be used to represent a self-attention mask, M-CA can be used to represent a cross-attention mask, M-TA can be used to represent a temporal attention mask, and ML can be used to represent a latent variable mask.

[0198] As shown in Table 1, the token-based video generation method (Tokenflow), image-to-video generation method (Pix2Video), re-render a video method (Rerender a Video), text-to-video generation method (Text2Video-Zero), video generation model method (FateZero), video processing and generation method (CoDef), video tuning method (Tune a Video), and video transmission technology (Video-P2P) are compared with the adaptive masking method FreeMask in this application in terms of mask SM for interpreting structural changes, mask TM for time step changes, self-attention mask M-SA, cross-attention mask M-CA, temporal attention mask M-TA, latent variable mask ML, zero-shot, and backbone. As can be seen from the table, the main components used in related technologies are stable diffusion (SD), stable diffusion depth (SD-depth), or control networks. However, the method in this application can implement any technology converter (Any T2V). The method in this embodiment can perform mask-guided editing on a comprehensive attention mechanism and utilizes a textual video model to improve temporal consistency. Other methods can only perform fusion at local time steps of some attention types, resulting in less than ideal editing effects.

[0199] Table 1 Comparison of FreeMask with other methods

[0200]

[0201] Figure 6 This is a schematic diagram illustrating the qualitative comparison results of FreeMask according to an embodiment of this application, as shown below. Figure 6 As shown in the qualitative comparison results between FreeMask and other state-of-the-art (SOTA) methods, it can be seen that FreeMask performs better in style transfer, attribute editing, and shape editing.

[0202] Table 2 compares the quantitative comparison results with the user study results. It can be seen that the method in this embodiment has better performance in terms of temporal consistency, alignment of produced video and text (CLIP), and semantic preservation capabilities such as mean peak signal-to-noise ratio (M.PSNR) and similarity (LPIPS). It also performs better in terms of editing capabilities, image quality, and overall fluency in the user study.

[0203] As shown in Table 2, the following video generation model methods in related technologies (FateZero, TokenFlow, Video-P2P, Time-to-Video, Control Video, Any V2V, and the combination of video generation model and mask matching cost method (FateZero+MMC)) are compared with the adaptive masking method FreeMask in this application in terms of temporal consistency (Temp), video-text alignment (CLIP), average peak signal-to-noise ratio (MPSNR), similarity (LPIPS), editing capability (Ed), image quality (Image), and overall smoothness (Quadium).

[0204] Table 2. Comparison of FreeMask with other methods in quantitative comparison and user survey.

[0205]

[0206] This embodiment takes into account the systematic semantic differences of cross-attention maps in video generation models. Its accuracy is positively correlated with the denoising time step, initially increasing and then decreasing with the increase in the number of cross-attention layers (referring to 3D-Unet). This embodiment proposes a mask matching cost (MMC), consisting of a time-dependent mask matching cost (TMMC) and a structure-dependent mask matching cost (LMMC), to utilize the systematic semantic differences of cross-attention maps for accuracy control and selection of cross-attention masks. It also improves mask structure-guided attention fusion based on comprehensive attention mechanisms (temporal attention, cross-attention, and self-attention), avoiding complex selection of hyperparameters such as fusion ratios, while also effectively decoupling attributes from non-editing parts. Furthermore, the method in this embodiment does not require retraining or additional structural assistance, allowing it to be seamlessly integrated into video editing methods and adaptable to various video models.

[0207] In this embodiment, in response to an input command applied to the operation interface, the original image and prompt information are displayed on the operation interface. The prompt information describes the processing performed on the original image. In response to a processing command applied to the operation interface, the target image corresponding to the prompt information is displayed on the operation interface. The target image is obtained by using a target mask to guide an image denoising model to edit the original image and prompt information. The target mask represents a mask that meets preset mask selection conditions. In other words, this embodiment generates a target mask that meets preset mask selection conditions. That is, the target mask is not an attention mask directly generated based on multiple attention features, but rather a mask selected from attention masks that meets preset mask selection conditions. Therefore, using this target mask to guide the image denoising model to edit the original image and prompt information avoids the inaccuracy of the processing results caused by directly using attention masks, thereby improving the image processing effect and solving the technical problem of poor image processing results.

[0208] Figure 7 This is a structural block diagram of a computing environment for an image processing method according to an embodiment of this application, such as... Figure 7 As shown, computing environment 701 includes multiple computing nodes (such as servers) running on a distributed network (represented as 710-1, 710-2, ..., in the diagram). Each computing node contains local processing and memory resources, and end user 702 can remotely run applications or store data within computing environment 701. Applications can be provided as multiple services 720-1, 720-2, 720-3, and 720-4 within computing environment 701, representing services "A", "D", "E", and "H", respectively.

[0209] End user 702 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 702 can be provided to ingress gateway 730. Ingress gateway 730 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 701).

[0210] The service is provided or deployed based on various virtualization technologies supported by the computing environment 701. In some embodiments, the service may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0211] In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, such as Figure 7 As shown, service 720-2 can be equipped with one or more Pods 740-1, 740-2, ..., 740-N (collectively referred to as Pods). A Pod can include a proxy 745 and one or more containers 742-1, 742-2, ..., 742-M (collectively referred to as containers). One or more containers within a Pod handle requests related to one or more corresponding functions of the service. The proxy 745 typically controls service-related network functions such as routing and load balancing. Other services can also be equipped with Pods similar to Pods.

[0212] During operation, executing a user request from end user 702 may require calling one or more services in computing environment 701, and executing one or more functions of one service may require calling one or more functions of another service. For example... Figure 7 As shown, service "A" 720-1 receives user requests from terminal user 702 from ingress gateway 730. Service "A" 720-1 can call service "D" 720-2, and service "D" 720-2 can request service "E" 720-3 to perform one or more functions.

[0213] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0214] According to embodiments of this application, a method for implementing the above is also provided. Figure 2 The image processing apparatus shown is an image processing method.

[0215] Figure 8 This is a schematic diagram of an image processing apparatus according to an embodiment of this application, such as... Figure 8 As shown, the image processing device 800 may include: a first display unit 802 and a second display unit 804.

[0216] The first display unit 802 is used to respond to input commands applied to the operation interface and display the original image and prompt information on the operation interface, wherein the prompt information is used to describe the processing performed on the original image.

[0217] The second display unit 804 is used to respond to the processing instructions applied to the operation interface and display the target image corresponding to the prompt information on the operation interface. The target image is obtained by using a target mask to guide the image denoising model to perform image editing on the original image and the prompt information. The target mask is used to represent the mask that meets the preset mask filtering conditions.

[0218] It should be noted that the first display unit 802 and the second display unit 804 mentioned above correspond to steps S202 to S204. The two units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed above. It should be noted that the above units can be hardware components or software components stored in memory (e.g., memory 1104) and processed by one or more processors (e.g., processors 1102a, 1102b, ..., 1102n). The above units can also be part of a device and run in computer terminal A.

[0219] In this image processing device, the first display unit 802 responds to input commands applied to the operation interface, displaying the original image and prompt information on the interface. The prompt information describes the processing performed on the original image. The second display unit 804 responds to processing commands applied to the operation interface, displaying the target image corresponding to the prompt information. The target image is obtained by using a target mask-guided image denoising model to edit the original image and prompt information. The target mask represents a mask that meets preset mask filtering conditions, thereby improving the image processing effect and solving the technical problem of poor image processing results.

[0220] According to embodiments of this application, a method for implementing the above is also provided. Figure 3 The video processing device shown is a video processing method.

[0221] Figure 9 This is a schematic diagram of a video processing apparatus according to an embodiment of this application, such as... Figure 9 As shown, the video processing device 900 may include a third display unit 902 and a fourth display unit 904.

[0222] The third display unit 902 is used to respond to input commands applied to the operation interface and display the original video and prompt information on the operation interface, wherein the prompt information is used to describe the processing performed on the original video.

[0223] The fourth display unit 904 is used to respond to the processing instructions applied to the operation interface and display the target video corresponding to the prompt information on the operation interface. The target video is obtained by using a target mask to guide a video denoising model to perform video editing on the original video and the prompt information. The target mask is used to represent the mask that meets the preset mask filtering conditions.

[0224] Here, the third display unit 902 and the fourth display unit 904 correspond to steps S302 to S304. The two units are implemented in the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed above. It should be noted that the above units can be hardware or software components stored in memory (e.g., memory 1104) and processed by one or more processors (e.g., processors 1102a, 1102b, ..., 1102n). The above units can also be run as part of a device in computer terminal A.

[0225] In this video processing device, the third display unit 902 responds to input commands applied to the operation interface, displaying the original video and prompt information on the interface. The prompt information describes the processing performed on the original video. The fourth display unit 904 responds to processing commands applied to the operation interface, displaying the target video corresponding to the prompt information. The target video is obtained by using a target mask to guide a video denoising model to edit the original video and prompt information. The target mask represents a mask that meets preset mask filtering conditions, thereby improving the image processing effect and solving the technical problem of poor image processing results.

[0226] According to an embodiment of this application, another method for implementing the above is also provided. Figure 4 The image processing apparatus shown is an image processing method.

[0227] Figure 10 This is a schematic diagram of another image processing apparatus according to an embodiment of this application, such as... Figure 10 As shown, the image processing device 1000 may include: a first acquisition unit 1002, a generation unit 1004, an editing unit 1006, and an output unit 1008.

[0228] The first acquisition unit 1002 is used to acquire the original image and prompt information by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the original image and the prompt information, and the prompt information is used to describe the processing performed on the original image.

[0229] The generation unit 1004 is used to generate a target mask that meets the preset mask filtering conditions.

[0230] The editing unit 1006 is used to guide the image denoising model with the target mask to perform image editing on the original image and the prompt information, and generate the target image corresponding to the prompt information.

[0231] The output unit 1008 is used to output the target image by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0232] It should be noted that the first acquisition unit 1002, generation unit 1004, editing unit 1006, and output unit 1008 correspond to steps S402 to S408. The four units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed above. It should be noted that the above units can be hardware or software components stored in memory (e.g., memory 1104) and processed by one or more processors (e.g., processors 1102a, 1102b, ..., 1102n). The above units can also be part of a device and run in computer terminal A.

[0233] In this image processing device, the first acquisition unit 1002 acquires the original image and prompt information by calling a first interface. The first interface includes a first parameter, the value of which includes the original image and the prompt information, which describes the processing performed on the original image. The generation unit 1004 generates a target mask that meets preset mask filtering conditions. The editing unit 1006 uses the target mask to guide an image denoising model to edit the original image and the prompt information, generating the target image corresponding to the prompt information. The output unit 1008 outputs the target image by calling a second interface. The second interface includes a second parameter, the value of which includes the target image. This improves the image processing effect and solves the technical problem of poor image processing quality.

[0234] Embodiments of this application may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced by a mobile terminal or other terminal device.

[0235] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0236] In this embodiment, the computer terminal described above can execute the program code for the following steps in the image processing method: responding to an input command applied to the operation interface, displaying the original image and prompt information on the operation interface, wherein the prompt information is used to describe the processing performed on the original image; responding to a processing command applied to the operation interface, displaying the target image corresponding to the prompt information on the operation interface, wherein the target image is obtained by using a target mask to guide an image denoising model to perform image editing on the original image and prompt information, and the target mask is used to characterize a mask that meets preset mask filtering conditions.

[0237] Optionally, Figure 11 This is a structural block diagram of a computer terminal according to an embodiment of this application, such as... Figure 11 As shown, the computer terminal A may include one or more (only one is shown in the figure) processors 1102, memory 1104, and transmission devices 1106.

[0238] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0239] Optionally, the processor may also execute program code for the following steps: during the processing of the original image and prompt information using the image inversion model, acquiring multiple attention features output by multiple cross-attention layers of the image inversion model at multiple time steps; generating multiple initial masks based on the object information in the prompt information and the multiple attention features, wherein different initial masks are used to characterize the position of the object feature corresponding to the object information in the corresponding attention feature; determining the target layer number and target time step corresponding to the image denoising model, wherein the target layer number is used to characterize the number of cross-attention layers corresponding to the mask that satisfies the preset mask selection conditions, and the target time step is used to characterize the time step corresponding to the mask that satisfies the preset mask selection conditions; and determining the target mask from the multiple initial masks based on the target layer number and target time step.

[0240] Optionally, the processor may also execute program code that performs the following steps: extracting multiple sub-features corresponding to the object information from multiple attention features based on the position of the object information in the prompt information; performing binarization processing on the multiple sub-features based on the threshold hyperparameter to obtain the processed sub-features; and performing shaping processing on the processed sub-features according to the resolution of the original image to generate multiple initial masks.

[0241] Optionally, the processor may also execute program code that performs the following steps: determining the processing type corresponding to the prompt information, wherein the processing type is used to characterize whether the shape of the object corresponding to the object information in the original image is changed; if the processing type characterizes that the shape of the object is changed, determining the target layer number and the mask corresponding to the target time step from multiple initial masks to obtain the target mask; if the processing type characterizes that the shape of the object is not changed, determining the mask corresponding to the target layer number from multiple initial masks to obtain the target mask.

[0242] Optionally, the processor may also execute program code that performs the following steps: acquiring a dataset, wherein the dataset contains a preset image, preset prompt information, and a preset mask; determining multiple preset attention features output by multiple cross-attention layers at multiple time steps during the processing of the preset image and preset prompt information using an image inversion model; generating multiple candidate masks based on the multiple preset attention features and preset prompt information; and determining the target layer number and target time step based on the multiple candidate masks and the preset mask.

[0243] Optionally, the processor may also execute program code that performs the following steps: obtaining performance metrics for multiple candidate masks based on the degree of overlap between multiple candidate masks and a preset mask, wherein different candidate masks correspond to different numbers of layers and / or time steps; summarizing the performance metrics of candidate masks corresponding to the same number of layers to obtain summarizing metrics for multiple numbers of layers, and summarizing the performance metrics of candidate masks corresponding to the same time step to obtain summarizing metrics for multiple time steps; determining the target number of layers based on the summarizing metrics for multiple numbers of layers, and obtaining the target time step based on the summarizing metrics for multiple time steps.

[0244] Optionally, the processor may also execute program code for the following steps: the image denoising model includes multiple types of attention layers, including at least two of the following types: cross-attention, self-attention, and temporal attention; the image inversion model is used to invert the original image and prompt information to obtain the image information of the original image; during the image editing process of the original image and prompt information using the image denoising model, the first attention feature output by each attention layer is obtained; during the image editing process of the image information and prompt information using the image denoising model, the second attention feature output by each attention layer is obtained; the target mask guides the image denoising model to perform feature fusion on the first and second attention features output by the same attention layer to obtain the fused feature corresponding to the same attention layer, wherein the fused feature corresponding to the last attention layer is the target feature output by the image denoising model; the target feature is used to reconstruct the image to generate the target image.

[0245] Optionally, the processor may also execute program code that performs the following steps: expands the target mask according to the spatial dimension to obtain the expanded mask; determines the first key vector and the first query vector corresponding to the first attention feature output by the same attention layer, and the second key vector and the second query vector corresponding to the second attention feature output by the same attention layer; and fuses the first key vector, the first query vector, the second key vector and the second query vector using the expanded mask to obtain the fused feature.

[0246] Optionally, the processor may also execute program code that performs the following steps: expands the target mask according to the spatial dimension to obtain the expanded mask; and uses the expanded mask to fuse the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature.

[0247] Optionally, the processor may also execute program code that performs the following steps: expands the target mask according to the spatial dimension to obtain the expanded mask; and uses the expanded mask to fuse the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature.

[0248] Optionally, the processor may also execute program code that performs the following steps: displaying identification information of at least one preset denoising model on the operation interface; and displaying identification information of the image denoising model corresponding to the selection instruction on the operation interface in response to a selection instruction applied to the identification information, wherein at least one preset denoising model includes an image denoising model.

[0249] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: responding to input commands applied to the operation interface, displaying the original video and prompt information on the operation interface, wherein the prompt information describes the processing performed on the original video; responding to processing commands applied to the operation interface, displaying the target video corresponding to the prompt information on the operation interface, wherein the target video is obtained by using a target mask to guide a video denoising model to perform video editing on the original video and prompt information, and the target mask is used to characterize a mask that meets preset mask filtering conditions.

[0250] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring the original image and prompt information by calling a first interface, wherein the first interface includes a first parameter, the parameter value of which includes the original image and the prompt information, and the prompt information is used to describe the processing performed on the original image; generating a target mask that meets preset mask filtering conditions; using the target mask to guide an image denoising model to perform image editing on the original image and the prompt information to generate a target image corresponding to the prompt information; and outputting the target image by calling a second interface, wherein the second interface includes a second parameter, the parameter value of which includes the target image.

[0251] Those skilled in the art will understand that Figure 11 The structure shown is for illustrative purposes only. Computer terminal A can also be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, PDA, mobile Internet device (MID), personal access display (PAD), or other terminal device. Figure 11 This does not limit the structure of the aforementioned computer terminal A. For example, computer terminal A may also include components that are more complex than those described above. Figure 11 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 11 The different configurations shown.

[0252] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0253] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the image processing method provided in Embodiment 1.

[0254] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0255] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: in response to an input command applied to the operation interface, displaying an original image and prompt information on the operation interface, wherein the prompt information describes the processing performed on the original image; in response to a processing command applied to the operation interface, displaying a target image corresponding to the prompt information on the operation interface, wherein the target image is obtained by using a target mask to guide an image denoising model to perform image editing on the original image and prompt information, and the target mask is a mask that satisfies preset mask filtering conditions and is generated based on multiple attention features of the image inversion model. The multiple attention features are output by multiple cross-attention layers of the image inversion model at multiple time steps during the processing of the original image and prompt information using the image inversion model.

[0256] Optionally, the aforementioned computer-readable storage medium may also execute program code that performs the following steps: during the processing of the original image and prompt information using an image inversion model, acquiring multiple attention features output by multiple cross-attention layers of the image inversion model at multiple time steps; generating multiple initial masks based on object information in the prompt information and multiple attention features, wherein different initial masks are used to characterize the position of object features corresponding to object information in corresponding attention features; determining the target layer number and target time step corresponding to the image denoising model, wherein the target layer number is used to characterize the number of cross-attention layers corresponding to the mask that satisfies the preset mask selection conditions, and the target time step is used to characterize the time step corresponding to the mask that satisfies the preset mask selection conditions; and determining the target mask from multiple initial masks based on the target layer number and target time step.

[0257] Optionally, the aforementioned computer-readable storage medium may also execute program code that performs the following steps: extracting multiple sub-features corresponding to the object information from multiple attention features based on the position of the object information in the prompt information; performing binarization processing on the multiple sub-features based on the threshold hyperparameter to obtain the processed sub-features; and performing shaping processing on the processed sub-features according to the resolution of the original image to generate multiple initial masks.

[0258] Optionally, the aforementioned computer-readable storage medium may also execute program code that performs the following steps: determining the processing type corresponding to the prompt information, wherein the processing type is used to characterize whether the shape of the object corresponding to the object information in the original image is changed; if the processing type characterizes that the shape of the object is changed, determining the target layer number and the mask corresponding to the target time step from multiple initial masks to obtain the target mask; if the processing type characterizes that the shape of the object is not changed, determining the mask corresponding to the target layer number from multiple initial masks to obtain the target mask.

[0259] Optionally, the aforementioned computer-readable storage medium may also execute program code that performs the following steps: acquiring a dataset, wherein the dataset contains a preset image, preset prompt information, and a preset mask; determining multiple preset attention features output by multiple cross-attention layers at multiple time steps during the processing of the preset image and preset prompt information using an image inversion model; generating multiple candidate masks based on the multiple preset attention features and preset prompt information; and determining the target layer number and target time step based on the multiple candidate masks and the preset mask.

[0260] Optionally, the aforementioned computer-readable storage medium may also execute program code that performs the following steps: obtaining performance metrics for multiple candidate masks based on the degree of overlap between multiple candidate masks and a preset mask, wherein different candidate masks correspond to different numbers of layers and / or time steps; summarizing the performance metrics of candidate masks corresponding to the same number of layers to obtain summarizing metrics for multiple numbers of layers, and summarizing the performance metrics of candidate masks corresponding to the same time step to obtain summarizing metrics for multiple time steps; determining the target number of layers based on the summarizing metrics for multiple numbers of layers, and obtaining the target time step based on the summarizing metrics for multiple time steps.

[0261] Optionally, the aforementioned computer-readable storage medium may also execute program code containing the following steps: An image denoising model includes multiple types of attention layers, including at least two of the following types: cross-attention, self-attention, and temporal attention; an image inversion model is used to invert the original image and prompt information to obtain the image information of the original image; during image editing of the original image and prompt information using the image denoising model, a first attention feature output by each attention layer is obtained; during image editing of the image information and prompt information using the image denoising model, a second attention feature output by each attention layer is obtained; a target mask is used to guide the image denoising model to fuse the first and second attention features output by the same attention layer to obtain a fused feature corresponding to the same attention layer, wherein the fused feature corresponding to the last attention layer is the target feature output by the image denoising model; image reconstruction is performed on the target feature to generate a target image.

[0262] Optionally, the computer-readable storage medium may also execute program code that performs the following steps: expanding the target mask according to the spatial dimension to obtain the expanded mask; determining the first key vector and the first query vector corresponding to the first attention feature output by the same attention layer, and the second key vector and the second query vector corresponding to the second attention feature output by the same attention layer; and fusing the first key vector, the first query vector, the second key vector, and the second query vector using the expanded mask to obtain the fused feature.

[0263] Optionally, the computer-readable storage medium may also execute program code that performs the following steps: unfolding the target mask according to the spatial dimension to obtain the unfolded mask; using the unfolded mask to fuse the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature.

[0264] Optionally, the computer-readable storage medium may also execute program code that performs the following steps: unfolding the target mask according to the spatial dimension to obtain the unfolded mask; using the unfolded mask to fuse the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature.

[0265] Optionally, the computer-readable storage medium may also execute program code that performs the following steps: displaying identification information of at least one preset denoising model on an operation interface; and displaying identification information of an image denoising model corresponding to a selection instruction on the operation interface in response to a selection instruction applied to the identification information, wherein at least one preset denoising model includes an image denoising model.

[0266] As an optional example, the computer-readable storage medium is configured to store program code for performing the following steps: in response to an input command applied to an interface, displaying an original video and a prompt message on the interface, wherein the prompt message describes the processing performed on the original video; in response to a processing command applied to the interface, displaying a target video corresponding to the prompt message on the interface, wherein the target video is obtained by using a target mask to guide a video denoising model to perform video editing on the original video and the prompt message, and the target mask is used to characterize a mask that meets preset mask filtering conditions.

[0267] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: acquiring an original image and prompt information by calling a first interface, wherein the first interface includes a first parameter, the value of which includes the original image and the prompt information, the prompt information describing the processing performed on the original image; generating a target mask that meets preset mask filtering conditions; using the target mask to guide an image denoising model to perform image editing on the original image and the prompt information, generating a target image corresponding to the prompt information; and outputting the target image by calling a second interface, wherein the second interface includes a second parameter, the value of which includes the target image.

[0268] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0269] Optionally, the computer program product described above may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the method provided in the above embodiments.

[0270] Embodiments of this application may provide an electronic device that may include a memory and a processor.

[0271] Figure 12 This is a block diagram of an electronic device for an image processing method according to an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0272] like Figure 12As shown, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1202 or a computer program loaded from storage unit 1208 into random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. An input / output (I / O) interface 1205 is also connected to bus 1204.

[0273] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0274] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by computing unit 1201, one or more steps of the image processing method described above can be performed. Alternatively, in other embodiments, computing unit 1201 can be configured to perform the image processing method by any other suitable means (e.g., by means of firmware).

[0275] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0276] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0277] The method embodiment provided in Embodiment 1 of this application can be executed in a mobile terminal, computer terminal or similar computing device. Figure 13 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method according to an embodiment of this application, such as... Figure 13As shown, the computer terminal 130 (or mobile device) may include one or more processors 1302 (shown as 1302a, 1302b, ..., 1302n in the figure) 1302 (processor 1302 may include, but is not limited to, a microprocessor (MCU) or a programmable gate array (FPGA), etc.), a memory 1304 for storing data, and a transmission device 1306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 13 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the computer terminal 130 may also include... Figure 13 The more or fewer components shown, or having the same Figure 13 The different configurations shown.

[0278] Figure 13 The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned computer terminal 130 (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 13 The use of the above is illustrated in a block diagram. Figure 13 The computer terminal 130 (or mobile device) shown is an embodiment of a computing node in computing environment 701.

[0279] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by an instruction execution system, apparatus, or device, or in combination with such an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0280] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD)) for displaying information to the user, a monitor; and a keyboard and pointing device (e.g., a mouse or pathball) through which the user provides input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0281] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0282] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0283] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0284] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0285] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0286] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0287] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0288] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory, random access memory, portable hard drive, magnetic disk, or optical disk.

[0289] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, include: In response to an input command applied to the user interface, the original image and prompt information are displayed on the user interface, wherein the prompt information describes the processing performed on the original image; In response to the processing instructions applied to the operation interface, the target image corresponding to the prompt information is displayed on the operation interface. The target image is obtained by using a target mask-guided image denoising model to perform image editing on the original image and the prompt information. The target mask is used to represent a mask that meets preset mask filtering conditions.

2. The method according to claim 1, characterized in that, In response to a processing instruction applied to the user interface, the method further includes: During the process of processing the original image and the prompt information using the image inversion model, multiple attention features output by multiple cross-attention layers of the image inversion model at multiple time steps are obtained; Based on the object information in the prompt information and the multiple attention features, multiple initial masks are generated, wherein different initial masks are used to characterize the position of the object feature corresponding to the object information in the corresponding attention feature; The target number of layers and the target time step corresponding to the image denoising model are determined, wherein the target number of layers is used to characterize the number of cross-attention layers corresponding to the mask that satisfies the preset mask filtering conditions, and the target time step is used to characterize the time step corresponding to the mask that satisfies the preset mask filtering conditions. The target mask is determined from the plurality of initial masks based on the target layer number and the target time step.

3. The method according to claim 2, characterized in that, The generation of multiple initial masks based on the object information in the prompt information and the multiple attention features includes: Based on the position of the object information in the prompt information, multiple sub-features corresponding to the object information are extracted from the multiple attention features; The multiple sub-features are binarized based on the threshold hyperparameter to obtain the processed sub-features. The processed sub-features are reshaped according to the resolution of the original image to generate the plurality of initial masks.

4. The method according to claim 2, characterized in that, The step of determining the target mask from the plurality of initial masks based on the target layer number and the target time step includes: Determine the processing type corresponding to the prompt information, wherein the processing type is used to characterize whether to change the shape of the object corresponding to the object information in the original image; When the processing type characterization changes the shape of the object, the target layer number and the mask corresponding to the target time step are determined from the plurality of initial masks to obtain the target mask; If the processing type characterization does not change the shape of the object, the target mask is obtained by determining the mask corresponding to the target layer number from the plurality of initial masks.

5. The method according to claim 2, characterized in that, Determining the target layer number and target time step corresponding to the image denoising model includes: Obtain a dataset, wherein the dataset contains a preset image, a preset prompt message, and a preset mask; During the process of processing the preset image and the preset prompt information using the image inversion model, multiple preset attention features output by the multiple cross-attention layers at multiple time steps are determined; Based on the multiple preset attention features and the preset prompt information, multiple candidate masks are generated; Based on the multiple candidate masks and the preset mask, the target layer number and the target time step are determined.

6. The method according to claim 5, characterized in that, The step of determining the target layer number and the target time step based on the plurality of candidate masks and the preset mask includes: Based on the degree of overlap between the multiple candidate masks and the preset mask, the performance index of the multiple candidate masks is obtained, wherein the number of layers and / or time steps corresponding to different candidate masks are different; The performance metrics of candidate masks corresponding to the same layer number are summarized to obtain the summary metrics corresponding to multiple layers, and the performance metrics of candidate masks corresponding to the same time step are summarized to obtain the summary metrics corresponding to multiple time steps. Based on the aggregated indicators corresponding to the multiple layers, the target layer is determined, and based on the aggregated indicators corresponding to the multiple time steps, the target time step is obtained.

7. The method according to claim 1, characterized in that, The image denoising model includes multiple types of attention layers, including at least two of the following: cross attention, self attention, and temporal attention; In response to a processing instruction applied to the user interface, the method further includes: The image inversion model is used to invert the original image and the prompt information to obtain the image information of the original image; During the image editing process of the original image and the prompt information using the image denoising model, the first attention feature output by each attention layer is obtained; During the process of image editing of the image information and the prompt information using the image denoising model, the second attention feature output by each attention layer is obtained; The target mask is used to guide the image denoising model to fuse the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature corresponding to the same attention layer. The fused feature corresponding to the last attention layer is the target feature output by the image denoising model. The target features are reconstructed to generate the target image.

8. The method according to claim 7, characterized in that, When the same attention layer is a temporal attention layer, the step of using the target mask to guide the image denoising model to fuse the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature corresponding to the same attention layer includes: The target mask is expanded according to the spatial dimension to obtain the expanded mask; Determine the first key vector and the first query vector corresponding to the first attention feature output by the same attention layer, and the second key vector and the second query vector corresponding to the second attention feature output by the same attention layer; The first key vector, the first query vector, the second key vector, and the second query vector are fused using the expanded mask to obtain the fused feature.

9. The method according to claim 7, characterized in that, When the same attention layer is a cross-attention layer, the step of using the target mask to guide the image denoising model to fuse the first attention features and the second attention features output by the same attention layer to obtain the fused features corresponding to the same attention layer includes: The target mask is expanded according to the spatial dimension to obtain the expanded mask; The first attention feature and the second attention feature output by the same attention layer are fused using the expanded mask to obtain the fused feature.

10. The method according to claim 7, characterized in that, When the same attention layer is a self-attention layer, the step of using the target mask to guide the image denoising model to fuse the first attention feature and the second attention feature output by the same attention layer to obtain the fused feature corresponding to the same attention layer includes: The target mask is expanded according to the spatial dimension to obtain the expanded mask; The first attention feature and the second attention feature output by the same attention layer are fused using the expanded mask to obtain the fused feature.

11. The method according to any one of claims 1 to 10, characterized in that, In response to a determination command applied to the user interface, the method further includes: The user interface displays the identification information of at least one preset denoising model; In response to a selection command applied to the identification information, the identification information of the image denoising model corresponding to the selection command is displayed on the operation interface, wherein the at least one preset denoising model includes the image denoising model.

12. A video processing method, characterized in that, include: In response to input commands applied to the operation interface, the original video and prompt information are displayed on the operation interface, wherein the prompt information is used to describe the processing performed on the original video; In response to the processing command applied to the operation interface, the target video corresponding to the prompt information is displayed on the operation interface. The target video is obtained by using a target mask to guide a video denoising model to perform video editing on the original video and the prompt information. The target mask is used to represent a mask that meets preset mask filtering conditions.

13. An image processing method, characterized in that, include: The original image and prompt information are obtained by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the original image and the prompt information, and the prompt information is used to describe the processing performed on the original image; Generate a target mask that meets the preset mask filtering conditions; The target mask guides the image denoising model to perform image editing on the original image and the prompt information, generating the target image corresponding to the prompt information; The target image is output by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

14. A computer terminal, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 13.

16. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 13.

Images