Image processing method and image processing model training method

Abstract

The embodiments of this specification provide an image processing method and a training method for an image processing model, the image processing method comprising: obtaining an image to be processed and preset empty text, the image to be processed comprising at least one group of point pairs to be processed, each group of point pairs to be processed comprising a starting point and a target point; inputting the image to be processed and the preset empty text into the image processing model, obtaining a target image corresponding to the image to be processed, the target image being obtained based on the noise feature data to be processed and the updated text feature data of the image to be processed, the updated text feature data being obtained based on the update of the empty text feature data by the feature data of the point to be processed, the feature data of the point to be processed being used to characterize the positional relationship between the starting point and the target point in each group of point pairs to be processed. By obtaining the image to be processed and the preset empty text, in the process of image processing the image to be processed, the image to be processed can be processed according to the starting point and the target point in the image to be processed, thereby improving the accuracy of image processing.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to an image processing method and a training method for an image processing model. Background Art

[0002] In current practical applications, image processing technology has been gradually applied to various technical fields. More and more people are beginning to use image editing software to edit images. Specifically, users can manually select the area that needs to be processed and then process the image area. However, this image processing method often results in the editing results not meeting user expectations, for example, erroneous processing of image areas not selected by users, etc., resulting in low accuracy of image processing. Therefore, a method is urgently needed to solve the problem of low accuracy of image processing. Summary of the invention

[0003] In view of this, the embodiments of this specification provide an image processing method and a training method for an image processing model. One or more embodiments of this specification also relate to an image processing device and a training device for an image processing model, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, there is provided an image processing method, including:

[0005] Acquire an image to be processed and preset empty text, wherein the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point;

[0006] The image to be processed and the preset empty text are input into an image processing model to obtain a target image corresponding to the image to be processed, wherein the target image is obtained based on the processed noise-added feature data and updated text feature data of the image to be processed, and the updated text feature data is obtained by updating the empty text feature data based on the feature data of the point to be processed, and the feature data of the point to be processed is used to characterize the positional relationship between the starting point and the target point in each group of points to be processed.

[0007] According to a second aspect of the embodiments of this specification, there is provided an image processing apparatus, including:

[0008] An acquisition module is configured to acquire an image to be processed and preset empty text, wherein the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point;

[0009] An input module is configured to input the image to be processed and the preset empty text into an image processing model to obtain a target image corresponding to the image to be processed, wherein the target image is obtained based on the processed noise-added feature data and updated text feature data of the image to be processed, and the updated text feature data is obtained by updating the empty text feature data based on the feature data of the point to be processed, and the feature data of the point to be processed is used to characterize the positional relationship between the starting point and the target point in each group of points to be processed.

[0010] According to a third aspect of an embodiment of this specification, a training method for an image processing model is provided, which is applied to a cloud-side device, including:

[0011] Acquire a sample training image and a sample empty text, wherein the sample training image includes a sample to-be-processed image and a sample target image corresponding to the sample to-be-processed image, the sample to-be-processed image includes at least one group of sample to-be-processed point pairs, and each group of sample to-be-processed point pairs includes a sample starting point and a sample target point;

[0012] Input the sample to-be-processed image and the sample empty text into an image processing model to obtain a predicted image, wherein the predicted image is obtained based on sample noise-added feature data and sample updated text feature data of the sample to-be-processed image, the sample updated text feature data is obtained by updating the sample empty text feature data based on the sample to-be-processed point feature data, and the sample to-be-processed point feature data is used to characterize the positional relationship between the sample starting point and the sample target point in each group of sample to-be-processed point pairs;

[0013] Calculating a loss value of the image processing model according to the sample target image and the predicted image;

[0014] Adjusting the model parameters of the image processing model according to the loss value, and continuing to train the image processing model until a model training stop condition is reached;

[0015] The model parameters of the trained image processing model are fed back to the target terminal.

[0016] According to a fourth aspect of an embodiment of this specification, a training device for an image processing model is provided, which is applied to a cloud-side device, including:

[0017] A sample acquisition module is configured to acquire a sample training image and a sample empty text, wherein the sample training image includes a sample to-be-processed image and a sample target image corresponding to the sample to-be-processed image, the sample to-be-processed image includes at least one group of sample to-be-processed point pairs, and each group of sample to-be-processed point pairs includes a sample starting point and a sample target point;

[0018] A sample input module is configured to input the sample to-be-processed image and the sample empty text into an image processing model to obtain a predicted image, wherein the predicted image is obtained based on sample noise-added feature data and sample updated text feature data of the sample to-be-processed image, the sample updated text feature data is obtained by updating the sample empty text feature data based on the sample to-be-processed point feature data, and the sample to-be-processed point feature data is used to characterize the positional relationship between the sample starting point and the sample target point in each group of sample to-be-processed point pairs;

[0019] A calculation module, configured to calculate a loss value of the image processing model according to the sample target image and the predicted image;

[0020] A training module, configured to adjust a model parameter of the image processing model according to the loss value, and continue to train the image processing model until a model training stop condition is reached;

[0021] The feedback module is configured to feed back the model parameters of the trained image processing model to the target terminal.

[0022] According to a fifth aspect of the embodiments of this specification, another image processing method is provided, including:

[0023] Receiving an image processing instruction sent by a target terminal, wherein the image processing instruction carries an image to be processed, and the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point;

[0024] In response to the image processing instruction, obtaining a preset empty text;

[0025] Input the image to be processed and the preset empty text into the image processing model to obtain a target image corresponding to the image to be processed, wherein the target image is obtained based on the processed noise feature data and the updated text feature data of the image to be processed, and the updated text feature data is obtained by updating the empty text feature data based on the feature data of the point to be processed, and the feature data of the point to be processed is used to characterize the positional relationship between the starting point and the target point in each group of the point to be processed;

[0026] The target image is sent to the target terminal.

[0027] According to a sixth aspect of the embodiments of this specification, another image processing device is provided, including:

[0028] A receiving module is configured to receive an image processing instruction sent by a target terminal, wherein the image processing instruction carries an image to be processed, and the image to be processed includes at least one group of point pairs to be processed, each group of point pairs to be processed includes a starting point and a target point;

[0029] A response module, configured to obtain preset empty text in response to the image processing instruction;

[0030] An acquisition module is configured to input the image to be processed and the preset empty text into an image processing model to obtain a target image corresponding to the image to be processed, wherein the target image is obtained based on the noise-added feature data to be processed and the updated text feature data of the image to be processed, the updated text feature data is obtained by updating the empty text feature data based on the feature data of the point to be processed, and the feature data of the point to be processed is used to characterize the positional relationship between the starting point and the target point in each group of the point to be processed;

[0031] The sending module is configured to send the target image to the target terminal.

[0032] According to a seventh aspect of an embodiment of this specification, there is provided a computing device, including:

[0033] Memory and processor;

[0034] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above-mentioned image processing method or the training method of the image processing model are implemented.

[0035] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned image processing method or the image processing model training method are implemented.

[0036] According to the ninth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned image processing method or image processing model training method.

[0037] An embodiment of the present specification realizes that, by acquiring an image to be processed and preset empty text, the image processing model can process the image to be processed according to the starting point and target point in the image to be processed during the image processing of the image to be processed, realize accurate positioning of the image processing area of ​​the image to be processed, and improve the accuracy of image processing. In the process of image processing of the image to be processed, the image processing model can update the empty text feature data of the preset empty text based on the feature data of the point to be processed to obtain the updated text feature data, so that the noise-added feature data to be processed of the image to be processed can be denoised according to the updated text feature data to obtain the target image corresponding to the image to be processed. The empty text feature data is updated by the feature data of the point to be processed to realize the combination of image features and text features. Since the text features contain semantic information, image processing of the image to be processed based on the semantic information can be realized, and the image area without semantic information will not be processed, thereby reducing the misprocessing of the image to be processed and improving the accuracy of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of an application scenario of an image processing method provided by an embodiment of this specification;

[0039] Figure 2 is a flow chart of an image processing method provided by an embodiment of this specification;

[0040] Figure 3 is a schematic diagram of an image to be processed provided by an embodiment of this specification;

[0041] Figure 4 is an architecture diagram of an image processing model provided by an embodiment of this specification;

[0042] Figure 5 It is a schematic diagram of obtaining updated text feature data provided by an embodiment of this specification;

[0043] Figure 6 is a processing flow chart of an image processing method provided by an embodiment of this specification;

[0044] Figure 7 is a structural schematic diagram of an image processing device provided by an embodiment of this specification;

[0045] Figure 8 is a flowchart of a training method for an image processing model provided by an embodiment of this specification;

[0046] Fig. 9 It is a structural schematic diagram of a training device for an image processing model provided by an embodiment of this specification;

[0047] Fig.10 is a flow chart of another image processing method provided by an embodiment of this specification;

[0048] Fig.11 is a schematic diagram of the structure of another image processing device provided by an embodiment of this specification;

[0049] Fig.12 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0050] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0051] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0052] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0053] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0054] First, the terms involved in one or more embodiments of this specification are explained.

[0055] Stab led iffusion: a vincent graph diffusion model.

[0056] DDIM: Deterministic Denois ing Diffus ion Implicit Model, deterministic denoising diffusion model.

[0057] DDIM Inversion: is an inverse process related to diffusion model-based image generation techniques. Specifically, DDIM Inversion involves reverse mapping a real-world image (or video) to the latent space of a diffusion model, i.e. finding the noise representation inside the model that corresponds to a given image.

[0058] In current practical applications, image processing technology has gradually been applied to various technical fields. More and more people are beginning to use image editing software to edit images. Specifically, users can manually select the area that needs to be processed and then process the image area. However, this image processing method often results in the editing results not meeting user expectations, resulting in low image processing accuracy.

[0059] In this specification, an image processing method and a training method for an image processing model are provided. This specification also relates to an image processing device and a training device for an image processing model, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0060] See also Figure 1 , Figure 1 The following is a schematic diagram showing an application scenario of an image processing method provided according to an embodiment of the present specification. The image processing method provided in the present specification can be applied to image editing scenarios, for example, Figure 1 As shown, before image processing, the person in the image needs to be moved from the solid line position to the dotted line position, and the image processing effect needs to be as follows Figure 1 The image after processing is shown in the figure. Figure 1 In the image processing scenario shown, in the process of moving the solid-line character, there may be position movement errors, or the solid-line character may be deformed during the movement.

[0061] In view of the above-mentioned process of misprocessing the image, in one or more embodiments provided in this specification, the image to be processed can be processed by determining the starting point and the target point in the image to be processed. At the same time, in order to avoid the image background and other areas that do not need to be processed from being operated, the image to be processed can be processed in combination with text data. In practical applications, since the user does not provide text data during the image processing process, it can be achieved by presetting empty text. Since there is no text data in the empty text, the image generated based on the empty text will be more random and uncertain. Therefore, in the process of processing the image to be processed based on the preset empty text, the preset empty text can be adjusted according to the starting point and the target point in the image to be processed to fill the text data in the preset empty text, so as to realize the processing of the image to be processed based on the text data, improve the accuracy of the image processing of the image to be processed, and improve the processing effect of the image after image processing.

[0062] See also Figure 2 , Figure 2 A flowchart of an image processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0063] Step 202: Acquire an image to be processed and preset empty text, wherein the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point.

[0064] The image to be processed refers to any image that needs to be processed. For example, if it is necessary to translate object P1 in image P, or rotate object P2 (such as 3D rotation), or scale object P3, or scale or reshape image P, then image P is the image to be processed.

[0065] The preset empty text refers to a preset empty text used to guide the image processing of the image to be processed. In practical applications, the preset empty text can be placed by default by the execution terminal without the need for user provision. The image processing method provided in this specification can be applied to any terminal, which can be a server, a smart phone, a laptop computer, a tablet computer, etc. This specification does not limit the specific form of the terminal.

[0066] It should be noted that the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point. The starting point can be any point in the image to be processed; the target point refers to the position of the point to which the starting point needs to be moved. A starting point and a target point constitute a group of point pairs to be processed. For example, there is a point A in the image P, and point A needs to be moved to point A', then point A and point A' constitute a group of point pairs to be processed, and point A is the starting point in the point pair to be processed, and point A' is the target point in the point pair to be processed.

[0067] In practical applications, when the user provides the image to be processed, he needs to determine at least one set of point pairs to be processed in the image to be processed, so as to facilitate the subsequent processing of the image to be processed based on the image processing model and improve the accuracy of image processing.

[0068] See also Figure 3 , Figure 3 FIG. 1 shows a schematic diagram of an image to be processed according to an embodiment of the present specification. Figure 3 As shown, the person in the image to be processed needs to be moved from the solid line position to the dotted line position. In the processed image, there are three groups of points to be processed, namely, the points to be processed (A, A'), (B, B'), and (C, C'). In the subsequent process of processing the image to be processed, the image to be processed can be processed based on the above three groups of points to be processed.

[0069] Furthermore, after the user determines multiple groups of point pairs to be processed in the image to be processed, the user terminal will generate an image processing instruction for the image to be processed based on the user's processing operation, and send the image processing instruction to the execution terminal of the image processing method, so that after receiving the image processing instruction, the execution terminal obtains the image to be processed and the preset empty text, thereby performing image processing on the image to be processed based on the preset empty text and the image processing model.

[0070] The image processing method provided in this specification can obtain an image to be processed with multiple groups of point pairs to be processed and preset empty text, and in the subsequent process, the image processing model processes the image to be processed based on the point pairs to be processed and the preset empty text, thereby improving the accuracy of subsequent image processing.

[0071] Step 204: Input the image to be processed and the preset empty text into an image processing model to obtain a target image corresponding to the image to be processed, wherein the target image is obtained based on the processed noise-added feature data and updated text feature data of the image to be processed, and the updated text feature data is obtained by updating the empty text feature data based on the feature data of the point to be processed, and the feature data of the point to be processed is used to characterize the positional relationship between the starting point and the target point in each group of points to be processed.

[0072] In practical applications, after obtaining the image to be processed and the preset empty text, the image to be processed and the preset empty text can be input into the image processing model to use the image processing model to perform image processing on the image to be processed and generate a target image after the image processing is performed on the image to be processed.

[0073] The image processing model is used to perform image processing on the image to be processed, which can be specifically to move the starting point in each group of the image to be processed to the target point, thereby generating a target image. The image processing model can specifically be a diffusion model, a stable diffusion model, etc. The target image refers to the image obtained after the image processing is performed on the image to be processed. Figure 1 In the example of , the image before image processing is the image to be processed, and the image after image processing is the target image.

[0074] The noise-added feature data to be processed refers to the feature vector or feature matrix obtained after the noise is added to the image to be processed. The empty text feature data refers to the empty text encoding feature vector or feature matrix obtained after encoding the preset empty text; the updated text feature data refers to the feature vector or feature matrix after the empty text feature data is updated. The feature point data to be processed is used to characterize the positional relationship between the starting point and the target point in each group of points to be processed; the feature point data to be processed is determined based on the starting point feature data of the starting point and the target point feature data of the target point in each group of points to be processed. The starting point feature data refers to the feature vector or feature matrix corresponding to each starting point, that is, the vector form expression or matrix form expression of each starting point; the target point feature data refers to the feature vector or feature matrix corresponding to each target point, that is, the vector form expression or matrix form expression of each target point.

[0075] See also Figure 4 , Figure 4 FIG. 1 shows an architecture diagram of an image processing model provided according to an embodiment of the present specification. Figure 4As shown, the image processing model provided by one or more embodiments of the present specification may specifically include a text encoder, an image encoder, an image information creator and an image decoder, wherein the image information creator includes multiple attention modules. The text encoder is used to encode the preset empty text and generate empty text feature data. The image encoder is used to encode the image to be processed and generate the feature data of the image to be processed. The feature data of the image to be processed is the image encoding feature vector or feature matrix obtained after encoding the image to be processed. The image information creator is used to generate target denoising feature data; the target denoising feature data refers to the feature vector or feature matrix obtained after denoising the noisy feature data to be processed with the updated text feature data as the denoising condition. The image decoder is used to decode the target denoising feature data and generate a target image. Specifically, the implementation method of image processing of the image to be processed and the preset empty text based on the image processing model is as follows:

[0076] In a specific implementation provided in this specification, the image processing model includes a text encoder, an image encoder, and an image information creator;

[0077] Inputting the image to be processed and the preset empty text into an image processing model to obtain a target image corresponding to the image to be processed, including:

[0078] Inputting the preset empty text into the text encoder to obtain empty text feature data corresponding to the preset empty text;

[0079] Inputting the image to be processed into the image encoder to obtain feature data of the image to be processed corresponding to the image to be processed;

[0080] Noising the image feature data to be processed according to the empty text feature data to generate noisy feature data to be processed;

[0081] The empty text feature data and the to-be-processed noise feature data are input into the image information creator to obtain target denoising feature data, and a target image is obtained based on the target denoising feature data.

[0082] After obtaining the image to be processed and the preset empty text, the image to be processed and the preset empty text can be input into the image processing model for image processing to generate the target image corresponding to the image to be processed. Specifically, the preset empty text can be input into the text encoder of the image processing model to obtain the empty text feature data corresponding to the preset empty text, and the image to be processed can be input into the image encoder of the image processing model to obtain the image feature data to be processed corresponding to the image to be processed. After obtaining the empty text feature data and the image feature data to be processed, the image feature data to be processed is denoised with the empty text feature data as the guiding condition to generate the noisy feature data to be processed. In practical applications, the DDIM Invers ion algorithm can be used to perform data inversion on the image feature data to be processed to obtain the noise representation corresponding to the image feature data to be processed inside the image processing model. Then, the obtained noisy feature data to be processed is input into the image information creator of the image processing model to realize the denoising of the noisy feature data to be processed and obtain the target denoised feature data. In practical applications, the image information creator can be specifically a Unet network structure. After obtaining the target denoising feature data of the image to be processed, the target denoising feature data is input into the image decoder of the image processing model to obtain the target image corresponding to the image to be processed.

[0083] Furthermore, the image information creator generates target denoised feature data based on the empty text feature data and the noise feature data to be processed as follows:

[0084] In a specific implementation provided in this specification, the empty text feature data and the to-be-processed noise feature data are input into the image information creator to obtain the target noise removal feature data, including:

[0085] Inputting the noise-added feature data to be processed into the image information creator to generate starting point feature data of the starting point and target point feature data of the target point in each group of point pairs to be processed;

[0086] Determine the feature data of the to-be-determined starting point and the feature data of the to-be-determined target point of the to-be-determined processing point pair according to the feature data of the to-be-determined starting point and the feature data of the to-be-determined target point, wherein the to-be-determined processing point pair is any one of the groups of processing point pairs, the feature data of the to-be-determined starting point is the feature data of the starting point of the to-be-determined processing point pair, and the feature data of the to-be-determined target point is the feature data of the target point of the to-be-determined processing point pair;

[0087] According to the feature data to be determined, the empty text feature data is updated until the feature data to be processed completes the update of the empty text feature data, thereby obtaining updated text feature data;

[0088] Based on the updated text feature data, the to-be-processed noisy feature data is denoised to obtain target denoised feature data.

[0089] Specifically, after the feature data to be processed and noisy is input into the image information creator, the image information creator processes the feature data to be processed and noisy. Since the image to be processed includes various groups of point pairs to be processed, the feature data to be processed and noisy obtained by noisy the image to be processed also includes feature data corresponding to various groups of point pairs to be processed. After processing the feature data to be processed and noisy, the image information creator can generate starting point feature data corresponding to the starting point in each group of point pairs to be processed and target point feature data corresponding to the target point.

[0090] Further, based on the starting point feature data and the target point feature data in each group of to-be-processed point pairs, feature data corresponding to each group of to-be-processed point pairs can be generated. After obtaining the feature data corresponding to each group of to-be-processed point pairs, the empty text feature data is updated and adjusted according to the feature data corresponding to each group of to-be-processed point pairs, until the feature data corresponding to each group of to-be-processed point pairs, i.e., the to-be-processed point feature data, completes the update of the empty text feature data, and the updated and adjusted updated text feature data can be obtained. After obtaining the updated text feature data, the updated text feature data is used as a denoising guide condition for the subsequent denoising of the to-be-processed noisy feature data, and the to-be-processed noisy feature data is denoised to obtain the target denoised feature data.

[0091] In practical applications, since the image processing model needs to move the starting point of each group of processed point pairs in the image to be processed to the target point, in order to ensure the accuracy of moving the starting point to the target point, the movement can be based on the positional relationship between the starting point and the target point. Furthermore, the feature data of the point pair to be processed can be determined by determining the feature distance between the feature data of the starting point and the feature data of the target point.

[0092] Based on this, in a specific implementation provided in this specification, determining the feature data to be determined of the to-be-determined to-be-processed point pair according to the feature data of the to-be-determined starting point and the feature data of the to-be-determined target point includes:

[0093] Calculating a to-be-determined characteristic distance between the to-be-determined starting point characteristic data and the to-be-determined target point characteristic data;

[0094] The to-be-determined characteristic distance is determined as the to-be-determined characteristic data of the to-be-determined to-be-processed point pair.

[0095] The feature distance to be determined refers to the feature distance between the feature data of the starting point to be determined and the feature data of the target point to be determined. In practical applications, the feature distance to be determined may be Manhattan distance or Euclidean distance.

[0096] Specifically, the feature distance to be determined between the feature data of the starting point to be determined and the feature data of the target point to be determined is calculated, that is, the Manhattan distance or Euclidean distance between the feature vector or feature matrix of the starting point to be determined and the feature vector or feature matrix of the target point to be determined is calculated, and the calculated feature distance to be determined (Manhattan distance or Euclidean distance) is determined as the feature data to be determined of the pair of points to be processed.

[0097] See also Figure 5 , Figure 5 FIG. 2 shows a schematic diagram of obtaining updated text feature data according to an embodiment of the present specification. Figure 5 As shown, taking the point pair to be processed (A, A') as an example for explanation, after obtaining the feature data corresponding to the point pair to be processed (A, A'), that is, the starting point feature data corresponding to the starting point A and the target point feature data corresponding to the target point A', calculate the feature distance between the starting point feature data corresponding to the starting point A and the target point feature data corresponding to the target point A', and based on the feature distance, adjust the empty text feature data until the processing point feature data corresponding to the processing point pair is completed. For example, if there are N groups of point pairs to be processed, it is necessary to adjust and update the empty text feature data according to each group of processing point feature data, that is, adjust the empty text feature data N times to obtain the updated updated text feature data.

[0098] Since the feature data of the points to be processed already include the image information of each group of points to be processed in the image to be processed, the updated text feature data obtained based on the update of the feature data of the points to be processed will also include information about the semantic level of the image to be processed. Therefore, in order to enable the starting point to be accurately moved to the target point, the updated text feature data can be used as a guiding condition for the subsequent denoising of the noisy feature data to be processed, thereby realizing the denoising of the noisy features to be processed.

[0099] In a specific implementation provided in this specification, the image information creator includes an attention module;

[0100] Based on the updated text feature data, denoising the to-be-processed denoised feature data to obtain target denoised feature data, including:

[0101] The updated text feature data and the to-be-processed denoised feature data are input into the attention module to obtain target denoised feature data.

[0102] In practical applications, after obtaining the updated text feature data, the updated text feature data is used as the denoising condition to achieve denoising of the to-be-processed noisy feature data. In one or more embodiments of the present specification, in order to further improve the effect of the generated target image, an attention module is added to the image information creator, and the to-be-processed noisy feature data is denoised by the attention module to obtain the target denoised data. That is, after obtaining the updated text feature data, the updated text feature data and the to-be-processed noisy feature data are input into the attention module in the image information creator to obtain the target denoised feature data.

[0103] Furthermore, the implementation process of denoising the processed noisy feature data based on the attention module is as follows:

[0104] In a specific implementation provided in this specification, the updated text feature data and the to-be-processed noise-added feature data are input into the attention module to obtain the target noise-removed feature data, including:

[0105] Inputting the empty text feature data, the updated text feature data and the to-be-processed noise feature data into the attention module;

[0106] Determine first key feature data and first value feature data, and second key feature data and second value feature data, wherein the first key feature data and the first value feature data are respectively key feature data and value feature data used by the empty text feature data to perform attention calculation on the to-be-processed noise feature data, and the second key feature data and the second value feature data are respectively key feature data and value feature data used by the updated text feature data to perform attention calculation on the to-be-processed noise feature data;

[0107] replacing the second key feature data with the first key feature data, and replacing the second value feature data with the first value feature data;

[0108] According to the first key feature data and the first value feature data, attention calculation is performed on the updated text feature data and the to-be-processed denoised feature data to obtain target denoised feature data.

[0109] Among them, the first key feature data refers to the key feature data used for denoising the processed noisy feature data based on the empty text feature data through attention calculation, that is, the key feature vector or key feature matrix, that is, the key value; the first value feature data refers to the value feature data used for denoising the processed noisy feature data based on the empty text feature data through attention calculation, that is, the value feature vector or value feature matrix, that is, the value value. The second key feature data refers to the key feature data used for denoising the processed noisy feature data based on the updated text feature data through attention calculation, that is, the key feature vector or key feature matrix, that is, the key value; the second value feature data refers to the value feature data used for denoising the processed noisy feature data based on the updated text feature data through attention calculation, that is, the value feature vector or value feature matrix, that is, the value value.

[0110] Specifically, the empty text feature data, the updated text feature data and the to-be-processed noisy feature data are input into the attention module, and based on the empty text feature data and the updated text feature data, the to-be-processed noisy feature data are denoised using the reference path and the edit path, respectively. The reference path is to denoise the to-be-processed noisy feature according to the empty text feature data, and the edit path is to denoise the to-be-processed noisy feature according to the updated text feature data. Then, the first key feature data and the first value feature data in the reference path denoising process, as well as the second key feature data and the second value feature data in the edit path denoising process are determined, and the second key feature data is replaced with the first key feature data, and the second value feature data is replaced with the first value feature data, and then based on the first key feature data and the first value feature data, the to-be-processed noisy feature data is denoised using the updated text feature data as the denoising condition, thereby obtaining the target denoised feature data.

[0111] By replacing the first key feature data with the second key feature data and replacing the first value feature data with the second value feature data, the appearance of the target image output by the image processing model can be made to correspond to the appearance of the image to be processed input to the image processing model, thereby further improving the accuracy of the target image output by the image processing model.

[0112] Further, in a specific implementation provided in this specification, the image processing model includes an image decoder;

[0113] Obtaining a target image based on the target denoising feature data includes:

[0114] The target denoising feature data is input into the image decoder to obtain a target image.

[0115] After obtaining the target denoising feature data, the obtained target denoising feature data is input into the image decoder of the image processing model to obtain the target image corresponding to the image to be processed.

[0116] The image processing method provided in this specification realizes learning of image information by updating empty text feature data. Since the updated text feature data contains semantic information, semantic-level image processing is performed on the image to be processed, thereby improving the accuracy of image processing.

[0117] Since the image processing model is the basis for implementing the image processing method provided in this specification, before performing image processing on the image to be processed based on the image processing model, the image processing model needs to be trained to obtain a trained image processing model, and the trained image processing model needs to be applied.

[0118] Based on this, in a specific implementation manner provided in this specification, the image processing model is trained based on the following method:

[0119] Acquire a sample training image and a sample empty text, wherein the sample training image includes a sample to-be-processed image and a sample target image corresponding to the sample to-be-processed image;

[0120] Inputting the sample to-be-processed image and the sample empty text into the image processing model to obtain a predicted image;

[0121] Calculating a loss value of the image processing model according to the sample target image and the predicted image;

[0122] The model parameters of the image processing model are adjusted according to the loss value, and the image processing model is continued to be trained until the model training stop condition is reached.

[0123] Among them, the sample training image refers to the sample image used to train the image processing model; the sample training image includes the sample to-be-processed image and the sample target image corresponding to the sample to-be-processed image; the sample to-be-processed image is the image that needs to be processed by the image processing model during the training of the image processing model; the sample target image is the actual image after the sample to-be-processed image is processed. The sample empty text refers to the preset empty text used to train the image processing model.

[0124] The sample training image can be an image obtained from a sample training image set (including a sample to-be-processed image and a sample target image), or can be an image pair (including a sample to-be-processed image and a sample target image) created in a customized manner before training the image processing model; the sample training image set refers to a set consisting of sample to-be-processed images and sample target images; the predicted image refers to a target image after image processing that is output when the sample to-be-processed image and sample empty text are input into the image processing model; the loss value refers to the difference value between the sample target image and the predicted image, which is used to measure the difference between the sample target image and the predicted image.

[0125] Specifically, through the above-mentioned method of obtaining the image to be processed and the preset empty text, a sample training image and a sample empty text are obtained, and the sample image to be processed and the sample empty text are input into the image processing model to obtain a predicted image.

[0126] The image processing model is used to determine the sample target image of the sample image to be processed. At this time, the image processing model is a model that has not been trained. There will be a deviation between the determined predicted image and the actual sample target image, and the model parameters of the image processing model need to be adjusted accordingly. Specifically, the loss value of the image processing model is calculated based on the output predicted image and the sample target image. The loss function for calculating the loss value can be a 0-1 loss function, a square loss function, a cross entropy loss function, etc. in actual applications. In this specification, preferably, the square loss function is selected as the loss function for calculating the loss value, and the model parameters of the image processing model are adjusted according to the loss value. The image processing model is continued to be trained based on the adjusted model parameters for the next batch of sample training images and sample empty text until the stop condition of the model training is reached.

[0127] Specifically, the model training stopping conditions include that the loss value is less than a preset threshold and / or the training rounds reach a preset round.

[0128] In a specific implementation provided in this specification, taking the loss value being less than a preset threshold as the training stop condition as an example, the preset threshold is 0.01. When the loss value is less than 0.01, it is considered that the image processing model training is completed.

[0129] In another specific implementation provided in the present specification, taking the preset number of training rounds as the training stop condition as an example, the preset number of training rounds is 50 rounds. When the number of training rounds of sample training images and sample empty texts reaches 50 rounds, the image processing model training is considered completed.

[0130] In another specific implementation provided in the present specification, two training stop conditions, a preset threshold and a preset training round, are set, and the loss value and the training round are determined at the same time. When either the loss value or the training round meets the training stop condition, the image processing model training is considered to be completed.

[0131] The image processing method provided in this specification includes: obtaining an image to be processed and preset empty text, wherein the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point; inputting the image to be processed and the preset empty text into an image processing model to obtain a target image corresponding to the image to be processed, wherein the target image is obtained based on the to-be-processed noise-added feature data and updated text feature data of the image to be processed, and the updated text feature data is obtained by updating the empty text feature data based on the to-be-processed point feature data, and the to-be-processed point feature data is used to characterize the positional relationship between the starting point and the target point in each group of to-be-processed point pairs.

[0132] An embodiment of the present specification realizes that, by acquiring an image to be processed and preset empty text, the image processing model can process the image to be processed according to the starting point and target point in the image to be processed during the image processing process, thereby realizing accurate positioning of the image processing area of ​​the image to be processed and improving the accuracy of image processing. The learning of image information is also realized by updating the empty text feature data, and because the updated text feature data contains semantic information, semantic-level image processing of the image to be processed is realized, thereby improving the accuracy of image processing.

[0133] The following combination Figure 6 , taking the application of the image processing method provided in this specification in the image editing scenario as an example, the image processing method is further described. Figure 6 FIG. 1 shows a process flow chart of an image processing method provided by an embodiment of the present specification. Figure 6As shown, the original image and empty text are obtained, the original image is input into the image encoder of the image processing model to obtain image coding feature data, the empty text is input into the text encoder of the image processing model to obtain text coding feature data, and based on the text coding feature data, the DDIM Inversion algorithm is used to invert and add noise to the image coding feature data to generate image noise feature data. The text coding feature data and the image coding feature data are input into the image information creator of the image processing model, and the image information creator processes the image noise feature data to generate the starting point feature data and the target point feature data corresponding to each group of points to be processed. And according to the starting point feature data and the target point feature data, the L1 distance (i.e., Manhattan distance, which is used as an example for explanation) between the starting point feature data and the target point feature data is calculated. The text coding feature data is adjusted by the calculated L1 distance, and after repeated iterations, the text coding feature adjustment and update of the L1 distance of each group of points to be processed is completed, and the updated text coding feature data is obtained. Furthermore, the obtained updated text encoding feature data, text encoding feature data and image denoising feature data are input into the attention module in the image information creator. Through attention calculation, the image denoising feature data is denoised with the updated text encoding feature data as the denoising condition of the image denoising feature data. The image denoising feature data output by the image information creator can be obtained. The image denoising feature data is then input into the image decoder of the image processing model. The image denoising feature data is decoded in the image decoder to obtain the edited image corresponding to the original image.

[0134] An embodiment of the present specification realizes that, by acquiring the original image and presetting the empty text, the image processing model can process the original image according to the starting point and the target point in the original image during the image processing of the original image, thereby realizing accurate positioning of the image processing area of ​​the original image and improving the accuracy of image processing. The learning of image information is also realized by updating the empty text encoding feature data. Since the updated text encoding feature data contains semantic information, semantic-level image processing of the original image is realized, thereby improving the accuracy of image processing.

[0135] Corresponding to the above method embodiment, this specification also provides an image processing device embodiment, Figure 7 FIG. 2 shows a schematic diagram of the structure of an image processing device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0136] The acquisition module 702 is configured to acquire an image to be processed and preset empty text, wherein the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point;

[0137] The input module 704 is configured to input the image to be processed and the preset empty text into an image processing model to obtain a target image corresponding to the image to be processed, wherein the target image is obtained based on the processed noise-added feature data and updated text feature data of the image to be processed, and the updated text feature data is obtained by updating the empty text feature data based on the feature data of the point to be processed, and the feature data of the point to be processed is used to characterize the positional relationship between the starting point and the target point in each group of points to be processed.

[0138] Optionally, the image processing model includes a text encoder, an image encoder and an image information creator;

[0139] The input module 704 is further configured to:

[0140] Inputting the preset empty text into the text encoder to obtain empty text feature data corresponding to the preset empty text;

[0141] Inputting the image to be processed into the image encoder to obtain feature data of the image to be processed corresponding to the image to be processed;

[0142] Noising the image feature data to be processed according to the empty text feature data to generate noisy feature data to be processed;

[0143] The empty text feature data and the to-be-processed noise feature data are input into the image information creator to obtain target denoising feature data, and a target image is obtained based on the target denoising feature data.

[0144] Optionally, the input module 704 is further configured to:

[0145] Inputting the noise-added feature data to be processed into the image information creator to generate starting point feature data of the starting point and target point feature data of the target point in each group of point pairs to be processed;

[0146] Determine the feature data of the to-be-determined starting point and the feature data of the to-be-determined target point of the to-be-determined processing point pair according to the feature data of the to-be-determined starting point and the feature data of the to-be-determined target point, wherein the to-be-determined processing point pair is any one of the groups of processing point pairs, the feature data of the to-be-determined starting point is the feature data of the starting point of the to-be-determined processing point pair, and the feature data of the to-be-determined target point is the feature data of the target point of the to-be-determined processing point pair;

[0147] According to the feature data to be determined, the empty text feature data is updated until the feature data to be processed completes the update of the empty text feature data, thereby obtaining updated text feature data;

[0148] Based on the updated text feature data, the to-be-processed noisy feature data is denoised to obtain target denoised feature data.

[0149] Optionally, the input module 704 is further configured to:

[0150] Calculating a to-be-determined characteristic distance between the to-be-determined starting point characteristic data and the to-be-determined target point characteristic data;

[0151] The to-be-determined characteristic distance is determined as the to-be-determined characteristic data of the to-be-determined to-be-processed point pair.

[0152] Optionally, the image information creator includes an attention module;

[0153] The input module 704 is further configured to:

[0154] The updated text feature data and the to-be-processed denoised feature data are input into the attention module to obtain target denoised feature data.

[0155] Optionally, the input module 704 is further configured to:

[0156] Inputting the empty text feature data, the updated text feature data and the to-be-processed noise feature data into the attention module;

[0157] Determine first key feature data and first value feature data, and second key feature data and second value feature data, wherein the first key feature data and the first value feature data are respectively key feature data and value feature data used by the empty text feature data to perform attention calculation on the to-be-processed noise feature data, and the second key feature data and the second value feature data are respectively key feature data and value feature data used by the updated text feature data to perform attention calculation on the to-be-processed noise feature data;

[0158] replacing the second key feature data with the first key feature data, and replacing the second value feature data with the first value feature data;

[0159] According to the first key feature data and the first value feature data, attention calculation is performed on the updated text feature data and the to-be-processed denoised feature data to obtain target denoised feature data.

[0160] Optionally, the image processing model includes an image decoder;

[0161] The input module 704 is further configured to:

[0162] The target denoising feature data is input into the image decoder to obtain a target image.

[0163] Optionally, the device further includes a model training module configured to:

[0164] Acquire a sample training image and a sample empty text, wherein the sample training image includes a sample to-be-processed image and a sample target image corresponding to the sample to-be-processed image;

[0165] Inputting the sample to-be-processed image and the sample empty text into the image processing model to obtain a predicted image;

[0166] Calculating a loss value of the image processing model according to the sample target image and the predicted image;

[0167] The model parameters of the image processing model are adjusted according to the loss value, and the image processing model is continued to be trained until the model training stop condition is reached.

[0168] The image processing device provided in this specification includes: an acquisition module, configured to acquire an image to be processed and preset empty text, wherein the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point; an input module, configured to input the image to be processed and the preset empty text into an image processing model to obtain a target image corresponding to the image to be processed, wherein the target image is obtained based on the to-be-processed noise-added feature data and updated text feature data of the image to be processed, and the updated text feature data is obtained by updating the empty text feature data based on the to-be-processed point feature data, and the to-be-processed point feature data is used to characterize the positional relationship between the starting point and the target point in each group of to-be-processed point pairs.

[0169] An embodiment of the present specification realizes that, by acquiring an image to be processed and preset empty text, the image processing model can process the image to be processed according to the starting point and target point in the image to be processed during the image processing process, thereby realizing accurate positioning of the image processing area of ​​the image to be processed and improving the accuracy of image processing. The learning of image information is also realized by updating the empty text feature data, and because the updated text feature data contains semantic information, semantic-level image processing of the image to be processed is realized, thereby improving the accuracy of image processing.

[0170] The above is a schematic scheme of an image processing device of this embodiment. It should be noted that the technical scheme of the image processing device and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the image processing device can be referred to the description of the technical scheme of the above-mentioned image processing method.

[0171] See also Figure 8 , Figure 8A flowchart of a method for training an image processing model provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0172] Step 802: Obtain sample training images and sample empty text.

[0173] The sample training image includes a sample to-be-processed image and a sample target image corresponding to the sample to-be-processed image, the sample to-be-processed image includes at least one group of sample to-be-processed point pairs, and each group of sample to-be-processed point pairs includes a sample starting point and a sample target point.

[0174] Step 804: input the sample image to be processed and the sample empty text into an image processing model to obtain a predicted image.

[0175] Among them, the predicted image is obtained based on the sample noise-added feature data and sample updated text feature data of the sample to be processed image, and the sample updated text feature data is obtained by updating the sample empty text feature data based on the sample to be processed point feature data, and the sample to be processed point feature data is used to characterize the positional relationship between the sample starting point and the sample target point in each group of sample to be processed point pairs.

[0176] Optionally, in a specific implementation manner provided in this specification, inputting the sample to-be-processed image and the sample empty text into an image processing model to obtain a predicted image includes:

[0177] Input the sample to-be-processed image into the image encoder of the image processing model to obtain sample image feature data, and input the sample empty text into the text encoder of the image processing model to obtain sample empty text feature data;

[0178] Noising the sample image feature data according to the sample empty text feature data to generate sample noisy feature data;

[0179] Inputting the sample empty text feature data and the sample noise-added feature data into the image information creator of the image processing model to obtain sample denoised feature data;

[0180] The sample denoising feature data is input into the image decoder of the image processing model to obtain a predicted image.

[0181] Step 806: Calculate the loss value of the image processing model based on the sample target image and the predicted image.

[0182] Step 808: Adjust the model parameters of the image processing model according to the loss value, and continue to train the image processing model until the model training stop condition is reached.

[0183] Step 810: Feedback the model parameters of the trained image processing model to the target terminal.

[0184] The training method of the image processing model provided in this specification is the same as the training method of the image processing model in the above-mentioned image processing method. The implementation process of the training method of the image processing model can refer to the training method of the image processing model in the above-mentioned image processing method, and this specification will not repeat it here.

[0185] It should be noted that the training method of the image processing model is applied to the cloud-side device. After the image processing model is trained, the model parameters of the trained image processing model need to be fed back to the target terminal.

[0186] The image processing model training method provided in this manual can use empty text and images as training data for training the image processing model at the same time, thereby improving the accuracy of image processing model training; after the image processing model training is completed, the trained image processing model parameters are fed back to the target terminal. The user does not need to train the image processing model by himself, but can directly obtain the model parameters of the image processing model, thereby improving the convenience of users using the image processing model.

[0187] Corresponding to the above method embodiment, this specification also provides an embodiment of a training device for an image processing model. Fig. 9 FIG. 1 is a schematic diagram showing a structure of a training device for an image processing model provided by an embodiment of the present specification. Fig. 9 As shown, the device is applied to a cloud-side device, including:

[0188] The sample acquisition module 902 is configured to acquire a sample training image and a sample empty text, wherein the sample training image includes a sample to-be-processed image and a sample target image corresponding to the sample to-be-processed image, and the sample to-be-processed image includes at least one group of sample to-be-processed point pairs, and each group of sample to-be-processed point pairs includes a sample starting point and a sample target point;

[0189] The sample input module 904 is configured to input the sample to-be-processed image and the sample empty text into an image processing model to obtain a predicted image, wherein the predicted image is obtained based on the sample noise-added feature data and the sample updated text feature data of the sample to-be-processed image, the sample updated text feature data is obtained by updating the sample empty text feature data based on the sample to-be-processed point feature data, and the sample to-be-processed point feature data is used to characterize the positional relationship between the sample starting point and the sample target point in each group of sample to-be-processed point pairs;

[0190] A calculation module 906 is configured to calculate a loss value of the image processing model according to the sample target image and the predicted image;

[0191] A training module 908 is configured to adjust the model parameters of the image processing model according to the loss value, and continue to train the image processing model until a model training stop condition is reached;

[0192] The feedback module 910 is configured to feed back the model parameters of the trained image processing model to the target terminal.

[0193] Optionally, the sample input module 904 is further configured to:

[0194] Input the sample to-be-processed image into the image encoder of the image processing model to obtain sample image feature data, and input the sample empty text into the text encoder of the image processing model to obtain sample empty text feature data;

[0195] Noising the sample image feature data according to the sample empty text feature data to generate sample noisy feature data;

[0196] Inputting the sample empty text feature data and the sample noise-added feature data into the image information creator of the image processing model to obtain sample denoised feature data;

[0197] The sample denoising feature data is input into the image decoder of the image processing model to obtain a predicted image.

[0198] The training device for the image processing model provided in this specification can use empty text and images as training data for training the image processing model at the same time, thereby improving the accuracy of the image processing model training; after the image processing model training is completed, the trained image processing model parameters are fed back to the target terminal. The user does not need to train the image processing model by himself, but can directly obtain the model parameters of the image processing model, thereby improving the convenience of the user in using the image processing model.

[0199] The above is a schematic scheme of a training device for an image processing model of this embodiment. It should be noted that the technical scheme of the training device for the image processing model and the technical scheme of the training method for the image processing model described above belong to the same concept, and the details not described in detail in the technical scheme of the training device for the image processing model can be found in the description of the technical scheme of the training method for the image processing model described above.

[0200] See also Fig.10 , Fig.10 A flowchart of another image processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0201] Step 1002: receiving an image processing instruction sent by a target terminal, wherein the image processing instruction carries an image to be processed, and the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point.

[0202] The image processing method is applied to the server. The image processing instruction refers to an instruction for performing image processing on the image to be processed. The target terminal is a terminal for generating the image processing instruction. When there is a need for image processing, the user can determine multiple groups of starting points and target points in the image to be processed, and based on the image to be processed containing the multiple groups of starting points and target points, generate an image processing instruction for performing image processing on the image to be processed on the target terminal, and send the image processing instruction to the server.

[0203] Step 1004: In response to the image processing instruction, obtain preset empty text.

[0204] Step 1006: Input the image to be processed and the preset empty text into an image processing model to obtain a target image corresponding to the image to be processed.

[0205] Among them, the target image is obtained based on the processed noise-added feature data and updated text feature data of the image to be processed, the updated text feature data is obtained by updating the empty text feature data based on the processed point feature data, and the processed point feature data is used to characterize the positional relationship between the starting point and the target point in each group of processed point pairs.

[0206] Optionally, in a specific implementation provided in this specification, the image processing model includes a text encoder, an image encoder, and an image information creator;

[0207] Inputting the image to be processed and the preset empty text into an image processing model to obtain a target image corresponding to the image to be processed, including:

[0208] Inputting the preset empty text into the text encoder to obtain empty text feature data corresponding to the preset empty text;

[0209] Inputting the image to be processed into the image encoder to obtain feature data of the image to be processed corresponding to the image to be processed;

[0210] Noising the image feature data to be processed according to the empty text feature data to generate noisy feature data to be processed;

[0211] The empty text feature data and the to-be-processed noise feature data are input into the image information creator to obtain target denoising feature data, and a target image is obtained based on the target denoising feature data.

[0212] Step 1008: Send the target image to the target terminal.

[0213] Another image processing method provided in this specification is similar to the execution process of the above-mentioned image processing method. In the image processing method, after receiving the image processing instruction sent by the target terminal, the server obtains the image to be processed and the preset empty text, and generates the target image corresponding to the image to be processed based on the above-mentioned image processing method. The implementation process of the image processing method can refer to the execution process of the above-mentioned image processing method, and this specification will not repeat it here. After the target image of the image to be processed is generated based on the image processing model, the generated target image is sent to the target terminal.

[0214] Another image processing method provided in this specification generates a target image corresponding to the image to be processed by receiving an image processing instruction sent by a target terminal. After the target image is generated, the target image is sent to the target terminal. Therefore, there is no need for the user to process the image to be processed by himself, thereby improving the accuracy of image processing and user experience.

[0215] Corresponding to the above method embodiment, this specification also provides another image processing device embodiment, Fig.11 FIG. 2 shows a schematic diagram of the structure of another image processing device provided by an embodiment of the present specification. Fig.11 As shown, the device comprises:

[0216] The receiving module 1102 is configured to receive an image processing instruction sent by a target terminal, wherein the image processing instruction carries an image to be processed, and the image to be processed includes at least one group of point pairs to be processed, each group of point pairs to be processed includes a starting point and a target point;

[0217] A response module 1104 is configured to obtain preset empty text in response to the image processing instruction;

[0218] The acquisition module 1106 is configured to input the image to be processed and the preset empty text into an image processing model to obtain a target image corresponding to the image to be processed, wherein the target image is obtained based on the noise-added feature data to be processed and the updated text feature data of the image to be processed, and the updated text feature data is obtained by updating the empty text feature data based on the feature data of the point to be processed, and the feature data of the point to be processed is used to characterize the positional relationship between the starting point and the target point in each group of the point to be processed;

[0219] The sending module 1108 is configured to send the target image to the target terminal.

[0220] Optionally, the image processing model includes a text encoder, an image encoder and an image information creator;

[0221] The obtaining module 1106 is further configured to:

[0222] Inputting the preset empty text into the text encoder to obtain empty text feature data corresponding to the preset empty text;

[0223] Inputting the image to be processed into the image encoder to obtain feature data of the image to be processed corresponding to the image to be processed;

[0224] Noising the image feature data to be processed according to the empty text feature data to generate noisy feature data to be processed;

[0225] The empty text feature data and the to-be-processed noise feature data are input into the image information creator to obtain target denoising feature data, and a target image is obtained based on the target denoising feature data.

[0226] Another image processing device provided in this specification generates a target image corresponding to the image to be processed by receiving an image processing instruction sent by a target terminal. After the target image is generated, the target image is sent to the target terminal. Therefore, there is no need for the user to process the image to be processed by himself, thereby improving the accuracy of image processing and user experience.

[0227] The above is a schematic scheme of another image processing device of this embodiment. It should be noted that the technical scheme of the image processing device and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the image processing device can be referred to the description of the technical scheme of the above-mentioned image processing method.

[0228] Fig.12 The block diagram of a computing device 1200 according to one embodiment of the present specification is shown. The components of the computing device 1200 include but are not limited to a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and the database 1250 is used to store data.

[0229] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0230] In one embodiment of the present specification, the above components of the computing device 1200 and Fig.12 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.12 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0231] The computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1200 may also be a mobile or stationary server.

[0232] Among them, the processor 1220 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned image processing method or image processing model training method.

[0233] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the image processing method or the training method embodiment of the image processing model, the description is relatively simple, and the relevant parts can be referred to the partial description of the image processing method or the training method embodiment of the image processing model.

[0234] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned image processing method or image processing model training method when executed by a processor.

[0235] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the image processing method or the image processing model training method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the image processing method or the image processing model training method embodiment.

[0236] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned image processing method or image processing model training method.

[0237] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned image processing method or image processing model training method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned image processing method or image processing model training method.

[0238] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0239] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0240] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0241] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0242] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Acquire an image to be processed and preset empty text, wherein the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point; Input the preset empty text into the text encoder in the image processing model to obtain the empty text feature data corresponding to the preset empty text; input the image to be processed into the image encoder in the image processing model to obtain the image feature data to be processed corresponding to the image to be processed; add noise to the image feature data to be processed according to the empty text feature data to generate the noisy feature data to be processed; input the empty text feature data and the noisy feature data to be processed into the image information creator in the image processing model to obtain the target denoising feature data, and obtain the target image based on the target denoising feature data, wherein the target image is obtained based on the noisy feature data to be processed and the updated text feature data of the image to be processed, and the updated text feature data is obtained by updating the empty text feature data based on the feature data of the point to be processed, and the feature data of the point to be processed is used to characterize the positional relationship between the starting point and the target point in each group of points to be processed.

2. The method according to claim 1, inputting the empty text feature data and the to-be-processed noise feature data into an image information creator in the image processing model to obtain target noise removal feature data, comprising: Inputting the noise-added feature data to be processed into the image information creator to generate starting point feature data of the starting point and target point feature data of the target point in each group of point pairs to be processed; Determine the feature data of the to-be-determined starting point and the feature data of the to-be-determined target point of the to-be-determined processing point pair according to the feature data of the to-be-determined starting point and the feature data of the to-be-determined target point, wherein the to-be-determined processing point pair is any one of the groups of processing point pairs, the feature data of the to-be-determined starting point is the feature data of the starting point of the to-be-determined processing point pair, and the feature data of the to-be-determined target point is the feature data of the target point of the to-be-determined processing point pair; According to the feature data to be determined, the empty text feature data is updated until the feature data to be processed completes the update of the empty text feature data, thereby obtaining updated text feature data; Based on the updated text feature data, the to-be-processed noisy feature data is denoised to obtain target denoised feature data.

3. The method according to claim 2, determining the feature data to be determined of the pair of points to be processed according to the feature data of the starting point to be determined and the feature data of the target point to be determined, comprising: Calculating a to-be-determined characteristic distance between the to-be-determined starting point characteristic data and the to-be-determined target point characteristic data; The to-be-determined characteristic distance is determined as the to-be-determined characteristic data of the to-be-determined to-be-processed point pair.

4. The method of claim 2, wherein the image information creator comprises an attention module; Based on the updated text feature data, denoising the to-be-processed denoised feature data to obtain target denoised feature data, including: The updated text feature data and the to-be-processed denoised feature data are input into the attention module to obtain target denoised feature data.

5. The method according to claim 4, inputting the updated text feature data and the to-be-processed denoised feature data into the attention module to obtain target denoised feature data, comprising: Inputting the empty text feature data, the updated text feature data and the to-be-processed noise feature data into the attention module; Determine first key feature data and first value feature data, and second key feature data and second value feature data, wherein the first key feature data and the first value feature data are respectively key feature data and value feature data used by the empty text feature data to perform attention calculation on the to-be-processed noise feature data, and the second key feature data and the second value feature data are respectively key feature data and value feature data used by the updated text feature data to perform attention calculation on the to-be-processed noise feature data; replacing the second key feature data with the first key feature data, and replacing the second value feature data with the first value feature data; According to the first key feature data and the first value feature data, attention calculation is performed on the updated text feature data and the to-be-processed denoised feature data to obtain target denoised feature data.

6. The method of claim 1, wherein the image processing model comprises an image decoder; Obtaining a target image based on the target denoising feature data includes: The target denoising feature data is input into the image decoder to obtain a target image.

7. The method according to claim 1, wherein the image processing model is trained based on the following method: Get sample training images and sample empty text, where, The sample training images include a sample to-be-processed image and a sample target image corresponding to the sample to-be-processed image; Inputting the sample to-be-processed image and the sample empty text into the image processing model to obtain a predicted image; Calculating a loss value of the image processing model according to the sample target image and the predicted image; The model parameters of the image processing model are adjusted according to the loss value, and the image processing model is continued to be trained until the model training stop condition is reached.

8. A training method for an image processing model, applied to a cloud-side device, comprising: Acquire a sample training image and a sample empty text, wherein the sample training image includes a sample to-be-processed image and a sample target image corresponding to the sample to-be-processed image, the sample to-be-processed image includes at least one group of sample to-be-processed point pairs, and each group of sample to-be-processed point pairs includes a sample starting point and a sample target point; Input the sample image to be processed into the image encoder of the image processing model to obtain sample image feature data, input the sample empty text into the text encoder of the image processing model to obtain sample empty text feature data; add noise to the sample image feature data according to the sample empty text feature data to generate sample noisy feature data; input the sample empty text feature data and the sample noisy feature data into the image information creator of the image processing model to obtain sample denoised feature data; input the sample denoised feature data into the image decoder of the image processing model to obtain a predicted image, wherein the predicted image is obtained based on the sample noisy feature data and the sample updated text feature data of the sample image to be processed, the sample updated text feature data is obtained by updating the sample empty text feature data based on the sample to be processed point feature data, and the sample to be processed point feature data is used to characterize the positional relationship between the sample starting point and the sample target point in each group of sample to be processed point pairs; Calculating a loss value of the image processing model according to the sample target image and the predicted image; Adjusting the model parameters of the image processing model according to the loss value, and continuing to train the image processing model until a model training stop condition is reached; The model parameters of the trained image processing model are fed back to the target terminal.

9. An image processing method, comprising: Receiving an image processing instruction sent by a target terminal, wherein the image processing instruction carries an image to be processed, and the image to be processed includes at least one group of point pairs to be processed, and each group of point pairs to be processed includes a starting point and a target point; In response to the image processing instruction, obtaining a preset empty text; Input the preset empty text into the text encoder in the image processing model to obtain the empty text feature data corresponding to the preset empty text; input the image to be processed into the image encoder in the image processing model to obtain the image feature data to be processed corresponding to the image to be processed; add noise to the image feature data to be processed according to the empty text feature data to generate the noisy feature data to be processed; input the empty text feature data and the noisy feature data to be processed into the image information creator in the image processing model to obtain the target denoising feature data, and obtain the target image based on the target denoising feature data, wherein the target image is obtained based on the noisy feature data to be processed and the updated text feature data of the image to be processed, and the updated text feature data is obtained by updating the empty text feature data based on the feature data of the point to be processed, and the feature data of the point to be processed is used to characterize the positional relationship between the starting point and the target point in each group of the point to be processed; The target image is sent to the target terminal.

10. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 9 are implemented.

11. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

12. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images