Image editing method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses an image editing method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining a to-be-edited image and an editing instruction; generating lexical elements of the target image based on the to-be-edited image and the editing instruction by using a pre-trained multi-modal large model; intercepting visual lexical elements of the target image from the lexical elements of the target image; the hidden space representation of the target image is generated by using a pre-trained diffusion model based on the hidden space representation of the to-be-edited image and the text lexical elements and the visual lexical elements corresponding to the editing instruction, and the pre-trained diffusion model has a multi-condition guide architecture; and decoding the hidden space representation of the target image to obtain the target image. According to the method, the visual lexical elements generated by the multi-modal large model are added to serve as constraint conditions, so that the generated target image is closer to an editing instruction of a user, and the image editing effect is improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of computer vision and artificial intelligence technology, and in particular to an image editing method, device, storage medium and electronic device. Background Art

[0002] Image editing refers to the process of modifying, adjusting and processing digital images through various techniques and tools to achieve specific visual effects, improve image quality or change image content. Image editing can involve partial or overall changes to an image, including but not limited to color adjustment, brightness and contrast adjustment, image synthesis, removal or addition of elements, shape deformation, texture synthesis, image restoration, etc.

[0003] In the related art, a diffusion model can be used to edit the input image based on the editing instruction input by the user, such as a command text input by the user. However, the editing effect of this image editing method is uncontrollable, and the image effect of the edited image is often inconsistent with the effect expected by the user. Summary of the invention

[0004] Embodiments of the present disclosure provide an image editing method, device, storage medium, and electronic device.

[0005] According to one aspect of an embodiment of the present disclosure, there is provided an image editing method, the method comprising: acquiring an image to be edited and an editing instruction; using a pre-trained multimodal large model to generate word-grams of a target image based on the image to be edited and the editing instruction; extracting visual word-grams of the target image from the word-grams of the target image; using a pre-trained diffusion model to generate a latent space representation of the target image based on the latent space representation of the image to be edited, the text word-grams corresponding to the editing instruction, and the visual word-grams, wherein the pre-trained diffusion model has a multi-conditional guided architecture; decoding the latent space representation of the target image to obtain the target image.

[0006] According to another aspect of an embodiment of the present disclosure, there is provided an image editing device, the device comprising: a first acquisition module, for acquiring an image to be edited and an editing instruction; a first generation module, for generating word elements of a target image based on the image to be edited and the editing instruction using a pre-trained multimodal large model; a reading module, for extracting visual word elements of the target image from the word elements of the target image; a second generation module, for generating a latent space representation of the target image based on the latent space representation of the image to be edited, the text word elements corresponding to the editing instruction, and the visual word elements using a pre-trained diffusion model, wherein the pre-trained diffusion model has a multi-conditional guided architecture; and a decoding module, for decoding the latent space representation of the target image to obtain the target image.

[0007] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to execute the above-mentioned image editing method.

[0008] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing processor executable instructions; the processor, configured to read the executable instructions from the memory and execute the instructions to implement the above-mentioned image editing method.

[0009] Based on the image editing method, device, storage medium and electronic device provided by the above-mentioned embodiments of the present disclosure, when image editing is required, the word element of the target image can be first generated based on the image to be edited and the editing instruction by using the pre-trained multimodal large model, and the visual word element of the target image can be intercepted from the word element of the target image. When the pre-trained diffusion model is used for image editing, the visual word element of the target image and the text word element corresponding to the editing instruction can be used as a guiding condition to guide the diffusion model to perform image editing, wherein the pre-trained diffusion model has a multi-condition guiding architecture. The technical solution of the present disclosure makes the generated target image closer to the user's editing instruction by adding the visual word element generated by the multimodal large model as a constraint condition, thereby improving the editing effect of the target image and the consistency with the editing instruction.

[0010] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 It is a flowchart of an image editing method provided by an exemplary embodiment of the present disclosure.

[0013] Figure 2 It is a schematic diagram of a generation framework of image editing provided by an exemplary embodiment of the present disclosure.

[0014] Figure 3 It is a schematic diagram of a model training process provided by an exemplary embodiment of the present disclosure.

[0015] Figure 4 It is a schematic diagram of a model training framework provided by an exemplary embodiment of the present disclosure.

[0016] Figure 5 It is a flowchart of an image editing method provided by another exemplary embodiment of the present disclosure.

[0017] Figure 6 It is a flowchart of an image editing method provided by another exemplary embodiment of the present disclosure.

[0018] Figure 7 It is a schematic diagram of a generation framework of image editing provided by another exemplary embodiment of the present disclosure.

[0019] Figure 8 It is a schematic diagram of generating reference graph word elements provided by an exemplary embodiment of the present disclosure.

[0020] Fig. 9 It is a schematic diagram of an image editing process of a diffusion model provided by an exemplary embodiment of the present disclosure.

[0021] Fig.10 It is a structural schematic diagram of an image editing device provided by an exemplary embodiment of the present disclosure.

[0022] Fig.11 It is a structural schematic diagram of an image editing device provided by another exemplary embodiment of the present disclosure.

[0023] Fig.12 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0025] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0026] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0027] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0028] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0029] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0030] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0031] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0032] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0033] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0034] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0035] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0036] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0037] SUMMARY OF THE DISCLOSURE

[0038] Current image editing solutions usually use diffusion models to directly perform the editing indicated by the editing instructions on the image to be edited based on the editing instructions input by the user, but the image effect of editing the image only according to the editing instructions is uncontrollable. The disclosed technical solution generates visual word elements of the target image based on the editing instructions and the image to be edited through a multimodal large model, and uses the visual word elements as constraints of the diffusion model, so that the generated target image is closer to the user's editing instructions, thereby improving the image editing effect.

[0039] In addition, reference image words can be obtained according to the reference image, and the reference image words can be used as constraints of the diffusion model to further improve the image editing effect of the diffusion model.

[0040] Exemplary Methods

[0041] Figure 1 FIG. 1 is a flowchart of an image editing method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as servers or mobile terminals, such as Figure 1 As shown, the following steps are included:

[0042] Step 101, obtaining an image to be edited and an editing instruction.

[0043] The image to be edited in the present disclosure is used to indicate an image that needs to be modified or adjusted, and the editing instruction is used to indicate an instruction for editing the image to be edited, for example, adjusting the image color, removing or adding image elements, replacing an image element in the image, etc.

[0044] The editing instruction may be a text instruction or a voice instruction. The voice instruction may be recognized as a text instruction through a voice recognition algorithm.

[0045] Step 102, using the pre-trained multimodal large model, based on the image to be edited and the editing instructions, generates word units of the target image.

[0046] In the present disclosure, the pre-trained multimodal large model is used to refer to a model that combines multimodal information such as text, images, and audio for training. The training process of the multimodal large model can be found in Figure 3 The illustrated embodiment will not be described in detail here.

[0047] In this embodiment, the trained multimodal large model can automatically generate word units of the target image after the image to be edited based on the editing instructions input by the user and the image to be edited.

[0048] The target image is an image obtained after the image to be edited is edited according to the editing instruction.

[0049] Step 103 , extracting visual word units of the target image from the word units of the target image.

[0050] The visual word-meta in the present disclosure is used to indicate the smaller feature vectors into which the target image is split. The visual word-meta can be used to calculate and match the diffusion model with the multi-dimensional vector of the latent space representation of the image to be edited, thereby guiding the diffusion model to perform image editing and denoising to generate the target image.

[0051] In specific implementation, text words and visual words can be respectively extracted from the words of the target image obtained in step 102. The text words are used to represent text features and are texts used to describe the target image.

[0052] Step 104, using a pre-trained diffusion model, based on the latent space representation of the image to be edited, the text word unit corresponding to the editing instruction, and the visual word unit, a latent space representation of the target image is generated, wherein the pre-trained diffusion model has a multi-conditional guided architecture.

[0053] In the present disclosure, the pre-trained diffusion model is a pre-trained model for text images, and the diffusion model can generate a latent space representation of a target image according to the latent space representation of the image to be edited. In the process of using the diffusion model to generate the latent space representation of the target image, the visual word unit and the text word unit corresponding to the editing instruction are used as guiding conditions to guide the diffusion model to generate an edited image that better meets the user's needs.

[0054] Among them, the multi-condition guided architecture means that the pre-trained diffusion model can use multiple guiding conditions to guide the image editing process.

[0055] The diffusion model mainly includes two processes: diffusion process and reverse process. The diffusion process is to start from the image to be edited and gradually add noise to it until the image to be edited becomes completely noisy. This process is usually described by a random process, such as Brownian motion. The reverse process is to start from completely noisy data and gradually remove noise to generate new data similar to the image to be edited (the edited target image).

[0056] In the present disclosure, in the process of image editing using a pre-trained diffusion model, visual words generated based on a multimodal large model and text words corresponding to editing instructions can be used as conditional guides to calculate the cross-attention feature representation of the latent space representation of the image of the cross-attention module of the pre-trained diffusion model, and obtain the latent space representation of the target image based on the cross-attention feature representation. In the process of generating the cross-attention feature representation, the cross-attention feature representation is obtained by assigning weights (i.e., weighted fusion weights) to the visual words and text words corresponding to the editing instructions respectively. Optionally, the weighted fusion weight is automatically learned when the diffusion model is pre-trained.

[0057] In this embodiment, the latent space representation of the image to be edited is a latent space expression obtained by encoding the image to be edited using an image encoder, such as a variational auto-encoder (VAE for short).

[0058] Step 105, decoding the latent space representation of the target image to obtain the target image.

[0059] In this embodiment, VAE can be used to decode the latent space representation of the target image to obtain the pixel space representation of the target image.

[0060] For example, see Figure 2 , which illustrates the generation framework of image editing. After the image to be edited and the editing instruction are processed by the multimodal large model, the word element of the target image is obtained, and the visual word element of the target image can be intercepted based on the word element of the target image; the image to be edited is encoded by the image encoder to obtain the latent space representation and then input into the diffusion model. The diffusion model uses the text word element corresponding to the editing instruction and the visual word element of the target image as guiding conditions to guide the diffusion model to output the latent space representation of the target image expected by the user; then it is decoded according to the image decoder to output the target image.

[0061] The method provided by the above-mentioned embodiment of the present disclosure can first generate word-grams of the target image based on the image to be edited and the editing instruction by using the pre-trained multimodal large model when image editing is required, and the visual word-grams of the target image can be intercepted from the word-grams of the target image. When the pre-trained diffusion model is used for image editing, the visual word-grams of the target image and the text word-grams corresponding to the editing instruction can be used as guiding conditions to guide the diffusion model for image editing, thereby improving the accuracy of the latent space representation of the generated target image, and thus improving the editing effect of the target image. The technical solution of the present disclosure makes the generated target image closer to the user's editing instruction by adding the visual word-grams generated by the multimodal large model as constraint conditions, thereby improving the editing effect of the target image and the consistency with the editing instruction.

[0062] In some optional implementations, a pre-trained diffusion model is used to generate a latent space representation of a target image based on the latent space representation of the image to be edited, the text words corresponding to the editing instructions, and the visual words, including: based on a cross-attention mechanism, a cross-attention feature representation is obtained according to the text words, visual words corresponding to the editing instructions, and the latent space representation of the image to be edited; using a pre-trained diffusion model, based on the cross-attention feature representation, the noise to be removed corresponding to each time step of the image to be edited is removed to obtain the latent space representation of the target image.

[0063] Among them, the cross-attention mechanism can assign attention weights to each feature according to the latent space representation of the input image to be edited, the text word corresponding to the editing instruction, and the visual word, so that the pre-trained diffusion model pays more attention to the key feature information with higher weights (i.e., higher feature correlation) during the generation process. In this way, the pre-trained diffusion model can better meet the user's editing instructions and retain the data features of the original data (the image to be edited) when generating the edited target image, thereby improving the image effect of the edited target image.

[0064] The attention weights assigned to each feature are automatically learned by the pre-trained diffusion model.

[0065] In some optional implementations, the multimodal large model and the diffusion model can be obtained by training, such as Figure 3 As shown, the training process includes the following steps.

[0066] Step 301, obtaining a training sample set, the training sample set includes at least one group of training samples, each group of training samples includes pre-editing image samples, editing instruction samples, real edited image samples and editing result description samples.

[0067] The editing instruction sample may be a text sample or an audio sample. If it is an audio sample, speech recognition may be performed on the audio sample to obtain the corresponding text. The real edited image sample is the edited image sample corresponding to the pre-edited image sample. The editing result description sample is information describing the sample effect of the real edited image sample in the sample.

[0068] Among them, the editing result description sample can be output through a trained text generation model (Generative Pre-Trained, abbreviated as GPT) based on the pre-editing image sample and the real edited image sample (the real edited image sample is obtained after the user edits the pre-editing image sample according to the editing instruction sample).

[0069] Step 302, for each group of pre-editing image samples and editing instruction samples in the training samples, use the first model to be trained to obtain the corresponding image word-grams, where the image word-grams include first text word-grams and first visual word-grams; and, based on the word-grams of the editing result description sample and the first text word-grams, obtain the first loss value of the training of the first model to be trained.

[0070] In the present disclosure, the first model to be trained can receive input of any at least one modality data, and predict corresponding image word units based on pre-editing image samples and editing instruction samples.

[0071] In this embodiment, the pre-edited image samples can be pre-processed first, and the pre-processed images can be converted into tensors. In specific implementation, the pre-edited image samples can be cropped, standardized, and other operations. Cropping can be used to crop images of different sizes to the same size, and standardization can be used to standardize image pixels with pixel values ​​of 0-255 to between -1 and 1, which helps to convert the image into a tensor of size (C, H, W), where C is channels (Channels) used to indicate the number of color channels of the image, H height (Height) is used to indicate the vertical size of the image, that is, the height of the image, and W width (Width) is used to indicate the horizontal size of the image, that is, the width of the image.

[0072] For the editing instruction sample, a lemma operation can be performed on it to obtain the text lemma corresponding to the editing instruction sample. The lemma operation can be understood as the process of dividing the editing instruction sample into a lemma sequence that can be recognized by the first model to be trained, which is used to provide recognizable and easy-to-recognize text data for the first model to be trained. By performing a lemma operation on the editing instruction, it is helpful for the first model to be trained to understand the text.

[0073] By converting the pre-editing image samples into tensors and converting the editing instruction samples into text tokens, it helps the first to-be-trained model to process.

[0074] In order to enable the first model to be trained to edit the editing instruction samples and the image samples before editing, an instruction format acceptable to the multimodal large model can be constructed based on the editing instruction samples and the image samples before editing, and the instruction can be input into the first model to be trained so that the first model to be trained can perform label prediction.

[0075] The first text word is used to represent text features, which are features of the description text used to express the edited image; the first visual word is used to represent visual features, which are image features used to express the edited image.

[0076] In specific implementation, the first set number of digits of the image word unit can be used as the first text word unit, and the second set number of digits can be used as the first visual word unit, or the second set number of digits of the image word unit can be used as the first text word unit, and the first set number of digits can be used as the first visual word unit. The image word unit is composed of the text word unit and the visual word unit, and the positions of the two in the image word unit can be obtained by learning.

[0077] In this embodiment, the editing result description samples are tokenized to obtain tokens of the editing result description samples. Then, a cross entropy loss function can be used to obtain a first loss value for training the first to-be-trained model based on each first text token and each token of the editing result description samples.

[0078] In some other implementations, each first text word may be decoded by a text decoder, and meaningless placeholders and sign symbols in the decoded text may be removed by filtering to obtain each first text. Then, a text encoder such as a multimodal pre-trained neural network (Contrastive Language-Image Pre-Training, referred to as CLIP model) is used to encode the first text to obtain the corresponding second text word. Then, a cross entropy loss function is used to obtain each first loss value based on each second text word and each word describing the editing result sample.

[0079] For example, see Figure 4 After the image samples before editing and the editing instruction samples are predicted by the first model to be trained, image tokens can be obtained. After intercepting the image tokens, the first text tokens (tokens) and the first visual tokens (tokens) can be obtained. After decoding and filtering the first text tokens, the first text can be obtained. After encoding the first text, the second text tokens can be obtained. Based on the tokens of the editing result description samples and the second text tokens, the cross entropy loss (first loss value) can be obtained.

[0080] Step 303, for each group of training samples, based on the corresponding first visual word unit, the text word unit of the editing instruction sample, and the latent space representation of the image sample before editing, obtain the second loss value of the second model to be trained; wherein the second model to be trained has a diffusion model architecture guided by multiple conditions.

[0081] The second model to be trained is an initial diffusion model. The image editing process of the diffusion model can be found in Fig. 9 , first convert the original image from pixel space to latent space through the VAE encoder, and then add noise to the latent space representation of the image to obtain z T Then, in the denoising process, the cross-attention mechanism is introduced, and the text vector representation corresponding to the editing instruction, the reference image word (which may or may not include the reference image word) and the above-mentioned first visual word are used as constraints to constrain the diffusion model's denoising image generation process, generate the latent space representation of the predicted edited image, and then decode the latent space representation of the predicted edited image through the VAE encoder to obtain the output image.

[0082] Among them, the cross-attention mechanism can assign a weight to the input full-image features (latent space representation of the image sample before editing), the text word corresponding to each editing instruction sample, and each first visual word, so that the model pays more attention to the key areas with higher weights (i.e., higher feature correlation) in the process of generating images. In this way, the model can better retain and reproduce the key features of the second image sample when generating the edited image, thereby improving the quality and similarity of the generated image. The calculation method of the cross-attention mechanism is as shown in formula (1).

[0083]

[0084] In formula (1), Attn(Q, K, V) represents the output of the attention mechanism, Q represents the weight matrix of the full-image features, K and V represent the weight matrices of the constraints (reference image words, visual words, and text vector representation of editing instructions), and the softmax function is used to represent the normalization process.

[0085] In the present disclosure, the guiding conditions of the cross-attention mechanism may include the text words corresponding to each editing instruction sample, the first visual words obtained based on the multimodal large model, and the reference image words (which may include the reference image words or may not include the reference image words). The guiding conditions can jointly guide the image editing process of the diffusion model, making image editing more accurate and effective, and meeting the user's editing needs.

[0086] In the present disclosure, the second loss value may include an image mean square error value determined based on the first loss function and a noise mean square error value determined based on the second loss function.

[0087] Among them, the first loss function can be used to determine the image mean square error value of the latent space representation of each predicted edited image and the latent space representation of each real edited image sample. Each real edited image sample is a sample label included in the training sample set, which is obtained by the user editing the pre-edited image sample based on the editing instruction. The latent space representation of the predicted edited image is the predicted sample output by the second to-be-trained model according to the above-mentioned guiding conditions and the pre-edited image sample. The image quality can be determined by determining the latent space representation of the real edited image sample and the latent space representation of the predicted edited image through the first loss function.

[0088] The second loss function may be a function commonly used in diffusion models to determine a loss value based on noise. The second loss function may be used to determine a noise mean square error value between a latent space representation of a predicted edited image and a true edited image sample.

[0089] Based on the image mean square error value determined by the first loss function and the noise mean square error value determined by the second loss function, a second loss value can be obtained.

[0090] It can be understood that if each training sample in the training sample set contains a reference image, the diffusion model obtained through training needs to use the reference image word element of the reference image as a guiding condition to guide the image generation process; if some training samples in the second training sample set contain a reference image and some training samples do not contain a reference image, the diffusion model obtained through training may use the reference image word element of the reference image as a guiding condition to guide the image generation process, or may not use the reference image word element of the reference image as a guiding condition.

[0091] Step 304: Based on the first loss value and the second loss value, guide the first model to be trained and the second model to be trained to obtain a pre-trained multimodal large model and a pre-trained diffusion model.

[0092] The method provided by the above-mentioned embodiments of the present disclosure improves the accuracy of the multimodal large model by using the editing result description text output by the trained text generation model as the sample label of the training sample for learning the multimodal large model when training the multimodal large model and the diffusion model, which helps to generate more accurate constraints (visual words) for the diffusion model, thereby improving the effect of the diffusion model in editing images; in addition, by introducing the cross-attention mechanism, the trained diffusion model can pay more attention to key areas with higher weights when editing images, thereby improving the quality of the generated images; in addition, when training the diffusion model, by adding the loss calculation of the image mean square error value on the basis of the original diffusion model's loss determined according to noise, the image generated by the trained diffusion model can be closer to the user's editing instructions, thereby improving the effect of the diffusion model in editing images.

[0093] Figure 5 is a flowchart of an image editing method provided by another exemplary embodiment of the present disclosure, such as Figure 5 As shown, the image editing method includes the following steps.

[0094] Step 501, obtaining an image to be edited and an editing instruction.

[0095] Step 502 , perform a lemma operation on the editing instruction to obtain instruction lemmas, and pre-process the image to be edited, and then convert the pre-processed image into a tensor.

[0096] In this embodiment, the lemma operation on the editing instruction can be understood as the process of segmenting the editing instruction into a sequence of lemmas that can be recognized by the multimodal large model, which is used to provide recognizable and easily recognizable text data for the multimodal large model. By performing the lemma operation on the editing instruction, it is helpful for the multimodal large model to understand and generate text.

[0097] In this embodiment, the preprocessing of the image to be edited may include operations such as cropping and normalization. Cropping can be used to crop images of different sizes into the same size, while normalization can be used to normalize image pixels with pixel values ​​of 0-255 to between -1 and 1, which helps to convert the image into a tensor of size (C, H, W).

[0098] Step 503, generate a model query instruction based on the instruction word unit and tensor.

[0099] In the present disclosure, the model query instruction is an instruction format acceptable to the multimodal large model. The model query instruction is generated by instruction words and image tensors. The editing instruction and the image to be edited can be constructed in the instruction so that the multimodal large model can generate and output results.

[0100] Exemplarily, the model query instruction is as follows: Conversation(system="A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.", roles=('USER', 'ASSISTANT'), messages=[['USER', "what will this image be like if′'remove bananas and add grapes'(in a short paragraph)\n<im_start><im_patch>......<im_patch><im_end>"], ['ASSISTANT', None]], offset=0, sep_style=<SeparatorStyle.TWO:2>, sep=",", sep2="", version='Unknown', skip_next=False), and input this instruction into the multimodal large model so that the multimodal large model can generate and output results.

[0101] The Chinese interpretation of the above model query instruction is "Conversation(system='A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.', roles=('user', 'assistant'), messages=[['user', "what will this image be like if'remove bananas and add grapes'(in a short paragraph)\n<im_start><im_patch>…<im_patch><im_end>"], ['assistant', None]], offset=0, sep_style=<SeparatorStyle.TWO:2>, sep='', sep2='', version='Unknown', skip_next=False)", that is, it is hoped that the multimodal large model will output an instruction to remove the banana element and add the grape element in the image.

[0102] Step 504, using the pre-trained multimodal large model, based on the model query instruction, generate the tokens of the target image, and intercept the visual tokens of the target image from the tokens of the target image.

[0103] Among them, the pre-trained multimodal large model can generate word-grams of the target image, and the text word-grams and visual word-grams of the target image can be respectively decomposed according to the word-grams, and the visual word-grams can be used in step 605, and the constrained diffusion model generates a latent space representation of the target image. The text word-gram is used to describe the result after the image is edited. For example, after inputting the instruction shown in step 603, the multimodal large model can output "Remove the bananas from the image and addgrapes. Edit the image to add the grapes and arrange them on the fruitstand. Finished".

[0104] Step 505 , using the pre-trained diffusion model, based on the latent space representation of the image to be edited, the text word-grams corresponding to the editing instructions, and the visual word-grams of the target image, a latent space representation of the target image is generated.

[0105] Step 506: Decode the latent space representation of the target image to obtain the target image.

[0106] In some embodiments, the implementation of steps 505 to 506 can be found in Figure 1 The description of steps 104 to 105 in the illustrated embodiment will not be described in detail here.

[0107] The method provided by the above-mentioned embodiment of the present disclosure discloses a specific implementation method of generating visual word units by a multimodal large model, which helps to constrain the process of generating a target image by a diffusion model through the visual word units generated by the multimodal large model.

[0108] In other implementations, additional reference images may be added to guide the process of generating the target image using the diffusion model. Figure 6 FIG. 1 is a flowchart of an image editing method provided by another exemplary embodiment of the present disclosure. This embodiment uses an example of how to edit an image to be edited by combining a reference image, visual words of a target image, and text words corresponding to an editing instruction. Figure 6 As shown, the image editing method includes the following steps.

[0109] Step 601: Acquire an image to be edited, an editing instruction, and at least one reference image.

[0110] In the present disclosure, the reference image includes at least one of the following information: local image information to be added or to be eliminated in the image to be edited, image style information of the image to be edited, and image filter information of the image to be edited.

[0111] Exemplarily, when the image background color, local color of the object, or face shape and image style of the image to be edited are required, the reference image can be obtained at the same time as the image to be edited and the editing instruction are obtained. For example, if the background color of the image to be edited is to be adjusted to the blue background color of the ID photo, a reference image with a blue background color can be obtained; for another example, if the image style of the image to be edited is to be adjusted to Disney style or pastoral style, a reference image with Disney style or pastoral style can be selected. The reference image can also be an image to which elements are to be added. For example, if an image element of a dog is to be added or removed in the image to be edited, an image of a dog can be selected as a reference image. In this embodiment, the accuracy and effectiveness of image editing are enhanced by adding constraints to the reference image.

[0112] Further, the reference image word-gram corresponding to the reference image can be obtained through step 602, and the visual word-gram of the target image can be obtained through step 603. The present disclosure does not limit the execution order of step 602 and step 603, and step 602 can be executed first and then step 603, or step 603 can be executed first and then step 602, or step 602 and step 603 can be executed simultaneously.

[0113] Step 602: extract features from at least one reference image to obtain reference image features, and generate reference image words based on the reference image features using a multi-layer perceptron.

[0114] In the present disclosure, the reference image feature is a feature vector obtained by encoding the reference image through a feature extractor based on a convolutional neural network. The reference image word is a graphic feature that can be used in a diffusion model obtained by capturing, refining, combining and mapping the reference image features through a multi-layer perceptron.

[0115] In specific implementation, the reference image may be preprocessed, for example, by adjusting the image size, cropping the image, and normalizing the image. Since the image sizes of different reference images may be different, the image sizes may be adjusted and cropped to make them the same, so as to improve the feature extraction efficiency of the model. After the image preprocessing operation is completed, the image may be input into a feature extractor, such as a Visual Geometry Group Network (VGG), for encoding to obtain reference image features.

[0116] In this embodiment, in order to enable the reference image features to be used as the guiding conditions of the diffusion model and guide the image generation process of the diffusion model, a mapping process may be further performed on the extracted reference image features. Figure 8, can be realized by a multilayer perceptron (MLP), which is a fully connected neural network structure consisting of multiple hidden layers (such as Figure 8 Hidden layer 1, hidden layer 2, hidden layer 3, hidden layer 4) in the image processing layer help capture the reference image features at different abstract levels, and then gradually refine and combine the features through layer-by-layer transmission and weighting, thereby obtaining a higher-level reference image attribute and feature representation. Through parameter training and fine-tuning of the multi-layer perceptron, the reference image features can be mapped from the original feature space to the image-text feature space of the diffusion model, which can more effectively supervise the diffusion model training.

[0117] Step 603, using the pre-trained multimodal large model, based on the image to be edited and the editing instruction, generates word units of the target image, and extracts visual word units of the target image from the word units of the target image.

[0118] The implementation of step 603 can be found in Figure 1 The description of step 102 to step 103 in the illustrated embodiment will not be described in detail here.

[0119] Step 604 , using the pre-trained diffusion model, based on the latent space representation of the image to be edited, the text word-grams corresponding to the editing instructions, the visual word-grams of the target image, and the reference image word-grams, a latent space representation of the target image is generated.

[0120] In the present disclosure, the latent space representation of the image to be edited is used to indicate the form of transforming the image to be edited from the pixel space to the latent space, which helps to improve the computational efficiency of the diffusion model. In specific implementation, the image to be edited can be encoded using an image encoder to obtain the latent space representation of the image to be edited.

[0121] In the present disclosure, the text word element corresponding to the editing instruction is used to indicate the conversion of the editing instruction from the instruction text to the text vector representation that can be combined with the latent space representation of the image, and the editing instruction is encoded by a text encoder to obtain the text word element corresponding to the editing instruction. In specific implementation, the CLIP model can be used to encode the text corresponding to the editing instruction to obtain the text word element corresponding to the editing instruction.

[0122] Among them, the CLIP model is a deep learning model that combines image and text information. It can be trained by considering both the image and the text description related to it, so as to represent them in a unified latent space. Using the CLIP model, a text vector representation can be output based on the input text. A vector representation of a text used in the diffusion model can be combined with the VAE latent space representation of the image to obtain an embedding that considers both image and text information, which can then be used as a guiding condition for the diffusion model to constrain the image generation process of the diffusion model.

[0123] Step 605: Decode the latent space representation of the target image to obtain the target image.

[0124] For example, see Figure 7 , which illustrates the generation framework of image editing. After feature extraction and mapping, the reference image can obtain the reference image word-gram. After the image to be edited and the editing instruction are processed by the multimodal large model, the word-gram of the target image is obtained, and the visual word-gram of the target image can be intercepted from the word-gram of the target image. The image to be edited is encoded by the image encoder to obtain the latent space representation and then input into the diffusion model. During the diffusion process, the diffusion model gradually adds noise to the latent space representation of the image to be edited to obtain a noisy image. Then, in the reverse process, the noisy image is gradually denoised. During the denoising process, it is constrained by the text vector representation corresponding to the editing instruction, the visual word-gram and the reference image word-gram to obtain the latent space representation of the target image. Then, the image decoder is used to decode the image and output the target image.

[0125] The method provided by the above-mentioned embodiment of the present disclosure can first generate word elements of the target image based on the image to be edited and the editing instructions by using the pre-trained multimodal large model when image editing is required, extract the visual word elements of the target image from the word elements of the target image, and generate reference image word elements based on the reference image, so that when the pre-trained diffusion model is used for image editing, the visual word elements of the target image and the reference image word elements of the reference image can be used as guiding conditions to guide the image editing process, improve the accuracy of the latent space representation of the generated target image, and thus improve the editing effect of the target image. The technical solution of the present disclosure can add the visual word elements and the reference image word elements generated by the multimodal large model as guiding conditions, more effectively supervise and guide the image editing process of the diffusion model, so that the generated target image is closer to the user's editing instructions, and improve the editing effect of the target image and the consistency with the editing instructions.

[0126] Exemplary Devices

[0127] Fig.10 is a structural diagram of an image editing device provided by an exemplary embodiment of the present disclosure, such as Fig.10As shown, the device may include:

[0128] A first acquisition module 111 is used to acquire an image to be edited and an editing instruction;

[0129] A first generating module 112 is used to generate word-grams of a target image based on the image to be edited and the editing instruction by using a pre-trained multimodal large model;

[0130] A reading module 113, used for extracting visual word units of a target image from word units of a target image;

[0131] A second generation module 114 is used to generate a latent space representation of a target image based on a latent space representation of the image to be edited, text words corresponding to the editing instruction, and visual words using a pre-trained diffusion model, wherein the pre-trained diffusion model has a multi-conditional guided architecture;

[0132] The decoding module 115 is used to decode the latent space representation of the target image to obtain the target image.

[0133] Fig.11 is a structural schematic diagram of an image editing device provided by another exemplary embodiment of the present disclosure, such as Fig.11 As shown, in Fig.10 Based on the illustrated embodiment, in some implementations, the second generation module 114 includes:

[0134] A cross-attention submodule 1141 is used to obtain a cross-attention feature representation based on a cross-attention mechanism according to latent space representations of text words, visual words and the image to be edited corresponding to the editing instruction;

[0135] The denoising submodule 1142 is used to use the pre-trained diffusion model to remove the noise to be removed corresponding to each time step of the image to be edited based on the cross-attention feature representation, and obtain the latent space representation of the target image.

[0136] In some embodiments, it further includes: a model training module 116, which is used for training to obtain a pre-trained multimodal large model and a pre-trained diffusion model;

[0137] The model training module 116 includes:

[0138] The first acquisition submodule 1161 is used to acquire a training sample set, wherein the training sample set includes at least one group of training samples, each group of training samples includes pre-editing image samples, editing instruction samples, real edited image samples, and editing result description samples;

[0139] The first prediction submodule 1162 is used to obtain corresponding image word-grams using the first to-be-trained model for each pre-editing image sample and the editing instruction sample in each group of training samples, wherein the image word-grams include first text word-grams and first visual word-grams; and obtain a first loss value for training the first to-be-trained model based on the word-grams of the editing result description sample and the first text word-grams;

[0140] A second prediction submodule 1163 is used to obtain, for each group of training samples, a second loss value of a second model to be trained based on the corresponding first visual word-gram, the text word-gram of the editing instruction sample, and the latent space representation of the image sample before editing; wherein the second model to be trained has a diffusion model architecture guided by multiple conditions;

[0141] The training submodule 1164 is used to guide the first model to be trained and the second model to be trained based on the first loss value and the second loss value to obtain the pre-trained multimodal large model and the pre-trained diffusion model.

[0142] In some embodiments, in response to each set of training samples further comprising reference image samples;

[0143] The second prediction submodule 1163 is used to obtain a second loss value of the second model to be trained for each group of training samples based on the corresponding first visual word-grams, the text word-grams of the editing instruction samples, the latent space representation of the image samples before editing, and the reference image word-grams.

[0144] In some embodiments, the device further comprises:

[0145] A third generation module 117, for generating a model query instruction based on the instruction word unit and the tensor;

[0146] The first generation module 112 is used to generate word units of a target image based on a model query instruction using a pre-trained multimodal large model.

[0147] In some embodiments, the device further comprises:

[0148] The second acquisition module 118 is used to acquire at least one reference image; wherein the reference image includes at least one of the following information: image local information to be added or to be deleted in the image to be edited, image style information of the image to be edited, and image filter information of the image to be edited;

[0149] A feature extraction module 119 is used to extract features from at least one reference image to obtain reference image features;

[0150] The fourth generation module 120 is used to generate reference graph word-grams based on reference graph features using a multi-layer perceptron.

[0151] In some embodiments, the second generation module 114 is used to generate a latent space representation of the target image based on the latent space representation of the image to be edited, the text word-grams corresponding to the editing instructions, the visual word-grams of the target image, and the reference image word-grams using a pre-trained diffusion model.

[0152] It should be pointed out that the modules in the present device can be decomposed and / or reassembled, and such decomposition and / or reassembly should be regarded as equivalent solutions of the present device.

[0153] The exemplary embodiment of the device corresponds to the exemplary method described above, and the relevant contents can be referenced and quoted to each other. The beneficial technical effects corresponding to the exemplary embodiment of the device can refer to the corresponding beneficial technical effects of the exemplary method described above, and will not be repeated here.

[0154] Exemplary Electronic Devices

[0155] Fig.12 A structural diagram of an electronic device provided in an embodiment of the present disclosure includes at least one processor 121 and a memory 122.

[0156] The processor 121 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 12 to perform desired functions.

[0157] The memory 122 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 121 may execute one or more computer program instructions to implement the vehicle posture detection method and / or other desired functions of the various embodiments of the present disclosure described above.

[0158] In one example, the electronic device may further include: an input device 123 and an output device 124 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0159] The input device 123 may also include, for example, a keyboard, a mouse, a touch screen, a sound pickup device (such as a microphone array), etc.

[0160] The output device 124 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0161] Of course, to simplify, Fig.12 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0162] Exemplary computer program products and computer-readable storage media

[0163] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the image editing method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0164] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0165] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the image editing method according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0166] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0167] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0168] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0169] The block diagrams of the devices, apparatuses, and equipment involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, and equipment can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", and the like are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0170] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0171] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0172] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0173] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An image editing method, characterized in that: include: Obtain the image to be edited and the editing instructions; Generate word units of a target image based on the image to be edited and the editing instruction using a pre-trained multimodal large model; Extracting visual word-grams of the target image from word-grams of the target image; Using a pre-trained diffusion model, based on the latent space representation of the image to be edited, the text word-gram corresponding to the editing instruction, and the visual word-gram, a latent space representation of the target image is generated; wherein the pre-trained diffusion model has a multi-conditional guided architecture; The latent space representation of the target image is decoded to obtain the target image.

2. The method according to claim 1, characterized in that The method of using the pre-trained diffusion model to generate a latent space representation of the target image based on the latent space representation of the image to be edited, the text word element corresponding to the editing instruction, and the visual word element comprises: Based on the cross-attention mechanism, a cross-attention feature representation is obtained according to the latent space representation of the text word-gram corresponding to the editing instruction, the visual word-gram and the image to be edited; The pre-trained diffusion model is used to remove the noise to be removed corresponding to each time step of the image to be edited based on the cross-attention feature representation, so as to obtain the latent space representation of the target image.

3. The method according to any one of claims 1-2, characterized in that: Before generating word-grams of a target image based on the image to be edited and the editing instruction by using the pre-trained multimodal large model, the method further includes: Generate a model query instruction based on the image to be edited and the editing instruction; The method of using the pre-trained multimodal large model to generate word-units of the target image based on the image to be edited and the editing instruction includes: The pre-trained multimodal large model is used to generate word-grams of the target image based on the model query instruction.

4. The method according to any one of claims 1 to 3, characterized in that: The step of obtaining the image to be edited and the editing instruction also includes: Acquire at least one reference image; wherein the reference image includes at least one of the following information: local image information to be added or to be deleted in the image to be edited, image style information of the image to be edited, and image filter information of the image to be edited; Performing feature extraction on the at least one reference image to obtain reference image features; A multi-layer perceptron is used to generate reference graph words based on the reference graph features.

5. The method according to claim 4, characterized in that The method of using the pre-trained diffusion model to generate a latent space representation of the target image based on the latent space representation of the image to be edited, the text word element corresponding to the editing instruction, and the visual word element comprises: By using a pre-trained diffusion model, a latent space representation of the target image is generated based on the latent space representation of the image to be edited, the text word-gram corresponding to the editing instruction, the visual word-gram of the target image, and the reference image word-gram.

6. The method according to any one of claims 1 to 5, characterized in that: The pre-trained multimodal large model and the pre-trained diffusion model are trained by the following steps: Acquire a training sample set, wherein the training sample set includes at least one group of training samples, each group of training samples includes pre-editing image samples, editing instruction samples, real edited image samples, and editing result description samples; For each pre-editing image sample and the editing instruction sample in each group of training samples, using the first to-be-trained model to obtain corresponding image word-grams, wherein the image word-grams include a first text word-gram and a first visual word-gram; And, based on the word-gram of the editing result description sample and the first text word-gram, a first loss value of training the first model to be trained is obtained; For each group of training samples, a second loss value of a second model to be trained is obtained based on the corresponding first visual word-gram, the text word-gram of the editing instruction sample, and the latent space representation of the image sample before editing; wherein the second model to be trained has a diffusion model architecture guided by multiple conditions; Based on the first loss value and the second loss value, the first model to be trained and the second model to be trained are guided to obtain the pre-trained multimodal large model and the pre-trained diffusion model.

7. The method according to claim 6, characterized in that In response to each set of training samples also including a reference image sample, obtaining a reference image word-gram of the reference image sample; The obtaining a second loss value of the second to-be-trained model includes: For each group of training samples, a second loss value of the second model to be trained is obtained based on the corresponding first visual word-gram, the text word-gram of the editing instruction sample, the latent space representation of the image sample before editing, and the reference image word-gram.

8. An image editing device, characterized in that: include: A first acquisition module, used for acquiring the image to be edited and the editing instruction; A first generation module, configured to generate word-grams of a target image based on the image to be edited and the editing instruction by using a pre-trained multimodal large model; A reading module, used for extracting the visual word element of the target image from the word element of the target image; A second generation module is used to generate a latent space representation of the target image based on the latent space representation of the image to be edited, the text word element corresponding to the editing instruction, and the visual word element by using a pre-trained diffusion model; wherein the pre-trained diffusion model has a multi-conditional guided architecture; A decoding module is used to decode the latent space representation of the target image to obtain the target image.

9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 7.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 7.

Images