Image completion method, device, system and computer-readable storage medium

Abstract

The present application relates to an image completion method, device, system and computer-readable storage medium, wherein the image completion method comprises: acquiring an image of a target object, and acquiring a target mask and a masked image based on the image; encoding a preset prompt word to obtain a text code, and determining a position code of the target object according to the target mask, wherein the prompt word is a character string without actual meaning; concatenating the text code and the position code to obtain a token, and inputting all the tokens into a mask self-attention module to obtain a conditional token; concatenating a target mask, a masked image and a pre-acquired noise image to obtain an image to be processed, and inputting the image to be processed and the conditional token into a BrushNet module to obtain a first intermediate result; using the first intermediate result, the noise image and the conditional token as initial input data, performing iterative processing for a set number of times through a freezing module to obtain an output result, and decoding the output result to obtain a completed image, thereby improving the accuracy of image completion.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image completion method, device, system and computer-readable storage medium. Background Art

[0002] Image completion is the use of deep learning technology to fill in missing areas in images. It can effectively restore damaged or incomplete images and make them complete and natural. At present, the image completion method of the BrushNet network uses mask images and noise images as two processing branches for image completion, and completes the image through specific semantic information of prompt words. However, the image completion method of the BrushNet network can have a good completion effect on common species or categories in nature. If image completion is performed on images in special scenes or categories that are not common in nature, texture and color problems are likely to occur, resulting in poor image completion effects. Therefore, improving the accuracy of image completion in specific scenes has become an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application provide an image completion method, device, system and computer-readable storage medium to at least solve the problem of low image completion accuracy in specific scenarios in the related art.

[0004] In a first aspect, an embodiment of the present application provides an image completion method, characterized in that the method is applied to a BrushNet network, the network includes a BrushNet module and a freeze module, and the method includes:

[0005] Acquire an image of a target object, and obtain a target mask and a masked image based on the image;

[0006] Encoding a preset prompt word to obtain a text code, and determining a position code of the target object according to the target mask, wherein the prompt word is a character string without actual meaning;

[0007] The text code and the position code are concatenated to obtain tokens, and all tokens are input into a masked self-attention module to obtain conditional tokens;

[0008] Concatenating the target mask, the masked image and the pre-acquired noise image to obtain an image to be processed, and inputting the image to be processed and the conditional token into the BrushNet module to obtain a first intermediate result;

[0009] The first intermediate result, the noise image and the conditional token are used as initial input data, and the freezing module is used to perform iterative processing for a set number of times to obtain an output result, and the output result is decoded to obtain a completed image.

[0010] In one embodiment, the step of concatenating the text encoding and the position encoding to obtain tokens, and inputting all tokens into a masked self-attention module to obtain conditional tokens comprises:

[0011] Concatenate the text code and the position code, and input the concatenated code into a multilayer perceptron to obtain the token;

[0012] All the tokens are cascaded and input into the masked self-attention module to obtain the conditional token.

[0013] In one embodiment, determining the position code of the target object according to the target mask includes:

[0014] Based on the target mask, sampling the edge of any target mask instance, and determining the center coordinates of the target mask instance according to the sampling result, and using the center coordinates as the position coordinates of the target mask instance, wherein each target mask instance corresponds to one target object;

[0015] Fourier mapping is performed on the position coordinates to obtain a position code of each target mask instance to determine a position code of each target object.

[0016] In one embodiment, the first intermediate result, the noise image and the conditional token are used as initial input data, the freezing module is iterated a set number of times to obtain an output result, and the output result is decoded to obtain a completed image, including:

[0017] outputting a first intermediate image through the freezing module according to the initial input data;

[0018] Updating the first intermediate image to the noise image, and updating the first intermediate result;

[0019] The conditional token, the updated noise image and the first intermediate result are used as input data for the next iterative processing until the set number of iterative processing is completed to obtain the output result.

[0020] In one embodiment, acquiring an image of the target object and obtaining a target mask and a masked image based on the image includes:

[0021] Acquire an image of the target object, and mark the image along the outline of the target object;

[0022] Based on the marked contour, the target mask and the masked image are obtained.

[0023] In one embodiment, the conditional token is input into the BrushNet module and the freezing module respectively, and the first intermediate result is input into the freezing module, which is specifically configured as follows:

[0024] Inputting the conditional token into the BrushNet module by means of zero convolution; and / or

[0025] Inputting the conditional token into the freezing module by means of zero convolution; and / or

[0026] The first intermediate result is input into the freezing module by means of zero convolution.

[0027] In a second aspect, an embodiment of the present application provides an image completion device, including:

[0028] A first acquisition module, used for acquiring an image of a target object, and obtaining a target mask and a masked image based on the image;

[0029] A determination module, used for encoding a preset prompt word to obtain a text code, and determining a position code of the target object according to the target mask, wherein the prompt word is a character string without actual meaning;

[0030] A second acquisition module is used to concatenate the text code and the position code to obtain tokens, and input all tokens into a masked self-attention module to obtain conditional tokens;

[0031] An input module, configured to obtain a to-be-processed image by concatenating the target mask, the masked image and the pre-acquired noise image, and input the to-be-processed image and the conditional token into the BrushNet module to obtain a first intermediate result;

[0032] The third acquisition module is used to use the first intermediate result, the noise image and the conditional token as initial input data, perform iterative processing for a set number of times through the freezing module to obtain an output result, and decode the output result to obtain a completed image.

[0033] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image completion method as described in the first aspect above is implemented.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image completion method as described in the first aspect above.

[0035] An image completion method, device, system, and computer-readable storage medium provided by the embodiments of the present application have at least the following technical effects.

[0036] A meaningless string is used as a prompt word, and a text encoding is obtained, and the position encoding of the target object is determined according to the target mask, so that the meaningless string is associated with the target object, and the target object in a specific scene can be determined according to the string. Therefore, based on the BrushNet network, the conditional token containing text information and position information is used as the input of the BrushNet module and the freeze module, so as to reduce the interference of the prompt word with specific meaning when completing the image. By inputting the conditional token into the BrushNet module and the freeze module, the regulating effect of the input condition in the BrushNet network is enhanced, so that the target object and the prompt word are better matched in a specific scene, thereby improving the accuracy of completing the image in a specific scene.

[0037] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 is a flowchart of an image completion method according to an exemplary embodiment;

[0040] Figure 2 is a schematic diagram showing an image completion method according to an exemplary embodiment;

[0041] Figure 3 is a schematic diagram of iterative processing according to an exemplary embodiment;

[0042] Figure 4 is a block diagram of an image completion device according to an exemplary embodiment;

[0043] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment.

[0044] In the above drawings, the meanings of the reference numerals are as follows:

[0045] 100. Pre-acquired noise image; 200. First image; 300. First intermediate image;

[0046] 400, BrushNet module; 500, Freeze module; DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0048] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0049] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0050] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0051] At present, the image completion method in the prior art is mainly for completing the common category targets in natural vision, and can only complete one target. However, when completing the target of a specific detection requirement scene, texture and color anomalies will appear in the completed image, resulting in poor completion image effect. The BrushNet method is a model that uses dual branches to process images, aiming to ensure and enhance the image restoration effect. In the BrushNet model, the masked image and the noise image are divided into two branches. After the images of the two branches are processed separately, the image information is merged and the image is completed according to the prompt word, thereby improving the image restoration effect. Using the BrushNet network for image completion requires three processing branches. The first processing branch is to process the mask image, and the second processing branch is to process the noise. In the first processing branch, the processed mask image, the processed mask image and the noise image are input into the BrushNet module for processing. In the second processing branch, the noise image is input into the freezing module for processing. The third processing branch is to input the prompt word into the freezing module. The Brushnet network can complete images based on prompt words. The more specific and clear the prompt words are, the better the effect of the completed image will be.

[0052] Although the image completion method based on the above-mentioned BrushNet network can obtain a completed image with good effect, it still has the problem of poor repair effect when completing the target in the scene required for specific detection. Therefore, this application optimizes the input of the BrushNet module and the freeze module of the BrushNet network and optimizes the input of the prompt word on the network structure based on the BrushNet method to improve the accuracy of the completed image in the specific requirement scene.

[0053] Based on the above situation, the embodiments of the present application provide an image completion method, device, system and computer-readable storage medium.

[0054] In a first aspect, an embodiment of the present application provides an image completion method. Figure 1 is a flowchart of an image completion method according to an exemplary embodiment. Figure 1 As shown, the image completion method includes:

[0055] Step S101: Acquire an image of a target object, and obtain a target mask and a masked image based on the image.

[0056] Specifically, multiple images containing the target object are obtained, and the multiple images are images in which the target object appears in different scenes. Among them, the target object is an uncommon target category in nature, which is affected by the existence time or depends on harsh formation conditions. The target objects specifically include tumors formed by uncontrolled proliferation, ice crystals that depend on low temperature and high humidity environments, double rainbows under specific sunlight and raindrop conditions, and irregular scratches caused by humans. The image of the target object is a picture of the same target object in different scenes. For example, if the target object is a pig tumor, the image of the target object is a picture of the pig tumor in different scenes.

[0057] In one example, step S101 includes:

[0058] Step S1011: Acquire an image of the target object, and mark the image along the outline of the target object.

[0059] Optionally, an image of the target object is obtained, and annotated along the contour of the target object in the image to form a closed figure. It should be noted that, when the image completion network provided in the embodiment of the present application is used to annotate the contour of the target object, a better completion effect can be achieved using images with a smaller number of frames. Specifically, the number of frames can be 3 frames, 5 frames, 10 frames, etc. The specific analysis will be carried out in detail below in conjunction with the structure of the image completion network.

[0060] Step S1012: Obtain a target mask and a masked image based on the marked contour.

[0061] According to the marked contour, the target mask and the masked image are generated. The target mask is a binary image. The pixel value of the area where the marked contour is located is assigned to 255, and the pixel value of other areas is assigned to 0, that is, the area where the contour is located is white and the background is black. The masked image is the image content of the target object with the pixel value of the area where the contour image is located assigned to 0, and the other areas are still the image of the target object. The target mask and the masked image are the same size as the image of the target object.

[0062] Step S102: Encode the preset prompt word to obtain a text code, determine the position code of the target object according to the target mask, and the prompt word is a character string without actual meaning.

[0063] To obtain the text encoding, the pre-set prompt word is encoded by the text encoder to obtain the text encoding. The prompt word is a string of letters without actual meaning, such as English letters, French letters, Latin letters, etc. However, the prompt word cannot be a random combination of alphabetic characters, numbers, and special characters. If English characters, special characters, and numbers are randomly combined, then in the image completion process, it is easy to use special characters or numbers as split points, and the prompt word is disassembled into multiple single characters. For example, the prompt words that cannot be used include sys2 and *sk#hs.

[0064] It should be noted that the prompt word has no specific meaning in the initial stage of image completion by the completion network, but after training the completion network, the prompt word can establish a connection with the target object to give the prompt word a clear meaning. Optionally, the prompt words include sks and linx_tum. For example, scratches on the surface of the battery are an uncommon category in nature. The preset prompt word is sks, and the target object is scratches on the surface of the battery. After training the completion network, sks is given the meaning of scratches on the surface of the battery, and the prompt word sks refers to scratches on the surface of the battery. If sks is entered, the completion network will complete the scratches on the surface of the battery in the preset area.

[0065] During the training of the image completion network, the prompt words without actual meaning can establish a corresponding connection with the target objects provided during the training process, so that the completion network can determine the corresponding target objects based on the prompt words, thereby avoiding the influence of the original semantics of the prompt words on image completion, so that the objects completed by the image completion network based on the prompt words are not limited to common species in nature. By establishing a connection between the prompt words and the target objects, the text content of the prompt words corresponds to the image, avoiding the situation in which the completion network cannot complete the corresponding target objects based on the prompt words in specific scenarios.

[0066] For obtaining the position code, based on the target mask, the edge of any target mask instance is sampled, and the center coordinates of the target mask instance are determined according to the sampling result, and the center coordinates are used as the position coordinates of the target mask instance. It should be noted that in the target mask, the target mask instance is the area of ​​each contour image, and each target mask instance corresponds to a target object.

[0067] In one embodiment, in the target mask, for any target mask instance, multiple points (e.g., 8, 10, 12, 15, etc.) are uniformly sampled along the edge of the target mask instance, and the coordinates of the geometric center of the corresponding target mask instance are determined based on the multiple points obtained by sampling, and the coordinates of the geometric center are the position coordinates of the target mask instance. Further, the position coordinates are Fourier mapped to obtain the position code of each target mask instance to determine the position code of each target object. Using the image completion network provided in the embodiment of the present application, multiple contours can be marked on the image of the target object, thereby determining the position code of each target mask instance based on the multiple contours, so that images at multiple positions can be completed.

[0068] The prompt word in step S102 uses a character string without actual meaning, that is, the prompt word has no meaning in the initial state, but after training the image completion network, the meaning of the prompt word is equivalent to the target object. The image completion network completes the corresponding target object in the image to be completed according to the prompt word. Since the prompt word has no actual meaning, it will not affect the image semantics of the completed image during the image completion process, thereby improving the accuracy of the final completed image. On the other hand, the position coding determines the position of the target object, so that in the image completion process, the position to be completed can be determined according to the position coding, so that the target object is accurately completed at the position determined by the position coding, thereby improving the accuracy of the completed image.

[0069] Step S103: Concatenate the text code and the position code to obtain tokens, and input all tokens into the masked self-attention module to obtain conditional tokens.

[0070] Obtaining a conditional token consists of the following two steps:

[0071] Step S301, concatenate the text code and the position code, and input the concatenated code into a multi-layer perceptron to obtain a token.

[0072] The text encoder is used to convert text information into a vector. Therefore, the text encoding obtained by the text encoder is a vector with a fixed length. Fourier mapping is to map low-dimensional coordinates to high-dimensional vectors to represent position information. Therefore, the position encoding obtained by Fourier mapping the position coordinates of the target mask instance is also a vector with a fixed length.

[0073] The text encoding and position encoding are connected in series, that is, the text encoding and position encoding are connected in sequence to form a longer feature vector. The feature dimension of the obtained feature vector becomes larger, and the feature vector contains the text information of the prompt word and the position information of the target mask instance. For example, the text encoding is a text vector with 1 column and 10 rows, and the position encoding is a position vector with 1 column and 20 rows. The text encoding and position encoding are connected in series to form a feature vector with 1 column and 30 rows.

[0074] It should be noted that the token only includes the text information of the prompt word and the location information of the target mask instance, but does not include other information, thus improving the foundation for subsequent efficient image completion processing.

[0075] The feature vector is input into MLP (Multilayer Perceptron) to obtain a token. The feature vector is input into MLP, and the output of MLP is still a vector. Among them, the token contains the text information of the prompt word and the location information of the target mask instance. Since there are at least two target mask instances in the image of the target object, there is a corresponding token for each target mask instance.

[0076] Step S302: Concatenate all tokens into the masked self-attention module to obtain conditional tokens.

[0077] Concatenate all tokens, that is, merge each token into a new vector. Concatenating all tokens maintains the feature dimension of the original tokens, that is, retains the original text information and position information, and increases the number of samples. For example, each token is a vector of 1 column and 40 rows, and the total number of tokens is 5. By concatenating these 5 tokens, a vector of 5 columns and 40 rows can be obtained.

[0078] In order to prevent information leakage between different target object instances, the new vector is input into the mask self-attention module, and the conditional token is obtained by processing all tokens through the mask self-attention module. The conditional token only includes the text information of the prompt word and the location information of the target mask instance. Therefore, the conditional token has more efficient targeting and simplicity, which not only simplifies subsequent processing, but also makes the image completion processing more accurate.

[0079] Step S103 obtains a token with text information and position information by concatenating text encoding and position encoding in series, thereby improving the matching of the token with the target object, and then obtains a conditional token by concatenating all tokens. The conditional token not only maintains the text information and position information of the original token, but also adds the information of several tokens, making the conditional token more efficient, targeted and simple, thereby improving the accuracy of image completion.

[0080] Step S104: The image to be processed is obtained by concatenating the target mask, the masked image and the pre-acquired noise image, and the image to be processed and the conditional token are input into the BrushNet module to obtain a first intermediate result.

[0081] The target mask is inverted and the inverted target mask is downsampled to obtain a target mask image of the target size. Inverting the target mask can enhance the information of the target mask instance. The target size is consistent with the size of the pre-acquired noise image. The masked image is encoded by a variational autoencoder (VAE) so that the masked image is spatially aligned with the pre-acquired noise image to obtain the processed masked image.

[0082] The inverted target mask, the processed masked image and the pre-acquired noise image are cascaded to obtain the image to be processed, wherein the inverted target mask, the processed masked image and the pre-acquired noise image have the same size.

[0083] In step S104, the image to be processed and the conditional token are input into the BrushNet module through zero convolution to obtain a first intermediate result.

[0084] Through zero convolution, the image to be processed and the conditional token are input into the BrushNet module, which strengthens and enriches the location information and text information of the target object, avoiding the problem of the target object's information being gradually weakened during network transmission.

[0085] Step S105: Use the first intermediate result, the noise image and the conditional token as initial input data, perform iterative processing a set number of times through the freezing module to obtain an output result, and decode the output result to obtain a completed image.

[0086] The first intermediate result, the pre-acquired noise image and the conditional token are used as initial input data, the initial input data is input into the freezing module, and an iterative process is performed a set number of times to obtain an output result. The output result after the iterative process is decoded to obtain a completed image. Among them, the number of iterative processes set is not specifically limited. In order to take into account the effect of the completed image and the efficiency of the completed image, the set number of times optionally includes 20 times, 30 times, 40 times, and 50 times. Preferably, the set number of times is set to 30 times. It should be noted that the pre-acquired noise image is a randomly generated noise image.

[0087] Before the set number of iterations, the conditional token has been generated and will not change with the update of the noise image. Therefore, the conditional token will not change during the iteration process. The set number of iterations specifically includes the following steps:

[0088] Step S501: Output a first intermediate image through a freezing module according to initial input data.

[0089] In any iterative process, the initial input data is input to the freezing module through zero convolution, and the first intermediate image output by the freezing module is obtained.

[0090] Step S502: Update the first intermediate image to a noise image, and update the first intermediate result.

[0091] The first intermediate image is updated to a noise image, that is, the pre-acquired noise image is replaced by the first intermediate image. The first intermediate image, the inverted target mask, and the processed masked image are concatenated to obtain the image to be processed. The image to be processed and the conditional token are input into the BrushNet module by zero convolution, thereby updating the first intermediate result.

[0092] Step S503: Use the conditional token, the updated noise image and the first intermediate result as input data for the next iterative processing, until the set number of iterative processing is completed to obtain the output result.

[0093] The conditional token, the updated noise image and the first intermediate result are used as input data for the next iterative process, and the iterative process is continuously performed according to the contents of step S501 to step S503 until the set number of iterative processes are completed. After the last iterative process is completed, the output result is obtained. The output result is decoded to obtain the completed image.

[0094] It should be noted that the method of inputting the conditional token into the BrushNet module and the frozen module is a zero convolution method. By adding the text information and position information in the conditional token to each layer of the BrushNet module and the frozen module in a zero convolution manner, the information of the target object in the two modules is strengthened and enriched, avoiding the problem in the prior art that the information of the target object is only added to one of the modules, thereby gradually weakening the information of the target object when performing multi-layer network propagation.

[0095] In addition, since the prompt word is a string of letters without actual meaning, it is an unfamiliar word for the text encoder. Therefore, the unfamiliar word has no initial semantics and will not be disturbed by other semantics. The prompt word can be associated with the target object through a small number of images of the target object. On the other hand, the conditional token carrying the prompt word text information is input into the BrushNet module, so that the BrushNet module can fully learn the key visual features of the target object and pass it to the freezing module so that the BrushNet network can generate the target object. Based on the above two aspects, the use of a string of letters without actual meaning accelerates the determination of the semantic consistency between the target object and the prompt word, so that a better completion effect can be achieved by using a smaller number of frames of images. For example, the specific meaning of tree is tree, and tree can be associated with the image features of various trees. If tree is used as the prompt word for tumor, then a large number of tumor images need to be used to train the tree prompt word, and it needs to be trained many times before it can change. Therefore, a large number of training images and an increase in the number of training times are required to achieve a better completion effect, otherwise it will be affected by the original semantics of tree, resulting in poor completion effect.

[0096] In one embodiment, the number of iterative processing is configured to be 30 times, and the pre-acquired noise image is a randomly generated noise image. After obtaining the conditional token, in the first iterative processing, the image to be processed is cascaded according to the negated target mask, the processed masked image and the pre-acquired noise image. The image to be processed and the conditional token are input into the BrushNet module to obtain the first intermediate result. The first intermediate result, the pre-acquired noise image and the conditional token are used as initial input data, and the initial input data is input into the freezing module to obtain the first intermediate image, which indicates that the first iterative processing is over. In the second iterative processing, the first intermediate image replaces the pre-acquired noise image, and the image to be processed is obtained according to the first intermediate image, the negated target mask and the processed masked image, and the image to be processed and the conditional token are input into the BrushNet module to obtain the second intermediate result. Since the initial data processed are different, the second intermediate result in the second iterative processing is different from the first intermediate result in the first iterative processing. Therefore, the second intermediate result, the first intermediate image and the conditional token are used as initial input data, and the initial input data is input into the freezing module to obtain the second intermediate image. The above iterative process is repeated until the final output result is obtained by freezing the module in the 30th iterative process.

[0097] Figure 2 is a schematic diagram of an image completion method according to an exemplary embodiment. Figure 2As shown, the top image of the positive electrode of the cylindrical battery is obtained, and the position where the battery scratches need to be completed is selected to obtain the target mask and the masked image. The preset prompt word is passed through the text encoder to obtain the text code, and the position code is determined according to the Fourier map and the target mask. The text code and the position code are connected in series to obtain the token, and all the tokens are cascaded and input into the mask self-attention module to obtain the conditional token. The target mask is inverted and downsampled to obtain the target mask of the target size. The masked image is encoded to obtain the processed masked image. The target mask of the target size, the processed masked image and the pre-acquired noise image are fused to obtain the image to be processed. The image to be processed and the conditional token are input into the BrushNet module 400 to obtain the first intermediate result. The first intermediate result and the pre-acquired noise image are input into the freezing module 500 to obtain the output result, and then the output result is encoded to obtain the completed image. Among them, in the process of obtaining the output result in the freezing module 500, a preset number of iterative processing is required, Figure 3 is a schematic diagram of iterative processing according to an exemplary embodiment. Figure 3 As shown, in the first iterative processing, the conditional token, the pre-acquired noise image 100 and the target mask of the target size, and the first image 200 formed by the processed masked image are input in series into the BrushNet module 400, and the pre-acquired noise image and the first intermediate result output by the BrushNet module 400 are input into the freezing module 500 to obtain the first intermediate image 300, completing the first iterative processing. The first intermediate image 300 is used as the noise image, and in the second iterative processing, the steps of the first iterative processing are continued, the difference being that the noise image is different, until the set number of iterative processing is completed, and the first intermediate image obtained by the last iterative processing is used as the output result, and the image of the scratch in the positive electrode of the cylindrical battery after completion is obtained.

[0098] In another embodiment, several frames of images to be completed that do not contain the target object are randomly selected, and the position where the target object is expected to be generated is marked in the image to be completed, and a target mask and a masked image are generated. Based on the connection established between the target object and the prompt word, the prompt word is input, and the image completion network can generate the target object at the marked position according to the prompt word. For example, the prompt word for the tumor is set to sks, and the correspondence between the tumor and the target object is established according to steps S101 to S105. In an image that does not contain a tumor, the position of the tumor that is expected to be generated is marked, and the corresponding target mask and masked image are generated. Based on the image completion network, the prompt word sks is input, and the tumor can be generated at the specified position in the image to be completed. And the specified position is not limited to one, it can be multiple.

[0099] Two frames of pig tumor images are selected as training images for the Dreambooth completion method and the image completion method of the present application. After training, the Dreambooth completion method and the image completion method of the present application can complete pig tumors. The Dreambooth completion method and the image completion method of the present application perform image completion on the image of the same pig. The position to be completed is marked on the image of the pig, that is, the position where the tumor is expected to be generated. The position to be completed is a closed figure, which represents the generation of a tumor in a closed figure. According to the position to be completed, a target mask is obtained. Among them, the position to be completed includes a first target position and a second target position. In the target mask, the pixel values ​​of the first target position and the second target position are assigned to 255, and the pixel values ​​of other areas are assigned to 0, that is, the area where the first target position and the second target position are located is white, and the background is black. Based on the image and target mask of the same pig, the image of the same pig is completed by the Dreambooth completion method and the image completion method of the present application.

[0100] Through the Dreambooth completion method, in the pig image, the tumor at the first target position is completed, but the second target position is not completed successfully, and a hole is left. Through the image completion method of the present application, in the pig image, the tumors at both target positions are completed successfully. Therefore, the image completion method of the present application can complete images at multiple positions.

[0101] In the Dreambooth completion method, although the tumor is completed at the first target position, the completed tumor does not completely match the range marked by the first target position, and the boundary between the completed tumor and the surrounding skin is obvious. In addition, the color and texture of the completed tumor cannot be distinguished from the surrounding skin, the characteristics of the original skin cannot be retained, there is a large difference from the surrounding skin, and the completed tumor has a large abruptness. In the image completion method of the present application, the two target positions are completed. The transition between the completed tumor and the surrounding skin is smooth, and the color and texture of the tumor are consistent with the surrounding environment, retaining the characteristics of the original skin, so that the completed image remains natural and smooth. According to the above comparison of the completion effects, the image completion method of the present application has a better completion effect, making the completed image more natural and realistic, and retaining the style of the original image.

[0102] In summary, the image completion method provided by the embodiment of the present application establishes a connection with the target object through a prompt word, so that the prompt word corresponds to the target object. At the same time, the position information of the target object and the text information of the prompt word are used as the dimensions of the conditional token, so that the text information and position information of the prompt word are contained in the conditional token. By inputting the conditional token into the BrushNet module and the freezing module respectively, the output result is obtained and the completion graph is obtained by decoding. By the above-mentioned method, the text information of the prompt word and the position information of the target object are input into the BrushNet module and the freezing module, the corresponding matching between the prompt word and the completion target object is improved, and the prompt word will not affect the image semantics of the completion image, and the completion target object is provided with accurate text information, and the position information can provide an accurate position for the completion target object. In the continuous iterative processing process, the two information are further deepened in the image to be completed, so that the image completion network can complete the target object according to the text information and the position information, thereby improving the accuracy of the completed image, and multiple positions can be positionally completed, thereby improving the accuracy of image completion.

[0103] In a second aspect, an embodiment of the present application provides an image completion device. Figure 4 FIG. 1 is a block diagram of an image completion device according to an exemplary embodiment. Figure 4 As shown, the image completion device includes:

[0104] A first acquisition module is used to acquire an image of a target object, and obtain a target mask and a masked image based on the image;

[0105] A determination module is used to encode the preset prompt word to obtain a text code, determine the position code of the target object according to the target mask, and the prompt word is a character string without actual meaning;

[0106] The second acquisition module is used to concatenate the text encoding and the position encoding to obtain tokens, and input all tokens into the masked self-attention module to obtain conditional tokens;

[0107] An input module, used for obtaining a to-be-processed image by concatenating the target mask, the masked image and the pre-acquired noise image, and inputting the to-be-processed image and the conditional token into a BrushNet module to obtain a first intermediate result;

[0108] The third acquisition module is used to use the first intermediate result, the noise image and the conditional token as initial input data, perform iterative processing for a set number of times through the freezing module to obtain an output result, and decode the output result to obtain a completed image.

[0109] In summary, the image completion device provided by the present application uses a character string without actual meaning as a prompt word, obtains a text encoding, and determines the position encoding of the target object according to the target mask, so that the character string without actual meaning is associated with the target object, thereby determining the target object in a specific scene according to the character string. Therefore, based on the BrushNet network, the conditional token containing text information and position information is used as the input of the BrushNet module and the freeze module, so as to reduce the interference of prompt words with specific meanings when completing the image. By inputting the conditional token into the BrushNet module and the freeze module, the regulating effect of the input conditions in the BrushNet network is enhanced, so that the target object and the prompt word are better matched in a specific scene, thereby improving the accuracy of completing the image in a specific scene.

[0110] It should be noted that the image completion device provided in this embodiment is used to implement the above-mentioned implementation mode, and the descriptions that have been made will not be repeated. As used above, the terms "module", "unit", "sub-unit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the device described in the above embodiment is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0111] In a third aspect, an embodiment of the present application provides an electronic device, Figure 5 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 5 As shown, the electronic device may include a processor 81 and a memory 82 storing computer program instructions.

[0112] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0113] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0114] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0115] The processor 81 implements any one of the image completion methods in the above embodiments by reading and executing computer program instructions stored in the memory 82 .

[0116] In one embodiment, the image completion device may further include a communication interface 83 and a bus 80. Figure 5 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0117] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0118] The bus 80 includes hardware, software or both, and couples the components of the image completion device to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0119] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the image completion method provided in the first aspect is implemented.

[0120] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0121] In a possible implementation manner, the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of the image completion method provided in the first aspect.

[0122] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on a user device, partially on a user device, as an independent software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0123] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image completion method, characterized in that: The method is applied to a BrushNet network, the network comprising a BrushNet module and a freeze module, and the method comprises: Acquire an image of a target object, and obtain a target mask and a masked image based on the image; Encoding a preset prompt word to obtain a text code, wherein the prompt word is a character string without actual meaning; Based on the target mask, sampling the edge of any target mask instance, and determining the center coordinates of the target mask instance according to the sampling result, and using the center coordinates as the position coordinates of the target mask instance, wherein each target mask instance corresponds to one target object; Performing Fourier mapping on the position coordinates to obtain a position code of each target mask instance to determine a position code of each target object; The text code and the position code are concatenated to obtain tokens, and all tokens are input into a masked self-attention module to obtain conditional tokens; Concatenating the target mask, the masked image and the pre-acquired noise image to obtain an image to be processed, and inputting the image to be processed and the conditional token into the BrushNet module to obtain a first intermediate result; The first intermediate result, the noise image and the conditional token are used as initial input data, and the freezing module is used to perform iterative processing for a set number of times to obtain an output result, and the output result is decoded to obtain a completed image.

2. The image completion method according to claim 1, characterized in that: The step of concatenating the text encoding and the position encoding to obtain tokens, and inputting all tokens into a masked self-attention module to obtain conditional tokens comprises: Concatenate the text code and the position code, and input the concatenated code into a multilayer perceptron to obtain the token; All the tokens are cascaded and input into the masked self-attention module to obtain the conditional token.

3. The image completion method according to claim 1, characterized in that: The method of using the first intermediate result, the noise image and the conditional token as initial input data, performing iterative processing for a set number of times through the freezing module to obtain an output result, and decoding the output result to obtain a completed image includes: outputting a first intermediate image through the freezing module according to the initial input data; Updating the first intermediate image to the noise image, and updating the first intermediate result; The conditional token, the updated noise image and the first intermediate result are used as input data for the next iterative processing until the set number of iterative processing is completed to obtain the output result.

4. The image completion method according to claim 1, characterized in that: The step of acquiring an image of the target object and obtaining a target mask and a masked image based on the image comprises: Acquire an image of the target object, and mark the image along the outline of the target object; Based on the marked contour, the target mask and the masked image are obtained.

5. The image completion method according to claim 1, characterized in that: The manner of inputting the conditional token into the BrushNet module and the freezing module respectively, and the manner of inputting the first intermediate result into the freezing module are specifically configured as follows: Inputting the conditional token into the BrushNet module by means of zero convolution; and / or Inputting the conditional token into the freezing module by means of zero convolution; and / or The first intermediate result is input into the freezing module by means of zero convolution.

6. An image completion device, characterized in that: The device is applied to a BrushNet network, which includes a BrushNet module and a freeze module, including: A first acquisition module, used for acquiring an image of a target object, and obtaining a target mask and a masked image based on the image; A text encoding module is obtained, which is used to encode a preset prompt word to obtain a text code, wherein the prompt word is a character string without actual meaning; A position coordinate obtaining module is used to sample the edge of any target mask instance based on the target mask, and determine the center coordinate of the target mask instance according to the sampling result, and use the center coordinate as the position coordinate of the target mask instance, wherein each target mask instance corresponds to one target object; A position coding module is used to perform Fourier mapping on the position coordinates to obtain the position coding of each target mask instance to determine the position coding of each target object; A conditional token obtaining module is used to obtain a token by serializing the text code and the position code, and input all the tokens into a masked self-attention module to obtain a conditional token; An input module, configured to obtain an image to be processed by concatenating the target mask, the masked image and the pre-acquired noise image, and input the image to be processed and the conditional token into the BrushNet module to obtain a first intermediate result; The module for obtaining a completed image is used to use the first intermediate result, the noise image and the conditional token as initial input data, perform iterative processing a set number of times through the freezing module to obtain an output result, and decode the output result to obtain a completed image.

7. An electronic device, characterized in that: The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image completion method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image completion method according to any one of claims 1 to 5 is implemented.

Images