Matting method and device, electronic equipment and storage medium

Abstract

The application provides a matting method and device, electronic equipment and storage medium. The method comprises: obtaining a to-be-processed image; obtaining text information input by a foreground of the to-be-processed image, and calling a text encoder to encode the text information to obtain a text embedding vector; calling an image encoder to encode the to-be-processed image to obtain an image embedding vector of the to-be-processed image; inputting the text embedding vector and the image embedding vector into a multi-modal cross self-attention model to obtain a target vector; and decoding the target vector by using a decoder to obtain a mask image of the to-be-processed image. The above method can improve the matting accuracy and effect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, relates to the field of image processing, and particularly relates to a matting method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the increasing development of e-commerce and live broadcast industries, it is crucial to obtain image editing materials (such as foreground images and video special effects). At present, image editing materials are mainly obtained by matting technology. If an image includes multiple foregrounds, without using a reference image of an original image for matting, the subject is not clear, which will result in inaccurate matting or poor matting effect, thereby affecting the quality of image editing materials and making it difficult to obtain a better image effect. SUMMARY

[0003] In view of the above, it is necessary to provide a matting method, device, electronic device, and storage medium, which can solve the technical problems of inaccurate matting and poor matting effect.

[0004] In one aspect, the present application provides a matting method, which includes: obtaining a to-be-processed image, obtaining text information input for a foreground of the to-be-processed image, and calling a text encoder to encode the text information to obtain a text embedding vector; calling an image encoder to encode the to-be-processed image to obtain an image embedding vector of the to-be-processed image; inputting the text embedding vector and the image embedding vector into a multi-modal cross self-attention model to obtain a target vector; and using a decoder to decode the target vector to obtain a mask image of the to-be-processed image.

[0005] In some embodiments of the present application, the method further includes: generating an image editing material according to pixel values of corresponding pixel points between the mask image and the to-be-processed image, and generating a video special effect and a commodity advertisement based on the image editing material.

[0006] In some embodiments of the present application, the text editor includes an embedding layer and a first self-attention encoder, and the calling of the text encoder to encode the text information to obtain a text embedding vector includes: preprocessing the text information to obtain a text sequence, calculating a first position encoding vector corresponding to each word vector in the text sequence, and adding the first position encoding vector to the text sequence to obtain a text position sequence; calling the embedding layer to convert the text position sequence to obtain a text conversion vector; inputting the text conversion vector into the first self-attention encoder to obtain a text encoding vector; and performing a pooling operation on the text encoding vector to obtain the text embedding vector.

[0007] In some embodiments of the present application, the adding the first position encoding vector into the text sequence to obtain a text position sequence comprises: generating an operation vector corresponding to each word vector according to the each word vector and the first position encoding vector corresponding to the each word vector, replacing the each word vector with the operation vector corresponding to the each word vector in the text sequence to obtain the text position sequence.

[0008] In some embodiments of the present application, the image encoder comprises a second self-attention encoder, and the encoding the image to be processed to obtain an image embedding vector of the image to be processed comprises: preprocessing the image to be processed to obtain a plurality of image blocks, calculating a second position encoding vector corresponding to each pixel value in each image block, generating an image vector of each image block according to the pixel values in the each image block, adding the second position encoding vector corresponding to each pixel value in the each image vector into the each image vector to obtain an image position vector corresponding to the each image vector, inputting each image position vector into the second self-attention encoder to obtain an image encoding vector corresponding to the each image position vector, and performing a pooling operation on a plurality of image encoding vectors to obtain the image embedding vector.

[0009] In some embodiments of the present application, the preprocessing the image to be processed to obtain a plurality of image blocks comprises: resizing the image to be processed, and dividing the resized image to be processed into a plurality of image blocks that do not overlap with each other.

[0010] In some embodiments of the present application, the multi-modal cross self-attention model comprises a self-attention mechanism and a feedforward neural network, and the inputting the text embedding vector and the image embedding vector into a preset multi-modal cross self-attention model to obtain a target vector comprises: performing cross-attention calculation on the text embedding vector and the image embedding vector by the self-attention mechanism to obtain a plurality of initial context feature vectors, mapping each initial context feature vector by the feedforward neural network to obtain a target context feature vector corresponding to the each initial context feature vector, and fusing a plurality of target context feature vectors to obtain the target vector.

[0011] In an aspect, the present disclosure provides a matting device. The matting device comprises: an obtaining unit configured to obtain a to-be-processed image; the obtaining unit is further configured to obtain text information of a foreground input for the to-be-processed image; an encoding unit configured to call a text encoder to encode the text information to obtain a text embedding vector; the encoding unit is further configured to call an image encoder to encode the to-be-processed image to obtain an image embedding vector of the to-be-processed image; an input unit configured to input the text embedding vector and the image embedding vector into a multi-modal cross self-attention model to obtain a target vector; and a decoding unit configured to decode the target vector using a decoder to obtain a mask image of the to-be-processed image.

[0012] In another aspect, the present disclosure provides an electronic device. The electronic device comprises: a memory configured to store at least one instruction; and a processor configured to execute the at least one instruction to implement the matting method.

[0013] In another aspect, the present disclosure provides a computer-readable storage medium. The computer-readable storage medium stores at least one instruction. The at least one instruction is executed by a processor in an electronic device to implement the matting method.

[0014] In the above embodiment, the text information of the foreground input is obtained, wherein the text information can reflect the matting demand of the user. The text embedding vector obtained by encoding the text information is input into the multi-modal cross self-attention model, which can make the multi-modal cross self-attention model clear the subject of matting, thereby ensuring the accuracy and flexibility of matting. The image encoder can learn the features of different scales of the to-be-processed image, so that the image embedding vector contains more local information and global information of the to-be-processed image. Since the multi-modal cross self-attention model can process the text image information at the same time, and the attention mechanism in the multi-modal cross self-attention model has strong learning ability, the multi-modal cross self-attention model can accurately fuse the matting demand of the user and the detail information and global information of the to-be-processed image. Since the target vector accurately fuses the matting demand of the user and the detail information and global information of the to-be-processed image, decoding the target vector can improve the matting accuracy and matting effect. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 FIG. 1 is a structural diagram of an electronic device according to an embodiment of the present disclosure.

[0016] Figure 2 FIG. 2 is a flowchart of a matting method according to an embodiment of the present disclosure.

[0017] Figure 3is a schematic diagram of a mask image provided by an embodiment of the present application.

[0018] Figure 4 is a flowchart of a method for generating a text embedding vector provided by an embodiment of the present application.

[0019] Figure 5 is a flowchart of a method for generating an image embedding vector provided by an embodiment of the present application.

[0020] Figure 6 is a flowchart of a method for generating a target context feature vector provided by an embodiment of the present application.

[0021] Figure 7 is a functional module diagram of a matting device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] It should be noted that, in the present application, "at least one" means one or more, and "multiple" means two or more than two. The "and / or" describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0024] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0025] With the increasing development of e-commerce and live broadcast industries, it is crucial to obtain image editing materials (such as foreground images and video special effects, etc.). Currently, image editing materials are mainly obtained by performing matting on images through matting technology. If an image includes multiple foregrounds, without using a reference image of an original image for matting, the subject will not be clear, which will result in inaccurate matting or poor matting effect, thereby affecting the quality of image editing materials and making it difficult to obtain a better image effect.

[0026] To solve the above technical problems, the present application provides a matting method and device, electronic equipment and storage medium, which can improve the matting accuracy and effect. The matting method provided by the present application can be applied to one or more electronic equipment.

[0027] As shown in Figure 1 Fig. 1 is a structural diagram of an electronic equipment provided by an embodiment of the present application. The electronic equipment 10 can be a mobile phone, a tablet computer, a notebook computer, a computer or the like, and the specific type of the electronic equipment is not limited in the present application.

[0028] As shown in Figure 1 The electronic equipment 10 can include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104 and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102 and the input / output interface 104 through the bus 105.

[0029] The communication module 101 can include a wired communication module and / or a wireless communication module. The wired communication module can provide one or more of the following wired communication solutions: universal serial bus (USB), controller area network bus (CAN), etc. The wireless communication module can provide one or more of the following wireless communication solutions: wireless fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0030] Memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The RAM can be directly read and written by the processor 103, and can be used to store executable programs (e.g., machine instructions) of other running programs, as well as user and application data. The RAM may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0031] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110. Non-volatile memory can include disk storage devices and flash memory.

[0032] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions that, when executed by the processor 103, can implement a matting method that is executed on the electronic device 10.

[0033] In other embodiments, such as Figure 1 The electronic device 10 shown also includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10.

[0034] The processor 103 can include one or more processing units, for example: the processor 103 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0035] The processor 103 provides computing and control capabilities, for example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned matting method.

[0036] The input / output interface 104 is used to provide a channel for user input or output, for example, the input / output interface 104 can be used to connect various input / output devices, such as a mouse, a keyboard, a touch device, a display screen, etc., so that the user can enter information, or make the information visualized.

[0037] The bus 105 is used to provide a communication channel between the communication module 101, the memory 102, the processor 103, and the input / output interface 104 in the electronic device 10.

[0038] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 can include more or fewer components than the illustration, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0039] As shown in Figure 2 , it is a flowchart of the matting method provided by an embodiment of the present application. According to different requirements, the order of each step in the flowchart can be adjusted according to actual requirements, and some steps can be omitted. The execution subject of the method is an electronic device, for example Figure 1 , the electronic device 10 shown.

[0040] S11, obtaining a to-be-processed image.

[0041] The matting technology involved in some embodiments of the present application is a technology for separating (extracting) the foreground from the image. Among them, the foreground refers to the more conspicuous object in the image, and the background refers to the other area in the image except the foreground.

[0042] In some embodiments of the present application, the image to be processed contains foreground and background. For example, if the image to be processed includes objects such as human body, puppy, grass, road, white cloud, and sky, the foreground can be the human body or the puppy, and the background can be the grass, road, white cloud, and sky.

[0043] In some embodiments of the present application, the electronic device can obtain the image to be processed in various ways. For example, the electronic device can obtain the image uploaded by the user on an e-commerce platform or a live broadcast platform as the image to be processed. Alternatively, the electronic device can obtain a video frame from a video as the image to be processed.

[0044] In the present embodiment, the above-mentioned process of obtaining the image to be processed is only for reference, and in actual application, it is not limited to the above-mentioned examples.

[0045] S12, obtain text information input by a user for the foreground of the image to be processed, and call a text encoder to encode the text information to obtain a text embedding vector.

[0046] In some embodiments of the present application, the text information can be the matting requirement information of the user for the foreground of the image to be processed, and the text information is used to specify the foreground in the image to be processed. For example, continuing the above-mentioned embodiment, if the image to be processed includes objects such as human body, puppy, grass, road, white cloud, and sky, when the user wants to specify the human body or the puppy as the foreground, the text information for the human body can be “separate out the human”, or the text information for the puppy can be “puppy” and the like. The above-mentioned examples are only examples, and the actual text information is not limited thereto.

[0047] In some embodiments of the present application, the electronic device can provide a visual interface, remind the user to input the text information through prompt information on the visual interface, and receive the information input by the user as the text information. The prompt information can be set by itself, and the present application does not limit it. For example, the prompt information can be “please input the matting object” and the like.

[0048] In other embodiments of the present application, the electronic device can also obtain the text information in other ways, and the present application does not limit the way of obtaining the text information.

[0049] In some embodiments of the present application, the text encoder includes, but is not limited to, a large vision transformer (Vision Transformer-Large, ViT-L / 14) and a large vision transformer (Vision Transformer-Large, ViT-L / 32), and the like.

[0050] In some embodiments of the present application, if the text encoder is ViT-L / 14, the text encoder comprises an embedding layer, a first self-attention encoder, and a multi-layer perceptron head. The embedding layer is used to convert the input data into a vector, the first self-attention encoder is used to add position encoding in the vector converted by the embedding layer, and the multi-layer perceptron head is used to perform classification processing (pooling processing) on the output of the first self-attention encoder. The multi-layer perceptron head comprises a layer normalization layer and two fully connected layers, and the Gaussian Error Linear Unit (GELU) is used as the activation function in the fully connected layer. The first self-attention encoder can be a multi-layer transformer encoder, and each layer of the transformer encoder comprises an attention mechanism and a feedforward neural network. The attention mechanism is a mechanism for calculating the correlation between different positions in the text encoder, which is used to capture global context features, and the feedforward neural network is a structure with linear transformation and nonlinear activation function. The nonlinear activation function in the feedforward neural network can be a ReLU function.

[0051] In the present embodiment, the text information is the user's matting demand information, and the text information is encoded by the text encoder, which can convert the matting demand information into a text embedding vector, providing a basis for indicating the subject of the matting in the to-be-processed image. In addition, ViT-L / 14 and ViT-L / 32 have strong semantic understanding ability and can quickly encode the text information to generate a text embedding vector with high quality.

[0052] S13, calling the image encoder to encode the to-be-processed image to obtain an image embedding vector of the to-be-processed image.

[0053] In some embodiments of the present application, the image encoder can be a spatial transformer autoencoder with self-attention (ViTAE-S).

[0054] In some embodiments of the present application, the image encoder comprises a second self-attention (Transformer) encoder, which can be a multi-layer transformer encoder, and each layer of the transformer encoder comprises an attention mechanism and a feedforward neural network.

[0055] In this embodiment, the image encoder uses the self-attention mechanism to capture global and local information (image information of different scales) in the image to be processed. Since the self-attention mechanism can better handle long-distance dependencies in the image, the encoding accuracy and robustness are improved, and the image embedding vector can better reflect the content in the image to be processed.

[0056] S14, input the text embedding vector and the image embedding vector into the multi-modal cross self-attention model to obtain a target vector.

[0057] In some embodiments of the present application, the multi-modal cross self-attention model can be a multi-layer modal mixed transformer encoder. For example, the multi-modal cross self-attention model can be a 6-layer modal mixed transformer encoder. The function of the 6-layer modal mixed transformer encoder is mainly to encode the input data and convert it into another representation form so that the subsequent model can better understand and process the data. In the multi-modal transformer, each layer is composed of a self-attention layer and a feedforward neural network layer. The self-attention layer allows the model to consider the information of the entire sequence when processing words, rather than just considering the previous word. The feedforward neural network layer provides the nonlinear ability of the model, so that the model can better handle long sequences and can more effectively establish connections between different modalities.

[0058] In the 6-layer modal mixed architecture, the self-attention layer of each layer can process data from different modalities and allow data from different modalities to interact with each other. This architecture allows the model to better integrate and understand information from different modalities, such as text and images.

[0059] In some embodiments of the present application, the electronic device can use the multi-modal cross self-attention model to fuse the text embedding vector and the image embedding vector to obtain a target vector.

[0060] In this embodiment, the target vector is a high-dimensional vector. The text embedding vector can reflect the user's matting requirements. Inputting the text embedding vector into the multi-modal cross self-attention model can make the multi-modal cross self-attention model clear about the subject of matting, thereby ensuring the accuracy and flexibility of matting. Since the subject of matting is clear, the problem that the multi-modal cross self-attention model cannot accurately perform matting due to matting confusion (semantic ambiguity) when the image to be processed has multiple foregrounds can be avoided. In addition, if the multi-modal cross self-attention model is trained by the text embedding vector and the image embedding vector, the problem of training failure caused by the multi-modal cross self-attention model not converging due to matting confusion (semantic ambiguity) can also be avoided.

[0061] S15, decoding the target vector using the decoder to obtain a mask image of the to-be-processed image.

[0062] In some embodiments of the present application, the network structure of the decoder can be symmetrical to the network structure of the image encoder, and the decoding process of the target vector using the decoder is the inverse process of the generation of the image embedding vector.

[0063] In some embodiments of the present application, the mask image is a binary image, if the pixel value of the corresponding pixel point of the foreground in the mask image is 1, the color displayed by the foreground is white, and if the pixel value of the corresponding pixel point of the background in the mask image is 0, the color displayed by the background is black. For example, if the text information specifies the foreground as a human body, as shown in FIG. 1, it is a schematic diagram of a mask image provided by an embodiment of the present application. In FIG. 1, the human body is white, and the background is black. Figure 3 Figure 3 In some embodiments of the present application, the mask image is a binary image, if the pixel value of the corresponding pixel point of the foreground in the mask image is 1, the color displayed by the foreground is white, and if the pixel value of the corresponding pixel point of the background in the mask image is 0, the color displayed by the background is black. For example, if the text information specifies the foreground as a human body, as shown in FIG. 1, it is a schematic diagram of a mask image provided by an embodiment of the present application. In FIG. 1, the human body is white, and the background is black.

[0064] In other embodiments of the present application, after obtaining the mask image (mask), the electronic device can generate an image editing material according to the pixel values of the corresponding pixel points between the mask image and the to-be-processed image, and generate a video special effect and a commodity advertisement based on the image editing material.

[0065] In some embodiments of the present application, the image editing material can be a foreground image, which can be used as an image editing material in the e-commerce industry and the live broadcast industry. For example, the foreground image can be used to make a video special effect in a live broadcast room or the foreground image can be directly used as a background in the live broadcast room to increase the visual effect of the live broadcast room, thereby improving the attraction of the live broadcast room. Alternatively, the foreground image can also be used to make a promotional advertisement of a commodity to show the packaging design and characteristics of the commodity, thereby improving the attraction of the commodity. The above examples of the foreground image are only examples, and the present application does not limit the use of the foreground image.

[0066] Specifically, the generation of the foreground image includes: the electronic device performs a multiplication operation on the pixel values of the corresponding pixel points between the mask image and the to-be-processed image, to obtain a target pixel value of each pixel point in the mask image, and replaces the pixel value of each pixel point in the mask image with the corresponding target pixel value, to obtain the foreground image.

[0067] ​In other embodiments of the present application, the electronic device can also perform addition, division, subtraction, etc. operations on the pixel values of the corresponding pixel points between the mask image and the to-be-processed image, so as to extract the foreground in the to-be-processed image. The present application does not limit the operation mode between the pixel values of the corresponding pixel points between the mask image and the to-be-processed image. In some embodiments of the present application, the generation of video special effects and commodity advertisements can refer to related technologies, and the present application does not limit this. For example, the electronic device generates a video special effect based on the foreground image, including: transforming the foreground image to obtain a transformed foreground image, and adding a visual effect to the foreground image to obtain a video special effect. The transformation of the foreground image includes color and shape transformation, and the visual effect includes, but is not limited to: blur, gradient, magnification, and rotation, etc.

[0068] Through the above implementation, the text information corresponding to the foreground input is obtained, wherein the text information can reflect the matting demand of the user, and the text embedding vector obtained by encoding the text information is input into the multi-modal cross self-attention model, which can make the multi-modal cross self-attention model clear the subject of matting, thereby ensuring the accuracy and flexibility of matting. The image encoder can learn the features of different scales of the to-be-processed image, so that the image embedding vector contains more local information and global information of the to-be-processed image. Since the multi-modal cross self-attention model can process text image information at the same time, and the attention mechanism in the multi-modal cross self-attention model has strong learning ability, the user's matting demand and the detail information and global information of the to-be-processed image can be accurately fused through the multi-modal cross self-attention model. Since the target vector accurately fuses the user's matting demand and the detail information and global information of the to-be-processed image, the matting accuracy and matting effect can be improved by decoding the target vector.

[0069] In some embodiments of the present application, if the text encoder is ViT-L / 14, the text editor includes an embedding layer and a first self-attention encoder. As shown in Figure 4 The flowchart of the method for generating a text embedding vector provided by an embodiment of the present application is shown in

[0070] S121, pre-processing the text information to obtain a text sequence.

[0071] In some embodiments of the present application, the preprocessing of the text information includes, but is not limited to, word segmentation, filtering, and encoding, etc. The electronic device can perform word segmentation on the text information to obtain one or more initial words, further, the electronic device filters the stop words in the one or more initial words to obtain one or more target words, and further, the electronic device encodes each target word to obtain an encoding vector of each target word, and combines the encoding vector of each target word according to the order of each target word in the text information to obtain a text sequence.

[0072] In some embodiments of the present application, the word segmentation and the filtering of the stop words are optional processes. The electronic device can use a word embedding technology (such as a Word2Vec algorithm, a GloVe algorithm, or a BERT algorithm, etc.) to map each target word to a high-dimensional encoding vector.

[0073] In some embodiments of the present application, the preprocessing of the text information can filter out the stop words without actual meaning, thereby improving the encoding speed of the target words.

[0074] In some embodiments of the present application, the electronic device adds the first position encoding vector to the text sequence to obtain a text position sequence.

[0075] In some embodiments of the present application, the process of calculating the first position encoding vector can refer to the process of calculating the position encoding according to the position of each target word corresponding to each word vector in the text information and the dimension index of each word vector in the related art.

[0076] In some embodiments of the present application, the electronic device adds the first position encoding vector to the text sequence to obtain a text position sequence includes: the electronic device generates an operation vector corresponding to each word vector according to each word vector and the first position encoding vector corresponding to each word vector, and replaces each word vector with the operation vector corresponding to each word vector in the text sequence to obtain the text position sequence.

[0077] In some embodiments of the present application, the electronic device can perform an operation on each word vector and the corresponding first position encoding vector to obtain an operation vector corresponding to each word vector. Since the dimensions of each word vector and the corresponding first position encoding vector can be the same, the electronic device can perform multiplication, addition, subtraction, and division operations on each word vector and the corresponding first position encoding vector to obtain an operation vector corresponding to each word vector, and each operation vector is used to replace the word vector corresponding to the operation vector in the text sequence. The present application does not limit the operation method.

[0078] In this embodiment, since the first position encoding vector includes the position information of the target word corresponding to each word vector in the text information, by adding the first position encoding vector to the text sequence, the text position sequence can include the position information of the target word corresponding to each word vector in the text information.

[0079] S123, calling an embedding layer to convert the text position sequence to obtain a text conversion vector.

[0080] In some embodiments of the present application, the electronic device can call an embedding layer to map the text position sequence to a high-dimensional space to obtain a text conversion vector.

[0081] In this embodiment, since the text position sequence includes the position information of the target word corresponding to each word vector in the text information, when the embedding layer is used to convert the text position sequence, the embedding layer can better understand the semantic information in the text position sequence.

[0082] S124, inputting the text conversion vector into a first self-attention encoder to obtain a text encoding vector. In some embodiments of the present application, the first self-attention encoder can include multiple layers of Transformer encoders, and each layer of Transformer encoder includes a self-attention mechanism and a feedforward neural network. Wherein, the process of the first self-attention encoder processing the text conversion vector to obtain the text encoding vector can refer to related technologies.

[0083] In this embodiment, since the text conversion vector is generated by the text position sequence, the position information of the target word corresponding to each word vector in the text information in the text conversion vector, when the first self-attention encoder is used to process the text conversion vector, the first self-attention encoder can better calculate the correlation of different positions.

[0084] S125, performing a pooling operation on the text encoding vector to obtain a text embedding vector.

[0085] In some embodiments of the present application, the electronic device can perform an average pooling operation or a maximum pooling operation on the text encoding vector to obtain a text embedding vector. The process of the average pooling operation or the maximum pooling operation can refer to related technologies.

[0086] In this embodiment, the text encoder is called to encode the text information to obtain the text embedding vector, in order to make the multi-modal cross self-attention model clear the subject of the cutout in the to-be-processed image, so that the foreground can be accurately extracted from the to-be-processed image.

[0087] In some embodiments of the present application, the image encoder includes a second self-attention encoder, such as Figure 5Fig. 1 is a flowchart of a method for generating an image embedding vector according to an embodiment of the present application. The method comprises the following steps:

[0088] S131, pre-processing the to-be-processed image to obtain a plurality of image blocks.

[0089] In some embodiments of the present application, the pre-processing of the to-be-processed image includes, but is not limited to, size adjustment, resolution adjustment, segmentation, etc. The electronic device can adjust the size and resolution of the to-be-processed image, and segment the adjusted to-be-processed image to obtain a plurality of image blocks. The plurality of image blocks do not overlap with each other.

[0090] In the present embodiment, the size adjustment of the to-be-processed image includes increasing or decreasing the size, and the resolution adjustment of the to-be-processed image includes increasing or decreasing the resolution. The adjustment of the size and resolution of the to-be-processed image is an optional process.

[0091] S132, calculating a second position encoding vector corresponding to each pixel value in each image block.

[0092] In some embodiments of the present application, the electronic device can calculate the first position encoding corresponding to each pixel value according to the position of each pixel value in the corresponding image block and the dimension index of each pixel value.

[0093] S133, generating an image vector of each image block according to the pixel values in each image block, and adding the second position encoding vector corresponding to each pixel value in each image vector to obtain an image position vector corresponding to each image vector.

[0094] In some embodiments of the present application, the electronic device can arrange the pixel values of each row of pixel points in each image block to obtain an image vector of each image block. The electronic device can multiply each pixel value in each image block with the corresponding second position encoding vector to obtain a multiplication vector corresponding to each pixel value. Further, the electronic device can replace each pixel value in each image vector with the multiplication vector corresponding to the pixel value to obtain an image position vector corresponding to each image vector.

[0095] In the present embodiment, the image position vector includes the position information of each pixel value in the corresponding image block.

[0096] S134, inputting each image position vector into a second self-attention encoder to obtain an image encoding vector corresponding to each image position vector.

[0097] In some embodiments of the present application, the second self-attention encoder can include a multi-layer Transformer encoder, each layer of the Transformer encoder including a self-attention mechanism and a feedforward neural network. The second self-attention encoder processes the image position vector to obtain the image encoding vector. The process of processing the text conversion vector to obtain the text encoding vector using the first self-attention encoder is described above.

[0098] In this embodiment, since the image position vector is generated by the second position encoding vector, the second position encoding vector includes the position information of each pixel value in the corresponding image block, and when the image position vector is processed using the second self-attention encoder, the second self-attention encoder can better obtain the image information of the image to be processed, thereby better calculating the correlation of different positions.

[0099] S135, performing a pooling operation on the plurality of image encoding vectors to obtain an image embedding vector.

[0100] In some embodiments of the present application, the electronic device can perform an average pooling operation or a maximum pooling operation on the image encoding vector to obtain the image embedding vector. The process of the average pooling operation or the maximum pooling operation can refer to related technologies.

[0101] In this embodiment, since the ViTAE-S is sensitive to the detailed information of the image, it can capture the global information of the image and better understand the context information and spatial information in the image to be processed, thereby improving the efficiency of identifying / classifying each pixel point in the image to be processed. In addition, the ViTAE-S combines the advantages of the Convolutional Neural Network (CNN) and the Transformer, which can ensure the feature extraction capability while having higher computational performance, thereby improving the generation efficiency of the image embedding vector. In some embodiments of the present application, the multi-modal cross self-attention model includes a self-attention mechanism and a feedforward neural network. As shown in Figure 6 The method for generating the target context feature vector includes the following steps:

[0102] S141, performing cross-attention calculation on the text embedding vector and the image embedding vector through the self-attention mechanism to obtain a plurality of initial context feature vectors.

[0103] In some embodiments of the present application, the electronic device performs cross-attention calculation on the text embedding vector and the image embedding vector through a self-attention mechanism to obtain a plurality of attention weights between the text embedding vector and the image embedding vector, and performs weighted summation operation on the text embedding vector and the image embedding vector through each attention weight to obtain a plurality of initial context feature vectors.

[0104] In S142, a feedforward neural network is used to map each initial context feature vector to obtain a target context feature vector corresponding to each initial context feature vector.

[0105] In some embodiments of the present application, the feedforward neural network includes a plurality of hidden layers and a plurality of activation functions, and each initial context feature vector is linearly transformed through the plurality of hidden layers and nonlinearly transformed through the plurality of activation functions after the linear transformation to obtain the target context feature vector.

[0106] In S143, the plurality of target context feature vectors are fused to obtain a target vector.

[0107] In some embodiments of the present application, the electronic device can fuse the plurality of target context feature vectors into a target vector in various ways. For example, the electronic device can perform weighted average operation on the plurality of target context feature vectors to obtain the target vector, or the electronic device can concatenate the plurality of target context feature vectors to obtain the target vector, or the electronic device can perform weighted fusion on the plurality of target context feature vectors through a self-attention mechanism to obtain the target vector.

[0108] In this embodiment, since the text embedding vector can explicitly extract the subject, cross-attention calculation on the text embedding vector and the image embedding vector through the multi-modal cross-self-attention model can make the multi-modal cross-self-attention model understand the image embedding vector more deeply, so as to accurately extract features of the image embedding vector, and make the target vector more accurate. Figure 7 As shown in FIG. 11, a functional module diagram of a matting device provided in an embodiment of the present application is shown. The matting device 11 includes an acquisition unit 110, an encoding unit 111, an input unit 112, and a decoding unit 113. The module / unit referred to in the present application refers to a series of computer readable instructions that can be acquired by the processor 103 in the computer device 100 and can complete a fixed function, and is stored in the memory 102 in the computer device 100. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments. Figure 1 The module / unit can be implemented in the form of software or hardware in the computer device 100. When the module / unit is implemented in the form of software, it can be stored in the memory 102 and executed by the processor 103 in the computer device 100. The memory 102 in the computer device 100 can include a number of storage media, for example, magnetic tape, floppy disk, zip disk, magnetic disk, optical disk, CD-ROM, RAM, and ROM. The memory 102 in the computer device 100 can be a volatile memory or a non-volatile memory. The volatile memory can be a random access memory (RAM) or a dynamic random access memory (DRAM). The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Figure 1 In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0109] The acquisition unit 110 is configured to acquire a to-be-processed image.

[0110] In some embodiments of the present application, the acquisition unit 110 is further configured to acquire text information of a foreground input of the image to be processed.

[0111] The encoding unit 111 is configured to call a text encoder to encode the text information to obtain a text embedding vector.

[0112] In some embodiments of the present application, the text editor comprises an embedding layer and a first self-attention encoder, and the encoding unit 111 is further configured to preprocess the text information to obtain a text sequence, calculate a first position encoding vector corresponding to each word vector in the text sequence, add the first position encoding vector to the text sequence to obtain a text position sequence, call the embedding layer to convert the text position sequence to obtain a text conversion vector, input the text conversion vector into the first self-attention encoder to obtain a text encoding vector, and perform a pooling operation on the text encoding vector to obtain the text embedding vector.

[0113] In some embodiments of the present application, the encoding unit 111 further comprises an adding subunit, which is configured to generate an operation vector corresponding to each word vector according to each word vector and the first position encoding vector corresponding to each word vector, and replace each word vector in the text sequence with the operation vector corresponding to each word vector to obtain the text position sequence.

[0114] In some embodiments of the present application, the encoding unit 111 is further configured to call an image encoder to encode the image to be processed to obtain an image embedding vector of the image to be processed.

[0115] In some embodiments of the present application, the image encoder comprises a second self-attention encoder, and the encoding unit 111 is further configured to preprocess the image to be processed to obtain a plurality of image blocks, calculate a second position encoding vector corresponding to each pixel value in each image block, generate an image vector of each image block according to the pixel values in each image block, and add the second position encoding vector corresponding to each pixel value in each image vector to obtain an image position vector corresponding to each image vector, input each image position vector into the second self-attention encoder to obtain an image encoding vector corresponding to each image position vector, and perform a pooling operation on a plurality of image encoding vectors to obtain the image embedding vector.

[0116] In some embodiments of the present application, the encoding unit 111 further comprises a preprocessing subunit configured to resize the image to be processed and divide the resized image to be processed into a plurality of image blocks that do not overlap with each other.

[0117] The input unit 112 is configured to input the text embedding vector and the image embedding vector into a multi-modal cross self-attention model to obtain a target vector.

[0118] In some embodiments of the present application, the multi-modal cross self-attention model comprises a self-attention mechanism and a feedforward neural network, and the decoding unit 112 is further configured to perform cross-attention calculation on the text embedding vectors and the image embedding vectors by using the self-attention mechanism to obtain a plurality of initial context feature vectors, map each initial context feature vector by using the feedforward neural network to obtain a target context feature vector corresponding to each initial context feature vector, and fuse the plurality of target context feature vectors to obtain a target vector.

[0119] The decoding unit 113 is configured to decode the target vector by using a decoder to obtain a mask image of the image to be processed.

[0120] In some embodiments of the present application, the decoding unit 113 further comprises a generating subunit configured to generate an image editing material according to pixel values of corresponding pixel points between the mask image and the image to be processed, and generate a video special effect and a commodity advertisement based on the image editing material.

[0121] The embodiments of the present application further provide a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program comprises program instructions. When the program instructions are executed, a method is implemented. The method can refer to the method in each of the above embodiments of the present application.

[0122] The computer readable storage medium can be an internal storage of the electronic device, such as a hard disk or a memory of the electronic device. The computer readable storage medium can also be an external storage of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0123] In some embodiments, the computer readable storage medium can include a program storage area and a data storage area. The program storage area can store an operating system, an application required by at least one function, etc. The data storage area can store data created according to the use of the electronic device, etc.

[0124] In the above embodiments, the description of each embodiment has its own focus. The parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0125] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0126] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / equipment and method can be implemented in other ways. For example, the apparatus / equipment embodiments described above are merely schematic. The division of the modules or units is merely a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0127] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0128] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of matting, characterized by, The method comprises: acquiring a to-be-processed image; acquiring text information of a foreground input for the to-be-processed image, and calling a text encoder to encode the text information to obtain a text embedding vector; calling an image encoder to encode the to-be-processed image to obtain an image embedding vector of the to-be-processed image, comprising: preprocessing the to-be-processed image to obtain a plurality of image blocks, calculating a position encoding vector corresponding to each pixel value in each image block, generating an image vector of each image block according to the pixel values in the image block, and adding the position encoding vector corresponding to each pixel value in each image vector to the image vector to obtain an image position vector corresponding to each image vector, inputting each image position vector into a self-attention encoder in the image encoder to obtain an image encoding vector corresponding to each image position vector, and performing a pooling operation on a plurality of image encoding vectors to obtain the image embedding vector; inputting the text embedding vector and the image embedding vector into a multi-modal cross self-attention model to obtain a target vector, comprising: performing cross-attention calculation on the text embedding vector and the image embedding vector through a self-attention mechanism in the multi-modal cross self-attention model to obtain a plurality of initial context feature vectors, mapping each initial context feature vector using a feedforward neural network in the multi-modal cross self-attention model to obtain a target context feature vector corresponding to each initial context feature vector, and fusing a plurality of target context feature vectors to obtain the target vector; decoding the target vector using a decoder to obtain a mask image of the to-be-processed image.

2. The matting method of claim 1, wherein, The method further comprises: generating an image editing material according to the pixel values of the corresponding pixel points between the mask image and the to-be-processed image, and generating a video special effect and a commodity advertisement based on the image editing material.

3. The matting method of claim 1, wherein, The calling of the text encoder to encode the text information to obtain the text embedding vector comprises: preprocessing the text information to obtain a text sequence; calculating a position encoding vector corresponding to each word vector in the text sequence, and adding the position encoding vector corresponding to each word vector to the text sequence to obtain a text position sequence; calling an embedding layer in the text encoder to convert the text position sequence to obtain a text conversion vector; inputting the text conversion vector into a self-attention encoder in the text encoder to obtain a text encoding vector; performing a pooling operation on the text encoding vector to obtain the text embedding vector.

4. The matting method of claim 3, wherein, The adding of the position encoding vector corresponding to each word vector to the text sequence to obtain a text position sequence comprises: generating an operation vector corresponding to each word vector according to the word vector and the position encoding vector corresponding to the word vector; replacing each word vector with the operation vector corresponding to the word vector in the text sequence to obtain the text position sequence.

5. The matting method of claim 1, wherein, The preprocessing of the to-be-processed image to obtain a plurality of image blocks comprises: The to-be-processed image is resized, and the resized to-be-processed image is segmented into the plurality of image blocks that do not overlap with each other.

6. A matting device, running on an electronic device, for implementing the matting method according to any one of claims 1 to 5, characterized in that, The matting device comprises: An acquisition unit is configured to acquire a to-be-processed image. The acquisition unit is further configured to acquire text information of foreground input for the to-be-processed image. An encoding unit is configured to call a text encoder to encode the text information to obtain a text embedding vector. The encoding unit is further configured to call an image encoder to encode the to-be-processed image to obtain an image embedding vector of the to-be-processed image. An input unit is configured to input the text embedding vector and the image embedding vector into a multi-modal cross self-attention model to obtain a target vector. A decoding unit is configured to use a decoder to decode the target vector to obtain a mask image of the to-be-processed image.

7. An electronic device, comprising: The electronic device comprises: a memory configured to store at least one instruction; and a processor configured to execute the at least one instruction to implement the matting method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the matting method according to any one of claims 1 to 5.

Images