Image matting method and device based on large-scale pre-training ViT model

Abstract

The invention discloses an image matting method and device based on a large-scale pre-training ViT model. The method comprises the following steps: acquiring an image to be matted; an image to be matted is input into the segmentation model, a corresponding ternary graph is generated through the segmentation model, the segmentation model comprises a backbone network and a bypass network, the backbone network adopts a ViT architecture, the weight pre-trained by the large model is applied to the backbone network in a key-value mapping mode, and the bypass network is used for assisting the backbone network in learning local space information; the ternary graph and the image to be matted are input into a matting model, so that a corresponding grayscale image is generated through the matting model, and the matting model has the same architecture as the segmentation model; fusing the to-be-matted image with the grayscale image to obtain a final matting result; therefore, the pre-training weight based on the large model is introduced into the ViT architecture, and the bypass network is added to assist the backbone network, so that the semantic recognition capability and matting precision of the model are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a cutout method based on a large-scale pre-trained ViT model, a computer-readable storage medium, a computer device, and a cutout device based on a large-scale pre-trained ViT model. Background Art

[0002] Cutting out refers to the process of removing the background from an image in order to accurately extract foreground objects in a still image or video image sequence. There are often challenges in the process of cutting out still images or video images, especially when dealing with complex scenes such as holes, mirrors, and small parts in e-commerce images, where it is difficult to achieve accurate recognition and fine cutouts.

[0003] In the related technology, although the infrastructure of some existing cutout models has adopted the ViT model, since the amount of pre-training data of the ViT model is only about 13 million data at most (ImageNet-22k dataset, mainly used for classification tasks), it has certain limitations in image feature extraction and semantic recognition, resulting in unsatisfactory cutout effects.

[0004] In addition, the ViT architecture uses a global attention mechanism, which focuses on capturing the global information of the image. However, since it only has single-scale feature information, it has certain limitations in learning local geometric details. Therefore, in the cutout task, the ViT architecture is difficult to fully focus on the local edge details of the foreground object, resulting in poor performance in the refined edge cutout effect. Summary of the invention

[0005] The present application aims to solve at least one of the technical problems in the above-mentioned technology to a certain extent. To this end, one purpose of the present application is to propose a cutout method based on a large-scale pre-trained ViT model, which effectively improves the semantic recognition ability and cutout accuracy of the model by introducing pre-trained weights based on a large model into the ViT architecture and adding a bypass network to assist the backbone network.

[0006] A second object of the present application is to provide a computer-readable storage medium.

[0007] The third objective of the present application is to provide a computer device.

[0008] The fourth objective of this application is to propose a cutout device based on a large-scale pre-trained ViT model.

[0009] To achieve the above-mentioned objectives, the first aspect of the present application proposes a cutout method based on a large-scale pre-trained ViT model, comprising the following steps: obtaining an image to be cutout; inputting the image to be cutout into a segmentation model to generate a corresponding ternary map through the segmentation model, wherein the segmentation model includes a backbone network and a bypass network, the backbone network adopts a ViT architecture, and applies the weights pre-trained by a large model to the backbone network through a key-value mapping method, and the bypass network is used to assist the backbone network in learning local spatial information; inputting the ternary map and the image to be cutout into the cutout model to generate a corresponding grayscale image through the cutout model, wherein the cutout model has the same architecture as the segmentation model; fusing the image to be cutout with the grayscale image to obtain the final cutout result.

[0010] According to the cutout method based on the large-scale pre-trained ViT model in the embodiment of the present application, the semantic recognition ability and accuracy of the cutout model are significantly improved by adopting the ViT architecture and bypass network with large-scale pre-trained weights; especially when processing complex details (such as holes, mirrors, and small components) in e-commerce images.

[0011] In addition, the cutout method based on the large-scale pre-trained ViT model proposed in the above embodiment of the present application may also have the following additional technical features:

[0012] Optionally, the backbone network includes an embedding module and multiple encoding modules; wherein the image to be cut out is encoded into a first embedding vector by the embedding module; and after adding position information to the first embedding vector, it is input into the encoding module for processing.

[0013] Optionally, the large model is an interactive segmentation model, comprising a prompt encoder, an image encoder and a lightweight mask decoder; wherein the prompt encoder is used to convert the input interactive prompt into a second embedding vector; the image encoder adopts a ViT architecture to capture the global and local features of the image; the lightweight mask decoder combines the outputs of the prompt encoder and the image encoder to generate a segmentation mask, which is used as the pre-trained weight of the large model.

[0014] Optionally, the bypass network includes a spatial prior module, a geometric prior feature injection module and a multi-scale feature extraction module; wherein the geometric prior features of the image to be cutout are extracted by the spatial prior module; the geometric prior feature injection module uses the first embedding vector after adding the position information as a query, and uses the geometric prior features as key and value, and performs a Cross Attention operation to obtain a new feature representation, so that the new feature representation can be used as the input of the next encoding module in the backbone network; the multi-scale feature extraction module uses the geometric prior features as a query, and uses the output of the encoding module as key and value, and performs a Cross Attention operation and then passes through the FFN layer to obtain multi-scale features.

[0015] To achieve the above-mentioned objectives, the second aspect embodiment of the present application proposes a computer-readable storage medium, on which a cutout program based on a large-scale pre-trained ViT model is stored. When the cutout program based on the large-scale pre-trained ViT model is executed by a processor, the cutout method based on the large-scale pre-trained ViT model as described above is implemented.

[0016] To achieve the above-mentioned objectives, the third aspect embodiment of the present application proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the program, it implements the above-mentioned cutout method based on the large-scale pre-trained ViT model.

[0017] To achieve the above-mentioned objectives, the fourth aspect of the present application proposes a cutout device based on a large-scale pre-trained ViT model, including: an acquisition module, used to acquire the image to be cutout; a segmentation module, used to input the image to be cutout into the segmentation model, so as to generate a corresponding ternary map through the segmentation model, wherein the segmentation model includes a backbone network and a bypass network, the backbone network adopts the ViT architecture, and applies the weights pre-trained by the large model to the backbone network through a key-value mapping method, and the bypass network is used to assist the backbone network in learning local spatial information; a cutout module, used to input the ternary map and the image to be cutout into the cutout model, so as to generate a corresponding grayscale image through the cutout model, wherein the cutout model has the same architecture as the segmentation model; a target generation module, used to fuse the image to be cutout with the grayscale image to obtain the final cutout result.

[0018] According to the cutout device based on the large-scale pre-trained ViT model in the embodiment of the present application, the semantic recognition ability and accuracy of the cutout model are significantly improved by adopting the ViT architecture and bypass network with large-scale pre-trained weights; especially when processing complex details (such as holes, mirrors, and small components) in e-commerce images.

[0019] In addition, the image cutout device based on the large-scale pre-trained ViT model proposed in the above embodiments of the present application may also have the following additional technical features:

[0020] Optionally, the backbone network includes an embedding module and multiple encoding modules; wherein the image to be cut out is encoded into a first embedding vector by the embedding module; and after adding position information to the first embedding vector, it is input into the encoding module for processing.

[0021] Optionally, the large model is an interactive segmentation model, comprising a prompt encoder, an image encoder and a lightweight mask decoder; wherein the prompt encoder is used to convert the input interactive prompt into a second embedding vector; the image encoder adopts a ViT architecture to capture the global and local features of the image; the lightweight mask decoder combines the outputs of the prompt encoder and the image encoder to generate a segmentation mask, which is used as the pre-trained weight of the large model.

[0022] Optionally, the bypass network includes a spatial prior module, a geometric prior feature injection module and a multi-scale feature extraction module; wherein the geometric prior features of the image to be cutout are extracted by the spatial prior module; the geometric prior feature injection module uses the first embedding vector after adding the position information as a query, and uses the geometric prior features as key and value, and performs a Cross Attention operation to obtain a new feature representation, so that the new feature representation can be used as the input of the next encoding module in the backbone network; the multi-scale feature extraction module uses the geometric prior features as a query, and uses the output of the encoding module as key and value, and performs a Cross Attention operation and then passes through the FFN layer to obtain multi-scale features. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of a process of a cutout method based on a large-scale pre-trained ViT model according to an embodiment of the present application;

[0024] Figure 2 A schematic diagram of the network structure of a Segmentation segmentation model according to an embodiment of the present application;

[0025] Figure 3A schematic diagram of the network structure of the SegmentAnything model according to an embodiment of the present application;

[0026] Figure 4 A schematic diagram of the network structure of a Matting cutout model according to an embodiment of the present application;

[0027] Figure 5 A schematic diagram of data input and output of an Injector module in a bypass network according to an embodiment of the present application;

[0028] Figure 6 A schematic diagram of data input and output of an Extractor module in a bypass network according to an embodiment of the present application;

[0029] Figure 7 It is a schematic diagram of a process of generating an Alpha Matte image according to an embodiment of the present application;

[0030] Figure 8 A schematic diagram of a process for generating a final cutout result according to an embodiment of the present application;

[0031] Fig. 9 It is a block diagram of a cutout device based on a large-scale pre-trained ViT model according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0033] In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0034] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0035] Figure 1 FIG. 1 is a flow chart of a method for cutting out a picture based on a large-scale pre-trained ViT model according to an embodiment of the present application. Figure 1 As shown, the cutout method based on the large-scale pre-trained ViT model includes the following steps:

[0036] S101, obtaining an image to be cut out.

[0037] It should be noted that the image to be cut out is a picture input by a user.

[0038] S102, input the image to be cut out into the segmentation model to generate the corresponding ternary map through the segmentation model, wherein the segmentation model includes a backbone network and a bypass network, the backbone network adopts the ViT architecture, and the weights pre-trained by the large model are applied to the backbone network through the key-value mapping method, and the bypass network is used to assist the backbone network in learning local spatial information.

[0039] It should be noted that if Figure 2 and Figure 7 As shown in the figure, the ternary image is called Trimap. The Trimap image is a mask with three categories (0: background, 1: unknown area, 2: foreground). Figure 7 As shown in the middle column, the black area is the background, the gray area is the unknown area (which can be understood as the transition area), and the white area is the foreground; the main function of Trimap is to provide prior information to the matting model (Matting) to assist in calculating the Alpha Matte.

[0040] In some embodiments, the backbone network includes an embedding module and multiple encoding modules; wherein the image to be cut out is encoded into a first embedding vector by the embedding module; and after adding position information to the first embedding vector, it is input into the encoding module for processing.

[0041] It should be noted that the key to semantic recognition lies in whether the Segmentation model can accurately generate the Trimap image, automatically identify the foreground object area in the image, and avoid misidentification or miscutting.

[0042] As a specific example, Figure 2 As shown in the figure, the Segmentation segmentation model includes two branch networks, in which the main network adopts the ViT architecture, which encodes the image to be cut out (input image) into Patch Embedding and adds Position Embedding, and then processes it through multiple Transformer Blocks.

[0043] Among them, Transformer Blocks are repeated modules that form the core of ViT, and each module includes the key components of the Transformer architecture. These modules are a collection of self-attention mechanisms (Self-Attention), feed-forward networks (FFN), layer normalization (Layer Normalization) and residual connections (ResidualConnections). They are stacked together through a series of Transformer Blocks, each of which processes the input sequence independently and passes the information to the next block. This stacking structure enables ViT to process image data effectively and achieve excellent performance in a variety of visual tasks; the output of each Transformer Block will be used as the input of the next Block. In this way, the model can gradually build and refine its understanding of the image.

[0044] In some embodiments, the large model is an interactive segmentation model, comprising a prompt encoder, an image encoder and a lightweight mask decoder; wherein the prompt encoder is used to convert the input interactive prompt into a second embedding vector; the image encoder adopts the ViT architecture to capture the global and local features of the image; the lightweight mask decoder combines the outputs of the prompt encoder and the image encoder to generate a segmentation mask, which is used as the pre-trained weight of the large model.

[0045] As a specific embodiment, in order to enhance the semantic recognition capability of ViT, Figure 3 The pre-trained parameters of the SegmentAnything large model are introduced into Figure 2 SegmentAnything is an interactive segmentation model that is good at learning a large amount of semantic information. It can realize interactive segmentation of the target area through multiple prompt input methods such as clicking, box selection, smearing, and language description. Figure 3 As shown in Figure 2, the network architecture of SegmentAnything consists of a prompt encoder and an image encoder. The final target area mask is generated by a lightweight mask decoder. Taking advantage of the billion-level semantic information of SegmentAnything, especially the rich semantic information in the image encoder, this part adopts the native ViT architecture, so its pre-trained weights can be applied to the target area mask through key-value mapping. Figure 2 It should be noted that SegmentAnything has obvious advantages in rich semantic recognition, but its ability to process edge details is relatively weak.

[0046] In some embodiments, the bypass network includes a spatial prior module, a geometric prior feature injection module and a multi-scale feature extraction module; wherein the geometric prior features of the image to be cutout are extracted through the spatial prior module; the geometric prior feature injection module uses the first embedding vector after adding the position information as a query, and uses the geometric prior features as key and value, and performs a Cross Attention operation to obtain a new feature representation, so that the new feature representation can be used as the input of the next encoding module in the backbone network; the multi-scale feature extraction module uses the geometric prior features as a query, and uses the output of the encoding module as key and value, performs a Cross Attention operation, and then passes through the FFN layer to obtain multi-scale features.

[0047] As a specific embodiment, the spatial prior module (Spatial Prior) refers to some network layers of ResNet-50, with the purpose of extracting multi-scale representation features of the image through convolution kernels. Table 1 shows the structure of the Spatial Adapter. The image size of the input network is 1024×1024. The network can be divided into 4 parts. Each part is downsampled at the beginning, and the stride of the downsampling is 2. Each convolution layer is followed by a ReLU layer and a Batch Normalization layer. Finally, a feature pyramid with three resolutions (256, 128, 64) is obtained, and these feature maps are flattened and spliced ​​to obtain the final geometric prior features.

[0048]

[0049] like Figure 5 As shown in the figure, the geometric prior feature injection module (Injector) converts the original input features of ViT into As a query, the geometric prior features As Key and Value, a new It is used as the input of the next transformer block in the ViT backbone network. The formula is as follows, where γ i The factor is a learnable parameter that is initialized to 0 to ensure The feature distribution of will not be suddenly modified drastically due to the injection of geometric priors, thereby better utilizing the pre-trained weights of ViT.

[0050]

[0051] like Figure 6As shown in Figure 1, the purpose of the multi-scale feature extraction module (Extractor) is to extract multi-scale features from the single-scale output of the ViT block. It consists of a Cross-Attention layer and a FFN layer. In contrast to the Injector, the Extractor uses geometric prior features As Query, ViT block outputs As the key and value for Cross-Attention, and then through the FFN layer to obtain the final multi-scale features. The formula is as follows:

[0052]

[0053] S103, inputting the ternary image and the image to be cutout into a cutout model to generate a corresponding grayscale image through the cutout model, wherein the cutout model has the same architecture as the segmentation model.

[0054] It should be noted that if Figure 4 As shown, in order to solve the problem of processing local edge details, this application introduces a Spatial Adapter (SA) bypass network for both the Segmentation model and the Matting model, aiming to help the Transformer better learn local spatial information with the help of the convolutional network.

[0055] In addition, Alpha Matte is a grayscale image that represents the transparency of each pixel in an RGB image. In an RGBA transparent image, RGB represents the three color channels of red, green, and blue, and A represents the Alpha channel, i.e. Alpha Matte. Each pixel value of Alpha Matte defines the transparency of the pixel at the corresponding position, ranging from [0,1], where 0 represents completely transparent and 1 represents completely opaque. Figure 7 As shown in the third column, the alpha matte image (A channel of RGBA transparent image) has a pixel transparency of 1 in the foreground area, 0 in the background, and a grayscale value between 0 and 1 at the foreground edge, thereby achieving edge refinement.

[0056] Therefore, if Figure 7 As shown, by inputting the image to be matted, the Trimap image is obtained through the Segmentation model (segmentation model), and then the image to be matted and the Trimap image are sent to the Matting model (matting model) to obtain the final Alpha Matte image.

[0057] S104, fusing the image to be cut out with the grayscale image to obtain a final cutout result.

[0058] Specifically, Figure 8 As shown in the figure, the alpha matte output by the cutout model needs to be fused with the image to be cutout and the alpha matte image through the following formula to obtain the final cutout result. Where α is the transparency value of each pixel position in the alpha matte image, F is the foreground object area, B is the background area, and I is the final cutout result. Where α=0 means that the pixel is completely transparent, α=1 is completely opaque, and α∈(0,1) is partially transparent.

[0059] I=αF+(1-α)B

[0060] It should be noted that during the training process, the basic network structure and pre-training parameters of the Segmentation model and the Matting model are basically the same. The main difference lies in the final segmentation head layer and the matting head layer. The output of the segmentation head layer is the segmentation mask of the trimap, which uses the Cross-Entropy Loss loss function; while the output of the matting head layer is the alpha matte of the final result, which uses the MSE Loss loss function. Both models use the Adam-based gradient descent method to solve the weight parameter w. In each iteration, the prediction result error is calculated and back-propagated to the model, and the gradient is calculated and the parameters of the network model are updated.

[0061] To summarize, according to the cutout method based on the large-scale pre-trained ViT model of the embodiment of the present application, first, the image to be cutout is obtained; then, the image to be cutout is input into the segmentation model to generate a corresponding ternary map through the segmentation model, wherein the segmentation model includes a backbone network and a bypass network, the backbone network adopts the ViT architecture, and the weights pre-trained by the large model are applied to the backbone network through the key-value mapping method, and the bypass network is used to assist the backbone network in learning local spatial information; then, the ternary map and the image to be cutout are input into the cutout model to generate a corresponding grayscale image through the cutout model, wherein the cutout model has the same architecture as the segmentation model; finally, the image to be cutout is fused with the grayscale image to obtain the final cutout result; thus, the present application adopts the ViT architecture with large-scale pre-trained weights and the SpatialAdapter Module module, which significantly improves the semantic recognition ability and accuracy of the cutout model. The qualified rate of cutouts on e-commerce images has been improved by 15% compared with traditional methods, especially when dealing with complex details in e-commerce images (such as holes, mirrors, and small parts). In addition, users do not need professional image processing skills, they only need to upload pictures, and the system can automatically complete high-precision cutout operations, which significantly lowers the usage threshold while ensuring high-quality cutout results.

[0062] In order to implement the above-mentioned embodiments, the embodiments of the present application also propose a computer-readable storage medium, on which a cutout program based on a large-scale pre-trained ViT model is stored. When the cutout program based on the large-scale pre-trained ViT model is executed by a processor, the cutout method based on the large-scale pre-trained ViT model as described above is implemented.

[0063] According to the computer-readable storage medium of the embodiment of the present application, by storing a cutout program based on a large-scale pre-trained ViT model, the processor implements the cutout method based on a large-scale pre-trained ViT model as described above when executing the cutout program based on the large-scale pre-trained ViT model. Thus, by introducing pre-trained weights based on a large model into the ViT architecture and adding a bypass network to assist the backbone network, the semantic recognition ability and cutout accuracy of the model are effectively improved.

[0064] In order to implement the above-mentioned embodiments, the embodiments of the present application also propose a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the program, the above-mentioned cutout method based on the large-scale pre-trained ViT model is implemented.

[0065] According to the computer device of the embodiment of the present application, the cutout program based on the large-scale pre-trained ViT model is stored in the memory, so that when the processor executes the cutout program based on the large-scale pre-trained ViT model, the cutout method based on the large-scale pre-trained ViT model as described above is implemented. Thus, by introducing the pre-trained weights based on the large model into the ViT architecture and adding a bypass network to assist the backbone network, the semantic recognition ability and cutout accuracy of the model are effectively improved.

[0066] In order to implement the above embodiment, the present application also proposes a cutout device based on a large-scale pre-trained ViT model, such as Fig. 9 As shown, the cutout device based on the large-scale pre-trained ViT model includes: an acquisition module 10, a segmentation module 20, a cutout module 30 and a target generation module 40.

[0067] Among them, the acquisition module 10 is used to acquire the image to be cut out; the segmentation module 20 is used to input the image to be cut out into the segmentation model to generate a corresponding ternary image through the segmentation model, wherein the segmentation model includes a backbone network and a bypass network, the backbone network adopts the ViT architecture, and applies the weights pre-trained by the large model to the backbone network through the key-value mapping method, and the bypass network is used to assist the backbone network in learning local spatial information; the cutout module 30 is used to input the ternary image and the image to be cut out into the cutout model to generate a corresponding grayscale image through the cutout model, wherein the cutout model has the same architecture as the segmentation model; the target generation module 40 is used to fuse the image to be cut out with the grayscale image to obtain the final cutout result.

[0068] It should be noted that the above Figure 1 The description of the cutout method based on the large-scale pre-trained ViT model in is also applicable to the cutout device based on the large-scale pre-trained ViT model, and will not be repeated here.

[0069] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0070] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0071] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0072] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0073] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present application may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0074] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0075] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0076] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of the present application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0077] In this application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0078] In the present application, unless otherwise clearly specified and limited, a first feature being “above” or “below” a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being “above”, “above”, and “above” a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being “below”, “below”, and “below” a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0079] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms should not be understood as necessarily being directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0080] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A cutout method based on a large-scale pre-trained ViT model, characterized in that: The following steps are involved: Get the image to be cut out; The image to be cut out is input into a segmentation model to generate a corresponding ternary map through the segmentation model, wherein the segmentation model includes a backbone network and a bypass network, the backbone network adopts a ViT architecture, and the weights pre-trained by the large model are applied to the backbone network through a key-value mapping method, and the bypass network is used to assist the backbone network in learning local spatial information; Inputting the ternary map and the image to be cutout into a cutout model to generate a corresponding grayscale image through the cutout model, wherein the cutout model has the same architecture as the segmentation model; The image to be cut out is merged with the grayscale image to obtain a final cutout result.

2. The image cutout method based on the large-scale pre-trained ViT model as claimed in claim 1, characterized in that: The backbone network includes an embedding module and a plurality of encoding modules; The image to be cut out is encoded into a first embedding vector by the embedding module; and the first embedding vector is input into the encoding module for processing after adding position information.

3. The image cutout method based on the large-scale pre-trained ViT model as claimed in claim 2, characterized in that: The large model is an interactive segmentation model, including a hint encoder, an image encoder, and a lightweight mask decoder; The prompt encoder is used to convert the input interactive prompt into a second embedding vector; The image encoder adopts the ViT architecture to capture the global and local features of the image; The lightweight mask decoder combines the outputs of the hint encoder and the image encoder to generate a segmentation mask, which serves as the pre-trained weights of the large model.

4. The image cutout method based on the large-scale pre-trained ViT model as claimed in claim 3, characterized in that: The bypass network includes a spatial priori module, a geometric priori feature injection module and a multi-scale feature extraction module; Wherein, the geometric prior features of the image to be cut out are extracted through the spatial prior module; The geometric prior feature injection module uses the first embedding vector after adding the position information as a query, and uses the geometric prior feature as a key and a value to perform a Cross Attention operation to obtain a new feature representation, so that the new feature representation can be used as an input of the next encoding module in the backbone network; The multi-scale feature extraction module uses the geometric prior features as queries, uses the output of the encoding module as keys and values, performs a Cross Attention operation, and then passes through the FFN layer to obtain multi-scale features.

5. A computer-readable storage medium, characterized in that: A cutout program based on a large-scale pre-trained ViT model is stored thereon, and when the cutout program based on a large-scale pre-trained ViT model is executed by a processor, a cutout method based on a large-scale pre-trained ViT model as described in any one of claims 1-4 is implemented.

6. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the cutout method based on the large-scale pre-trained ViT model as described in any one of claims 1 to 4 is implemented.

7. A cutout device based on a large-scale pre-trained ViT model, characterized in that: include: An acquisition module is used to acquire the image to be cut out; A segmentation module, used for inputting the image to be cut out into a segmentation model to generate a corresponding ternary map through the segmentation model, wherein the segmentation model includes a backbone network and a bypass network, the backbone network adopts a ViT architecture, and the weights pre-trained by the large model are applied to the backbone network through a key-value mapping method, and the bypass network is used to assist the backbone network in learning local spatial information; A cutout module, used for inputting the ternary map and the image to be cutout into a cutout model, so as to generate a corresponding grayscale image through the cutout model, wherein the cutout model has the same architecture as the segmentation model; The target generation module is used to fuse the image to be cutout with the grayscale image to obtain a final cutout result.

8. The image cutout device based on the large-scale pre-trained ViT model according to claim 7, characterized in that: The backbone network includes an embedding module and a plurality of encoding modules; The image to be cut out is encoded into a first embedding vector by the embedding module; and the first embedding vector is input into the encoding module for processing after adding position information.

9. The image cutout device based on the large-scale pre-trained ViT model according to claim 8, characterized in that: The large model is an interactive segmentation model, including a hint encoder, an image encoder, and a lightweight mask decoder; The prompt encoder is used to convert the input interactive prompt into a second embedding vector; The image encoder adopts the ViT architecture to capture the global and local features of the image; The lightweight mask decoder combines the outputs of the hint encoder and the image encoder to generate a segmentation mask, which serves as the pre-trained weights of the large model.

10. The image cutout device based on the large-scale pre-trained ViT model according to claim 9, characterized in that: The bypass network includes a spatial priori module, a geometric priori feature injection module and a multi-scale feature extraction module; Wherein, the geometric prior features of the image to be cut out are extracted through the spatial prior module; The geometric prior feature injection module uses the first embedding vector after adding the position information as a query, and uses the geometric prior feature as a key and a value to perform a Cross Attention operation to obtain a new feature representation, so that the new feature representation can be used as an input of the next encoding module in the backbone network; The multi-scale feature extraction module uses the geometric prior features as queries, uses the output of the encoding module as keys and values, performs a Cross Attention operation, and then passes through the FFN layer to obtain multi-scale features.

Images