A reference image segmentation method and device based on a base model and a storage medium

Abstract

The application discloses a kind of based on basic model's referential image segmentation method, device and storage medium, belong to artificial intelligence technical field.The method includes: obtaining image, text as input data, and the target mask corresponding to text as label data;Referential image segmentation neural network is constructed;Image feature and text feature are respectively extracted by image encoder and text encoder;Image feature is fused with global text feature, and first multimodal feature is obtained;First multimodal feature is fused with sequence text feature, and second multimodal feature is obtained;According to second multimodal feature, sparse point sampling method is used to individually supervise the boundary point of target, and output mask is obtained;The loss of output mask and target mask is calculated, and referential image segmentation neural network is trained according to the loss.The application can solve the problem that high-precision segmentation mask cannot be obtained in referential image segmentation task, and improve referential image segmentation performance.

Description

Technical Field

[0001] This invention relates to the fields of pattern recognition and artificial intelligence, and in particular to a method, apparatus and storage medium for reference image segmentation based on a basic model. Background Technology

[0002] With the rapid development of computer technology, a multimodal image segmentation task utilizing text-guided segmentation—referential image segmentation—has emerged in the field of image segmentation. This task can be specifically applied in human-computer interaction, interactive image editing, autonomous driving, and other common application scenarios of image segmentation tasks.

[0003] Meanwhile, thanks to the massive amounts of internet data and abundant computing resources, research on foundational models has developed rapidly. By pre-training on large-scale data, foundational models have demonstrated remarkable potential across multiple downstream tasks. However, few works have applied foundational models to dense prediction tasks, such as image segmentation, especially algebraic image segmentation. Furthermore, existing algebraic image segmentation work often focuses on fusing information from text and images to better locate the target, but fails to achieve high-precision segmentation, i.e., outputting a mask with clear boundaries. Summary of the Invention

[0004] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, apparatus and storage medium for reference image segmentation based on a basic model.

[0005] The technical solution adopted in this invention is:

[0006] A method for representing image segmentation based on a base model includes the following steps:

[0007] Acquire images and text as input data, and use the target mask corresponding to the text as label data;

[0008] Construct a neural network for denotating images and input the obtained images and text into the neural network;

[0009] Image features are extracted from images using an image encoder to obtain multi-scale image features; text features are extracted from text using a text encoder to obtain global text features and sequence text features.

[0010] The first multimodal feature is obtained by fusing image features with global text features using a multimodal feature pyramid network.

[0011] The first multimodal feature is fused with the sequence text feature using the Transformer Decoder to obtain the second multimodal feature;

[0012] Based on the second multimodal feature, a sparse point sampling method is used to individually supervise the boundary points of the target to obtain the output mask;

[0013] Calculate the loss between the output mask and the target mask, and train the representational image segmentation neural network based on the loss.

[0014] Furthermore, the feature extraction of the image through the image encoder includes:

[0015] Image feature extraction is performed using the CLIP-RN101 multimodal base model image encoder;

[0016] The feature extraction of text through a text encoder includes:

[0017] The text encoder uses the multimodal base model CLIP-RN101 to extract features from the text.

[0018] Furthermore, the process of fusing image features with global text features to obtain the first multimodal feature includes:

[0019] A multimodal feature pyramid network is used to fuse image features at multiple scales output by the image encoder. Pooling or interpolation operations are used to unify the size of image features at different scales. In this process, global text features are incorporated through dot product to obtain the first multimodal feature F. m1 .

[0020] Furthermore, the process of fusing image features with global text features to obtain the first multimodal feature includes:

[0021] First, the image features F output by the image encoder are... v3 The image features F from the three multi-scale features are then fused with global text features through a dot product operation. v1 The fused image features F v3 The size is scaled to the image feature F through interpolation or pooling operations. v2 The image features are then concatenated using three identically sized image features and passed through an aggregation network. This aggregation network consists of a 1x1 convolutional layer, a batch normalization layer, and a ReLU nonlinear activation function. Finally, to enhance the positional information of the image features, the Grid-based two-dimensional positional information is summed with the concatenated features to obtain the first multimodal feature F. m1 .

[0022] Furthermore, the process of fusing the first multimodal feature with the sequence text feature to obtain the second multimodal feature includes:

[0023] The first multimodal feature F is processed using the Transformer Decoder. m1 With sequence text features F t Interact with the system to complete the second fusion and obtain the second multimodal features;

[0024] The interaction process uses an attention mechanism, with the first multimodal feature F m1 As the query Q, the sequence text feature F t The calculation formula is as follows, with key K and value V as the input:

[0025]

[0026] Among them, W q W k W v These are the linear transformation weights for query Q, key K, and value V, respectively.

[0027] Furthermore, in the first multimodal feature F m1 Before the sequential text features are fed into the Transformer DecoderLayer for secondary fusion interaction, since the Transformer's attention mechanism is independent of position, positional encoding is needed to represent the positional information of the image and text. For image-dimensional features, two-dimensional sine and cosine positional encoding is used; for sequence-dimensional features, one-dimensional sine and cosine positional encoding is used.

[0028] Furthermore, the step of individually supervising the boundary points of the target using a sparse point sampling method based on the second multimodal features to obtain the output mask includes:

[0029] The obtained second multimodal feature F m2 The multimodal features are fed into a cascaded upsampling layer and a 3x3 convolutional layer to restore the dimension of the original input image, and a preliminary output mask M1 is obtained through a 1x1 convolutional layer.

[0030] During the upsampling process, the low-level image features F, which contain rich texture information, are stitched together from the image encoder. v4 Integrate into multimodal features to enhance target boundary features;

[0031] Sparse point matching: First, randomly oversample the points of the output mask M1 using a uniform distribution; then, based on the classification score of the output mask M1, select the K points among the points sampled by the uniform distribution that are closest to the segmentation threshold; finally, use the global text features F obtained from the text encoder. s With the sampled pixel features F p The dot product is calculated using the following formula:

[0032] P = F p ·F s

[0033] Finally, P is used as the new point classification score and returned to its original position.

[0034] Furthermore, during training, the calculated loss consists of two parts: the initial mask loss, which is composed of BCE Loss and DICE Loss; and the sparse point matching loss, which is composed of BCE Loss.

[0035] The expression for the total loss is as follows:

[0036] L total =L M +L P

[0037] =L bce (M1,M gt )+L dice (M1,M gt )+L bce (P,P gt )

[0038] Among them, M gt With P gt These are the mask label data and the mask sampling point label data, respectively.

[0039] Another technical solution adopted in this invention is:

[0040] A reference image segmentation device based on a fundamental model, comprising:

[0041] At least one processor;

[0042] At least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor performs the method as described above.

[0044] Another technical solution adopted in this invention is:

[0045] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0046] The beneficial effects of this invention are: this invention effectively leverages the potential of the multimodal basic model CLIP in the task of referential image segmentation, making full use of its multimodal representation capabilities, and transferring its text feature retrieval capabilities from the image dimension to the pixel dimension in the form of sparse pixel matching, so that the entire model method can focus on the boundary information of the target to output a target mask with clear boundaries. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart of the steps of a referential image segmentation method based on a basic model in an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the multimodal feature pyramid network in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the CLIP-based segmentation and refining module in an embodiment of the present invention;

[0051] Figure 4 This is a segmentation effect diagram for high-precision referential image segmentation in an embodiment of the present invention. Detailed Implementation

[0052] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0054] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0055] Furthermore, in the description of this invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0056] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0057] Terminology Explanation:

[0058] CLIP, short for Contrastive Language-Image Pre-training, is a multimodal foundational model proposed by OpenAI in 2021. This model learns through contrastive learning on approximately 400 million image-text pairs, enabling it to understand the relationships between images and text. It has demonstrated impressive performance scores and generalization abilities on numerous datasets, and its excellent multimodal representation capabilities can be readily transferred to various downstream tasks. The CLIP model primarily consists of an image encoder and a text encoder. This application aims to fully leverage the potential of the pre-trained CLIP foundational model in image segmentation methods.

[0059] like Figure 1 As shown, this example provides a high-quality referential image segmentation method based on a fundamental model to address the problem of obtaining high-precision segmentation masks in referential image segmentation tasks, thereby improving the performance of referential image segmentation. The method specifically includes the following steps:

[0060] S1. Obtain the dataset and perform data preprocessing: Obtain images and text as input data, and use the target mask corresponding to the text as label data.

[0061] In this example, the network is trained using image, text, and mask annotation data from the RefCOCO, RefCOCO+, and RefCOCOg public datasets. Before training, the training data is processed using the LMDB library to store it as a binary file, accelerating data retrieval during training. During training, the image data needs to be uniformly scaled to 512x512 and normalized. Simultaneously, padding is applied to text sequences of different lengths to ensure that text sequences within the same batch have the same length of 30 characters.

[0062] S2. Construct a reference image segmentation network based on the multimodal basic model CLIP, and use the pre-trained CLIP as the backbone network to extract image features and text features respectively.

[0063] Before training begins, the CLIP model is initialized using pre-trained weights provided by OpenAI. During training, the CLIP image encoder extracts image features, and the CLIP text encoder extracts text features, which include a sequence text feature and a global text feature. The specific network structures of the image encoder and text encoder are shown in Table 1 below:

[0064] Table 1

[0065]

[0066]

[0067] S3. A multimodal feature pyramid network is used to fuse multi-scale image features with global text features.

[0068] like Figure 2 As shown, in one fusion process, firstly, the image encoder outputs the image features F. v3 The image features are interactively fused with global text features through dot product operations. Next, the image features F from the three multi-scale features are... v1 The fused image features F v3 The size is scaled to the image feature F through interpolation or pooling operations. v2 The dimensions are 32×32. Then, the three features of the same size are concatenated and passed through an aggregation network. This aggregation network includes a 1x1 convolutional layer, a batch normalization layer, and a ReLU non-linear activation function. Finally, to enhance the positional information of the image features, the 2D positional information based on the grid is summed with the fused features to obtain the first multimodal feature F after one fusion. m1 .

[0069] S4. Use Transformer Decoder to perform secondary fusion of the first multimodal features and the sequence text features.

[0070] During the secondary fusion process, an attention-based Transformer Decoder was used to process the first multimodal feature F. m1 A second multimodal feature F is obtained by performing secondary fusion interaction with the sequence text features. m2 The Transformer Decoder used consists of three Transformer Decoder Layers. Each Transformer Decoder Layer includes a self-attention layer, a cross-attention layer, and a feedforward neural network layer. The specific network structure of the Transformer Decoder Layer is shown in Table 2 below:

[0071] Table 2

[0072] Input layer - 1*30*512 Self-attention layer Multi-head attention mechanism 1*30*512 Layer normalization layer Stable training 1*30*512 Cross attention layer Multi-head attention mechanism 1*30*512 Layer normalization layer Stable training 1*30*512 Feedforward Neural Network Nonlinear transformation and feature extraction 1*30*512

[0073] The first multimodal feature F m1 Before the sequential text features are fed into the Transformer Decoder Layer for secondary fusion, positional encoding is needed to represent the positional information of the image and text because the Transformer's attention mechanism is position-independent. For image-dimensional features, two-dimensional sine and cosine positional encoding is used. For sequence-dimensional features, one-dimensional sine and cosine positional encoding is used. The formulas for calculating sine and cosine positional encoding are shown below:

[0074]

[0075]

[0076] The interaction process primarily utilizes an attention mechanism, with the first multimodal feature F m1 As a query, F t As keys and values, the following calculation formula applies:

[0077]

[0078] Among them, W q W k W υ These are the linear transformation weights for Query, Key, and Value, respectively.

[0079] S5. Design a CLIP-based segmentation and refinement module, which enhances the texture information of the mask features by utilizing the low-level features from the image encoder, and uses a sparse point sampling method to supervise the boundary points of the target separately.

[0080] See Figure 3 As an optional implementation, step S5 specifically includes steps S51-S53:

[0081] S51, the second multimodal feature F obtained in step S4 m2 The multimodal feature dimensions are fed into a cascaded upsampling layer and a 3x3 convolutional layer to restore the dimensions of the original input image, and a preliminary output mask M1 is obtained through a 1x1 convolutional layer.

[0082] S52. During the upsampling process, the low-level features of the image encoder with rich texture information are incorporated into the multimodal features by splicing to enhance the target boundary features.

[0083] S53. Sparse Point Matching: First, randomly oversample the points of the output mask M1 using a uniform distribution. Then, based on the classification score of M1, select the K points from the uniformly distributed samples that are closest to the segmentation threshold of 0.5 (generally, points with classification scores near the segmentation threshold are often points on the target boundary). Utilize the global text features F obtained from the text encoder... s With the sampled pixel features F p The dot product is calculated using the following formula:

[0084] P = F p ·F s

[0085] Finally, P is used as the new point classification score and returned to its original position.

[0086] S6. Calculate the loss between the output mask and the label mask data, and train the entire reference image segmentation neural network based on the loss.

[0087] Segmentation is essentially a pixel-level classification task, and for image segmentation, the categories are only foreground and background. Therefore, binary cross-entropy loss (BCE Loss) is used instead of multi-class cross-entropy loss (CE Loss) in the loss calculation. The loss consists of two parts: the initial mask loss, composed of BCE Loss and DICE Loss; and the sparse point matching loss, also composed of BCE Loss. Therefore, the total loss can be expressed by the following formula:

[0088] L total =L M +L P

[0089] =L bce (M1,M gt )+L dice (M1,M gt )+L bce (P,P gt )

[0090] Among them, M gt With P gt These are the mask label data and the mask sampling point label data, respectively.

[0091] In the deep learning network, the image and text encoders are initialized using pre-trained CLIP model parameters, while the remaining networks are trained with randomly initialized parameters. The training process employs the gradient descent algorithm, calculating gradients from the last layer and propagating them layer by layer to update all network parameters, thus achieving the goal of training the network. The Adam training optimizer is used, with the initial learning rate and weight decay set to 0.00005 and 0.01, respectively, and a cosine annealing learning rate decay strategy is employed.

[0092] like Figure 4 As shown, the method presented in this example can output high-quality target masks with clear boundaries, achieving more detailed segmentation results compared to existing superior methods like CRIS. Furthermore, it can output target masks with clear boundaries well for targets of different sizes.

[0093] In summary, the method of the present invention has at least the following advantages and effective gains compared with the prior art:

[0094] (1) The high-quality reference image segmentation method based on the basic model designed in this invention can effectively utilize the multimodal basic model CLIP and give full play to its potential in dense prediction tasks.

[0095] (2) The CLIP-based refining module designed in this invention makes full use of CLIP’s multimodal representation capabilities, enabling the model to focus more on target boundary features and output a high-quality target mask with clear boundaries.

[0096] (3) Based on CLIP, this invention inherits the powerful generalization performance of CLIP trained on 400M image-text pairs, making the method of this invention more applicable to real-world scenarios.

[0097] (4) The method of using the basic model involved in this invention provides a very meaningful precedent for other researchers on how to use the basic model to better enhance downstream tasks.

[0098] This embodiment also provides a reference image segmentation device based on a basic model, including:

[0099] At least one processor;

[0100] At least one memory for storing at least one program;

[0101] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.

[0102] This embodiment of the device for representing image segmentation based on a base model can execute the method for representing image segmentation based on a base model provided in the method embodiment of the present invention. It can execute any combination of implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0103] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0104] This embodiment also provides a storage medium storing instructions or programs that can execute the reference image segmentation method based on a basic model provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0105] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0106] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0107] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0109] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0110] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0111] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0113] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for representing image segmentation based on a fundamental model, characterized in that, Includes the following steps: Acquire images and text as input data, and use the target mask corresponding to the text as label data; Construct a neural network for denotating images and input the obtained images and text into the neural network; Image features are extracted from images using an image encoder to obtain multi-scale image features; text features are extracted from text using a text encoder to obtain global text features and sequence text features. The first multimodal feature is obtained by fusing image features with global text features; The first multimodal feature is fused with the sequence text feature to obtain the second multimodal feature; Based on the second multimodal feature, a sparse point sampling method is used to individually supervise the boundary points of the target to obtain the output mask; Calculate the loss between the output mask and the target mask, and train the reference image segmentation neural network based on the loss. The process of fusing image features with global text features to obtain the first multimodal feature includes: First, the image features output by the image encoder are... The image features are then fused with global text features through a dot product operation; next, the image features from the three multi-scale features are combined. The fused image features Scaling the size to the image features using interpolation or pooling operations. The dimensions of the three images are determined; then, the features of the three images of the same size are concatenated and passed through an aggregation network; finally, the 2D location information based on the grid is summed with the concatenated features to obtain the first multimodal feature. ; The process of fusing the first multimodal feature with the sequence text feature to obtain the second multimodal feature includes: Use Transformer Decoder to process the first multimodal features. Features of Sequence Text Interact with the system to complete the second fusion and obtain the second multimodal features; The interaction process uses an attention mechanism, with the first multimodal feature As query Q, sequence text features As the key K and value V, the calculation formula is as follows: in, , , These are the linear transformation weights for query Q, key K, and value V, respectively. In the first multimodal feature Before the sequential text features are fed into the Transformer Decoder Layer for secondary fusion interaction, since the Transformer's attention mechanism is independent of position, positional encoding is needed to represent the positional information of the image and text. For image-dimensional features, two-dimensional sine and cosine positional encoding is used; for sequence-dimensional features, one-dimensional sine and cosine positional encoding is used. The step of individually supervising the boundary points of the target using a sparse point sampling method based on the second multimodal features to obtain the output mask includes: The obtained second multimodal features The multimodal features are fed into a cascaded upsampling layer and a 3x3 convolutional layer to restore the dimension of the original input image, and a preliminary output mask is obtained through a 1x1 convolutional layer. ; During the upsampling process, the low-level image features of the image encoder are stitched together. Integrate into multimodal features to enhance target boundary features; Sparse point matching: Use a uniform distribution to first randomly oversample and output a mask. The point; based on the output mask The classification score is used to select the K points closest to the segmentation threshold from the uniformly distributed sampled points; the global text features obtained from the text encoder are then used. Features of the sampled pixels The dot product is calculated using the following formula: Finally, P is used as the new point classification score and returned to its original position.

2. The method for referential image segmentation based on a fundamental model according to claim 1, characterized in that, The feature extraction of the image through the image encoder includes: Image feature extraction is performed using the CLIP-RN101 multimodal base model image encoder; The feature extraction of text through a text encoder includes: The text encoder uses the multimodal base model CLIP-RN101 to extract features from the text.

3. The method for referential image segmentation based on a fundamental model according to claim 1, characterized in that, The process of fusing image features with global text features to obtain the first multimodal feature includes: A multimodal feature pyramid network is used to fuse image features at multiple scales output by the image encoder. Pooling or interpolation operations are used to unify the size of image features at different scales, and global text features are incorporated in the process to obtain the first multimodal feature. .

4. The method for referential image segmentation based on a fundamental model according to claim 1, characterized in that, During training, the calculated loss consists of two parts: the initial mask loss, which is composed of BCE Loss and DICE Loss; and the sparse point matching loss, which is composed of BCE Loss. The expression for the total loss is as follows: in, and These are the mask label data and the mask sampling point label data, respectively.

5. A referential image segmentation device based on a fundamental model, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-4.

6. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-4.

Images