Training method and device for image panoptic segmentation model based on open vocabulary segmentation

Abstract

The present invention relates to a training method and device for an image panoramic segmentation model based on open vocabulary segmentation, which obtains a training sample set and a preset point grid; the training sample set includes at least one image and a pixel-level category label corresponding to each image in the at least one image; based on the preset point grid, at least one image and the pixel-level category label corresponding to each image, a query embedding vector corresponding to each point in the preset point grid is determined; based on at least one image and the query embedding vector corresponding to each point, a predicted mask region vector corresponding to each point is generated in combination with a pre-trained feature extraction and mask generation model; based on at least one image and the predicted mask region vector corresponding to each point, a preset cosine similarity loss function is combined to update the parameters of a multi-scale feature adapter. Without consuming a large amount of manual annotation costs, image panoramic segmentation can be completed and the segmentation performance of image segmentation can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image panoramic segmentation, and in particular to a training method and device for an image panoramic segmentation model based on open vocabulary segmentation. Background Art

[0002] To overcome the limitations of closed vocabulary segmentation, open vocabulary segmentation was proposed. Open vocabulary segmentation uses text embeddings of category names represented in natural language as label embeddings instead of learning them from a training dataset. By doing so, the model can classify a wider range of vocabulary, thereby improving the ability to handle a wider range of categories. To ensure that meaningful embeddings are provided, a pre-trained text encoder is usually used. This encoder can effectively capture the semantic meaning of words and phrases, which is critical for open vocabulary segmentation. Multimodal models such as (Contrastive language-image Pre-Training, CLIP) have shown potential in open vocabulary segmentation because they are able to learn aligned image-text feature representations from large-scale Internet data.

[0003] Currently, image semantic segmentation based on open vocabulary segmentation usually relies on image-mask-text triples, but this method requires a lot of manual effort to annotate the correspondence between mask and text, which leads to expensive annotation costs.

[0004] Although some weakly supervised methods have been proposed in the prior art, such as text supervision to reduce annotation costs, the incompleteness of supervision severely limits the versatility and performance. Among them, text supervision only utilizes image and text pairs for semantic segmentation, which is insufficient in capturing complex spatial details, which is not optimal for dense prediction. In addition, this type of supervision, text supervision, lacks location information, making it difficult for the model to distinguish different instances with the same semantic class. These problems severely limit the versatility and segmentation performance of existing weakly supervised methods.

[0005] Therefore, the existing techniques for image segmentation based on open vocabulary segmentation require expensive annotation costs, while limiting the versatility and segmentation performance of image segmentation. Summary of the invention

[0006] The present invention provides a training method and device for an image panoramic segmentation model based on open vocabulary segmentation, so as to solve the problem that in the prior art, expensive annotation costs are required during the training process of the image panoramic segmentation model based on open vocabulary segmentation, while limiting the versatility and segmentation performance of image segmentation. The present invention achieves the versatility of image segmentation and improves the segmentation performance of image segmentation without consuming a large amount of manual cost to annotate the relationship between images, masks and texts.

[0007] A training method for an image panoramic segmentation model based on open vocabulary segmentation, the method comprising: obtaining a training sample set and a preset point grid; the training sample set comprising at least one image and a pixel-level category label corresponding to each image in the at least one image; the preset point grid comprising at least one point; based on the preset point grid, the at least one image and the pixel-level category label corresponding to each image, determining a query embedding vector corresponding to each point in the preset point grid; wherein the query embedding vector corresponding to each point comprises a position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in each image; based on the The query embedding vector corresponding to the at least one image and each point is combined with a pre-trained feature extraction and mask generation model to generate a predicted mask area vector corresponding to each point; based on the at least one image and the predicted mask area vector corresponding to each point, combined with a preset cosine similarity loss function, the parameters of the multi-scale feature adapter are updated; the multi-scale feature adapter and the multi-scale pixel decoder in the pre-trained feature extraction and mask generation model are used to enhance the visual embedding of the predicted mask; the preset cosine similarity loss function is used to realize the determination of the text embedding vector corresponding to the predicted mask area vector corresponding to each point from the at least one text embedding vector corresponding to each image.

[0008] In one of the embodiments, the pre-trained feature extraction and mask generation model also includes: a visual cue encoder and a preset visual encoder; the query embedding vector corresponding to each point in the preset point grid is determined based on the preset point grid, the at least one image and the pixel-level category label corresponding to each image, including: for each point in the preset point grid, encoding each point into two position embedding vectors based on the visual cue encoder, and determining at least one pixel-level real mask embedding vector corresponding to each point in each image based on the pixel-level category label corresponding to each image and the visual cue encoder; inputting each image into the preset visual encoder to determine the content embedding vector corresponding to the at least one real mask embedding vector; for each point in the preset point grid, concatenating the two position embedding vectors corresponding to each point and the at least one pixel-level real mask embedding vector corresponding to each point in each image, and combining them with the preset query type corresponding to each point and the content embedding vector corresponding to the at least one real mask embedding vector to obtain the query embedding vector corresponding to each point.

[0009] In one of the embodiments, before combining the pre-trained feature extraction and mask generation model to generate a predicted mask area vector corresponding to each point, the method further includes: based on the at least one image and the query embedding vector corresponding to each point, combining the mask generation and semantic segmentation loss function, updating the pre-trained feature extraction and mask generation model.

[0010] In one embodiment, the feature extraction and mask generation model also includes a visual cue encoder, a preset visual encoder and a mask decoder, and the pre-trained feature extraction and mask generation model is updated based on the at least one image and the query embedding vector corresponding to each point, combined with the mask generation and semantic segmentation loss function, including: for each image in the at least one image, feature extraction is performed through the preset visual encoder and the multi-scale pixel decoder to obtain a multi-scale pixel feature vector of each image; based on the query embedding vector corresponding to each point, the multi-scale pixel feature vector of each image and the mask generation and semantic segmentation loss function, the parameters of the visual cue encoder, mask decoder and multi-scale pixel decoder are updated to obtain a pre-trained feature extraction and mask generation model.

[0011] In one of the embodiments, the updating of the parameters of the multi-scale feature adapter based on the at least one image and the predicted mask area vector corresponding to each point, combined with a preset cosine similarity loss function, includes: for each image in the at least one image, inputting each image into a preset enhanced image text extraction model to determine at least one text embedding vector corresponding to each image; inputting each image in the at least one image into a preset visual encoder in the feature extraction and mask generation model to obtain an initial pixel feature vector for each point; and inputting the initial pixel feature vector of each point and the predicted mask area vector corresponding to each point into the multi-scale feature adapter to obtain a pixel feature vector for each point; and updating the parameters of the multi-scale feature adapter based on at least one text embedding vector corresponding to all images in the training sample set, the pixel feature vector of each point in the point grid, and the preset cosine similarity loss function.

[0012] In one of the embodiments, the preset enhanced image text extraction model includes: a preset visual language model, a preset text language enhancement model and a preset text encoder; the step of inputting each image into the preset enhanced image text extraction model to determine at least one text embedding vector corresponding to each image includes: inputting each image into the preset visual language model to determine an initial text embedding vector of each image; the initial text embedding vector of each image includes a descriptive text feature representation of each image; the initial text embedding vector of each image is input into the preset text language enhancement model to determine at least one enhanced text represented by the text features of each image; and the at least one enhanced text represented by the text features of each image is input into the preset text encoder to obtain at least one text embedding vector corresponding to each image.

[0013] In one of the embodiments, the updating of the parameters of the multi-scale feature adapter based on at least one text embedding vector corresponding to all images in the training sample set, the pixel feature vector of each point in the point grid and the preset cosine similarity loss function includes: for each point, based on the arc cosine similarity between the pixel feature vector of each point and each text embedding vector corresponding to each image, determining the cost matrix between the pixel feature vector of each point and each text embedding vector corresponding to each image; based on the cost matrix between the pixel feature vector of each point and all text embedding vectors corresponding to each image, and a bipartite graph matching algorithm, determining the best matching pixel feature vector and text embedding vector corresponding to each point; inputting the best matching pixel feature vector and text embedding vector corresponding to each point into the preset cosine similarity loss function to obtain the loss function value corresponding to each point, and updating the parameters of the multi-scale feature adapter based on the loss function value corresponding to each point.

[0014] In one embodiment, the expression corresponding to the cost matrix between the predicted mask region vector corresponding to each point and each text embedding vector corresponding to each image is: Among them, δ' i,j represents the arc cosine similarity between the pixel feature vector corresponding to the i-th point and the j-th text embedding vector corresponding to each image, r i Represents the pixel feature vector of the i-th point, t j represents the jth text embedding vector corresponding to each image; the preset cosine similarity loss function is: Among them, r i ' represents the best matching pixel feature vector corresponding to the i-th point, t k Represents the kth text embedding vector of the best match corresponding to the i-th point.

[0015] The present invention also provides a training device for an image panoramic segmentation model based on open vocabulary segmentation, the device comprising: an acquisition module, used to acquire a training sample set and a preset point grid; the training sample set comprises at least one image, and a pixel-level category label corresponding to each image in the at least one image; the preset point grid comprises at least one point; a determination module, used to determine a query embedding vector corresponding to each point in the preset point grid based on the preset point grid, the at least one image and the pixel-level category label corresponding to each image; wherein the query embedding vector corresponding to each point comprises a position embedding vector of each point, and at least one pixel-level true mask embedding vector corresponding to each point in each image; A generation module is used to generate a prediction mask area vector corresponding to each point based on the at least one image and the query embedding vector corresponding to each point, combined with a pre-trained feature extraction and mask generation model; an update module is used to update the parameters of the multi-scale feature adapter based on the at least one image and the prediction mask area vector corresponding to each point, combined with a preset cosine similarity loss function; the multi-scale feature adapter and the multi-scale pixel decoder in the pre-trained feature extraction and mask generation model are used to enhance the visual embedding of the prediction mask; the preset cosine similarity loss function is used to realize the determination of the text embedding vector corresponding to the prediction mask area vector corresponding to each point from the at least one text embedding vector corresponding to each image.

[0016] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned training method of the image panoramic segmentation model based on open vocabulary segmentation.

[0017] The present invention also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, causes the one or more processors to execute the steps of the above-mentioned training method for the image panoramic segmentation model based on open vocabulary segmentation.

[0018] The above-mentioned training method and device of the image panoramic segmentation model based on open vocabulary segmentation, by providing a preset point grid and the pixel-level category label corresponding to each image, determines the query embedding vector of each point in the preset point grid, realizes the correspondence between the image pixel-level category label and the position embedding vector, and further combines the pre-trained feature extraction and mask generation model to determine the predicted mask area vector corresponding to each point, and realizes the pixel-level mask division of the image. Combined with the preset cosine similarity loss function, the predicted mask area vector corresponding to each point is matched with the text embedding vector corresponding to each image, and the entity alignment in the text description corresponding to each image is realized, so as to realize the pixel-level segmentation of each image under the condition of decoupling the mask and the text. In this way, the decoupling of the mask and the text is realized, and the relationship between the corresponding image, the mask and the text is not spent a lot of labor costs, so as to realize the use of independent image-mask and image-text pairs to liberate the strict correspondence between the mask and the text. In addition, while realizing the decoupling of the mask and the text, it is possible not only to realize the semantic segmentation of each image, but also to distinguish the instances of each image. Furthermore, in order to reduce the inherent noise in the correspondence between masks and entities, corresponding multi-scale feature adapters and multi-scale pixel decoders are added to enhance the visual embedding of the predicted masks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 One of the flow charts of the training method of the image panoptic segmentation model based on open vocabulary segmentation provided by the present invention;

[0020] Figure 2 A second flow chart of a method for training an image panoptic segmentation model based on open vocabulary segmentation provided by the present invention;

[0021] Figure 3 A third flowchart of a method for training an image panoptic segmentation model based on open vocabulary segmentation provided by the present invention;

[0022] Figure 4 A fourth flowchart diagram of a training method for an image panoptic segmentation model based on open vocabulary segmentation provided by the present invention;

[0023] Figure 5 A fifth flowchart of a method for training an image panoptic segmentation model based on open vocabulary segmentation provided by the present invention;

[0024] Figure 6 A schematic diagram of the framework of the training method of the image panoptic segmentation model based on open vocabulary segmentation provided by the present invention;

[0025] Figure 7 A schematic diagram of the structure of the mask decoder layer provided by the present invention;

[0026] Figure 8 A schematic diagram of the framework of the multi-scale feature adapter provided by the present invention;

[0027] Fig. 9 A comparison chart of experimental results of the training method of the image panoptic segmentation model based on open vocabulary segmentation provided by the present invention and other methods;

[0028] Fig.10 A schematic diagram of the framework of a training device for an image panoptic segmentation model based on open vocabulary segmentation provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0031] To facilitate understanding, the inventive concept of the present invention is described.

[0032] Currently, image panoptic segmentation models based on open vocabulary segmentation are usually trained by relying on image-mask-text triplets, but this method requires a lot of manual effort to annotate the correspondence between mask and text, which leads to expensive annotation costs.

[0033] Although some weak supervision methods have been proposed in the prior art, such as reducing annotation costs through text supervision, the incompleteness of supervision severely limits the versatility and performance. Among them, text supervision only uses image and text pairs for semantic segmentation, which is insufficient in capturing complex spatial details, which is not optimal for dense prediction. In addition, this type of supervision, text supervision, lacks location information, making it difficult for the model to distinguish different instances with the same semantic class, that is, this type is only suitable for ordinary semantic segmentation and is difficult to apply to instance segmentation or panoramic segmentation. These problems severely limit the versatility and segmentation performance of existing weak supervision methods.

[0034] Therefore, in view of the above problems, the training method and device of the image panoramic segmentation model based on open vocabulary segmentation provided by the present invention, by providing a preset point grid and the pixel-level category label corresponding to each image, determine the query embedding vector of each point in the preset point grid, realize the correspondence between the image pixel-level category label and the position embedding vector, further combine the pre-trained feature extraction and mask generation model, determine the predicted mask area vector corresponding to each point, and realize the pixel-level mask division of the image. Combined with the preset cosine similarity loss function, the predicted mask area vector corresponding to each point is matched with the text embedding vector corresponding to each image, and the entity alignment in the text description corresponding to each image is realized, so as to realize the pixel-level segmentation of each image under the condition of decoupling the mask and the text. In this way, the decoupling of the mask and the text is realized, and the relationship between the corresponding image, the mask and the text does not need to be expended a lot of labor costs, so as to realize the use of independent image-mask and image-text pairs to liberate the strict correspondence between the mask and the text. Moreover, by decoupling the mask from the text, not only can the semantic segmentation of each image be realized, but also the instance distinction of each image can be realized. In addition, in order to reduce the inherent noise in the correspondence between the mask and the entity, the corresponding multi-scale feature adapter and multi-scale pixel decoder are added to enhance the visual embedding of the predicted mask.

[0035] The following describes the training method and device of the image panoptic segmentation model based on open vocabulary segmentation provided by the present invention in conjunction with the accompanying drawings.

[0036] Figure 1 A flow chart of a training method for an image panoramic segmentation model based on open vocabulary segmentation provided by the present invention. It can be understood that the training method for an image panoramic segmentation model based on open vocabulary segmentation can be performed by a training device for an image panoramic segmentation model based on open vocabulary segmentation. The training device for an image panoramic segmentation model based on open vocabulary segmentation can be a computer device. The image panoramic segmentation model based on open vocabulary segmentation includes at least: a feature extraction and mask generation model and a multi-scale feature adapter.

[0037] like Figure 1 As shown, in one embodiment, a training method for an image panoptic segmentation model based on open vocabulary segmentation is proposed, which may specifically include the following steps:

[0038] Step 110, obtaining a training sample set and a preset point grid; the training sample set includes at least one image and a pixel-level category label corresponding to each image; the preset point grid includes at least one point.

[0039] Each of the images is an image to be panoptically segmented, for example, it can be an image in the field of medical imaging or self-driving cars. The pixel-level category label corresponding to each image can be understood as each pixel corresponding to each image has a corresponding category label, including not only the category to which each pixel belongs, but also the instance identifier of each pixel. Each pixel corresponding to each image has a corresponding category label, for example, it can be person 1, person 2, cat 1 or cat 2. Moreover, the category label is a category label expressed in text.

[0040] The preset point grid is used as a visual cue for each image to determine the predicted mask region vector, mainly used to align the position information of the mask and each image. The preset point grid includes a plurality of evenly distributed points. The preset point grid is a grid of size h×w, with a height of h≤4 and a width of w≤4, and the grid is aligned with the center of the pixel.

[0041] Step 120, based on the preset point grid, the at least one image and the pixel-level category label corresponding to each image, determine the query embedding vector corresponding to each point in the preset point grid; wherein the query embedding vector corresponding to each point includes the position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in each image.

[0042] It can be understood that in order to use independent image-mask and image-text pairs to liberate the strict correspondence between mask and text, and at the same time to be able to capture complex spatial details, obtain more location information, and distinguish different instances of the same semantic class in each image, by setting a preset point grid, based on the preset point grid and the pixel-level category label corresponding to each image, the query embedding vector of each point in the preset point grid can be learned, thereby obtaining the pixel-level true mask embedding vector corresponding to each image. Therefore, based on the query embedding vector corresponding to each point, the predicted mask area vector corresponding to each point is determined, which can not only realize semantic segmentation of each image, but also instance segmentation and panorama segmentation of each image, thereby realizing the use of weak supervision methods, reducing the cost of manual annotation, while ensuring the versatility and segmentation performance of image segmentation.

[0043] Furthermore, in order to subsequently extract features of the image and mask based on the query embedding vector, and to perform preliminary classification and semantic segmentation of the mask of each pixel in the image based on the pixel-level category label corresponding to each image, so as to facilitate further matching of the mask with the text, step 130 may be further executed.

[0044] Step 130, based on the at least one image and the query embedding vector corresponding to each point, combined with a pre-trained feature extraction and mask generation model, generate a predicted mask region vector corresponding to each point.

[0045] Among them, the pre-trained feature extraction and mask generation model is used to complete the extraction of visual embedding features of the mask, and has the functions of pixel-level mask classification and segmentation.

[0046] Step 140: Based on the at least one image and the predicted mask region vector corresponding to each point, in combination with a preset cosine similarity loss function, update the parameters of the multi-scale feature adapter.

[0047] Among them, the multi-scale feature adapter and the multi-scale pixel decoder in the pre-trained feature extraction and mask generation model are used to enhance the visual embedding of the prediction mask; the preset cosine similarity loss function is used to determine the text embedding vector corresponding to the prediction mask area vector corresponding to each point from at least one text embedding vector corresponding to each image.

[0048] It can be understood that since the present invention uses independent image-mask and image-text pairs, and due to different shooting angles, instances in images of the same scene may be located at different positions in the image, resulting in inherent noise in the correspondence between the mask and the entity (instance) in the image corresponding to the same scene. Therefore, in order to learn a richer visual embedding of the mask, the present invention uses a multi-scale pixel decoder and a multi-scale feature adapter to improve the quality of the visual embedding and effectively solve the inherent noise in the correspondence between the mask and the entity, thereby stabilizing the matching process, and the use of a multi-scale feature adapter is intended to enhance the visual embedding of the predicted mask and further align it with the text embedding of the corresponding entity.

[0049] The training method of the image panoramic segmentation model based on open vocabulary segmentation provided by the present invention provides a preset point grid and a pixel-level category label corresponding to each image, determines the query embedding vector of each point in the preset point grid, realizes the correspondence between the image pixel-level category label and the position embedding vector, further combines the pre-trained feature extraction and mask generation model, determines the predicted mask area vector corresponding to each point, realizes the pixel-level mask division of the image, and then combines the preset cosine similarity loss function to match the predicted mask area vector corresponding to each point with the text embedding vector corresponding to each image, realizes the entity alignment in the text description corresponding to the mask and each image, thereby realizing the pixel-level segmentation of each image under the condition of decoupling the mask and the text. In this way, the decoupling of the mask and the text is realized, and the relationship between the corresponding image, the mask and the text does not need to be expended a lot of labor costs, thereby realizing the use of independent image-mask and image-text pairs to liberate the strict correspondence between the mask and the text. In addition, while realizing the decoupling of the mask and the text, it is possible not only to realize the semantic segmentation of each image, but also to distinguish the instances of each image. Furthermore, in order to reduce the inherent noise in the correspondence between masks and entities, corresponding multi-scale feature adapters and multi-scale pixel decoders are added to enhance the visual embedding of the predicted masks.

[0050] In one embodiment, if Figure 2 As shown, the pre-trained feature extraction and mask generation model also includes: a visual cue encoder and a preset visual encoder; the determining of the query embedding vector corresponding to each point in the preset point grid based on the preset point grid, the at least one image and the pixel-level category label corresponding to each image includes:

[0051] Step 210, for each point in the preset point grid, encode each point into two position embedding vectors based on the visual cue encoder, and determine at least one pixel-level true mask embedding vector corresponding to each point in each image based on the pixel-level category label and the visual cue encoder corresponding to each image.

[0052] Specifically, two position embedding vectors for each point can be determined based on the sinusoidal encoding in the visual cue encoder.

[0053] The pre-trained visual cue encoder is used to encode the visual cue (point or bounding box). The pre-trained visual cue encoder is determined based on mask generation and semantic segmentation loss function.

[0054] Step 220: input each of the images into the preset visual encoder to determine a content embedding vector corresponding to the at least one true mask embedding vector.

[0055] Among them, a preset visual encoder is used to extract visual features in each image, for example, it can be a CLIP model based on ConvNext. Specifically, the preset visual encoder can be configured as a ConvNext large model, including four stages. Each stage contains a different number of blocks: 3 in the first stage, 3 in the second stage, 27 in the third stage, and 3 in the fourth stage. Multi-scale features are extracted using the preset visual encoder. These features are represented as feature maps of different widths and scales: the 192-wide feature map is reduced by 4 times, the 384-wide feature map is reduced by 8 times, the 768-wide feature map is reduced by 16 times, and the 1536-wide feature map is reduced by 32 times.

[0056] Step 230, for each point in the preset point grid, concatenate the two position embedding vectors corresponding to each point and at least one pixel-level true mask embedding vector corresponding to each point in each image, and combine them with the preset query type corresponding to each point and the content embedding vector corresponding to the at least one true mask embedding vector to obtain the query embedding vector corresponding to each point.

[0057] Specifically, the expression of the query embedding vector corresponding to the i-th point is: in, Represents the two position embedding vectors corresponding to the i-th point, q msk ∈R M×dim represents the M true mask embedding vectors corresponding to the i-th point in each image, q type ∈R dim Indicates the preset query type corresponding to the i-th point, which can be a point or a box. feat ∈R dimRepresents a content embedding vector corresponding to the at least one true mask embedding vector.

[0058] In one embodiment, before step 130, the method further includes: updating a pre-trained feature extraction and mask generation model based on the at least one image and the query embedding vector corresponding to each point, combined with mask generation and semantic segmentation loss functions.

[0059] Among them, the mask generation and semantic segmentation loss function is used to enable the feature extraction and mask generation model to have pixel-level mask classification and segmentation functions.

[0060] Specifically, the mask generation and semantic segmentation loss functions can be:

[0061] where λ bce and λ dice are two hyperparameters used to balance the binary cross entropy loss and semantic segmentation loss Each point m i The corresponding predicted mask region vector, Includes k real mask region vectors m j , and the pixel-level category label c corresponding to each true mask region vector in each image j .

[0062] In one embodiment, if Figure 3 As shown, the feature extraction and mask generation model also includes a visual cue encoder, a preset visual encoder and a mask decoder, and the pre-trained feature extraction and mask generation model is updated based on the at least one image and the query embedding vector corresponding to each point, combined with the mask generation and semantic segmentation loss function, including:

[0063] Step 310: For each image in the at least one image, feature extraction is performed through the preset visual encoder and the multi-scale pixel decoder to obtain a multi-scale pixel feature vector of each image.

[0064] Among them, the multi-scale pixel decoder is used to enhance the visual features in each image extracted from the preset visual encoder, thereby reducing the inherent noise in the correspondence between the mask and the entity. The pre-trained multi-scale pixel decoder includes a feature pyramid network (FPN) architecture. First, a 6-layer multi-scale deformable transformer is used on the multi-scale features to aggregate contextual information. Then, the low-resolution feature map in the multi-scale pixel decoder is enlarged by 2 times and then combined with the corresponding resolution feature map from the backbone, which has been projected to match the channel dimension. The projection is achieved through a 1×1 convolutional layer, followed by group normalization (GroupNorm). Subsequently, the merged features are further processed by an additional 3×3 convolutional layer, supplemented by GroupNorm and ReLU activation. This process is applied iteratively, starting with a feature map that is 32 times smaller, until we obtain a final feature map that is 4 times smaller. In order to generate a pixel-by-pixel embedding, a single 1×1 convolutional layer is applied at the end. Throughout the multi-scale pixel decoder, all feature maps maintain a consistent dimension of 256 channels.

[0065] Step 320, based on the query embedding vector corresponding to each point, the multi-scale pixel feature vector of each image and the mask generation and semantic segmentation loss function, update the parameters of the visual cue encoder, mask decoder and multi-scale pixel decoder to obtain a pre-trained feature extraction and mask generation model.

[0066] Specifically, the query embedding vector corresponding to each point and the multi-scale pixel feature vector of each image can be input into the mask decoder to obtain the initial predicted mask area vector corresponding to each point, and then based on the initial predicted mask area vector corresponding to each point, the parameters of the visual cue encoder, mask decoder and multi-scale pixel decoder are iteratively updated.

[0067] Among them, the initial prediction mask area vector corresponding to each point can be obtained based on the matrix multiplication of the multi-scale pixel feature vector corresponding to each point and the query embedding vector corresponding to each point. The corresponding calculation formula can be: Among them, q i represents the query embedding vector corresponding to the i-th point, The multi-scale pixel feature vector corresponding to the i-th point, sigmoid(·) is the sigmoid function that normalizes the mask value to [0,1].

[0068] It can be understood that in actual scenarios, multiple points can be queried at the same time, so the query embedding vectors corresponding to one or more points can be used as a set of query embeddings, that is, Q = q1, q2, ... q n, then the corresponding calculation formula for the prediction mask area vector is: For example, it can be m n =sigmoid(Q;x p ), m n ∈R M×H×W ,

[0069] In the present invention, a transformer decoder can be used as a mask decoder. Specifically, the present invention uses six Figure 7 The mask decoder layer is shown, and the same loss function is applied after each layer. The specific loss function can refer to the first preset loss function below.

[0070] In one embodiment, if Figure 4 As shown, the updating of the parameters of the multi-scale feature adapter based on the at least one image and the predicted mask region vector corresponding to each point in combination with a preset cosine similarity loss function includes:

[0071] Step 410: for each image in the at least one image, input each image into a preset enhanced image text extraction model to determine at least one text embedding vector corresponding to each image.

[0072] The preset enhanced image text extraction model includes: a preset visual language model, a preset text language enhancement model and a preset text encoder.

[0073] Specifically, step 410 includes step 4110 to step 4130.

[0074] Step 4110, input each of the images into the preset visual language model to determine the initial text embedding vector of each image; the initial text embedding vector of each image includes a description text feature representation of each image.

[0075] Among them, the preset visual language model is used to refine the text description and improve the quality of the text description. It can be understood that since the collected image-text pairs usually contain some text that does not match the image, which leads to incorrect correspondence between the mask and the entity, directly aligning these incorrect correspondences will lead to poor segmentation performance. Therefore, the present invention improves the quality of the text description by setting a corresponding preset visual language model, so that the initial text embedding vector obtained for each image is a feature representation of the text description that matches each image more closely.

[0076] Preferably, the preset visual language model may be a Large Language and Vision Assistant (LLaVa) model. It is understood that the preset visual language model may also adopt other large visual language models, such as Wenxinyiyan, and the present invention is not limited thereto.

[0077] It can be understood that the preset text language enhancement model combined with the preset visual language model can extract accurate entities from each image, thereby obtaining accurate image-text pairs, laying the foundation for the subsequent alignment of the image text with the image mask.

[0078] Step 4120, input the initial text embedding vector of each image into a preset text language enhancement model to determine at least one enhanced text represented by the text feature of each image.

[0079] The preset text language enhancement model is used for accurate entity extraction.

[0080] Preferably, the preset text language enhancement model may be, for example, a (Chat Generative Pre-trained Transformer, ChatGPT) parser. It is understood that the preset text language enhancement model may also adopt other models, and the present invention is not limited to this.

[0081] It can be understood that the preset text language enhancement model combined with the preset visual language model can extract accurate entities from each image, thereby obtaining accurate image-text pairs, laying the foundation for the subsequent alignment of the image text with the image mask.

[0082] Step 4130, inputting at least one enhanced text represented by the text features of each image into the preset text encoder respectively to obtain at least one text embedding vector corresponding to each image.

[0083] The preset text encoder, for example, can be a CLIP model based on ConvNext. Specifically, the preset text encoder is constructed as a 16-layer Transformer, each layer has 768 neural units, and has 12 attention heads.

[0084] Step 420, input each image of the at least one image into the visual encoder preset in the feature extraction and mask generation model to obtain the initial pixel feature vector of each point; and input the initial pixel feature vector of each point and the predicted mask area vector corresponding to each point into the multi-scale feature adapter to obtain the pixel feature vector of each point.

[0085] Specifically, each image in the at least one image can be input into the preset visual encoder to obtain the initial pixel feature vector of each point; and the initial pixel feature vector of each point and the predicted mask region vector corresponding to each point are sequentially subjected to mask pooling and multi-scale feature adapter to obtain the pixel feature vector of each point. The expression corresponding to the pixel feature vector of each point is: i =Fv(P(x v ,m i ), where P() represents flooding pooling, Fv() represents multi-scale feature adapter, and x v Represents the initial pixel feature vector of each point, m i Represents the predicted mask region vector corresponding to each point.

[0086] Among them, the pre-trained multi-scale pixel decoder is used to enhance the visual features in each image extracted from the preset visual encoder, thereby reducing the inherent noise in the correspondence between the mask and the entity. The pre-trained multi-scale pixel decoder includes a feature pyramid network (FPN) architecture. First, a 6-layer multi-scale deformable transformer is used on the multi-scale features to aggregate contextual information. Then, the low-resolution feature map in the multi-scale pixel decoder is enlarged by 2 times and then combined with the corresponding resolution feature map from the backbone, which has been projected to match the channel dimension. The projection is achieved by a 1×1 convolution layer, followed by group normalization (GroupNorm). Subsequently, the merged features are further processed by an additional 3×3 convolution layer, supplemented by GroupNorm and ReLU activation. This process is applied iteratively, starting with a feature map that is 32 times smaller, until we obtain a final feature map that is 4 times smaller. In order to generate a pixel-by-pixel embedding, a single 1×1 convolution layer is applied at the end. Throughout the multi-scale pixel decoder, all feature maps maintain a consistent dimension of 256 channels.

[0087] Among them, the pre-trained multi-scale feature adapter is used to match the pre-trained multi-scale pixel decoder, and is used to process the feature maps of different scales output from the pre-trained multi-scale pixel decoder respectively, and refine and enhance the pixel features of various scales.

[0088] Specifically, the specific structure of the multi-scale feature adapter can refer to Figure 8 .

[0089] Step 430 , updating the parameters of the multi-scale feature adapter based on at least one text embedding vector corresponding to all images in the training sample set, the pixel feature vector of each point in the point grid, and the preset cosine similarity loss function.

[0090] In one embodiment, if Figure 5 As shown, the above step 430 includes:

[0091] Step 510, for each point, based on the arc cosine similarity between the pixel feature vector of each point and each text embedding vector corresponding to each image, determine a cost matrix between the pixel feature vector of each point and each text embedding vector corresponding to each image.

[0092] The expression corresponding to the cost matrix between the predicted mask region vector corresponding to each point and each text embedding vector corresponding to each image is: Among them, δ' i,j represents the arc cosine similarity between the pixel feature vector corresponding to the i-th point and the j-th text embedding vector corresponding to each image, r i Represents the pixel feature vector of the i-th point, t j represents the j-th text embedding vector corresponding to each image.

[0093] It can be understood that the pixel feature vector corresponding to the i-th point may include pixel feature vectors of multiple scales; therefore, the expression corresponding to the cost matrix between the predicted mask region vector corresponding to each point and each text embedding vector corresponding to each image can be: Among them, S represents the scale number, Δ s is the cost matrix of the s-th scale. Specifically, the cost matrix of the s-th scale Δ s Represents the cost matrix between the predicted mask region vector corresponding to each point and each text embedding vector corresponding to each image at a scale.

[0094] Step 520, based on the cost matrix between the pixel feature vector of each point and all text embedding vectors corresponding to each image, and a bipartite graph matching algorithm, determine the best matching pixel feature vector and text embedding vector corresponding to each point.

[0095] Step 530, input the best matching pixel feature vector and text embedding vector corresponding to each point into the preset cosine similarity loss function to obtain the loss function value corresponding to each point, and update the parameters of the multi-scale feature adapter based on the loss function value corresponding to each point.

[0096] Among them, the preset cosine similarity loss function is:

[0097] Among them, r i ' represents the best matching pixel feature vector corresponding to the i-th point, t k Represents the kth text embedding vector of the best match corresponding to the i-th point.

[0098] Figure 6 A framework schematic diagram of a training method for an image panoptic segmentation model based on open vocabulary segmentation provided by the present invention. For image-mask pairs, a branch of mask generation, including a visual cue encoder, a pixel decoder, and a mask decoder, is used to predict a set of binary masks of the input image. For image-text pairs, mask-text bipartite graph matching is used to utilize confidence pairs between predicted masks and entities in text descriptions. Afterwards, a multi-scale feature adapter is used to enhance the mask visual embedding, which is further aligned with the relevant entity embedding based on confidence pairs. Since the collected image-text pairs usually contain some text that does not match the image, the correspondence between the mask and the entity is incorrect. Therefore, a large visual-text model can be used to improve the quality of the text description, and combined with a ChatGPT-based parser for accurate entity extraction. For mask-text alignment, mask-text bipartite graph matching is introduced to utilize confidence pairs of predicted masks and entities. The multi-scale feature adapter aims to enhance the visual embedding of the predicted mask, which is further aligned with the text embedding of the corresponding entity. Since the mask and text are allowed to be unpaired in the present invention, multi-scale integration is introduced to improve the quality of visual embedding and effectively solve the inherent noise in the correspondence between mask and entity, thereby stabilizing the matching process.

[0099] like Figure 7 As shown, each mask decoder layer includes a cross attention layer. The cross attention layer is crucial because it ensures that the final pixel features are enriched with basic geometric information such as point coordinates and bounding boxes. In addition, in each decoder layer, a cross attention layer is used, which ensures that the final pixel feature vector has access to key geometric information (e.g., point coordinates and boxes). Finally, the visual cue embedding (query embedding vector) is fed to the layer for normalization and then processed by a multi-layer perceptron (MLP). These processed query embedding vectors are then dot-producted with the pixel feature vector to ultimately generate a predicted mask (predicted mask region vector). The mask decoder layer updates the visual cue embedding and pixel features through a cross attention layer. The self-attention layer is used to update the visual cue. At each attention layer, the position encoding is added to the pixel feature, and the entire original visual cue (including the position encoding) is added to the updated visual cue.

[0100] Figure 8 This is a schematic diagram of the framework of the multi-scale feature adapter provided by the present invention. Figure 8The Multi-Scale Feature Adapter shown aims to refine and enhance pixel features at various scales. The adapter consists of multiple Low-Level Adapters (LoRA Adapters)

[27] , each of which is tailored to update features at a specific scale. Each LoRA adapter contains two linear layers, denoted as A and B, with a non-linear activation function placed in between. The linear layers A and B are used to linearly transform the input data, while the activation function introduces non-linearity, allowing more complex relationships in the data to be modeled. Each adapter is responsible for processing features at a specific scale, suggesting a hierarchical approach to feature refinement. This modular design allows for focused and specialized processing of features based on their resolution and semantic complexity, which is particularly beneficial for dense prediction tasks.

[0101] For ease of understanding, the following experiments are combined to illustrate the effects that the present invention can achieve. It is mainly compared with the full supervision and weak supervision methods. For specific comparison results, please refer to Fig. 9. Among them, "COCO S.", "COCO P." and "COCO C." represent the COCO dataset, the panoramic dataset and the caption dataset, respectively. "IN 1K" represents the ImageNet-1K image dataset. Among them, the existing fully supervised methods involved can be referenced to: (Openvocabulary universal image segmentation with mask clip, MaskCLIP), (Open-vocabulary panoptic segmentation with text-to-image diffusion models, ODISE), (Open-vocabulary segmentation with single frozen convolutional clip, FC-CLIP), (Open-vocabulary panoptic segmentation with embedding modulation, OPSNet). The existing weakly supervised methods involved can be referenced to: (Cut and learn for unsupervised object detection and instance segmentation, CutLER+STEGO), (Unsupervised universal image segmentation, U2Seg). The experimental results include PQ, SQ and RQ of all datasets. Specifically, the model of the present invention is evaluated on the COCO, ADE20K and Cityscapes datasets with zero samples. The existing weakly supervised methods only use text supervision, which brings challenges to distinguishing different instances with the same semantic class. The present invention proposes panoramic segmentation based on open vocabulary segmentation under weak supervision. Compared with the unsupervised method, the method of the present invention is significantly better than U2Seg 1.9% PQ, 1.5% SQ and 4.4% RQ on the COCO dataset. Unfortunately, the image mask pairs used in the present invention contain multiple granularity masks, such as object masks and partial masks, which is different from the panoramic segmentation dataset. This difference leads to many false positive results, limiting the performance of the method of the present invention.

[0102] The following describes a training device for an image panoramic segmentation model based on open vocabulary segmentation and an image classification device provided by the present invention. The training device for an image panoramic segmentation model based on open vocabulary segmentation described below and the training method for an image panoramic segmentation model based on open vocabulary segmentation described above can refer to each other.

[0103] like Fig.10As shown, in one embodiment, a training device for an image panoptic segmentation model based on open vocabulary segmentation is provided. The training device for an image panoptic segmentation model based on open vocabulary segmentation may include:

[0104] The acquisition module 1010 is used to acquire a training sample set and a preset point grid; the training sample set includes at least one image and a pixel-level category label corresponding to each image in the at least one image; the preset point grid includes at least one point;

[0105] A determination module 1020 is used to determine a query embedding vector corresponding to each point in the preset point grid based on the preset point grid, the at least one image and the pixel-level category label corresponding to each image; wherein the query embedding vector corresponding to each point includes a position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in each image;

[0106] A generating module 1030, configured to generate a predicted mask region vector corresponding to each point based on the at least one image and the query embedding vector corresponding to each point, in combination with a pre-trained feature extraction and mask generation model;

[0107] The updating module 1040 is used to update the parameters of the multi-scale feature adapter based on the at least one image and the predicted mask area vector corresponding to each point, combined with a preset cosine similarity loss function; the multi-scale feature adapter and the multi-scale pixel decoder in the pre-trained feature extraction and mask generation model are used to enhance the visual embedding of the predicted mask; the preset cosine similarity loss function is used to realize the determination of the text embedding vector corresponding to the predicted mask area vector corresponding to each point from the at least one text embedding vector corresponding to each image.

[0108] The training device of the image panoramic segmentation model based on open vocabulary segmentation provided by the present invention provides a preset point grid and a pixel-level category label corresponding to each image, determines the query embedding vector of each point in the preset point grid, realizes the correspondence between the image pixel-level category label and the position embedding vector, and further combines the pre-trained feature extraction and mask generation model to determine the predicted mask area vector corresponding to each point, so as to realize the pixel-level mask division of the image. Combined with the preset cosine similarity loss function, the predicted mask area vector corresponding to each point is matched with the text embedding vector corresponding to each image, so as to realize the entity alignment in the text description corresponding to each image, thereby realizing the pixel-level segmentation of each image under the condition of decoupling the mask and the text. In this way, the decoupling of the mask and the text is realized, and the relationship between the corresponding image, the mask and the text does not need to be expended a lot of labor costs, so as to realize the use of independent image-mask and image-text pairs to liberate the strict correspondence between the mask and the text. In addition, while realizing the decoupling of the mask and the text, it is possible not only to realize the semantic segmentation of each image, but also to distinguish the instances of each image. Furthermore, in order to reduce the inherent noise in the correspondence between masks and entities, corresponding multi-scale feature adapters and multi-scale pixel decoders are added to enhance the visual embedding of the predicted masks.

[0109] In addition, the present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the training method of the image panoramic segmentation model based on open vocabulary segmentation provided by the present invention, and the training method of the image panoramic segmentation model based on open vocabulary segmentation includes: obtaining a training sample set and a preset point grid; the training sample set includes at least one image and a pixel-level category label corresponding to each image in the at least one image; the preset point grid includes at least one point; based on the preset point grid, the at least one image and the pixel-level category label corresponding to each image, determining a query embedding vector corresponding to each point in the preset point grid; wherein the query embedding vector corresponding to each point includes each The method comprises the following steps: generating a position embedding vector of each point, and generating at least one pixel-level true mask embedding vector corresponding to each point in each image; generating a predicted mask region vector corresponding to each point based on the at least one image and the query embedding vector corresponding to each point, in combination with a pre-trained feature extraction and mask generation model; updating parameters of a multi-scale feature adapter based on the at least one image and the predicted mask region vector corresponding to each point, in combination with a preset cosine similarity loss function; the multi-scale feature adapter and the multi-scale pixel decoder in the pre-trained feature extraction and mask generation model are used to enhance the visual embedding of the predicted mask; the preset cosine similarity loss function is used to realize the determination of a text embedding vector corresponding to the predicted mask region vector corresponding to each point from at least one text embedding vector corresponding to each image.

[0110] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the training method of the image panoramic segmentation model based on open vocabulary segmentation provided by the present invention, and the training method of the image panoramic segmentation model based on open vocabulary segmentation includes: obtaining a training sample set and a preset point grid; the training sample set includes at least one image, and a pixel-level category label corresponding to each image in the at least one image; the preset point grid includes at least one point; based on the preset point grid, the at least one image and the pixel-level category label corresponding to each image, determining the query embedding vector corresponding to each point in the preset point grid; wherein each point corresponds to The query embedding vector includes the position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in each image; based on the at least one image and the query embedding vector corresponding to each point, combined with a pre-trained feature extraction and mask generation model, a predicted mask area vector corresponding to each point is generated; based on the at least one image and the predicted mask area vector corresponding to each point, combined with a preset cosine similarity loss function, the parameters of the multi-scale feature adapter are updated; the multi-scale feature adapter and the multi-scale pixel decoder in the pre-trained feature extraction and mask generation model are used to enhance the visual embedding of the predicted mask; the preset cosine similarity loss function is used to realize the determination of the text embedding vector corresponding to the predicted mask area vector corresponding to each point from the at least one text embedding vector corresponding to each image.

[0111] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the training method of the image panoramic segmentation model based on open vocabulary segmentation provided by the present invention, the training method of the image panoramic segmentation model based on open vocabulary segmentation comprising: obtaining a training sample set and a preset point grid; the training sample set comprises at least one image, and a pixel-level category label corresponding to each image in the at least one image; the preset point grid comprises at least one point; based on the preset point grid, the at least one image and the pixel-level category label corresponding to each image, determining a query embedding vector corresponding to each point in the preset point grid; wherein the query embedding vector corresponding to each point comprises a position embedding of each point. vector, and at least one pixel-level true mask embedding vector corresponding to each point in each image; based on the at least one image and the query embedding vector corresponding to each point, combined with a pre-trained feature extraction and mask generation model, generate a predicted mask area vector corresponding to each point; based on the at least one image and the predicted mask area vector corresponding to each point, combined with a preset cosine similarity loss function, update the parameters of the multi-scale feature adapter; the multi-scale feature adapter and the multi-scale pixel decoder in the pre-trained feature extraction and mask generation model are used to enhance the visual embedding of the predicted mask; the preset cosine similarity loss function is used to realize the determination of the text embedding vector corresponding to the predicted mask area vector corresponding to each point from the at least one text embedding vector corresponding to each image.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] It can be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for an image panoptic segmentation model based on open vocabulary segmentation, characterized in that: The method comprises: Acquire a training sample set and a preset point grid; the training sample set includes at least one image and a pixel-level category label corresponding to each image in the at least one image; the preset point grid includes at least one point; Based on the preset point grid, the at least one image and the pixel-level category label corresponding to each image, determine the query embedding vector corresponding to each point in the preset point grid; wherein the query embedding vector corresponding to each point includes the position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in each image; Based on the at least one image and the query embedding vector corresponding to each point, combined with a pre-trained feature extraction and mask generation model, generate a predicted mask region vector corresponding to each point; Based on the at least one image and the predicted mask area vector corresponding to each point, combined with a preset cosine similarity loss function, the parameters of the multi-scale feature adapter are updated; the multi-scale feature adapter and the multi-scale pixel decoder in the pre-trained feature extraction and mask generation model are used to enhance the visual embedding of the predicted mask; the preset cosine similarity loss function is used to realize the determination of the text embedding vector corresponding to the predicted mask area vector corresponding to each point from the at least one text embedding vector corresponding to each image.

2. The method for training an image panoptic segmentation model based on open vocabulary segmentation according to claim 1, characterized in that: The pre-trained feature extraction and mask generation model further includes: a visual cue encoder and a preset visual encoder; the step of determining a query embedding vector corresponding to each point in the preset point grid based on the preset point grid, the at least one image, and a pixel-level category label corresponding to each image, comprising: For each point in the preset point grid, encoding each point into two position embedding vectors based on the visual cue encoder, and determining at least one pixel-level true mask embedding vector corresponding to each point in each image based on the pixel-level category label corresponding to each image and the visual cue encoder; Inputting each of the images into the preset visual encoder to determine a content embedding vector corresponding to the at least one true mask embedding vector; For each point in the preset point grid, the two position embedding vectors corresponding to each point and at least one pixel-level true mask embedding vector corresponding to each point in each image are spliced, and then combined with the preset query type corresponding to each point and the content embedding vector corresponding to the at least one true mask embedding vector to obtain the query embedding vector corresponding to each point.

3. The method for training an image panoptic segmentation model based on open vocabulary segmentation according to claim 1, characterized in that: Before generating a predicted mask area vector corresponding to each point by combining the pre-trained feature extraction and mask generation model, the method further includes: updating the pre-trained feature extraction and mask generation model based on the at least one image and the query embedding vector corresponding to each point, combining the mask generation and semantic segmentation loss function.

4. The method for training an image panoptic segmentation model based on open vocabulary segmentation as claimed in claim 3, characterized in that: The feature extraction and mask generation model further includes a visual cue encoder, a preset visual encoder and a mask decoder, and the pre-trained feature extraction and mask generation model is updated based on the at least one image and the query embedding vector corresponding to each point, combined with the mask generation and semantic segmentation loss function, including: For each image of the at least one image, extract features through the preset visual encoder and the multi-scale pixel decoder to obtain a multi-scale pixel feature vector of each image; Based on the query embedding vector corresponding to each point, the multi-scale pixel feature vector of each image and the mask generation and semantic segmentation loss function, the parameters of the visual cue encoder, mask decoder and multi-scale pixel decoder are updated to obtain a pre-trained feature extraction and mask generation model.

5. The method for training an image panoptic segmentation model based on open vocabulary segmentation according to claim 1, characterized in that: The updating of the parameters of the multi-scale feature adapter based on the at least one image and the predicted mask region vector corresponding to each point in combination with a preset cosine similarity loss function includes: For each image in the at least one image, inputting each image into a preset enhanced image text extraction model to determine at least one text embedding vector corresponding to each image; Input each image of the at least one image into a visual encoder preset in the feature extraction and mask generation model to obtain an initial pixel feature vector of each point; and input the initial pixel feature vector of each point and the predicted mask region vector corresponding to each point into a multi-scale feature adapter to obtain a pixel feature vector of each point; Based on at least one text embedding vector corresponding to all images in the training sample set, a pixel feature vector of each point in the point grid, and the preset cosine similarity loss function, parameters of the multi-scale feature adapter are updated.

6. The method for training an image panoptic segmentation model based on open vocabulary segmentation as claimed in claim 5, characterized in that: The preset enhanced image text extraction model includes: a preset visual language model, a preset text language enhancement model and a preset text encoder; the step of inputting each image into the preset enhanced image text extraction model to determine at least one text embedding vector corresponding to each image includes: inputting each image into the preset visual language model to determine an initial text embedding vector of each image; the initial text embedding vector of each image includes a description text feature representation of each image; Embedding the initial text vector of each image into a preset text language enhancement model to determine at least one enhanced text represented by the text feature of each image; The at least one enhanced text represented by the text feature of each image is respectively input into the preset text encoder to obtain at least one text embedding vector corresponding to each image.

7. The method for training an image panoptic segmentation model based on open vocabulary segmentation as claimed in claim 5, characterized in that: The updating of the parameters of the multi-scale feature adapter based on at least one text embedding vector corresponding to all images in the training sample set, the pixel feature vector of each point in the point grid and the preset cosine similarity loss function comprises: For each point, determining a cost matrix between the pixel feature vector of each point and each text embedding vector corresponding to each image based on the arc cosine similarity between the pixel feature vector of each point and each text embedding vector corresponding to each image; Based on the cost matrix between the pixel feature vector of each point and all text embedding vectors corresponding to each image, and a bipartite graph matching algorithm, determine the best matching pixel feature vector and text embedding vector corresponding to each point; The best matching pixel feature vector and text embedding vector corresponding to each point are input into the preset cosine similarity loss function to obtain the loss function value corresponding to each point, and the parameters of the multi-scale feature adapter are updated based on the loss function value corresponding to each point.

8. The method for training an image panoptic segmentation model based on open vocabulary segmentation as claimed in claim 7, characterized in that: The expression corresponding to the cost matrix between the predicted mask region vector corresponding to each point and each text embedding vector corresponding to each image is: Among them, δ′ i，j represents the arc cosine similarity between the pixel feature vector corresponding to the i-th point and the j-th text embedding vector corresponding to each image, r i Represents the pixel feature vector of the i-th point, t j represents the jth text embedding vector corresponding to each image; The preset cosine similarity loss function is: Among them, r i ′ represents the best matching pixel feature vector corresponding to the i-th point, t k Represents the kth text embedding vector of the best match corresponding to the i-th point.

9. A training device for an image panoptic segmentation model based on open vocabulary segmentation, characterized in that: The device comprises: An acquisition module, used to acquire a training sample set and a preset point grid; the training sample set includes at least one image and a pixel-level category label corresponding to each image in the at least one image; the preset point grid includes at least one point; A determination module, configured to determine a query embedding vector corresponding to each point in the preset point grid based on the preset point grid, the at least one image, and a pixel-level category label corresponding to each image; wherein the query embedding vector corresponding to each point includes a position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in each image; A generation module, configured to generate a predicted mask region vector corresponding to each point based on the at least one image and the query embedding vector corresponding to each point, in combination with a pre-trained feature extraction and mask generation model; An updating module is used to update the parameters of the multi-scale feature adapter based on the at least one image and the predicted mask area vector corresponding to each point, combined with a preset cosine similarity loss function; the multi-scale feature adapter and the multi-scale pixel decoder in the pre-trained feature extraction and mask generation model are used to enhance the visual embedding of the predicted mask; the preset cosine similarity loss function is used to realize the determination of the text embedding vector corresponding to the predicted mask area vector corresponding to each point from the at least one text embedding vector corresponding to each image.

10. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the method for training an image panoramic segmentation model based on open vocabulary segmentation as described in any one of claims 1 to 8.

11. A storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for training an image panoramic segmentation model based on open vocabulary segmentation as described in any one of claims 1 to 8.

Images