Visual localization and referent segmentation method, system, device and storage medium based on mask referent modeling
By using mask-based reference modeling methods in visual positioning and reference segmentation tasks, unifying visual and language feature spaces, the problems of inconsistency and complexity of visual and language features in the prior art are solved, and task performance is improved and model efficiency.
Patent Information
- Application Number
- CN202411220270.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The existing visual positioning and reference segmentation techniques rely on independent coding of visual and language features, resulting in inconsistency, increasing computational burden and implementation complexity, making it difficult to effectively capture the fine-grained reference relationship between images and text, limiting the Task performance.
Using a method based on mask reference modeling, pre-training is performed through a multimodal encoder, using a dynamic image masking strategy and text masking strategy of reference perception, unifying visual and language feature space, and learning general reference capabilities. , avoid dependence on complex encoder/decoder architectures and early interaction technologies.
It improves the performance of visual positioning and reference segmentation tasks, simplifies the model architecture, reduces the computational burden and implementation complexity, and enhances the model's understanding of the reference relationship between images and text.
Smart Images

Figure CN118734091B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to a method, system, device and storage medium for visual localization and representative segmentation based on mask representative modeling. Background Art
[0002] Currently, visual localization and referential segmentation techniques aim to locate the area referred to by the query text in the image, including referential expression understanding (REC), phrase localization (PG), and referential expression image segmentation (RES). In these task steps, REC and PG usually use rectangular bounding boxes to locate the target area, while referential expression image segmentation uses more sophisticated segmentation masks to represent the target area. These tasks require the model to understand the semantics of the query text and locate or segment specific areas accordingly. Therefore, visual localization is a multimodal interaction and alignment task that is highly dependent on visual and language features.
[0003] However, existing visual localization and representative segmentation techniques usually rely on independent encoding of visual and linguistic features, which leads to inconsistency between visual and linguistic features. In addition, in order to achieve effective multimodal fusion, existing technologies often require complex fusion encoder / decoder architectures based on multiple Transformers and multiple early interaction techniques. Although this architecture is powerful, it also increases the computational burden and implementation complexity, making it difficult for existing masked visual text modeling methods to effectively capture the fine-grained representative relationship between images and texts, limiting their performance in visual localization and representative segmentation tasks. Summary of the invention
[0004] The embodiments of the present application provide a visual localization and coreference segmentation method, system, device and storage medium based on mask coreference modeling, which can pre-train a unified and modally shared multimodal encoder based on aggregated visual words and aggregated text words, so that the model can learn general coreference capabilities and unify the visual and language feature spaces, thereby enabling the model to perform downstream fine-tuning for visual localization and coreference segmentation tasks to obtain high-precision localization or coreference segmentation results, avoiding the model's dependence on interactive technologies such as encoder / decoder architectures, and improving the model's performance in visual localization and coreference segmentation tasks.
[0005] To achieve the above-mentioned objectives, the first aspect of an embodiment of the present application provides a visual localization and reference segmentation method based on mask reference modeling, including: obtaining a first image and a first text for training; obtaining visual words, aggregated visual words, text words, and aggregated text words of the first image and the first text through a multimodal encoder, and performing the following pre-training process on the multimodal encoder: applying a reference-aware dynamic image masking strategy, masking the image block of the first image based on the image area referred to by the first text, and reconstructing the masked image block and the visual target association score corresponding to the image block based on the masked visual words and the aggregated text words of the first image to achieve reference for the multimodal encoder. Pre-training of the representative-aware masked image modeling part; applying a reference-aware text masking strategy to mask the text words of the first text based on the reference subject of the first text, and reconstructing the masked text words and the semantic target association scores corresponding to the text words according to the text words and the aggregated visual words of the masked first text, so as to achieve pre-training of the reference-aware masked text modeling part of the multimodal encoder; obtaining a second image and a second text to be processed, inputting the second image and the second text into the pre-trained multimodal encoder, and fine-tuning the multimodal encoder based on the visual localization task head or the reference segmentation task head to obtain the visual localization task result or the reference segmentation task result.
[0006] In some embodiments, the method of obtaining visual words, aggregated visual words, text words, and aggregated text words of the first image and the first text through a multimodal encoder includes: dividing the first image into multiple image blocks, and encoding each of the image blocks into visual words through the multimodal encoder; dividing the first text into multiple text words, and encoding each of the text words into text words through the multimodal encoder; outputting aggregated visual words and aggregated text words corresponding to multiple visual words and text words through the self-attention mechanism of the multimodal encoder; wherein the aggregated visual words represent the overall visual features corresponding to the first text, and the aggregated text words represent the overall text features corresponding to the first image.
[0007] In some embodiments, the application of the reference-aware dynamic image masking strategy masks the image blocks of the first image based on the image area referred to by the first text, including: obtaining the image area referred to by the first text in the first image; applying the reference-aware dynamic image masking strategy to block-mask a plurality of the image blocks located within the image area with a first masking rate, and randomly masking a plurality of the image blocks located outside the image area with a second masking rate, wherein the first masking rate is greater than the second masking rate.
[0008] In some embodiments, the reconstructing the masked image block and the visual target association score corresponding to the image block according to the visual word-grams and the aggregated text word-grams of the masked first image includes: encoding the masked first image through the multimodal encoder to obtain visual word-grams corresponding to multiple image blocks in the masked first image; performing dot product operations on the multiple visual word-grams and the aggregated text word-grams to obtain visual association word-grams between each image block and the reference image area corresponding to the first text; reconstructing the multiple masked image blocks in the first image according to the multiple visual association word-grams to obtain reconstructed visual word-grams; reconstructing the visual target association scores corresponding to the image blocks according to the multiple visual association word-grams to obtain reconstructed visual target association scores; calculating the visual modality loss function according to the reconstructed visual word-grams and the reconstructed visual target association scores, and optimizing the reference-aware masked image modeling part of the multimodal encoder based on the visual modality loss function; wherein the visual target association score represents the positional relationship between the image block and the image area referred to by the first text.
[0009] In some embodiments, the visual target association score corresponding to the image block is reconstructed according to the multiple visually associated words to obtain a reconstructed visual target association score, including: reconstructing the horizontal distance mask, vertical distance mask, width ratio mask and height ratio mask between each of the image blocks and the referenced image area according to the visually associated words; and obtaining the reconstructed visual target association score according to the horizontal distance mask, the vertical distance mask, the width ratio mask and the height ratio mask.
[0010] In some embodiments, the masking of text words of the first text based on the reference subject of the first text includes: determining subject text words from the first text according to the reference subject of the first text; preferentially masking the subject text words based on a target masking rate corresponding to the first text, and then randomly masking the remaining non-subject text words.
[0011] In some embodiments, the reconstruction of masked text words and semantic target association scores corresponding to the text words based on the text words and the aggregated visual words of the masked first text includes: encoding the masked first text through the multimodal encoder to obtain text words corresponding to multiple text words in the masked first text; performing dot product operation on the multiple text words and the aggregated visual words to obtain text association words between each text word and the reference text area corresponding to the first image; reconstructing multiple masked text words in the first text based on the multiple text association words to obtain reconstructed text words; reconstructing the text target association scores corresponding to the text words based on the multiple text association words to obtain reconstructed text target association scores; calculating a text modality loss function based on the reconstructed text words and the reconstructed text target association scores, and optimizing the reference-aware mask image modeling part of the multimodal encoder based on the text modality loss function; wherein the text target association score represents the correlation between the text word and the text target referred to by the first image.
[0012] In some embodiments, the text target association score corresponding to the text word is reconstructed according to the multiple text-associated word-grams to obtain a reconstructed text target association score, including: calculating a first similarity between the text word-gram of each text word and the aggregated visual word-gram according to the multiple text-associated word-grams, and calculating a second similarity between the text word-gram of each text word and the aggregated visual word-gram referring to the image area referred to by the first text; and obtaining the reconstructed text target association score according to the first similarity and the second similarity.
[0013] In some embodiments, the multimodal encoder is fine-tuned based on the head position of the visual localization task to obtain the visual localization task result, including: obtaining the target visual word element, target text word element, target aggregate visual word element and target aggregate text word element of the second image and the second text based on the multimodal encoder; obtaining the similarity mask between the image block of the second image and the referential target of the second text according to the target visual word element, the target text word element, the target aggregate visual word element and the target aggregate text word element; cross-modally aligning multiple target visual word elements based on the similarity mask, and summing the aligned target visual word elements to obtain a compact visual word element; performing bounding box regression prediction based on the compact visual word element based on a multi-layer perceptron to obtain a prediction mask; applying the prediction mask to the second image to obtain the task result of visual localization.
[0014] In some embodiments, the multimodal encoder is fine-tuned based on the reference segmentation task head to obtain the reference segmentation task result, including: obtaining the target visual word-grams, target text word-grams, target aggregated visual word-grams and target aggregated text word-grams of the second image and the second text based on the multimodal encoder, and upsampling the target visual word-grams using multi-layer deconvolution operations to obtain high-resolution visual word-grams; performing cosine similarity calculation on the high-resolution visual word-grams and the target aggregated text word-grams to obtain a similarity mask; adjusting the similarity mask to the target resolution through bilinear interpolation to obtain a prediction mask; applying the prediction mask to the second image to obtain the task result of reference segmentation.
[0015] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application provides a visual localization and reference segmentation system based on mask reference modeling, and the system is used to implement the visual localization and reference segmentation method based on mask reference modeling as described in any one of the first aspects.
[0016] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application provides an electronic device, comprising: at least one processor; at least one memory for storing at least one program; when at least one of the programs is executed by at least one of the processors, a visual positioning and reference segmentation method based on mask reference modeling as described in any one of the first aspects is implemented.
[0017] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the visual localization and reference segmentation method based on mask reference modeling as described in any one of the fourth aspects.
[0018] The embodiments of the present application provide a method, system, device and storage medium for visual localization and reference segmentation based on masked reference modeling, which can obtain a first image and a first text for training; obtain visual word-grams, aggregated visual word-grams, text word-grams and aggregated text word-grams of the first image and the first text through a multimodal encoder; apply a reference-aware dynamic image masking strategy to mask the image block of the first image based on the image area referred to by the first text, and reconstruct the masked image block and the visual target association score corresponding to the image block according to the masked visual word-grams and aggregated text word-grams of the first image, so as to achieve pre-training of the reference-aware masked image modeling part of the multimodal encoder; apply a reference-aware text masking strategy to mask the text words of the first text based on the reference subject of the first text, and reconstruct the masked text words and the semantic target association score corresponding to the text words according to the masked text word-grams and aggregated visual word-grams of the first text, so as to achieve pre-training of the reference-aware masked text modeling part of the multimodal encoder; Obtain a second image and a second text to be processed, and input the second image and the second text into a pre-trained multimodal encoder; fine-tune the multimodal encoder based on a visual localization task head or a reference segmentation task head to obtain a visual localization task result or a reference segmentation task result; it can be understood that the method of the present application reconstructs the encoded mask image block or mask text word by pre-training the multimodal encoder to optimize the reference-aware mask image modeling part and the mask text modeling part in the multimodal encoder, so that the multimodal encoder can learn the common reference ability between images and texts to eliminate the inconsistency between visual and language features, and then the pre-trained model can be configured with a corresponding task head for the visual localization task or the reference segmentation task, and corresponding downstream fine-tuning can be performed to obtain high-precision localization or reference segmentation results, which can effectively avoid dependence on complex encoder / decoder architectures and early interaction technologies, thereby simplifying the overall architecture of the model and reducing the computational burden and implementation complexity of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 A schematic diagram of a single-tower architecture of a visual positioning and representative segmentation method based on mask representative modeling provided in an embodiment of the present application;
[0020] Figure 2 A flowchart of a visual localization and representative segmentation method based on mask representative modeling provided in an embodiment of the present application;
[0021] Figure 3 A flowchart of obtaining aggregated visual word-grams and aggregated text word-grams provided in an embodiment of the present application;
[0022] Figure 4 A schematic diagram of constructing mask reference modeling provided by an embodiment of the present application;
[0023] Figure 5 A flowchart of obtaining visually associated word units and image mask position information provided by an embodiment of the present application;
[0024] Figure 6 A schematic diagram of a dynamic image mask provided by an embodiment of the present application;
[0025] Figure 7 A flowchart of obtaining a task result of visual positioning or reference segmentation provided in an embodiment of the present application;
[0026] Figure 8 A schematic diagram of an embodiment of the present application used to indicate an expression understanding task;
[0027] Fig. 9 A schematic diagram used in the present application to refer to a segmentation and understanding task provided in an embodiment of the present application;
[0028] Fig.10 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0030] In some embodiments, although the functional modules are divided in the schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the schematic diagram or the order in the flowchart. The terms first, second, etc. in the specification, claims, and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0031] In addition, unless otherwise clearly specified and limited, the term "connected" should be understood in a broad sense. For example, it can be a fixed connection or a movable connection, a detachable connection or a non-detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or the two can communicate with each other; it can be a direct connection or an indirect connection through an intermediate medium.
[0032] In the description of the embodiments of the present application, the description of reference terms "one embodiment / implementation", "another embodiment / implementation" or "certain embodiments / implementations", "in the above embodiments / implementations", etc. means that the specific features, structures, materials or characteristics described in combination with the implementation or example are included in at least two embodiments or implementations disclosed in the present application. In the disclosure of this application, the schematic representation of the above terms does not necessarily refer to the same embodiment or implementation. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that in the flowchart.
[0033] Currently, visual localization and coreference segmentation techniques aim to locate the region referred to by the query text in the image, including referential expression understanding (REC), phrase localization (PG), and referential expression image segmentation. In these task steps, REC and PG usually use rectangular bounding boxes to locate the target region, while referential expression image segmentation uses more refined segmentation masks to represent the target region. These tasks require the model to understand the semantics of the query text and locate or segment specific regions accordingly. Therefore, visual localization is a multimodal interaction and alignment task that is highly dependent on visual and language features. However, existing visual localization and coreference segmentation techniques usually rely on independent encoding of visual and language features, which leads to inconsistency between visual and language features. In addition, in order to achieve effective multimodal fusion, existing technologies often require complex fusion encoder / decoder architectures based on multiple Transformers and multiple early interaction techniques. Although this architecture is powerful, it also increases the computational burden and implementation complexity, making it difficult for existing masked visual text modeling methods to effectively capture the fine-grained coreference relationship between images and texts, limiting their performance in visual localization and coreference segmentation tasks.
[0034] It is understandable that in order to overcome the limitations of the existing technology, especially to improve the model architecture and efficiency problems in visual localization and reference segmentation tasks, in the existing technology, visual localization and reference segmentation work is usually limited to the independent encoding of visual and language features, which leads to the inconsistency between visual and language features. In addition, the current masked visual text modeling (MVLM) cannot effectively capture the fine-grained reference relationship between image and text in the reference task. The present application provides a method, system, device and storage medium for visual localization and reference segmentation based on masked reference modeling, the method comprising: pre-training a multimodal encoder through a first image and a first text, wherein the construction process of masked reference modeling comprises: segmenting the first image into multiple image blocks, and encoding each image block into a visual word element through a multimodal encoder; applying a reference-aware dynamic image masking strategy, masking the image block of the first image based on the image area referred to by the first text, so as to perform masked image modeling; based on aggregated visual word elements and reference text fragments corresponding to the first image, obtaining reference text association information and text mask position information of multiple text words in the first text, so as to perform masked reference modeling, thereby unifying the visual and language feature spaces, and directly regressing the reference results in combination with masked reference modeling.
[0035] The embodiments of the present application are further described below in conjunction with the accompanying drawings.
[0036] refer to Figure 1 , Figure 1 A single-tower architecture schematic diagram of a visual localization and coreference segmentation method based on mask coreference modeling provided for an embodiment of the present application; A first aspect of an embodiment of the present application provides a visual localization and coreference segmentation method based on mask coreference modeling, comprising: obtaining a first image and a first text for training; obtaining visual words, aggregated visual words, text words, and aggregated text words of the first image and the first text through a multimodal encoder, and pre-training the multimodal encoder; obtaining a second image and a second text to be processed, and inputting the second image and the second text into the multimodal encoder to obtain a task result of visual localization or coreference segmentation.
[0037] It can be understood that the multimodal encoder in the present application scheme is a single-tower architecture based on masked reference modeling (MRefM), which can model visual and language features in a unified feature space, thereby avoiding the use of complex interaction modules, bulky fusion Transformer / decoder and special positioning tokens. Among them, the multimodal encoder is a shared modality Transformer encoder, which can process image and text data. The goal of pre-training the encoder is to obtain masked reference modeling (Masked-Referring-Modeling, MRefM), which includes referring-aware mask image modeling (RefMIM for short, referring MIM) and referring-aware mask text modeling (RefMLM for short, referring MLM). These two modeling methods can help the model better understand and learn the referring relationship between images and texts, as well as the correspondence between them.
[0038] In some embodiments, the first image is an image sample for training; the first text is a text description paired with the first image, which is used to refer to a specific area in the image; the multimodal encoder is a neural network model that can process data from different modalities (such as images and texts) and can fuse data from different modalities together through a self-attention mechanism; masked reference modeling is a modeling method that learns the reference relationship between images and texts through masking technology; the visual localization task represents the task of finding the specific area referred to by the text description in the image and marking it with a bounding box, and the reference segmentation task represents locating the area referred to by the text description and segmenting the area.
[0039] Among them, the reference-aware masked image modeling part in the multimodal encoder can perform image masking, and try to reconstruct this part of the image in combination with the text description to learn the association between the image and the text; the reference-aware masked text modeling part in the multimodal encoder can perform text masking, and try to reconstruct this part of the text in combination with the image to learn the association between the image and the text, so that the present application can unify the visual and language feature spaces through a single-tower architecture, simplify the overall architecture, and thus reduce the computational burden and implementation complexity. In practical applications, the user can provide a new image and text pair (a second image and a second text), and these data will be input into the pre-trained multimodal encoder to perform visual localization or reference segmentation tasks to obtain visual localization or reference segmentation task results. It can be understood that the pre-trained multimodal encoder already has reference capabilities, so there is no need to modify the structure of the model encoder itself like other models. It only needs to connect the corresponding task head to the outside of the encoder to fine-tune the corresponding task, so as to obtain high-precision positioning capabilities or reference segmentation capabilities.
[0040] refer to Figure 2 , Figure 2 A flowchart of a method for visual localization and representative segmentation based on mask representative modeling provided in an embodiment of the present application; the process of pre-training a multimodal encoder and obtaining a task result includes but is not limited to the following steps:
[0041] Step S210, applying a reference-aware dynamic image masking strategy, masking an image block of the first image based on the image region referenced by the first text, and reconstructing the masked image block and the visual target association score corresponding to the image block according to the masked visual word-grams and the aggregated text word-grams of the first image, so as to implement pre-training of the reference-aware masked image modeling part of the multimodal encoder;
[0042] Step S220, applying a reference-aware text masking strategy, masking text words of the first text based on the reference subject of the first text, and reconstructing the masked text words and the semantic target association scores corresponding to the text words according to the masked text words and the aggregated visual words of the first text, so as to achieve pre-training of the reference-aware masked text modeling part of the multimodal encoder;
[0043] Step S230, obtain the second image and the second text to be processed, input the second image and the second text into the pre-trained multimodal encoder, and fine-tune the multimodal encoder based on the visual localization task head or the reference segmentation task head to obtain the visual localization task result or the reference segmentation task result.
[0044] Among them, aggregating visual word-grams includes: dividing the first image into multiple image blocks, and converting each image block into a visual word-gram, which are then input into the multimodal encoder and processed through a self-attention mechanism to obtain a comprehensive visual representation, namely, aggregated visual word-grams: Aggregating text word-grams: dividing the first text into multiple text words, and encoding each text word into a text word-gram through a multimodal encoder, which are also input into the multimodal encoder and processed through a self-attention mechanism to obtain a comprehensive text representation, namely, aggregated text word-grams. It can be imagined that a single "word-gram" in this application represents a single token.
[0045] In some embodiments, visual words, aggregated visual words, text words, and aggregated text words are the output results of the multimodal encoder. The present application can perform mask image modeling and mask reference modeling based on each output result of the multimodal encoder during the training process to enhance the model's understanding of the referential relationship between images and texts, thereby achieving better visual positioning and referential segmentation tasks.
[0046] It can be understood that the method of the present application adopts a unified and modality-sharing multimodal encoder, and obtains mask reference modeling through the pre-training process corresponding to steps S210 to S230. The reference-aware mask image modeling and the reference-aware mask text modeling in the mask reference modeling can effectively capture the fine-grained reference relationship between image and text, realize the unification of visual and language features, eliminate the inconsistency between visual and language features in traditional methods, and enhance the model's ability to understand the reference relationship.
[0047] In some embodiments, corresponding to step S220, by performing a dot product operation with the aggregated text words, the degree of association between each image block and the image area referred to by the text can be obtained, and then based on the association information, it can be determined which image blocks need to be masked. The application of a reference-aware dynamic image masking strategy can enable the present application to be adjusted according to the content of the text description and the image area referred to by the text, so as to better focus on the image area referred to by the text, wherein visually associated words are used to characterize the degree of association between each image block and the image area referred to by the text, which can be obtained through a dot product operation, and the application of a reference-aware dynamic image masking strategy is used to determine which image blocks should be masked and how to mask them.
[0048] refer to Figure 3 , Figure 3 A flowchart of obtaining aggregated visual word-grams and aggregated text word-grams is provided for an embodiment of the present application; in some embodiments, a first image is segmented into a plurality of image blocks, and each image block is encoded into a visual word-gram by a multimodal encoder, including but not limited to the following steps:
[0049] Step S310, dividing the first image into a plurality of image blocks, and encoding each image block into a visual word unit by a multimodal encoder;
[0050] Step S320, dividing the first text into a plurality of text words, and encoding each text word into a text word unit by a multimodal encoder;
[0051] Step S330, outputting aggregated visual words and aggregated text words corresponding to multiple visual words and text words through the self-attention mechanism of the multimodal encoder; wherein the aggregated visual words represent the overall visual features corresponding to the first text, and the aggregated text words represent the overall text features corresponding to the first image.
[0052] Wherein, corresponding to step S310, the input first image is first divided into multiple smaller image blocks (patches), and each image block is then mapped into a visual word unit through an encoding process, which can be implemented with the help of a convolutional neural network (CNN) or other image processing technology, so that the entire image is represented as a series of visual word units; further corresponding to step S320, the input first text is decomposed into a series of text words through a word segmenter. Each text word is then mapped into a text word unit, and this mapping process can be completed by word embedding technology, such as pre-trained models such as Word2Vec, GloVe or BERT; further, corresponding to step S330, all previously generated visual word units and text word units are input into a multimodal encoder together, and the multimodal encoder can capture the complex relationship between these data through a self-attention mechanism. The output aggregated visual word unit represents the comprehensive understanding of the overall visual features of the image by the entire text, reflecting the association between the text description and the image content, and the aggregated text word unit is the comprehensive understanding of the overall text features of the text by the entire image, reflecting the association between the image content and the text description.
[0053] It can be understood that the pre-training of masked reference modeling is to enable the model to learn the fine-grained reference relationship between image and text through the process of masking and reconstruction, and then optimize the visual word-grams, text word-grams, aggregated visual word-grams and aggregated text word-grams output by the multimodal encoder, so that the aggregated visual word-grams can correspond to the reference information in the text description, and the aggregated text word-grams can correspond to the image area referred to by the text, so that the model can learn richer cross-modal feature representations, thereby improving the performance in visual localization and reference segmentation tasks.
[0054] refer to Figure 4 , Figure 4A schematic diagram of constructing masked reference modeling provided for an embodiment of the present application; it can be understood that the present application can adopt a multimodal and modality-sharing Transformer as the underlying backbone network based on the currently existing BEiT-3 method, and the Transformer is the multimodal encoder in the present application. The multimodal encoder can use masked image modeling (MIM), masked text modeling (MLM) and multi-view masked text modeling (MVLM) to process images, texts and image-text pairs, and realize the learning of universal representations through the mixed expert (MoE) head and shared multi-head self-attention mechanism, and perform original masked image modeling (Vanilla-MaskImage-Modeling) and original masked text modeling (Vanilla-Mask-Language-Modeling). Then, the present application can perform reference-aware masked image modeling (RefMIM) and reference-aware masked text modeling (RefMLM) on the basis of the original masked image modeling and the original masked text modeling to realize the construction process of masked reference modeling.
[0055] In some embodiments, during the modeling of the original mask image, the input image is first converted into a series of visual patches through convolutional projection, where and Represents the width and height of the image respectively. is the block size, is the hidden dimension in the unified feature space. Then, according to a specific masking strategy, a certain proportion of image blocks are selected for masking, and the position of the mask is given by Definition. The masking operation is implemented by replacing the image patch embedding at the mask position with a shared learnable mask embedding, i.e., for all , embed the image block into Replace with mask embedding In addition, a learnable special embedding token is added at the beginning of the sequence as category embeddings. These embeddings are then fed into a multimodal encoder. To recover the masked image patches, this application can use a MIM head consisting of a multi-layer perceptron (MLP) and a Softmax classifier for prediction. The visual tokens of the original image can be generated using the image tokenizer proposed in BEiTv2, which provides a self-supervisory signal for MIM. This application represents the visual tokens of the original image as , Represents the set of pre-trained images, and then the training loss of MIM can be defined as:
[0056] ;
[0057] The training loss of MIM can be reduced during the modeling of the original mask image As the goal, iterative training is performed to obtain the original mask image model.
[0058] In some embodiments, the original masked text modeling involves tokenizing the input text words through a sentence tokenizer and mapping each word to a corresponding word embedding. The corresponding vocabulary size can be 64010, where , represents a real number matrix with M rows and D columns, where M is the length of the text word after word segmentation. Then, following BEiT-3, this application uses a fixed mask ratio The text tokens are randomly masked. The masked positions are called For all , embed the masked word into token Replaced with shared learnable mask embeddings Similarly, special embeddings are added at the beginning and end of the sequence to learn special tokens and a sequence-terminating token , these embeddings are fed into a multimodal encoder. The multimodal encoder in this application can use a monotonic projection layer as an MLM head to predict the original text word embeddings Text token at mask position, original text word embedding Can be generated by a text tokenizer. Further, given the original text token is recorded as , represents the pre-trained text words, then the training loss of MLM can be defined as:
[0059] ;
[0060] The training loss of MLM can be reduced during the original mask text modeling process As the goal, iterative training is performed to obtain the original mask language model.
[0061] Based on the above original mask image model and original mask language model, the present application can be performed as follows: Figure 4In the process of constructing masked reference modeling, the input image is first segmented into a series of image blocks and converted into visual word units through convolutional projection. According to a specific reference-aware masking strategy, a part of the image blocks are selected for masking, and the embeddings of these masked image blocks are replaced with shared learnable masked embeddings; the input text is segmented by a word segmenter, and each word is mapped to the corresponding word embedding, and according to the reference-aware masking strategy, a part of the text word units are selected for masking and replaced with shared learnable masked embeddings; the processed image and text are input into the multimodal encoder, which can obtain the aggregated visual word unit token , aggregate text word token , and then predict the masked image blocks for the image part through the above-mentioned reference MIM head. The reference MIM head can be composed of a multi-layer perceptron (MLP) and a Softmax classifier, which is used to restore the masked image blocks, and through the visual words predicted by the reference MIM head, the model can reconstruct the visual content related to the text description in the image, thereby learning the association between the image and the text; conversely, through the reference MLM head for the text part, it is responsible for predicting the masked text words. This reference MLM head is composed of an MLP, which is used to restore the masked text words, and through the text words predicted by the reference MLM head, the model can reconstruct the semantic content related to the image in the text, thereby learning the association between the text and the image, so that the model can learn the reference relationship between the image and the text in the pre-training stage.
[0062] refer to Figure 5 , Figure 5 A flowchart for obtaining visually associated word-grams and image mask position information provided in an embodiment of the present application; in some embodiments, the mask image block and the visual target association score corresponding to the image block are reconstructed according to the visual word-grams and aggregated text word-grams of the masked first image, including but not limited to the following steps:
[0063] Step S510, encoding the masked first image by a multimodal encoder to obtain visual word units corresponding to a plurality of image blocks in the masked first image;
[0064] Step S520, performing a dot product operation on the multiple visual word-grams and the aggregated text word-grams to obtain a visually associated word-gram between each image block and the referring image region corresponding to the first text;
[0065] Step S530, reconstructing a plurality of mask image blocks in the first image according to a plurality of visually associated word-grams to obtain reconstructed visual word-grams;
[0066] Step S540, reconstructing the visual target relevance score corresponding to the image block according to the plurality of visually associated word-grams to obtain a reconstructed visual target relevance score;
[0067] Step S550, calculating a visual modality loss function according to the reconstructed visual word unit and the reconstructed visual target association score, and optimizing the reference-aware mask image modeling part of the multimodal encoder based on the visual modality loss function;
[0068] The visual target association score represents the positional relationship between the image block and the image region referred to by the first text.
[0069] In some embodiments, the present application encodes the masked first image through a multimodal encoder to obtain visual word elements corresponding to multiple image blocks in the masked first image, and then each image block can be mapped to a visual word element, and then these visual word elements are dot-producted with aggregated text words to evaluate the degree of association between each image block and the image area referred to in the text description to calculate the visual association word element between each image block and the image area referred to by the text; further, multiple masked image blocks in the first image are reconstructed according to multiple visual association words to obtain reconstructed visual word elements, and visual target association scores corresponding to the image blocks are reconstructed according to multiple visual association words to obtain reconstructed visual target association scores, and by comparing the reconstructed visual word elements with the visual word elements corresponding to the original image, and comparing the reconstructed visual target association scores with the visual target association scores corresponding to the original image, a visual modality loss function technology is implemented to optimize the reference-aware mask image modeling part of the multimodal encoder according to the corresponding loss value, wherein the visual word element corresponding to the original image and the visual target association score can be obtained by the image tokenizer proposed in BEiTv2. The word segmenter provides supervision for the self-supervised learning of MIM.
[0070] It can be understood that in the mask image modeling of reference perception, the present application not only needs to reconstruct the visual word itself, but also needs to reconstruct the association information between the visual word and the image region referred to by the text in the text description. To this end, the present application reconstructs the visual target association score corresponding to the image block according to multiple visual association words to obtain a reconstructed visual target association score, including: reconstructing the horizontal distance mask, vertical distance mask, width ratio mask and height ratio mask of each image block and the referred image area according to the visual association word; obtaining the reconstructed visual target association score according to the horizontal distance mask, vertical distance mask, width ratio mask and height ratio mask. Specifically, the present application can introduce a visual target association head composed of a three-layer perceptron (MLP) to predict the association score between each image block and the image area referred to by the text. This score represents the distance and ratio information between each image block and the image area referred to by the text. The standardized distance represents the standardized horizontal and vertical distance between the center coordinates of each image block and the center coordinates of the image area referred to by the text. The ratio information represents the ratio of the size of each image block to the size of the image area referred to by the text. This information helps the model understand the geometric relationship between the image block and the image area referred to by the text, and then the training loss of the reference MIM can be defined as:
[0071] ;
[0072] Where ⊙ represents the dot product operation. It can be understood that in referring to MIM, this application does not use a single-modal visual token like the previous method, but proposes to use an aggregated text token. The visual tokens that perform dot product operations are reconstructed, indicating that MIM not only needs to reconstruct the relevant content of the model , we also need to reconstruct the score of visual target association , this application uses a visual object association head composed of a three-layer perceptron (MLP) to predict the score. The score represents the token of each image patch The image area referred to by the text The distance between Represents the center coordinates, width, and height of the image region referred to by the text. It contains four masks: horizontal distance mask, vertical distance mask, width ratio mask, and height ratio mask, i.e. x-, y-, w-, h-masks , which represent the standardized horizontal and vertical distances of each patch from the reference center, namely , and the ratio of the width and height of each patch to the image area referred to by the text, that is, ,in Represents the center coordinates of each patch (image block), where it is worth noting that the text in the above process refers to the image area is the image area pre-set for the original image, and the image area referred to by the text in the above process can be Calculate the correlation score.
[0073] Through the above steps and the definition of the training loss of the reference MIM, the reference-aware mask image modeling proposed in this application can more effectively learn the reference relationship between images and texts, thereby improving the performance in visual localization and reference segmentation tasks. It is worth noting that the training loss function of the reference MIM compares the reconstructed word units and the associated scores to achieve pre-training of the reference-aware mask image modeling part of the multimodal encoder.
[0074] refer to Figure 6 , Figure 6 A schematic diagram of a dynamic image mask provided for an embodiment of the present application; in some embodiments, a reference-aware dynamic image mask strategy is applied, and the reference-aware dynamic image mask strategy is applied to mask the image blocks of the first image based on the image area referred to by the first text, including: obtaining the image area referred to by the first text in the first image; applying the reference-aware dynamic image mask strategy to block-mask multiple image blocks located in the image area with a first mask rate, and randomly mask multiple image blocks located outside the image area with a second mask rate, wherein the first mask rate is greater than the second mask rate.
[0075] exist Figure 6 In the paper, the present application shows schematic diagrams of three masking strategies, namely (a) random masking (MAE), (b) block masking (BEiT-3), and (c) the dynamic image masking method based on reference proposed in the present application. Among them, α represents the proportion of the overall mask, while β and γ refer to the mask rate outside and within the image area referenced by the text, respectively. It is worth noting that the strategy proposed in the present application effectively circumvents the limitations of the aforementioned methods and successfully guides the model to focus on the image area referenced by the text.
[0076] Specifically, the size of the image after being divided into image blocks is represented as h and w in this application, where , , is the size of the image block, is the total number of image blocks. In order to maximize the image area referred to by the text (from the starting coordinates and size The mask effect of the text is defined as follows. This application introduces a margin m around it and expresses the coordinates of the image area block referred to by the expanded text as ,Right now , and The calculation method of and Therefore, the number of image blocks contained in the image area referred to by the text It can be expressed as Among them, in order to ensure that the model can allocate appropriate attention to the visual information around the image area referred to by the text, this application uses a relatively low masking rate to randomly mask its surrounding environment. Specifically, the dynamic image masking strategy of reference perception is applied to dynamically mask multiple image blocks based on visually associated words and the referred image area. The algorithm process corresponding to the image mask position information of multiple image blocks is as follows:
[0077] First, enter image blocks (patch), The image blocks referred to are ,exist Random selection Number c, then create a new mask , satisfying the conditions , then mask: mask Set to ; Further, start to construct the mask of the image area referred to by the text , create a mask of all 0s , perform circular mask generation: in the mask The absolute value of is less than or equal to Before, repeat the following: : Randomly select an integer s between 1 and η·N_r minus the number of elements in the current mask Mr_r; : Randomly select a real number r, ranging from a to 1 / a; calculate the size of the mask area, and calculate the width of the mask area based on r and high : ; Determine the location of the mask area: randomly select the x coordinate of the upper left corner of the mask area and the y coordinate : ; Update mask: set mask , if i belongs to Range, j belongs to range; finally, when exiting the loop, merge the masks: Add to On the image, get the position of the dynamic mask .
[0078] Among them, since the image area referred to by the text is different in different image-text pairs, the overall mask rate of each sample is is determined dynamically:
[0079] ;
[0080] α represents the entire mask rate, β represents the second mask rate beyond the reference area, and γ represents the first mask rate within the reference area. The first mask rate is greater than the second mask rate. It can be understood that the present application uses a relatively low mask rate β for random masking around the image area referred to by the text. Correspondingly, in the extended area of the image area referred to by the text, the present application uses a block mask strategy with a higher mask rate γ to ensure that the model can allocate appropriate attention to the visual information around the image area referred to by the text.
[0081] refer to Figure 7 , Figure 7 A schematic diagram of a text mask provided in an embodiment of the present application; in some embodiments, the masked text words and the semantic target association scores corresponding to the text words are reconstructed according to the text words and the aggregated visual words of the masked first text, but not limited to the following steps:
[0082] Step S710, encoding the masked first text by a multimodal encoder to obtain text word-grams corresponding to a plurality of text words in the masked first text;
[0083] Step S720, performing a dot product operation on the plurality of text words and the aggregated visual word-gram to obtain a text-associated word-gram between each text word and the referring text region corresponding to the first image;
[0084] Step S730, reconstructing a plurality of masked text words in the first text according to a plurality of text-associated word-grams to obtain reconstructed text word-grams;
[0085] Step S740, reconstructing the text target relevance score corresponding to the text word according to the plurality of text-related word-grams to obtain a reconstructed text target relevance score;
[0086] Step S750, calculating a text modality loss function based on the reconstructed text word unit and the reconstructed text target association score, and optimizing the reference-aware mask image modeling part of the multimodal encoder based on the text modality loss function; wherein the text target association score represents the correlation between the text word and the text target referred to by the first image.
[0087] It can be understood that in the reference-aware masked text modeling, the goal is not only to restore the masked text words, but also to consider the cross-modal association between text and image, wherein the aggregated visual word-meta is obtained by aggregating these visual word-meta to obtain the aggregated visual word-meta representing the entire image or the image area referred to by a specific text. After the masked first text is encoded by a multimodal encoder to obtain text words corresponding to multiple text words in the masked first text, the text word-meta corresponding to each text word is subjected to a dot product operation with the aggregated visual word-meta, and a text association word-meta representing the association between each text word-meta and the image can be obtained. The word-gram reconstructs multiple masked text words in the first text to obtain reconstructed text word-grams, and reconstructs the text target association scores corresponding to the text words to obtain reconstructed text target association scores, and by comparing the reconstructed text word-grams with the text word-grams corresponding to the original text, as well as comparing the reconstructed text target association scores with the text target association scores corresponding to the original text, a technology of text modality loss function is implemented to optimize the reference-aware masked text modeling part of the multimodal encoder according to the corresponding loss values, wherein the text word-grams corresponding to the original text and the text target association scores can be obtained based on the supervision tools provided by the self-supervised learning of the original MLM.
[0088] In some embodiments, the text words of the first text are masked based on the referential subject of the first text, including: determining the subject text words from the first text according to the referential subject of the first text; preferentially masking the subject text words based on the target masking rate corresponding to the first text, and then randomly masking the remaining non-subject text words, wherein it can be understood that the referential subject is the text word used to act as the subject in the sentence, and the referential-aware text masking strategy is to preferentially mask the text words corresponding to the subject of the sentence on the basis of random masking, and then randomly mask the remaining text words, and the final masking ratio is fixed like the original random text masking.
[0089] In some embodiments, the text target association score corresponding to the text word is reconstructed according to multiple text-associated word-grams to obtain a reconstructed text target association score, including: calculating a first similarity between the text word-gram of each text word and the aggregated visual word-gram according to the multiple text-associated word-grams, and calculating a second similarity between the text word-gram of each text word and the aggregated visual word-gram referring to the image area referred to by the first text; obtaining a reconstructed text target association score according to the first similarity and the second similarity, wherein the first similarity is the similarity between each text word-gram and the aggregated visual word-gram, and the second similarity is the similarity between each text word-gram and the aggregated visual word-gram referring to the image area, and combining the first similarity and the second similarity to obtain the reference text association information of each text word-gram, and then based on the text-associated word-gram of each text word, the text can be randomly masked to obtain text mask position information of each text word for subsequent mask language model training.
[0090] Specifically, the training process of reference-aware masked text modeling includes using a teacher model (such as the BEiT-3 model fine-tuned by image-text contrastive learning) to calculate the similarity between each text token and the aggregated visual token, using cosine similarity to measure the similarity, and obtaining scores through softmax normalization, the scores reflect the relevance of the text token to the reference target; further, predicting these scores through a text semantic target association head consisting of three layers of multi-layer perceptron (MLP) and softmax normalization, these scores are used to guide the model to learn which text tokens are closely related to the image region referred to by the text; further, using Kullback-Leibler divergence (KL divergence) to measure the difference between the predicted scores and the true labels, this loss function helps the model learn how to correctly predict mask tokens while taking into account their relevance to the image region referred to by the text in the image, for example, by calculating the language token Aggregate visual tokens with the image region referred to by the text and the aggregate visual token of the entire image The weighted sum of the normalized similarities is:
[0091] ;
[0092] in, represents the cosine similarity operation, represents softmax normalization. In addition, this application can use the text semantic target association head composed of three layers of MLP and softmax normalization to predict the score. Finally, the training loss of the reference MLM is defined as:
[0093] ;
[0094] in, It indicates that Kullback-Leibler divergence is used for probability prediction. Through the above steps and the definition of the training loss of the reference MLM, the reference-aware masked text modeling proposed in this application can more effectively learn the reference relationship between images and texts, thereby improving the performance in visual localization and reference segmentation tasks. It is worth noting that the training loss function of the reference MLM compares the reconstructed word units and the associated scores to achieve pre-training of the reference-aware masked text modeling part of the multimodal encoder.
[0095] In some embodiments, it is worth noting that the pre-training process of the reference-aware masked image modeling part of the multimodal encoder and the pre-training process of the reference-aware masked text modeling part of the multimodal encoder in the present application are performed alternately, that is, when the reference-aware masked image modeling part is masked and reconstructed, the reference-aware masked text modeling part is encoded, and the reference-aware masked image modeling part performs a corresponding image word-meta reconstruction process based on the aggregated text word-meta obtained by encoding the reference-aware masked text modeling part to complete the parameter optimization of the reference-aware masked text modeling part. Conversely, when the reference-aware masked text modeling part is masked and reconstructed, the reference-aware masked image modeling part is encoded, and the reference-aware masked text modeling part performs a corresponding text word-meta reconstruction process based on the aggregated visual word-meta obtained by encoding the reference-aware masked image modeling part to complete the parameter optimization of the reference-aware masked text modeling part. The pre-training process of the multimodal encoder is completed by cyclically alternatingly optimizing the model parameters of the two parts.
[0096] In some embodiments, the multimodal encoder is fine-tuned based on the head position of the visual localization task to obtain the visual localization task result, including: obtaining the target visual word element, target text word element, target aggregate visual word element and target aggregate text word element of the second image and the second text based on the multimodal encoder; obtaining the similarity mask between the image block of the second image and the referential target of the second text according to the target visual word element, target text word element, target aggregate visual word element and target aggregate text word element; cross-modally aligning multiple target visual word elements based on the similarity mask, and summing the aligned target visual word elements to obtain a compact visual word element; performing bounding box regression prediction based on the compact visual word element based on a multilayer perceptron to obtain a prediction mask; applying the prediction mask to the second image to obtain the task result of visual localization.
[0097] In some embodiments, the multimodal encoder is fine-tuned based on the reference segmentation task head to obtain the reference segmentation task result, including: obtaining the target visual word-grams, target text word-grams, target aggregated visual word-grams and target aggregated text word-grams of the second image and the second text based on the multimodal encoder, and upsampling the target visual word-grams by multi-layer deconvolution operations to obtain high-resolution visual word-grams; performing cosine similarity calculation on the high-resolution visual word-grams and the target aggregated text word-grams to obtain a similarity mask; adjusting the similarity mask to the target resolution through bilinear interpolation to obtain a prediction mask; applying the prediction mask to the second image to obtain the task result of reference segmentation.
[0098] Among them, the target aggregate visual word unit is the aggregate visual feature representation obtained after processing by the multimodal encoder for a given image in the practical application stage of pre-training. The target aggregate text word unit is the aggregate text feature representation obtained after processing by the multimodal encoder for a given text in the practical application stage of pre-training. The similarity mask represents the similarity between each area in the image and the target described in the text, which can be used to determine the position of the target. The prediction mask is a binary mask used to represent the exact position and shape of the target in the image. It can be understood that, given a second image and a second text, the input image and text are processed using the pre-trained multimodal encoder. , after obtaining the target visual word, the target text word, the target aggregate visual word and the target aggregate text word, which are respectively the aggregate feature representations of the image and the text, the present application can calculate the similarity between each image block in the image and the target described in the text according to the target visual word and the target aggregate text word, obtain a similarity mask, which represents the correlation between each area in the image and the target described in the text, and generate a prediction mask, generate a prediction mask based on the similarity mask, which represents the position and shape of the target in the image, and apply the prediction mask to the second image to obtain the task result of visual positioning or reference segmentation.
[0099] In some embodiments, a dot product operation or other similarity measurement method can be used to calculate the similarity between the target visual word and the target aggregate text word to generate a similarity mask, and then when predicting the mask, for the visual localization task, a multi-layer perceptron (MLP) can be used to perform bounding box regression prediction on the compact visual feature representation to generate a prediction mask. For the reference segmentation task, the visual features can be upsampled using a deconvolution operation, and then cosine similarity calculations are performed with the aggregate text word to generate a prediction mask, which is adjusted to the target resolution through bilinear interpolation.
[0100] refer to Figure 8 , Figure 8A schematic diagram of the present application for a referring-expression-comprehension task provided for an embodiment of the present application; a multimodal encoder is fine-tuned based on a visual localization task head to obtain a visual localization task result, wherein a referring-expression-comprehension (REC)-based REC task head corresponding to the referring-expression-comprehension (REC) task head is used to obtain a visual localization task result, a similarity mask represents a matrix of similarities between each visual token and an aggregated language token, and is normalized by Softmax; a compact visual feature represents a single feature vector obtained by summing the aligned visual tokens, which is used for subsequent bounding box regression; bounding box regression is a computer vision technique used to predict the position and size of a target object in an image; a multi-layer perceptron (MLP) is a simple feedforward neural network used for classification and regression tasks.
[0101] In some embodiments, corresponding to Figure 8 The present application can perform similarity operations (such as dot products) on each pair of target visual tokens and target aggregated language tokens to output a Softmax-normalized similarity mask, and then copy the similarity mask and multiply it element-by-element with each hidden dimension of the visual token to achieve cross-modal alignment, and sum all aligned visual tokens to obtain a compact visual feature representation; further, a bounding box regression prediction is performed on the compact visual feature representation through a multi-layer perceptron (MLP) to predict a mask, including the position and size of the target object in the image. Specifically, based on the reference MIM, the present application first performs bounding box regression prediction on the visual token (segmentation) and aggregation language token Perform similarity operation to obtain a SoftMax normalized similarity mask This mask is then copied and multiplied back to each hidden dimension of the visual token. Subsequently, the visual tokens are summed to obtain a simplified token, and finally a 3-layer MLP is used to predict the box Perform regression:
[0102]
[0103] Among them, in order to improve the performance of the model, the above process needs to define a loss function to optimize the model parameters, and the loss function in the REC task head combines the positioning loss and the segmentation loss: the positioning loss includes the smooth L1 loss and the GIoU loss, which are used for the accuracy of bounding box regression, and the segmentation loss: including the Focal loss (FocalLoss) and the Dice / F1 loss, which are used to deal with the uneven data distribution and evaluate the segmentation quality. The final loss function is the weighted sum of the two losses. The weights can be adjusted according to actual needs. Specifically, the similarity can be used as the bounding box mask of the coarse-grained downsampling. , and the similarity mask after Sigmoid activation The segmentation constraint is imposed. Therefore, the loss function of the REC task can be reformulated as the weighted sum of the original localization loss (i.e., smooth L1 loss and Giou loss) and the segmentation loss (i.e., Focal loss and Dice / F-1 loss):
[0104]
[0105] Based on the above loss function, this application can obtain a compact visual feature representation through cross-modal alignment, and use MLP to perform bounding box regression prediction while ensuring the accuracy of visual positioning and segmentation.
[0106] refer to Fig. 9 , Fig. 9 A schematic diagram of a referring-expression-segmentation task provided in the present application for an embodiment of the present application; in some embodiments, a multimodal encoder is fine-tuned based on a referring-expression-segmentation task head to obtain a referring-expression-segmentation task result, wherein a referring-expression-segmentation task head (RES) corresponding to the referring-expression-segmentation task is used to obtain a referring-expression-segmentation task result, and a multi-layer deconvolution operation can be used to upsample visual words to restore the spatial resolution of features to obtain high-resolution visual words; further, a cosine similarity is calculated between each high-resolution visual word and an aggregated language word to obtain a similarity matrix, a similarity mask is obtained, and bilinear interpolation is used to adjust the similarity mask to the target resolution to match the actual size of the image, and a predicted mask is output.
[0107] It is understandable that for the reference segmentation task, a loss function needs to be defined to measure the gap between the predicted mask and the true label. The loss function of the reference-based RES task head can be a binary cross entropy loss or a Dice loss to measure the similarity between two binary images. Specifically, the present application uses a 3-layer deconvolution to upsample the visual token to . Then, the cosine similarity operation is performed on the upsampled visual token and the aggregated language token. Then the similarity mask obtained above is used as the final prediction mask after one layer of bilinear interpolation. This application denotes the ground-truth segmentation mask as , then the loss function of RES is defined as:
[0108]
[0109] It can be understood that by calculating the cosine similarity between the upsampled visual token and the aggregated language token, a similarity mask is obtained. Finally, the similarity mask is adjusted to the target resolution through bilinear interpolation to obtain a predicted mask, which helps to improve the segmentation accuracy for fine-grained target segmentation tasks.
[0110] The second aspect of the embodiment of the present application provides a visual localization and reference segmentation system based on mask reference modeling. The system is used to implement the visual localization and reference segmentation method based on mask reference modeling as any one of the first aspects, so it will not be repeated here.
[0111] In summary, the new pre-training paradigm of masked reference modeling in the multimodal encoder proposed in the present application applies mask modeling technology to reference tasks. This paradigm includes masked image modeling of reference and masked text modeling of reference. In addition, in the process of masked modeling of reference, a dynamic image masking strategy based on reference perception and a text masking strategy based on reference perception are introduced, so that the multimodal encoder can efficiently model the complex reference relationship between vision and language. The corresponding framework of the multimodal encoder of the present application realizes positioning and reference segmentation in a unified modal shared feature space without relying on traditional modal interaction technology, modal fusion Transformer encoder / decoder or special positioning token, thereby achieving significant simplification and optimization in the model architecture and improving the performance in visual localization and reference segmentation tasks.
[0112] Some embodiments of the present application provide an electronic device, Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, referring to Fig.10 The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the visual localization and referential segmentation method based on mask referential modeling of any one of the above embodiments is implemented, for example, the above described Figure 2 Steps S210 to S230 of the method, Figure 3 Steps S310 to S330 of the method, Figure 5 Steps S510 to S550 of the method, Figure 7 Method steps S710 to S750.
[0113] The electronic device 1000 of the embodiment of the present application includes one or more processors 1010 and a memory 1020. Fig.10 In the figure, a processor 1010 and a memory 1020 are taken as an example.
[0114] The processor 1010 and the memory 1020 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.
[0115] The memory 1020, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 1020 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1020 may optionally include a memory 1020 remotely disposed relative to the processor 1010, and these remote memories may be connected to the electronic device 1000 via a network. At the same time, examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.
[0116] In some embodiments, when the processor executes the computer program, the visual localization and reference segmentation method based on mask reference modeling of any of the above embodiments is performed at preset intervals.
[0117] Those skilled in the art will understand that Fig.10 The device structure shown in the figure does not constitute a limitation on the electronic device 1000, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0118] exist Fig.10 In the electronic device 1000 shown, the processor 1010 can be used to call the visual positioning and reference segmentation method based on mask reference modeling stored in the memory 1020, so as to implement the visual positioning and reference segmentation method based on mask reference modeling.
[0119] Based on the hardware structure of the above-mentioned electronic device 1000, various embodiments of the visual positioning and reference segmentation system based on mask reference modeling of the present application are proposed. At the same time, the non-transient software programs and instructions required to implement the visual positioning and reference segmentation method based on mask reference modeling of the above-mentioned embodiments are stored in the memory. When executed by the processor, the visual positioning and reference segmentation method based on mask reference modeling of the above-mentioned embodiments is executed.
[0120] The present application also provides a computer-readable storage medium storing computer-executable instructions for executing the above-mentioned visual positioning and reference segmentation method based on mask reference modeling, which can enable the above-mentioned one or more processors to execute the visual positioning and reference segmentation method based on mask reference modeling of any of the above-mentioned embodiments, for example, executing the above-mentioned Figure 2 Steps S210 to S230 of the method, Figure 3 Steps S310 to S330 of the method, Figure 5 Steps S510 to S550 of the method, Figure 7 Method steps S710 to S750.
[0121] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the visual localization and reference segmentation method based on mask reference modeling to implement any one of the above embodiments, for example, executing the above described Figure 2 Steps S210 to S230 of the method, Figure 3 Steps S310 to S330 of the method, Figure 5 Steps S510 to S550 of the method, Figure 7 Method steps S710 to S750.
[0122] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network nodes. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0123] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer-readable storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer-readable storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD to ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0124] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.
Claims
1. A visual localization and coreference segmentation method based on mask coreference modeling, characterized in that: include: Obtaining a first image and a first text for training; Obtain visual word-grams, aggregated visual word-grams, text word-grams, and aggregated text word-grams of the first image and the first text through a multimodal encoder, and perform the following pre-training process on the multimodal encoder: Applying a reference-aware dynamic image masking strategy, masking an image block of the first image based on the image region referenced by the first text, and reconstructing the masked image block and the visual target association score corresponding to the image block according to the visual word-gram and the aggregated text word-gram of the masked first image, so as to implement pre-training of the reference-aware masked image modeling part of the multimodal encoder; Applying a reference-aware text masking strategy, masking text words of the first text based on the reference subject of the first text, and reconstructing the masked text words and the semantic target association scores corresponding to the text words according to the masked text word-grams and the aggregated visual word-grams of the first text, so as to implement pre-training of the reference-aware masked text modeling part of the multimodal encoder; Acquire a second image and a second text to be processed, input the second image and the second text into the pre-trained multimodal encoder, and fine-tune the multimodal encoder based on the visual localization task head or the reference segmentation task head to obtain a visual localization task result or a reference segmentation task result.
2. The visual localization and representative segmentation method based on mask representative modeling according to claim 1, characterized in that: The step of obtaining visual word-grams, aggregated visual word-grams, text word-grams, and aggregated text word-grams of the first image and the first text through a multimodal encoder includes: Segmenting the first image into a plurality of image blocks, and encoding each of the image blocks into a visual word unit by the multimodal encoder; Dividing the first text into a plurality of text words, and encoding each of the text words into text words by the multimodal encoder; Outputting, through the self-attention mechanism of the multimodal encoder, a plurality of aggregated visual word-grams and aggregated text word-grams corresponding to the visual word-grams and the text word-grams; The aggregated visual word-gram represents the overall visual features corresponding to the first text, and the aggregated text word-gram represents the overall text features corresponding to the first image.
3. The visual localization and representative segmentation method based on mask representative modeling according to claim 1, characterized in that: The application of the reference-aware dynamic image masking strategy, masking the image block of the first image based on the image area referred to by the first text, includes: Acquire the image area referred to by the first text in the first image; A reference-aware dynamic image masking strategy is applied to block-mask a plurality of image blocks located within the image area at a first masking rate, and to randomly mask a plurality of image blocks located outside the image area at a second masking rate, wherein the first masking rate is greater than the second masking rate.
4. The visual localization and representative segmentation method based on mask representative modeling according to claim 1, characterized in that: The step of reconstructing the masked image block and the visual target association score corresponding to the image block according to the visual word-gram and the aggregated text word-gram of the masked first image includes: encoding the masked first image by the multimodal encoder to obtain visual word units corresponding to the plurality of image blocks in the masked first image; Performing a dot product operation on the plurality of visual words and the aggregated text word to obtain a visually associated word between each of the image blocks and a referring image region corresponding to the first text; Reconstructing a plurality of mask image blocks in the first image according to the plurality of visually associated word-grams to obtain reconstructed visual word-grams; Reconstructing the visual target relevance score corresponding to the image block according to the plurality of visually associated word-grams to obtain a reconstructed visual target relevance score; Calculating a visual modality loss function according to the reconstructed visual word unit and the reconstructed visual target association score, and optimizing a referentially-aware mask image modeling portion of the multimodal encoder based on the visual modality loss function; The visual target association score represents the positional relationship between the image block and the image region referred to by the first text.
5. The visual localization and representative segmentation method based on mask representative modeling according to claim 4, characterized in that: The reconstructing the visual target relevance score corresponding to the image block according to the plurality of visually associated word-grams to obtain a reconstructed visual target relevance score includes: Reconstructing a horizontal distance mask, a vertical distance mask, a width ratio mask and a height ratio mask between each of the image blocks and the reference image area according to the visually associated word; A reconstructed visual target association score is obtained according to the horizontal distance mask, the vertical distance mask, the width ratio mask and the height ratio mask.
6. The visual localization and representative segmentation method based on mask representative modeling according to claim 1, characterized in that: The step of masking the text words of the first text based on the referential subject of the first text includes: determining a subject text word from the first text according to a referent subject of the first text; The subject text words are masked first based on the target masking rate corresponding to the first text, and then the remaining non-subject text words are randomly masked.
7. The method for visual localization and representative segmentation based on mask representative modeling according to claim 1, characterized in that: The step of reconstructing the masked text words and the semantic target association scores corresponding to the text words according to the masked text words and the aggregated visual words of the first text includes: Encoding the masked first text by the multimodal encoder to obtain text word-grams corresponding to the plurality of text words in the masked first text; Performing a dot product operation on the plurality of text words and the aggregated visual word-gram to obtain a text-associated word-gram between each of the text words and a referring text region corresponding to the first image; Reconstructing a plurality of masked text words in the first text according to a plurality of the text-associated word-grams to obtain reconstructed text word-grams; Reconstructing the text target relevance score corresponding to the text word according to the plurality of text-related word-grams to obtain a reconstructed text target relevance score; Calculating a text modality loss function according to the reconstructed text word and the reconstructed text target association score, and optimizing the reference-aware mask image modeling part of the multimodal encoder based on the text modality loss function; The text target association score represents the correlation between the text word and the text target referred to by the first image.
8. The visual localization and representative segmentation method based on mask representative modeling according to claim 7, characterized in that: The step of reconstructing the text target relevance score corresponding to the text word according to the plurality of text-related word-grams to obtain a reconstructed text target relevance score includes: Calculating a first similarity between a text word-gram of each of the text words and the aggregate visual word-gram according to the plurality of text-associated word-grams, and calculating a second similarity between a text word-gram of each of the text words and the aggregate visual word-gram referring to the image region referred to by the first text; A reconstructed text target association score is obtained according to the first similarity and the second similarity.
9. The method for visual localization and representative segmentation based on mask representative modeling according to claim 1, characterized in that: The multimodal encoder is fine-tuned based on the visual positioning task head position to obtain a visual positioning task result, including: Obtaining target visual word-grams, target text word-grams, target aggregated visual word-grams, and target aggregated text word-grams for the second image and the second text based on the multimodal encoder; Obtaining a similarity mask between an image block of the second image and a referent target of the second text according to the target visual word-gram, the target text word-gram, the target aggregate visual word-gram and the target aggregate text word-gram; Performing cross-modal alignment on the plurality of target visual word-grams based on the similarity mask, and summing the aligned target visual word-grams to obtain a compact visual word-gram; Performing bounding box regression prediction based on the compact visual word-element based on a multi-layer perceptron to obtain a prediction mask; The predicted mask is applied to the second image to obtain a visual localization task result.
10. The visual localization and representative segmentation method based on mask representative modeling according to claim 1, characterized in that: Fine-tuning the multimodal encoder based on the coreference segmentation task head to obtain a coreference segmentation task result includes: Obtaining target visual word-grams, target text word-grams, target aggregated visual word-grams, and target aggregated text word-grams of the second image and the second text based on the multimodal encoder, and upsampling the target visual word-grams by multi-layer deconvolution operations to obtain high-resolution visual word-grams; Performing cosine similarity calculation on the high-resolution visual word-unit and the target aggregated text word-unit to obtain a similarity mask; Adjusting the similarity mask to a target resolution by bilinear interpolation to obtain a prediction mask; The predicted mask is applied to the second image to obtain a task result representing segmentation.
11. A visual localization and referent segmentation system based on mask referent modeling, characterized in that: The system is used to implement the visual localization and representative segmentation method based on mask representative modeling as described in any one of claims 1 to 10.
12. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, the visual localization and representative segmentation method based on mask representative modeling as described in any one of claims 1 to 10 is implemented.
13. A computer-readable storage medium storing computer-executable instructions for executing the visual localization and representative segmentation method based on mask representative modeling as described in any one of claims 1 to 10.
