A method and system for referential image segmentation based on reiteration-interleaving
By using a reiteration-interweaving mechanism to process visual and textual features, this technology addresses the problem of insufficient utilization of textual feature contextual relationships in existing technologies, thereby improving image segmentation accuracy and robustness and enabling efficient image segmentation in complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI ARTIFICIAL INTELLIGENCE INNOVATION CENT
- Filing Date
- 2025-03-18
- Publication Date
- 2026-06-09
Smart Images
Figure CN120388038B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and image processing technology, and in particular to a method and system for reference image segmentation based on reiteration-interweaving. Background Technology
[0002] Image segmentation is a fundamental task in computer vision, aiming to classify and label different regions within an image. Traditional image segmentation tasks primarily focus on the visual information of the image itself, while in recent years, multimodal learning incorporating linguistic information has gradually become a research hotspot. In more challenging problems, the model not only needs to process image data but also needs to perform accurate segmentation based on textual descriptions. This problem is called referential image segmentation.
[0003] To address this issue, some progress has been made in research on cross-modal alignment mechanisms. Chinese patent application publication CN116912837A discloses a detail- and boundary-driven method for reference image segmentation, which enhances feature alignment by using boundary, detail, and saliency detection methods. Chinese patent application publication CN116704506A proposes a reference image segmentation method that adaptively adjusts multimodal correspondences based on different global semantic features, enhancing the model's ability to understand cross-modal information. However, these methods typically assume that the text description is a short phrase or a simple structure. This assumption oversimplifies the diversity and flexibility of descriptive language in real-world situations, limiting the ability to understand complex semantic expressions and thus restricting their practical application. In real-world scenarios, such methods still have significant drawbacks:
[0004] (1) Existing models oversimplify the processing of text features and lack the ability to model complex syntactic structures and semantic dependencies, resulting in the implicit contextual relationships in the description not being fully utilized.
[0005] (2) There is a serious imbalance in the sequence length between visual and textual features. The visual features of long sequences tend to mask the semantic information of short texts, weakening the fine-grainedness of cross-modal interaction.
[0006] (3) Traditional architectures (such as Transformer) suffer from low computational efficiency when processing long sequences, making them unsuitable for real-time applications. These issues cause existing methods to experience a significant decrease in segmentation accuracy and insufficient robustness when dealing with complex and lengthy text descriptions in real-world scenarios.
[0007] Therefore, there is an urgent need for a technical solution that can efficiently balance multimodal sequence interactions, deeply analyze complex text semantics, and improve cross-modal alignment quality in order to break through the current performance bottleneck of referential image segmentation. Summary of the Invention
[0008] The purpose of this invention is to overcome the shortcomings of the prior art by providing a reference image segmentation method and system based on reiteration-interweaving, so as to solve or partially solve the problem that the implicit contextual relationships in text descriptions are not fully utilized, thus affecting the segmentation effect.
[0009] The objective of this invention can be achieved through the following technical solutions:
[0010] One aspect of the present invention provides a reference image segmentation method based on reiteration-interweaving, which obtains segmentation results based on input images and text using a trained reference image segmentation model, wherein the training process of the trained reference image segmentation model includes the following steps:
[0011] Obtain the target image and its attribute text;
[0012] The target image is encoded into visual features through visual feature extraction;
[0013] By extracting text features, the attribute text is encoded into text features;
[0014] Based on the visual features, grouping is performed, and based on the text features, stacking is performed to complete the reiteration operation. Based on the grouped visual features and the stacked text features, splicing, scanning fusion, rearrangement, and aggregation based on learnable scaling parameters are performed to obtain new visual features and text features, and the interleaving operation is completed. By repeating the reiteration-interleaving operation multiple times, the visual features after each reiteration-interleaving operation are obtained, and the segmentation result is obtained according to the probability map.
[0015] Based on the segmentation results and the obtained true segmentation mask, the image segmentation model is trained.
[0016] As a preferred technical solution, the process of encoding the target image into visual features includes the following steps:
[0017] The target image is divided into multiple non-overlapping slices, and each slice is mapped to the feature space through linear projection or convolution to obtain a feature sequence.
[0018] The feature sequence is processed based on learnable relative position bias;
[0019] For the feature sequence after bias processing, a sliding window self-attention model is used to obtain the output visual features through multi-layer window self-attention, using windows as windows and step-by-step downsampling.
[0020] As a preferred technical solution, the process of encoding the attribute text into text features includes the following steps:
[0021] Based on the aforementioned text features, a word segmenter is used to generate word-level segmentation codes;
[0022] The word segmentation encoding is converted into an embedding vector and added to the absolute position encoding and segment encoding;
[0023] Based on the summed features, a text sub-attention model is used to obtain word-level features describing the text encoding, which are then used as text features.
[0024] As a preferred technical solution, the process of grouping based on the visual features includes the following steps:
[0025] The visual features are used as input to a two-dimensional state-space model to obtain updated visual features that do not change shape.
[0026] Based on the updated visual features, the windows are regrouped and flattened into a sequence according to the groups to obtain the regrouped visual features.
[0027] As a preferred technical solution, the stacking process based on the text features includes the following steps:
[0028] The text feature encoding features are repeated several times and stacked to obtain new text features, where the number of repetitions matches the number of groups of visual features.
[0029] As a preferred technical solution, the process of splicing, scanning fusion, rearranging, and aggregation based on learnable scaling parameters, based on the grouped visual features and stacked text features, includes the following steps:
[0030] The grouped visual features and the stacked text features are concatenated to obtain a hybrid multimodal sequence;
[0031] The hybrid multimodal sequence is input into a two-dimensional state space model to obtain a cross-modal fusion hybrid multimodal sequence with invariant shape, resulting in updated visual features and updated text features.
[0032] The updated visual features are rearranged to obtain the modality-fused visual features, which are then used as the new visual features.
[0033] The updated text features are aggregated according to learnable proportional parameters to obtain the aggregated text features after modal fusion, which are then used as new text features.
[0034] As a preferred technical solution, the process of obtaining the visual features after each reiteration-interleaving operation by repeatedly performing the reiteration-interleaving operation, and then obtaining the segmentation result based on the probability map, includes the following steps:
[0035] By repeatedly performing the reiteration-interleaving operation, the visual features after each reiteration-interleaving operation are obtained, forming a visual feature sequence.
[0036] Using the visual feature sequence as input to a multi-layer convolutional neural network, multi-layer, multi-modal fused visual features are obtained;
[0037] Based on the multi-level and multi-modal visual features, the images are restored to a size that matches the target image through interpolation to obtain a segmentation probability map. The segmentation result is then obtained by thresholding.
[0038] As a preferred technical solution, the process of training the denotated image segmentation model based on the segmentation results and the obtained true segmentation mask includes the following steps:
[0039] Based on the segmentation results and the obtained true segmentation mask, the Dessian loss and focus loss are calculated to obtain the comprehensive loss.
[0040] Based on the aforementioned comprehensive loss, the referential image segmentation model is trained using gradient descent.
[0041] As a preferred technical solution, the overall loss is:
[0042] L = Dice(y,m) + Focal(y,m)
[0043] Where y is the predicted segmentation result, m is the actual segmentation result of the image, Dice represents the Dice loss function, and Focal represents the focal loss function.
[0044] Another aspect of the present invention provides a reference image segmentation system based on reiteration-interlacing for implementing the aforementioned reference image segmentation method based on reiteration-interlacing. The reference image segmentation system includes:
[0045] The visual feature extraction module is used to encode the acquired target image into visual features through visual feature extraction.
[0046] The text feature extraction module is used to encode the acquired attribute text into text features through text feature extraction;
[0047] The reiteration-interweaving module is used to perform grouping processing based on the visual features, stacking processing based on the text features, and completing the reiteration operation. Based on the grouped visual features and the stacked text features, it performs splicing, scan fusion, rearrangement, and aggregation processing based on learnable scaling parameters to obtain new visual features and text features, and completes the interweaving operation. By repeating the reiteration-interweaving operation multiple times, the visual features after each reiteration-interweaving operation are obtained, and the segmentation result is obtained according to the probability map.
[0048] The training module is used to train the image segmentation model based on the segmentation results and the obtained real segmentation mask;
[0049] The denotation image segmentation module is used to obtain segmentation results based on the input image and text using the trained denotation image segmentation model.
[0050] Compared with the prior art, the present invention has at least one of the following beneficial effects:
[0051] (1) Fully understand the multi-level semantics in the text: For long sentences with complex structure or ambiguous semantics, this invention can better understand the multi-level semantics in the text through the reiteration and interleaving processing mechanism, and make full use of the contextual relationships implied in the text description, thereby avoiding the drawbacks of oversimplification of text in traditional methods. By integrating images and text into a unified multimodal sequence, the deep interaction between images and text is further enhanced. This innovative reiteration and interleaving mechanism enables image and text information to be more closely combined, thereby improving the accuracy and robustness of image segmentation.
[0052] (2) Good stability and accuracy of semantic text: In response to the problem of unbalanced ratio of image and text information, this invention performs splicing, scanning fusion, rearrangement and aggregation based on learnable ratio parameters on visual features after grouping and text features after stacking to obtain new visual features and text features. By balancing the ratio of image and text in multimodal sequences, the phenomenon of image information covering text semantics is avoided, thereby ensuring the stability and accuracy of text semantics.
[0053] (3) Strong cross-modal understanding ability: This invention processes visual and textual features through reiteration and interleaving processing mechanisms, which enhances cross-modal understanding ability for descriptions containing referential, context-dependent or ambiguous information, and shows higher robustness, enabling accurate image segmentation in complex scenes. Attached Figure Description
[0054] Figure 1 This is a flowchart of the reference image segmentation method based on reiteration-interweaving in the embodiments;
[0055] Figure 2 This is a schematic diagram representing the image segmentation model in the embodiment;
[0056] Figure 3 This is a schematic diagram of the reference image segmentation system based on reiteration-interweaving in the embodiment;
[0057] Figure 4 This is a schematic diagram of the electronic device in the embodiment. Detailed Implementation
[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0059] Example 1
[0060] To address the problems existing in the prior art, this embodiment provides a reference image segmentation method based on reiteration-interweaving, which constructs and trains a method such as... Figure 2 The image segmentation model shown uses a trained model to segment images representing their meaning. (See also...) Figure 1 The method includes the following steps:
[0061] Step S1: Obtain the target image and attribute text.
[0062] Step S2 involves encoding the target image into visual features using a visual feature extractor. Specifically, step S2 may include steps S201-S203:
[0063] Step S201: Transfer the target image x∈R H×W×3 The slice is divided into non-overlapping 2D slices, each slice being P×P in size. Each slice is mapped to a D-dimensional feature space through linear projection or 1×1 convolution to obtain serialized features.
[0064] Step S202: Combine the serialized features with the learnable relative position bias to obtain new features.
[0065] Step S203: Input the above features into the Swin Transformer, and after multiple layers of window self-attention, shifted window, and step-by-step downsampling (Patch Merging), output the visual features V∈R of the last layer. h×w×d , where h and w are the length and width dimensions of the output visual features, and d is the final number of channels.
[0066] Step S3: Encode the attribute text into text features using a text feature extractor. Specifically, step S3 may include steps S301-S303:
[0067] Step S301: Feed the description text into the tokenizer to generate a subword-level word segmentation sequence and add special tags.
[0068] Step S302: Convert the word segmentation encoding into an embedding vector and add it to the absolute position encoding (PositionEmbeddings) and segment encoding (Segment Embeddings).
[0069] Step S303: Feed the above features into the text self-attention model (text transformer) to obtain word-level features T∈R that describe the text encoding. N×d , where N is the number of word-level features in the text encoding.
[0070] Step S4 involves feeding both visual and textual features into a modal interleaving network to obtain a segmentation probability map, which is then converted into a segmentation result using a thresholding method. Step S4 may include steps S401-S410:
[0071] Step S401: Encode the visual features V∈R of the target image h×w×d First, a two-dimensional state-space model (SS2D) is input to obtain the updated visual features V with unchanged shape.
[0072] Step S402: Regroup the updated visual features V by window and flatten them into a sequence by group to obtain the regrouped visual features. Where P is the total number of window groups, and L is the number of elements in each window.
[0073] Step S403: Describe the text encoding features T∈R N×d Repeat this process several times, stacking the results to obtain new descriptive text. Where P is the total number of window groups. Steps S401-S403 implement the "reaffirm" operation.
[0074] Step S404: Process the above-mentioned visual features and textual features The sequences are spliced together to form a mixed multimodal sequence H∈R P×(L+N)×d .
[0075] Step S405: Input the above hybrid multimodal sequence H into another two-dimensional state-space model (SS2D) to obtain an updated hybrid multimodal sequence H with invariant shape, and then obtain the updated visual features. and updated text features
[0076] Step S406: Update the visual features Rearranged, the visual features V'∈R after modal fusion are obtained. h ×w×d And serve as the visual feature for the next iteration.
[0077] Step S407: Update the text features Aggregate according to the learnable proportional parameter a to obtain the aggregated text features T'∈R N×d It can be expressed by the formula as follows: This serves as the textual feature for the next iteration. Steps S404-S407 implement the "interweaving" operation.
[0078] Step S408: Repeat steps S401-S407M times, and obtain the intermediate output result {V'1,V'2,…,V'} based on the final visual features of each round. M}
[0079] Step S409: Input the above multi-level intermediate results into a multi-layer convolutional neural network (Convolution) and fuse them from top to bottom to obtain the final visual features V after multimodal fusion of the reiteration-interleaving network. out .
[0080] Step S410: For the final visual feature V out The image is restored to its original dimensions using interpolation to obtain a segmentation probability map, which is then further converted into a segmentation result y using a thresholding method.
[0081] Step S5: Calculate the loss between the predicted segmentation result and the actual segmentation mask, and train using gradient descent. Specifically, step S5 may include steps S501-S502:
[0082] Step S501: Calculate the loss function L = Dice(y,m) + Focal(y,m), where Dice represents the Dice Loss function, Focal represents the Focal Loss function, y is the predicted segmentation result, and m is the actual segmentation result of the image.
[0083] Step S502: Update L using gradient descent.
[0084] Step S6: Use the trained model to perform inference on the test scene image and test description text to obtain the test image segmentation result.
[0085] In summary, this method designs a referential image segmentation framework. Specifically, in the stage of handling complex text descriptions, firstly, for long sentences with complex structures or ambiguous semantics, the "reiteration and interleaving" mechanism can better understand the multi-level semantics in the text, thus avoiding the drawback of oversimplification of text in traditional methods. Secondly, for cases where the ratio of image and text information is unbalanced, by balancing the proportion of image and text in the multimodal sequence, the phenomenon of image information overriding text semantics is avoided, thus ensuring the stability and accuracy of text semantics. For descriptions containing referentials, context dependencies, or ambiguities, the cross-modal understanding capability is enhanced, exhibiting higher robustness and enabling accurate image segmentation in complex scenarios. In the multimodal interaction stage, this method further enhances the deep interaction between images and text by integrating images and text into a unified multimodal sequence. This innovative reiteration and interleaving mechanism allows image and text information to be more tightly combined, thereby improving the accuracy and robustness of image segmentation.
[0086] Example 2
[0087] Based on Example 1, see Figure 3 This embodiment provides a reference image segmentation system based on reiteration-interweaving, used to implement the reference image segmentation method based on reiteration-interweaving as described in Embodiment 1. The reference image segmentation system includes:
[0088] (1) Visual feature extraction module, used to encode the acquired target image into visual features through visual feature extraction.
[0089] (2) Text feature extraction module, which is used to encode the obtained attribute text into text features through text feature extraction.
[0090] (3) Reiteration-interweaving processing module, used to perform grouping processing based on the visual features, stacking processing based on the text features, and complete the reiteration operation. Based on the grouped visual features and the stacked text features, splicing, scanning fusion, rearrangement and aggregation processing based on learnable scale parameters are performed to obtain new visual features and text features, and complete the interweaving operation. By repeating the reiteration-interweaving operation multiple times, the visual features after each reiteration-interweaving operation are obtained, and the segmentation result is obtained according to the probability map.
[0091] (4) Training module, used to train the image segmentation model based on the segmentation results and the obtained real segmentation mask.
[0092] (5) The reference image segmentation module is used to obtain the segmentation result based on the input image and text using the trained reference image segmentation model.
[0093] Example 3
[0094] Based on the foregoing embodiments, see Figure 4 This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the reference image segmentation method based on reaffirmation-interweaving as described in Embodiment 1.
[0095] like Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The method described herein. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0096] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0097] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0098] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A reference image segmentation method based on reiteration-interweaving, characterized in that, Based on the input image and text, a segmentation result is obtained using a trained referential image segmentation model. The training process of the trained referential image segmentation model includes the following steps: Obtain the target image and its attribute text; The target image is encoded into visual features through visual feature extraction; By extracting text features, the attribute text is encoded into text features; Based on the visual features, grouping is performed, and based on the text features, stacking is performed to complete the reiteration operation. Based on the grouped visual features and the stacked text features, splicing, scanning fusion, rearrangement, and aggregation based on learnable scaling parameters are performed to obtain new visual features and text features, and the interleaving operation is completed. By repeating the reiteration-interleaving operation multiple times, the visual features after each reiteration-interleaving operation are obtained, and the segmentation result is obtained according to the probability map. Based on the segmentation results and the obtained true segmentation mask, the referential image segmentation model is trained. The process of grouping based on the visual features includes the following steps: The visual features are used as input to a two-dimensional state-space model to obtain updated visual features that do not change shape. Based on the updated visual features, the windows are regrouped, and the groups are flattened into sequences to obtain the regrouped visual features. The stacking process based on the text features includes the following steps: The text feature encoding features are repeated several times and stacked to obtain new text features. The number of repetitions matches the number of groups of visual features. The process of splicing, scan fusion, rearrangement, and aggregation based on learnable scaling parameters, based on the grouped visual features and stacked text features, includes the following steps: The grouped visual features and the stacked text features are concatenated to obtain a hybrid multimodal sequence; The hybrid multimodal sequence is input into a two-dimensional state space model to obtain a cross-modal fusion hybrid multimodal sequence with invariant shape, resulting in updated visual features and updated text features. The updated visual features are rearranged to obtain the modality-fused visual features, which are then used as the new visual features. The updated text features are aggregated according to learnable proportional parameters to obtain the aggregated text features after modal fusion, which are then used as new text features.
2. The reference image segmentation method based on reiteration-interweaving according to claim 1, characterized in that, The process of encoding the target image into visual features includes the following steps: The target image is divided into multiple non-overlapping slices, and each slice is mapped to the feature space through linear projection or convolution to obtain a feature sequence. The feature sequence is processed based on learnable relative position bias; For the feature sequence after bias processing, a sliding window self-attention model is used to obtain the output visual features through multi-layer window self-attention, using windows as windows and step-by-step downsampling.
3. The reference image segmentation method based on reiteration-interweaving according to claim 1, characterized in that, The process of encoding the attribute text into text features includes the following steps: Based on the aforementioned text features, a word segmenter is used to generate word-level segmentation codes; The word segmentation encoding is converted into an embedding vector and added to the absolute position encoding and segment encoding; Based on the summed features, a text sub-attention model is used to obtain word-level features describing the text encoding, which are then used as text features.
4. The reference image segmentation method based on reiteration-interweaving according to claim 1, characterized in that, The process of obtaining the segmentation result based on the probability map by repeatedly performing the reiteration-interleaving operation to obtain the visual features after each reiteration-interleaving operation includes the following steps: By repeatedly performing the reiteration-interleaving operation, the visual features after each reiteration-interleaving operation are obtained, forming a visual feature sequence. Using the visual feature sequence as input to a multi-layer convolutional neural network, multi-layer, multi-modal fused visual features are obtained; Based on the multi-level and multi-modal visual features, the images are restored to a size that matches the target image through interpolation to obtain a segmentation probability map. The segmentation result is then obtained by thresholding.
5. The reference image segmentation method based on reiteration-interweaving according to claim 1, characterized in that, The process of training the referential image segmentation model based on the segmentation results and the obtained true segmentation mask includes the following steps: Based on the segmentation results and the obtained true segmentation mask, the Dessian loss and focus loss are calculated to obtain the comprehensive loss. Based on the aforementioned comprehensive loss, the referential image segmentation model is trained using gradient descent.
6. The reference image segmentation method based on reiteration-interweaving according to claim 5, characterized in that, The comprehensive loss mentioned above is: in, To predict the segmentation results, The image represents the actual segmentation result. Dice represents the Dice loss function, and Focal represents the focal loss function.
7. A referential image segmentation system based on reiteration-interweaving, characterized in that, For implementing the reference image segmentation method based on reiteration-interweaving as described in any one of claims 1-6, the reference image segmentation system comprises: The visual feature extraction module is used to encode the acquired target image into visual features through visual feature extraction. The text feature extraction module is used to encode the acquired attribute text into text features through text feature extraction; The reiteration-interweaving module is used to perform grouping processing based on the visual features, stacking processing based on the text features, and completing the reiteration operation. Based on the grouped visual features and the stacked text features, it performs splicing, scan fusion, rearrangement, and aggregation processing based on learnable scaling parameters to obtain new visual features and text features, and completes the interweaving operation. By repeating the reiteration-interweaving operation multiple times, the visual features after each reiteration-interweaving operation are obtained, and the segmentation result is obtained according to the probability map. The training module is used to train the image segmentation model based on the segmentation results and the obtained real segmentation mask; The denotation image segmentation module is used to obtain segmentation results based on the input image and text using the trained denotation image segmentation model.