Multimodal large model image segmentation method and device based on hierarchical word element representation

Abstract

The embodiment of the application provides a kind of multi-modal large model image segmentation method and device based on hierarchical word representation, mask image is encoded into word sequence by designing mask marker, and progressive generation from shape prototype to local detail is realized by causal attention mechanism.Three-stage training strategy is adopted, first, the marker is trained through the mask reconstruction task, then the mask word is integrated into the large multi-modal model and jointly trained, finally, it is fine-tuned using high-resolution data.Multi-level supervision is carried out based on hierarchical mask loss function, and accurate segmentation of natural language description target is realized.The method effectively solves the deficiencies of traditional technology in complex scene understanding, model training and mask generation, and significantly improves the performance of multi-modal image segmentation.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a multimodal large model image segmentation method and apparatus based on hierarchical lexical representation. Background Technology

[0002] Existing multimodal image segmentation methods have significant shortcomings. Traditional systems perform poorly when handling complex scenes and detailed features, and struggle to accurately understand and locate target objects described in natural language.

[0003] Furthermore, existing technologies face bottlenecks in model training. Most systems lack hierarchical learning of mask representations and fail to effectively integrate the semantic understanding capabilities of large-scale pre-trained models, resulting in insufficient accuracy and generalization of segmentation results.

[0004] Existing systems have technical shortcomings in mask generation. The lack of a causal attention-based sequential generation mechanism makes it difficult to progressively refine mask features, affecting the precision of segmentation results. Solving these problems is crucial for improving the performance of multimodal image segmentation. Summary of the Invention

[0005] To address the problems in existing technologies, this application provides a multimodal large model image segmentation method and apparatus based on hierarchical lexical representation, which can effectively solve the shortcomings of traditional technologies in complex scene understanding, model training and mask generation, and significantly improve the performance of multimodal image segmentation.

[0006] To solve at least one of the above problems, this application provides the following technical solution:

[0007] Firstly, this application provides a multimodal large-model image segmentation method based on hierarchical lexical representation, including:

[0008] The input mask image is encoded into a one-dimensional mask term sequence by a mask marker. The mask marker generates the current mask term using a causal attention mechanism based on the mask block corresponding to the mask image and the generated mask terms. The first term in the mask term sequence represents the position and shape prototype of the target region, and the last term represents the local detail features of the target region. Each mask term in the mask term sequence is conditionally generated based on its preceding mask term. The mask term sequence is input to the vector quantization layer for term quantization to obtain the mask term code. The mask decoder reconstructs the mask image based on the mask term code.

[0009] The mask tokenizer is trained in the first stage by a mask reconstruction task, and in the second stage by a large multimodal model. The mask token codes are expanded into the vocabulary of the large multimodal model. The mask tokens are integrated into the large multimodal model based on low-resolution image data. Joint training is performed based on the segmentation dataset and the general dataset. The mask token sequence is supervised and trained at different levels using a hierarchical mask loss function. The third stage of fine-tuning training is performed using high-resolution image data to obtain the mask semantic segmentation model.

[0010] The system receives the image to be processed and the corresponding target object description from the segmentation request of the referential expression. It inputs the image to be processed and the target object description into the mask semantic segmentation model. The mask semantic segmentation model generates the corresponding mask word sequence based on the target object description. The mask decoder generates the corresponding segmentation mask image based on the mask word sequence. The mask semantic segmentation model identifies the target region referred to by the target object description in the image to be processed based on the segmentation mask image.

[0011] Furthermore, it also includes: dividing the input mask image into blocks to obtain a mask image block sequence, inputting the mask image block sequence together with several fixed latent vectors into the encoder network of the mask tagger for joint self-attention calculation, and obtaining the target mask word after the latent vectors have undergone the above encoding process.

[0012] The encoder network employs conditional autoregressive modeling. Each attention layer exhibits bidirectional attention between mask image blocks and unidirectional attention between latent vectors. Each latent vector is computed based on the mask image block features and all preceding latent vectors. This ensures that the latent vectors can represent the mask image from coarse to fine granular, thereby encoding the latent vectors into an accurate one-dimensional mask term sequence.

[0013] The conditionally generated masked word sequence is input into a vector quantization layer. The vector quantization layer maintains a preset number of codebook vectors. By calculating the Euclidean distance between each masked word in the masked word sequence and the codebook vector, the codebook vector with the smallest distance is selected as the corresponding masked word code. The mask decoder uses a Transformer deep learning model to perform feature mapping and upsampling on the masked word code, and restores the feature mapping result to a reconstructed image with the same resolution as the input masked image.

[0014] Furthermore, it also includes: constructing a mask reconstruction loss function to train the mask tagger, wherein the mask reconstruction loss function includes a reconstruction error term and a codebook commitment term, wherein the reconstruction error term calculates the binary cross-entropy loss and region overlap loss between the reconstructed mask image and the original mask image, wherein the region overlap loss term calculates the cross-union ratio score between the predicted mask and the real mask at different scale levels, wherein the codebook commitment term calculates the Euclidean distance between the quantized features of the mask words and the nearest neighbor codebook vector, and optimizes the mask reconstruction loss function based on the Adam optimizer to update the network parameters and codebook vector of the mask tagger;

[0015] The masked lexical codes are mapped to fixed-dimensional lexical embedding vectors. The word embedding matrix of the large-scale multimodal model is expanded to accommodate the lexical embedding vectors. Training samples containing masked lexical codes are constructed based on low-resolution images. The input sequence of the training samples consists of image feature lexical codes and masked lexical codes. The large-scale multimodal model is trained on the training samples using a cross-entropy loss function and a hierarchical masking loss function, enabling the model to learn the semantic correspondence between masked lexical codes and lexical codes of other modalities. The hierarchical masking loss function supervises the training of the masked lexical sequence at different hierarchical levels. Each hierarchical level consists of several sub-lexical sequences counted from the beginning. The sub-lexical sequences are processed by a masking decoder to obtain the predicted mask for the corresponding level. The loss function at each hierarchical level calculates the binary cross-entropy loss and the region overlap loss term between the predicted mask and the real mask at the corresponding level. The final hierarchical masking loss is equal to the sum of the loss functions at each hierarchical level.

[0016] Furthermore, it also includes: constructing mixed training data batches, each training batch containing a preset ratio of segmentation data samples and general data samples; using high-resolution image data to fine-tune the model after the second stage of training; treating ordinary text and masked words equally in the large-scale multimodal model; and using cross-entropy loss as the training loss function to obtain the final masked semantic segmentation model.

[0017] Furthermore, it also includes: using an image encoder to extract features from the input image to be processed to obtain an image feature sequence, passing the image feature sequence through a multilayer perceptron module to obtain a visual representation sequence aligned with the input layer feature space of the large language model, and inputting the aligned visual representation sequence into the large language model network of the mask semantic segmentation model;

[0018] The large language model network of the mask semantic segmentation model generates a mask word sequence using an autoregressive decoding method. At each decoding time step, the generated mask words are multimodally interacted with the visual feature representation to obtain interaction features. Based on the interaction features, the probability distribution of the next mask word is predicted. Mask words are selected from the probability distribution through a sampling or greedy decoding strategy until a complete mask word sequence is generated.

[0019] Furthermore, it also includes: inputting the generated mask word sequence into a mask decoder, which is a Transformer model with a bidirectional attention mechanism. Along with the mask word sequence, several initial hidden states are also input. After passing through the mask decoder, the initial hidden states map the mask word sequence into an initial feature map. The spatial resolution of the feature map is restored through an upsampling module. Each upsampling module includes a nearest neighbor upsampling layer and a convolutional layer. The upsampled feature map is normalized to obtain a probability mask map. The probability mask map is converted into a binary mask to obtain a segmentation mask image.

[0020] The segmentation mask image is subjected to morphological processing and boundary optimization. The morphological processing includes opening and closing operations on the segmentation mask to remove noise and smooth the boundaries. The boundary optimization is based on image gradient information to locally adjust the segmentation boundaries. The optimized segmentation mask is fused with the original image at the pixel level. The contour and boundary of the target region are marked on the image to be processed to generate a segmentation result image containing the target region identifier.

[0021] Secondly, this application provides a multimodal large-model image segmentation apparatus based on hierarchical lexical representation, comprising:

[0022] The word representation module is used to encode the input mask image into a one-dimensional mask word sequence through a mask marker. The mask marker generates the current mask word based on the mask block corresponding to the mask image and the generated mask words using a causal attention mechanism. The first word of the mask word sequence represents the position and shape prototype of the target region, and the last word represents the local detail features of the target region. Each mask word in the mask word sequence is conditionally generated based on its preceding mask word. The mask word sequence is input to the vector quantization layer for word quantization to obtain the mask word code. The mask decoder reconstructs the mask image based on the mask word code.

[0023] The model training module is used to perform the first stage training of the mask tokenizer through the mask reconstruction task, the second stage training using a large multimodal model, expanding the mask lexical code into the vocabulary of the large multimodal model, integrating the mask lexical into the large multimodal model based on low-resolution image data, jointly training according to the segmentation dataset and the general dataset, using a hierarchical mask loss function to supervise the training of the mask lexical sequence at different levels, and performing the third stage fine-tuning training using high-resolution image data to obtain the mask semantic segmentation model.

[0024] An image segmentation module is used to receive the image to be processed and the corresponding target object description in a segmentation request for a referential expression, input the image to be processed and the target object description into the mask semantic segmentation model, the mask semantic segmentation model generates the corresponding mask lexical sequence based on the target object description, the mask decoder generates the corresponding segmentation mask image based on the mask lexical sequence, and the mask semantic segmentation model identifies the target region referred to by the target object description in the image to be processed based on the segmentation mask image.

[0025] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal large model image segmentation method based on hierarchical lexical representation.

[0026] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal large model image segmentation method based on hierarchical lexical representation.

[0027] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the multimodal large model image segmentation method based on hierarchical lexical representation.

[0028] As described above, this application provides a multimodal large-scale image segmentation method and apparatus based on hierarchical lexical representation. It encodes masked images into lexical sequences using a mask marker and achieves progressive generation from shape prototypes to local details through a causal attention mechanism. A three-stage training strategy is employed: first, the marker is trained through a mask reconstruction task; then, masked lexical sequences are integrated into a large-scale multimodal model for joint training; and finally, high-resolution data is used for fine-tuning. Multi-level supervision is implemented based on a hierarchical mask loss function to achieve accurate segmentation of natural language-described targets. This method effectively addresses the shortcomings of traditional techniques in complex scene understanding, model training, and mask generation, significantly improving the performance of multimodal image segmentation. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating the multimodal large model image segmentation method based on hierarchical lexical representation in the embodiments of this application;

[0031] Figure 2 This is a structural diagram of the multimodal large model image segmentation device based on hierarchical lexical representation in the embodiments of this application;

[0032] Figure 3 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0033] Figure label:

[0034] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0037] To address the problems existing in current technologies, this application provides a multimodal large-scale image segmentation method and apparatus based on hierarchical lexical representation. The method encodes masked images into lexical sequences using a mask marker, and achieves progressive generation from shape prototypes to local details through a causal attention mechanism. A three-stage training strategy is employed: first, the marker is trained through a mask reconstruction task; then, masked lexical representations are integrated into a large-scale multimodal model for joint training; and finally, high-resolution data is used for fine-tuning. Multi-level supervision is implemented based on a hierarchical mask loss function to achieve accurate segmentation of natural language-described targets. This method effectively solves the shortcomings of traditional techniques in complex scene understanding, model training, and mask generation, significantly improving the performance of multimodal image segmentation.

[0038] To effectively address the shortcomings of traditional techniques in complex scene understanding, model training, and mask generation, and to significantly improve the performance of multimodal image segmentation, this application provides an embodiment of a multimodal large-model image segmentation method based on hierarchical lexical representation. See [link to embodiment]. Figure 1 The multimodal large model image segmentation method based on hierarchical lexical representation specifically includes the following:

[0039] Step S101: The input mask image is encoded into a one-dimensional mask term sequence by a mask marker. The mask marker generates the current mask term using a causal attention mechanism based on the mask block corresponding to the mask image and the generated mask terms. The first term in the mask term sequence represents the position and shape prototype of the target region, and the last term represents the local detail features of the target region. Each mask term in the mask term sequence is conditionally generated based on its preceding mask term. The mask term sequence is input to the vector quantization layer for term quantization to obtain the mask term code. The mask decoder reconstructs the mask image based on the mask term code.

[0040] Optionally, this embodiment addresses the challenges of mask representation in image segmentation tasks by innovatively designing a mask encoding method based on hierarchical lexical units. Traditional methods directly model the mask image at the pixel level, making it difficult to effectively capture the hierarchical semantic information of the target region. To address this, this embodiment constructs a dedicated mask marker that converts the two-dimensional mask image into a one-dimensional lexical sequence representation with a clear semantic hierarchy.

[0041] The mask marker in this embodiment first performs uniform block division on the input mask image. The block division results in a sequence of mask blocks, each containing position coordinates and pixel value information.

[0042] This embodiment innovatively employs a causal attention mechanism to generate masked tokens in the masking tokenizer. The attention calculation uses the following formula:

[0043] Attention(Q,K,V)=softmax(QK^T / √d)V

[0044] Where Q, K, and V represent the query, the key-value pair matrix, and d is the dimension of the attention head, respectively. Causal masking ensures that the current position can only notice information from previous positions, thus achieving autoregressive generation. This design allows each masked term to fully utilize the contextual information in already generated terms.

[0045] This embodiment implements a hierarchical encoding strategy for masked lexical sequences. The first few lexical units of the sequence focus on capturing macroscopic features such as the location and shape of the target region. These units obtain a rough outline and spatial distribution information of the target region by downsampling and contour extraction of the masked image. The later lexical units are responsible for encoding local detail features, including microscopic information such as the precise location of boundaries and texture variations. This hierarchical lexical organization is similar to the human visual perception process, first capturing the overall picture and then focusing on the details.

[0046] This embodiment innovatively designs a conditional generation mechanism. The generation of each masked lexical is conditionalized based on its preceding lexical, and a conditional embedding network converts the information of the preceding lexical into a conditional vector. This conditional vector is fused with the features of the current position to guide the generation of new lexicals. This design ensures the coherence of the generation process and maintains semantic consistency between adjacent lexicals.

[0047] This embodiment constructs an efficient vector quantization layer. This layer maintains a dictionary containing K codebook vectors, where K is set according to the complexity of the application scenario. The quantization process calculates the Euclidean distance between the lexical features and the codebook vectors, selecting the nearest neighbor codebook vector as the quantization result. This discretization representation simplifies feature complexity while preserving necessary semantic information.

[0048] This embodiment implements a precise mask reconstruction mechanism. The mask decoder employs a Transformer model with a bidirectional attention mechanism. Along with the mask word sequence, several initial hidden states are input. After passing through the mask decoder, the initial hidden states map the mask word sequence into an initial feature map. The spatial resolution of the feature map is restored through an upsampling module. Each upsampling module includes a nearest neighbor upsampling layer and a convolutional layer. The upsampled feature map is normalized to obtain a probability mask map. The probability mask map is then converted into a binary mask to obtain a segmentation mask image.

[0049] This embodiment effectively addresses the shortcomings of traditional mask representation methods in terms of semantic expression and computational efficiency through the above-mentioned technological innovations. This scheme can compress two-dimensional mask information into a compact one-dimensional sequence while maintaining a clear semantic hierarchy. In practical applications, this representation method significantly reduces storage and computational overhead, and improves the training efficiency of the model.

[0050] This embodiment has significant practical implications for the field of image segmentation. By converting mask information into a sequence of terms, it lays the foundation for subsequent fusion with large-scale language models. This design enables the model to better understand and generate segmentation results that conform to natural language descriptions, providing a new solution for cross-modal image understanding. Through testing and verification in different scenarios, this scheme demonstrates good generalization ability and practical value.

[0051] Step S102: The mask tokenizer is trained in the first stage by the mask reconstruction task, and a large multimodal model is used for the second stage of training. The mask token codes are expanded into the vocabulary of the large multimodal model. The mask tokens are integrated into the large multimodal model based on low-resolution image data. Joint training is performed based on the segmentation dataset and the general dataset. The mask token sequence is supervised and trained at different levels using a hierarchical mask loss function. The third stage of fine-tuning training is performed using high-resolution image data to obtain the mask semantic segmentation model.

[0052] Optionally, this embodiment designs an innovative three-stage training strategy, effectively addressing the challenge of fusing masked terms with large-scale multimodal models. First, a mask reconstruction task is constructed to pre-train the masked tokenizer. The mask reconstruction loss function L_r is defined as:

[0053] L_r=MSE(M_rec,M_orig)+λ1||z_q-sg(e)||2+λ2||sg(z_q)-e||2

[0054] Where M_rec is the reconstruction mask, M_orig is the original mask, z_q is the quantization feature, e is the codebook vector, sg represents the gradient truncation operation, and λ1 and λ2 are balancing parameters. This loss function simultaneously optimizes reconstruction quality and codebook utilization efficiency.

[0055] This embodiment implements an adaptive learning rate adjustment mechanism in the first stage of training. The training process monitors the trend of reconstruction error changes, and when the rate of error decrease slows, the learning rate is adjusted using a cosine annealing strategy. Simultaneously, an early stopping mechanism is introduced, stopping training promptly when performance on the validation set no longer improves, thus avoiding overfitting. This training strategy ensures that the mask tagger can learn a stable and effective mask encoding representation.

[0056] This embodiment innovatively designs a method for integrating masked terms with a large-scale multimodal model. First, the masked term codes are mapped to vector representations with the same dimension as the model's word embeddings. The mapping function F_map employs a multilayer perceptron structure:

[0057] F_map(x) = MLP(Embedding(x))

[0058] Where x represents the lexical code, Embedding is the learnable embedding layer, and MLP is the nonlinear transformation network. This design ensures that the masked lexical code can be represented in the same semantic space as the original lexical code in the model.

[0059] This embodiment implements an efficient training strategy based on low-resolution images. A dedicated training data generation pipeline is constructed to downsample the original images to an appropriate resolution while maintaining the integrity of the masking information. The training samples contain image feature sequences and corresponding masking term sequences. The model learns the semantic correspondence between the two modalities through a masking language modeling task.

[0060] This embodiment innovatively designs a joint training mechanism. Each training batch contains a preset ratio of segmented data and general data, and the joint loss function L_joint is defined as:

[0061] L_joint=αL_mask+γ*L_lm

[0062] Where L_mask is the mask prediction loss, L_lm is the language modeling loss, and α and γ are weight coefficients. This multi-task learning framework ensures both the model's segmentation ability and its general language understanding ability.

[0063] This embodiment constructs a hierarchical mask loss function. The difference between the predicted mask and the true mask is calculated at different spatial scales. The loss function L_hier contains multiple hierarchical terms:

[0064] L_hier=Σw_i*(IoU_i+BCE_i)

[0065] Where i represents different scale levels, w_i is the hierarchical weight, IoU_i measures the region overlap (i.e., Dice Loss), and BCE_i evaluates the accuracy of pixel-level predictions (i.e., BCE Loss). This multi-scale supervision ensures that the model can generate segmentation results of different granularities using mask terms of different lengths.

[0066] This embodiment implements a fine-tuning training strategy for high-resolution images. A progressive learning approach is adopted, first training at a lower resolution and then gradually increasing the image resolution. Each time the resolution is increased, the model parameters from the previous stage are inherited, and the learning rate and batch size are adjusted accordingly. This strategy ensures both training efficiency and the model's ability to handle high-resolution inputs.

[0067] This embodiment effectively addresses the limitations of traditional image segmentation models in semantic understanding and cross-modal fusion through the aforementioned technological innovations. This training scheme enables the model to possess both accurate segmentation capabilities and the ability to understand natural language descriptions, making flexible, interactive image segmentation possible. In practical applications, this scheme significantly improves the model's ability to understand complex scenes and its segmentation accuracy.

[0068] The innovations of this embodiment are mainly reflected in the training strategy design and loss function construction. Through a carefully designed three-stage training process, effective fusion of masked terms and large-scale multimodal models is achieved. This scheme provides a new solution for improving the semantic understanding capability of image segmentation models and has significant application value in the field of computer vision. Experimental verification shows that this training strategy can effectively improve model performance, especially demonstrating significant advantages in complex scenarios requiring accurate semantic understanding.

[0069] Step S103: Receive the image to be processed and the corresponding target object description from the referential expression segmentation request; input the image to be processed and the target object description into the mask semantic segmentation model; the mask semantic segmentation model generates the corresponding mask lexical sequence based on the target object description; the mask decoder generates the corresponding segmentation mask image based on the mask lexical sequence; and the mask semantic segmentation model identifies the target region referred to by the target object description in the image to be processed based on the segmentation mask image.

[0070] Optionally, this embodiment designs a complete processing flow for the language-guided segmentation problem in multimodal image segmentation tasks. In practical application scenarios, users often need to specify the target object to be segmented through natural language descriptions, which requires the model to accurately understand the language descriptions and locate the corresponding regions in the image.

[0071] This embodiment first implements a preprocessing mechanism for multimodal input. Upon receiving a segmentation request, the image to be processed is standardized, including resizing, color space conversion, and pixel value normalization. For the target object description text, a pre-trained language model is used for word segmentation and feature encoding to obtain the semantic representation of the text. This standardized preprocessing ensures the consistency of the input data, laying the foundation for subsequent feature extraction and unified multimodal modeling.

[0072] This embodiment constructs an innovative masked word generation strategy. Based on the fused cross-modal features, the model uses an autoregressive decoding method to generate masked words one by one. During the generation process, a beam search strategy is used to maintain multiple candidate sequences, and the optimal sequence is selected based on the cumulative probability score. This design improves the quality and diversity of the generated results, and can better adapt to different segmentation scenarios.

[0073] This embodiment optimizes the mask decoding process. The mask decoder adopts a multi-scale feature fusion architecture, containing multiple feature reconstruction modules. Each module gradually recovers the spatial resolution of the feature map through transposed convolution operations and preserves detailed information using residual connections. A spatial attention mechanism is also introduced during the decoding process to dynamically adjust the weight distribution of feature reconstruction based on contextual information.

[0074] This embodiment implements a precise boundary optimization mechanism. After generating the initial segmentation mask, a Conditional Random Field (CRF) is used for post-processing. CRF considers the spatial relationships and color similarity between pixels to optimize the smoothness and accuracy of the segmentation boundaries. Simultaneously, edge information from the image is incorporated to locally adjust the segmentation results, improving the accuracy of boundary localization.

[0075] This embodiment designs a visual annotation scheme. The optimized segmentation mask is fused with the original image and displayed, using a semi-transparent overlay to highlight the target region. The boundaries of the target region are outlined to improve visual recognition. Simultaneously, annotation information corresponding to the verbal description is added to the image to facilitate user understanding and verification of the segmentation results.

[0076] This embodiment effectively addresses the shortcomings of traditional image segmentation methods in language understanding and cross-modal alignment through the aforementioned technological innovations. This scheme can accurately understand the user's verbal descriptions and precisely locate the corresponding target regions in the image. In practical applications, this method significantly improves the accuracy and user-friendliness of interactive image segmentation.

[0077] This embodiment is of significant value to the fields of intelligent image editing and visual content understanding. Through language-guided precise segmentation capabilities, it provides a reliable foundation for subsequent image editing, content analysis, and other tasks. This solution demonstrates promising application prospects in professional fields such as medical image analysis and remote sensing image processing, making a significant contribution to improving the level of automation in these areas.

[0078] As described above, the multimodal large-scale model image segmentation method based on hierarchical lexical representation provided in this application can encode masked images into lexical sequences by designing a mask marker, and achieve progressive generation from shape prototypes to local details through a causal attention mechanism. A three-stage training strategy is adopted: first, the marker is trained through a mask reconstruction task; then, the masked lexical sequences are integrated into a large-scale multimodal model and jointly trained; finally, high-resolution data is used for fine-tuning. Multi-level supervision is performed based on a hierarchical mask loss function to achieve accurate segmentation of natural language description targets. This method effectively solves the shortcomings of traditional techniques in complex scene understanding, model training, and mask generation, significantly improving the performance of multimodal image segmentation.

[0079] In one embodiment of the multimodal large model image segmentation method based on hierarchical lexical representation in this application, the method may further include the following:

[0080] Step S201: The input mask image is divided into blocks to obtain a sequence of mask image blocks. This sequence, along with several fixed latent vectors, is then input into the encoder network of the mask tokenizer for joint self-attention computation. After the above encoding process, the latent vectors yield the target mask terms. The encoder network employs conditional autoregressive modeling. Each attention layer exhibits bidirectional attention between mask image blocks and unidirectional attention between latent vectors. This ensures that the latent vectors can represent the mask image from coarse-grained to fine-grained levels, thereby encoding the latent vectors into an accurate one-dimensional mask term sequence.

[0081] Optionally, this embodiment innovatively implements a hierarchical encoding process for mask images. Addressing the limitations of traditional methods that directly process pixel-level masks, a processing scheme based on block segmentation and lexicalization is designed. First, a uniform block strategy is applied to the input mask image, with each block including pixel information and position coordinate information.

[0082] This embodiment innovatively designs a dedicated mask encoder network structure. The network adopts the standard Transformer structure, and the input is a mask image patch and several latent vectors. Each attention layer has bidirectional attention between mask image patches, but unidirectional attention between latent vectors. This ensures that the latent vectors can represent the mask image from coarse to fine granular, so that the latent vectors are encoded into accurate mask terms.

[0083] This embodiment employs an attention masking mechanism. The attention mask matrix M is calculated using the following formula: M = softmax(WQ * WK^T / √d)

[0084] Where WQ and WK are feature projection matrices, and d is the feature dimension. This mechanism enables the model to selectively focus on important image patch features while suppressing interference from irrelevant regions. The distribution of attention weights reflects the importance of different image patches to the target representation.

[0085] This embodiment innovatively implements a combination strategy for word sequences. The combination of preceding and following words takes into account semantic coherence, and the ability to express the mask image from coarse to fine granular is ensured through conditional autoregressive modeling and a hierarchical mask loss function.

[0086] This embodiment effectively overcomes the limitations of traditional mask representation methods in terms of semantic expression and computational efficiency through the above-mentioned technological innovations. This scheme can convert two-dimensional mask information into a one-dimensional sequence with a clear semantic hierarchy, significantly improving the efficiency of subsequent processing. In practical applications, this representation method provides a new solution for the compression and transmission of mask information.

[0087] The innovations of this embodiment are mainly reflected in the block segmentation strategy, feature extraction, and lexical generation. Through a carefully designed processing flow, efficient encoding of mask information is achieved. This scheme provides significant support for improving the performance of image segmentation models and has broad application prospects in the field of computer vision. Experimental verification shows that this method significantly reduces computational and storage overhead while maintaining segmentation accuracy.

[0088] In one embodiment of the multimodal large model image segmentation method based on hierarchical lexical representation in this application, the method may further include the following:

[0089] Step S301: The input mask image is divided into blocks to obtain a sequence of mask image blocks. The sequence of mask image blocks and several fixed latent vectors are then input into the encoder network of the mask tokenizer for joint self-attention calculation. After the above encoding process, the latent vectors yield the target mask terms. The encoder network uses conditional autoregressive modeling. Each attention layer has bidirectional attention between mask image blocks and unidirectional attention between latent vectors. This ensures that the latent vectors can represent the mask image from coarse to fine granular, thus encoding the latent vectors into the accurate one-dimensional mask term sequence.

[0090] Step S302: Input the conditionally generated masked word sequence into the vector quantization layer. The vector quantization layer maintains a preset number of codebook vectors. By calculating the Euclidean distance between each masked word in the masked word sequence and the codebook vector, the codebook vector with the smallest distance is selected as the corresponding masked word code. The mask decoder uses the Transformer deep learning model to perform feature mapping and upsampling on the masked word code, and restores the feature mapping result to a reconstructed image with the same resolution as the input masked image.

[0091] Optionally, this embodiment innovatively designs a conditional autoregressive modeling scheme to address the problem of generating masked lexical sequences. Traditional methods often ignore the dependencies between lexical units, resulting in a lack of coherence in the generated results. To address this, this embodiment constructs a conditional autoregressive model based on the transformer architecture, capturing long-range dependencies between lexical units through a position-aware attention mechanism.

[0092] This embodiment implements a precise positional encoding mechanism. For each position i in the sequence, its d-dimensional positional encoding PE is calculated:

[0093] PE(i,2k)=sin(i / 10000^(2k / d))

[0094] PE(i,2k+1)=cos(i / 10000^(2k / d))

[0095] Where k is the dimension index. This encoding method enables the model to perceive the relative positional relationships of words, which helps to maintain the spatial coherence of the generated sequence.

[0096] This embodiment innovatively designs a conditional autoregressive model for masked word sequences. Each masked word is calculated using a Transformer with a causal attention mechanism based on the masked image block and its preceding masked word subsequence.

[0097] p(m_1,...,m_K|M)=∏p(m_k|M,m_1,...,m_(k-1))

[0098] Where p represents the probability distribution, m_i represents the i-th mask word, K is the total number of mask words, M represents the mask image block, and ∏ represents the cumulative product (k from 1 to K).

[0099] This embodiment constructs an efficient vector quantization layer. This layer maintains a dictionary containing K codebook vectors, with the value of K dynamically set according to the complexity of the application scenario. The quantization process employs nearest neighbor search, selecting the best-matching codebook vector using Euclidean distance. To improve search efficiency, a fast approximate nearest neighbor search based on locality-sensitive hashing is implemented.

[0100] This embodiment effectively addresses the shortcomings of traditional mask generation methods in terms of coherence and accuracy through the aforementioned technological innovations. This scheme can generate semantically coherent and detail-rich mask representations, providing a reliable foundation for subsequent image segmentation tasks. In practical applications, this method significantly improves the quality and efficiency of mask generation.

[0101] The innovations of this embodiment are mainly reflected in conditional modeling, feature quantization, and decoding reconstruction. Through a carefully designed processing flow, high-quality generation and reconstruction of mask information are achieved. This scheme provides significant support for improving the performance of image segmentation models and has broad application prospects in the field of computer vision. Experimental verification shows that this method significantly improves the efficiency and stability of the generation process while maintaining reconstruction quality.

[0102] In one embodiment of the multimodal large model image segmentation method based on hierarchical lexical representation in this application, the method may further include the following:

[0103] Step S401: Construct a mask reconstruction loss function to train the mask tagger. The mask reconstruction loss function includes a reconstruction error term and a codebook commitment term. The reconstruction error term calculates the binary cross-entropy loss and region overlap loss between the reconstructed mask image and the original mask image. The region overlap loss term calculates the cross-union ratio score between the predicted mask and the real mask at different scale levels. The codebook commitment term calculates the Euclidean distance between the quantized features of the mask words and the nearest neighbor codebook vector. Optimize the mask reconstruction loss function based on the Adam optimizer and update the network parameters and codebook vector of the mask tagger.

[0104] Step S402: Map the masked lexical code to a fixed-dimensional lexical embedding vector, expand the word embedding matrix of the large multimodal model to accommodate the lexical embedding vector, construct training samples containing masked lexicals based on low-resolution images, the input sequence of the training samples consists of image feature lexicals and masked lexicals, the large multimodal model is trained on the training samples using a cross-entropy loss function and a hierarchical masking loss function, enabling the model to learn the semantic correspondence between masked lexicals and other modal lexicals, the hierarchical masking loss function supervises the training of the masked lexical sequence at different hierarchical levels, each hierarchical level's lexical sequence is a number of sub-lexical sequences counted from the beginning, the sub-lexical sequences are processed by a masking decoder to obtain the corresponding level's predicted mask, the loss function at each hierarchical level calculates the binary cross-entropy loss and region overlap loss term between the corresponding level's predicted mask and the real mask, the final hierarchical masking loss is equal to the sum of the loss functions at each hierarchical level.

[0105] Optionally, this embodiment innovatively designs a dual-constraint mask reconstruction loss function. This loss function, L_total, consists of two key components:

[0106] L_total=λ_r*L_recon+λ_c*L_commit

[0107] Where L_recon is the reconstruction error term, L_commit is the codebook commitment term, and λ_r and λ_c are balancing factors. This design ensures both reconstruction quality and constrains the distribution of quantization features.

[0108] This embodiment implements adaptive reconstruction error calculation. The reconstruction error is expressed as a weighted mean square error: L_recon=-Σm_i(M_i log(M'_i)+(1-M_i)log(1-M'_i))

[0109] Where M is the target mask, M' is the generated mask, and m is the positional weight. The weight values ​​are dynamically assigned based on the semantic importance of the pixels, allowing the model to focus more on the reconstruction quality of key regions.

[0110] This embodiment optimizes the calculation method for the codebook commitment term. The commitment loss is calculated using the following formula: L_commit=λ1||z_q-sg(e)||2+λ2||sg(z_q)-e||2

[0111] Where z_q represents the features before quantization, e is the nearest neighbor codebook vector, sg represents the gradient truncation operation, and λ1 and λ2 are balancing parameters. This design encourages quantized features to be close to the codebook vector, while avoiding codebook collapse through gradient truncation.

[0112] This embodiment uses the Adam optimizer to update model parameters, and the learning rate is dynamically adjusted according to the training progress.

[0113] lr=lr_0*cos(π*t / T)

[0114] Where lr_0 is the initial learning rate, t is the current step number, and T is the total number of steps. This cosine annealing strategy makes the training process more stable.

[0115] This embodiment implements an efficient lexical mapping mechanism. A learnable mapping network converts masked lexical codes into fixed-dimensional embedding vectors.

[0116] E = MLP(OneHot(code))

[0117] Here, `code` represents the lexical code, and `MLP` stands for Multilayer Perceptron. The parameters of the mapping network are synchronously optimized through backpropagation to ensure that the generated embedding vectors have good semantic representation capabilities.

[0118] This embodiment constructs a dynamic expansion scheme for the word embedding matrix. While keeping the original word embeddings unchanged, a new embedding space is allocated for the masked words. The expansion process uses orthogonal initialization to ensure that the newly added embedding vectors maintain appropriate discriminability with the original embeddings. Simultaneously, a dynamic adjustment mechanism for the embedding space is implemented, adaptively allocating the embedding dimension according to training requirements.

[0119] This embodiment optimizes the training sample construction strategy. Training samples constructed based on low-resolution images contain multiple modalities:

[0120] S = [T_img; T_mask; T_text]

[0121] Where T_img represents image feature terms, T_mask represents mask terms, and T_text represents text terms. The sample construction process considers the temporal correspondence between different modalities to ensure that the model can learn the correct cross-modal mapping.

[0122] This embodiment uses the cross-entropy loss function to optimize the category distribution of masked words:

[0123] L_ce = -Σy_i*log(p_i)

[0124] Where y_i represents the true label and p_i represents the predicted probability. Label smoothing techniques are used to alleviate overfitting and improve the model's generalization ability.

[0125] This embodiment effectively addresses the shortcomings of traditional methods in mask representation learning and multimodal fusion through the above-mentioned technological innovations. This scheme can learn high-quality mask representations and successfully integrate them into large-scale multimodal models. In practical applications, this method significantly improves the model's ability to understand and generate mask information.

[0126] The innovations of this embodiment are mainly reflected in the design of the loss function, parameter optimization, and multimodal modeling. Through a carefully designed training strategy, effective integration of mask information with other modalities is achieved. This approach provides a new solution for improving the semantic understanding capability of image segmentation models and has significant application value in the field of computer vision. Experimental results show that this method can effectively improve the segmentation performance of the model, especially demonstrating a significant advantage in tasks requiring the understanding of complex semantic scenes.

[0127] In one embodiment of the multimodal large model image segmentation method based on hierarchical lexical representation in this application, the method may further include the following:

[0128] Step S501: Construct mixed training data batches. Each training batch contains segmentation data samples and general data samples with a preset ratio. High-resolution image data is used to fine-tune the model after the second stage of training. The large multimodal model treats ordinary text and masked words equally. The training loss function adopts cross-entropy loss, thereby obtaining the final masked semantic segmentation model.

[0129] Optionally, this embodiment innovatively designs a hybrid batch training strategy. In each training batch, the split data and general data are mixed in a ratio of r:(1-r), where r is a dynamically adjusted mixing ratio: r=r_0*(1+α*cos(2πt / T))

[0130] Where r_0 is the basic mixing ratio, t is the current iteration number, T is the total number of iterations, and α is the adjustment coefficient. This periodic adjustment strategy enables the model to achieve a balance between specialized segmentation capabilities and general language understanding capabilities.

[0131] This embodiment implements a multi-task joint training mechanism. The joint training objective function L_joint is constructed as follows: L_joint = αL_mask + γ*L_lm

[0132] Where L_mask is the mask prediction loss, L_lm is the language modeling loss, and α and γ are weight coefficients. This multi-task learning framework ensures both the model's segmentation ability and its general language understanding capability.

[0133] This embodiment innovatively designs a mask reconstruction loss. Pixel-level differences are calculated using a binary cross-entropy loss function:

[0134] L_recon=-Σm_i(M_i log(M'_i)+(1-M_i)log(1-M'_i))

[0135] Where M is the target mask, M' is the generated mask, and m is the positional weight. The weight values ​​are dynamically assigned based on the semantic importance of the pixels, allowing the model to focus more on the reconstruction quality of key regions.

[0136] This embodiment implements a hierarchical masking loss function. This loss function, L_hier, comprehensively considers monitoring signals from multiple levels:

[0137] L_hier=Σw_i*(IoU_i+BCE_i)

[0138] Where i represents different scale levels, w_i is the hierarchical weight, IoU_i measures the region overlap (i.e., Dice Loss), and BCE_i evaluates the accuracy of pixel-level predictions (i.e., BCE Loss). This multi-scale supervision ensures that the model can generate segmentation results of different granularities using mask terms of different lengths.

[0139] This embodiment constructs an adaptive IoU calculation mechanism. At each scale level, the region overlap is calculated using soft IoU:

[0140] IoU=(P∩G+ε) / (P∪G+ε)

[0141] Where P is the predicted mask, G is the true mask, and ε is the smoothing factor. The use of soft IoU makes the loss function differentiable, which facilitates backpropagation optimization.

[0142] This embodiment optimizes the method for evaluating boundary accuracy. The boundary distance metric uses a weighted Hausdorff distance:

[0143] D_b=max{sup_x∈P inf_y∈G d(x,y),sup_y∈G inf_x∈P d(x,y)}

[0144] Where d(x,y) is the Euclidean distance between pixels. This metric can accurately reflect the degree of deviation of the segmentation boundary.

[0145] This embodiment implements an efficient fine-tuning training strategy. When fine-tuning on high-resolution data, a progressive learning scheme is adopted, gradually increasing the resolution of the input image. Each resolution stage inherits the model parameters from the previous stage, and the learning rate and batch size are adjusted accordingly. This strategy ensures both training efficiency and model performance.

[0146] This embodiment effectively addresses the shortcomings of traditional image segmentation models in multi-task learning and refined segmentation through the above-mentioned technological innovations. This scheme can simultaneously balance segmentation accuracy and model versatility, providing a new solution for achieving high-quality semantic segmentation. In practical applications, this method significantly improves the model's segmentation performance and generalization ability.

[0147] The innovations of this embodiment are mainly reflected in the training strategy design and loss function construction. Through a carefully designed training process and supervision mechanism, a comprehensive improvement in model performance is achieved. This scheme provides crucial support for improving the accuracy and robustness of image segmentation models and has broad application prospects in the field of computer vision. Experimental verification shows that this method can effectively improve model performance, especially demonstrating significant advantages in complex scenes requiring accurate segmentation.

[0148] In one embodiment of the multimodal large model image segmentation method based on hierarchical lexical representation in this application, the method may further include the following:

[0149] Step S601: Use an image encoder to extract features from the input image to be processed to obtain an image feature sequence. Pass the image feature sequence through a multilayer perceptron module to obtain a visual representation sequence aligned with the input layer feature space of the large language model. Input the aligned visual representation sequence into the large language model network of the mask semantic segmentation model.

[0150] Step S602: The large language model network of the mask semantic segmentation model generates a mask word sequence using an autoregressive decoding method. For each decoding time step, the generated mask words are multimodally interacted with the visual feature representation to obtain interaction features. Based on the interaction features, the probability distribution of the next mask word is predicted. Mask words are selected from the probability distribution through a sampling or greedy decoding strategy until a complete mask word sequence is generated.

[0151] The decoder in this embodiment adopts an autoregressive transformer architecture, containing multiple layers of self-attention and cross-modal attention modules. The computation process of each layer includes feature update and normalization:

[0152] H'=LayerNorm(H+MultiHead(H))

[0153] Here, H represents the hidden state features, and MultiHead represents the multi-head attention operation. This design enables the model to effectively integrate multimodal information.

[0154] This embodiment implements an adaptive masked term generation mechanism. At each decoding time step, the generated masked term sequence S_t is interacted with the visual feature V in a multimodal manner:

[0155] F_inter = Interaction(S_t, V)

[0156] The Interaction module includes attention calculation and feature fusion. Interaction features carry rich contextual information, providing guidance for subsequent lexical generation.

[0157] This embodiment constructs a probabilistic prediction network. It predicts the probability distribution of the next mask word based on interaction features:

[0158] P(t+1)=softmax(MLP(F_inter) / τ)

[0159] Here, MLP stands for Multilayer Perceptron, and τ is the temperature parameter. Adjusting the temperature parameter affects the diversity of sampling; lower temperatures tend to select higher-probability words.

[0160] This embodiment optimizes the decoding strategy selection mechanism. It dynamically selects between sampling and greedy decoding based on task requirements: token = argmax(P) # Greedy decoding

[0161] token ~ multinomial(P) # random sampling

[0162] Greedy decoding is suitable for scenarios that require deterministic results, while random sampling helps to generate diverse segmentation results.

[0163] This embodiment effectively addresses the shortcomings of traditional image segmentation methods in semantic understanding and feature fusion through the above-mentioned technological innovations. This scheme can accurately understand text descriptions and locate and segment corresponding target regions in images. In practical applications, this method significantly improves the accuracy of interactive image segmentation and the user experience.

[0164] The innovations of this embodiment are mainly reflected in feature alignment, decoding mechanism, and strategy selection. Through a carefully designed processing flow, accurate segmentation guided by text is achieved. This scheme provides a new solution for improving the semantic understanding capability of image segmentation models and has significant application value in the field of computer vision. Experimental verification shows that this method can effectively improve segmentation performance, especially demonstrating significant advantages in complex scenes requiring accurate semantic understanding.

[0165] In one embodiment of the multimodal large model image segmentation method based on hierarchical lexical representation in this application, the method may further include the following:

[0166] Step S701: Input the generated mask word sequence into the mask decoder. The mask decoder is a Transformer model with a bidirectional attention mechanism. Along with the mask word sequence, several initial hidden states are also input. After the initial hidden states pass through the mask decoder, the mask word sequence is mapped to an initial feature map. The spatial resolution of the feature map is restored through an upsampling module. Each upsampling module contains a nearest neighbor upsampling layer and a convolutional layer. The upsampled feature map is normalized to obtain a probability mask map. The probability mask map is converted into a binary mask to obtain a segmentation mask image.

[0167] Step S702: Perform morphological processing and boundary optimization on the segmentation mask image. The morphological processing includes performing opening and closing operations on the segmentation mask to remove noise and smooth the boundaries. The boundary optimization is based on image gradient information to locally adjust the segmentation boundaries. The optimized segmentation mask is fused with the original image at the pixel level. The contour and boundary of the target region are marked on the image to be processed to generate a segmentation result image containing the target region identifier.

[0168] Optionally, this embodiment implements an innovative mask reconstruction mechanism. The input to the mask decoder is a sequence of masked terms and several initial hidden state sequences. The generation of the initial feature map is accomplished through a bidirectional attention Transformer:

[0169] F_init=Transformer(Concat(tokens,Embed))

[0170] Where tokens is the masked word sequence, Embed is the initial hidden state sequence, and Concat is the sequence concatenation operation. F_init is the feature corresponding to Embed after passing through the Transformer.

[0171] The upsampling module used in this embodiment is the nearest neighbor upsampling and convolution module. The upsampling operation is represented as: F_up = Conv(Upsample(F_in))

[0172] Here, F_in represents the input features, and Upsample and Conv represent the nearest neighbor upsampling layer and convolutional layer, respectively. This design ensures smooth upsampling of features while maintaining the transmission of detailed information.

[0173] This embodiment implements a probability mask generation strategy. The feature map is converted into a probability distribution after normalization:

[0174] P_mask = sigmoid(F_final)

[0175] Where F_final is the final feature map. The probability mask reflects the confidence that each pixel belongs to the target region, providing a reliable basis for subsequent binarization.

[0176] This embodiment constructs an adaptive binarization method. The binarization of the probability mask adopts a dynamic threshold strategy: M_binary = threshold(P_mask,t)

[0177] Here, t is an adaptive threshold, dynamically determined based on the probability distribution using the OTSU algorithm. This method can better adapt to the segmentation needs of different scenarios.

[0178] This embodiment optimizes the morphological processing workflow. The kernel size for opening and closing operations is dynamically adjusted according to the image resolution.

[0179] k=max(3,round(min(H,W) / 100))

[0180] Where H and W represent the image height and width. Morphological operations effectively remove noise and holes in the segmentation results, improving the quality of the segmentation mask.

[0181] This embodiment implements a precise boundary optimization mechanism. A boundary energy function is constructed based on image gradient information:

[0182] in Let be the image gradient, smoothness be the smoothing term, and w be the weighting coefficient. The segmentation boundary is optimized by minimizing the energy function to better fit the target contour.

[0183] This embodiment innovatively designs a mask fusion strategy. The optimized segmentation mask is alpha-blended with the original image:

[0184] I_result=α*I_orig+(1-α)*M_color

[0185] Where I_orig is the original image, M_color is the colored mask, and α is the transparency parameter. This visualization method intuitively shows the segmentation results.

[0186] This embodiment effectively addresses the shortcomings of traditional segmentation methods in mask reconstruction and boundary optimization through the above-mentioned technological innovations. This scheme can generate high-quality segmentation results, and further improves the segmentation quality through post-processing. In practical applications, this method significantly improves the visual quality and usability of the segmentation results.

[0187] The innovations of this embodiment are mainly reflected in feature reconstruction, boundary optimization, and visualization. Through a carefully designed processing flow, a high-quality conversion from masked terms to accurate segmentation results is achieved. This scheme provides significant support for improving the performance of image segmentation models and has broad application prospects in the field of computer vision. Experimental verification shows that this method can generate segmentation results with accurate boundaries and excellent visual effects, especially demonstrating significant advantages in application scenarios requiring fine segmentation.

[0188] To effectively address the shortcomings of traditional techniques in complex scene understanding, model training, and mask generation, and to significantly improve the performance of multimodal image segmentation, this application provides an embodiment of a multimodal large model image segmentation apparatus based on hierarchical lexical representation for implementing all or part of the aforementioned multimodal large model image segmentation method based on hierarchical lexical representation. See [link to embodiment]. Figure 2 The multimodal large model image segmentation device based on hierarchical lexical representation specifically includes the following components:

[0189] The word representation module 10 is used to encode the input mask image into a one-dimensional mask word sequence through a mask marker. The mask marker generates the current mask word using a causal attention mechanism based on the mask block corresponding to the mask image and the generated mask words. The first word in the mask word sequence represents the position and shape prototype of the target region, and the last word represents the local detail features of the target region. Each mask word in the mask word sequence is conditionally generated based on its preceding mask word. The mask word sequence is input to the vector quantization layer for word quantization to obtain the mask word code. The mask decoder reconstructs the mask image based on the mask word code.

[0190] The model training module 20 is used to perform a first-stage training of the mask tokenizer through a mask reconstruction task, a second-stage training using a large multimodal model, expand the mask lexical code into the vocabulary of the large multimodal model, integrate the mask lexical into the large multimodal model based on low-resolution image data, perform joint training based on the segmentation dataset and the general dataset, supervise the training of the mask lexical sequence at different levels using a hierarchical mask loss function, and perform a third-stage fine-tuning training using high-resolution image data to obtain the mask semantic segmentation model.

[0191] The image segmentation module 30 is used to receive the image to be processed and the corresponding target object description in the referential expression segmentation request, input the image to be processed and the target object description into the mask semantic segmentation model, the mask semantic segmentation model generates the corresponding mask word sequence based on the target object description, the mask decoder generates the corresponding segmentation mask image based on the mask word sequence, and the mask semantic segmentation model identifies the target region referred to by the target object description in the image to be processed based on the segmentation mask image.

[0192] As described above, the multimodal large-scale image segmentation device based on hierarchical lexical representation provided in this application can encode masked images into lexical sequences by designing a mask marker, and achieve progressive generation from shape prototypes to local details through a causal attention mechanism. A three-stage training strategy is adopted: first, the marker is trained through a mask reconstruction task; then, masked lexical representations are integrated into a large-scale multimodal model and jointly trained; finally, high-resolution data is used for fine-tuning. Multi-level supervision is performed based on a hierarchical mask loss function to achieve accurate segmentation of natural language-described targets. This method effectively solves the shortcomings of traditional techniques in complex scene understanding, model training, and mask generation, significantly improving the performance of multimodal image segmentation.

[0193] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in complex scene understanding, model training, and mask generation, and significantly improve the performance of multimodal image segmentation, this application provides an embodiment of an electronic device for implementing all or part of the multimodal large model image segmentation method based on hierarchical lexical representation. The electronic device specifically includes the following components:

[0194] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the multimodal large-scale image segmentation device based on hierarchical lexical representation and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the multimodal large-scale image segmentation method based on hierarchical lexical representation and the embodiments of the multimodal large-scale image segmentation device based on hierarchical lexical representation in the embodiments, the contents of which are incorporated herein, and repeated details will not be described again.

[0195] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0196] In practical applications, parts of the multimodal large-model image segmentation method based on hierarchical lexical representation can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific choice depends on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations in this regard. If all operations are completed in the client device, the client device may further include a processor.

[0197] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0198] Figure 3 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 3 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0199] In one embodiment, the functionality of the multimodal large model image segmentation method based on hierarchical lexical representation can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0200] Step S101: The input mask image is encoded into a one-dimensional mask term sequence by a mask marker. The mask marker generates the current mask term using a causal attention mechanism based on the mask block corresponding to the mask image and the generated mask terms. The first term in the mask term sequence represents the position and shape prototype of the target region, and the last term represents the local detail features of the target region. Each mask term in the mask term sequence is conditionally generated based on its preceding mask term. The mask term sequence is input to the vector quantization layer for term quantization to obtain the mask term code. The mask decoder reconstructs the mask image based on the mask term code.

[0201] Step S102: The mask tokenizer is trained in the first stage by the mask reconstruction task, and a large multimodal model is used for the second stage of training. The mask token codes are expanded into the vocabulary of the large multimodal model. The mask tokens are integrated into the large multimodal model based on low-resolution image data. Joint training is performed based on the segmentation dataset and the general dataset. The mask token sequence is supervised and trained at different levels using a hierarchical mask loss function. The third stage of fine-tuning training is performed using high-resolution image data to obtain the mask semantic segmentation model.

[0202] Step S103: Receive the image to be processed and the corresponding target object description from the referential expression segmentation request; input the image to be processed and the target object description into the mask semantic segmentation model; the mask semantic segmentation model generates the corresponding mask lexical sequence based on the target object description; the mask decoder generates the corresponding segmentation mask image based on the mask lexical sequence; and the mask semantic segmentation model identifies the target region referred to by the target object description in the image to be processed based on the segmentation mask image.

[0203] As described above, the electronic device provided in this application encodes a mask image into a sequence of terms by designing a mask marker, and achieves progressive generation from shape prototype to local details through a causal attention mechanism. A three-stage training strategy is adopted: first, the marker is trained through a mask reconstruction task; then, the mask terms are integrated into a large-scale multimodal model and jointly trained; finally, high-resolution data is used for fine-tuning. Multi-level supervision is performed based on a hierarchical mask loss function to achieve accurate segmentation of natural language description targets. This method effectively solves the shortcomings of traditional techniques in complex scene understanding, model training, and mask generation, significantly improving the performance of multimodal image segmentation.

[0204] In another embodiment, the multimodal large model image segmentation device based on hierarchical lexical representation can be configured separately from the central processing unit 9100. For example, the multimodal large model image segmentation device based on hierarchical lexical representation can be configured as a chip connected to the central processing unit 9100, and the function of the multimodal large model image segmentation method based on hierarchical lexical representation can be realized through the control of the central processing unit.

[0205] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 3 All components shown; in addition, the electronic device 9600 may also include Figure 3 For components not shown, please refer to existing technologies.

[0206] like Figure 3 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0207] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0208] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0209] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0210] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0211] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0212] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0213] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the multimodal large-scale image segmentation method based on hierarchical lexical representation, where the execution subject is a server or client, as described in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the multimodal large-scale image segmentation method based on hierarchical lexical representation, where the execution subject is a server or client, as described in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0214] Step S101: The input mask image is encoded into a one-dimensional mask term sequence by a mask marker. The mask marker generates the current mask term using a causal attention mechanism based on the mask block corresponding to the mask image and the generated mask terms. The first term in the mask term sequence represents the position and shape prototype of the target region, and the last term represents the local detail features of the target region. Each mask term in the mask term sequence is conditionally generated based on its preceding mask term. The mask term sequence is input to the vector quantization layer for term quantization to obtain the mask term code. The mask decoder reconstructs the mask image based on the mask term code.

[0215] Step S102: The mask tokenizer is trained in the first stage by the mask reconstruction task, and a large multimodal model is used for the second stage of training. The mask token codes are expanded into the vocabulary of the large multimodal model. The mask tokens are integrated into the large multimodal model based on low-resolution image data. Joint training is performed based on the segmentation dataset and the general dataset. The mask token sequence is supervised and trained at different levels using a hierarchical mask loss function. The third stage of fine-tuning training is performed using high-resolution image data to obtain the mask semantic segmentation model.

[0216] Step S103: Receive the image to be processed and the corresponding target object description from the referential expression segmentation request; input the image to be processed and the target object description into the mask semantic segmentation model; the mask semantic segmentation model generates the corresponding mask lexical sequence based on the target object description; the mask decoder generates the corresponding segmentation mask image based on the mask lexical sequence; and the mask semantic segmentation model identifies the target region referred to by the target object description in the image to be processed based on the segmentation mask image.

[0217] As described above, the computer-readable storage medium provided in this application encodes a mask image into a sequence of terms by designing a mask marker, and achieves progressive generation from shape prototypes to local details through a causal attention mechanism. A three-stage training strategy is employed: first, the marker is trained through a mask reconstruction task; then, the mask terms are integrated into a large-scale multimodal model and jointly trained; finally, high-resolution data is used for fine-tuning. Multi-level supervision is performed based on a hierarchical mask loss function to achieve accurate segmentation of natural language-described targets. This method effectively addresses the shortcomings of traditional techniques in complex scene understanding, model training, and mask generation, significantly improving the performance of multimodal image segmentation.

[0218] Embodiments of this application also provide a computer program product capable of implementing all steps of the multimodal large model image segmentation method based on hierarchical lexical representation, where the execution subject is a server or client, as described in the above embodiments. When executed by a processor, this computer program / instruction implements the steps of the multimodal large model image segmentation method based on hierarchical lexical representation. For example, the computer program / instruction implements the following steps:

[0219] Step S101: The input mask image is encoded into a one-dimensional mask term sequence by a mask marker. The mask marker generates the current mask term using a causal attention mechanism based on the mask block corresponding to the mask image and the generated mask terms. The first term in the mask term sequence represents the position and shape prototype of the target region, and the last term represents the local detail features of the target region. Each mask term in the mask term sequence is conditionally generated based on its preceding mask term. The mask term sequence is input to the vector quantization layer for term quantization to obtain the mask term code. The mask decoder reconstructs the mask image based on the mask term code.

[0220] Step S102: The mask tokenizer is trained in the first stage by the mask reconstruction task, and a large multimodal model is used for the second stage of training. The mask token codes are expanded into the vocabulary of the large multimodal model. The mask tokens are integrated into the large multimodal model based on low-resolution image data. Joint training is performed based on the segmentation dataset and the general dataset. The mask token sequence is supervised and trained at different levels using a hierarchical mask loss function. The third stage of fine-tuning training is performed using high-resolution image data to obtain the mask semantic segmentation model.

[0221] Step S103: Receive the image to be processed and the corresponding target object description from the referential expression segmentation request; input the image to be processed and the target object description into the mask semantic segmentation model; the mask semantic segmentation model generates the corresponding mask lexical sequence based on the target object description; the mask decoder generates the corresponding segmentation mask image based on the mask lexical sequence; and the mask semantic segmentation model identifies the target region referred to by the target object description in the image to be processed based on the segmentation mask image.

[0222] As described above, the computer program product provided in this application encodes a mask image into a sequence of terms by designing a mask marker, and achieves progressive generation from shape prototypes to local details through a causal attention mechanism. A three-stage training strategy is adopted: first, the marker is trained through a mask reconstruction task; then, the mask terms are integrated into a large-scale multimodal model and jointly trained; finally, high-resolution data is used for fine-tuning. Multi-level supervision is performed based on a hierarchical mask loss function to achieve accurate segmentation of natural language description targets. This method effectively solves the shortcomings of traditional techniques in complex scene understanding, model training, and mask generation, significantly improving the performance of multimodal image segmentation.

[0223] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0224] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0225] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0226] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0227] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A multimodal large-scale image segmentation method based on hierarchical lexical representation, characterized in that, The method includes: The input mask image is encoded into a one-dimensional mask term sequence by a mask marker. The mask marker generates the current mask term using a causal attention mechanism based on the mask block corresponding to the mask image and the generated mask terms. The first term in the mask term sequence represents the position and shape prototype of the target region, and the last term represents the local detail features of the target region. Each mask term in the mask term sequence is conditionally generated based on its preceding mask term. The mask term sequence is input to the vector quantization layer for term quantization to obtain the mask term code. The mask decoder reconstructs the mask image based on the mask term code. The mask tokenizer is trained in the first stage by a mask reconstruction task, and in the second stage by a large multimodal model. The mask token codes are expanded into the vocabulary of the large multimodal model. The mask tokens are integrated into the large multimodal model based on low-resolution image data. Joint training is performed based on the segmentation dataset and the general dataset. The mask token sequence is supervised and trained at different levels using a hierarchical mask loss function. The third stage of fine-tuning training is performed using high-resolution image data to obtain the mask semantic segmentation model. The system receives the image to be processed and the corresponding target object description from the segmentation request of the referential expression. It inputs the image to be processed and the target object description into the mask semantic segmentation model. The mask semantic segmentation model generates the corresponding mask word sequence based on the target object description. The mask decoder generates the corresponding segmentation mask image based on the mask word sequence. The mask semantic segmentation model identifies the target region referred to by the target object description in the image to be processed based on the segmentation mask image.

2. The multimodal large model image segmentation method based on hierarchical lexical representation according to claim 1, characterized in that, The input mask image is encoded into a one-dimensional mask term sequence using a mask marker. The mask marker generates the current mask term using a causal attention mechanism based on the mask block corresponding to the mask image and the previously generated mask terms. The first term in the mask term sequence represents the position and shape prototype of the target region, and the last term represents the local detail features of the target region, including: The input mask image is divided into blocks to obtain a sequence of mask image blocks. This sequence of mask image blocks, along with several fixed latent vectors, is then input into the encoder network of the mask tokenizer for joint self-attention computation. After the latent vectors undergo the above encoding process, the target mask words are obtained. The encoder network employs conditional autoregressive modeling. Each attention layer exhibits bidirectional attention between mask image blocks and unidirectional attention between latent vectors. This ensures that the latent vectors can represent the mask image from coarse to fine granular, thereby encoding the latent vectors into an accurate one-dimensional mask term sequence.

3. The multimodal large-scale image segmentation method based on hierarchical lexical representation according to claim 1, characterized in that, Each masking term in the masking term sequence is input to a vector quantization layer for term quantization to obtain a masking term code. The masking decoder reconstructs the masked image based on the masking term code, including: The conditionally generated masked word sequence is input into a vector quantization layer. The vector quantization layer maintains a preset number of codebook vectors. By calculating the Euclidean distance between each masked word in the masked word sequence and the codebook vector, the codebook vector with the smallest distance is selected as the corresponding masked word code. The mask decoder uses a Transformer deep learning model to perform feature mapping and upsampling on the masked word code, and restores the feature mapping result to a reconstructed image with the same resolution as the input masked image.

4. The multimodal large model image segmentation method based on hierarchical lexical representation according to claim 1, characterized in that, The process of training the mask tokenizer in the first stage using a mask reconstruction task, and then training it in the second stage using a large-scale multimodal model, expanding the vocabulary of the large-scale multimodal model with the mask lexical codes, and integrating the mask lexical codes into the large-scale multimodal model based on low-resolution image data, includes: A mask reconstruction loss function is constructed to train the mask tagger. The mask reconstruction loss function includes a reconstruction error term and a codebook commitment term. The reconstruction error term calculates the binary cross-entropy loss and region overlap loss between the reconstructed mask image and the original mask image. The region overlap loss term calculates the cross-union ratio score between the predicted mask and the real mask at different scale levels. The codebook commitment term calculates the Euclidean distance between the quantized features of the mask words and the nearest neighbor codebook vector. The mask reconstruction loss function is optimized based on the Adam optimizer to update the network parameters and codebook vector of the mask tagger. The masked lexical codes are mapped to fixed-dimensional lexical embedding vectors. The word embedding matrix of the large-scale multimodal model is expanded to accommodate the lexical embedding vectors. Training samples containing masked lexical codes are constructed based on low-resolution images. The input sequence of the training samples consists of image feature lexical codes and masked lexical codes. The large-scale multimodal model is trained on the training samples using a cross-entropy loss function and a hierarchical masking loss function, enabling the model to learn the semantic correspondence between masked lexical codes and lexical codes of other modalities. The hierarchical masking loss function supervises the training of the masked lexical sequence at different hierarchical levels. Each hierarchical level consists of several sub-lexical sequences counted from the beginning. The sub-lexical sequences are processed by a masking decoder to obtain the predicted mask for the corresponding level. The loss function at each hierarchical level calculates the binary cross-entropy loss and the region overlap loss term between the predicted mask and the real mask at the corresponding level. The final hierarchical masking loss is equal to the sum of the loss functions at each hierarchical level.

5. The multimodal large model image segmentation method based on hierarchical lexical representation according to claim 1, characterized in that, The process of jointly training the segmentation dataset and the general dataset, followed by a third-stage fine-tuning training using high-resolution image data, yields a masked semantic segmentation model, including: A mixed training data batch is constructed, with each batch containing a preset ratio of segmentation data samples and general data samples. High-resolution image data is used to fine-tune the model after the second stage of training. The large-scale multimodal model treats ordinary text and masked words equally. The training loss function adopts cross-entropy loss, thereby obtaining the final masked semantic segmentation model.

6. The multimodal large model image segmentation method based on hierarchical lexical representation according to claim 1, characterized in that, The process of receiving the image to be processed and the corresponding target object description from the referential expression segmentation request, inputting the image to be processed and the target object description into the mask semantic segmentation model, and the mask semantic segmentation model generating the corresponding mask lexical sequence based on the target object description includes: An image encoder is used to extract features from the input image to be processed to obtain an image feature sequence. The image feature sequence is then passed through a multilayer perceptron module to obtain a visual representation sequence aligned with the input layer feature space of the large language model. The aligned visual representation sequence is then input into the large language model network of the mask semantic segmentation model. The large language model network of the mask semantic segmentation model generates a mask word sequence using an autoregressive decoding method. At each decoding time step, the generated mask words are multimodally interacted with the visual feature representation to obtain interaction features. Based on the interaction features, the probability distribution of the next mask word is predicted. Mask words are selected from the probability distribution through a sampling or greedy decoding strategy until a complete mask word sequence is generated.

7. The multimodal large-model image segmentation method based on hierarchical lexical representation according to claim 1, characterized in that, The mask decoder generates a corresponding segmentation mask image based on the mask term sequence, and the mask semantic segmentation model identifies the target region referred to by the target object description in the image to be processed based on the segmentation mask image, including: The generated masked word sequence is input into the mask decoder, which is a Transformer model with a bidirectional attention mechanism. Along with the masked word sequence, several initial hidden states are also input. After passing through the mask decoder, the initial hidden states map the masked word sequence into an initial feature map. The spatial resolution of the feature map is restored through an upsampling module. Each upsampling module contains a nearest neighbor upsampling layer and a convolutional layer. The upsampled feature map is normalized to obtain a probability mask map. The probability mask map is then converted into a binary mask to obtain a segmentation mask image. The segmentation mask image is subjected to morphological processing and boundary optimization. The morphological processing includes opening and closing operations on the segmentation mask to remove noise and smooth the boundaries. The boundary optimization is based on image gradient information to locally adjust the segmentation boundaries. The optimized segmentation mask is fused with the original image at the pixel level. The contour and boundary of the target region are marked on the image to be processed to generate a segmentation result image containing the target region identifier.

8. A multimodal large-scale image segmentation device based on hierarchical lexical representation, characterized in that, The device includes: The word representation module is used to encode the input mask image into a one-dimensional mask word sequence through a mask marker. The mask marker generates the current mask word based on the mask block corresponding to the mask image and the generated mask words using a causal attention mechanism. The first word of the mask word sequence represents the position and shape prototype of the target region, and the last word represents the local detail features of the target region. Each mask word in the mask word sequence is conditionally generated based on its preceding mask word. The mask word sequence is input to the vector quantization layer for word quantization to obtain the mask word code. The mask decoder reconstructs the mask image based on the mask word code. The model training module is used to perform the first stage training of the mask tokenizer through the mask reconstruction task, the second stage training using a large multimodal model, expanding the mask lexical code into the vocabulary of the large multimodal model, integrating the mask lexical into the large multimodal model based on low-resolution image data, jointly training according to the segmentation dataset and the general dataset, using a hierarchical mask loss function to supervise the training of the mask lexical sequence at different levels, and performing the third stage fine-tuning training using high-resolution image data to obtain the mask semantic segmentation model. An image segmentation module is used to receive the image to be processed and the corresponding target object description in a segmentation request for a referential expression, input the image to be processed and the target object description into the mask semantic segmentation model, the mask semantic segmentation model generates the corresponding mask lexical sequence based on the target object description, the mask decoder generates the corresponding segmentation mask image based on the mask lexical sequence, and the mask semantic segmentation model identifies the target region referred to by the target object description in the image to be processed based on the segmentation mask image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal large model image segmentation method based on hierarchical lexical representation as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the multimodal large model image segmentation method based on hierarchical lexical representation as described in any one of claims 1 to 7.

Images