Image segmentation method based on Qianwen large model to guide backbone network

Abstract

The present invention discloses a method for image segmentation based on the guidance of the backbone network by the Qianwen large model, comprising: inputting the document image in the training set into the UNet backbone segmentation network to obtain a feature map; inputting the document image and the manually designed prompt into the Qianwen 2 multimodal large model to obtain the feature information of its hidden layer; putting the feature information into the deformable agent attention mechanism for fusion to obtain more complete information of the fusion of semantic and visual features; putting the learnable query vector and the fusion information into the decoder layer, etc. The present invention accelerates the convergence speed of the model and makes the model training more stable. It can not only reduce the computational overhead and the use of video memory, but also enhance the flexibility of the agent acquisition, so that the model can learn reasonable and useful knowledge more adaptively and obtain more flexible attention. Through the guidance of Qianwen, the performance of the backbone segmentation network is effectively improved, and the document image is segmented more clearly.

Description

Technical Field

[0001] The present invention belongs to the field of document image segmentation, and specifically relates to a method for image segmentation based on a Qianwen large model to guide a backbone network. Background Art

[0002] With the rapid development of internet technology and the deepening of digital transformation, documents, as the core medium for information storage and transmission in human society, have undergone profound changes in their form and application. Whether paper documents, electronic contracts, scientific research papers, or multimodal composite documents containing images, tables, and text, they are all experiencing exponential growth. According to International Data Corporation (IDC), the total global data volume is expected to reach 175 zettabytes by 2025, of which unstructured document data will account for over 80% (IDC, 2023). Faced with the demand for storing, retrieving, and analyzing massive amounts of documents, traditional manual processing methods are no longer efficient and accurate enough. For example, the financial industry processes hundreds of millions of invoices annually, yet manual review has an error rate as high as 3-5% and can take weeks (Gartner, 2024). Against this backdrop, document analysis and recognition (DAR) technology, as a key bridge connecting unstructured data with computer intelligence, has become a research hotspot in the fields of artificial intelligence and computer vision.

[0003] The development of document analysis technology dates back to the mid-20th century. Early research focused on single-character recognition (CR). For example, in 1954, the IBM 726 system recognized check digits through template matching, marking the beginning of optical character recognition (OCR) technology. By the 1980s, with the rise of statistical learning methods, the combination of feature engineering techniques such as HOG and SIFT with support vector machine (SVM) classifiers pushed document analysis beyond single-character recognition to encompass simpler structures. Typical applications include newspaper digitization and license plate recognition. In the 21st century, breakthroughs in deep learning technology revolutionized the document analysis paradigm. The AlexNet model, proposed by Hinton's team in the 2012 ImageNet competition, boosted image recognition accuracy to 85% through end-to-end feature learning, providing new insights into document layout analysis and the parsing of complex structures. Currently, document analysis technology has formed a complete processing chain: from image preprocessing, physical layout analysis, content recognition, to semantic information extraction and knowledge graph construction, gradually realizing the full-process automated conversion from pixel-level data to structured information.

[0004] A complete document analysis system consists of four steps: image preprocessing, layout analysis (including physical and logical layout analysis), content recognition, and post-processing and semantic analysis. Physical layout analysis, also known as document image segmentation, is a sub-field of image segmentation. The difficulty lies in the complexity of document layout structures, such as the large number of regions, irregular shapes, varying sizes, and similar visual features between regions.

[0005] Researchers in various fields have proposed numerous models and methods for physical layout analysis, which can be broadly categorized into non-deep learning methods (top-down, bottom-up, and hybrid approaches) and deep learning methods (various deep neural networks and large-scale models). Non-deep learning methods began to emerge in the 1980s, and most of them rely on various rules and prior knowledge to merge or segment document regions. Deep learning methods, on the other hand, emerged after Hinton's team achieved impressive results in the ImageNet competition in 2012. Unlike non-deep learning methods, deep learning methods, based on training with large amounts of data and the powerful feature extraction capabilities of large models, enable more precise region segmentation.

[0006] Methods based on deep neural networks can be divided into two types according to task categories: object detection-based analysis and semantic segmentation / instance segmentation-based analysis.

[0007] Object detection-based methods use general-purpose object detectors (such as the Yolo series, Mask R-CNN, and Faster R-CNN) to locate regions of various categories within a document. This approach aims to extract features from images using a convolutional network, then decode these features to predict the location of the object (represented by a bounding box) and its category. However, these methods are only effective at predicting rectangular regions and regions with moderate aspect ratios, and struggle to predict regions of arbitrary shapes, extreme aspect ratios, or free-form shapes. Semantic segmentation / instance segmentation-based methods use general-purpose segmentation frameworks (such as FCN and Mask R-CNN) to divide document parts into free-form shapes. Compared to object detection, these methods can parse documents more precisely and freely.

[0008] Furthermore, with the introduction of the Transformer by Google in 2017, self-attention and cross-attention mechanisms for long sequences have swept across various AI fields, including computer vision (CV) and natural language processing (NLP). Naturally, the Transformer has also begun to appear in document image segmentation tasks. Leveraging the powerful feature extraction capabilities of the attention mechanism, many effective models and methods for image understanding and segmentation have been developed, such as MaskFormer proposed by Cheng et al. in 2021, LayoutLM proposed by Xu et al. in 2020, and SAM proposed by Kirillov et al. in 2023, all of which have achieved good image segmentation results.

[0009] Although relevant research has achieved certain results, the following problems still exist in the models and methods:

[0010] 1) Deep neural networks lack interpretability, which makes their performance sometimes difficult to predict and makes the models difficult to transfer to different fields;

[0011] 2) Many models for document layout analysis rely on OCR technology. While this can enhance the model's understanding of documents to a certain extent, it also introduces OCR errors, causing the model's performance to be strongly dependent on OCR results.

[0012] 3) When the attention mechanism is applied to images, the amount of computation is too large and consumes a lot of computing power;

[0013] 4) When the model is trained on a document dataset, the convergence speed is slow and the convergence process is unstable. Summary of the Invention

[0014] In response to the above-mentioned technical problems existing in the prior art, the present invention proposes a method for image segmentation based on the Qianwen large model to guide the backbone network. Specifically, it is a method based on the Qianwen 2 multimodal large model to guide the UNet backbone segmentation network to perform document physical layout analysis. It is trained with a Chinese document dataset to achieve more refined and more adaptive image segmentation mask results.

[0015] The present invention is achieved through the following technical solutions:

[0016] The present invention discloses a method for image segmentation based on a Qianwen large model to guide a backbone network, comprising:

[0017] 1) Build the overall model, which includes the Qianwen2 multimodal model, the UNet backbone segmentation network, the fusion module, and the decoding module;

[0018] The Qianwen-2 multimodal large model is used to obtain document images and prompts, and output visual feature information and semantic feature information to the fusion module;

[0019] The UNet backbone segmentation network is used to obtain document images and generate coarse-grained feature maps for the final prediction;

[0020] The fusion module is used to receive the visual feature information and semantic feature information output by the Qianwen-2 multimodal large model, and output high-level fusion features to the decoding module;

[0021] The decoding module is used to receive the learnable query vector and high-level fusion features, output the learned query vector, and then generate a prediction mask by combining the learned query vector with the coarse-grained feature map;

[0022] 2) Obtain document images from the training set and use the UNet backbone segmentation network to obtain feature maps. Simultaneously, obtain document images and prompts, and use the Qianwen-2 multimodal model to obtain the feature information of their hidden layers, including visual and semantic features.

[0023] 3) The feature information of the hidden layer is fused through the deformable proxy attention mechanism to obtain a more complete high-level fusion feature that integrates semantic feature information and visual feature information;

[0024] 4) The learnable query vector and the more complete high-level fusion features obtained in step 3) are passed through the decoding module of the overall model. The learnable query vector learns the knowledge in the high-level fusion features to obtain the learned query vector.

[0025] 5) Multiply the learned query vector with the feature map output by the UNet backbone segmentation network in step 2) to obtain the final predicted mask.

[0026] As a further improvement, the deformable proxy attention mechanism in step 3) of the present invention is a process in which the proxy attention mechanism generates one element of the proxy amount by fixing the values ​​of four positions each time through a pooling method, and adds a deformable sampling method.

[0027] As a further improvement, the specific operation of the deformable sampling method described in the present invention is to allow each element in the proxy quantity to flexibly select four values ​​from the query vector through a linear layer, and flexibly assign weights to these four values, and then perform weighted average of these four values ​​to obtain the element's own proxy value. During the deformable sampling process, when each element in the proxy quantity freely selects four values, if a value with a non-integer coordinate is selected, the value is obtained through bilinear interpolation.

[0028] As a further improvement, the feature information of the hidden layer of the Qianwen-2 multimodal large model in step 2) described in the present invention is that the Qianwen-2 multimodal large model makes answers based on the input document image and prompts, and the hidden layer state of the first layer output in the 29-layer decoder of the Qianwen-2 multimodal large model is obtained as visual feature information, and the hidden layer states of the last 28 layers are obtained and stacked as semantic feature information.

[0029] As a further improvement, the advanced fusion features in step 3) described in the present invention use visual feature information as a query vector and semantic feature information as a key-value vector, which are input into a deformable proxy attention mechanism for interaction to obtain values. In the fusion process, there are multiple layers of deformable proxy attention mechanisms, that is, the fusion part uses a multi-layer stacked structure, and the visual feature information interacts with the semantic feature information multiple times to finally obtain an advanced fusion feature, and the size of the advanced fusion feature is the same as the size of the visual feature information.

[0030] As a further improvement, the learnable query vector in step 4) of the present invention is a learnable parameter initialized by sinusoidal position encoding. The learnable query vector is used to obtain the most useful part of the high-level fusion features, condense the high-level fusion features, and ultimately efficiently guide the UNet backbone segmentation network to perform task processing.

[0031] As a further improvement, the decoding module in step 4) of the present invention is a multi-layer structure, wherein each layer is composed of a cross-attention deformable proxy attention and a self-attention deformable proxy attention. The input of the cross-attention deformable proxy attention is a learnable query vector and high-level fusion features, and the learnable query vector is used as the query vector of the attention mechanism, and the high-level fusion features are used as the key-value vector of the attention mechanism; the input of the self-attention deformable proxy attention is a learnable query vector, so that the learnable query vector can communicate with each other and perform efficient learning.

[0032] As a further improvement, the feature map in step 5) of the present invention is generated by inputting the document image into the UNet backbone segmentation network. It is a coarse-grained feature map that serves as the basis for the entire model segmentation. The multiplication of the feature map and the learned query vector is performed in the channel dimension.

[0033] As a further improvement, the training set in step 2) of the present invention is a Chinese document analysis dataset, namely, the CDLA dataset.

[0034] As a further improvement, the prompts in step 2) of the present invention include prompting the Qianwen 2 multimodal large model to extract text from the document image, extract non-text parts from the document image, summarize the document content and the specific output format.

[0035] The beneficial effects of the present invention are as follows:

[0036] 1. Based on the primary image visual information used by the backbone segmentation network, the Qianwen-2 multimodal model is introduced. Its powerful multimodal feature alignment, document understanding, and document question-answering capabilities are utilized to obtain visual and semantic information from the decoder feature space, thus avoiding the drawbacks of information omissions caused by the use of OCR technology. The more abstract and advanced feature information output by the Qianwen-2 multimodal model is used to guide the backbone segmentation network to perform more detailed document image segmentation tasks, strengthening the backbone segmentation network's visual understanding of the input document images. At the same time, the entire model gains semantic understanding of document images. The two understanding capabilities guide each other and align feature information, thereby giving the model a deeper understanding of the content and visual structure of document images and achieving better performance on the task.

[0037] 2. A fusion and decoding module is designed after the backbone segmentation network and the Qianwen-2 multimodal model. After obtaining the Qianwen-2 multimodal model's understanding of the document image's visual structure and semantic content, the fusion module receives these two more abstract and advanced feature information, primarily visual feature information and supplemented by semantic feature information, interacting with each other to further align the two features and fuse them to obtain advanced fused feature information. Next, a learnable query vector is set up within the decoding module. This learnable query vector is primarily used, supplemented by advanced fused features, for knowledge learning. This allows the learnable query vector to learn the most useful and condensed knowledge from the advanced fused features, extracting its essence and removing its dross, reducing interference from useless information. The learned query vector is then used as the final guidance information to guide the backbone segmentation network for the final, detailed document image segmentation. The designed fusion and decoding module can align semantic and visual feature information and only extract the most refined portion of the advanced fused feature information, accelerating model convergence and making model training more stable.

[0038] 3. A deformable proxy attention mechanism is introduced in the feature fusion and decoding stages. A deformable sampling method is added on top of the proxy attention mechanism. This not only reduces computational overhead and memory usage, but also enhances the flexibility of proxy acquisition, allowing the model to more adaptively learn reasonable and useful knowledge and obtain more flexible attention. The deformable proxy attention mechanism also solves the problem that proxy attention cannot be used in cross-attention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of the proposed model is provided;

[0040] Figure 2 Flowchart of the deformable agent attention mechanism. DETAILED DESCRIPTION

[0041] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] The model process of the present invention is as follows Figure 1 As shown, Figure 1 Flowchart of the proposed model.

[0043] A method for image segmentation based on a Qianwen large model to guide a backbone network, comprising:

[0044] 1) Build the overall model, which includes the Qianwen2 multimodal model, the UNet backbone segmentation network, the fusion module, and the decoding module;

[0045] The Qianwen-2 multimodal large model is used to obtain document images and prompts, and output visual feature information and semantic feature information to the fusion module;

[0046] The UNet backbone segmentation network is used to obtain document images and generate coarse-grained feature maps for the final prediction;

[0047] The fusion module is used to receive the visual feature information and semantic feature information output by the Qianwen-2 multimodal large model, and output high-level fusion features to the decoding module;

[0048] The decoding module is used to receive the learnable query vector and high-level fusion features, output the learned query vector, and then generate a prediction mask by combining the learned query vector with the coarse-grained feature map;

[0049] 2) Obtain the document images in the training set, obtain the feature map through the UNet backbone segmentation network, obtain the document images and prompts at the same time, and obtain the feature information of its hidden layer through the Qianwen-2 multimodal model. The feature information includes visual feature information and semantic feature information; the training set is the Chinese document analysis dataset, namely the CDLA dataset. It includes 5,000 training images of Chinese documents and 1,000 test images of Chinese documents. The categories included are text, title, picture, picture title, table, table title, reference, header, footer and formula, a total of 10 categories. The feature information of the hidden layer of the Qianwen-2 multimodal model is that the Qianwen-2 multimodal model answers according to the input document images and prompts, and then obtains the hidden layer state of the first layer output in the 29-layer decoder of the Qianwen-2 multimodal model as the visual feature information, and obtains the hidden layer states of the output of the last 28 layers and stacks them as the semantic feature information. The shapes of the visual information and semantic information are respectively and ,in represents the batch size for training, , Represent the length of visual and semantic features respectively, Represents the dimension of the feature. In the specific work, the dimensions of visual feature information and semantic feature information are both 1536, the lengths are set to 1024 and 256 respectively, and the training batch is set to 2. The feature information of the hidden layer generated by the Qianwen 2 multimodal large model serves as the guiding knowledge for the UNet backbone segmentation network. The feature map is generated by the UNet backbone segmentation network by inputting the document image. It is a coarse-grained feature map that serves as the basis for the segmentation of the entire model. The multiplication of the feature map and the learned query vector is performed on the channel dimension, and the final output is a prediction mask with the size of [training batch, number of classification categories, predicted mask height, predicted mask width].

[0050]

[0051] in is the predicted mask, Represents the feature map output by UNet, represents the learned query vector.

[0052] The prompts should include prompting the Qianwen 2 multimodal large model to extract text from document images, extract non-text parts from document images, summarize document content and specific output format.

[0053] The specific cases are as follows:

[0054] {"type": "text", "text": '''

[0055] Please analyze the following document image and complete the following tasks:

[0056] 1. **Extract all visible text**:

[0057] - Extract text line by line / paragraph from top to bottom and left to right.

[0058] - If there are multiple columns of text, sort them in column order.

[0059] - Label the approximate location of each text segment (e.g., "top left corner," "center").

[0060] 2. **Identify non-text elements**:

[0061] - Label the location and type of non-text areas such as images, tables, and charts (e.g., “There is a table in the lower right corner”).

[0062] - Describe the general content of non-text elements (e.g., "the table contains 3 rows and 4 columns of data").

[0063] 3. **Summarize the document**:

[0064] - Briefly summarize the core information and structure of the document.

[0065] - If there are complex diagrams or formulas, try to describe their meaning in words.

[0066] Please output in the following format:

[0067] {

[0068] "text_content": [

[0069] {"position": "Position description", "text": "Extracted text content"}, ...

[0071] ],

[0072] "non_text_elements": [

[0073] {"type": "Type (such as table, picture)", "position": "Position description", "description": "Content description"}, ...

[0075] ],

[0076] "summary": "Summary of document contents"

[0077] }

[0078] '''}.

[0079] 3) The feature information of the hidden layer is fused through the deformable proxy attention mechanism to obtain a more complete high-level fusion feature that integrates the semantic feature information and the visual feature information. The deformable proxy attention mechanism is to add a deformable sampling method to the process of generating one element of the proxy amount by fixing the value of four positions at a time through the pooling method. The process of the deformable proxy attention mechanism is as follows Figure 2 As shown, Figure 2 This is a flowchart of the deformable proxy attention mechanism. Proxy attention is a proxy designed in ordinary attention. , which can be used as The amount of agency, from Learn knowledge from It is a A much smaller amount (typically 49 pixel values) is then used instead and and Interaction, communication Since the proxy can use dozens of values ​​instead of thousands of values ​​for interactive calculations, this design can greatly reduce the overhead of attention calculations in the image field without excessively damaging the interactive ability and feature extraction ability of attention. The specific operation of the deformable sampling method is to allow each element in the proxy to flexibly select four values ​​from the query vector through a linear layer, flexibly assign weights to these four values, and then take the weighted average of these four values ​​to obtain the element's own proxy value. Compared with the pooling method in the proxy attention mechanism that fixes four values, the deformable sampling method allows the proxy to more arbitrarily and globally extract and condense knowledge to form proxy features. In the deformable sampling process, when each element in the proxy freely selects four values, if a value with a non-integer coordinate is selected, the value is obtained through bilinear interpolation.

[0080] High-level fusion features are derived by using visual features as query vectors and semantic features as key-value vectors, which are fed into a deformable proxy attention mechanism for interaction. The fusion process involves multiple layers of the deformable proxy attention mechanism, meaning the fusion process uses a multi-layer stacked structure. Visual features are repeatedly interacted with semantic features, ultimately resulting in high-level fusion features. The size of the high-level fusion features is the same as that of the visual features. Visual features that undergo cross-attention interactions incorporate semantic information, making them more comprehensive and instructive.

[0081]

[0082] in represents the fusion feature, () represents the cross-interaction deformable agent attention calculation, , They are visual features and semantic features respectively.

[0083] 4) The learnable query vector and the more complete high-level fusion features obtained in step 3) are passed through the decoding module of the overall model. The learnable query vector learns the knowledge in the high-level fusion features and obtains the learned query vector; the learnable query vector is a learnable parameter initialized by the sinusoidal position encoding and has a size of ,in is the training batch size, is the number of learnable query vectors, is the dimension of the learnable query vector. The learnable query vector is used to obtain the most useful part of the high-level fusion features, condense the high-level fusion features, and finally efficiently guide the UNet backbone segmentation network to perform task processing.

[0084] The decoding module has a multi-layer structure, in which each layer consists of a cross-attention deformable proxy attention and a self-attention deformable proxy attention. The input of the cross-attention deformable proxy attention is a learnable query vector and high-level fusion features. The learnable query vector is used as the query vector of the attention mechanism, and the high-level fusion features are used as the key-value vector of the attention mechanism; the input of the self-attention deformable proxy attention is the learnable query vector, so that the learnable query vector can communicate with each other for efficient learning.

[0085]

[0086] in Represent the query vectors before and after learning, Represents attention computation for self-interacting deformable agents.

[0087] 5) Multiply the learned query vector by the feature map output by the UNet backbone segmentation network in step 2) to obtain the final predicted mask. The size of the final predicted mask should be [training batch, number of classification categories = 10, predicted mask height, predicted mask width].

[0088] In terms of loss function, the present invention optimizes the model by combining the loss functions of cross entropy loss, focal loss, and Dice loss, with the weights all set to 1. The use of a combined loss function enables the network to learn pixel-level and region-level features, thereby enhancing performance.

[0089] The loss function is set as follows:

[0090]

[0091] The various parts of the total loss function are as follows:

[0092]

[0093]

[0094]

[0095] The following experiments are conducted based on the CDLA dataset to verify the effectiveness of the present invention. The experimental configuration is as follows: RTX4090 (24G) and Quadro RTX 8000 (50G) are used for training, with 10 training rounds and a batch size of 8 (2 4), the initial learning rate is We used the Adam optimizer and an exponential learning rate decay strategy, with the learning rate decayed every 1250 steps and the exponential coefficient set to 0.85 (step_size=1250, gamma=0.85). We set the fusion module and decoding module to 4 layers, and the number of learnable query vectors to 100.

[0096] The model clearly segments each part of the document, especially small target areas, which can be well identified. The quantitative analysis experimental data is as follows:

[0097] Table 1 is the performance table of this embodiment in various categories and indicators

[0098]

[0099] Table 2 is the performance table of UNet in various categories and indicators

[0100]

[0101] As can be seen from the table above, compared to the UNet backbone segmentation network, the image segmentation method based on the Qianwen model-guided backbone network performs comparable to UNet on large samples (such as text, images, tables, etc.), but significantly outperforms UNet on small samples (such as table titles, footers, image titles, and formulas), with an F1 score significantly higher than UNet. It is also worth noting that on these smaller samples, the present invention does not necessarily achieve higher precision than UNet, but the recall rate is significantly higher than UNet. This shows that the image segmentation method based on the Qianwen model-guided backbone network can better consider extreme sample conditions when predicting categories and better exploit the features of small samples. Furthermore, it is less conservative than UNet. Due to the extraction of more complete features, it is more confident in predicting masks for small samples, which can maintain a high recall rate.

[0102] The present invention completes the guidance of the backbone segmentation network by introducing the guidance of Qianwen, and obtains better performance than the backbone segmentation network.

[0103] The above description of the embodiments is intended to facilitate understanding and application of the present invention by those skilled in the art. It will be apparent that those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without requiring inventive effort. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for image segmentation based on a Qianwen model to guide a backbone network, characterized in that: include: 1) Constructing an overall model, which includes the Qianwen2 multimodal large model, the UNet backbone segmentation network, the fusion module, and the decoding module; The Qianwen2 multimodal large model is used to obtain document images and prompts, output visual feature information and semantic feature information and send them to the fusion module; The UNet backbone segmentation network is used to obtain document images and generate coarse-grained feature maps for the final prediction; The fusion module is used to receive the visual feature information and semantic feature information output by the Qianwen-2 multimodal large model, and output high-level fusion features to the decoding module; The decoding module is used to receive a learnable query vector and high-level fusion features, output a learned query vector, and then generate a prediction mask by combining the learned query vector with the coarse-grained feature map; 2) Obtain document images from the training set, obtain feature maps through the UNet backbone segmentation network, and simultaneously obtain document images and prompts, and obtain their hidden layer feature information through the Qianwen2 multimodal large model. The feature information includes visual feature information and semantic feature information; 3) The feature information of the hidden layer is fused through the deformable proxy attention mechanism to obtain a more complete high-level fusion feature that integrates semantic feature information and visual feature information; 4) The learnable query vector and the more complete high-level fusion features obtained in step 3) are passed through the decoding module of the overall model. The learnable query vector learns the knowledge in the high-level fusion features to obtain a learned query vector. 5) Multiply the learned query vector with the feature map output by the UNet backbone segmentation network in step 2) to obtain the final predicted mask; The advanced fusion features in step 3) are obtained by inputting the visual feature information as the query vector and the semantic feature information as the key-value vector into the deformable proxy attention mechanism for interaction. In the fusion process, there are multiple layers of deformable proxy attention mechanisms, that is, the fusion part uses a multi-layer stacked structure. The visual feature information interacts with the semantic feature information multiple times, and finally obtains the advanced fusion features. The size of the advanced fusion features is the same as the size of the visual feature information.

2. The method for image segmentation based on the Qianwen model-guided backbone network according to claim 1, characterized in that: The deformable proxy attention mechanism in step 3) is a process in which the proxy attention mechanism generates one element of the proxy amount by fixing the values ​​of four positions each time through a pooling method, and adds a deformable sampling method.

3. The method for image segmentation based on the Qianwen model-guided backbone network according to claim 2, characterized in that: The specific operation of the deformable sampling method is to allow each element in the proxy quantity to flexibly select four values ​​from the query vector through a linear layer, flexibly assign weights to these four values, and then perform a weighted average of these four values ​​to obtain the element's own proxy value. During the deformable sampling process, when each element in the proxy quantity freely selects four values, if a value with a non-integer coordinate is selected, the value is obtained through bilinear interpolation.

4. The method for image segmentation based on the Qianwen model-guided backbone network according to claim 1, 2 or 3, characterized in that: The feature information of the hidden layer of the Qianwen-2 multimodal model in the step 2) is that the Qianwen-2 multimodal model makes answers based on the input document image and prompts, obtains the hidden layer state of the first layer output in the 29-layer decoder of the Qianwen-2 multimodal model as visual feature information, and obtains the hidden layer states of the last 28 layers output and stacks them as semantic feature information.

5. The method for image segmentation based on the Qianwen model-guided backbone network according to claim 4, characterized in that: The learnable query vector in step 4) is a learnable parameter initialized by sinusoidal position encoding. The learnable query vector is used to obtain the most useful part of the high-level fusion features, condense the high-level fusion features, and ultimately efficiently guide the UNet backbone segmentation network to perform task processing.

6. The method for image segmentation based on the Qianwen model-guided backbone network according to claim 1, 2, 3 or 5, characterized in that: The decoding module in step 4) is a multi-layer structure, wherein each layer consists of a cross-attention deformable proxy attention and a self-attention deformable proxy attention. The input of the cross-attention deformable proxy attention is a learnable query vector and high-level fusion features, and the learnable query vector is used as the query vector of the attention mechanism, and the high-level fusion features are used as the key-value vector of the attention mechanism; the input of the self-attention deformable proxy attention is a learnable query vector, so that the learnable query vector can communicate with each other and perform efficient learning.

7. The method for image segmentation based on the Qianwen model-guided backbone network according to claim 6, characterized in that: The feature map in step 5) is generated by inputting the document image into the UNet backbone segmentation network. It is a coarse-grained feature map and serves as the basis for the entire model segmentation. The multiplication of the feature map and the learned query vector is performed in the channel dimension.

8. The method for image segmentation based on the Qianwen model-guided backbone network according to claim 7, characterized in that: The training set in step 2) is the Chinese document analysis dataset, namely the CDLA dataset.

9. The method for image segmentation based on the Qianwen model-guided backbone network according to claim 7 or 8, characterized in that: The prompts in step 2) include prompting the Qianwen 2 multimodal large model to extract text from the document image, extract the non-text part of the document image, summarize the document content and the specific output format.

Images