Handwritten text segmentation method based on SAM model

Abstract

The invention discloses a handwritten text segmentation method based on an SAM model, and belongs to the technical field of electric digital data processing. The method comprises the following steps: obtaining noise interference data and text background data of a handwritten text; performing comprehensive analysis according to the noise interference data to obtain a noise interference evaluation value, performing comprehensive analysis according to the text background data to obtain a text background evaluation value, performing comprehensive analysis according to the noise interference evaluation value and the text background evaluation value to obtain a text noise evaluation value, and performing adjustment feedback according to the text noise evaluation value; the method comprises the following steps: acquiring an encoder feature map of a handwritten text, processing the encoder feature map according to a noise interference evaluation value and a text background evaluation value to obtain a fused feature map, and outputting a handwritten text segmentation result according to the fused feature map, thereby solving the problem that a traditional handwritten text line segmentation method is difficult to effectively distinguish document low-quality scenes; and the handwritten text segmentation accuracy is improved.

Description

Technical Field

[0001] The invention relates to the technical field of electronic digital data processing, and in particular to a handwritten text segmentation method based on a SAM model. Background Art

[0002] The SAM model is a major advancement in the field of image segmentation and provides a foundational model for computer vision tasks. With its acceptable design, efficiency, and zero-shot generalizability, SAM continues to revolutionize segmentation models, making it a powerful tool for a variety of computer vision applications. As technology continues to develop, the SAM model is expected to play an important role in more fields.

[0003] Existing handwritten text segmentation methods include graph clustering, codebook-based methods, water flow algorithms, and deep learning techniques using fully convolutional networks. These methods aim to improve the accuracy and efficiency of text line segmentation in handwritten documents and address challenges such as touch lines, tilted text, and heavy components.

[0004] For example, the invention patent announcement with announcement number: CN112507714B discloses a text segmentation method and device, comprising: obtaining a target text, performing sentence segmentation processing on the target text, and generating a sentence segmentation result; in response to any sentence segmentation result, determining that the length of the sentence segmentation result exceeds a preset length threshold, performing word segmentation processing on the sentence segmentation result to obtain at least three words; performing word combination on at least two of the at least three words to obtain a phrase; comparing the length of the current phrase with the preset length threshold; in response to the length of the phrase not exceeding the preset length threshold, using the phrase as a segmentation result of the target text.

[0005] For example, a patent application with publication number: CN118627465A discloses a method and system for segmenting science test paper text, including: obtaining a database of historical test papers, randomly extracting parts of the historical test papers in the database, and generating test paper text according to a preset typesetting format; identifying formulas and pictures in the test paper text, converting the formulas and pictures into corresponding characters, and performing sentence segmentation based on target symbols; automatically annotating each sentence, and inputting the annotation results into a constructed pointer annotation neural network model for training; inputting the test paper text to be segmented into the trained neural network model, and outputting a segmented test question list, wherein the sentences are encoded by modeling longer contextual relationships, and a hierarchical attention mechanism from the word level to the sentence level is established to achieve deep semantic understanding, and the setting and interaction of the start position pointer and the end position pointer in the pointer annotation neural network model are used during model training.

[0006] However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the present application found that the above technology has at least the following technical problems: In the prior art, traditional handwritten text line segmentation methods are difficult to effectively distinguish low-quality document scenes, resulting in segmentation errors and recognition difficulties, and there is a problem of low handwritten text segmentation accuracy. Summary of the invention

[0007] The present invention provides a handwritten text segmentation method based on the SAM model, which solves the problem that the traditional handwritten text line segmentation method in the prior art is difficult to effectively distinguish low-quality document scenes, resulting in segmentation errors and recognition difficulties, and the problem of low handwritten text segmentation accuracy, thereby achieving improved processing capabilities for handwritten text segmentation.

[0008] The present invention provides a handwritten text segmentation method based on a SAM model, comprising the following steps: obtaining noise interference data and text background data of a handwritten text; obtaining a noise interference evaluation value according to a comprehensive analysis of the noise interference data, obtaining a text background evaluation value according to a comprehensive analysis of the text background data, obtaining a text noise evaluation value according to a comprehensive analysis of the noise interference evaluation value and the text background evaluation value, and performing adjustment feedback according to the text noise evaluation value; obtaining an encoder feature map of the handwritten text, processing the encoder feature map according to the noise interference evaluation value and the text background evaluation value to obtain a fused feature map, and outputting a handwritten text segmentation result according to the fused feature map.

[0009] Furthermore, the step of obtaining a noise interference evaluation value based on a comprehensive analysis of the noise interference data includes: the noise interference data includes character angle, edge clarity and character overlap area; obtaining a reference character angle, allowable deviation character angle, critical edge clarity and critical character overlap area from a text segmentation database; and obtaining a noise interference evaluation value through a comprehensive analysis.

[0010] Furthermore, the step of obtaining a text background evaluation value based on a comprehensive analysis of the text background data includes: the text background data includes background color difference, background complexity and background brightness difference; obtaining a critical background color difference, a critical background complexity and a critical background brightness difference from a text segmentation database; and obtaining a text background evaluation value through a comprehensive analysis.

[0011] Furthermore, the step of obtaining a text noise evaluation value by comprehensive analysis based on the noise interference evaluation value and the text background evaluation value includes: obtaining a critical noise interference evaluation value and a critical text background evaluation value from a text segmentation database; and obtaining a text noise evaluation value by comprehensive analysis.

[0012] Furthermore, the text noise evaluation value is obtained as follows: ; In the formula, ξN represents the text noise evaluation value, α7 represents the text noise evaluation influence factor corresponding to the noise interference evaluation value, α8 represents the text noise evaluation influence factor corresponding to the text background evaluation value, ξI1 represents the noise interference evaluation value, ξI0 represents the critical noise interference evaluation value, ξB1 represents the text background evaluation value, and ξB0 represents the critical text background evaluation value.

[0013] Furthermore, the step of adjusting feedback according to the text noise evaluation value includes: obtaining a text noise evaluation threshold from a text segmentation database; comparing the text noise evaluation value with the text noise evaluation threshold, if the text noise evaluation value is greater than the text noise evaluation threshold, providing early warning feedback to the handwritten text image, if the text noise evaluation value is less than or equal to the text noise evaluation threshold, performing a handwritten text segmentation operation.

[0014] Furthermore, the encoder feature map is processed according to the noise interference evaluation value and the text background evaluation value to obtain a fused feature map, the step comprising: dividing the noise interference evaluation value by the text background evaluation value to obtain a feature ratio; obtaining the initial weights of each feature layer of the preset encoder feature map, and comparing the feature ratio with 1, if the feature ratio is greater than 1, increasing the weight of the low-level feature, if the feature ratio is equal to 1, no additional processing is performed, if the feature ratio is less than 1, increasing the weight of the high-level feature; fusing the adjusted encoder feature map to obtain a fused feature map.

[0015] Furthermore, the step of increasing the weight of the low-level features includes: marking the subtraction of the feature ratio and 1 as the deviation feature ratio; matching the deviation feature ratio with the weight adjustment value corresponding to each deviation feature ratio range preset in the text segmentation database to obtain the weight adjustment value of the low-level feature; adding the weight adjustment value of the low-level feature to the weight of the low-level feature to obtain the current weight value of the low-level feature.

[0016] Furthermore, the step of outputting the handwritten text segmentation result according to the fused feature map includes: obtaining each pixel value and pixel threshold of the fused feature map; comparing each pixel value of the fused feature map with the pixel threshold, if a pixel value of the fused feature map is greater than the pixel threshold, then marking the pixel as a foreground pixel, if a pixel value of the fused feature map is less than or equal to the pixel threshold, then marking the pixel as a background pixel, counting each foreground pixel, and outputting the handwritten text segmentation result according to each foreground pixel.

[0017] Furthermore, the step of outputting the handwritten text segmentation result according to each foreground pixel includes: projecting each foreground pixel area to extract feature information in each foreground pixel; obtaining each segmentation pixel point according to the feature information, and using each segmentation pixel point to segment the fused feature map to obtain the handwritten text segmentation result.

[0018] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The present invention provides a handwritten text segmentation method based on the SAM model, thereby utilizing the noise interference evaluation value and the text background evaluation value to process the encoder feature map, thereby achieving improved accuracy of handwritten text segmentation and enhanced robustness and adaptability of the system.

[0019] 2. The present invention comprehensively analyzes the noise interference evaluation value and the text background evaluation value, so as to more comprehensively understand the noise situation and background characteristics in the handwritten text, thereby realizing the accurate calculation of the text noise evaluation value and accurately reflecting the degree of noise interference on the handwritten text.

[0020] 3. The present invention uses fused feature maps to segment handwritten text, thereby combining the advantages of low-level detail features and high-level semantic features, thereby achieving more accurate and detailed segmentation of handwritten text, avoiding segmentation errors or omissions that may be caused by a single feature level in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flow chart of a handwritten text segmentation method based on the SAM model provided in an embodiment of the present application.

[0022] Figure 2 A graph showing changes in text noise evaluation values ​​for a handwritten text segmentation method based on the SAM model provided in an embodiment of the present application.

[0023] Figure 3 The SAM framework diagram of a handwritten text segmentation method based on the SAM model is shown. DETAILED DESCRIPTION

[0024] The embodiment of the present application provides a handwritten text segmentation method based on the SAM model, which solves the problem that the traditional handwritten text line segmentation method in the prior art is difficult to effectively distinguish low-quality document scenes, resulting in segmentation errors and recognition difficulties, and the problem of low handwritten text segmentation accuracy. A noise interference evaluation value is obtained by comprehensive analysis based on noise interference data, a text background evaluation value is obtained by comprehensive analysis based on text background data, a text noise evaluation value is obtained by comprehensive analysis based on the noise interference evaluation value and the text background evaluation value, and feedback is adjusted according to the text noise evaluation value; an encoder feature map of the handwritten text is obtained, and the encoder feature map is processed according to the noise interference evaluation value and the text background evaluation value to obtain a fused feature map, and a handwritten text segmentation result is output according to the fused feature map, thereby achieving improved processing capabilities for handwritten text segmentation.

[0025] The technical solution in the embodiment of the present application is to solve the problem that the above-mentioned traditional handwritten text line segmentation method is difficult to effectively distinguish low-quality document scenes, resulting in segmentation errors and recognition difficulties, and there is a problem of low accuracy in handwritten text segmentation. The overall idea is as follows: By obtaining noise interference data and text background data of handwritten text; obtaining a noise interference evaluation value based on a comprehensive analysis of the noise interference data, obtaining a text background evaluation value based on a comprehensive analysis of the text background data, obtaining a text noise evaluation value based on a comprehensive analysis of the noise interference evaluation value and the text background evaluation value, and performing adjustment feedback based on the text noise evaluation value; obtaining an encoder feature map of the handwritten text, and processing the encoder feature map based on the noise interference evaluation value and the text background evaluation value to obtain a fused feature map, and outputting a handwritten text segmentation result based on the fused feature map, thereby achieving a handwritten text segmentation result with high precision and high robustness.

[0026] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0027] like Figure 1 As shown, it is a flow chart of a handwritten text segmentation method based on the SAM model provided in an embodiment of the present application, the method comprising the following steps: obtaining noise interference data and text background data of the handwritten text; obtaining a noise interference evaluation value according to a comprehensive analysis of the noise interference data, obtaining a text background evaluation value according to a comprehensive analysis of the text background data, obtaining a text noise evaluation value according to a comprehensive analysis of the noise interference evaluation value and the text background evaluation value, and performing adjustment feedback according to the text noise evaluation value; obtaining an encoder feature map of the handwritten text, and processing the encoder feature map according to the noise interference evaluation value and the text background evaluation value to obtain a fused feature map, and outputting a handwritten text segmentation result according to the fused feature map.

[0028] In this embodiment, SAM (full name Segment Anything Model) is a general segmentation model with efficient image segmentation capabilities, and can perform accurate object segmentation without the need for a large amount of labeled data. When obtaining the encoder feature map of the handwritten text, an Adapter module is inserted into the ViT (full name Vision Transformer, i.e., image encoder) of SAM. The Adapter module is a lightweight neural network module that can be inserted into a pre-trained Transformer model to achieve efficient parameter fine-tuning. The present invention has significant benefits in improving the accuracy of handwritten text segmentation, enhancing robustness, optimizing feature representation, improving segmentation efficiency, and supporting a variety of application scenarios.

[0029] In addition, the text segmentation database is used to store data related to the handwritten text segmentation method based on the SAM model, including: reference character angle, allowable deviation character angle, critical edge clarity, critical character overlapping area, noise interference assessment influencing factor corresponding to the character angle, critical noise interference assessment value and text noise assessment threshold, etc. The data in the text segmentation database can be directly queried through public databases such as the MNIST database and the EMNIST database, and can be obtained through the websites of relevant academic research institutions or data set platforms such as Kaggle and OpenML.

[0030] Furthermore, the step of obtaining a noise interference evaluation value based on a comprehensive analysis of the noise interference data includes: the noise interference data includes character angle, edge clarity and character overlap area; obtaining a reference character angle, allowable deviation character angle, critical edge clarity and critical character overlap area from a text segmentation database; and obtaining a noise interference evaluation value through a comprehensive analysis.

[0031] The noise interference evaluation value is obtained as follows: ; In the formula, ξI1 represents the noise interference evaluation value, α1 represents the noise interference evaluation influence factor corresponding to the character angle, α2 represents the noise interference evaluation influence factor corresponding to the edge clarity, α3 represents the noise interference evaluation influence factor corresponding to the character overlapping area, A1 represents the character angle, A0 represents the reference character angle, A2 represents the allowable deviation character angle, D1 represents the edge clarity, D0 represents the critical edge clarity, O1 represents the character overlapping area, and O0 represents the critical character overlapping area.

[0032] α1, α2 and α3 are the noise interference assessment impact factors corresponding to the character angle, edge clarity and character overlap area preset in the text segmentation database, respectively. These impact factors are numerical indicators that measure the impact of the above noise factors on the noise interference assessment value. Specifically, there is a mapping relationship table for each character angle, edge clarity and character overlap area, which records each possible noise factor value and its corresponding noise interference assessment impact factor. These mapping relationships can be one-to-one or many-to-one. In practical applications, when it is necessary to evaluate a certain noise interference assessment value, the measured character angle, edge clarity and character overlap area can be input into their respective corresponding mapping relationship tables, and the noise interference assessment impact factors corresponding to these values ​​can be quickly found, where the value range of the impact factor is between 0 and 1.

[0033] In this embodiment, the character angle refers to the degree of inclination of the character relative to the numerical direction of the image, which can be obtained by detecting the outline of the character through image processing technology (such as Hough transform, edge detection, etc.) and measuring its direction or inclination angle; edge clarity reflects the sharpness or blurriness of the character edge, which can be measured by calculating the gradient of the character edge, Laplace operator, etc. For example, the Tenengrad gradient function is used to calculate the edge clarity of a character image. First, the edge detection operator such as Sobel is applied to the image to detect the edge, and then the gradient amplitude of the detected edge is calculated. The gradient amplitude of the edge is obtained as follows: ; In the formula, G represents the gradient amplitude, Gx and Gy represent the gradient values ​​in the horizontal and vertical directions respectively, and the sum of the gradient amplitudes is calculated as the edge clarity of the image. The larger the value, the sharper the character edge and the higher the edge clarity. The character overlapping area refers to the sum of the total area occupied by the overlapping parts between different characters in the image. The character overlapping area is analyzed by connected domains, scanning the pixels in the binary image by row or column. For each unlabeled pixel, if it is a foreground pixel, a new connected region is created and a unique label is assigned to the pixel. Then, the neighboring pixels of the pixel (usually 4 or 8 neighbors) are checked. If the neighboring pixels are also foreground pixels and are unlabeled, they are added to the current connected region and assigned the same label. At the same time, the pixel connected regions with the same neighborhood relationship are merged during the scanning process. Identify each character region in the handwritten text and obtain it by calculating the overlapping parts between adjacent character regions. The three parameters of character angle, edge clarity and character overlap area are interrelated in text segmentation and jointly affect the segmentation effect and noise interference evaluation. For example, deviation of character angle may lead to character recognition errors; insufficient edge clarity may make it difficult to accurately segment characters; and excessive character overlap area may increase segmentation difficulty and noise interference. The noise interference evaluation value obtained through comprehensive analysis can reflect the difficulty of character segmentation in the image and the degree of noise interference, thereby providing strong support for subsequent text segmentation and recognition tasks.

[0034] Furthermore, the step of obtaining a text background evaluation value based on a comprehensive analysis of the text background data includes: the text background data includes background color difference, background complexity and background brightness difference; obtaining a critical background color difference, a critical background complexity and a critical background brightness difference from a text segmentation database; and obtaining a text background evaluation value through a comprehensive analysis.

[0035] The text background evaluation value is obtained as follows: ; Wherein, ξB1 represents the text background evaluation value, α4 represents the text background evaluation influence factor corresponding to the background color difference, α5 represents the text background evaluation influence factor corresponding to the background complexity, α6 represents the text background evaluation influence factor corresponding to the background brightness difference, C1 represents the background color difference, C0 represents the critical background color difference, S1 represents the background complexity, S0 represents the critical background complexity, L1 represents the background brightness difference, L0 represents the critical background brightness difference, and e is a natural constant.

[0036] α4, α5 and α6 are the text background assessment influencing factors corresponding to the background color difference, background complexity and background brightness difference preset in the text segmentation database, respectively. These influencing factors are numerical indicators that measure the influence of the above background factors on the text background assessment value. Specifically, there is a mapping relationship table for each background color difference, background complexity and background brightness difference, which records each possible background factor value and its corresponding text background assessment influencing factor. These mapping relationships can be one-to-one or many-to-one. In practical applications, when it is necessary to evaluate a text background assessment value, the measured background color difference, background complexity and background brightness difference can be input into their respective corresponding mapping relationship tables, and the text background assessment influencing factors corresponding to these values ​​can be quickly found, where the value range of the influencing factor is between 0 and 1.

[0037] In this embodiment, the background color difference can be measured using a color measuring instrument such as a colorimeter or image processing software; the background complexity can be obtained by using image processing software or algorithms to extract the features of the background area (such as color histogram, texture features, etc.), divide the color space into several intervals, and count the number of pixels in each interval, use the distance metric of the color space (such as Euclidean distance, Manhattan distance, etc.) to quantify the color, and then use the gray level co-occurrence matrix to analyze the spatial relationship of the gray levels in the image to quantify the texture features, and calculate based on these features. The background complexity is obtained as follows: ; In the formula, S1 represents the background complexity, β1 represents the background complexity influence factor corresponding to the color feature, β2 represents the background complexity influence factor corresponding to the texture feature, and F i1 represents the quantized value of the i-th color feature, F j1 represents the quantized value of the jth texture feature, i is the number of each color feature, i=1,2,3,...N, N is the number of color features, j is the number of each texture feature, j=1,2,3,...,M, M is the number of texture features. Among them, the background complexity influence factor corresponding to the color feature and the background complexity influence factor corresponding to the texture feature are directly obtained from the text segmentation database.

[0038] The background brightness difference can be measured using a brightness measuring instrument (such as a photometer) or image processing software. The three factors affect each other. For example, a highly complex background may mask color differences or brightness differences, making text segmentation more difficult. The text background evaluation value obtained through comprehensive analysis helps to understand the difficulty of text segmentation under different background conditions. The larger the text background evaluation value, the easier it is to segment the text, which provides guidance for further optimizing the text segmentation algorithm.

[0039] Furthermore, the step of obtaining a text noise evaluation value by comprehensive analysis based on the noise interference evaluation value and the text background evaluation value includes: obtaining a critical noise interference evaluation value and a critical text background evaluation value from a text segmentation database; and obtaining a text noise evaluation value by comprehensive analysis.

[0040] In this embodiment, the noise interference evaluation value may be affected by the text background evaluation value. For example, when the text background evaluation value shows that the text language style is complex or the vocabulary is widely distributed, the noise interference evaluation value may increase accordingly, because the complex language structure and wide vocabulary increase the difficulty of noise identification and processing; on the contrary, if the text background evaluation value shows that the text language style is simple and the vocabulary is concentrated, the noise interference evaluation value may decrease, because the simple language structure and limited vocabulary make noise easier to identify and remove. The text noise evaluation value obtained by comprehensive analysis can accurately obtain the noise level in the text. This helps to evaluate the quality of the text and determine whether further text denoising is needed.

[0041] Furthermore, the text noise evaluation value is obtained as follows: ; In the formula, ξN represents the text noise evaluation value, α7 represents the text noise evaluation influence factor corresponding to the noise interference evaluation value, α8 represents the text noise evaluation influence factor corresponding to the text background evaluation value, ξI1 represents the noise interference evaluation value, ξI0 represents the critical noise interference evaluation value, ξB1 represents the text background evaluation value, and ξB0 represents the critical text background evaluation value.

[0042] In this embodiment, α7 and α8 are the text noise assessment impact factors corresponding to the noise interference assessment value and the text background assessment value preset in the text segmentation database, respectively. These impact factors are numerical indicators that measure the impact of the above-mentioned interference factors on the text noise assessment value. Specifically, there is a mapping relationship table for each noise interference assessment value and the text background assessment value, and the table records each possible interference factor value and its corresponding text noise assessment impact factor. These mapping relationships can be one-to-one or many-to-one. In practical applications, when it is necessary to evaluate a certain text noise assessment value, the measured noise interference assessment value and the text background assessment value can be input into their respective corresponding mapping relationship tables, and the text noise assessment impact factors corresponding to these values ​​can be quickly found, where the value range of the impact factor is between 0 and 1.

[0043] The text noise evaluation factor corresponding to the noise interference evaluation value is set to 0.4, the text noise evaluation factor corresponding to the text background evaluation value is set to 0.6, the noise interference evaluation value is set to 0.8, the critical noise interference evaluation value is set to 0.76, and the critical text background evaluation value is set to 1.2. When the text background evaluation value continues to increase, the text noise evaluation value is calculated. This is shown in Table 1, the text noise evaluation value data table of the handwritten text segmentation method based on the SAM model.

[0044] Table 1 Text noise evaluation value data table of handwritten text segmentation method based on SAM model like Figure 2 As shown in Table 1 and Figure 2 It can be seen that while the text noise evaluation factor corresponding to the noise interference evaluation value, the text noise evaluation factor corresponding to the text background evaluation value, the noise interference evaluation value, the critical noise interference evaluation value and the critical text background evaluation value remain unchanged, and the text background evaluation value continues to increase, the text noise evaluation value continues to decrease.

[0045] Furthermore, the step of adjusting feedback according to the text noise evaluation value includes: obtaining a text noise evaluation threshold from a text segmentation database; comparing the text noise evaluation value with the text noise evaluation threshold, if the text noise evaluation value is greater than the text noise evaluation threshold, providing early warning feedback to the handwritten text image, if the text noise evaluation value is less than or equal to the text noise evaluation threshold, performing a handwritten text segmentation operation.

[0046] In this embodiment, the text noise evaluation threshold is a parameter used to determine whether the noise level of the current text exceeds an acceptable range. When the text noise evaluation value is greater than the text noise evaluation threshold, it means that the noise level of the current text is too high, which may have an adverse effect on the subsequent text segmentation operation. Therefore, the system will give early warning feedback to the handwritten text image, prompting the user or system administrator to pay attention and possibly take further denoising processing; when the text noise evaluation value is less than or equal to the text noise evaluation threshold, it means that the noise level of the current text is within an acceptable range and will not have a significant impact on the subsequent text segmentation operation. The system will continue the handwritten text segmentation operation. By setting the text noise evaluation threshold and giving early warning feedback when the noise level is too high, text segmentation can be avoided in the case of severe noise interference, thereby improving the accuracy of text segmentation.

[0047] Furthermore, the encoder feature map is processed according to the noise interference evaluation value and the text background evaluation value to obtain a fused feature map, the step comprising: dividing the noise interference evaluation value by the text background evaluation value to obtain a feature ratio; obtaining the initial weights of each feature layer of the preset encoder feature map, and comparing the feature ratio with 1, if the feature ratio is greater than 1, increasing the weight of the low-level feature, if the feature ratio is equal to 1, no additional processing is performed, if the feature ratio is less than 1, increasing the weight of the high-level feature; fusing the adjusted encoder feature map to obtain a fused feature map.

[0048] In this embodiment, the division method of low-level features and high-level features is: the encoder network features are divided into four equal parts according to the number of layers, wherein the feature parts containing detail information (including color, shape, texture, edge, etc.) are low-level features, and the feature parts containing semantic information (object category, attribute features, position relationship, etc.) are high-level features. The determination of weights can be achieved through a variety of methods, such as experience-based methods, machine learning algorithm-based methods (such as least squares method, gradient descent method, etc.). When extracting low-level feature information of handwritten text, a CNN (full name Convolutional Neural Network, i.e., convolutional neural network) branch is introduced. CNN has a strong ability in processing image texture and local features; at the same time, it is processed in coordination with the ViT branch to improve segmentation accuracy. ViT is good at capturing global context information as high-level feature information for extracting handwritten text. When the feature ratio is greater than 1, it means that the text boundary is blurred, and the model will increase the weight of low-level features because these features usually contain more detail information, which helps to locate the text boundary more accurately; when the feature ratio is less than 1, it means that the background interference is strong, and the model will rely more on high-level features to capture semantic information. The present invention designs a lightweight mask decoder, proposes a dynamic feature enhancement strategy, and realizes dynamic weighted processing of different feature layers through cross-branch interaction modules. When fusing the adjusted encoder feature maps, the feature maps of the two encoders are uniformly processed through feature alignment and weighted fusion, which not only retains the global context information, but also enhances the ability to capture fine-grained edge features. Feature alignment is the process of ensuring that the feature maps output by the two encoders can be compared and fused in the same space or standard. Upsampling (such as bilinear interpolation) or downsampling (such as pooling or convolutional layers) are used to make the feature maps of CNN and ViT have the same spatial size (for example, the same height and width), and then the number of channels is adjusted through a 1x1 convolutional layer to make their number of channels consistent (for CNN output, 1x1 convolution can be used to reduce or increase the number of channels, and for ViT output, a similar method can be used to adjust the number of channels); weighted fusion is to weight the features according to their importance or contribution to obtain a more comprehensive and accurate feature representation. When the feature maps are aligned, the feature values ​​at each pixel position can be weighted. A common method is to calculate the importance of each input feature map by learning a weight matrix (for example, through a fully connected layer or convolutional layer) and weight them accordingly. For example, there are two feature maps F CNN and F VIT , the steps of weighted fusion are as follows: Where, F fusion represents the fusion feature map to be processed, μ represents the weight of the low-level feature, and F CNN represents the feature map of the CNN encoder, (1-μ) represents the weight of the high-level features, and FVIT Represents the feature map of the VIT encoder. After weighted fusion, ReLU or other activation functions are applied to enhance the nonlinear expression ability of the network and finally obtain the fused feature map.

[0049] Furthermore, the step of increasing the weight of the low-level features includes: marking the subtraction of the feature ratio and 1 as the deviation feature ratio; matching the deviation feature ratio with the weight adjustment value corresponding to each deviation feature ratio range preset in the text segmentation database to obtain the weight adjustment value of the low-level feature; adding the weight adjustment value of the low-level feature to the weight of the low-level feature to obtain the current weight value of the low-level feature.

[0050] In this embodiment, the sum of the weight of the low-level feature and the weight adjustment value is the current weight value of the low-level feature. The deviation feature ratio and the weight adjustment value corresponding to each deviation feature ratio range preset in the text segmentation database form a mapping set, and the weight adjustment value corresponding to the deviation feature ratio is obtained according to the deviation feature ratio input mapping set, wherein the mapping relationship is a many-to-one or one-to-one relationship. By increasing the weight of the low-level feature, the model will pay more attention to the changes of the low-level feature during the training process, thereby improving the sensitivity to the low-level feature, which helps the model to better capture the local details and structural information in the handwritten text; appropriate weight adjustment can optimize the performance of the model, making it more accurate and efficient when processing handwritten text segmentation tasks. By increasing the weight of the low-level feature, the model can more effectively utilize the useful information in the input image and improve the segmentation accuracy and speed. In addition, the step of increasing the weight of the high-level feature is also the same. 1 is subtracted from the feature ratio and marked as the deviation feature ratio, and then the deviation feature ratio is matched with the weight adjustment value corresponding to each deviation feature ratio range preset in the text segmentation database to obtain the weight adjustment value of the low-level feature; the weight adjustment value of the high-level feature is added to the weight of the high-level feature to obtain the current weight value of the high-level feature.

[0051] Furthermore, the step of outputting the handwritten text segmentation result according to the fused feature map includes: obtaining each pixel value and pixel threshold of the fused feature map; comparing each pixel value of the fused feature map with the pixel threshold, if a pixel value of the fused feature map is greater than the pixel threshold, then marking the pixel as a foreground pixel, if a pixel value of the fused feature map is less than or equal to the pixel threshold, then marking the pixel as a background pixel, counting each foreground pixel, and outputting the handwritten text segmentation result according to each foreground pixel.

[0052] In this embodiment, the pixel threshold is a value used to distinguish foreground and background pixels, which is directly obtained from the text segmentation database; the pixel values ​​of the fused feature map can access the value of each pixel through the interface provided by the image processing library or framework, and these values ​​are usually stored in a two-dimensional array or matrix, where each element corresponds to a pixel in the fused feature map. By using the fused feature map and the pixel threshold for classification, multiple feature information can be fully utilized to improve the accuracy of segmentation.

[0053] Furthermore, the step of outputting the handwritten text segmentation result according to each foreground pixel includes: projecting each foreground pixel area to extract feature information in each foreground pixel; obtaining each segmentation pixel point according to the feature information, and using each segmentation pixel point to segment the fused feature map to obtain the handwritten text segmentation result.

[0054] In this embodiment, the feature information includes edge information, texture information, and text features. The feature information in each foreground pixel is extracted based on the CNN encoder and the VIT encoder. At the same time, the Grounding DINO model can be used to generate the detection box coordinate information of the text line and integrate it into the Prompt encoder of SAM. These coordinate information can be used as the input of the Prompt encoder to help the model understand the structure and layout of the handwritten text more accurately. When obtaining the segmentation pixel points according to the feature information and segmenting the fused feature map, the pixels can be classified into text areas and background areas by a pixel-level classifier (such as FCN or U-Net). These segmentation pixels will directly determine the boundaries of the handwritten text area and help generate text segmentation results. The features extracted by the Prompt encoder are combined with the detection box coordinate information to guide the optimization of the feature maps of the VIT encoder and the CNN encoder. The Prompt features are used to enhance the semantic consistency of the cross-branch features, making the collaborative processing between branches more intelligent and improving the accuracy of the text segmentation task. The specific framework diagram is as follows: Figure 3 A SAM framework diagram of a handwritten text segmentation method based on the SAM model is shown.

[0055] In summary, the embodiments of the present application obtain a noise interference evaluation value by comprehensive analysis based on noise interference data, obtain a text background evaluation value by comprehensive analysis based on text background data, obtain a text noise evaluation value by comprehensive analysis based on the noise interference evaluation value and the text background evaluation value, and perform adjustment feedback based on the text noise evaluation value; obtain an encoder feature map of the handwritten text, and process the encoder feature map based on the noise interference evaluation value and the text background evaluation value to obtain a fused feature map, and output a handwritten text segmentation result based on the fused feature map, thereby achieving improved processing capabilities for handwritten text segmentation.

[0056] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0058] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0060] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0061] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A handwritten text segmentation method based on the SAM model, characterized in that: The following steps are involved: Obtain noise interference data and text background data of handwritten text; A noise interference evaluation value is obtained based on a comprehensive analysis of the noise interference data, a text background evaluation value is obtained based on a comprehensive analysis of the text background data, a text noise evaluation value is obtained based on a comprehensive analysis of the noise interference evaluation value and the text background evaluation value, and adjustment feedback is performed based on the text noise evaluation value; The encoder feature map of the handwritten text is obtained, and the encoder feature map is processed according to the noise interference evaluation value and the text background evaluation value to obtain a fused feature map, and the handwritten text segmentation result is output according to the fused feature map.

2. The handwritten text segmentation method based on the SAM model as claimed in claim 1, characterized in that: The step of obtaining the noise interference evaluation value based on the comprehensive analysis of the noise interference data comprises: The noise interference data includes character angle, edge clarity and character overlap area; Obtaining a reference character angle, an allowable deviation character angle, a critical edge definition, and a critical character overlap area from a text segmentation database; The noise interference assessment value is obtained through comprehensive analysis.

3. The handwritten text segmentation method based on the SAM model as claimed in claim 1, characterized in that: The step of obtaining the text background evaluation value based on comprehensive analysis of the text background data comprises: The text background data includes background color difference, background complexity and background brightness difference; Get the critical background color difference, critical background complexity and critical background brightness difference from the text segmentation database; The text background evaluation value is obtained through comprehensive analysis.

4. The handwritten text segmentation method based on the SAM model as claimed in claim 1, characterized in that: The step of obtaining the text noise evaluation value by comprehensive analysis of the noise interference evaluation value and the text background evaluation value comprises: Obtain a critical noise interference evaluation value and a critical text background evaluation value from a text segmentation database; The text noise evaluation value is obtained through comprehensive analysis.

5. The handwritten text segmentation method based on the SAM model as claimed in claim 4, characterized in that: The text noise evaluation value is obtained as follows: ； In the formula, ξN represents the text noise evaluation value, α7 represents the text noise evaluation influence factor corresponding to the noise interference evaluation value, α8 represents the text noise evaluation influence factor corresponding to the text background evaluation value, ξI1 represents the noise interference evaluation value, ξI0 represents the critical noise interference evaluation value, ξB1 represents the text background evaluation value, and ξB0 represents the critical text background evaluation value.

6. The handwritten text segmentation method based on the SAM model as claimed in claim 1, characterized in that: The step of adjusting feedback according to the text noise evaluation value comprises: Get text noise evaluation threshold from text segmentation database; The text noise evaluation value is compared with the text noise evaluation threshold. If the text noise evaluation value is greater than the text noise evaluation threshold, a warning feedback is given to the handwritten text image. If the text noise evaluation value is less than or equal to the text noise evaluation threshold, a handwritten text segmentation operation is performed.

7. The handwritten text segmentation method based on the SAM model as claimed in claim 1, characterized in that: The step of processing the encoder feature map according to the noise interference evaluation value and the text background evaluation value to obtain a fused feature map comprises: Divide the noise interference evaluation value by the text background evaluation value to get the feature ratio; Get the preset initial weights of each feature layer, and compare the feature ratio with 1. If the feature ratio is greater than 1, increase the weight of the low-level features. If the feature ratio is equal to 1, no additional processing is performed. If the feature ratio is less than 1, increase the weight of the high-level features. The adjusted encoder feature maps are fused to obtain a fused feature map.

8. The handwritten text segmentation method based on the SAM model as claimed in claim 7, characterized in that: The step of increasing the weight of the low-level features includes: The subtraction of the feature ratio and 1 is marked as the deviation feature ratio; Matching the deviation feature ratio with the weight adjustment value corresponding to each deviation feature ratio range preset in the text segmentation database to obtain the weight adjustment value of the low-level feature; The weight adjustment value of the low-level feature is added to the weight of the low-level feature to obtain the current weight value of the low-level feature.

9. The handwritten text segmentation method based on the SAM model as claimed in claim 1, characterized in that: The step of outputting the handwritten text segmentation result according to the fused feature map comprises: Get the pixel value and pixel threshold of the fused feature map; Compare each pixel value of the fused feature map with the pixel threshold. If a pixel value of the fused feature map is greater than the pixel threshold, the pixel is marked as a foreground pixel. If a pixel value of the fused feature map is less than or equal to the pixel threshold, the pixel is marked as a background pixel. Count the foreground pixels and output the handwritten text segmentation result based on each foreground pixel.

10. The handwritten text segmentation method based on the SAM model as claimed in claim 9, characterized in that: The step of outputting the handwritten text segmentation result according to each foreground pixel comprises: Projecting each foreground pixel area to extract feature information from each foreground pixel; Each segmentation pixel point is obtained according to the feature information, and each segmentation pixel point is used to segment the fused feature map to obtain the handwritten text segmentation result.

Images