A Handwritten Text Detection Method and System Based on Early Fusion of Multi-Source Images

Abstract

This invention discloses a handwritten text detection method and system based on early fusion of multi-source images. The method introduces a blank template image and a question-answering region mask as prior information, forming three inputs with the user-captured image. These three inputs are then independently embedded and concatenated along the channel dimension, followed by early fusion via 1×1 convolution. This allows the network to autonomously learn the correspondence between the three source features, distinguishing between handwritten and printed content without pixel-level precise alignment. The fused features are then processed by a Swin Transformer backbone network and a feature pyramid network to extract multi-scale features, and the text segmentation result is output by a detection head based on differentiable binarization. During the training phase, a region weight coefficient map is generated using the question-answering region mask, and the loss function is differentially weighted, guiding the model to focus on the question-answering region while maintaining full-image detection capability. This invention can accurately distinguish between handwritten and printed text in complex shooting scenarios, improving the handwritten detection accuracy in key areas.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and document image processing technology, and relates to a handwritten text detection method based on early fusion of multi-source images. Background Technology

[0002] In automated document image processing scenarios, such as exam paper grading and form entry, the main task is to accurately detect handwritten text in user images. Existing technologies face a fundamental contradiction: distinguishing between handwritten and printed content requires a blank template as a background reference. However, template comparison methods based on image difference demand extremely high pixel-level alignment accuracy. Factors commonly present in real-world shooting scenarios, such as perspective distortion, paper curvature, and uneven lighting, make precise alignment difficult to achieve, and the difference results are easily drowned out by noise, making it impossible to reliably extract handwritten characters. If template difference is not relied upon, and only a single user image is used as input, the model lacks prior information to distinguish between printed and handwritten characters that are similar in shape and coexist in the same area, making it difficult to overcome false positives and false negatives.

[0003] Therefore, how to effectively inject prior knowledge such as blank templates and answer areas without requiring pixel-level precise alignment, so that the model can autonomously distinguish between handwritten and printed content, has become a technical problem that needs to be solved. Summary of the Invention

[0004] To address the problems existing in the background technology, this invention proposes a handwritten text detection method based on early fusion of multi-source images.

[0005] To achieve the above objectives, the technical solution adopted by this invention is as follows: a handwritten text detection method based on early fusion of multi-source images, comprising:

[0006] Obtain the user image to be detected, the blank template image corresponding to the user image, and the pre-annotated answer area mask;

[0007] The user image is subjected to feature embedding processing to obtain a first feature map; the blank template image is subjected to feature embedding processing to obtain a second feature map; and the answer area mask is subjected to feature embedding processing to obtain a third feature map.

[0008] The first feature map, the second feature map, and the third feature map are concatenated along the channel dimension to obtain a concatenated feature map.

[0009] The spliced ​​feature map is reduced in dimensionality by using a 1×1 convolution kernel to obtain a fused feature map.

[0010] A backbone feature extraction network is used to perform multi-scale feature extraction on the fused feature map to obtain a multi-scale feature map.

[0011] A text detection network is used to detect and decode the multi-scale feature map to obtain the handwritten text detection result.

[0012] Further, the step of acquiring the user image to be detected, the blank template image corresponding to the user image, and the pre-annotated answer area mask includes:

[0013] Acquire the user image to be detected;

[0014] Based on the content features of the user image, the corresponding blank template image and the associated answer area mask are retrieved from the pre-built template database;

[0015] The user image and the blank template image are coarsely aligned in space, and the answer area mask is mapped to the spatial position corresponding to the user image.

[0016] Furthermore, the step of using a backbone feature extraction network to perform multi-scale feature extraction on the fused feature map to obtain a multi-scale feature map includes:

[0017] Feature extraction is performed on the fused feature map based on a hierarchical self-attention mechanism to obtain initial multi-scale features;

[0018] The initial multi-scale features are fused across scales and channels are unified using a feature pyramid network to obtain the multi-scale feature map.

[0019] Furthermore, the step of using a text detection network to detect and decode the multi-scale feature map to obtain the handwritten text detection result includes:

[0020] The multi-scale feature map is subjected to differentiable binarization to output a text region probability map and an adaptive threshold map.

[0021] Calculate a binarized text segmentation map based on the text region probability map and the adaptive threshold map;

[0022] The bounding boxes of the handwritten text are extracted based on the binarized text segmentation map and used as the handwritten text detection result.

[0023] Furthermore, the backbone feature extraction network and the text detection network are executed based on a handwritten text detection model, which is obtained through a model training phase, including:

[0024] Acquire user images for training, blank template images for training, answer area masks for training, and handwritten real-value annotations;

[0025] The training user image, the training blank template image, and the training answer area mask are respectively processed by feature embedding. After concatenation in the channel dimension, dimensionality reduction is performed to obtain the training fusion feature map.

[0026] The backbone feature extraction network is used to perform multi-scale feature extraction on the training fusion feature map, and the text detection network is used for detection and decoding to obtain prediction results including text region probability map, adaptive threshold map and binarized text segmentation map;

[0027] A region weight coefficient map is generated based on the training answer region mask;

[0028] Based on the region weighting coefficient map, a weighted loss calculation is performed on the difference between the prediction result and the handwritten true value annotation to obtain the total loss function;

[0029] Update network parameters based on the total loss function.

[0030] Further, the step of generating a region weight coefficient map based on the training answer region mask includes:

[0031] Assign a first weight coefficient to the internal pixels of the training answer region mask;

[0032] Assign a second weighting coefficient to the outer pixels of the training answer region mask;

[0033] The first weight coefficient is greater than the second weight coefficient, and the second weight coefficient is greater than zero.

[0034] Furthermore, the total loss function is obtained by summing the probability graph loss component, the threshold graph loss component, and the binary graph loss component;

[0035] The probability map loss component is obtained by multiplying the pixel-by-pixel basic error value between the text region probability map and the corresponding truth label by the weight coefficient of the corresponding spatial position in the region weight coefficient map and summing them up.

[0036] The threshold map loss component is obtained by multiplying the pixel-by-pixel basic error value between the adaptive threshold map and the corresponding ground truth label by the weight coefficient of the corresponding spatial position in the region weight coefficient map and summing them up.

[0037] The binary image loss component is obtained by multiplying the pixel-by-pixel basic error value between the binary text segmentation image and the corresponding ground truth label by the weight coefficient of the corresponding spatial position in the region weight coefficient image and summing them up.

[0038] This technical solution also provides a handwritten text detection system based on early fusion of multi-source images, including:

[0039] The multi-source input acquisition module is configured to acquire the user image to be detected, the blank template image corresponding to the user image, and the pre-annotated answer area mask;

[0040] The feature embedding module includes a first feature embedding unit, a second feature embedding unit, and a third feature embedding unit; the first feature embedding unit is configured to perform feature embedding processing on the user image to obtain a first feature map; the second feature embedding unit is configured to perform feature embedding processing on the blank template image to obtain a second feature map; and the third feature embedding unit is configured to perform feature embedding processing on the answer area mask to obtain a third feature map.

[0041] The early stitching module is configured to stitch the first feature map, the second feature map, and the third feature map along the channel dimension to obtain a stitched feature map;

[0042] The dimension reduction and fusion module is configured to perform convolutional dimension reduction on the spliced ​​feature map using a 1×1 convolutional kernel to obtain a fused feature map.

[0043] The feature extraction module is configured to use a backbone feature extraction network to perform multi-scale feature extraction on the fused feature map to obtain a multi-scale feature map;

[0044] The detection output module is configured to use a text detection network to detect and decode the multi-scale feature map to obtain the handwritten text detection result.

[0045] Furthermore, the multi-source input acquisition module is also configured as follows:

[0046] Based on the content features of the user image, the corresponding blank template image and the associated answer area mask are retrieved from the pre-built template database;

[0047] The user image and the blank template image are coarsely aligned in space, and the answer area mask is mapped to the spatial position corresponding to the user image.

[0048] Furthermore, a handwritten text detection system based on early fusion of multi-source images also includes a model training module, comprising:

[0049] The training data acquisition unit is configured to acquire user images for training, blank template images for training, answer area masks for training, and handwritten real value annotations.

[0050] The training forward propagation unit is configured to perform feature embedding processing on the training user image, the training blank template image, and the training answer region mask respectively, and perform convolutional dimensionality reduction after concatenation in the channel dimension to obtain a training fused feature map; the backbone feature extraction network is used to perform multi-scale feature extraction on the training fused feature map, and the text detection network is used for detection and decoding to obtain the prediction result;

[0051] The region weight generation unit is configured to generate a region weight coefficient map based on the training answer region mask, wherein a first weight coefficient is assigned to the inner pixels of the training answer region mask, and a second weight coefficient is assigned to the outer pixels of the training answer region mask, wherein the first weight coefficient is greater than the second weight coefficient and the second weight coefficient is greater than zero.

[0052] The weighted loss calculation unit is configured to perform weighted loss calculation on the difference between the prediction result and the handwritten true value annotation based on the regional weight coefficient map, and obtain the total loss function.

[0053] The parameter update unit is configured to update network parameters based on the total loss function.

[0054] Compared with the prior art, the present invention has the following beneficial effects: First, by introducing a blank template image as a background reference and replacing the traditional image difference method with an early fusion method of channel splicing plus 1×1 convolution, the model can autonomously establish the correspondence between the user image and the template during the feature learning stage, accurately distinguish between handwritten content and printed background coexisting in the same area, reduce the false detection rate caused by the similarity between printed and handwritten characters, and at the same time have strong robustness to factors such as perspective distortion and uneven lighting in actual shooting scenes.

[0055] Second, by introducing a mask for the answer area and assigning differentiated weights to the loss function, the model becomes more sensitive to the detection error of key answer areas during training. While focusing on improving the handwriting detection accuracy within the answer area, it maintains the ability to detect handwritten content in non-answer areas of the entire image, avoiding false detections of printing due to completely ignoring non-answer areas, thus achieving a balance between detection accuracy and detection completeness. Attached Figure Description

[0056] Figure 1 This is a flowchart of a handwritten text detection method based on early fusion of multi-source images according to the present invention;

[0057] Figure 2 This is a schematic diagram of the network architecture of the handwritten text detection model of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] like Figures 1-2 As shown, the technical solution adopted in this invention is as follows: A handwritten text detection method based on early fusion of multi-source images, comprising:

[0060] S1: Obtain the user image to be detected, the blank template image corresponding to the user image, and the pre-annotated answer area mask.

[0061] The user image is a document image submitted by the user via a camera, containing the handwritten answer to be detected and a printed background. The printed background includes question numbers, options, explanatory text, etc. The user image serves as the target image to be detected in the method.

[0062] The blank template image is a blank version of the original document corresponding to the user image. It contains only printed content and no handwritten marks. It serves as a background reference to help the model distinguish between the original printed content and the newly added handwritten content.

[0063] The answer area mask is a pre-labeled image that marks the location of the area in the user image where handwritten answers are expected to be filled in. The labeling content includes the boundaries of answer boxes, horizontal lines, blank areas, etc. This mask serves as spatial prior information and guides the model to focus on key answer areas during the model training phase.

[0064] Specifically, the step of acquiring the user image to be detected, the blank template image corresponding to the user image, and the pre-annotated answer area mask includes:

[0065] Obtain the user image to be detected.

[0066] Based on the content features of the user image, the corresponding blank template image and the associated answer area mask are retrieved from the pre-built template database.

[0067] The user image and the blank template image are coarsely aligned in space, and the answer area mask is mapped to the spatial position corresponding to the user image.

[0068] Specifically, the content features of the user image are first extracted. These content features include, but are not limited to, layout features, QR code or barcode identifiers, page fingerprints extracted based on scale-invariant feature transformation or directional rapid rotation feature points, or key text fragments obtained through optical character recognition.

[0069] Based on the extracted content features, the user image is first subjected to content feature extraction. The content features include, but are not limited to, layout features (such as text line distribution and title position), QR code or barcode identification, page fingerprint formed by visual bag-of-words vectors extracted based on scale-invariant feature transformation or directional fast rotation feature points, or key text fragments obtained through optical character recognition.

[0070] Then, the extracted content features are used for similarity matching in a pre-constructed template database, which stores all possible blank template images and answer region masks associated with each blank template image. When the content features of the user image have the highest similarity to the features of a record in the template database and exceed a preset threshold, the blank template image and associated answer region mask corresponding to that record are determined as the retrieval result and output. Since the answer region mask and the blank template image are stored in association in the database, the same template corresponds to a fixed mask label, ensuring the consistency of spatial prior information.

[0071] The step of coarsely aligning the user image and the blank template image involves applying perspective transformation or thin-plate spline transformation to the two images to make the document corners roughly correspond. This alignment operation does not require pixel-level precision. The transformation matrix required for the geometric transformation is calculated by detecting and matching document corners or scale-invariant feature transformation feature points in the two images.

[0072] After completing the coarse spatial alignment, the same calculated geometric transformation matrix is ​​used to map the answer region mask from its original blank template image coordinate system to the corresponding spatial position in the user image. This ensures that the marked answer areas in the mask roughly cover the corresponding answer boxes, lines, or blank areas in the user image. This coarse alignment operation reduces the precision requirements of image preprocessing. Subsequently, the neural network autonomously adapts to subtle spatial deviations during feature learning, resulting in stronger robustness in actual shooting scenarios. This avoids the problem of traditional pixel-difference-based template comparison methods introducing a large amount of noise due to inaccurate alignment.

[0073] S2: Perform feature embedding processing on the user image to obtain a first feature map; perform feature embedding processing on the blank template image to obtain a second feature map; perform feature embedding processing on the answer area mask to obtain a third feature map.

[0074] Specifically, the user image, the blank template image, and the answer region mask are each processed by an independent feature embedding module (Patch Embedding module). The three feature embedding modules have the same structural parameters, including the same block size and the same embedding dimension, to ensure that the feature maps output by each module remain aligned in the spatial dimension, thereby satisfying the prerequisite for channel-dimensional stitching.

[0075] The process is illustrated using the feature embedding processing of the user image as an example: Let the user image be... Its size is ,in Image height, 3 represents the image width, and 3 represents the number of RGB color channels. After the user image is input into the first feature embedding processing module, this module divides the input image into several non-overlapping image patches through a convolution operation, and maps each image patch to a length of 3. The embedding vector. The specific components of the first feature embedding processing module include: a convolutional layer with a kernel size of... pixels, convolution stride of 4 pixels, padding of 3 pixels, output channel number of A LayerNorm is used to normalize the features output by the convolution to stabilize the training process; and a Gaussian Error Linear Unit (GELU) activation function is used to introduce non-linear transformation capability. After processing by the first feature embedding module, the output first feature map is denoted as... Its spatial dimensions are Since the convolution stride is 4, the first feature map is downsampled by a factor of 4 in both the height and width directions.

[0076] Similarly, the blank template image is input into the second feature embedding processing module, and undergoes a processing flow with the same structural parameters as the first feature embedding processing module, that is, it sequentially passes through a convolution kernel with a size of... After processing by a convolutional layer with a pixel count, a stride of 4 pixels, padding of 3 pixels, and 128 output channels, a normalization layer, and a Gaussian error linear unit activation function, a second feature map is output. The second feature map has the same dimensions as the first feature map, i.e., its height and width are each a fraction of the blank template image. The number of channels is 128.

[0077] The answer region mask is a single-channel image, input to the third feature embedding processing module, and undergoes the same processing flow as the two modules mentioned above, outputting a third feature map. The size of the third feature map is the same as the first and second feature maps, i.e., its height and width are each a fraction of the answer region mask. The number of channels is 128.

[0078] Three independent feature embedding processing modules process the three inputs in parallel, each outputting a feature map with the same spatial resolution and number of channels. This design aims to maintain the independence of the three inputs at the lowest level of feature extraction, ensuring that the unique information of each input (such as the mixed handwriting and printing information of the user image, the pure printing background information of the blank template image, and the spatial prior information of the answer area mask) is fully preserved, avoiding premature information mixing that could lead to feature contamination. Simultaneously, the identical downsampling rate and embedding dimension ensure accurate alignment of the three feature maps in subsequent channel-level concatenation operations.

[0079] In one implementation, when the input image size is uniformly scaled to... At pixel level, the feature map space size output by each feature embedding processing module is 1. Each feature map has 128 pixels and 128 channels. The three feature maps represent information from the same document page from different perspectives: the first feature map... The second feature map encodes the features of a mixed image, including handwritten answers and a printed background, actually taken by the user; The third feature map encodes the blank template background features containing only printed content corresponding to the user image; The spatial location features of the answer area where handwritten answers are expected to appear are encoded. These three types of feature maps constitute the input for subsequent early fusion operations.

[0080] S3: The first feature map, the second feature map, and the third feature map are concatenated along the channel dimension to obtain a concatenated feature map.

[0081] The first feature map is denoted as Its size is ,in The height of the input image. The width of the input image. The downsampling factor of the feature embedding processing module. The embedding channel cardinality is denoted as . The second feature map is denoted as . Its size is similar to the first feature map Same, for The third feature map is denoted as... Its size is similar to the first feature map and the second feature map Same, for The three feature maps have the same height and width in the spatial dimension and the same number of channels in the channel dimension. This alignment characteristic is a prerequisite for performing channel-dimensional stitching operations.

[0082] The process of the channel dimension splicing operation is as follows: in the channel directions of the three feature maps, the... , and They are cascaded sequentially. Because each of the three feature maps has... Each channel, after being stitched together, yields a stitched feature map denoted as... Its spatial dimensions remain as The number of channels remains unchanged, while the number of channels changes. .

[0083] In one specific implementation, when the input image size is uniformly scaled to... Pixels, downsampling rate Embedded channel base At that time, the first feature map The second feature map and the third feature map All space dimensions are The pixel count and channel count are both 128. After concatenation along the channel dimension, the concatenated feature map... The space dimensions are Pixels, with 384 channels (i.e.) ).

[0084] The technical significance of this channel-dimensional concatenation operation lies in achieving non-differential fusion. Unlike traditional image differencing methods (i.e., performing pixel-by-pixel subtraction between the user image and the template image to obtain the difference region), this scheme does not use pixel-by-pixel subtraction; instead, it concatenates the three feature maps along the channel dimension. This design enables the neural network to perform subsequent... During convolutional dimensionality reduction, the model automatically establishes the correspondence between the three input features through learning, rather than relying on manually designed difference rules. Therefore, the channel dimension concatenation operation makes the model robust to pixel misalignment between the user image and the blank template image, eliminating the need for pixel-level precise alignment during the input stage and thus reducing the accuracy requirements of image preprocessing.

[0085] The splicing feature map A complete representation containing three sources of information: the first feature map It provides mixed features of handwritten answers and printed backgrounds in images actually captured by the user; the second feature map A blank template background feature containing only printed content is provided; the third feature map Prior features of the spatial location of the answer area are provided. These three features are jointly represented in the channel dimension, forming the basis for early fusion, and subsequent fusion is achieved through... Convolutional kernels perform channel dimensionality reduction, enabling the network to automatically learn the mapping relationship between the three source features, thereby inputting the three source information into the subsequent backbone feature extraction network in a fused form.

[0086] S4: Perform convolutional dimensionality reduction on the spliced ​​feature map using a 1×1 convolution kernel to obtain the fused feature map.

[0087] The 1×1 convolutional kernel is a convolutional kernel with a size of [missing value]. The convolutional layer for pixels is configured with the following parameters: the number of input channels equals the number of stitched feature maps. The number of channels, i.e. The number of output channels equals the number of fused channels. The convolution stride is 1 pixel; the padding is 0 pixels.

[0088] The mathematical essence of the 1×1 convolution kernel is to perform a linear combination of the pixel vectors at each spatial location along the channel dimension, that is, to concatenate the feature map. In spatial location The length at is The channel vector, multiplied by a size of The weight matrix produces a matrix of length [missing information]. The new channel vector.

[0089] In one specific implementation, the embedded channel base The value is 128, and the spliced ​​feature map The number of channels is 384, and the number of fused channels is... Set to the number of embedded channels The values ​​are equal, i.e., 128. At this time, the 1×1 convolution kernel has 384 input channels, 128 output channels, a convolution stride of 1, and padding of 0.

[0090] The fused feature map is output after processing by the 1×1 convolution kernel. Spatial dimensions and the splicing feature map Same, for ,in The height of the input image. The width of the input image. The downsampling factor of the feature embedding processing module; the fused feature map The number of channels was reduced from 384 to 128. When the input image size is... Pixels, downsampling rate At that time, the fused feature map The space dimensions are Pixels, 128 channels.

[0091] The technical effects achieved by the 1×1 convolution dimensionality reduction operation include two aspects.

[0092] First, channel dimensionality reduction: reducing the number of channels after splicing and expansion. Compress back to the same number of channels as a single-path feature map This effectively controls the computational complexity of the subsequent backbone feature extraction network, avoiding a significant increase in the number of parameters and computational load caused by the three-source splicing.

[0093] Second, channel mixing and information fusion: the 1×1 convolutional kernels perform linear combination in the channel dimension, enabling the network to automatically learn the first feature map. The second feature map and the third feature map The optimal linear combination method between channels. This process achieves non-differential adaptive fusion, that is, the network learns to establish a mapping relationship by which channel features of the blank template image should be referenced for a feature at a certain location in the user image, and whether it is affected by the prior of the mask space of the answer region, without relying on manually designed pixel-level difference rules.

[0094] Therefore, the fused feature map Not only does it preserve complete spatial structure information, but it also injects the three sources of prior knowledge into all subsequent network layers in the form of joint features, forming the core link of early fusion.

[0095] In another implementation, the number of fusion channels It can be set to the number of embedded channels. The number of fusion channels can be an integer multiple, such as twice, to provide greater feature capacity; or in deployment scenarios with limited computing resources, the number of fusion channels can be... It can be set to the number of embedded channels. Half of the number of fusion channels, to reduce computational overhead. The specific value can be adjusted according to the actual task requirements and hardware constraints.

[0096] S5: Use a backbone feature extraction network to perform multi-scale feature extraction on the fused feature map to obtain a multi-scale feature map.

[0097] The backbone feature extraction network is built on the Swing Transformer architecture, characterized by a hierarchical structure and a shifted window attention mechanism to achieve multi-scale feature extraction from the input feature map, from local to global. The backbone feature extraction network consists of multiple cascaded stages, each containing several Swing Transformer modules. The core of each Swing Transformer module is an alternating arrangement of Window Multi-head Self-Attention (W-MSA) and Shifted Window Multi-head Self-Attention (SW-MSA). Patch Merging operations are used between adjacent stages to downsampling the spatial resolution and expanding the channel dimension, thus forming a pyramid-shaped hierarchical feature representation.

[0098] In one specific implementation, the backbone feature extraction network configured with Swin-Tiny has the following structural parameters: the window size is set to... Pixels; comprising four stages, each stage containing 2, 2, 6, and 2 SwinTransformer modules respectively; the feature maps output by each stage are denoted as... , , , .

[0099] The input for the first stage is the fused feature map. Its spatial dimensions are The number of channels is ,in The height of the input image. The width of the input image. The downsampling factor of the feature embedding processing module. This represents the number of fusion channels. The feature map output from the first stage. The spatial resolution is the input fused feature map of The number of channels is 128.

[0100] The feature map output in the second stage The spatial resolution is the input fused feature map of The number of channels is 256.

[0101] The feature map output in the third stage The spatial resolution is the input fused feature map of The number of channels is 512.

[0102] Feature map output in the fourth stage The spatial resolution is the input fused feature map of The number of channels is 1024. When the input image size is... Pixels, downsampling rate Number of fusion channels At that time, the spatial dimensions of the output feature maps at each stage are as follows: Pixels ), Pixels ), Pixels ), Pixels ).

[0103] The multi-scale feature map , , , This constitutes the initial multi-scale features, capturing visual features under different receptive fields: low-level feature maps (such as...) It has high spatial resolution and a small receptive field, preserving rich spatial details and edge information; high-level feature maps (such as...) This set of feature maps has lower spatial resolution and a larger receptive field, encoding stronger semantic information and global context. It will be fed into a subsequent feature pyramid network for cross-scale fusion and channel unification to further enhance the representation capability of multi-scale features.

[0104] Specifically, the step of using a backbone feature extraction network to perform multi-scale feature extraction on the fused feature map to obtain a multi-scale feature map includes:

[0105] Feature extraction is performed on the fused feature map based on a hierarchical self-attention mechanism to obtain initial multi-scale features.

[0106] The initial multi-scale features are fused across scales and channels are unified using a feature pyramid network to obtain the multi-scale feature map.

[0107] The multi-scale feature extraction process of the backbone feature extraction network is decomposed into two sub-steps: hierarchical self-attention feature extraction and cross-scale fusion of feature pyramids.

[0108] The first sub-step involves processing the fused feature map based on a hierarchical self-attention mechanism. Feature extraction is performed to obtain initial multi-scale features. This sub-step is completed by the four stages of the aforementioned Swin Transformer backbone network.

[0109] The hierarchical self-attention mechanism works as follows: Within each Swing Transformer module, self-attention is calculated within a fixed-size window using window multi-head self-attention (W-MSA), ensuring that each image patch interacts only with other image patches within the same window, thus acquiring local context information with linear computational complexity. Subsequently, shifted window multi-head self-attention (SW-MSA) spatially shifts the window, enabling image patches originally located in different windows to interact across windows, thereby expanding the receptive field without significantly increasing computational overhead and achieving feature aggregation from local to global perspectives.

[0110] Patch merging operations between stages will be adjacent The features of the spatial blocks are concatenated along the channel dimension and then subjected to dimensionality reduction through a linear transformation, achieving a halving of spatial resolution and a doubling of the number of channels, thereby constructing a pyramid-shaped hierarchical feature structure. The initial multi-scale features output by this sub-step are the aforementioned feature map. , , , .

[0111] The second sub-step involves using a Feature Pyramid Network (FPN) to perform cross-scale feature fusion and channel unification on the initial multi-scale features, resulting in the multi-scale feature map. The function of the Feature Pyramid Network is to progressively transfer the strong semantic information of the high-level feature maps to the lower-level feature maps via a top-down path, while maintaining the spatial accuracy of each layer's features through lateral connections, ultimately outputting an enhanced feature map that incorporates multi-scale information.

[0112] In one specific implementation, the feature pyramid network receives a three-layer feature map from the initial multi-scale features output by the backbone feature extraction network. , , The processing is performed, and its spatial resolution is respectively the input fused feature map. of , , The processing flow of the feature pyramid network is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] conduct The convolution operation adjusts the number of channels to 256, and then the spatial resolution is improved to match the feature map through an upsampling operation. Same; for feature maps conduct The convolution operation adjusts the number of channels to 256, and then compares it with the upsampled... The features are added element by element to obtain intermediate features. ; for the intermediate features pass After smoothing the convolution, it is upsampled again to match the feature map. Same spatial resolution; for feature maps conduct The convolution operation adjusts the number of channels to 256, and then compares it with the upsampled... The features are added element by element to obtain intermediate features. .

[0113] Then, the intermediate features of each layer were processed separately. Convolution smoothing, output feature map , , The number of channels is uniformly set to 256. Finally, the feature map... , , All were upsampled to feature maps. Spatial resolution, stitched along the channel dimension and then... Convolution reduces the number of channels to 256, yielding the final fused multi-scale feature map. .

[0114] The cross-scale fusion mechanism of the feature pyramid network enables low-level feature maps to enhance their ability to discriminate small-scale handwritten text by leveraging the semantic guidance of high-level feature maps. Simultaneously, high-level feature maps can accurately locate text boundaries using detailed information from low-level feature maps, thereby improving the overall detection capability for handwritten text of different scales and shapes. The final fused multi-scale feature map... The text will be input into the subsequent detection head module for handwritten text detection and decoding.

[0115] S6: Use a text detection network to detect and decode the multi-scale feature map to obtain the handwritten text detection result.

[0116] The text detection network is constructed based on the differentiable binarization module of DBNet. The text detection network receives a fused multi-scale feature map output by the feature pyramid network. As input, the spatial resolution of the feature map and the feature map Same, the number of channels is 256.

[0117] The text detection network contains two parallel prediction branches: a probability graph branch and a threshold graph branch. The probability graph branch consists of two... Convolutional layer and one The convolutional layers consist of, in which the first one... The convolutional layer reduces the number of channels from 256 to 128, the second one... The convolutional layer reduces the number of channels from 128 to 64, and finally... The probability map output by the convolutional layer in a single channel is processed by the sigmoid activation function and then upsampled to the input image size using bilinear interpolation to obtain the text region probability map. .

[0118] The threshold graph branch also consists of two. Convolutional layer and one The system consists of convolutional layers that output a single-channel adaptive threshold map. After processing with a sigmoid activation function, the image is upsampled to the input image size to obtain the adaptive threshold map. In one specific implementation, when the input image size is... The probability map of the text region at pixel level and the adaptive threshold map All dimensions are Pixel.

[0119] The differentiable binarization mechanism replaces the step function in the standard binarization operation with a differentiable sigmoid function, thereby enabling the binarization process to participate in gradient backpropagation during the training phase. Based on the text region probability map... and the adaptive threshold map The binarized text segmentation map is obtained by performing a differentiable binarization operation. In this binary text segmentation image, the value of each pixel is in the range [0,1], which represents the probability that the pixel belongs to the handwritten text region.

[0120] After obtaining the binary text segmentation map Subsequently, the text detection network performs post-processing to extract structured handwritten text detection results. The post-processing procedure includes: processing the binarized text segmentation map... Threshold filtering is performed, and a preset threshold (ranging from 0.3 to 0.5) is used to obtain a binary mask. Connectivity analysis is then performed on the binary mask to extract the contour of each connected region. For each contour, the Douglas-Peucker algorithm is used for polygon approximation to obtain an approximate bounding box. Non-Maximum Suppression (NMS) is applied to all detected bounding boxes, with the Intersection over Union (IoU) threshold set to 0.5 to remove duplicate detections. Finally, the four coordinates and confidence score of each detection box are output, where the confidence score is derived from the probability map of the text region within the detection box. The average value is calculated. The output handwritten text detection result is a set of structured data containing bounding box coordinates and confidence scores, identifying all detected handwritten text regions in the user image.

[0121] Specifically, the step of using a text detection network to detect and decode the multi-scale feature map to obtain the handwritten text detection result includes:

[0122] The multi-scale feature map is subjected to differentiable binarization to output a text region probability map and an adaptive threshold map.

[0123] A binarized text segmentation map is calculated based on the text region probability map and the adaptive threshold map.

[0124] The bounding boxes of the handwritten text are extracted based on the binarized text segmentation map and used as the handwritten text detection result.

[0125] Specifically, the detection and decoding process of the text detection network is decomposed into three sub-steps: differentiable binarization processing to generate a probability map and a threshold map, calculating a binarized text segmentation map, and post-processing to extract bounding boxes.

[0126] The first sub-step involves performing differentiable binarization on the multi-scale feature map, outputting a text region probability map and an adaptive threshold map. The multi-scale feature map is the fused multi-scale feature map output by the feature pyramid network. After being input into the two parallel branches of the text detection network, it undergoes convolution processing through the probability graph branch and the threshold graph branch, respectively.

[0127] The probability graph of the text region output by the probability graph branch In the diagram, each pixel takes a value in the range [0,1], representing the probability that the pixel is predicted to be a handwritten text region. The adaptive threshold map output by the threshold map branch... In this model, each pixel's value is also within the range [0,1], representing the adaptive threshold used when performing differentiable binarization at that pixel location. Unlike traditional fixed threshold binarization, this adaptive threshold map... This allows text pixels at different spatial locations to use different binarization thresholds based on their local features, thereby better handling text boundary determination under complex conditions such as uneven lighting and variations in stroke thickness.

[0128] The second sub-step involves calculating a binarized text segmentation map based on the text region probability map and the adaptive threshold map. The binarized text segmentation map... It is calculated through differentiable binarization.

[0129] The principle of the differentiable binarization operation is to transform the probability map of the text region. The probability value of each pixel in the adaptive threshold map The threshold values ​​at corresponding positions are differentially analyzed, and the difference result is multiplied by a preset amplification factor k. Then, the result is mapped to the [0,1] interval using the Sigmoid function. The amplification factor k controls the sharpness of the binarization transition: when the value of k is large, the output of the Sigmoid function approaches the binary output of the step function, making the binarization result more distinct. In one specific embodiment, the amplification factor k is set to 50.

[0130] The computational process of the differentiable binarization operation can be described by the following mathematical expression:

[0131] ;

[0132] in: This represents a binary text segmentation map with dimensions of [size missing]. ; This represents a probability map of a text region, with dimensions of [size missing]. Each pixel value is in the range [0,1], representing the probability that the pixel belongs to the handwritten text area; This represents an adaptive threshold map with dimensions of [size missing]. Each pixel value is in the range [0,1], representing the differentiable binarization adaptive threshold at that pixel location; k represents a preset magnification factor used to control the sharpness of the binarization transition; The sigmoid activation function is expressed as follows: , is used to map the input to the (0,1) interval.

[0133] The third sub-step involves, based on the binarized text segmentation map... The bounding boxes of the handwritten text are extracted and used as the handwritten text detection result. This sub-step converts the pixel-level segmentation result into instance-level bounding boxes through a post-processing workflow.

[0134] First, the binarized text segmentation image is processed... Threshold filtering is performed to obtain a binary mask by setting a preset threshold. Then, connected component analysis is performed on the binary mask to extract the contour of each connected region. Next, the Douglas-Peucker algorithm is used to approximate each contour into a polygon bounding box. Non-maximum suppression is then applied to all detected bounding boxes, with an intersection-over-union (IoU) threshold set to 0.5 to remove duplicate detections. Finally, the four coordinates and confidence score of each detection box are output, where the confidence score is derived from the probability map of the text region within the detection box. The average value.

[0135] The handwritten text detection results are represented in the form of structured data. Each detection box contains four coordinates (corresponding to the four corner points of the text box) and a confidence score, which fully identifies the location and detection reliability of all detected handwritten text regions in the user image.

[0136] Specifically, the backbone feature extraction network and the text detection network are executed based on a handwritten text detection model, which is obtained through a model training phase, including:

[0137] Acquire user images for training, blank template images for training, answer area masks for training, and handwritten real-value annotations.

[0138] The user image used for training, the blank template image used for training, and the answer area mask used for training are respectively subjected to feature embedding processing. After concatenation in the channel dimension, dimensionality reduction is performed to obtain the fused feature map used for training.

[0139] The backbone feature extraction network is used to perform multi-scale feature extraction on the training fusion feature map, and the text detection network is used for detection and decoding to obtain prediction results including a text region probability map, an adaptive threshold map, and a binarized text segmentation map.

[0140] A region weight coefficient map is generated based on the training answer region mask.

[0141] The total loss function is obtained by calculating the weighted loss between the predicted result and the handwritten true value annotation based on the regional weight coefficient map.

[0142] Update network parameters based on the total loss function.

[0143] The model training phase includes six sub-steps:

[0144] The first sub-step: Specifically, the acquired training data includes four categories: user images for training, blank template images for training, answer area masks for training, and handwritten real value annotations.

[0145] The training user image is a document image containing handwritten answers and a printed background; the training blank template image is a blank document image containing only printed content, corresponding to the training user image; the training answer area mask is a label image marking the position of the answer area; the handwritten true value annotation is a pixel-level binary segmentation label, wherein the handwritten stroke area is marked as 1 and the background area is marked as 0.

[0146] In one specific implementation, the training dataset contains 10,000 real user photos of exam papers, each photo corresponding to a blank PDF template. The answer area mask is generated by semi-automatic annotation, and the handwritten real-value annotations are pixel-level binary segmentation images. During the data preprocessing stage, the user images used for training and the blank template images used for training are coarsely aligned spatially, and the image sizes are uniformly scaled to [size missing]. Pixels; apply the same data augmentation operation synchronously to all three inputs, including random rotation (angle range of ). The enhancements include random scaling (ranging from 0.9 to 1.1 times), random cropping, brightness jitter (adjustment coefficient ranging from 0.8 to 1.2), and Gaussian noise addition (standard deviation ranging from 0 to 5). All enhancement operations are applied synchronously to the three inputs through the same random seed to ensure that the spatial correspondence remains unchanged.

[0147] The second sub-step: Specifically, feature embedding processing is performed on the user image used for training to obtain a first feature map, feature embedding processing is performed on the blank template image used for training to obtain a second feature map, and feature embedding processing is performed on the answer region mask used for training to obtain a third feature map; the three feature maps are then concatenated along the channel dimension to obtain a concatenated feature map; using... The convolution kernel performs convolutional dimensionality reduction to obtain the fused feature map for training. The generation method of the fused feature map for training is the same as that of the fused feature map during the inference phase. Totally consistent.

[0148] The third sub-step involves: specifically, using the backbone feature extraction network to extract multi-scale features from the fused feature map used for training, obtaining initial multi-scale features; using a feature pyramid network to perform cross-scale feature fusion and channel unification on the initial multi-scale features, obtaining a fused multi-scale feature map; and using the text detection network (based on the DBNet differentiable binarization module) to detect and decode the fused multi-scale feature map, outputting a text region probability map. Adaptive threshold map and the binarized text segmentation map calculated through differentiable binarization operation. These three types of outputs constitute the prediction result.

[0149] Fourth sub-step: The region weight coefficient map The generation rule is as follows: For pixels inside the training answer region mask (i.e., the answer region marked by the mask), a first weight coefficient is assigned. For pixels outside the training answer region mask (i.e., non-answer region), a second weight coefficient is assigned. Wherein, the first weighting coefficient The value is greater than the second weighting coefficient. The value of , and the second weighting coefficient Greater than zero.

[0150] In one specific implementation, the first weighting coefficient The value is 3, the second weighting coefficient The value is set to 1. The purpose of this differentiated weight allocation is to impose a higher penalty (weight) on detection errors within the answer area. This makes the model more sensitive to handwriting detection accuracy in key answer areas during training; and maintains a normal proportion of penalty (weight) for detection errors in non-answer areas. This ensures the model retains its ability to detect handwritten content across the entire image, preventing misidentification of printed text as handwritten. The weights for non-answer regions are not zero to allow the model to learn to distinguish between handwritten and printed content in these areas, avoiding misidentifications caused by completely ignoring non-answer regions.

[0151] Fifth sub-step: Based on the aforementioned regional weight coefficient map The difference between the predicted result and the handwritten true value annotation is weighted and summed to obtain the total loss function. The total loss function... It consists of a weighted sum of three loss components: the probabilistic graphical loss component. Threshold map loss components Binary plot loss components The weighting coefficients corresponding to each loss component are denoted as follows: , and In one implementation, all three weighting coefficients are set to 1.

[0152] The probability graph loss component The calculation method is as follows: First, calculate the probability map of the text region. The pixel-by-pixel binary classification cross-entropy base error between the predicted probability value of each pixel in the handwritten true value annotation and the true label of the corresponding pixel in the annotation.

[0153] Then, the base error value of each pixel is multiplied by the weight coefficient of the corresponding spatial location in the region weight coefficient map $W$ to obtain the weighted error value.

[0154] Finally, the weighted error values ​​of all pixels are summed. The threshold map loss component... The calculation method is as follows: First, calculate the adaptive threshold map. The pixel-wise Smooth L1 baseline error value between the prediction threshold and the corresponding ground truth threshold label for each pixel is calculated. Then, the baseline error value of each pixel is multiplied by the weight coefficient of the corresponding spatial location in the region weight coefficient map $W$ to obtain the weighted error value.

[0155] Finally, the weighted error values ​​of all pixels are summed. The truth threshold label is typically set as an outward Gaussian decaying label centered on the handwritten text region.

[0156] The binary image loss component The calculation method is as follows: First, calculate the binarized text segmentation map. The pixel-wise binary cross-entropy base error value between the predicted value of each pixel and the true label of the corresponding pixel in the handwritten true value annotation is calculated; then, the base error value of each pixel is multiplied by the region weight coefficient map. The weighted error value is obtained by assigning weight coefficients to the corresponding spatial locations. Finally, the weighted error values ​​of all pixels are summed. The total loss function... Equal to the probability map loss component The threshold map loss component and the binary graph loss component sum.

[0157] Sixth sub-step: Based on the total loss function The gradients of each network parameter are calculated using the backpropagation algorithm, and the network parameters are updated using the optimizer to minimize the total loss function.

[0158] In one specific implementation, the AdamW optimizer is used for parameter updates, with the initial learning rate set to... The momentum parameter is set to (0.9, 0.999), and the weight decay coefficient is set to... The number of training cycles is set to 50 epochs; the batch size is set to 8 (which can be adjusted to 4 or 16 depending on the GPU memory).

[0159] The learning rate employs a cosine annealing decay scheduling strategy, where the maximum number of cycles is... Set to 50, minimum learning rate Set as Gradient clipping is performed during training, with the maximum norm threshold set to 1.0.

[0160] In each training batch, after completing the first to fifth sub-steps, the sixth sub-step is executed. This involves calculating the partial derivatives of the loss function with respect to the network parameters of each layer via backpropagation, and then updating the network parameters using the AdamW optimizer based on the gradient and a preset learning rate. After a preset 50 training epochs, the model converges, the optimal weight parameters are saved, and the resulting model can then be used for handwritten text detection during the inference phase.

[0161] It is worth noting that there are the following key differences between the model training phase and the inference phase: the training phase requires the handwritten ground truth annotations as supervision signals and calculates the total loss function to update the network parameters; the inference phase does not require calculating the loss function, nor does it require backpropagation and parameter updates. It only requires the input of the user image, the blank template image, and the answer region mask, and directly outputs the text region probability map or the binarized text segmentation map through forward propagation, and then obtains the handwritten text detection result through post-processing.

[0162] Furthermore, the differentiated weighted loss mechanism only takes effect during the training phase, through the region weight coefficient map. The model is guided to focus on the answer area, and the area weight coefficient map is no longer needed in the inference stage, as the model has learned to pay different attention to different areas through training.

[0163] Specifically, generating a region weight coefficient map based on the training answer region mask includes: assigning a first weight coefficient to the internal pixels of the training answer region mask.

[0164] Assign a second weighting coefficient to the outer pixels of the training answer region mask.

[0165] The first weight coefficient is greater than the second weight coefficient, and the second weight coefficient is greater than zero.

[0166] The regional weight coefficient map It is a two-dimensional weight matrix with the same spatial dimensions as the input image. pixels, of which The height of the input image. The input image width. The region weight coefficient map. The weight coefficient for each position is determined based on whether the pixel is located inside the training answer area mask.

[0167] The internal pixels of the training answer region mask refer to the pixels corresponding to the spatial positions marked as the answer region (i.e., mask pixel values ​​of 1) in the training answer region mask. A first weighting coefficient is assigned to these pixels. .

[0168] The first weighting coefficient A value greater than 1 has the technical significance of amplifying the detection error within the answer area, making the model more sensitive to the handwriting detection accuracy of key answer areas during training, thereby shifting attention resources towards the answer areas where high-precision detection is required by business needs. In one specific implementation, the first weight coefficient... The value is 3.

[0169] The outer pixels of the training answer region mask refer to the pixels corresponding to the spatial locations marked as non-answer regions (i.e., mask pixel values ​​of 0) in the training answer region mask. A second weighting coefficient is assigned to these pixels. The second weighting coefficient The value of is equal to 1, which technically means that the detection error in non-answer areas is calculated according to a normal proportion, so that the model can still learn the handwriting detection ability in these positions during training. In one specific implementation, the second weight coefficient The value is 1.

[0170] The first weighting coefficient With the second weighting coefficient The following numerical constraints must be satisfied between them: the first weight coefficient Greater than the second weighting coefficient And the second weighting coefficient The value is greater than zero. This constraint ensures that the loss weight of pixels inside the answer region is greater than the loss weight of pixels outside the answer region, making the model more sensitive to detection errors in the answer region; at the same time, the loss weight of pixels outside the answer region is not zero, i.e., the second weight coefficient. A value greater than zero ensures that the model retains its ability to distinguish between handwritten and printed content in non-answer areas, preventing the complete neglect of non-answer areas and the misidentification of printed text as handwritten text.

[0171] In more complex application scenarios, fine-grained differentiated weighting schemes can be designed. For example, in scenarios involving handwritten annotations on contracts or forms, different weight coefficients can be assigned to different answer area types: the signature area is assigned a weight coefficient of 5, the date area is assigned a weight coefficient of 3, the ordinary annotation area is assigned a weight coefficient of 2, and the non-masked area is still assigned a weight coefficient of 1. In this case, the area weight coefficient diagram... The value of is no longer just two discrete values, but is assigned spatially according to the weight coefficients corresponding to each region type.

[0172] The regional weight coefficient map After generation, the pixel-wise errors between the text region probability map, the adaptive threshold map, the binarized text segmentation map, and the handwritten ground truth annotation are weighted and summed for subsequent weighted loss calculation steps. This differential weighted loss mechanism only takes effect during the model training phase; the region weight coefficient map is no longer needed during the inference phase. The model has learned to pay different levels of attention to different regions through differential weighted training.

[0173] Specifically, the total loss function is obtained by summing the probabilistic graph loss component, the threshold graph loss component, and the binary graph loss component.

[0174] The probability map loss component is obtained by multiplying the pixel-by-pixel basic error value between the text region probability map and the corresponding ground truth label by the weight coefficient of the corresponding spatial position in the region weight coefficient map and summing them up.

[0175] The threshold map loss component is obtained by multiplying the pixel-by-pixel basic error value between the adaptive threshold map and the corresponding ground truth label by the weight coefficient of the corresponding spatial position in the region weight coefficient map and summing them up.

[0176] The binary image loss component is obtained by multiplying the pixel-by-pixel basic error value between the binary text segmentation image and the corresponding ground truth label by the weight coefficient of the corresponding spatial position in the region weight coefficient image and summing them up.

[0177] The total loss function It is composed of three independent loss components combined through addition: probabilistic graphical loss component Threshold map loss components Binary plot loss components .

[0178] Wherein, the probability graph loss component Used to measure the probability map of the text region The difference between the predicted probability value and the corresponding ground truth label for each pixel; the threshold map loss component Used to measure the adaptive threshold map The difference between the predicted threshold and the corresponding ground truth threshold label for each pixel; the binary map loss component. Used to measure the binarized text segmentation map The difference between the predicted value and the corresponding ground truth label for each pixel is calculated. Three loss components supervise the model's training from three levels: text region probability prediction, adaptive threshold prediction, and final binarized segmentation result, respectively, so that the model can converge to the optimal state on all three outputs.

[0179] In one specific implementation, the weight coefficient of each loss component in the total loss function is set to 1, meaning that the three loss components participate in the calculation of the total loss with equal contributions. Equal to the probability map loss component The threshold map loss component and the binary graph loss component The sum of the three. The technical significance of this configuration is that it gives equal importance to the supervision signal strength of text region probability prediction, threshold prediction and binarization segmentation results, so that the model can be optimized in a balanced way across all output branches.

[0180] The probability graph loss component The calculation process consists of three steps.

[0181] The first step is to calculate the pixel-by-pixel basic error value: for the probability map of the text region... Each spatial location in Calculate the predicted probability value for this location. The truth label corresponding to the position in the handwritten truth label The binary cross-entropy baseline error value between the two. The truth label. For binary labels, when the position The value is 1 when it belongs to the handwritten text area and 0 when it belongs to the background. The calculation method for the basic error value of the binary classification cross-entropy is as follows: when When the value is 1, the basic error value is equal to ;when When it is 0, the basic error value is equal to .

[0182] The second step, differential weighting: weighting the positions... The base error value multiplied by the regional weighting coefficient map Corresponding spatial position Weighting coefficients The weighted error value is obtained. The weighting coefficients... The rule for determining the value is: if the position If the answer area is located inside the training area mask, then the weight coefficient is equal to the first weight coefficient. If position If the answer area is located outside the training area mask, then the weight coefficient is equal to the second weight coefficient. ;in Greater than and Greater than zero.

[0183] The third step, spatial summation: sum the weighted error values ​​of all spatial locations to obtain the probability map loss components. .

[0184] The effect of the differential weighting is as follows: for pixels within the answer area (weight coefficient is...) If the value is 3), the detection error is amplified, making the model more sensitive to the handwriting detection accuracy at these locations, thus shifting attention resources towards the key answer area; for pixels outside the answer area (weight coefficient is... ... If the value is 1, the detection error is calculated according to the normal ratio, so that the model still maintains the ability to detect handwritten content in the whole picture and prevents printed text from being misdetected as handwritten text.

[0185] The threshold map loss component The calculation process consists of three steps.

[0186] The first step is to calculate the pixel-by-pixel basic error value: for the adaptive threshold map... Each spatial location in Calculate the prediction threshold for this location. With the corresponding truth threshold label The Smooth L1 baseline error value between the prediction threshold and the target value. The Smooth L1 baseline error value is a regression loss function that is insensitive to outliers, and it is calculated as follows: when the prediction threshold is reached... With truth threshold label When the absolute difference between them is less than a preset threshold (usually set to 1), the basic error value is equal to... Otherwise, the basic error value equals .

[0187] The truth threshold label It is not directly obtained from manual annotation, but is automatically generated based on the boundaries of the handwritten text region: taking the handwritten text region as the core, it is dilated outwards using Gaussian decay, with the distance from the text region boundary being... The truth threshold label of a pixel within a pixel is set to ,in In one implementation method, the preset maximum attenuation distance is used. The value is 10 pixels.

[0188] The second step, differential weighting: weighting the positions... The Smooth L1 baseline error value multiplied by the region weighting coefficient map Corresponding spatial position Weighting coefficients The weighted error value is obtained.

[0189] The third step, spatial summation: sum the weighted error values ​​of all spatial locations to obtain the threshold map loss components. .

[0190] The threshold map loss component The differentiated weighting mechanism allows the model to have higher threshold prediction accuracy requirements within the answer region, while maintaining normal threshold prediction accuracy requirements outside the answer region, thereby guiding the adaptive threshold map. It generates more accurate binary boundary determination in key areas.

[0191] The binary image loss component The calculation process consists of three steps.

[0192] The first step is to calculate the pixel-by-pixel basic error value: for the binarized text segmentation image... Each spatial location in Calculate the predicted value for this location. The truth label corresponding to the position in the handwritten truth label The basic error value of the binary cross-entropy between the predicted values. It is calculated by differentiable binarization operation, and its value is in the interval [0,1].

[0193] The calculation method for the basic error value of the binary classification cross-entropy is as follows: when When the value is 1, the basic error value is equal to ;when When it is 0, the basic error value is equal to .

[0194] The second step, differential weighting: weighting the positions... The base error value multiplied by the regional weighting coefficient map Corresponding spatial position Weighting coefficients The weighted error value is obtained.

[0195] The third step is spatial summation: summing the weighted error values ​​of all spatial locations to obtain the binary map loss components. .

[0196] The binary image loss component Directly supervising the final binarized segmentation result output by the model is a key loss term to ensure that the model can accurately distinguish between handwritten text pixels and background pixels. Through a differential weighting mechanism, the model has higher requirements for the binarization segmentation accuracy within the answer region, while maintaining normal requirements for the binarization segmentation accuracy of non-answer regions. This allows the model to focus on improving the detection performance of key regions while maintaining the overall image detection capability.

[0197] All three loss components employ the same differentiated weighting mechanism, i.e., they all use the aforementioned regional weight coefficient map. The pixel-by-pixel base error values ​​are weighted. This unified design enables the model to focus attention on key regions at three levels: text region probability prediction, adaptive threshold prediction, and binarization segmentation, while maintaining full-image detection capabilities. The differential weighted loss mechanism only takes effect during the model training phase and is not used during the inference phase.

[0198] This technical solution also provides a handwritten text detection system based on early fusion of multi-source images, including:

[0199] The multi-source input acquisition module is configured to acquire the user image to be detected, the blank template image corresponding to the user image, and the pre-annotated answer area mask.

[0200] The feature embedding module includes a first feature embedding unit, a second feature embedding unit, and a third feature embedding unit. The first feature embedding unit is configured to perform feature embedding processing on the user image to obtain a first feature map. The second feature embedding unit is configured to perform feature embedding processing on the blank template image to obtain a second feature map. The third feature embedding unit is configured to perform feature embedding processing on the answer area mask to obtain a third feature map.

[0201] The early stitching module is configured to stitch the first feature map, the second feature map, and the third feature map along the channel dimension to obtain a stitched feature map.

[0202] The dimension reduction and fusion module is configured to perform convolutional dimension reduction on the spliced ​​feature map using a 1×1 convolution kernel to obtain a fused feature map.

[0203] The feature extraction module is configured to use a backbone feature extraction network to perform multi-scale feature extraction on the fused feature map to obtain a multi-scale feature map.

[0204] The detection output module is configured to use a text detection network to detect and decode the multi-scale feature map to obtain the handwritten text detection result.

[0205] Specifically, the multi-source input acquisition module is further configured as follows:

[0206] Based on the content features of the user image, the corresponding blank template image and the associated answer area mask are retrieved from the pre-built template database.

[0207] The user image and the blank template image are coarsely aligned in space, and the answer area mask is mapped to the spatial position corresponding to the user image.

[0208] Specifically, a handwritten text detection system based on early fusion of multi-source images further includes a model training module, which includes:

[0209] The training data acquisition unit is configured to acquire user images for training, blank template images for training, answer area masks for training, and handwritten real value annotations.

[0210] The training forward propagation unit is configured to perform feature embedding processing on the training user image, the training blank template image, and the training answer region mask, respectively. After concatenation along the channel dimension, convolutional dimensionality reduction is performed to obtain a training fused feature map. The backbone feature extraction network is used to perform multi-scale feature extraction on the training fused feature map, and the text detection network is used for detection and decoding to obtain the prediction result.

[0211] The region weight generation unit is configured to generate a region weight coefficient map based on the training answer region mask, wherein a first weight coefficient is assigned to the inner pixels of the training answer region mask, and a second weight coefficient is assigned to the outer pixels of the training answer region mask, wherein the first weight coefficient is greater than the second weight coefficient and the second weight coefficient is greater than zero.

[0212] The weighted loss calculation unit is configured to perform weighted loss calculation on the difference between the prediction result and the handwritten true value annotation based on the regional weight coefficient map, and obtain the total loss function.

[0213] The parameter update unit is configured to update network parameters based on the total loss function.

[0214] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A handwritten text detection method based on early fusion of multi-source images, characterized in that, include: Obtain the user image to be detected, the blank template image corresponding to the user image, and the pre-annotated answer area mask; The user image is subjected to feature embedding processing to obtain a first feature map; The blank template image is subjected to feature embedding processing to obtain a second feature map; the answer area mask is subjected to feature embedding processing to obtain a third feature map; The first feature map, the second feature map, and the third feature map are concatenated along the channel dimension to obtain a concatenated feature map. The spliced ​​feature map is reduced in dimensionality by using a 1×1 convolution kernel to obtain a fused feature map. A backbone feature extraction network is used to perform multi-scale feature extraction on the fused feature map to obtain a multi-scale feature map. A text detection network is used to detect and decode the multi-scale feature map to obtain the handwritten text detection result.

2. The handwritten text detection method based on early fusion of multi-source images according to claim 1, characterized in that, The process of acquiring the user image to be detected, the blank template image corresponding to the user image, and the pre-annotated answer area mask includes: Acquire the user image to be detected; Based on the content features of the user image, the corresponding blank template image and the associated answer area mask are retrieved from the pre-built template database; The user image and the blank template image are coarsely aligned in space, and the answer area mask is mapped to the spatial position corresponding to the user image.

3. The handwritten text detection method based on early fusion of multi-source images according to claim 1, characterized in that, The process involves using a backbone feature extraction network to perform multi-scale feature extraction on the fused feature map, resulting in a multi-scale feature map, including: Feature extraction is performed on the fused feature map based on a hierarchical self-attention mechanism to obtain initial multi-scale features; The initial multi-scale features are fused across scales and channels are unified using a feature pyramid network to obtain the multi-scale feature map.

4. The handwritten text detection method based on early fusion of multi-source images according to claim 1, characterized in that, The step of using a text detection network to detect and decode the multi-scale feature map to obtain handwritten text detection results includes: The multi-scale feature map is subjected to differentiable binarization to output a text region probability map and an adaptive threshold map. Calculate a binarized text segmentation map based on the text region probability map and the adaptive threshold map; The bounding boxes of the handwritten text are extracted based on the binarized text segmentation map and used as the handwritten text detection result.

5. The handwritten text detection method based on early fusion of multi-source images according to claim 1, characterized in that, The backbone feature extraction network and the text detection network are executed based on a handwritten text detection model, which is obtained through a model training phase, including: Acquire user images for training, blank template images for training, answer area masks for training, and handwritten real-value annotations; The training user image, the training blank template image, and the training answer area mask are respectively processed by feature embedding. After concatenation in the channel dimension, dimensionality reduction is performed to obtain the training fusion feature map. The backbone feature extraction network is used to perform multi-scale feature extraction on the training fusion feature map, and the text detection network is used for detection and decoding to obtain prediction results including text region probability map, adaptive threshold map and binarized text segmentation map; A region weight coefficient map is generated based on the training answer region mask; Based on the region weighting coefficient map, a weighted loss calculation is performed on the difference between the prediction result and the handwritten true value annotation to obtain the total loss function; Update network parameters based on the total loss function.

6. The handwritten text detection method based on early fusion of multi-source images according to claim 5, characterized in that, The generation of the region weight coefficient map based on the training answer region mask includes: Assign a first weight coefficient to the internal pixels of the training answer region mask; Assign a second weighting coefficient to the outer pixels of the training answer region mask; The first weight coefficient is greater than the second weight coefficient, and the second weight coefficient is greater than zero.

7. The handwritten text detection method based on early fusion of multi-source images according to claim 5, characterized in that, The total loss function is obtained by summing the probability graph loss component, the threshold graph loss component, and the binary graph loss component. The probability map loss component is obtained by multiplying the pixel-by-pixel basic error value between the text region probability map and the corresponding truth label by the weight coefficient of the corresponding spatial position in the region weight coefficient map and summing them up. The threshold map loss component is obtained by multiplying the pixel-by-pixel basic error value between the adaptive threshold map and the corresponding ground truth label by the weight coefficient of the corresponding spatial position in the region weight coefficient map and summing them up. The binary image loss component is obtained by multiplying the pixel-by-pixel basic error value between the binary text segmentation image and the corresponding ground truth label by the weight coefficient of the corresponding spatial position in the region weight coefficient image and summing them up.

8. A handwritten text detection system based on early fusion of multi-source images, characterized in that, include: The multi-source input acquisition module is configured to acquire the user image to be detected, the blank template image corresponding to the user image, and the pre-annotated answer area mask; The feature embedding module includes a first feature embedding unit, a second feature embedding unit, and a third feature embedding unit; the first feature embedding unit is configured to perform feature embedding processing on the user image to obtain a first feature map; the second feature embedding unit is configured to perform feature embedding processing on the blank template image to obtain a second feature map; and the third feature embedding unit is configured to perform feature embedding processing on the answer area mask to obtain a third feature map. The early stitching module is configured to stitch the first feature map, the second feature map, and the third feature map along the channel dimension to obtain a stitched feature map; The dimension reduction and fusion module is configured to perform convolutional dimension reduction on the spliced ​​feature map using a 1×1 convolutional kernel to obtain a fused feature map. The feature extraction module is configured to use a backbone feature extraction network to perform multi-scale feature extraction on the fused feature map to obtain a multi-scale feature map; The detection output module is configured to use a text detection network to detect and decode the multi-scale feature map to obtain the handwritten text detection result.

9. A handwritten text detection system based on early fusion of multi-source images according to claim 8, characterized in that, The multi-source input acquisition module is further configured to: Based on the content features of the user image, the corresponding blank template image and the associated answer area mask are retrieved from the pre-built template database; The user image and the blank template image are coarsely aligned in space, and the answer area mask is mapped to the spatial position corresponding to the user image.

10. A handwritten text detection system based on early fusion of multi-source images according to claim 8, characterized in that, It also includes a model training module, which includes: The training data acquisition unit is configured to acquire user images for training, blank template images for training, answer area masks for training, and handwritten real value annotations. The training forward propagation unit is configured to perform feature embedding processing on the training user image, the training blank template image, and the training answer region mask respectively, and perform convolutional dimensionality reduction after concatenation in the channel dimension to obtain a training fused feature map; the backbone feature extraction network is used to perform multi-scale feature extraction on the training fused feature map, and the text detection network is used for detection and decoding to obtain the prediction result; The region weight generation unit is configured to generate a region weight coefficient map based on the training answer region mask, wherein a first weight coefficient is assigned to the inner pixels of the training answer region mask, and a second weight coefficient is assigned to the outer pixels of the training answer region mask, wherein the first weight coefficient is greater than the second weight coefficient and the second weight coefficient is greater than zero. The weighted loss calculation unit is configured to perform weighted loss calculation on the difference between the prediction result and the handwritten true value annotation based on the regional weight coefficient map, and obtain the total loss function. The parameter update unit is configured to update network parameters based on the total loss function.

Images