Method for pointing object text positioning based on iterative semantic visual association

Abstract

The application discloses a pointing object text positioning method based on iterative semantic visual association, which comprises the following steps: obtaining a document image containing a pointing object and a natural language instruction; extracting visual and language features, predicting a pointing object boundary box and generating a pointing mask to extract pointing context features; splicing the pointing context features and the instruction semantics to generate channel scaling coefficients, and modulating the visual features in space and channel; using the instruction semantics to generate FiLM parameters to modulate the enhanced visual features, and fusing the features through cross-modal attention to predict an initial target boundary box; starting an iterative refinement process, generating a target mask according to the current predicted box and calculating a semantic-visual consistency score, combining geometric difference and semantic information to modulate and fuse features to predict a better boundary box, and outputting the result with the highest score after iteration until the termination condition is met. The application enhances the fine-grained spatial semantic understanding and high-precision iterative positioning capability.

Description

Technical Field

[0001] This invention belongs to the field of document image understanding and interactive text localization technology, specifically involving a locator text localization method based on iterative semantic visual association. Background Technology

[0002] With the development of computer vision technology, text detection and recognition in images has become an important research direction. Among them, visual grounding aims to locate target regions in images based on natural language descriptions and has been widely used in fields such as human-computer interaction. Existing methods are mainly divided into two-stage and one-stage methods: two-stage methods first generate candidate regions and then perform matching, which has high computational cost and is sensitive to missed detections; one-stage methods directly predict bounding boxes based on fused features, which is more efficient but has limited ability to model complex spatial relationships.

[0003] However, the methods described above are primarily designed for sparse objects with significant appearance differences in natural scenes (such as vehicles and pedestrians). When applied to the specific task of Pointer-based Document Text Grounding (PDTG), they exhibit significant limitations: 1. Dense and similar text: The words, lines and paragraphs in the document image are densely arranged and have highly similar visual features. It is difficult to distinguish adjacent text units, which can easily lead to the positioning box covering the wrong area.

[0004] 2. Weak spatial semantic understanding: PDTG instructions often contain complex spatial relationships with reference to the pointing position, such as "previous word" and "next paragraph". Existing methods lack effective modeling of hierarchical document structure and relative positional relationships.

[0005] 3. Extremely high positioning accuracy requirements: PDTG requires near-character-level bounding box accuracy; even slight inclusion of adjacent characters or lines is considered an error. Existing methods often use IoU thresholds (such as 0.5) that are insufficient to meet such high accuracy requirements and lack a stepwise optimization mechanism.

[0006] Furthermore, while traditional optical character recognition (OCR) technology can detect all text, it cannot combine the location of the object with the language command for targeted localization, and it is not suitable for PDTG tasks.

[0007] In summary, there is an urgent need for a new method that can integrate directional information and language instructions to achieve fine-grained spatial semantic understanding and high-precision iterative localization in dense document images, in order to solve the bottlenecks faced by existing technologies in directional interactive text localization tasks. Summary of the Invention

[0008] This invention provides a directional text localization method based on iterative semantic visual association, aiming to solve the problems of existing visual text localization methods, such as difficulty in deeply integrating directional information and language instructions in dense document images, lack of fine-grained spatial semantic understanding capabilities, and insufficient localization accuracy.

[0009] This invention provides a method for locating directional text based on iterative semantic visual association, comprising the following steps: S1: Obtain the document image containing the pointer and the natural language instructions used to describe the target text region; S2: Extract visual features of the document image through a visual encoder, extract linguistic features of the natural language instructions through a language encoder, and extract instruction semantic vectors from the linguistic features; S3: Based on the visual features, predict the bounding box of the pointing object using a regression prediction head; based on the bounding box of the pointing object, generate a pointing mask, and use the pointing mask to perform weighted pooling on the visual features to extract pointing context features; S4: After concatenating the pointing context feature with the instruction semantic vector, input it into a two-layer perceptron network and process it through the Sigmoid activation function to generate channel scaling coefficients; use the pointing mask to spatially modulate the visual features, and use the channel scaling coefficients to channel modulate the spatially modulated visual features to obtain enhanced visual features; S5: Based on the instruction semantic vector, channel scaling parameters and channel bias parameters are generated as FiLM modulation parameters through two independent learnable mapping functions, and the enhanced visual features are channel-modulated using the FiLM modulation parameters; the modulated visual features are used as keys and values, and the language features are used as queries to perform cross-modal attention calculation to obtain fused features; the initial target bounding box is obtained by regression prediction based on the fused features; S6: Starting from the initial target bounding box, perform an iterative refinement process, which includes: generating a target mask based on the target bounding box of the current iteration, and calculating a semantic-visual consistency score based on the target mask and the fusion features of the current iteration; when the score of the current iteration is higher than the score of the previous iteration by more than a preset threshold, generating channel modulation parameters for channel modulation based on the geometric feature difference between the current target bounding box and the bounding box of the pointing object, the semantic information of the spatial relationship category and the target object type obtained from the instruction semantic vector, and the instruction semantic vector, and using the channel modulation parameters and the pointing mask to enhance the current fusion features to obtain optimized features, and then predicting the target bounding box of the next iteration based on the optimized features through regression; repeating the iteration until the termination condition is met, and outputting the target bounding box with the highest semantic-visual consistency score as the final localization result.

[0010] Optionally, in step S3, generating a pointing mask based on the bounding box of the pointing object includes: calculating the diagonal length of the bounding box of the pointing object, using a preset ratio of the diagonal length as the expansion radius, expanding the bounding box, and generating an expanded pointing mask.

[0011] Optionally, the preset ratio is 0.3.

[0012] Optionally, in step S4, the modulation of the visual features using the pointing mask and the channel scaling factor is achieved by the following formula:

[0013] in, As a visual feature, For pointing to the mask, This is the channel scaling factor. Indicates a broadcast operation. and For learnable scaling parameters, This represents element-wise multiplication. To enhance visual features.

[0014] Optionally, in step S2, extracting the instruction semantic vector from the language features includes: extracting the vector corresponding to the CLS marker output by the language encoder, and projecting it through a linear projection layer to obtain the instruction semantic vector.

[0015] Optionally, the spatial relationship category and the target object type are obtained based on the instruction semantic vector by decoding the instruction semantic vector in the following way: inputting the instruction semantic vector into a spatial relationship classifier and an object type classifier to obtain the probability distribution of the spatial relationship category and the probability distribution of the target object type; the spatial relationship category includes current, previous, and next; the target object type includes word, line, and paragraph.

[0016] Optionally, in step S6, calculating the semantic-visual consistency score based on the target mask and the fusion features of the current iteration includes: using the target mask to perform mask-weighted pooling on the fusion features of the current iteration to obtain local features, performing global pooling on the fusion features to obtain global features, concatenating the local features, the global features, and the instruction semantic vector, inputting them into a multilayer perceptron composed of multiple fully connected layers and processing them through a Sigmoid activation function to obtain the semantic-visual consistency score.

[0017] Optionally, in step S6, generating channel modulation parameters for channel modulation includes: calculating the difference between the center coordinates of the current target bounding box and the center coordinates of the pointing object bounding box, as well as the logarithmic ratio of their width and height, to form a geometric descriptor; concatenating the geometric descriptor, the probability distribution of the spatial relationship category, the probability distribution of the target object type, and the instruction semantic vector, and obtaining FiLM parameters for channel scaling and channel bias through nonlinear mapping as the channel modulation parameters; wherein, the parameters for channel scaling are processed by the hyperbolic tangent function.

[0018] Optionally, the termination condition for the iterative refinement process is: the number of iterations reaches a preset maximum value, or the improvement of the semantic-visual consistency score in two consecutive iterations does not exceed the preset threshold.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the locator text localization method based on iterative semantic visual association as described above.

[0020] The beneficial effects of this invention lie in the fact that, through the synergistic effect of various technical features, it systematically solves the three core problems of locating interactive text pointing to objects in dense document images, specifically in the following ways: First, this invention utilizes the coordinates of the pointing object as prior information to generate a pointing mask and extract pointing context features accordingly. Then, it performs spatial and semantic joint modulation on the visual features, effectively integrating the pointing position information with the semantics of the language instructions. This enhances the model's attention to the pointing-related region, thereby significantly improving the ability to lock the initial target region in dense and similar-looking text units and solving the problem of insufficient utilization of pointing information.

[0021] Second, this invention employs a combination of FiLM modulation mechanism based on instruction semantics and cross-modal attention fusion. It first enhances relevant visual features at the channel level based on semantics, and then establishes fine-grained semantic-visual associations at the spatial level. This achieves deep semantic alignment and spatial reasoning between visual features and language instructions, enabling the model to accurately understand complex spatial relationship instructions such as current, previous, and next. This significantly improves the fine-grained spatial semantic understanding of document hierarchical structures and solves the problem of difficult spatial semantic understanding.

[0022] Third, this invention proposes an iterative enhancement localization mechanism based on a scorer neural network, constructing a closed-loop process of prediction, semantic-visual consistency scoring, feature optimization, and re-prediction. This mechanism can adaptively decide whether to continue iterating based on the scoring results, and dynamically optimize features using geometric differences and semantic information, thereby gradually correcting bounding box deviations and achieving iterative improvement in localization accuracy. Ultimately, it meets the stringent requirements of document text localization tasks for character-level accuracy, and solves the problems of unstable localization and insufficient accuracy in traditional one-time prediction methods.

[0023] In summary, this invention achieves the goal of deeply fusing directional information and natural language instructions in dense document images, thereby enabling fine-grained spatial semantic understanding and high-precision iterative localization. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram illustrating the working principle of the adaptive iterative refinement module provided by the present invention.

[0026] Figure 2 The overall framework diagram of AIVG for object-oriented interactive text localization provided by the present invention.

[0027] Figure 3 A structural diagram of the pointing guide module provided by the present invention.

[0028] Figure 4 The diagram shows the structure of the cross-modal feature fusion module provided by this invention.

[0029] Figure 5 A schematic diagram of geometry-semantic consistency modulation provided by the present invention.

[0030] Figure 6 This is an overview diagram of the test results on real photographs of documents provided by the present invention. Detailed Implementation

[0031] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1 This embodiment provides a locator text localization method based on iterative semantic visual association. This method aims to solve the problem of accurately understanding and integrating user-provided locator position information and natural language commands in densely typed document images, thereby achieving high-precision and robust localization of text targets at different granularities such as words, lines, and paragraphs. Figure 2 A schematic diagram illustrating the overall framework of the method provided in this embodiment is shown. The method in this embodiment specifically includes the following steps: S1: Obtain the document image I provided by the user, which contains a clear pointing object, such as a finger or pen tip. Simultaneously, obtain the natural language instruction T input by the user, which describes the target text region to be located. The document image I and the natural language instruction T together constitute a pointing object interactive text localization request.

[0033] Natural language commands typically express a user's location intent. For systematic modeling, commands can be categorized into various typical scenarios resulting from combinations of spatial relationships and text unit types. Specifically, spatial relationships include three categories: current, previous, and next; text unit types include words, lines, and paragraphs. The combinations of these two types form nine different location scenarios, as shown in the table below: Table 1 Examples of Natural Language Instruction Classification

[0034] This classification scheme covers most document text location needs and can be extended to more complex interactive scenarios through combination.

[0035] S2: Extract visual features from document image I using a visual encoder. This embodiment uses a ResNet-50 network as the visual backbone. The input image is processed by the network to obtain a high-dimensional visual feature map, which can be represented as follows:

[0036] in, , and This indicates the spatial resolution of the downsampled feature map; This represents the visual encoding function. To facilitate subsequent processing and reduce computational cost, the visual features can be further unified into a single function using adaptive average pooling and 1x1 convolution operations. Embedded representation.

[0037] Simultaneously, linguistic features of the natural language instruction T are extracted using a language encoder. This embodiment employs the BERT model. Inputting the text instruction T into BERT yields a word-level semantic embedding sequence, the process of which can be represented as follows:

[0038] in, The language encoding function is used to extract the global semantic vector corresponding to the [CLS] tag in its output. As a holistic representation of the instruction. To align linguistic and visual features in the semantic space, a linear projection layer is used to... Projection into instruction semantic vector :

[0039] in, and These are learnable parameters.

[0040] Furthermore, this step also includes decoding the implicit spatial relationships and target type information from the instruction semantic vector. Specifically, the instruction semantic vector... The inputs are given to a spatial relation classifier and an object type classifier, respectively. The spatial relation classifier outputs the probability distribution of spatial relation categories. The categories include current, previous, and next. The object type classifier outputs the probability distribution of the target object type. The types include words, lines, and paragraphs. The probability calculation formula is as follows:

[0041] in, and The weight matrix is ​​for a learnable classifier. and They represent the first The spatial relationship category and the first Normalized probabilities of each object type category.

[0042] This design enables the model to explicitly understand and utilize the structured semantic constraints in the instructions, providing crucial guidance for subsequent spatial reasoning and feature modulation. It addresses the problem of the model's limited ability to understand complex spatial semantic instructions and enhances the explicitness of semantic parsing through explicit classification, thereby helping the model to more accurately distinguish adjacent words, lines, or paragraphs in hierarchical document structures.

[0043] S3: This step corresponds to Figure 3 The pointing guide module is shown.

[0044] First, based on the visual feature map obtained in step S2 Predicts the bounding box of the pointing object using a lightweight convolutional regression head. ,in The normalized center coordinates of the pointing region , These represent the normalized width and height of the bounding box, respectively, with values ​​normalized to the [0,1] interval. This lightweight convolutional regression head employs an anchor-free design, consisting of two 3×3 convolutional layers (the number of channels is reduced from 256 to 128 and then to 64), one 1×1 convolutional layer (outputting 4-dimensional bounding box parameters), a global average pooling layer, and a sigmoid activation function, connected sequentially.

[0045] Secondly, a pointing mask is generated based on the predicted pointing object bounding box. To cover the surrounding area where user attention may extend, the pointer bounding box is expanded. Specifically, the diagonal length of the pointer bounding box is calculated, and the bounding box is expanded with a preset proportion of this length as the expansion radius. This preset proportion can be adjusted according to the density of layout elements in the document image. As a preferred embodiment of the present invention, experimental verification has shown that setting this proportion to 0.3 achieves a good balance between covering the effective attention area and avoiding the introduction of excessive noise.

[0046]

[0047]

[0048] Expanded rectangular area It can be represented as: .based on Generate initial pointer mask To smooth the boundary, for Perform convolution and sigmoid activation to obtain a smooth pointing mask. :

[0049] in, For the Sigmoid function; Indicates that the convolution kernel is The convolutional processing is performed. The parameters (weights) of this convolutional layer are learnable and are initialized using the Kaiming Uniform method during the initial stage of model training.

[0050] Next, using a pointer mask Visual feature maps Perform weighted pooling to extract contextual features. The calculation formula is:

[0051] in, , representing the area pointing to the mask; vector Visual features that indicate the fusion of information.

[0052] S4: Pointing to context features With instruction semantic vector splicing, through a fully connected network Then, after processing with the Sigmoid activation function, channel scaling factors are generated. :

[0053] in, This represents vector concatenation. For the sigmoid function. Fully connected networks. Specifically, it is a two-layer perceptron, whose structure is as follows: a first fully connected layer (FC), a ReLU activation function, and a second fully connected layer (FC). In a preferred embodiment, the number of visual feature channels C=256, and the dimension of the instruction semantic vector is 256. Therefore, the input dimension of the first fully connected layer is 512 (i.e., 256+256), and the output dimension is 256; the input and output dimensions of the second fully connected layer are both 256.

[0054] This operation injects instruction semantics into the pointing representation, enabling channel gating to consider both position and semantic information simultaneously, thus overcoming the positioning drift problem that may occur if only the pointing position is relied upon.

[0055] Using pointer masks and channel scaling factor Original visual features By performing joint modulation in both spatial and channel dimensions, enhanced visual features can be obtained. The modulation formula is:

[0056] in, This represents element-wise multiplication. This indicates a broadcast operation to match feature dimensions. and Learnable scaling parameters are in scalar form. At the start of model training, these learning scaling parameters are used to ensure stability during training and provide a suitable starting point. and The initial values ​​for both parameters are set to 0.5. To prevent over-modulation of features during training, L2 regularization constraints are applied to these two parameters in the model's total loss function. This technique addresses the problem of effectively utilizing the strong spatial prior of the pointing object. Through dual spatial and channel modulation, visual features are guided to regions highly correlated with the pointing position and instruction semantics, significantly improving the ability to lock onto the initial target in dense and visually similar text units, laying a solid foundation for subsequent accurate understanding of spatial relationship instructions.

[0057] All of the above learnable parameters, including scaling parameters , All of them are trained end-to-end together with the other parts of the model, without a separate training phase.

[0058] S5: This step corresponds to Figure 4 The cross-modal feature fusion module is shown. First, it is based on the instruction semantic vector. Channel scaling parameters are generated through two independent learnable mapping functions. and channel offset parameters This is used as a parameter for Feature Linear Modulation (FiLM). The enhanced visual features obtained in step S4 are then processed using this set of parameters. Perform channel modulation:

[0059] in, For channel multiplication, and It is a learnable mapping function generated from language features. Adjust the response intensity of each visual channel, enhance the channel that is relevant to the instruction semantics, and suppress irrelevant information; Injecting semantic biases into visual features guides the model to incorporate semantic priors into the feature layer.

[0060] Therefore, FiLM modulation selectively enhances visual channels that are semantically relevant to the instructions at the global channel level, while suppressing irrelevant information.

[0061] However, global channel modulation alone is insufficient to establish fine-grained semantic-spatial correspondences. Therefore, a cross-modal attention mechanism is further introduced. This mechanism integrates the modulated visual features... As the key K and value V, the word-level features output by the language encoder are... As the query Q, compute cross-modal attention:

[0062] in, , , For learnable parameter matrix, The feature dimension is defined by the attention matrix A, which establishes the association weights between each linguistic lexical unit and each position in the visual feature map. The attention-fused features are then normalized and linearly projected to obtain a deeply fused cross-modal feature map. .

[0063] Finally, based on the fused feature map Predict the initial target bounding box using a lightweight regression head: ,in and For normalized center coordinates, and This is used to predict the normalized width and height of the target bounding box. This bounding box provides an approximate range of the target region, serving as a starting point for subsequent iterations of refinement.

[0064] This technology solves the semantic drift problem caused by the simple fusion of visual and linguistic features. By first performing FiLM global channel modulation and then performing cross-modal attention local alignment, a deep semantic-spatial association is achieved, enabling the model to accurately understand complex relative spatial relationship instructions such as previous, next, and current paragraph, providing key support for hierarchical spatial reasoning in dense layouts.

[0065] S6: This step corresponds to Figure 1 The adaptive iterative refinement principle is illustrated. It uses the initial target bounding box... Starting with this, a closed-loop iterative process of prediction-scoring-optimization is initiated, aiming to gradually correct the positioning deviation and achieve near-character-level bounding box accuracy required for document text positioning.

[0066] For the t-th iteration (t starts from 0), the current fused feature map is: (initial The current prediction box is Perform the following sub-steps: 1. Generate target mask and calculate semantic-visual consistency score: Based on the current prediction frame Generate a soft target mask The value inside the box is close to 1, while the value outside the box decreases with distance. Using... right Perform masked weighted pooling to extract local target features:

[0067] in, At the same time, for Global pooling is used to obtain global features: .Will , With instruction semantic vector splice together and construct a joint representation. .

[0068] Will Input a nonlinear mapping network The network consists of three fully connected layers: the first fully connected layer has an input dimension of 768 and an output dimension of 256, followed by a ReLU activation function; the second fully connected layer has an input dimension of 256 and an output dimension of 256, followed by a ReLU activation function; and the third fully connected layer has an input dimension of 256 and an output dimension of 1. Finally, the output of the third layer is processed by a Sigmoid activation function to obtain the semantic-visual consistency score.

[0069] The rating It quantifies the comprehensive semantic and visual matching degree between the current prediction box and the input command, providing a reliable optimization guide for the iterative process.

[0070] 2. Determine whether to perform feature enhancement and the next round of prediction: Compare current ratings Rating compared to the previous iteration If the score increases by more than the preset threshold... If the iteration succeeds, the feature enhancement and prediction steps begin; otherwise, the iteration tends to terminate.

[0071] When further iterations are needed, feature enhancement should be performed first. For example... Figure 5 As shown, calculate the current target box. With the pointing frame Geometric feature differences :

[0072] To improve training stability, Scale normalization and boundedness are performed to bring the numerical range within a reasonable range. Then, the processed geometric descriptor... The spatial relationship probability distribution obtained in step S2 Object type probability distribution and instruction semantic vector By splicing together, a joint description is formed. .here, The dimension is 4. and Each dimension has 3. The dimension is 256, therefore The dimension is 4+3+3+256=266.

[0073] Will Input a nonlinear mapping network Generate FiLM parameters for channel modulation. and The network The specific structure is as follows: First fully connected layer, input dimension 266, output dimension 256; Second fully connected layer, input dimension 256, output dimension 256; Third fully connected layer, input dimension 256, output dimension... , among which the former each element as ,back each element as :

[0074] These parameters are used to modulate the current fused features:

[0075] in, This represents element-wise multiplication; This indicates a broadcast operation that extends the channel parameters to the spatial dimension of the feature map (e.g., by...). dimensional vector reshaping to And broadcast to (This allows all elements within the same channel to share the same value). This represents the hyperbolic tangent function.

[0076] At the same time, the pointer mask is used again. Computational space gating And combine instruction semantics to compute semantic gating The modulated features are then subjected to residual amplification to obtain the optimized feature map. Among them, spatial gating generation network With semantically gated generative networks The specific structure is as follows:

[0077] Pointer mask for single-channel network reception First, go through a The convolutional layer increases the number of channels to 16, and after ReLU activation, it goes through another... The convolutional layer reduces the number of channels back to 1, and finally outputs spatial gating through the sigmoid activation function. .

[0078]

[0079] The network first calculates In pointing to the mask The pooling features of the region are then compared with the instruction semantic vector. The input vector is obtained by concatenation. This vector is then passed through a fully connected layer (input dimension 512, output dimension 256) and ReLU activation, followed by another fully connected layer (input dimension 256, output dimension 256) and Sigmoid activation, outputting a semantically gated vector. .

[0080] This process enables targeted feature optimization based on the geometric relationship between the target and the object of reference, and the explicit spatial semantic constraints of the instructions.

[0081] Subsequently, based on the optimized feature map Predict the target bounding box for the next iteration using the regression head. .

[0082] 3. Iteration Termination and Result Output: Repeat the above steps. The iteration termination condition is set as follows: the number of iterations reaches a preset maximum value, or the semantic-visual consistency score does not improve by more than a threshold in two consecutive iterations. .

[0083] Finally, the target bounding box with the highest semantic-visual consistency score throughout the entire iteration process is selected as the final localization result output.

[0084] The adaptive iterative refinement mechanism is one of the core innovations of this invention. It solves the inherent problem of insufficient localization accuracy in dense document scenarios caused by traditional one-time prediction methods. By introducing semantic-visual consistency scoring as an internal optimization objective and constructing a regression-scoring-enhancement closed loop, predictions can be progressively focused and corrected. This mechanism is particularly suitable for scenarios with extremely high accuracy requirements in document text localization, where even slight inclusion of adjacent characters or lines is considered unacceptable. Experiments show that this mechanism can significantly improve recall at high IoU thresholds, meeting the stringent requirements of the PDTG task for high-precision localization.

[0085] To enable the model to learn the above capabilities, end-to-end training is required. The overall loss L of this invention comprises four parts: target text localization loss. Pointing and positioning loss Joint classification loss of spatial relationships and object types And iterative scoring and schedule loss specifically designed for iterative processes. Each loss is weighted. To achieve a balance, the overall loss can be expressed as:

[0086] In a preferred embodiment, the weights are set to , , , It should be noted that the specific weight values ​​mentioned above are recommended values ​​given after ablation experiments and parameter adjustments on the validation set. Their main purpose is to balance the orders of magnitude of various losses and ensure the stability of model training convergence. In practical applications, those skilled in the art can make appropriate adjustments around the above recommended values ​​according to the characteristics of a specific dataset or the evaluation metrics of interest.

[0087] Iteration loss in this embodiment The design of the bounding box is particularly crucial, as it supervises the model to learn the correct localization quality concept. For a batch containing N samples, let the maximum number of iterations be T. For the predicted bounding box of the i-th sample in the t-th iteration... Calculate its relationship with the ground truth bounding box. The Intersection over Union (IoU) ratio is used as the ideal score for this prediction. ,Right now .

[0088] Rating loss Defined as the semantic-visual consistency score predicted by the model. Compared to the ideal score The mean square error between the two values ​​indicates that the supervised prediction score approximates the true positioning quality.

[0089] Schedule loss The aim is to suppress the degradation of localization quality during iteration (i.e., the prediction results of subsequent iterations are worse than those of previous iterations). Specifically, for two adjacent iterations t and t+1, if the ideal score decreases ( If the condition is true, then a penalty is imposed. To achieve differentiable optimization of this conditional judgment, the loss term is constructed using the modified linear unit (ReLU) function:

[0090] in, This function only applies when quality degrades ( This generates non-zero gradients, driving model optimization to avoid such degradation.

[0091] The final iterative loss is the weighted sum of the two losses mentioned above:

[0092] In a preferred embodiment, It is set to 0.2. This loss design encourages the iterative process to continuously improve prediction quality.

[0093] To illustrate the method of this invention, consider a specific scenario: A user points to a word in a document and issues the command "Locate the next word." First, image and text features are extracted using a visual encoder and a language encoder, respectively. The pointing guidance module predicts the finger region and modulates the visual features based on the command semantics, initially highlighting the relevant region. The cross-modal fusion module deeply fuses the spatial relational semantics of "next" with the visual features, initially selecting candidate next word regions. Subsequently, the iterative refinement module is initiated: the first iteration evaluates the matching score of the initial bounding box. If the height is insufficient, the geometric offset between the current bounding box and the finger bounding box is calculated. Combined with the decoded semantic information of "next" and "word", the features are enhanced to predict a more accurate bounding box. And calculate the new score .like Significantly higher than If the prediction box is not found, the iteration continues. This process is repeated until the prediction box closely matches the target word and the score no longer increases significantly, at which point the final result is output.

[0094] To systematically evaluate the method of this invention, an automatically generated PDTG dataset was constructed. The method for constructing the dataset is as follows: The process involves parsing word, line, and paragraph-level bounding boxes from a PDF document and rendering the page to generate a high-resolution image consistent with a real-world reading scenario. The workflow is as follows: (1) Parse each page of PDF, extract word, line and paragraph bounding boxes, and record their relationships and hierarchical structure.

[0095] (2) Randomly select a word on each page as the initial pointing target, overlay a pointing mask below the word, and record the coordinates of the fingertip (pointing tip). The pointing object data is synthesized in the following way: take photos of real pointing objects (such as fingers) from different angles, and extract the pointing object mask with an alpha channel using image processing software; synthesize the mask image into the document image with the pointing tip aligned with the randomly selected word in the PDF document; calculate the pointing object bounding box as the annotation based on the mask image size, tip coordinates, and document image size.

[0096] (3) Randomly sample one text instruction from the nine categories of instructions: this, previous, next × word, line, paragraph. Each instruction corresponds to a spatial relationship label. (representing this, previous, and next respectively) and an object type label. (Representing word, line, and paragraph respectively). During the data generation phase, based on the PDF parsing results, nine candidate bounding boxes (such as "current word," "previous line," "next paragraph," etc.) are pre-calculated for each target word under different spatial relationships and object types. Then, the parsed data is processed according to the instructions. Label pairs are automatically retrieved from these nine pre-computed candidate boxes to determine unique target bounding boxes. If certain label combinations are invalid in the current page context (e.g., "Previous Line" does not exist in the first line of the page), the combination is excluded during the sampling phase.

[0097] (4) Apply slight perturbations to improve generalization ability. Specific parameters for visual data augmentation include: random rotation angle at... Within the range; brightness adjustment factor in Random sampling within the interval; contrast adjustment factor in Random sampling is performed within the interval. In terms of language, a synonym substitution method based on a pre-defined vocabulary is used to rewrite some instructions to increase the diversity of language expression. The vocabulary is organized according to semantic categories, such as: a. Spatial relation words: {"this", "this", "this", "this", "current", "this one"} corresponds to "this"; {"previous", "the one before", "the one before"} corresponds to "previous"; {"next", "next", "the one after", "the one after"} corresponds to "next".

[0098] b. Object type words: {"word", "word"} corresponds to "word"; {"line", "this line", "this line"} corresponds to "line"; {"paragraph", "this paragraph", "this paragraph", "paragraph"} corresponds to "paragraph".

[0099] c. Action verbs: {"locate", "find", "output", "mark", "display", "point out", "confirm", "give"} etc.

[0100] By replacing keywords in the instructions, instruction variants with the same semantics but different expressions are generated.

[0101] Therefore, this dataset covers the nine types of instructions mentioned above and includes visual perturbations and language rewriting to improve generalization ability. Each sample in the dataset contains three parts: document image, language instruction, and target bounding box. A total of 30,000 samples were generated and divided into training, validation, and test sets at 80%, 10%, and 10% respectively.

[0102] In this embodiment, the AdamW optimizer is used during training, and the initial learning rate is set to... The weight decays to The batch size was set to 16, and the training lasted for 80 epochs. Gradient pruning and ReduceLROnPlateau learning rate scheduling strategies were used.

[0103] For fine-tuning the parameters of the BERT language encoder, a phased unfreezing strategy is adopted: In the first 20% of the total training epochs (i.e., the first 16 epochs), all BERT parameters are frozen, and only newly added modules (such as cross-modal fusion modules and bounding box prediction heads) are trained; in subsequent training, the last four layers of the BERT encoder are unfrozen for fine-tuning, while the remaining BERT parameters remain frozen to achieve a stable convergence process. The minimum number of iterations is set to 2, the maximum to 5, and the threshold... Set as The maximum number of iterations and the score improvement threshold are empirical values ​​determined based on experimental observations, aiming to balance positioning accuracy and computational efficiency. All input images are scaled to [size missing] while maintaining aspect ratio. The comparative experiment was conducted using PyTorch on an NVIDIA RTX 4090 GPU and an Intel(R) Xeon(R) Silver 4310 CPU. The specific comparison results are shown in the table below: Table 2 Performance comparison results with mainstream advanced methods

[0104] As shown in the table above, experiments on this dataset demonstrate that the complete method of this invention (AIVG) significantly outperforms several mainstream visual text localization methods in terms of word-avg-IoU, line-avg-IoU, paragraph-avg-IoU, and Recall@0.5. In word-level localization, the word-avg-IoU of the AIVG method is 0.7644, significantly higher than the best-performing comparison method TransVG++ (0.4388), and the improvement is more pronounced compared to other methods. This indicates that the proposed locator-guided semantic modulation, cross-modal feature fusion, and adaptive iterative refinement (AIR) effectively enhance the fine-grained target discrimination capability in dense text scenes.

[0105] In line-level and paragraph-level localization, the AIVG method of this invention achieves average line-avg-IoU of 0.8950 and average paragraph-avg-IoU of 0.9635, respectively. This indicates that the model can correctly understand spatial relationships such as this / previous / next and accurately align linguistic semantics with visual regions in hierarchical text structures (lines and paragraphs).

[0106] In terms of recall, the proposed AIVG method achieves 0.9873 on a recall scale of 0.5, covering almost all locatable samples, while the best comparison method, TransVG++, only achieves 0.6273, with most methods falling between 0.40 and 0.54. This gap indicates that traditional visual text localization models in PDTG often only coarsely locate the target region, while the proposed adaptive iterative refinement mechanism can continuously correct coarse predictions, significantly reducing missed detections and large offsets, thereby improving recall. It maintains stable predictions under the text localization standard.

[0107] Ablation experiments further validated the effectiveness of each module, and the key results are shown in the table below: Table 3 Main Results of Ablation Experiment

[0108] From the table above, we can see that: (1) After removing the pointer guidance module, the word-average IoU (word-avg-IoU) decreased from 0.7644 to 0.7503, and the Recall@0.7 decreased from 0.8455 to 0.8400, while the line-level and paragraph-level targets only fluctuated slightly. This indicates that pointer guidance mainly helps with word-level localization: in scenarios where targets are small and dense, using pointer priors can help the model lock the correct word.

[0109] (2) Removing FiLM or attention mechanisms from cross-modal fusion significantly reduces line-level and paragraph-level localization accuracy. After removing FiLM, the average line-avg IoU and paragraph-level average IoU decreased from 0.8950 and 0.9635 to 0.8443 and 0.9413, respectively, and Recall@0.7 decreased from 0.8455 to 0.8242. After removing cross-attention, the average line-avg IoU and paragraph-level average IoU decreased to 0.8585 and 0.9536, respectively. These results validate the effectiveness of cross-modal fusion. The synergistic effect of FiLM and cross-attention allows the model to selectively amplify instruction-related regions at the channel (global) level and then strengthen alignment at the spatial (local) level, thereby significantly improving line-level and paragraph-level localization performance.

[0110] (3) After removing the iterative refinement module, Recall@0.7 decreased from 0.8455 to 0.8358, and both the word-average IoU and line-average IoU decreased, while the paragraph-average IoU slightly increased to 0.9700. This is because the paragraph target is larger, and the coarse prediction can already locate it well; subsequent iterations may bring slight offsets, resulting in small fluctuations in IoU. The overall trend shows that the iterative module continuously corrects the coarse location through the closed-loop mechanism of "prediction-scoring-optimization", enabling more predictions to be more rigorously refined. It remains effective under the standard, thereby improving robustness and stability.

[0111] (4) After removing the spatial / object classification branch, the word-avg IoU, line-avg IoU, and paragraph-avg IoU decreased to 0.7525, 0.8772, and 0.9525, respectively, and the Recall@0.7 decreased to 0.8327. This indicates that explicitly learning spatial relationship categories (this / previous / next) and target unit types (word / line / paragraph) helps the model distinguish between target scale and relative position.

[0112] The above ablation experiment results show that the various technical modules of this invention are not simply superimposed, but deeply coupled and functionally complementary, forming an organic whole, specifically manifested as follows: (1) The core function of the pointing object guidance module is to provide accurate spatial priors: This module mainly improves the localization accuracy of word-level targets (its removal leads to the most significant decrease in word-avg-IoU). This shows that in scenarios where targets are dense and similar in appearance (word level), the technical feature of pointing position is closely combined with the algorithm features such as mask generation and feature modulation, which locks the initial attention focus for the model and is the key entry point and spatial foundation for solving the technical problem of "dense text differentiation".

[0113] (2) The core function of the cross-modal fusion module (FiLM and attention) is to achieve semantic-spatial alignment: the removal of this module has a greater impact on line and paragraph-level localization (line-avg-IoU and para-avg-IoU decrease significantly). This indicates that the global channel modulation of FiLM and the local spatial alignment of cross-modal attention work together to enhance the technical feature of visual and language fusion, enabling the model to understand complex spatial relationship semantics such as "previous / next". Their function is to transform ambiguous instructions into precise spatial offsets.

[0114] (3) The core function of the adaptive iterative refinement module is to achieve accuracy convergence and stability: the removal of this module leads to a decrease in the high-threshold index Recall@0.7, indicating that its core value lies in continuously optimizing the localization accuracy. This module iteratively combines multiple algorithms, such as the geometric information of the predicted bounding box, the semantic-visual consistency judgment of the scorer, and the dynamic modulation of the feature optimization network, to form a closed-loop technology system of "prediction-evaluation-correction". The algorithms interact with each other in this system to ensure the high reliability of the final output.

[0115] In summary, the removal of any single module will disrupt the performance balance and accuracy limit of the entire technical solution across different target levels (words, lines, and paragraphs). This demonstrates that there is a close functional dependency and synergistic effect between the algorithmic features and technical features proposed in this invention, which together contribute to the overall technical effect that far exceeds that of a single module or existing technologies.

[0116] In addition, such as Figure 6 As shown, tests were conducted on real-life document photos, and the results demonstrate that the model trained on the generated dataset generalizes well to real images. The first line uses the instruction "locate this word"; the second line uses "locate the previous line" and "locate the previous paragraph"; and the third line uses "locate this paragraph". Since real photos are typically noisier than synthetic datasets, the predicted bounding boxes may have slight offsets, but the overall localization remains reliable.

[0117] In summary, the method of the present invention has reliable positioning performance and demonstrates good generalization ability and practical value.

[0118] Example 2 This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the locator text localization method based on iterative semantic visual association as described in Embodiment 1.

[0119] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A locator text localization method based on iterative semantic visual association, characterized in that, Includes the following steps: S1: Obtain the document image containing the pointer and the natural language instructions describing the target text region; S2: Extract the visual features of the document image and the linguistic features of the natural language instructions, and extract the instruction semantic vector; S3: Predict the bounding box of the pointing object based on the visual features and generate a pointing mask accordingly, so as to perform weighted pooling on the visual features and extract the pointing context features; S4: After concatenating the pointing context feature with the instruction semantic vector, input the two-layer perceptron network to generate channel scaling coefficients; use the pointing mask and the channel scaling coefficients to perform spatial modulation and channel modulation on the visual features respectively to obtain enhanced visual features; S5: Generate FiLM modulation parameters based on the instruction semantic vector and perform channel modulation on the enhanced visual features; The modulated visual features are used as keys and values, and the language features are used as queries for cross-modal attention fusion. The initial target bounding box is obtained based on fusion feature regression; S6: Starting from the initial target bounding box, perform an iterative refinement process, including: generating a target mask based on the current target bounding box, and calculating a semantic-visual consistency score based on the current fusion features; when the score increases beyond a preset threshold, generating channel modulation parameters based on the geometric feature differences between the current target bounding box and the pointing object bounding box, the semantic information of the spatial relationship category and the target object type, and the instruction semantic vector, enhancing the current fusion features, and then predicting the target bounding box for the next iteration through regression; repeating the iteration until the termination condition is met, and outputting the target bounding box with the highest semantic-visual consistency score as the final localization result.

2. The object text localization method based on iterative semantic visual association according to claim 1, characterized in that, The step S3 of generating the pointing mask includes: calculating the diagonal length of the bounding box of the pointing object, using a preset ratio of the diagonal length as the expansion radius, expanding the bounding box, and generating an expanded pointing mask.

3. The object text localization method based on iterative semantic visual association according to claim 2, characterized in that, The preset ratio is 0.

3.

4. The object text localization method based on iterative semantic visual association according to claim 1, characterized in that, In step S4, the modulation of the visual features using the pointing mask and the channel scaling factor is achieved by the following formula: in, As a visual feature, For pointing to the mask, This is the channel scaling factor. Indicates a broadcast operation. and For learnable scaling parameters, This represents element-wise multiplication. To enhance visual features.

5. The object text localization method based on iterative semantic visual association according to claim 1, characterized in that, In step S2, extracting the instruction semantic vector from the language features includes: extracting the vector corresponding to the CLS marker output by the language encoder, and projecting it through a linear projection layer to obtain the instruction semantic vector.

6. The object text localization method based on iterative semantic visual association according to claim 1, characterized in that, The spatial relationship category and target object type are obtained by decoding the instruction semantic vector in the following way: the instruction semantic vector is input into the spatial relationship classifier and the object type classifier to obtain the probability distribution of the spatial relationship category and the probability distribution of the target object type; the spatial relationship category includes current, previous, and next; the target object type includes word, line, and paragraph.

7. The object text localization method based on iterative semantic visual association according to claim 1, characterized in that, In step S6, calculating the semantic-visual consistency score based on the target mask and the fusion features of the current iteration includes: using the target mask to perform mask-weighted pooling on the fusion features of the current iteration to obtain local features, performing global pooling on the fusion features to obtain global features, concatenating the local features, the global features and the instruction semantic vector, inputting them into a multilayer perceptron composed of multiple fully connected layers and processing them through the Sigmoid activation function to obtain the semantic-visual consistency score.

8. The object text localization method based on iterative semantic visual association according to claim 1, characterized in that, In step S6, generating channel modulation parameters for channel modulation includes: calculating the difference between the center coordinates of the current target bounding box and the center coordinates of the pointing object bounding box, as well as the logarithmic ratio of their width and height, to form a geometric descriptor; concatenating the geometric descriptor, the probability distribution of the spatial relationship category, the probability distribution of the target object type, and the instruction semantic vector, and obtaining FiLM parameters for channel scaling and channel bias through nonlinear mapping as the channel modulation parameters; wherein, the parameters for channel scaling are processed by the hyperbolic tangent function.

9. The object text localization method based on iterative semantic visual association according to claim 1, characterized in that, The termination condition for the iterative refinement process is: the number of iterations reaches a preset maximum value, or the improvement of the semantic-visual consistency score in two consecutive iterations does not exceed the preset threshold.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the locator text localization method based on iterative semantic visual association as described in any one of claims 1 to 9.

Images