An image recognition-based power engineering list analysis method, device and medium

Abstract

This invention discloses a method, device, and medium for parsing power engineering lists based on image recognition, comprising: acquiring an image of the power engineering list to be recognized and extracting a visual feature map; decoding character by character with a decoder, obtaining the hidden state of the current decoding step and predicting the current character and confidence level; acquiring cohesive words, generating a modulation signal and adding it to the cohesive words to obtain an enhanced query vector; using the enhanced query vector as a query vector and performing cross-attention calculation with the visual feature map to obtain global visual context features, while simultaneously acquiring local sequence features from the self-attention output; generating gating coefficients, and determining the current character output after weighted fusion of global and local features based on the gating coefficients; and finally generating a recognition result. This invention, through learnable cohesive words and a dynamic gating mechanism, achieves on-demand fusion of global context and local features in long text recognition, effectively improving the recognition accuracy of long texts such as long equipment names in power engineering lists.

Description

Technical Field

[0001] This invention relates to the field of image processing and text recognition technology, and in particular to a method, device and medium for parsing power engineering inventory based on image recognition. Background Technology

[0002] A power engineering inventory is a standardized document used during the construction of power projects to record key information such as equipment name, specifications, quantity, and unit. It is typically presented in tabular form and contains numerous long text fields. For example, the equipment name field often contains long text entries containing dozens of characters, such as "10kV cross-linked polyethylene insulated armored power cable" or "sulfur hexafluoride enclosed combined electrical circuit breaker." In practical engineering applications, the power engineering inventory is usually in the form of scanned copies or on-site photographs. It needs to be converted into structured data using image recognition technology for subsequent processing in a power materials management system (PMS) or geographic information system (GIS).

[0003] Existing deep learning-based text recognition methods typically employ convolutional neural networks (CNNs) to extract visual features from images, and then output the recognition results character by character through recurrent neural networks (RNNs) or transformer-based decoders. Classical methods, represented by CRNNs (Convolutional Recurrent Neural Networks), have achieved good results in conventional text recognition tasks.

[0004] However, the long text in power engineering project lists presents unique challenges to existing methods. On the one hand, due to the limited receptive field of CNNs, when the text length exceeds the coverage of the receptive field, the network struggles to extract features from long text sequences spanning hundreds of pixels as a whole, easily leading to feature fragmentation. On the other hand, while existing Transformer-based decoders can capture long-distance dependencies to some extent through self-attention mechanisms, they utilize visual features in the same way at each decoding step, lacking the ability to perceive and adaptively adjust to the recognition difficulty of the current decoding step. When the model encounters long equipment names in power engineering project lists, if a character has a low prediction confidence due to deformation, blurring, or other reasons, existing models cannot proactively introduce additional global contextual information for correction, and can only rely on a fixed attention distribution for decoding, limiting recognition accuracy.

[0005] To address this, existing technologies have implemented several improvements. For instance, some methods expand the model's perception range for long texts by designing feature extraction networks with larger receptive fields, but this increases the number of model parameters and computational overhead. Other methods correct the recognition results after completion using post-processing rules or language models, but these are reactive measures that cannot dynamically intervene during the recognition process and are highly dependent on the completeness of the rule design and the domain adaptability of the language model. Still other methods employ gating mechanisms to fuse multi-source features, but existing gating mechanisms typically only input the confidence level of the current decoding step or a single modality feature, failing to fully utilize the synergistic relationship between the decoder's internal state and learnable prior knowledge, making it difficult to achieve optimal global-local feature dynamic balance in long text recognition scenarios.

[0006] In summary, existing text recognition methods, in the specific scenario of power engineering lists, lack a technical solution that can dynamically perceive the recognition difficulty of the current decoding step during the recognition process and introduce global contextual information as needed to assist in character judgment. Summary of the Invention

[0007] The purpose of this invention is to solve the problem that when existing technologies are applied to image recognition of power engineering lists, the recognition accuracy of some medium and long texts decreases due to visual feature breaks.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0009] A method for parsing power engineering bills of quantities based on image recognition, comprising:

[0010] Obtain the image to be identified from the list of power engineering projects;

[0011] The image to be identified is subjected to text region detection to obtain at least one text region;

[0012] Feature extraction is performed on the text region to obtain a visual feature map of the text region;

[0013] The visual feature map is input into the decoder for character-by-character decoding. During the decoding process, for the current decoding step, the hidden state of the current decoding step is obtained, and the current character and its confidence are predicted based on the hidden state, and the cohesive word is obtained; wherein, the cohesive word is a learnable parameter vector and is pre-stored in the trainable parameter set of the decoder.

[0014] The cohesive lexical is used as the base query vector, the hidden state is mapped to a modulation signal through a linear transformation, and the modulation signal is added to the base query vector to generate an enhanced query vector.

[0015] Parallel execution uses the enhanced query vector as the query vector and performs cross-attention calculation with the visual feature map to obtain global visual context features;

[0016] The local sequence features of the current decoding step are obtained; wherein the local sequence features are the features output by the self-attention module of the decoder, which are used to characterize the context dependencies of the currently decoded character sequence.

[0017] The confidence level and the norm of the enhanced query vector are used as scalar inputs and input into the gating network to obtain the gating input vector.

[0018] The gated input vector is input into a gated network to generate gated coefficients; wherein the gated coefficients are negatively correlated with the confidence level.

[0019] According to the gating coefficient, the global visual context features and the local sequence features are weighted and fused to obtain fused features, and the character output of the current decoding step is determined according to the fused features; wherein, the gating coefficient is used to control the weight of the global visual context features during fusion, and the lower the confidence level and the larger the norm of the enhanced query vector, the greater the weight of the global visual context features;

[0020] Based on the character output of each decoding step, the recognition result of the text region is generated.

[0021] Furthermore, the step of generating an enhanced query vector based on the hidden state and the cohesive tokens includes:

[0022] The hidden state is mapped to a modulation signal through a linear transformation; the modulation signal is added element by element to the cohesive word to obtain the enhanced query vector;

[0023] The cohesive lexical units serve as the basic components of the enhanced query vector and are updated through backpropagation during training to learn the common preferences of global contextual retrieval.

[0024] Furthermore, the gating network generates gating coefficients based on a preset analytical function. The analytical function is:

[0025] ;

[0026] in, The confidence level is... To enhance query vectors L2 norm after layer normalization; , , These are learnable scalar parameters, and and During training, weights are clamped to force positive values; the weighted fusion is performed according to the following formula:

[0027] Fusion features = (1- )×Local sequence features+ ×Global visual context features.

[0028] Furthermore, the global visual context features and the local sequence features are aligned in the feature dimension;

[0029] The decoder is a transformer-based decoder, which includes parallel self-attention branches and cross-attention branches;

[0030] The self-attention branch uses causal masking to limit the visibility of future characters, and the self-attention branch uses rotational position encoding to maintain temporal order; the cross-attention branch uses learnable two-dimensional absolute position encoding or two-dimensional relative position bias for the visual feature map.

[0031] The cross-attention branch uses the enhanced query vector as an independent query vector and calculates the global visual context features with the visual feature map;

[0032] The self-attention branch and the cross-attention branch are executed in parallel during forward computation; their outputs are weighted and fused by the gating coefficients and then jointly input into the subsequent feedforward network layer.

[0033] Furthermore, the method also includes a training phase and an inference phase:

[0034] During the training phase, the cohesive lexical units always participate in forward computation and backward propagation, and their parameters are updated normally through gradient descent.

[0035] During the inference phase, the gating coefficients are calculated in real time by the analytical function to adaptively control the fusion weights of the global visual context features and the local sequence features.

[0036] Furthermore, the step of performing text region detection on the image to be identified to obtain at least one text region includes:

[0037] Feature extraction is performed on the image to be identified to obtain an image feature map;

[0038] The image feature map is input into a differentiable binarization segmentation network or a segmentation-based text detection network to obtain a text region probability map and a threshold map.

[0039] Based on the text region probability map and the threshold map, an approximate binary mask is generated through a differentiable binarization operation;

[0040] Perform connected component analysis or contour detection on the approximate binarized mask to obtain the initial text region;

[0041] The initial text region is fitted with a polygon, and it is determined whether the fitted polygon is a curved text region. If it is determined to be a curved region, the curved text region is corrected to a horizontal rectangular region by thin plate spline transformation or perspective transformation. If it is determined to be a non-curved region, its tilted bounding box is directly determined.

[0042] Based on the corrected horizontal rectangular area or the tilted bounding box, determine the final position information of the at least one text area to accommodate the tilted table lines or curved text in the power engineering list.

[0043] Furthermore, the calculation of the confidence level of the current character prediction includes:

[0044] Obtain the probability distribution of the output of the current decoding step;

[0045] The input confidence C of the gated network adopts the maximum original probability value of the probability distribution;

[0046] The probability distribution is calibrated to obtain the calibrated probability distribution.

[0047] Furthermore, the visual feature map is obtained by extracting features from the text region through a convolutional neural network or a visual transformer; the character-by-character decoding includes character prediction through the self-attention mechanism and cross-attention mechanism of the decoder, and during the decoding process, the decoding process is terminated when the cumulative number of decoding steps exceeds a preset maximum length threshold or a terminator is predicted.

[0048] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor runs the program, it implements the image recognition-based power engineering inventory parsing method.

[0049] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image recognition-based power engineering inventory parsing method.

[0050] The beneficial effects of this invention are:

[0051] First, this invention provides a power engineering inventory parsing method based on image recognition. By setting learnable cohesive terms as the basic query vector for global context retrieval, and dynamically modulating the cohesive terms using a modulation signal generated based on the hidden state of the current decoding step, an enhanced query vector is obtained. This enhanced query vector is then used to perform cross-attention calculation with a visual feature map to acquire global visual context features. This solves the problems in existing technologies where global context retrieval lacks guidance from the current decoding step state and the retrieval results do not match the current recognition requirements. Compared to the fixed cross-attention mechanism or simple gating schemes based solely on confidence in existing technologies, this invention achieves greater accuracy and personalization in global context retrieval through a dual mechanism of "learnable prior + current state modulation," effectively improving the recognition accuracy of difficult characters in long texts.

[0052] Secondly, the power engineering inventory parsing method based on image recognition provided by this invention generates a gating coefficient negatively correlated with confidence level, and uses this gating coefficient to perform weighted fusion of global visual context features and local sequence features. This solves the problem in existing technologies where the gating mechanism relies solely on a single confidence level indicator and cannot fully reflect the quality of global context retrieval. Compared to existing gating schemes with fixed fusion weights or based solely on confidence level, this invention achieves refined and adaptive control of the degree of fusion between global and local features through a dual-input gating design of "confidence level + query vector norm," resulting in superior recognition performance in long text recognition scenarios for power engineering inventory. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the overall process of a power engineering bill of quantities parsing method based on image recognition provided by the present invention. Detailed Implementation

[0054] To make the above-mentioned objects, features and advantages of the present invention more readily understood, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0055] In this invention, the "cohesive lexicon" refers to a learnable parameter vector, named for its "cohesive" role at the feature level: In long text recognition, due to the limited receptive field of convolutional neural networks, the visual feature maps of long texts often present scattered and localized representations in space, making it difficult to form a unified perception of the complete long text. The cohesive lexicon, as a learnable prior query vector, works in conjunction with the hidden state of the current decoding step during the decoding process. Through a cross-attention mechanism, it retrieves and aggregates global contextual information related to the current character from the visual feature map, re-associating the scattered and localized visual representations into a holistic feature representation with contextual coherence at the feature level, thus playing a technical role in "coheding" fragmented visual features.

[0056] Example 1

[0057] The power engineering inventory parsing method based on image recognition provided by this invention includes the following main stages in its overall processing flow:

[0058] Step S100: Image acquisition and preprocessing.

[0059] The process involves acquiring the image of the power project inventory list to be identified. This image can be obtained from various sources, including scanning a paper list, photographing a paper list with a digital camera or smartphone, and screenshots of electronic documents exported from the power project management system. After acquiring the image, optional preprocessing operations include image grayscale conversion, contrast enhancement, illumination normalization, and geometric correction to improve the accuracy of subsequent text region detection. For example, for images with perspective distortion caused by the shooting angle, perspective transformation can be used to correct them into a front view.

[0060] Specifically, the images to be identified from the power engineering project inventory can come from various sources. In a typical application scenario, on-site power engineering workers use mobile devices (such as smartphones or tablets) to photograph the paper power engineering inventory, and the resulting images are uploaded to the server via a wireless network for recognition processing. In another application scenario, electronic documents (such as PDF inventory files) stored in the power engineering management system are converted into image files, which are then input into this method as the images to be identified.

[0061] Optional preprocessing operations include, but are not limited to, the following:

[0062] Image grayscale conversion: Converting a color image to a grayscale image to reduce the computational burden of subsequent processing. For color images, a weighted average method can be used to convert the values ​​of the three RGB channels into a single grayscale value.

[0063] Contrast enhancement: By using methods such as histogram equalization or adaptive histogram equalization (CLAHE), the contrast of the image is enhanced, making the difference between the text area and the background area more obvious.

[0064] Illumination normalization: For images with uneven illumination, illumination normalization can be performed using homomorphic filtering or the Retinex algorithm to reduce the impact of illumination changes on text detection and recognition.

[0065] Geometric correction: For perspective distortion caused by shooting angle, the image can be corrected to a front view by detecting quadrilateral regions in the image (such as the boundary of a sheet of paper) and calculating the perspective transformation matrix.

[0066] Step S200: Perform text region detection on the preprocessed image to be recognized, locating one or more regions in the image that contain text. Text region detection can be implemented using segmentation-based methods or regression-based methods.

[0067] Taking a segmentation-based method as an example, the image feature map of the image to be identified is first extracted by a convolutional neural network. Then, the image feature map is input into a differentiable binarization segmentation network or a segmentation-based text detection network (such as PSENet, PAN, etc.) to obtain a text region probability map and a threshold map.

[0068] Then, based on the text region probability map and threshold map, an approximate binarized mask is generated through differentiable binarization. Connectivity analysis or contour detection is performed on the approximate binarized mask to obtain the initial text region. Polygon fitting is performed on the initial text region to determine whether the fitted polygon is a curved text region. If it is determined to be a curved region, the curved text region is corrected to a horizontal rectangular region through thin plate spline (TPS) transformation or perspective transformation. If it is determined to be a non-curved region, its tilted bounding box is directly determined. Based on the corrected horizontal rectangular region or tilted bounding box, the final position information of each text region is determined to adapt to the tilted table lines or curved text that may exist in the power engineering bill of quantities.

[0069] Specifically, text region detection is a preliminary step in text recognition, and its goal is to locate all regions containing text in the image to be recognized. In the scenario of power engineering bills, text regions are usually presented as horizontally arranged table cells, but there may also be cases where they are tilted (e.g., table lines are slightly rotated) or curved (e.g., the bill of contents paper is wrinkled).

[0070] In a preferred embodiment of this example, text region detection employs a segmentation-based method, specifically including the following sub-steps:

[0071] Sub-step S210: Using a convolutional neural network (such as ResNet50) as the backbone network, multi-scale feature extraction is performed on the image to be recognized to obtain an image feature map. This feature map preserves the spatial structure information of the image and has a high level of semantic abstraction.

[0072] Sub-step S220: Input the image feature map into a differentiable binarization segmentation network (DBNet) to obtain a text region probability map and a threshold map. Each pixel value in the text region probability map represents the probability that the pixel belongs to the text region; each pixel value in the threshold map represents the adaptive threshold used for binarization at that location.

[0073] Sub-step S230: Based on the text region probability map and threshold map, generate an approximate binarized mask through a differentiable binarization operation. The formula for the differentiable binarization operation is:

[0074] ;

[0075] in, Let be the value at position (i,j) in the probability graph. The values ​​at the same position in the threshold graph. This is a magnification factor (usually set to 50). The difference between the probability map and the threshold map is then mapped to the [0,1] interval using the Sigmoid function, resulting in an approximately binarized mask. Since this operation is differentiable, the gradient can be backpropagated to the prediction networks of the probability map and the threshold map, thus enabling end-to-end training.

[0076] Sub-step S240: Perform connected component analysis or contour detection on the approximate binarized mask to aggregate interconnected text pixels into an initial text region. Each connected component corresponds to a candidate text instance.

[0077] Sub-step S250: Perform polygon fitting (e.g., using the minimum bounding rectangle or minimum bounding polygon) on each initial text region to obtain the fitted polygon. Determine whether the fitted polygon is a curved text region. The determination criteria may include: whether the number of sides of the polygon exceeds a preset threshold (e.g., if the number of sides is greater than 6, it may be curved text), whether the aspect ratio of the polygon is within a specific range, and whether each vertex of the polygon deviates significantly from a straight line, etc. If it is determined to be a curved region, the curved text region is corrected to a horizontal rectangular region through Thin Plate Spline Transform (TPS); if it is determined to be a non-curved region, its tilted bounding box (i.e., the minimum bounding rotated rectangle) is directly determined.

[0078] Sub-step S260: Based on the corrected horizontal rectangular area or tilted bounding box, determine the final position information of each text area (including center point coordinates, width, height and rotation angle), as the basis for text area cropping in the subsequent step S300.

[0079] Step S300: Extract features from each detected text region to obtain a visual feature map of that text region. Visual feature extraction can be achieved using convolutional neural networks (such as ResNet, VGG, etc.) or visual transformers (VisionTransformer, ViT).

[0080] In a preferred embodiment of this invention, a convolutional neural network (CNN) is used as an example. The cropped text region image is input into the CNN, processed through multiple convolutional and downsampling layers, and outputs a three-dimensional feature tensor with spatial dimensions (height H × width W) and channel dimensions C, which is the visual feature map. This visual feature map preserves the visual texture information at each spatial location within the text region and serves as the basis for subsequent character-by-character recognition by the decoder. Optionally, a Feature Pyramid Network (FPN) can be used for multi-scale feature fusion during feature extraction to enhance the perception of characters of different sizes.

[0081] In one specific implementation, visual feature extraction is achieved using a visual transformer (ViT). Specifically, the text region image (corrected to a horizontal orientation) detected and cropped in step S200 is divided into multiple image patches, each patch being P×P pixels in size (e.g., P=8 or P=16). Each image patch is flattened into a one-dimensional vector, linearly projected to obtain the image patch embedding, and then superimposed with a learnable two-dimensional absolute position encoding or a two-dimensional relative position bias. This embedding is then input into a multi-head self-attention transformer for processing. The feature tensor output by the transformer is the visual feature map, whose spatial dimension corresponds to the number of image patches (H patches in the height direction × W patches in the width direction), and the channel dimension is the hidden layer dimension of the transformer.

[0082] In another specific implementation, visual feature extraction is achieved using a convolutional neural network (CNN). The text region image is input into the CNN (such as ResNet34), processed through multiple convolutional layers, pooling layers, and activation function layers, outputting a feature tensor of dimension C×H×W as the visual feature map. Here, H and W are the height and width of the feature map (downsampled relative to the original image), and C is the number of channels.

[0083] Regardless of the implementation method used, the visual feature map retains the visual texture information of each spatial location within the text area, which is the basis for subsequent cross-attention calculation.

[0084] Step S400: Input the visual feature map into the transformer-based decoder to begin character-by-character decoding. The decoder includes a multi-head self-attention module, a multi-head cross-attention module, and a feed-forward network module.

[0085] Before decoding begins, the decoder input needs to be initialized, typically using the start character as the input for the first decoding step. The decoder employs an autoregressive approach, outputting character by character; that is, the output of each decoding step depends on the output of all previous decoding steps. During decoding, the process terminates when the cumulative number of decoding steps exceeds a preset maximum length threshold (e.g., 25 characters) or when a stop character is predicted. The preset maximum length threshold can be set based on the actual length distribution of text fields in the power engineering bill of quantities; for example, it could be set to 50 characters to cover the length range of most equipment names.

[0086] Step S500: Current decoding step state acquisition and character prediction stage. In the current decoding step... Get the hidden state of the decoder in the current step. The hidden state refers to the feature vector generated by the decoder after undergoing multiple transformations such as self-attention, cross-attention, and feedforward networks. It encodes the contextual information of the current decoding step, including the semantic information of the decoded character sequence and the visual focus information of the current step.

[0087] Based on the hidden state The probability distribution of the current character is predicted through a linear classification layer (usually a fully connected layer with a softmax activation function). And from this, we can obtain the current predicted character and its confidence level. Among them, confidence level The maximum original probability value of the probability distribution can be used, i.e. ;

[0088] Alternatively, to obtain a more reliable confidence estimate, the probability distribution can be calibrated (e.g., by temperature scaling or Platt scaling) to obtain a calibrated probability distribution, and the maximum value can then be taken as the confidence level. The value range is [0,1]. A larger value indicates a more accurate prediction from the model.

[0089] At the current decoding step t, the decoder outputs a probability distribution. ,in The size of the character set (i.e., the number of all possible characters). Let represent the probability of the i-th character, and .

[0090] In a specific embodiment, the confidence level directly adopts the maximum original probability value of the probability distribution, that is . For example, if in the probability distribution output at the current decoding step, the probability of the character "dian" is 0.85, and the probabilities of the remaining characters are all less than 0.85, then the confidence level = 0.85.

[0091] In another specific embodiment, in order to improve the reliability of confidence level estimation, the probability distribution is first calibrated, and then the maximum value of the calibrated probability distribution is taken as the confidence level. The temperature scaling method can be adopted for the calibration process: , where is the temperature parameter ( > 0). When >1, the probability distribution becomes smoother (that is, the gap between each probability value is narrowed), which helps to alleviate the problem of over-confidence of the model; when <1, the probability distribution becomes sharper. After temperature scaling, the maximum value of the calibrated probability distribution is taken as the confidence level .

[0092] The input confidence level of the gating network adopts the value calculated by the above method. During the training process, the confidence level participates in the forward calculation as one of the inputs of the gating network, and its gradient will be transmitted back to the decoder through the gating network, thereby affecting the parameter update of the decoder.

[0093] Step S600: Obtain the pre-stored Bonding Token . The bonding token is a learnable parameter vector, whose dimension is consistent with the dimension of the query vector in the decoder. The bonding token is pre-stored in the set of trainable parameters of the decoder, and is updated through back propagation and gradient descent during the model training process. It does not depend on any specific input image or decoding step, but serves as a globally shared learnable prior, for encoding the common preference of global context retrieval in the long text scenario of power engineering lists.

[0094] The bonding token described in the present invention is a learnable parameter vector with the same dimension as the query vector, its function is to serve as the basic query vector for global context retrieval, and its dimension is denoted as (consistent with the dimension of the query vector in the decoder). At the time of model initialization, random initialization (such as sampling from a normal distribution with a mean of 0 and a variance of 0.02) or zero initialization may be adopted. During the training process, As one of the trainable parameters of the model, it is updated through backpropagation and gradient descent.

[0095] Hidden state of the current decoding step The dimension is denoted as (Usually consistent with the hidden layer dimension of the decoder). Mapped to a modulation signal through linear transformation : ,in It is A learnable weight matrix of dimension 1. This linear transformation can be understood as mapping the hidden state from the latent space of the decoder to the query vector space.

[0096] modulated signal With adhesive elements Adding elements one by one yields the enhanced query vector. This addition operation requires... , and Having the same vector dimension The physical meaning of adding elements one by one is: It provides a common foundation for global retrieval (the optimal retrieval direction learned during training). It provides a personalized offset for the current decoding step (fine-tuning the search direction based on the current state), and the result is obtained by superimposing the two. It possesses both global prior knowledge and the specificity of the current step.

[0097] Specifically, during training, the cohesive lexical units continuously adjust their parameter values ​​to learn the knowledge of "which types of global visual information are valuable for auxiliary character judgment in long text recognition".

[0098] It should be noted that the cohesive lexical units described in this invention differ from conventional learnable parameters in deep learning models (such as convolutional kernel weights, fully connected layer weights, positional encodings, etc.) in the following aspects:

[0099] (i) Initialization methods. The initialization of conventional learnable parameters aims at numerical stability, employing random initialization or Xavier / He initialization, with initial values ​​not carrying task-specific semantics. In contrast, the initialization of cohesive lexical units can be based on pre-trained model transfer or domain data statistical priors, so that their initial values ​​carry semantic prior information that is helpful for global retrieval of long texts.

[0100] (II) Update Rules. The update of conventional learnable parameters is directly driven by the final loss function, with each parameter updated independently. However, after the glued tokens generate the enhanced query vector, the norm of this vector is used as one of the inputs to the gating network in the calculation of the gating coefficients. The gating coefficients, in turn, determine the fusion weight of global visual context features in the final prediction, thus affecting the gradient backpropagation of the loss function to the glued tokens. Therefore, the update of the glued tokens is indirectly modulated by the gating network, exhibiting task-driven adaptive characteristics.

[0101] (III) Dimension Setting. The dimension of conventional learnable parameters is determined by the input-output structure of the network layer. However, the dimension of cohesive terms is determined by the functional requirement of aligning with the decoder's query vector space, and its dimension selection is directly related to the richness of global retrieval prior information that can be encoded.

[0102] (iv) Connection Method. Conventional learnable parameters are usually connected to input features through multiplication (such as matrix multiplication) and participate in a single, fixed data stream. However, cohesive lexical units are fused with the modulation signal of the current decoding step through element-wise addition. The resulting enhanced query vector participates in two data streams simultaneously: first, it is used as a query vector to perform cross-attention calculation with the visual feature map to obtain global visual context features; second, its norm is used as one of the inputs to the gating network to participate in the generation of gating coefficients. This connection method of "adding the basic query vector to the modulation signal" is different from the conventional method of "obtaining the query vector through linear transformation of input features".

[0103] Furthermore, this invention enhances the query vector through the design of "basic query vector + modulation signal". It retains the common preferences of global contextual retrieval learned by the cohesive tokens during training (basic components) while incorporating personalized state information of the current decoding step (modulation components), enabling it to adaptively adjust the retrieval focus based on the position and state of the character currently being decoded. Furthermore, as the basic components of the enhanced query vector, the cohesive tokens can also be updated through backpropagation during training to learn the common preferences of global contextual retrieval.

[0104] Step S700: The enhanced query vector generated in step S600... As the query vector, cross-attention is calculated with the visual feature map obtained in step S300 to obtain the global visual context features. Specifically, the visual feature maps are first transformed into key matrices through linear transformations. Sum matrix Then calculate the enhanced query vector. and The correlation scores between each position are normalized using softmax to obtain the attention weight distribution; finally, the attention weights are used to... We perform weighted summation to obtain the global visual context features. This feature aggregates visual information from the spatial locations in the visual feature map that are most relevant to the current decoding step, providing global visual context support for the recognition of the current character.

[0105] The difference between the above cross-attention calculation process and the conventional cross-attention calculation in the decoder is that the query vector in conventional cross-attention is the hidden state of the current decoding step. Or its linear transformation, while the query vector in this step is an enhanced query vector modulated by glued lexical units. .because By incorporating learnable global retrieval prior knowledge, the cross-attention mechanism guided by it can more effectively retrieve global contextual information that is helpful for long text recognition from visual feature maps.

[0106] Step S800: Local Sequence Feature Acquisition Stage. This step is logically parallel to step S700 in the forward computation, and there is no data dependency between them. Acquire the local sequence features of the current decoding step. Local sequence features These are the features output by the decoder's self-attention module, used to characterize the contextual dependencies of the currently decoded character sequence. Specifically, in the decoder's self-attention module, the decoded character sequence (including all outputs before the current step) is converted into a sequence vector through an embedding layer, and then self-attention computation with causal masking is performed to ensure that each position can only focus on its preceding positions (i.e., maintaining autoregressive properties). The output is the local sequence feature. This feature reflects the local semantic dependencies between decoded characters. For example, when recognizing "crosslinked polyethylene," the decoded "crosslinked" will have a semantic expectation of the currently decoded "poly". Optionally, the self-attention module can employ rotational position encoding to maintain the temporal order of the sequence.

[0107] Specifically, in the self-attention module of the decoder, the character sequence that has been output before the current decoding step is... ,(in The start symbol is converted into a sequence vector through the embedding layer, and then self-attention computation with causal mask is performed.

[0108] The purpose of causal masking is to ensure that when calculating the attention weights at the current position, only sequence information preceding that position (including the position itself) is considered, while future information is ignored, thus preserving the autoregressive decoding characteristics. The output of the self-attention module is the local sequence feature. , dimension . The encoding captures the contextual dependencies of the decoded character sequence, reflecting the local semantic coherence within the sequence. For example, when recognizing "electric cable," after decoding the character "electricity," the prediction of the character "force" will be influenced by the semantics of "electricity," and this influence is transmitted through local sequence features.

[0109] Optionally, the self-attention module can employ Rotation Position Encoding (RoPE) to encode the positional information of characters in the sequence. RoPE rotates the query vector and key vector using a rotation matrix, implicitly incorporating relative positional information into the attention calculation, thereby enhancing the model's ability to perceive the sequence order.

[0110] Step S900: Gating coefficient generation stage. The confidence level of the current decoding step is... and enhanced query vectors The norm of the input is used as a scalar input to the gating network to generate gating coefficients. .

[0111] Specifically, the enhanced query vector is first calculated. The L2 norm is calculated, and layer normalization is applied to this norm to obtain the normalized norm value. Confidence level. The value obtained in step S500 is used directly. and As two scalar inputs, they are input to the gating network.

[0112] Gated networks generate gating coefficients based on a pre-defined analytical function. In one specific implementation, the parsing function is:

[0113] ;

[0114] in, For confidence level, To enhance query vectors L2 norm after layer normalization; , , These are learnable scalar parameters, and and During training, the weights are clamped to force the values ​​to be positive (i.e., > 0, >0), to ensure the gating coefficient is 0. With confidence level Negative correlation and enhanced query vector norm Positive correlation. The specific implementation of weight clamping is as follows: after each parameter update, check... and If the value is less than 0, it is set to 0.

[0115] The specific operation of layer normalization is as follows: ,in and They are respectively The mean and standard deviation of each dimension value (calculated during layer normalization). After layer normalization. It is a scalar value.

[0116] It should be noted that the gating coefficient The value range is [0,1], representing the weight ratio of global visual context features during fusion. The lower the confidence level and the larger the norm of the enhanced query vector, the higher the gating coefficient. The larger the value, the greater the weight of the global visual context features.

[0117] Step S1000: Feature fusion and character output stage. The gating coefficients are... Global visual context features and local sequence features Weighted fusion is performed to obtain fusion features. The specific calculation method for weighted fusion is as follows:

[0118] ;

[0119] in, and The features are pre-aligned (i.e., both have the same vector dimension) to ensure the feasibility of the weighted summation operation.

[0120] Furthermore, when the gating coefficient g is close to 0, the fused features mainly consist of local sequence features, and the model relies on the sequence context of the decoded characters for character prediction. When the gating coefficient g is close to 1, the fused features mainly consist of global visual context features, and the model relies on global visual information retrieved from the visual feature map for character prediction. When g is between 0 and 1, the two are mixed proportionally. Through this dynamic weighted fusion mechanism, the model can adaptively adjust the relative weights of global and local information at each decoding step according to the current recognition difficulty.

[0121] Obtain fusion features Then, it is input into the feedforward network layer of the decoder for further nonlinear transformation. Then, the probability distribution of the final character is calculated through the linear classification layer (fully connected layer plus softmax), and the character output of the current decoding step is determined accordingly (i.e., the character with the highest probability is taken as the output).

[0122] Step S1100: Decoding Termination Judgment and Recognition Result Generation Stage. Repeat steps S500 to S1000 until the decoding termination condition is met. The decoding termination condition includes: (1) the current decoding step predicts a termination character; (2) the cumulative number of decoding steps exceeds a preset maximum length threshold (e.g., 50). When decoding terminates, the characters output by each decoding step are concatenated in order to form the complete recognition result string of the text region. If the image to be recognized contains multiple text regions, steps S300 to S1100 are performed on each text region respectively, and the recognition results of all text regions are finally output.

[0123] It should be noted that the models involved in the above methods (including the visual feature extraction network, decoder, cohesive lexical units, gating network parameters, etc.) need to be trained on a large amount of labeled data. The main difference between the training phase and the inference phase is:

[0124] During the training phase, cohesive lexical units It consistently participates in both forward and backpropagation, and its parameters are updated normally via gradient descent. Specifically, for each training sample (including the input image and its corresponding ground truth text label), the model performs a complete forward computation (including all steps S100 to S1000 above), calculates the loss between the predicted result and the ground truth label (such as cross-entropy loss), then calculates the gradients of each parameter through backpropagation, and updates all trainable parameters, including cohesive terms, using an optimizer (such as AdamW). The linear transformation matrix of the modulated signal Parameters of gating networks , , And all parameters of the visual feature extraction network and decoder.

[0125] During training, the parameters of the gated network and By using weighted clamping to force a positive constraint, the gating coefficient is ensured. With confidence level Maintaining a negative correlation between them and enhancing the query vector norm A positive correlation is maintained between them. This constraint helps the gating network learn a fusion strategy that aligns with the design expectations.

[0126] During the inference phase (i.e., the practical application phase), the model parameters are fixed, and the cohesive lexical units are no longer updated. The gating coefficient g is determined by the analytical function based on the confidence level of the current decoding step. and Enhanced query vector norm The weights for fusion of global visual context features and local sequence features are calculated in real time and used to adaptively control the fusion. Since backpropagation and parameter updates are not required during the inference phase, computational efficiency is high, meeting the practical application requirements of power engineering inventory parsing.

[0127] It should be further noted that, in one specific implementation, the decoder is a transformer-based decoder, which includes parallel self-attention branches and cross-attention branches. The self-attention branches and cross-attention branches are executed in parallel during forward computation, and their outputs are weighted and fused by gating coefficients before being jointly input into subsequent feedforward network layers.

[0128] The self-attention branch uses causal masks to limit the visibility of future characters, ensuring the correctness of autoregressive decoding. Simultaneously, the self-attention branch employs Rotated Position Encoding (RoPE) to maintain the temporal order of the sequence.

[0129] The cross-attention branch employs learnable 2D absolute position encoding or 2D relative position bias on the visual feature map to enhance the model's ability to perceive spatial relationships among different spatial locations within the visual feature map. The cross-attention branch uses the enhanced query vector as an independent query vector to compute global visual context features with the visual feature map.

[0130] Example 2

[0131] The following example of processing a specific power engineering bill of quantities illustrates the complete processing flow of this invention.

[0132] Suppose the input image is a scanned photograph of an electrical engineering bill of quantities, which contains a text area that reads "10kV cross-linked polyethylene insulated armored power cable".

[0133] Step S100: Obtain the scanned photo as the image to be identified, and perform grayscale conversion and contrast enhancement preprocessing on it.

[0134] Step S200: Input the preprocessed image into the DBNet text detection network to detect a text region. Its location information is a rectangle in the image coordinate system (top left corner coordinates (100, 200), width 800 pixels, height 40 pixels). Based on shape judgment, this region is non-curved horizontal text, and its tilted bounding box (rotation angle close to 0 degrees) is directly determined.

[0135] Step S300: Crop the text region (800×40 pixels) from the original image, input it into the Vision Transformer (ViT) for feature extraction, and obtain a visual feature map with dimensions of 256×25×5 (256-dimensional channels, 25 blocks in the width direction, and 5 blocks in the height direction).

[0136] Step S400: Input the visual feature map into the Transformer decoder, using the start character [BOS] as the initial input, and begin character-by-character decoding. The preset maximum length threshold is set to 50.

[0137] Step S500 (taking the 4th character "k" as an example): Current decoding step t=4 (the characters "1", "0", and "k" have been decoded), the decoder outputs the hidden state. The classification layer predicts the probability distribution of the current character. The probability of the character "V" is 0.45, the probability of the character "k" is 0.30, and the probabilities of the remaining characters are dispersed, with the maximum probability value C=0.45. Since C=0.45 is lower than a preset threshold (e.g., 0.7), the model's prediction of this character is not sufficiently confident.

[0138] Step S600: Obtain pre-stored cohesive lexical units (Dimension 256), hide state Through linear transformation Obtain the modulated signal Calculate the enhanced query vector (Dimension 256).

[0139] Step S700: ... As a query vector, cross-attention is calculated with the visual feature map. The attention weights are concentrated on the spatial location in the visual feature map related to "kV" (because we are currently decoding the word "k", and "kV" is a common unit in power engineering lists; the cohesive lexical unit has learned this prior during training), thus obtaining the global visual context features. .

[0140] Step S800: In parallel, the decoder's self-attention module performs self-attention computation with causal masking on the decoded "1", "0", and "k", and outputs local sequence features. This feature encodes the sequence context information of the prefix "10k".

[0141] Step S900: Calculation The L2 norm was obtained and then normalized. Set C=0.45 and Input the gated network, and use the analytical function to calculate the gate control coefficient g=0.72 (g is biased towards 1 due to the low confidence level).

[0142] Step S1000: Calculate fusion features Since g is relatively large, the fused features are mainly composed of global visual context features. The fused features are input into the FFN and classification layer to recalculate the character probability distribution. At this point, thanks to the guidance of "kV" related information in the global visual context features, the probability of the character "V" increases to 0.78, and the model outputs "V" as the current character.

[0143] Step S1100: Continue decoding the subsequent characters "cross-linked polyethylene insulated armored power cable" until a terminator is predicted or the maximum length is reached. Finally, concatenate and output the complete string "10kV cross-linked polyethylene insulated armored power cable".

[0144] As can be seen from the above process, when the model is not confident enough in predicting the character "k" (confidence level C=0.45), the gating mechanism of this invention automatically increases the fusion weight of the global visual context features (g=0.72), enabling the model to retrieve the global context information of the common power unit "kV" from the visual feature map by using cohesive lexical units, thereby assisting in making the correct character judgment. When predicting simple characters (such as the common numbers "1" and "0"), the confidence level is higher, the gating coefficient g is smaller, and the model mainly relies on local sequence features for fast decoding, avoiding unnecessary computational overhead.

[0145] Furthermore, compared to existing technologies, this invention, in the scenario of long text recognition for power engineering bills, can automatically increase the fusion weight of global visual context features when the model encounters characters with low prediction confidence due to deformation, ambiguity, or other reasons. This allows the model to retrieve global context information related to the current character from the visual feature map using cohesive lexical units, assisting in making correct character judgments. This mechanism avoids the shortcomings of existing technologies that "indiscriminately introduce all characters into the global context" or "only perform post-processing corrections after recognition," achieving refined and adaptive control of the degree of fusion between global and local features.

[0146] Meanwhile, since the cohesive lexical units are learnable parameter vectors, they can learn common character combination patterns in power engineering terminology (such as "kV" and "cross-linked polyethylene") through a large number of power engineering inventory samples during training. This allows them to provide more domain-specific global contextual support for low-confidence characters in actual recognition. This domain-adaptive capability is not possessed by existing general text recognition methods.

[0147] Furthermore, since the gating coefficient only increases the weight of global features when the confidence level is low and keeps the weight of global features small when the confidence level is high, this invention ensures that the model maintains high decoding efficiency in the recognition of most simple characters, and introduces additional global context calculation only when necessary, thus achieving a good balance between recognition accuracy and computational efficiency.

[0148] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for parsing power engineering bill of quantities based on image recognition, characterized in that, include: Obtain the image to be identified from the list of power engineering projects; The image to be identified is subjected to text region detection to obtain at least one text region; Feature extraction is performed on the text region to obtain a visual feature map of the text region; The visual feature map is input into the decoder for character-by-character decoding. During the decoding process, for the current decoding step, the hidden state of the current decoding step is obtained, and the current character and its confidence are predicted based on the hidden state, and the cohesive word is obtained; wherein, the cohesive word is a learnable parameter vector and is pre-stored in the trainable parameter set of the decoder. The cohesive lexical is used as the base query vector, the hidden state is mapped to a modulation signal through a linear transformation, and the modulation signal is added to the base query vector to generate an enhanced query vector. Parallel execution uses the enhanced query vector as the query vector and performs cross-attention calculation with the visual feature map to obtain global visual context features; The local sequence features of the current decoding step are obtained; wherein the local sequence features are the features output by the self-attention module of the decoder, which are used to characterize the context dependencies of the currently decoded character sequence. The confidence level and the norm of the enhanced query vector are used as scalar inputs and input into the gating network to obtain the gating input vector. The gated input vector is input into a gated network to generate gated coefficients; wherein the gated coefficients are negatively correlated with the confidence level. According to the gating coefficient, the global visual context features and the local sequence features are weighted and fused to obtain fused features, and the character output of the current decoding step is determined according to the fused features; wherein, the gating coefficient is used to control the weight of the global visual context features during fusion, and the lower the confidence level and the larger the norm of the enhanced query vector, the greater the weight of the global visual context features; Based on the character output of each decoding step, the recognition result of the text region is generated.

2. The method according to claim 1, characterized in that, The step of generating an enhanced query vector based on the hidden state and the cohesive tokens includes: The hidden state is mapped to a modulation signal through a linear transformation; the modulation signal is added element by element to the cohesive word to obtain the enhanced query vector; The cohesive lexical units serve as the basic components of the enhanced query vector and are updated through backpropagation during training to learn the common preferences of global contextual retrieval.

3. The method according to claim 1, characterized in that, The gating network generates gating coefficients based on a preset analytical function. The analytical function is: ； in, The confidence level is... To enhance query vectors L2 norm after layer normalization; , , These are learnable scalar parameters, and and During training, weights are clamped to force positive values; the weighted fusion is performed according to the following formula: Fusion features = (1- )×Local sequence features+ ×Global visual context features.

4. The method according to claim 1, characterized in that, The global visual context features are aligned with the local sequence features in the feature dimension; The decoder is a transformer-based decoder, which includes parallel self-attention branches and cross-attention branches; The self-attention branch uses causal masking to limit the visibility of future characters, and the self-attention branch uses rotational position encoding to maintain temporal order; the cross-attention branch uses learnable two-dimensional absolute position encoding or two-dimensional relative position bias for the visual feature map. The cross-attention branch uses the enhanced query vector as an independent query vector and calculates the global visual context features with the visual feature map; The self-attention branch and the cross-attention branch are executed in parallel during forward computation; their outputs are weighted and fused by the gating coefficients and then jointly input into the subsequent feedforward network layer.

5. The method according to claim 1, characterized in that, The method also includes a training phase and an inference phase: During the training phase, the cohesive lexical units always participate in forward computation and backward propagation, and their parameters are updated normally through gradient descent. During the inference phase, the gating coefficients are calculated in real time by the analytical function to adaptively control the fusion weights of the global visual context features and the local sequence features.

6. The method according to claim 1, characterized in that, The step of performing text region detection on the image to be identified to obtain at least one text region includes: Feature extraction is performed on the image to be identified to obtain an image feature map; The image feature map is input into a differentiable binarization segmentation network or a segmentation-based text detection network to obtain a text region probability map and a threshold map. Based on the text region probability map and the threshold map, an approximate binary mask is generated through a differentiable binarization operation; Perform connected component analysis or contour detection on the approximate binarized mask to obtain the initial text region; The initial text region is fitted with a polygon, and it is determined whether the fitted polygon is a curved text region. If it is determined to be a curved region, the curved text region is corrected to a horizontal rectangular region by thin plate spline transformation or perspective transformation. If it is determined to be a non-curved region, its tilted bounding box is directly determined. Based on the corrected horizontal rectangular area or the tilted bounding box, determine the final position information of the at least one text area to accommodate the tilted table lines or curved text in the power engineering list.

7. The method according to claim 1, characterized in that, The calculation of the confidence score for the current character prediction includes: Obtain the probability distribution of the output of the current decoding step; The input confidence C of the gated network adopts the maximum original probability value of the probability distribution; The probability distribution is calibrated to obtain the calibrated probability distribution.

8. The method according to claim 4, characterized in that, The visual feature map is obtained by extracting features from the text region through a convolutional neural network or a visual transformer; the character-by-character decoding includes character prediction through the self-attention mechanism and cross-attention mechanism of the decoder, and the decoding process is terminated when the cumulative number of decoding steps exceeds the preset maximum length threshold or when a terminator is predicted.

9. An electronic device, characterized in that: The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the program, it implements the power engineering inventory parsing method based on image recognition as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the power engineering inventory parsing method based on image recognition as described in any one of claims 1 to 8.

Images