Method and system for extracting information from power communication construction drawings based on large models

Abstract

This application discloses a method and system for extracting information from power communication construction drawings based on a large model, including: obtaining an image of the power communication construction drawing to be identified and preprocessing it; using a constructed and trained CRNN model to perform text recognition on the preprocessed construction drawing image and post-processing the recognized text, and then converting the post-processed text into supervisory instruction data; fine-tuning the large model based on the supervisory instruction data to obtain a fine-tuned large model; and using the fine-tuned large model to extract key information from the text identified based on the CRNN model. This application fine-tunes the large model and combines it with computer vision technology to achieve automated, high-precision recognition of key information in power communication construction drawings, improving recognition accuracy and efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric power communication construction, and in particular relates to a method and system for extracting electric power communication construction drawing information based on a large model. Background Art

[0002] Power communication construction drawings contain a large amount of complex information, such as equipment layout, line direction, connection relationships, etc. Traditional manual identification methods are not only time-consuming and labor-intensive, but also prone to errors. The recognition accuracy and efficiency need to be improved.

[0003] Prior art document 1 (CN118015648A) discloses a drawing information extraction method, device, equipment and storage medium, including obtaining a drawing to be processed; extracting graphic element information from the drawing to be processed based on a pre-trained graphic element extraction model; extracting relationship information between graphic elements in the drawing to be processed based on the graphic element information and a pre-trained relationship extraction model; and determining the component name corresponding to each graphic element based on the graphic element information, the relationship information and a pre-trained component recognition model.

[0004] Prior art document 2 (CN111860348A) discloses a weakly supervised OCR recognition method for power drawings based on deep learning, which includes using a pre-trained text detection model to detect the image to be recognized and predicting the text area box at the entire word level; performing text recognition on the predicted text area box, using character segmentation to obtain single character text for vertical text, and directly using text lines for horizontal text, and then recognizing it through the CNN+BiLSTM+CTC model; post-processing the recognition results, and judging and modifying the results based on prior knowledge to improve the accuracy.

[0005] However, the disadvantage of prior art document 1 is that while convolutional neural networks are used to extract network element information, CNNs have a weak understanding of global context. The convolutional layers in AlexNet use large convolution kernels and strides, which may lose some fine-grained local features, and the extraction effect may be poor, especially for fine graphics.

[0006] The shortcoming of prior art document 2 is that, while BiLSTM can capture bidirectional dependencies in sequences, its ability to capture long-distance dependencies gradually weakens when processing long sequences. This may cause the model to fail to accurately understand the overall semantics of the text, thus affecting recognition accuracy. Summary of the Invention

[0007] In order to address the deficiencies in the prior art, the present invention provides a method and system for extracting information from power communication construction drawings based on a large model. By constructing an efficient large model and combining it with computer vision technology, the method realizes automated and high-precision recognition of key information in power communication construction drawings, thereby improving recognition accuracy and efficiency.

[0008] The present invention adopts the following technical solutions.

[0009] A first aspect of the present invention provides a method for extracting information from power communication construction drawings based on a large model, comprising:

[0010] S1: Acquire the image of the power communication construction drawing to be identified and perform preprocessing;

[0011] S2: Use the constructed and trained CRNN model to perform text recognition on the construction drawing images pre-processed in S1 and post-process the recognized text, and then convert the post-processed text into supervision instruction data;

[0012] S3: Fine-tune the large model based on the supervised instruction data to obtain a fine-tuned large model;

[0013] S4: Use the fine-tuned large model to extract key information from the text recognized by the CRNN model.

[0014] Optionally, in S1, the preprocessing includes part or all of binarization, grayscale, denoising, and rotation correction.

[0015] Optionally, in S2, training the CRNN model includes:

[0016] Acquire various types of electric power communication construction drawing images, annotate each character in each electric power communication construction drawing image, and obtain original electric power communication construction drawing images and annotated electric power communication construction drawing images;

[0017] Perform image preprocessing on the original power communication construction drawing images and the annotated power communication construction drawing images to form a data set, and divide the data set into a training set, a test set, and a validation set according to a preset ratio;

[0018] Input the training set into the constructed CRNN model for training, and construct the CRNN model loss function based on the output results of the CRNN model and the training labels;

[0019] Use the validation set to judge the performance of the CRNN model on unseen data through the first evaluation indicator during the CRNN model training process, adjust the hyperparameters according to the performance, and obtain the optimal CRNN model;

[0020] The test set is used to evaluate the recognition accuracy of the trained CRNN model based on the second evaluation indicator.

[0021] Optionally, perform image preprocessing on the power communication construction drawings, including:

[0022] Convert the original image of the electric power communication construction drawing into a grayscale image, convert the grayscale image into a binary image, and then remove the noise in the binary image;

[0023] Use edge algorithms to identify straight lines in the denoised binary image, use Hough transform to extract line parameters from the denoised binary image, and estimate the tilt angle of the original image of the power communication construction drawing based on the line parameters;

[0024] The original image of the electric power communication construction drawing is rotationally corrected according to the estimated tilt angle so that the text lines in the electric power communication construction drawing are aligned.

[0025] Optionally, construct the CRNN model loss function according to the following formula:

[0026]

[0027] in, is the loss value, is the target label sequence All permutations of is the time step The feature vector corresponding to the preprocessed construction drawing image, is the time step The predicted label, For in time The predicted label is The probability of , T is the length of the input sequence, is the target label sequence.

[0028] Optionally, during the CRNN model training process, the validation set is used to determine the performance of the CRNN model on unseen data using a first evaluation metric, and hyperparameters are adjusted based on the performance to obtain the optimal CRNN model, including:

[0029] Constructing an objective function based on the first evaluation indicator and its corresponding weight;

[0030] Randomly sample in the hyperparameter space to construct multiple CRNN models. Use the validation set to train multiple CRNN models to obtain the objective functions corresponding to different hyperparameter combinations, and sort the objective functions corresponding to different hyperparameter combinations from small to large.

[0031] Based on the Bayesian optimization algorithm, the models corresponding to the hyperparameter combinations corresponding to the top k objective functions are trained, and the model corresponding to the hyperparameter combination corresponding to the minimum objective function is selected as the optimal CRNN model.

[0032] Optionally, the constructed CRNN model includes a CNN layer, a feature fusion layer, a two-head self-attention mechanism, a bidirectional LSTM, and a CTC layer;

[0033] The CNN layer is used to extract features from the image text lines of the power communication construction drawings to obtain a feature map, wherein the features in the feature map include the shape of the characters, the edges of the characters, the texture of the characters, the layout between the characters, and the structure of the characters; the CNN layer includes multiple convolutional layers;

[0034] The feature fusion layer is connected to multiple convolutional layers to fuse the feature maps extracted by the multiple convolutional layers;

[0035] The fused feature map is converted into a feature sequence and then input into the dual-head attention mechanism. The output of the two heads is weighted summed to obtain the final feature sequence.

[0036] The forward LSTM in the bidirectional LSTM reads the forward information of the final feature sequence, and the reverse LSTM reads the reverse information of the final feature sequence. The outputs of the forward LSTM and the reverse LSTM are concatenated at each time step to obtain the feature vector of each time step and output the probability distribution of all characters.

[0037] The CTC layer aligns the sequence according to the probability distribution of the characters and calculates the loss, and the final character sequence is extracted from the output of the CTC layer through the decoding strategy.

[0038] Optionally, the feature maps extracted from multiple convolutional layers are fused in the feature fusion layer according to the following formula:

[0039]

[0040] in, Indicates the Feature map after layer fusion, Indicates the The feature map of the layer, represents a channel attention mechanism, For the general The processed feature map is spliced ​​with the feature map of the previous layer. Indicates the maximum pooling operation on the spliced ​​feature map. , H and W represent the height and width of the feature map respectively, and C is the number of channels;

[0041] Among them, the channel attention mechanism dynamically adjusts the weight of the channel by learning the importance of each channel, and combines the weight of each channel obtained with the first Feature map of the layer Multiply to obtain the weighted feature map .

[0042] Optionally, the feature fusion layer includes multiple fully connected layers, and the channel attention mechanism dynamically adjusts the weight of each channel by learning the importance of each channel, specifically including:

[0043] Use channel attention mechanism to process the feature map of layer i , perform global average pooling on each channel to obtain the channel description vector;

[0044] The channel description vector is compressed through the first fully connected layer, and then the compressed channel description vector is input into the ReLU activation function;

[0045] The output of the ReLU activation function is input into the second fully connected layer, and the output of the second fully connected layer is passed through the sigmoid activation function to obtain the weight of each channel.

[0046] Optionally, the large model is fine-tuned using the improved LoRA technology. During the fine-tuning process, Hadamard products are performed using sub-matrices of matrix A and matrix B. The parameter matrix is ​​obtained based on the Hadamard product, and the number of Hadamard products is dynamically adjusted.

[0047] Optionally, during fine-tuning, a Hadamard product is performed using submatrices of matrix A and matrix B, a parameter matrix is ​​obtained based on the Hadamard product, and the number of Hadamard products is dynamically adjusted, including:

[0048] Slice the matrix B by columns to obtain the submatrices of B. When r / s is an integer, each submatrix of B contains s columns, and r / s submatrices of B are obtained. When r / s is not an integer, r / s+1 submatrices of B are obtained. Slice the matrix A by rows to obtain the submatrix of A. When r / s is an integer, each submatrix of A contains s rows, and r / s submatrices of A are obtained. When r / s is not an integer, r / s+1 submatrices of A are obtained, where r is the rank of the A matrix and the B matrix, and s is the step size.

[0049] Perform matrix product of the corresponding sub-matrices of B and A:

[0050]

[0051] in, , Indicates the first sub-matrices, Indicates the first sub-matrices, N represents the number of sub-matrices;

[0052] N Perform the Hadamard product in order to obtain the parameter matrix:

[0053]

[0054] in, The parameter matrix representing the parameter update during fine-tuning, represents the Hadamard product.

[0055] Optionally, the method further includes:

[0056] The key information extracted from the large model is converted into structured data, and the converted key information and power communication construction drawings are stored in the database in a set format.

[0057] A second aspect of the present invention provides a large-scale model-based power communication construction drawing information extraction system, comprising:

[0058] A data acquisition module, used to obtain images of power communication construction drawings to be identified;

[0059] The drawing recognition module is used to use the constructed and trained CRNN model to perform text recognition on images of power communication construction drawings, post-process the recognized text, and then convert the post-processed text into supervision instruction data;

[0060] A fine-tuning module is used to fine-tune the large model based on the supervision instruction data to obtain a fine-tuned large model;

[0061] The large model information extraction module is used to extract key information from text recognized by the CRNN model using the fine-tuned large model.

[0062] Optionally, the system also includes an output module and a data storage module, wherein the output module is used to convert the key information output by the large model information extraction module into structured data, and the data storage module is used to store the structured data and power communication construction drawings in a set format in a database.

[0063] The third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the method for extracting information from power communication construction drawings based on a large model is implemented.

[0064] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for extracting power communication construction drawing information based on a large model.

[0065] Compared with the prior art, the beneficial effects of the present invention include at least:

[0066] This invention combines visual technology with large-scale modeling to automatically and accurately extract information from power and telecommunications construction drawings, reducing manual recognition errors and improving recognition accuracy. This automated recognition process not only significantly shortens construction preparation time and improves construction efficiency, but also reduces reliance on specialized personnel and lowers labor costs.

[0067] This paper adds a downsampling branch to the convolutional layer to fuse features from different convolutional layers, dynamically adjusts the learning rate, adds a dual-head self-attention mechanism before the BiLSTM, uses more stringent evaluation metrics to assess the drawing recognition effect, and fine-tunes large models using an improved LoRA algorithm. This paper improves feature extraction and multi-scale perception capabilities, enabling the model to more comprehensively capture low-level and high-level image features and adapt to complex drawing recognition tasks. The improved LoRA fine-tuning algorithm enhances the model's expressiveness, while the step size mechanism increases training flexibility and helps the model converge more efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0069] Figure 1 A schematic diagram of the architecture of a large-scale model-based power communication construction drawing information extraction system provided by an embodiment of the present invention;

[0070] Figure 2 A schematic diagram of a large-scale model-based process for extracting information from power communication construction drawings provided by an embodiment of the present invention;

[0071] Figure 3 A schematic diagram of constructing a CRNN model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. The embodiments described in this application are only part of the embodiments of the present invention, not all of them. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0073] Combine Figures 1 to 3 As shown, embodiment 1 of the present invention provides a method for extracting information from power communication construction drawings based on a large model, comprising the following steps:

[0074] S1: Acquire an image of the power communication construction drawing to be identified and perform preprocessing.

[0075] The image format of the power communication construction drawing in the embodiment of the present invention may be JPEG, BMP, or PNG, which is not limited here.

[0076] In an exemplary implementation of an embodiment of the present invention, the drawing information extraction system may obtain drawings to be identified uploaded by a user through a preset drawing upload interface, wherein the drawings to be identified may be multiple drawings of a construction object.

[0077] Optionally, obtaining the image of the power communication construction drawing includes: obtaining the original image of the power communication construction drawing; performing a preprocessing operation on the original image to obtain the image of the power communication construction drawing to be identified, and the preprocessing operation includes part or all of grayscale, binarization, denoising, and rotation correction.

[0078] In the disclosed embodiment, the image quality can be improved by processing the collected original image of the electric power communication construction drawing.

[0079] S2: Use the constructed and trained CRNN model to perform text recognition on the construction drawing graphics preprocessed in S1 and post-process the recognized text, and then convert the post-processed text into supervision instruction data.

[0080] Optionally, the CRNN model includes a feature extraction model and a drawing recognition model. The feature extraction model can be a convolutional neural network (CNN), and the drawing recognition model can be a recurrent neural network (RNN). Both are trained using massive amounts of power communication construction drawing data. The CRNN model can learn features such as Chinese and English characters, symbols, and numbers in construction drawings, enabling it to accurately recognize these characters, symbols, and numbers.

[0081] Specifically, the model is pre-trained using public datasets from other fields to initialize the model parameters. Then, a large number of annotated power communication construction drawing datasets are used to train the pre-trained model. By adjusting the model parameters, optimizing the network structure, and introducing the attention mechanism, the recognition accuracy and generalization ability of the model are improved. The training of the CRNN model in S2 includes:

[0082] S2.1: Acquire various types of electric power communication construction drawing images, annotate each character in each electric power communication construction drawing image, and obtain the original electric power communication construction drawing image and the annotated electric power communication construction drawing image text.

[0083] Specifically, the annotated data is saved in a specific format, which can be json, xml, or txt, and the name is consistent with the drawing name.

[0084] S2.2: Perform image preprocessing on the original power communication construction images, combine the processed images with the corresponding annotated text to form a dataset, and divide the dataset into a training set, a test set, and a validation set according to a preset ratio.

[0085] Specifically, the dataset is divided into training set, test set, and validation set in a ratio of 8:1:1.

[0086] Optional, combined Figure 2 As shown, image preprocessing is performed on the power communication construction drawings, including:

[0087] Convert the original image of the electric power communication construction drawing into a grayscale image, convert the grayscale image into a binary image, and then remove the noise in the binary image;

[0088] Use edge algorithms to identify straight lines in the denoised binary image, use Hough transform to extract line parameters from the denoised binary image, and estimate the tilt angle of the original image of the power communication construction drawing based on the line parameters;

[0089] The original image of the electric power communication construction drawing is rotationally corrected according to the estimated tilt angle so that the text lines in the electric power communication construction drawing are aligned.

[0090] Specifically, image preprocessing of power communication construction drawings includes the following steps:

[0091] S2.2.1: Convert the original image of the power communication construction drawing to be identified into a grayscale image. Grayscale images contain only brightness information, not color information, making subsequent processing more efficient. Grayscale conversion uses a weighted average method.

[0092] S2.2.2: Use adaptive thresholding to convert the grayscale image into a binary image to highlight the text area for subsequent text recognition. Calculate different thresholds based on the local mean of each small block of the image. For example, within a small window, the average of all pixels is used as the threshold for that area.

[0093] S2.2.3: Remove noise from the binarized image and retain useful text information. The denoising method can be a filter-based method.

[0094] S2.2.4: If the image is not horizontal or vertical, correct the tilt of the image so that the text in the image is horizontally aligned. The specific steps are as follows:

[0095] (1) Use the Canny edge detection algorithm to determine the edges in the image by finding the gradient changes in the image, and obtain an edge image in which the edge positions will be marked.

[0096] (2) Use Hough transform to detect straight lines in edge images. In image space, each edge point ( ) can be expressed as a parameterized polar coordinate line equation:

[0097]

[0098] in( is a point in image space, is the distance from the line to the origin, is the tilt angle of the line, which ranges from [0,180]. For each point in the image, traverse all possible , calculate the corresponding Value, count all ( ), when the number reaches a threshold (e.g. 10% of the total number of edge points), it is considered as a straight line in space, and all straight lines in the image are extracted.

[0099] (3) Calculate the inclination angle of the text area and obtain the inclination angles of all straight lines according to (2) , count all The most frequent occurrences The angle at which the text will be tilted.

[0100] (4) Image rotation correction: Use the rotation matrix to rotate the image. The rotation matrix can rotate a point around the center point by a certain angle. The form of the rotation matrix is:

[0101]

[0102] in Rotate the image by the rotation matrix to the angle you want to rotate. Angle returns the text area to a horizontal state.

[0103] S2.3: Input the training set into the constructed CRNN model for training, and construct the CRNN model loss function based on the output results of the CRNN model and the training labels.

[0104] The CRNN model is trained based on the preprocessed training set, and the validation set is used to evaluate the model performance during the training process. Specifically:

[0105] The constructed loss function is used in S2.3 to optimize the model, which can handle the problem of inconsistent lengths of input and output sequences.

[0106]

[0107] in, is the loss value, is the target label sequence All possible alignments of , that is, all possible permutations of the labels, is the time step The feature vector corresponding to the preprocessed construction drawing image is is the time step The predicted label, It's in time The predicted label is The probability of , T is the length of the input sequence, It is a target tag sequence, which can be text, symbols, etc. of time in construction drawings.

[0108] Combine Figure 3 As shown in Figure 2, the CRNN network structure constructed in S2.3 specifically includes CNN layer, feature fusion layer, dual-head self-attention mechanism, bidirectional LSTM and CTC layer; among them,

[0109] The CNN layer is used to extract features from the text lines in the image of the power communication construction drawing, generating a feature map. The key features in the feature map include the shape of the characters, the edges of the characters, the texture characteristics of the characters, the layout and structure between the characters, and other features. The present invention uses four convolutional layers for feature extraction. The input is a preprocessed drawing with dimensions of (H, W, C) = (32, 150, 1). After four convolution operations, the sizes are (3, 3, 64), (3, 3, 128), (3, 3, 256), and (3, 3, 256), respectively.

[0110] The feature fusion layer connects multiple convolutional layers and is used to fuse the feature maps extracted by multiple convolutional layers. Specifically, a downsampling branch is added to the convolutional layer to fuse the feature maps from different convolutional layers, which is expressed as the following formula:

[0111]

[0112] in, Indicates the Feature map after layer fusion, Indicates the The feature map of the layer, represents a channel attention mechanism, , H and W represent the height and width of the feature map respectively, C is the number of channels, and the feature map selects channels that are more helpful for recognition through a channel attention mechanism SE-Layer. The channel refers to the depth dimension of the feature map output by the convolutional layer, thereby reducing the number of unnecessary channels during feature fusion. The maximum pooling operation is performed on the concatenated feature maps to reduce the spatial dimension of the feature maps. The specific steps of feature fusion are described below.

[0113] (1) Use the channel attention mechanism SE-Layer to process the Feature map of the layer , perform global average pooling on each channel to obtain the channel description vector.

[0114]

[0115] Where H and W represent the height and width of the feature map respectively. is the eigenvalue of the i-th row, j-th column, and c-th channel.

[0116] (2) The feature fusion layer includes a fully connected layer, which learns channel weights through a fully connected layer and a nonlinear activation function. The channel description vector is compressed, then passed through the ReLU activation function, and then through a fully connected layer To expand, and finally get the weight of each channel through the sigmoid activation function.

[0117]

[0118] in, is the weight vector of all channels, represents the weight of channel c, is the sigmoid function, and is the weight matrix of the fully connected layer, and z is the matrix describing all channels.

[0119] The channel attention mechanism learns the importance of each channel and dynamically adjusts the channel weights, thereby enhancing channel features that are helpful to the task and suppressing unimportant channel features.

[0120] (3) Combine the learned weights with the original feature map Multiply them together to get the weighted feature map.

[0121]

[0122] in represents the weight of channel c, Represents the characteristics of channel c.

[0123] (4) and Splicing in the channel dimension, we get .

[0124] (5) Perform the maximum pooling operation on the above spliced ​​feature map to reduce the spatial dimension of the feature map and obtain the fused feature map .

[0125] By fusing features from different convolutional layers, the embodiments of the present invention can combine low-level detail information with high-level abstract information, thereby improving the model's ability to understand the overall structure of the data.

[0126] Two-headed self-attention mechanism: The fused sequence is fed into the two-headed attention mechanism, and the outputs of both heads are weighted summed to produce the final feature sequence. This parallel computation by the two attention heads captures global dependencies and enhances the model's ability to model long-term dependencies in the input data. This sequence is then fed into the two-headed self-attention mechanism. Attention scores are calculated for each feature position using the query, key, and value. The two-headed self-attention mechanism captures important contextual information from different perspectives. The weighted sum of the outputs of both heads is used as the final feature.

[0127] Bidirectional LSTM: The forward LSTM reads the forward information of the sequence, and the reverse LSTM reads the reverse information of the sequence. The outputs of the two are spliced ​​at each time step to obtain the feature vector of each time step, and finally output the softmax probability distribution of all characters.

[0128] It can be understood that the bidirectional LSTM belongs to the RNN layer in the CRNN model.

[0129] CTC layer: The CTC layer aligns the sequence and calculates the loss according to the probability distribution of the characters, and extracts the final character sequence from the output of CTC through the decoding strategy.

[0130] The decoding strategy includes one or more of greedy decoding, beam set search decoding, prefix beam set search decoding, Viterbi decoding, and non-autoregressive decoding.

[0131] In the data iterative training process in S2.3, the preprocessed drawings and corresponding label data are input, the training rounds are set to 60 times, the initial learning rate is 0.001, the learning rate is dynamically adjusted, and the learning rate is decayed to half of the original every 6 rounds. The training batch size is 16.

[0132] S2.4: Use the validation set to determine the performance of the CRNN model on unseen data using the first evaluation metric during CRNN model training. Adjust the hyperparameters based on the performance to obtain the optimal CRNN model.

[0133] The first evaluation metric includes a loss function and character accuracy, and reflects the generalization ability of the model.

[0134] In this embodiment, the following objective function can be set according to the first evaluation index:

[0135] ,

[0136] in, 、 Represent the weights of the CRNN model loss function and character accuracy, respectively. represents the CRNN model loss function, In this way, the CRNN model is trained by combining the loss function and the character accuracy to minimize the objective function and obtain the optimal CRNN model, thereby improving the recognition accuracy.

[0137] Specifically, hyperparameters include learning rate, batch size, and number of LSTM hidden units. First, random sampling is performed in the hyperparameter space to obtain the objective functions corresponding to different hyperparameter combinations. These objective functions are then sorted from small to large. Then, based on the Bayesian optimization algorithm, the models corresponding to the hyperparameter combinations corresponding to the first k objective functions are further fine-tuned. The model corresponding to the hyperparameter combination with the minimum objective function is selected as the optimal CRNN model. In this way, by combining random search and the Bayesian optimization algorithm to adjust the hyperparameters of the CRNN model, the optimal CRNN model can be obtained more quickly and accurately.

[0138] S2.5: Use the test set to evaluate the trained CRNN model and evaluate the recognition accuracy of the CRNN model based on the second evaluation indicator.

[0139] The second evaluation index includes character accuracy and text accuracy. The second evaluation index is used to reflect the accuracy of model recognition.

[0140] Specifically, the test set is used to evaluate the performance of the trained model in order to further improve the algorithm, accuracy and generalization ability. Multiple indicators are used, including character accuracy based on edit distance. , text accuracy .

[0141]

[0142] in, Predicted text and real text The edit distance between Represents real text N is the total number of characters in the text.

[0143]

[0144] in, is the predicted text, is the real text, and N is the total number of texts.

[0145] Specifically, if character accuracy exceeds 95% but text accuracy is less than 60%, this indicates that the model is prone to errors in certain characters. You can increase the training samples for specific characters and improve the diversity of the training data. For example, you can use data augmentation techniques to increase the diversity of training data and increase the training weights for these error-prone character samples. If character accuracy is less than 60%, increase the number of LSTM hidden units to improve sequence modeling capabilities.

[0146] In S2, post-processing is to optimize and proofread the recognition results, including removing extra spaces, correcting typos, and other operations to improve the quality of generated supervisory instruction data. Based on the CRNN model, the extracted text is converted into instruction data. The instruction data is in the following format:

[0147] [{"instruction":"Extract the cable naming service information from the text given below and output it in a specific format.",

[0148] "input": " 1. This project is designed based on the access system and preliminary design review opinions, relevant meeting spirit, etc. The design scope of this volume includes the relevant content of optical fiber communication of photovoltaic power stations. The installed capacity of the starting photovoltaic power station is 100MW, and it is connected to the 110kV side of the target substation through a 110kV grid-connected line with a length of 18km. 2. System communication plan 2.1 Optical fiber communication plan (1) Optical cable construction plan Along the 110kV line from the starting photovoltaic power station to the target substation, two 48-core OPGW optical cables with a core of G.652D and a length of 2*18km are installed. ……",

[0149] "output":{

[0150] "A-end site": "Starting PV power station",

[0151] "Z-end site": "target substation"

[0152] "Primary Line Voltage Level": "110kV",

[0153] "Number of cores": "48 cores",

[0154] "Optical Cable Type": "OPGW Cable",

[0155] "Core Model": "G.652D",

[0156] "cable length": "18km"}

[0157] } ]

[0159] S3: Fine-tune the large model based on the supervised instruction data to obtain the fine-tuned large model.

[0160] The process of model fine-tuning involves adjusting the parameters of the model to better suit the needs of a specific task. This process not only improves the performance of the model on specific tasks, but also helps the model understand and generate more accurate text content.

[0161] Optionally, the large model can be fine-tuned using the improved LoRA (Low-Rank Adaptation) technology. The improvement of LoRA is that during the fine-tuning process, the Hadamard product is performed on the sub-matrices of matrix A and matrix B, the parameter matrix is ​​obtained based on the Hadamard product, and the number of Hadamard products is dynamically adjusted.

[0162] Specifically, we use the improved LoRA technology to fine-tune the large model. The process is as follows:

[0163] S3.1: For the pre-trained weight matrix , LoRA limits its update method, that is, the incremental parameter matrix of all parameter fine-tuning Represented as two matrices with smaller parameters and A low-rank approximation of :

[0164]

[0165] in, or Represents the pre-trained weight matrix, whose dimension is , this matrix represents the weights in the original model. and They are the two low-rank matrices introduced in LoRA fine-tuning, and ,rank Much smaller than .

[0166] S3.2: During training, initialize the low-rank adaptation matrices B and A:

[0167] The matrix A is initialized by the Gaussian function, , the matrix B is initialized to all 0, .

[0168] S3.3: Dynamically adjust the number of Hadamard products. Starting from the initial B and A, use their submatrices to perform Hadamard products. To make the number of Hadamard products more flexible, introduce a step size s, for example, s = 2, r = 8.

[0169] Matrix B is sliced ​​by columns to obtain submatrices. When r / s is an integer, each submatrix contains s columns, and r / s submatrices are obtained. When r / s is not an integer, r / s+1 submatrices are obtained. The number of columns of the last submatrix is ​​less than s. The number of submatrices is recorded as N. Similarly, matrix A is sliced ​​by rows to obtain the same number of submatrices.

[0170] Perform matrix product of the corresponding submatrices of B and A to obtain:

[0171]

[0172] in, Indicates the first sub-matrices, Indicates the first sub-matrices, , N represents the number of sub-matrices. Perform the Hadamard product in sequence to obtain the final parameter matrix:

[0173]

[0174] S3.4: Given Input , output after adding LoRA :

[0175]

[0176] in, represents the pre-trained weight matrix, The parameter matrix representing the parameter update during fine-tuning, represents the Hadamard product.

[0177] S4: Use the fine-tuned large model to extract key information from the text recognized by the CRNN model.

[0178] Specifically, when receiving new power communication construction drawings, the text within them must first be recognized using a CRNN model. This recognized text is then passed as input to a fine-tuned large model. The large model understands the user's intent based on a prompt template and extracts key information. Based on this extracted information, the required text output is generated. This process not only improves information processing efficiency but also ensures that the generated text is accurate and meets practical requirements.

[0179] Specifically, the prompt template can be: Please extract the optical cable naming service information from the following text, including the A-end site, the Z-end site, the primary line voltage level, the number of cores, the optical cable type, the fiber core model, and the optical cable length. If no specific information is found in the provided text, the value of the information should be set to None and output in JSON format.

[0180] Specifically, the key information of the power communication construction drawings includes construction project information and business information. The construction project information includes the construction project name and construction project address; the business information includes optical cable naming, networking information, automation or scheduling IAD information, protection business information, telephone number and construction site information.

[0181] Specifically, the key information is explained as follows:

[0182] (1) Optical cable naming includes basic information such as the type, number of cores, length, name, and the names of the starting and ending sites. This part is crucial for the configuration and management of optical cables, ensuring that the cable identification is consistent with the actual physical location.

[0183] (2) Networking information involves the overall structure of the network, including the names of the starting and ending sites, the usage of the equipment at the starting and ending sites, the transmission system, the usage of optical cables, etc., to help design and implement network connections.

[0184] (3) Automation / scheduling IAD information mainly includes basic information such as transmission system, starting and ending site names, transmission channel type, and intermediate channel resource usage.

[0185] (4) Protection business information involves the protection measures in the power communication system, mainly including information such as protection device model, transmission system, single-port / dual-port, and transmission channel type. This information is very important for ensuring the safety and reliability of the system.

[0186] (5) The telephone number shall include the starting and ending site names, names, application instructions and other information.

[0187] S5: Convert the key information extracted from the large model into structured data, and store the converted key information and power communication construction drawings in the database in a set format.

[0188] Specifically, power and telecommunications construction drawings are typically stored in image format. To ensure the integrity and accessibility of these files, the drawing files are first stored in the file system. Each file should be assigned a unique file name and path to facilitate subsequent retrieval and management. Tables are created directly to store the extracted structured data.

[0189] By converting the unstructured text information output by large models into structured data, it is easier to store and manage. The power communication construction drawings and the extracted structured data can be stored in the database efficiently and securely.

[0190] Combine Figure 1 As shown, embodiment 2 of the present invention provides a large-model-based power communication construction drawing information extraction system, which runs the method described in embodiment 1. The system includes:

[0191] A data acquisition module, used to obtain images of power communication construction drawings to be identified;

[0192] The drawing recognition module is used to use the constructed and trained CRNN model to perform text recognition on images of power communication construction drawings, post-process the recognized text, and then convert the post-processed text into supervision instruction data;

[0193] A fine-tuning module is used to fine-tune the large model based on the supervision instruction data to obtain a fine-tuned large model;

[0194] The large model information extraction module is used to extract key information from text recognized by the CRNN model using the fine-tuned large model.

[0195] Optionally, the system also includes an output module and a data storage module, wherein the output module is used to convert the key information output by the large model information extraction module into structured data, and the data storage module is used to store the structured data and power communication construction drawings in a set format in a database.

[0196] Optionally, the system further includes a preprocessing module, which preprocesses the original image of the power communication construction drawing, including part or all of binarization, grayscale, denoising, and rotation correction.

[0197] Embodiment 3 of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the method for extracting information from power communication construction drawings based on a large model described in embodiment 1 is implemented.

[0198] Embodiment 4 of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements a large model-based power communication construction drawing information extraction method according to embodiment 1.

[0199] Compared with the prior art, the beneficial effects of the present invention include at least:

[0200] This invention combines visual technology with large-scale modeling to automatically and accurately extract information from power and telecommunications construction drawings, reducing manual recognition errors and improving recognition accuracy. This automated recognition process not only significantly shortens construction preparation time and improves construction efficiency, but also reduces reliance on specialized personnel and lowers labor costs.

[0201] This paper adds a downsampling branch to the convolutional layer to fuse features from different convolutional layers, dynamically adjusts the learning rate, adds a dual-head self-attention mechanism before the BiLSTM, uses more stringent evaluation metrics to evaluate the drawing recognition effect, and fine-tunes large models using an improved LoRA. This paper improves feature extraction and multi-scale perception capabilities, enabling the model to more comprehensively capture low-level and high-level image features and adapt to complex drawing recognition tasks. The improved LoRA fine-tuning algorithm enhances the model's expressiveness, while the step size mechanism increases training flexibility and helps the model converge more efficiently.

[0202] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0203] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0204] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punched card or raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0205] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0206] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, the state information of the computer-readable program instructions is used to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), so that the electronic circuit can execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for extracting information from power communication construction drawings based on a large model, characterized in that: include: S1: Acquire the image of the power communication construction drawing to be identified and perform preprocessing; S2: Use the constructed and trained CRNN model to perform text recognition on the construction drawing images preprocessed in S1 and post-process the recognized text, and then convert the post-processed text into supervision instruction data. The CRNN model adds a downsampling branch on the basis of the convolutional layer to fuse the feature maps from different convolutional layers. S3: Fine-tune the large model based on the supervisory instruction data to obtain a fine-tuned large model; and use the improved LoRA technology to fine-tune the large model. During the fine-tuning process, use the sub-matrices of matrix A and matrix B to perform Hadamard products, obtain the parameter matrix based on the Hadamard product, and dynamically adjust the number of Hadamard products; S4: Use the fine-tuned large model to extract key information from the text recognized by the CRNN model; the constructed CRNN model includes a CNN layer, a feature fusion layer, a two-head self-attention mechanism, a bidirectional LSTM, and a CTC layer; The CNN layer is used to extract features from the image text lines of the power communication construction drawings to obtain a feature map, wherein the features in the feature map include the shape of the characters, the edges of the characters, the texture of the characters, the layout between the characters, and the structure of the characters; the CNN layer includes multiple convolutional layers; The feature fusion layer is connected to multiple convolutional layers to fuse the feature maps extracted by the multiple convolutional layers; The fused feature map is converted into a feature sequence and then input into the dual-head attention mechanism. The output of the two heads is weighted summed to obtain the final feature sequence. The forward LSTM in the bidirectional LSTM reads the forward information of the final feature sequence, and the reverse LSTM reads the reverse information of the final feature sequence. The outputs of the forward LSTM and the reverse LSTM are concatenated at each time step to obtain the feature vector of each time step and output the probability distribution of all characters. The CTC layer aligns the sequence according to the probability distribution of the characters and calculates the loss, and the final character sequence is extracted from the output of the CTC layer through the decoding strategy.

2. The method for extracting information from power communication construction drawings based on a large model according to claim 1 is characterized in that: In S1, the preprocessing includes part or all of binarization, grayscale, denoising, and rotation correction.

3. The method for extracting information from power communication construction drawings based on a large model according to claim 1, characterized in that: In S2, training the CRNN model includes: Acquire various types of electric power communication construction drawing images, annotate each character in each electric power communication construction drawing image, and obtain original electric power communication construction drawing images and annotated electric power communication construction drawing images; Perform image preprocessing on the original power communication construction drawing images and the annotated power communication construction drawing images to form a data set, and divide the data set into a training set, a test set, and a validation set according to a preset ratio; Input the training set into the constructed CRNN model for training, and construct the CRNN model loss function based on the output results of the CRNN model and the training labels; Use the validation set to judge the performance of the CRNN model on unseen data through the first evaluation indicator during the CRNN model training process, adjust the hyperparameters according to the performance, and obtain the optimal CRNN model; The test set is used to evaluate the recognition accuracy of the trained CRNN model based on the second evaluation indicator.

4. The method for extracting information from power communication construction drawings based on a large model according to claim 3 is characterized in that: Image preprocessing of power communication construction drawings, including: Convert the original image of the electric power communication construction drawing into a grayscale image, convert the grayscale image into a binary image, and then remove the noise in the binary image; Use edge algorithms to identify straight lines in the denoised binary image, use Hough transform to extract line parameters from the denoised binary image, and estimate the tilt angle of the original image of the power communication construction drawing based on the line parameters; The original image of the electric power communication construction drawing is rotationally corrected according to the estimated tilt angle so that the text lines in the electric power communication construction drawing are aligned.

5. The method for extracting information from power communication construction drawings based on a large model according to claim 3 is characterized in that: Construct the CRNN model loss function according to the following formula: in, is the loss value, is the target label sequence All permutations of is the time step The feature vector corresponding to the preprocessed construction drawing image, is the time step The predicted label, For in time The predicted label is The probability of , T is the length of the input sequence, is the target label sequence.

6. The method for extracting information from power communication construction drawings based on a large model according to claim 3 is characterized in that: During the CRNN model training process, the validation set is used to determine the performance of the CRNN model on unseen data using the first evaluation metric. Hyperparameters are adjusted based on the performance to obtain the optimal CRNN model, including: Constructing an objective function based on the first evaluation indicator and its corresponding weight; Randomly sample in the hyperparameter space to construct multiple CRNN models. Use the validation set to train multiple CRNN models to obtain the objective functions corresponding to different hyperparameter combinations, and sort the objective functions corresponding to different hyperparameter combinations from small to large. Based on the Bayesian optimization algorithm, the models corresponding to the hyperparameter combinations corresponding to the top k objective functions are trained, and the model corresponding to the hyperparameter combination corresponding to the minimum objective function is selected as the optimal CRNN model.

7. The method for extracting information from power communication construction drawings based on a large model according to claim 1, characterized in that: The feature maps extracted from multiple convolutional layers are fused in the feature fusion layer according to the following formula: in, Indicates the Feature map after layer fusion, Indicates the The feature map of the layer, represents a channel attention mechanism, For the general The processed feature map is spliced ​​with the feature map of the previous layer. Indicates the maximum pooling operation on the spliced ​​feature map. , H and W represent the height and width of the feature map respectively, and C is the number of channels; Among them, the channel attention mechanism dynamically adjusts the weight of the channel by learning the importance of each channel, and combines the weight of each channel obtained with the first Feature map of the layer Multiply to obtain the weighted feature map .

8. The method for extracting information from power communication construction drawings based on a large model according to claim 7, characterized in that: The feature fusion layer includes multiple fully connected layers. The channel attention mechanism dynamically adjusts the weight of each channel by learning the importance of each channel, specifically including: Use channel attention mechanism to process the feature map of layer i , perform global average pooling on each channel to obtain the channel description vector; The channel description vector is compressed through the first fully connected layer, and then the compressed channel description vector is input into the ReLU activation function; The output of the ReLU activation function is input into the second fully connected layer, and the output of the second fully connected layer is passed through the sigmoid activation function to obtain the weight of each channel.

9. The method for extracting information from power communication construction drawings based on a large model according to claim 1, characterized in that: During the fine-tuning process, the Hadamard product is performed using the submatrices of matrix A and matrix B. The parameter matrix is ​​obtained based on the Hadamard product, and the number of Hadamard products is dynamically adjusted, including: Slice the matrix B by columns to obtain the submatrices of B. When r / s is an integer, each submatrix of B contains s columns, and r / s submatrices of B are obtained. When r / s is not an integer, r / s+1 submatrices of B are obtained. Slice the matrix A by rows to obtain the submatrix of A. When r / s is an integer, each submatrix of A contains s rows, and r / s submatrices of A are obtained. When r / s is not an integer, r / s+1 submatrices of A are obtained, where r is the rank of the A matrix and the B matrix, and s is the step size. Perform matrix product of the corresponding sub-matrices of B and A: in, , Indicates the first sub-matrices, Indicates the first sub-matrices, N represents the number of sub-matrices; N Perform the Hadamard product in order to obtain the parameter matrix: in, The parameter matrix representing the parameter update during fine-tuning, represents the Hadamard product.

10. The method for extracting information from power communication construction drawings based on a large model according to any one of claims 1 to 9, characterized in that: The method further comprises: The key information extracted from the large model is converted into structured data, and the converted key information and power communication construction drawings are stored in the database in a set format.

11. A large-scale model-based power communication construction drawing information extraction system, characterized in that: include: A data acquisition module, used to obtain images of power communication construction drawings to be identified; The drawing recognition module uses a constructed and trained CRNN model to perform text recognition on images of power communication construction drawings, post-process the recognized text, and then convert the post-processed text into supervisory instruction data. The CRNN model adds a downsampling branch to the convolutional layer to fuse feature maps from different convolutional layers. The fine-tuning module is used to fine-tune the large model based on the supervision instruction data to obtain the fine-tuned large model; and the improved LoRA technology is used to fine-tune the large model. During the fine-tuning process, the sub-matrices of matrix A and matrix B are used to perform Hadamard products, and the parameter matrix is ​​obtained according to the Hadamard product. The number of Hadamard products is dynamically adjusted; The large model information extraction module is used to extract key information from the text recognized by the CRNN model using the fine-tuned large model; Among them, the constructed CRNN model includes CNN layer, feature fusion layer, dual-head self-attention mechanism, bidirectional LSTM and CTC layer; The CNN layer is used to extract features from the image text lines of the power communication construction drawings to obtain a feature map, wherein the features in the feature map include the shape of the characters, the edges of the characters, the texture of the characters, the layout between the characters, and the structure of the characters; the CNN layer includes multiple convolutional layers; The feature fusion layer is connected to multiple convolutional layers to fuse the feature maps extracted by the multiple convolutional layers; The fused feature map is converted into a feature sequence and then input into the dual-head attention mechanism. The output of the two heads is weighted summed to obtain the final feature sequence. The forward LSTM in the bidirectional LSTM reads the forward information of the final feature sequence, and the reverse LSTM reads the reverse information of the final feature sequence. The outputs of the forward LSTM and the reverse LSTM are concatenated at each time step to obtain the feature vector of each time step and output the probability distribution of all characters. The CTC layer aligns the sequence according to the probability distribution of the characters and calculates the loss, and the final character sequence is extracted from the output of the CTC layer through the decoding strategy.

12. The large model-based power communication construction drawing information extraction system according to claim 11, characterized in that: The system also includes an output module and a data storage module. The output module is used to convert the key information output by the large model information extraction module into structured data. The data storage module is used to store the structured data and power communication construction drawings in a set format in a database.

13. An electronic device comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method for extracting power communication construction drawing information based on a large model according to any one of claims 1-10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for extracting information from power communication construction drawings based on a large model as described in any one of claims 1 to 10 are implemented.

Images