A multimodal model fine-tuning method and system for equipment nameplate image text extraction

Abstract

The present invention proposes a multimodal model fine-tuning method and system for equipment nameplate image text extraction, including: data collection and intelligent annotation; text information extraction data set generation based on reinforcement learning; preliminary training of multi-level cross-modal large models; fine-tuning of adaptive multimodal large models; model reasoning and verification. This method and system combines a variety of advanced technologies, including multi-angle shooting, reinforcement learning, generative adversarial networks, multi-level feature extraction, cross-attention mechanism and adaptive optimization algorithm, which significantly improves the efficiency and accuracy of power equipment nameplate image text information extraction. The system can not only process large amounts of data quickly and efficiently, but also ensure high performance and high reliability in practical applications through continuous optimization and verification, and has strong practical value and promotion prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment image processing, and in particular to a multimodal model fine-tuning method and system for equipment nameplate image text extraction. Background Art

[0002] With the progress and development of social economy, the electricity load is growing, and the types and number of new power equipment are also increasing. At present, the transcription method of equipment nameplate information is still highly dependent on manpower, which significantly exposes problems such as low efficiency, high labor cost and information errors. Therefore, it is increasingly important to study an efficient and accurate automatic extraction technology for power equipment nameplate text information.

[0003] Existing related technologies mainly rely on image text detection and recognition technology (OCR), and the process generally includes three parts: training text detection models, text recognition models, and key information extraction. For the various types of power equipment nameplates, training targeted OCR models requires location and content annotation of the text. After the model is trained, it can detect multiple paragraphs of text information on the nameplate and extract key information from it through design rules.

[0004] However, this multi-step processing method has significant problems. First, given the wide variety of image quality, OCR model performance, and nameplate types, it becomes extremely complicated to design effective rules, and recognition results are prone to noise and errors. In addition, each link may introduce errors, which gradually accumulate in the processing chain, resulting in inaccurate final recognition results. At the same time, since the text needs to be annotated one by one, the annotation workload is large and time-consuming. These problems directly affect recognition efficiency and accuracy, increase labor costs, and bring the risk of information errors. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention proposes a multimodal model fine-tuning method and system for equipment nameplate image text extraction, which uses parallel computing and batch processing technology to speed up model reasoning, improve the system's processing efficiency, and quickly respond to actual application needs. The model output results are accurately verified through the verification set to ensure that the output text information is accurate and consistent, enhance the reliability and practicality of the system, and ensure high performance in practical applications.

[0006] To achieve the above object, the present invention provides a multimodal model fine-tuning method for equipment nameplate image text extraction, comprising: Step 1: Data collection and intelligent labeling.

[0007] Step 2: Generation of text information extraction dataset based on reinforcement learning.

[0008] Step 3: Preliminary training of a multi-level cross-modal large model.

[0009] Step 4: Fine-tuning of the adaptive multimodal large model.

[0010] Step 5: Efficient reasoning and verification.

[0011] Furthermore, the method of data collection and intelligent labeling in step 1 includes: Through system configuration and connection with front-end shooting equipment, multi-angle and multi-focal length automatic shooting technology is adopted to obtain high-resolution nameplate images.

[0012] The reinforcement learning algorithm is used for intelligent labeling to automatically label the nameplate image with text categories and contents. The labeling accuracy is further improved by manually verifying and adjusting the preliminary labeling results.

[0013] A multimodal large model is used to recognize the text content in nameplate images and semantically annotate them in combination with image context information.

[0014] Cross-validate preliminary annotation results through artificial intelligence algorithms to ensure high accuracy and consistency of annotations Furthermore, the method for generating a text information extraction data set based on reinforcement learning in step 2 includes: Based on the annotated text categories and contents, generative adversarial networks (GANs) are used to generate text combinations of various prompts and answers, which are then merged with the corresponding nameplate images to form an enhanced image-text mapping dataset.

[0015] The dataset is randomly divided into a training set and a validation set in a ratio of 8:2 for model training and testing, and the diversity and representativeness of the dataset are improved through reinforcement learning.

[0016] Furthermore, the preliminary training method of the multi-level cross-modal large model in step 3 includes: The image data in the training set is preprocessed in multiple stages, including image enhancement, feature extraction, and noise filtering, to improve image quality and feature extraction effects.

[0017] The preprocessed image and the corresponding prompt text are input into a multi-level cross-modal large model. A progressive training strategy is adopted to extract image and text features at the basic feature layer, intermediate feature layer, and advanced feature layer, and then fuse them layer by layer. The methods of image data preprocessing include: The image is resized and normalized to a fixed size and the pixel values ​​are normalized to the range of 0-1.

[0018] Enhance the diversity and robustness of image data through data enhancement processing, including random rotation and adding Gaussian noise Furthermore, the fine-tuning method of the adaptive multimodal large model in step 4 includes: Use an adaptive optimization algorithm to compare the text output by the model with the corresponding annotation text and dynamically adjust the model parameters; The preprocessed image and text features are input into the initialized large model, and feature fusion is performed using an advanced cross-attention mechanism.

[0019] The loss is calculated for the output after feature fusion, and the multimodal large model is adaptively fine-tuned, with a focus on optimizing the language model part of information extraction. The loss value is continuously reduced through iterative training, and the model is iteratively trained through an adaptive optimization algorithm to optimize the parameters of the language model part to improve the accuracy and reliability of information extraction.

[0020] Furthermore, the efficient reasoning and verification method in step 5 includes: The nameplate image to be tested and the corresponding prompt text are input into the trained multimodal large model, and efficient reasoning is achieved through parallel computing and batch processing technology.

[0021] The model output results are accurately verified through the validation set to ensure the accuracy and consistency of the output text information.

[0022] A system using a multimodal model fine-tuning method for image text extraction is applicable to the above-mentioned multimodal model fine-tuning method for equipment nameplate image text extraction, including a data acquisition module, an intelligent annotation module, a data set generation module, a multimodal large model training module, a multimodal large model fine-tuning module and a reasoning and verification module; The data acquisition module includes: Shooting device: used to automatically shoot images of nameplates of power equipment from multiple angles and focal lengths.

[0023] Image processing unit: used to pre-process the acquired images, including resizing, standardization and noise filtering.

[0024] The intelligent annotation module includes: Reinforcement learning annotation unit: used to automatically annotate the text category and content in the image through reinforcement learning algorithm.

[0025] Manual verification unit: used to manually check and modify the automatic annotation results to ensure the accuracy of the annotation data.

[0026] Furthermore, the dataset generation module includes: Generative Adversarial Networks (GANs): used to generate text combinations of various prompts and answers, and merge them with the corresponding nameplate images to form a picture-text mapping dataset.

[0027] Dataset division unit: used to divide the image-text mapping dataset into a training set and a validation set in a ratio of 8:2.

[0028] The multimodal large model training module includes: Multi-level feature extraction unit: used to extract image and text features at the basic feature layer, intermediate feature layer and advanced feature layer.

[0029] Feature fusion unit: used to fuse the extracted image features and text features to generate a joint feature representation.

[0030] Furthermore, the multimodal large model fine-tuning module includes: Adaptive optimization unit: used to calculate the loss value between the text output by the model and the annotated text, and dynamically adjust the model parameters.

[0031] Iterative training unit: used to iteratively train the multimodal large model and optimize the language model part to improve the accuracy of information extraction.

[0032] The reasoning and verification modules include: Reasoning unit: used to input the nameplate image to be tested and the corresponding prompt text into the trained multimodal large model for end-to-end information extraction.

[0033] Verification unit: used to test the model output results through the verification set to ensure the accuracy and consistency of the output text information.

[0034] The image processing unit further comprises: Data enhancement unit: used to perform data enhancement processing such as random rotation and adding Gaussian noise on the image to improve the diversity and robustness of the image data.

[0035] Furthermore, the adaptive optimization unit includes: Cross-attention mechanism: used to optimize the interaction between image features and text features during feature fusion and improve the quality of joint feature representation; Loss calculation unit: used to calculate the loss of the output after feature fusion and iteratively train the model parameters through an adaptive optimization algorithm.

[0036] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention provides a multimodal model fine-tuning method and system for extracting text from equipment nameplate images. Through multi-angle and multi-focal length automatic shooting technology, the nameplate of power equipment can be photographed from different angles and focal lengths to ensure the comprehensiveness and clarity of the image. The intelligent annotation module uses a reinforcement learning algorithm to automatically annotate the collected nameplate images, which improves the annotation efficiency and accuracy and reduces the need for manual intervention.

[0037] 2. The present invention provides a multimodal model fine-tuning method and system for extracting text from equipment nameplate images, which uses generative adversarial networks (GANs) to generate a variety of prompt and answer text combinations to enhance the diversity and representativeness of the data set. In this way, more scenarios and situations can be covered, and the generalization ability of the model can be enhanced, making it perform better in different practical applications.

[0038] 3. The present invention provides a multimodal model fine-tuning method and system for equipment nameplate image text extraction. The multi-level cross-modal large model extracts features through the basic feature layer, intermediate feature layer and advanced feature layer, and extracts the feature information of the image and text layer by layer. Through the cross-attention mechanism, the image features and text features are interactively fused to generate a joint feature representation, thereby improving the representation ability of the model and the information extraction effect.

[0039] 4. The present invention provides a multimodal model fine-tuning method and system for equipment nameplate image text extraction, which uses an adaptive optimization algorithm (such as Adam or RMSprop) to dynamically adjust model parameters, minimize loss values, and improve model accuracy. Through iterative training, the language model part is continuously optimized to make its performance in text information extraction more accurate and stable, ensuring that the model can handle various complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 The method for extracting information from electric nameplate images based on fine-tuning of a multi-modal large model is as follows: Figure 2 This is a case study of the text annotation of electric nameplate images used in the present invention for information extraction multimodal large model training. Figure 3 The multimodal large model framework and recognition steps for extracting power nameplate image information of the present invention Figure 4 This is a case study of secondary packaging of electric nameplate image text for information extraction multimodal large model training in this invention DETAILED DESCRIPTION The technical solution of the present invention will be more clearly and completely explained below through description of preferred embodiments of the present invention in combination with the accompanying drawings.

[0042] like Figure 1 As shown, the present invention is specifically: Step 1: Data collection and intelligent labeling Data collection Automatic shooting at multiple angles and focal lengths: Use high-resolution cameras to automatically shoot nameplates of power equipment from different angles and focal lengths. The camera captures multiple nameplate images through pre-set angle and focal length parameters to ensure that every detail can be clearly recorded.

[0043] Image processing: Size resizing: Scale the image to a fixed size (e.g. 448x448 pixels) to facilitate model processing.

[0044] Standardization: Normalize the image pixel values ​​to the range of 0-1 to unify the grayscale of the image.

[0045] Noise filtering: Apply filtering algorithms (such as Gaussian filtering) to remove noise from images and improve image quality.

[0046] Smart Labeling Reinforcement learning annotation: Use reinforcement learning algorithms to intelligently annotate the collected nameplate images. The reinforcement learning model continuously optimizes its annotation strategy through interaction with the environment. For example, the model will receive rewards or penalties based on the accuracy of the annotation, thereby learning how to more accurately annotate the text category and content in the image.

[0047] Manual verification and adjustment: The preliminary annotation results generated by the reinforcement learning algorithm are presented to manual reviewers for verification and adjustment. Manual reviewers will check the accuracy of the annotations and make necessary modifications to ensure the accuracy and consistency of the annotation data.

[0048] Step 2: Generate a dataset for text information extraction based on Generative Adversarial Networks (GANs) Generative Adversarial Networks (GANs) Text combination generation: Generative adversarial networks (GANs) are used to generate text combinations of various prompts and answers. These combinations include different types of prompts (such as "device model", "manufacturer", etc.) and corresponding answers (such as specific device models and manufacturers).

[0049] Image-text mapping dataset generation: The generated text combinations are merged with the collected nameplate images to form an image-text mapping dataset. GANs can enhance the diversity and representativeness of the dataset by generating realistic text-image pairs.

[0050] Dataset Partitioning Training set and validation set division: The generated image-text mapping dataset is randomly divided into a training set (80%) and a validation set (20%) for subsequent model training and testing. During the division process, ensure that the training set and validation set are representative and diverse.

[0051] Step 3: Preliminary training of a multi-layer cross-modal large model Multi-stage preprocessing: Resize: Scale the image to a fixed size (e.g. 448x448 pixels).

[0052] Normalize: Normalize image pixel values ​​to the range of 0-1.

[0053] Data augmentation: Data augmentation techniques such as random rotation (±45°), adding Gaussian noise, etc. are applied to enhance the diversity and robustness of image data.

[0054] Feature extraction and fusion Multi-level feature extraction: Use a multi-level cross-modal large model (such as a model based on the Transformer structure) to extract image and text features at the basic feature layer, intermediate feature layer, and advanced feature layer. For example, the basic feature layer may extract edge and color information of the image, the intermediate feature layer may extract texture and shape information, and the advanced feature layer may extract semantic information.

[0055] Feature fusion: The extracted image features and text features are fused layer by layer. Through the cross-attention mechanism, the image features and text features interact to generate a joint feature representation. The feature fusion process can be achieved through a multi-head self-attention mechanism to ensure the effective integration of features from different modalities.

[0056] Step 4: Fine-tuning of the adaptive multimodal large model Adaptive Optimization Algorithm Loss calculation and parameter adjustment: Compare the text output by the model with the corresponding annotated text and calculate the loss value (such as cross entropy loss). Use adaptive optimization algorithms (such as Adam or RMSprop) to dynamically adjust the model parameters to minimize the loss value.

[0057] Iterative training Model fine-tuning: Iteratively train the multimodal large model, focusing on optimizing the language model part used for information extraction. Through continuous training and verification, the accuracy and robustness of the model are gradually improved. The specific steps are as follows: Input data: Input the preprocessed images and corresponding text in the training set into the model.

[0058] Forward Propagation: Propagate forward through the model to generate predicted outputs.

[0059] Loss calculation: Calculate the loss between the predicted output and the true annotation.

[0060] Backpropagation and parameter update: Gradients are calculated through the backpropagation algorithm and the model parameters are updated using an adaptive optimization algorithm.

[0061] Step 5: Efficient Reasoning and Verification Efficient reasoning Parallel computing and batch processing: In the inference stage, the nameplate image to be tested and the corresponding prompt text are input into the trained multimodal large model. Use parallel computing and batch processing technology to improve the efficiency of inference. For example, GPU acceleration technology and batch processing mechanism can quickly process a large amount of input data.

[0062] Accurate verification Validation set test: Test the model output results through the validation set to ensure the accuracy and consistency of the output text information. The specific steps include: Input validation data: Input the preprocessed images and corresponding text in the validation set into the model.

[0063] Prediction output: Generates prediction output through the model.

[0064] Performance evaluation: Calculate the accuracy, recall, and F1 score of the predicted output to evaluate the performance of the model on the validation set.

[0065] Comprehensively evaluate the performance of the model, including precision, recall, and F1 score, to ensure the reliability of the model in practical applications. For example, analyze the error types of the model through the confusion matrix, and further optimize the model based on the evaluation results.

[0066] As a specific implementation method: Step 1: Collect nameplate images and annotate data. Control the system by configuring and connecting the front-end camera, and then focus on the nameplate of the photographed power equipment by changing its angle and focal length, configure the shooting points for clear imaging and record the configuration, and then the system calls the camera to obtain the nameplate image data; after collecting data on various types of power equipment nameplates, all texts in each nameplate image are annotated by category and content. The annotation samples are as follows: Figure 2 There is no restriction on the annotation method. In the experimental case of the present invention, the annotation mainly uses the existing open source multimodal large model chatgpt-4 to identify all text content and type of each nameplate image. After saving it as txt text, it is manually checked and modified one by one.

[0067] Step 2: Design of text question-answering rules and generation of information extraction data set. Based on the nameplate image and text annotation categories, an image text question-answering queue is generated according to the parameter transmission rules of the multimodal model to be trained. In the experimental case of the present invention, the question-answering text case is generated using five types of key information, namely, equipment category, equipment model, manufacturer, production date, and production number, as follows: { "instruction": "Manufacturer", "input": numpy, "output": {Manufacturer: XXX} } Among them, "instruction" is the input prompt text, which serves as a reference for the required text information and question input, "input": numpy is the input image data, and "output" is the output of the required text information output and answer, such as Figure 3 and Figure 4 As shown, in the experimental case, the present invention randomly divides the collected and constructed data sets into training sets and validation sets in a ratio of 8:2, which are 800 and 200 image data and corresponding text annotations, respectively, for fine-tuning training and testing of the multimodal model of the present invention.

[0068] Step 3: Input the image data and corresponding prompt text in the training set into the cross-modal large model for information extraction, as shown in Figure 3. In the experimental case, Qwen-VL is used as an experimental model to experimentally verify the present invention. Other types of multimodal large models in rights protection are also applicable to this step.

[0069] In view of the diversity and uncertainty of input images and prompt words in actual production environments, during the training process, the image data in the annotated dataset is first preprocessed, including resizing, standardization, and data augmentation. In the experimental case, the size is scaled and padded to 448*448, and standardized to a numerical range of 0-1. During training, the images are randomly and dynamically rotated within the range of ±45°, and Gaussian noise is added. Online enhancement strategies are used and the images are input into the image feature extraction component of the large model with pre-trained weight initialization to obtain the image feature sequence. At the same time, based on the rich prior knowledge of text in the big model, the present invention designs 20 prompt input text rules in the experimental case. In the training, the type of the annotated text is repackaged according to the designed 20 text rules, and the prompt word text is randomly selected and switched as the big model input prompt word ("instruction"), which is input into the big model text encoding component with weight initialization to obtain the text feature sequence. The example of the prompt word text repackage rule is shown in Figure 4, which is specifically combined with the actual application scenario, such as the inquiry method of the nameplate design type is proposed by 10 power station staff according to their work experience and the 20 with the highest votes are selected after voting; The large model used in the case of the present invention is Tongyi Qianwen, whose image features and text encoding components are mainly typical Transformer structures. Specifically, the image feature extraction branch adopts the VIT-BigG structure to extract image features. When the two feature modalities are fused, in order to further reduce the image feature dimension and better integrate with the text features, the module uses a cross-attention mechanism to compress the image features to a fixed length, and then superimposes the two modal features to obtain multimodal features, inputs the large language model for information extraction, and outputs the required text information. In this experimental case, the large language model used for fine-tuning is also the Quen-7B model with pre-trained weights in Tongyi Qianwen, and its structure is a typical Transformer structure.

[0070] Step 4: Fine-tune the multimodal large model. The output text of the multimodal large model and the answer information content of the corresponding annotated text are calculated for loss, which is used to iteratively train the large language model (LLM) in the multimodal large model for information extraction from fusion features; the training loss used is the contrast loss, which is as follows:

[0071] in, and Represents two samples, is the dimension of the feature, N is the number of samples, and in the experimental example, the present invention sets N=256, which is adjustable, and P is associated with the dictionary length, which is 152K.

[0072] During the training process, the AdamW optimizer was used to fine-tune the large text model. A total of 50 rounds of iterative training were performed, with the learning rate linearly decayed from the maximum 1e-5 to 1e-6, and the batch size was 128.

[0073] Step 5: Post-training model inference: Input the nameplate image to be tested and the corresponding prompt text into the trained nameplate information extraction multimodal large model, and directly output the required text information end-to-end.

[0074] A multimodal large model fine-tuning system for extracting text information from nameplate images of power equipment A multimodal large model fine-tuning system for extracting text information from nameplate images of power equipment, comprising a data acquisition module, an intelligent annotation module, a data set generation module, a multimodal large model training module, a multimodal large model fine-tuning module, and a reasoning and verification module The data acquisition module includes: Shooting device: used to automatically shoot images of nameplates of power equipment from multiple angles and focal lengths.

[0075] Image processing unit: used to pre-process the acquired images, including resizing, standardization and noise filtering.

[0076] The intelligent annotation module includes: Reinforcement learning annotation unit: used to automatically annotate the text category and content in the image through reinforcement learning algorithm.

[0077] Manual verification unit: used to manually check and modify the automatic annotation results to ensure the accuracy of the annotation data.

[0078] The dataset generation module includes: Generative Adversarial Networks (GANs): used to generate text combinations of various prompts and answers, and merge them with the corresponding nameplate images to form a picture-text mapping dataset.

[0079] Dataset division unit: used to divide the image-text mapping dataset into a training set and a validation set in a ratio of 8:2.

[0080] The multimodal large model training module includes: Multi-level feature extraction unit: used to extract image and text features at the basic feature layer, intermediate feature layer and advanced feature layer.

[0081] Feature fusion unit: used to fuse the extracted image features and text features to generate a joint feature representation.

[0082] The multimodal large model fine-tuning module includes: Adaptive optimization unit: used to calculate the loss value between the text output by the model and the annotated text, and dynamically adjust the model parameters.

[0083] Iterative training unit: used to iteratively train the multimodal large model and optimize the language model part to improve the accuracy of information extraction.

[0084] The reasoning and verification modules include: Reasoning unit: used to input the nameplate image to be tested and the corresponding prompt text into the trained multimodal large model for end-to-end information extraction.

[0085] Verification unit: used to test the model output results through the verification set to ensure the accuracy and consistency of the output text information.

[0086] The image processing unit further comprises: Data enhancement unit: used to perform data enhancement processing such as random rotation and adding Gaussian noise on the image to improve the diversity and robustness of the image data.

[0087] The adaptive optimization unit includes: Cross-attention mechanism: used to optimize the interaction between image features and text features during feature fusion and improve the quality of joint feature representation.

[0088] Loss calculation unit: used to calculate the loss of the output after feature fusion and iteratively train the model parameters through an adaptive optimization algorithm.

[0089] As a specific embodiment, extract the manufacturer information on a nameplate of an electric equipment: The data acquisition module automatically captures high-resolution nameplate images through multi-angle and multi-focal length photography. The intelligent annotation module uses a reinforcement learning algorithm to automatically annotate the text categories and content in the image, which are then checked and modified by a manual verification unit. The dataset generation module uses GANs to generate a variety of prompt and answer text combinations, merges them with the nameplate image, forms a text-image mapping dataset, and divides it into a training set and a validation set.

[0090] The multimodal large model training module performs multi-stage preprocessing on image data and performs model training through multi-level feature extraction units and feature fusion units. The multimodal large model fine-tuning module uses adaptive optimization units and iterative training units to fine-tune the model, reduce loss values, and optimize the language model part.

[0091] The reasoning and verification module inputs the nameplate image to be tested and the prompt text "Manufacturer" into the trained multimodal large model, performs reasoning through parallel computing and batch processing technology, and ensures the accuracy of the results through verification set testing.

[0092] The above specific implementations are only descriptions of the preferred implementations of the present invention, and do not limit the protection scope of the present invention. Without departing from the design concept and spirit of the present invention, various modifications, substitutions and improvements made by ordinary technicians in this field to the technical solution of the present invention based on the text description and drawings provided by the present invention should all fall within the protection scope of the present invention. The protection scope of the present invention is determined by the claims.

Claims

1. A multimodal model fine-tuning method for equipment nameplate image text extraction, characterized in that: include: Step 1: Data collection and intelligent labeling; Step 2: Generation of text information extraction dataset based on reinforcement learning; Step 3: Preliminary training of a multi-level cross-modal large model; Step 4: Fine-tuning of the adaptive multimodal large model; Step 5: Model reasoning and verification.

2. The multimodal model fine-tuning method for extracting text from equipment nameplate images according to claim 1 is characterized in that: The data collection and intelligent labeling methods in step 1 include: Through system configuration and connection with front-end shooting equipment, multi-angle and multi-focal length automatic shooting technology is adopted to obtain nameplate images; Use reinforcement learning algorithms for intelligent labeling, automatically label nameplate images with text categories and content, and further improve labeling accuracy by manually verifying and adjusting preliminary labeling results; Also includes: Use a multimodal large model to recognize the text content in the nameplate image and perform semantic annotation based on the image context information; The preliminary annotation results are cross-validated using artificial intelligence algorithms.

3. The multimodal model fine-tuning method for extracting text from equipment nameplate images according to claim 2 is characterized in that: The method for generating a text information extraction data set based on reinforcement learning in step 2 includes: Based on the annotated text categories and contents, a generative adversarial network is used to generate text combinations of multiple prompts and answers, and merged with the corresponding nameplate images to form an enhanced image-text mapping dataset; The dataset is randomly divided into a training set and a validation set in a ratio of 8:2 for model training and testing, and the diversity and representativeness of the dataset are improved through reinforcement learning.

4. The multimodal model fine-tuning method for extracting text from equipment nameplate images according to claim 3 is characterized in that: The preliminary training method of the multi-level cross-modal large model in step 3 includes: Perform multi-stage preprocessing on the image data in the training set, including image enhancement, feature extraction, and noise filtering; The preprocessed image and the corresponding prompt text are input into a multi-level cross-modal large model. A progressive training strategy is adopted to extract image and text features at the basic feature layer, intermediate feature layer, and advanced feature layer, and then fuse them layer by layer. The methods of image data preprocessing include: Scaling and normalizing the image, adjusting the image size to a fixed size and normalizing the pixel values ​​to the range of 0-1; Through data enhancement processing, the diversity and robustness of image data are enhanced.

5. The multimodal model fine-tuning method for extracting text from equipment nameplate images according to claim 4 is characterized in that: The fine-tuning method of the adaptive multimodal large model in step 4 includes: Use an adaptive optimization algorithm to compare the text output by the model with the corresponding annotation text and dynamically adjust the model parameters; The preprocessed image and text features are input into the initialized large model, and the cross-attention mechanism is used for feature fusion; The loss is calculated for the output after feature fusion, the multimodal large model is adaptively fine-tuned, the language model part of information extraction is optimized, the loss value is continuously reduced through iterative training, and the model is iteratively trained through an adaptive optimization algorithm to optimize the parameters of the language model part to improve the accuracy and reliability of information extraction.

6. The multimodal model fine-tuning method for extracting text from equipment nameplate images according to claim 5 is characterized in that: The efficient reasoning and verification methods in step 5 include: Input the nameplate image to be tested and the corresponding prompt text into the trained multimodal large model, and realize reasoning through parallel computing and batch processing technology; The model output results are accurately verified through the validation set.

7. A system using a multimodal model fine-tuning method for image text extraction is applicable to a multimodal model fine-tuning method for equipment nameplate image text extraction according to claims 1-6, characterized in that: It includes data acquisition module, intelligent labeling module, data set generation module, multimodal large model training module, multimodal large model fine-tuning module and reasoning and verification module; The data acquisition module includes: Photographic device: used to automatically photograph the nameplate images of power equipment from multiple angles and focal lengths; Image processing unit: used to pre-process the acquired images, including size scaling, standardization and noise filtering; The intelligent annotation module includes: Reinforcement learning annotation unit: used to automatically annotate the text category and content in the image through reinforcement learning algorithm; Manual verification unit: used to manually check and modify the automatic annotation results to ensure the accuracy of the annotation data.

8. The system using the multimodal model fine-tuning method for image text extraction according to claim 7, characterized in that: The dataset generation module includes: Generative adversarial network: used to generate a variety of prompt and answer text combinations and merge them with the corresponding nameplate images to form a picture-text mapping dataset; Dataset division unit: used to divide the image-text mapping dataset into a training set and a validation set in a ratio of 8:2; The multimodal large model training module includes: Multi-level feature extraction unit: used to extract image and text features at the basic feature layer, intermediate feature layer and advanced feature layer; Feature fusion unit: used to fuse the extracted image features and text features to generate a joint feature representation.

9. The system using the multimodal model fine-tuning method for image text extraction according to claim 7, characterized in that: The multimodal large model fine-tuning module includes: Adaptive optimization unit: used to calculate the loss value between the text output by the model and the annotated text, and dynamically adjust the model parameters; Iterative training unit: used to iteratively train the multimodal large model and optimize the language model part to improve the accuracy of information extraction; The reasoning and verification modules include: Reasoning unit: used to input the nameplate image to be tested and the corresponding prompt text into the trained multimodal large model for end-to-end information extraction; Verification unit: used to test the model output results through the verification set to ensure the accuracy and consistency of the output text information; The image processing unit further comprises: Data enhancement unit: used to perform data enhancement processing on images to improve the diversity and robustness of image data.

10. The system using the multimodal model fine-tuning method for image text extraction according to claim 9, characterized in that: The adaptive optimization unit includes: Cross-attention mechanism: used to optimize the interaction between image features and text features during feature fusion and improve the quality of joint feature representation; Loss calculation unit: used to calculate the loss of the output after feature fusion and iteratively train the model parameters through an adaptive optimization algorithm.

Images