Power equipment nameplate automatic identification method and system based on visual large model VLM

Abstract

The invention discloses a power equipment nameplate automatic identification method and system based on a visual large model VLM, and the method comprises the steps: S1, carrying out the preprocessing operation of collected images, generating a corresponding label for each image, and obtaining an image data set with the labels; s2, dividing the image data set attached with the label into a training set and a test set, and performing model training and evaluation on the visual large model according to the image data set to obtain a trained visual large model VLM; s3, performing feature extraction on the preprocessed image by adopting a pre-trained visual large model VLM to obtain an identified character information key value pair; and S4, reasoning the recognized character information key value pairs, obtaining the type of the power equipment according to a preset power equipment classification model, and outputting corresponding structured information. According to the invention, the problem that the existing power equipment nameplate identification technical scheme has obvious deficiencies in the aspects of efficiency, accuracy and cost efficiency is solved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a method and a system for automatically identifying nameplates of electric equipment based on a visual large model (VLM). Background Art

[0002] The existing technical solutions for nameplate recognition of power equipment mainly include manual input, template matching, optical character recognition (OCR), and simple machine learning models. Among them, manual input is done by staff manually entering the information on the nameplate into the system. This solution is applicable to nameplates of any format and is not affected by font or background interference. However, this solution requires a lot of manual time and effort, and is extremely inefficient when facing a large number of equipment; and there are many input errors caused by human factors, which affects the accuracy of the data; in the long run, the labor cost is high, especially in areas with high labor costs. Template matching technology uses computer vision technology to match and identify the text on the nameplate according to preset templates. The advantage of this solution is that it has a good recognition effect on nameplates of fixed formats; but the scope of application of this solution is limited, and it can only recognize nameplates of specific formats, and the recognition effect is not good for non-standard or deformed nameplates; in actual environments, nameplates may fail to be recognized due to various factors (such as contamination, changes in light, etc.), and the robustness is poor; each new nameplate type needs to be reconfigured, which increases the workload and cost of maintenance.

[0003] Optical character recognition (OCR) technology automatically recognizes text content in images through OCR software. This technology can automatically recognize text and has a fast processing speed. However, in some cases, such as complex fonts and severe background interference, the recognition accuracy of OCR will drop significantly; and OCR cannot handle a variety of font styles and typesetting layouts well; due to the high recognition error rate, manual verification and correction are required, which increases additional post-processing costs.

[0004] Machine learning model technology uses some basic machine learning algorithms (such as support vector machines, decision trees, etc.) to perform image classification and text recognition. Compared with traditional template matching, machine learning models have certain generalization capabilities. However, when the training data set is small or not diverse enough, the model's generalization ability is poor and it is difficult to adapt to new nameplate styles; compared with deep learning models, simple models may require more manual feature engineering, and the training process is more cumbersome; and as data accumulates, the model needs to be continuously adjusted and optimized, which increases maintenance costs.

[0005] The existing technology solutions for nameplate recognition of power equipment have obvious deficiencies in efficiency, accuracy and cost efficiency. It is urgent to adopt a large-scale visual model recognition technology to significantly improve the accuracy and processing speed of recognition, while reducing labor costs and maintenance costs, thereby improving economic benefits.

[0006] The Chinese patent with publication number CN111126380A discloses a method and system for recognizing nameplate text of electric equipment, wherein the recognition method includes the steps of: S1. obtaining a target image with nameplate text; S2. performing contrast enhancement processing on the target image; S3. using an OCR recognition method for recognition to obtain natural text content. Although the invention performs contrast processing and brightness detection on the acquired target image, the use of OCR for recognition still has interference from font diversity or similar characters, resulting in recognition errors or failures. Summary of the invention

[0007] The present invention provides a method and system for automatic recognition of nameplates of electric equipment based on a visual large model (VLM), aiming to solve the problem that existing technical solutions for recognition of nameplates of electric equipment have obvious deficiencies in efficiency, accuracy and cost efficiency.

[0008] In order to solve the above technical problems, the present invention provides a method for automatic recognition of nameplates of electric equipment based on a visual large model (VLM), comprising the following steps:

[0009] S1: Use a photographic device to capture images of nameplates of power equipment, perform preprocessing operations on the captured images, and generate corresponding labels for each image to obtain a labeled image dataset.

[0010] S2: Divide the labeled image dataset into a training set and a test set, perform model training and evaluation on the visual large model according to the image dataset, and obtain a trained visual large model VLM.

[0011] S3: Use the pre-trained visual large model VLM to extract features from the pre-processed image to obtain the recognized text information key-value pairs.

[0012] S4: Infer the recognized text information key-value pairs, obtain the type of power equipment according to the preset power equipment classification model and output the corresponding structured information.

[0013] Preferably, the method also includes: opening terminal editing permissions, automatically filling the identified structured information into a table file, and the terminal user editing the output structured information in the table file, expanding the image data set with the image information edited by the terminal user, and enhancing the adaptability of the visual large model VLM to different scenes or devices.

[0014] Preferably, the preprocessing operation includes: image enhancement, multiple smoothing processing and image transformation.

[0015] The image enhancement includes: size standardization, denoising, and brightness adjustment.

[0016] The multiple smoothing processes include: using median filtering to remove salt and pepper noise, using moderate Gaussian blurring to reduce high frequency noise, and using guided filtering to optimize edge preservation.

[0017] The image transformation includes: rotating, scaling, shearing, and flipping the image.

[0018] Preferably, the step S3 further includes:

[0019] For different business scenarios, after using the visual large model VLM to obtain the recognized original text information key-value pairs, understand the business scenarios behind the recognized text information key-value pairs, merge the recognized original text information key-value pairs with the set business scenario templates, and generate the required result text information key-value pairs.

[0020] Preferably, the step S4 is specifically:

[0021] The power equipment information template is set according to the preset power equipment type, and the prompt engineering PromptEngineering is used to guide the reasoning of the recognized text information key-value pairs according to the power equipment information template to obtain the power equipment type and output the corresponding structured information.

[0022] On the other hand, the present invention provides an automatic recognition system for nameplates of electric equipment based on a visual large model (VLM), comprising an image acquisition and preprocessing module, a data partitioning and model training module, a feature extraction module, and an information reasoning and output module.

[0023] The image acquisition and preprocessing module is used to use a photographic device to capture images of the nameplates of power equipment, perform preprocessing operations on the acquired images, and generate corresponding labels for each image to obtain a labeled image data set.

[0024] The data partitioning and model training module is used to divide the labeled image data set into a training set and a test set, and perform model training and evaluation on the visual large model according to the image data set to obtain a trained visual large model VLM.

[0025] The feature extraction module is used to extract features from the preprocessed image using a pre-trained visual large model VLM to obtain recognized text information key-value pairs.

[0026] The information reasoning and output module is used to reason about the recognized text information key-value pairs, obtain the type of power equipment according to the preset power equipment classification model and output the corresponding structured information.

[0027] Preferably, the system also includes a data set expansion module; the data set expansion module is used to open terminal editing permissions, automatically fill the identified structured information into a table file, and the terminal user edits the output structured information in the table file, and expands the image data set with the image information edited by the terminal user, thereby enhancing the adaptability of the model to different scenarios or devices.

[0028] Preferably, the preprocessing operations of the image acquisition and preprocessing module include: image enhancement, multiple smoothing processing and image transformation.

[0029] The image enhancement includes: size standardization, denoising, and brightness adjustment.

[0030] The multiple smoothing processes include: using median filtering to remove salt and pepper noise, using moderate Gaussian blurring to reduce high frequency noise, and using guided filtering to optimize edge preservation.

[0031] The image transformation includes: rotating, scaling, shearing, and flipping the image.

[0032] Preferably, the feature extraction module further includes:

[0033] For different business scenarios, after using the visual large model VLM to obtain the recognized original text information key-value pairs, understand the business scenarios behind the recognized text information key-value pairs, merge the recognized original text information key-value pairs with the set business scenario templates, and generate the required result text information key-value pairs.

[0034] On the other hand, the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for automatic identification of nameplates of electric equipment based on a visual large model (VLM) as described in any embodiment of the present invention is implemented.

[0035] Compared with the prior art, the present invention has the following technical effects:

[0036] The method of the present invention realizes high-precision text recognition through efficient image preprocessing and feature extraction, and can ensure the accurate acquisition of nameplate information in complex environments. In addition, the editing function of the terminal user to output structured information enables the system to continuously optimize the data set and enhance the model's adaptability to different scenarios and equipment, thereby improving flexibility and practicality. Through the automated recognition process, work efficiency is significantly improved, and structured information of power equipment is quickly generated. At the same time, the feedback mechanism ensures that the information provided by the user is integrated into the data set, forming a closed loop of continuous improvement, and further improving the long-term performance and recognition ability of the model. Overall, the method of the present invention not only realizes efficient and accurate recognition of nameplate information through the powerful feature extraction and understanding capabilities of the large visual model, but also outputs specified information in combination with business scenarios, thereby improving the overall efficiency, accuracy and cost-efficiency of nameplate recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the overall flow chart of automatic recognition of electric equipment nameplate based on visual large model VLM described in the present invention;

[0038] Figure 2 is a schematic diagram of the nameplate image of the present invention;

[0039] Figure 3 2 is a schematic diagram of algorithm comparison of the present invention. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in combination with specific embodiments of the present application and with reference to the accompanying drawings.

[0041] Embodiment 1

[0042] This embodiment provides a method for automatic recognition of electric equipment nameplates based on a visual large model VLM. Figure 1 As shown, the following steps are included:

[0043] S1: Use a photographic device to capture images of nameplates of power equipment, perform preprocessing operations on the captured images to ensure that the image quality is suitable for subsequent processing, and generate corresponding labels for each image to obtain a labeled image dataset.

[0044] Specifically, you can use high-resolution industrial cameras or mobile phone cameras and other photographic equipment to take photos of the nameplates of power equipment; the images must be clear, and the text and logos on the nameplates must be clearly visible. The images must be collected from different environments and angles. The images must cover various equipment types to ensure sample diversity.

[0045] As a preferred implementation of this embodiment, the preprocessing operation includes image enhancement, multiple smoothing processing and image transformation.

[0046] The image enhancement includes but is not limited to: size standardization, denoising, brightness adjustment, etc.

[0047] The multiple smoothing processes include: using median filtering to remove salt and pepper noise, using moderate Gaussian blurring to reduce high frequency noise, and using guided filtering to optimize edge retention. In practical applications, the window size of the median filter, Gaussian blurring, and guided filtering filters can usually be set to 3X3 or 5X5.

[0048] The image transformation includes: rotating, scaling, shearing, and flipping the image.

[0049] This can increase the generalization ability of the model during subsequent large visual model VLM training.

[0050] Label generation needs to include all text information on the nameplate. This step can be done manually or semi-automatically to ensure the accuracy of the label.

[0051] During the collection process, attention should be paid to factors such as lighting conditions and shooting angles to minimize recognition errors caused by environmental factors.

[0052] S2: Divide the labeled image dataset into a training set and a test set, perform model training and evaluation on the visual large model according to the image dataset, and obtain a trained visual large model VLM.

[0053] S3: Use the pre-trained large visual model VLM to extract features from the pre-processed image to obtain the recognized text information key-value pairs. Specifically, the deployment of the large visual model VLM can use LMDeploy to deploy the model, allowing multiple users to access it at the same time (concurrent access), and provide a RESTful API compatible with OpenAI, especially the / v1 / chat / completions interface, to support image format interaction, and can process image input, allowing the client to make inference requests through the API.

[0054] S4: Infer the key-value pairs of the recognized text information, obtain the type of power equipment according to the power equipment classification model preset in advance by the user (such as transformer, ring main unit, cable branch box, etc.) and output the preset corresponding structured information. Specifically, you can use Prompt Engineering to guide the inference of the power equipment type and return the preset template field set of the specified equipment type in a fixed JSON format. Figure 2The following is a schematic diagram of the collected nameplate image. After the visual large model VLM performs feature extraction and uses the prompt engineering to guide the inference of the type of power equipment, the structured information obtained is as follows:

[0055] {

[0056] "Brand":"Xiamen Siming Shuang'an Equipment Co., Ltd."

[0057] "Manufacturer":"Beijing Schneider Low Voltage Electric Co., Ltd."

[0058] "Model":"RM6-12",

[0059] "Product Type":"Outdoor High Voltage Ring Network Switchgear",

[0060] "Rated voltage":"12kV",

[0061] "Frequency":"50Hz",

[0062] "Insulation level":"42 / 50 / AC1min / kV",

[0063] "Comply with standard":"GB3906·IEC298",

[0064] "Protection level":"IP33",

[0065] "Product Number":"20011.208",

[0066] "Date":"January 2007"

[0067] }

[0068] As a preferred implementation of this embodiment, for different business scenarios, this embodiment can also use the visual big model VLM to obtain the original text information key-value pairs, understand the business scenarios behind the recognized text information key-value pairs, merge the recognized original text information key-value pairs with the set business scenario template, generate the required result text information key-value pairs, and then return structured information in a specific JSON format by using the prompt project. This combination makes the output of the visual big model more structured and usable, which is convenient for subsequent processing and application. Figure 2 The schematic diagram of the nameplate image of the power equipment shown in the figure, the structured information further returned by the above steps is as follows, among which sblx automatically adds when recognizing the template:

[0069] {

[0070] "Brand":"Xiamen Siming Shuang'an Equipment Co., Ltd."

[0071] "Manufacturer":"Beijing Schneider Low Voltage Electric Co., Ltd."

[0072] "Model":"RM6-12",

[0073] "Product_Type":"Outdoor high voltage ring network switch equipment",

[0074] "Rated_Voltage":"12kV",

[0075] "Frequency":"50Hz",

[0076] "Insulation_Level":"42 / 50 / AC1min / kV",

[0077] "Standards_Compliant":"GB3906·IEC298",

[0078] "Protection_Level":"IP33",

[0079] "Product_Number":"20011.208",

[0080] "Date":"January 2007",

[0081] "sblx":"Ring network switchgear"

[0082] }

[0083] As a preferred implementation of this embodiment, the method further includes: opening terminal editing permissions, automatically filling the recognized message into the table, allowing the terminal user to edit the output structured information in the table file, and expanding the image data set with the image information edited by the terminal user, thereby enhancing the adaptability of the visual large model VLM to different scenes or devices. Specifically, based on user feedback, the model can be retrained or the existing model parameters can be adjusted. The visual large model VLM is regularly updated on the server side, and new versions are continuously optimized and released to ensure that the model is always in the best recognition state.

[0084] In order to verify the effectiveness and superiority of the method provided in this embodiment, some specific cases are provided below:

[0085] The image samples selected for the experimental evaluation are 900 on-site nameplate images, see Figure 3 As shown, compared with the PP-OCR deep learning framework, the automatic recognition method for electric equipment nameplate based on the visual large model VLM described in this embodiment has a better recognition effect.

[0086] Embodiment 2

[0087] Accordingly, this embodiment provides an automatic nameplate recognition system for electric equipment based on a visual large model (VLM), including an image acquisition and preprocessing module, a data partitioning and model training module, a feature extraction module, and an information reasoning and output module.

[0088] The image acquisition and preprocessing module is used to use a photographic device to capture images of nameplates of power equipment, perform preprocessing operations on the acquired images, and generate corresponding labels for each image to obtain a labeled image data set. This module is used to implement the function of step S1 in Example 1 and will not be repeated here.

[0089] The data partitioning and model training module is used to divide the labeled image data set into a training set and a test set, and perform model training and evaluation on the visual large model according to the image data set to obtain a trained visual large model VLM. This module is used to implement the function of step S2 in Example 1 and will not be repeated here.

[0090] The feature extraction module is used to extract features from the preprocessed image using a pre-trained visual large model VLM to obtain recognized text information key-value pairs. This module is used to implement the function of step S3 in Example 1 and will not be described in detail here.

[0091] The information reasoning and output module is used to reason about the recognized text information key-value pairs, obtain the power equipment type according to the preset power equipment classification model and output the corresponding structured information. This module is used to implement the function of step S4 in Example 1 and will not be repeated here.

[0092] Embodiment 3

[0093] This embodiment provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for automatic recognition of nameplates of electric equipment based on a visual large model (VLM) as described in any embodiment of the present invention.

[0094] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0095] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0097] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0098] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for automatic recognition of nameplates of electric equipment based on visual large model (VLM), characterized in that: The following steps are involved: S1: Use a photographic device to take images of the nameplates of power equipment, perform preprocessing operations on the collected images, and generate corresponding labels for each image to obtain a labeled image dataset; S2: Divide the labeled image dataset into a training set and a test set, train and evaluate the visual large model according to the image dataset, and obtain a trained visual large model VLM; S3: Use the pre-trained visual large model VLM to extract features from the pre-processed image to obtain the recognized text information key-value pairs; S4: Infer the recognized text information key-value pairs, obtain the type of power equipment according to the preset power equipment classification model and output the corresponding structured information.

2. The method for automatic recognition of electric equipment nameplate based on visual large model VLM according to claim 1 is characterized in that: The method also includes: opening terminal editing permissions, automatically filling the identified structured information into a table file, allowing the terminal user to edit the output structured information in the table file, expanding the image data set with the image information edited by the terminal user, and enhancing the adaptability of the visual large model VLM to different scenes or devices.

3. The method for automatic recognition of electric equipment nameplate based on visual large model VLM according to claim 1 is characterized in that: The preprocessing operations include: image enhancement, multiple smoothing processing and image transformation; The image enhancement includes: size standardization, denoising, and brightness adjustment; The multiple smoothing processes include: using median filtering to remove salt and pepper noise, using moderate Gaussian blurring to reduce high frequency noise, and using guided filtering to optimize edge retention; The image transformation includes: rotating, scaling, shearing, and flipping the image.

4. The method for automatic recognition of electric equipment nameplate based on visual large model VLM according to claim 1 is characterized in that: The step S3 also includes: For different business scenarios, after using the visual large model VLM to obtain the recognized original text information key-value pairs, understand the business scenarios behind the recognized text information key-value pairs, merge the recognized original text information key-value pairs with the set business scenario templates, and generate the required result text information key-value pairs.

5. The method for automatic recognition of electric equipment nameplate based on visual large model VLM according to claim 1 is characterized in that: The step S4 is specifically as follows: The power equipment information template is set according to the preset power equipment type, and Prompt Engineering is used to guide the reasoning of the recognized text information key-value pairs according to the power equipment information template to obtain the power equipment type and output the corresponding structured information.

6. An automatic recognition system for nameplates of electric equipment based on visual large model VLM, characterized in that: It includes image acquisition and preprocessing module, data partitioning and model training module, feature extraction module and information reasoning and output module; The image acquisition and preprocessing module is used to use a photographic device to capture images of nameplates of power equipment, perform preprocessing operations on the acquired images, and generate corresponding labels for each image to obtain a labeled image data set; The data partitioning and model training module is used to divide the labeled image data set into a training set and a test set, and perform model training and evaluation on the visual large model according to the image data set to obtain a trained visual large model VLM; A feature extraction module is used to extract features from the pre-processed image using a pre-trained visual large model VLM to obtain recognized text information key-value pairs; The information reasoning and output module is used to reason about the recognized text information key-value pairs, obtain the type of power equipment according to the preset power equipment classification model and output the corresponding structured information.

7. The automatic recognition system for electric equipment nameplate based on visual large model (VLM) according to claim 6 is characterized in that: The system also includes a data set expansion module; the data set expansion module is used to open terminal editing permissions, automatically fill the identified structured information into a table file, and the terminal user edits the output structured information in the table file, and expands the image data set with the image information edited by the terminal user, thereby enhancing the adaptability of the model to different scenarios or devices.

8. The automatic recognition system for electric equipment nameplate based on visual large model (VLM) according to claim 6 is characterized in that: The preprocessing operations of the image acquisition and preprocessing module include: image enhancement, multiple smoothing processing and image transformation; The image enhancement includes: size standardization, denoising, and brightness adjustment; The multiple smoothing processes include: using median filtering to remove salt and pepper noise, using moderate Gaussian blurring to reduce high frequency noise, and using guided filtering to optimize edge retention; The image transformation includes: rotating, scaling, shearing, and flipping the image.

9. The automatic recognition system for electric equipment nameplate based on visual large model (VLM) according to claim 6 is characterized in that: The feature extraction module also includes: For different business scenarios, after using the visual large model VLM to obtain the recognized original text information key-value pairs, understand the business scenarios behind the recognized text information key-value pairs, merge the recognized original text information key-value pairs with the set business scenario templates, and generate the required result text information key-value pairs.

10. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for automatic recognition of nameplates of electric equipment based on a visual large model (VLM) as described in any one of claims 1 to 5 when executing the computer program.

Images