Automatic image reasoning and recognition system and method based on large model

Abstract

The invention relates to an automatic image reasoning recognition system and method based on a large model, and the method comprises the steps: obtaining a to-be-processed image, and carrying out the preprocessing of the image, so as to improve the quality of the image; performing global feature extraction on the optimized image through a large-scale pre-training model, and generating a global mask of the image; and according to the global mask of the image, performing automatic identification and labeling on each region in the image, identifying and extracting a target object or a target region in the image through a classification model, performing classification, and according to a classification result, labeling each target object or target region and outputting the labeled target object or target region. According to the method, the identification efficiency and accuracy can be effectively improved, the specific event relationship can be analyzed more reliably, and the development of an intelligent system can be promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an automatic image reasoning and recognition system and method based on a large model. Background Art

[0002] In modern society, image processing technology has been widely used in many fields such as construction site safety monitoring, drone inspection, and high and medium voltage line maintenance. In these application scenarios, it is crucial to quickly and accurately label and identify target objects in images. The commonly used technical means currently rely mainly on deep learning models, especially structures such as convolutional neural networks (CNNs). These technologies have made significant progress in the past few years and have been widely used in practical scenarios. However, with the continuous increase in application demand, its shortcomings have also emerged. For example, the strong dependence on local features leads to the inability to understand complex scenes, the model training is long, the data cost is high, the global understanding ability is poor, the ability to process the correlation between features is weak, the noise is large, the interactivity is poor, etc. In other words, the existing technology has obvious shortcomings in capturing global information of images, consuming computing resources, generalization ability, model interpretability, and robustness to noise. Summary of the invention

[0003] The purpose of the present invention is to overcome at least one of the above-mentioned technical defects and provide an image automatic reasoning and recognition system and method based on a large model, which can effectively improve the efficiency and accuracy of image processing and recognition, while reducing the cost investment in the process.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is: The present invention provides an image automatic reasoning and recognition system based on a large model, which comprises: A data preprocessing module, connected to an external input unit, for acquiring an image to be processed and preprocessing the image, for optimizing and improving the quality of the image; A large model inference module performs global feature extraction on the optimized image output by the data preprocessing module through a large-scale pre-trained model, and generates a global mask of the image; An identification and annotation module, which automatically identifies and annotates each region in the image according to the global mask of the image, is used to identify and extract the target object or target region in the image and classify it, and annotate each target object or target region according to the classification result; The data output module is used to output the labeled image data.

[0005] For the above technical solution, the applicant has further specific optimized implementation methods.

[0006] Optionally, the system includes a post-processing module, the input side of which is connected to the recognition and labeling module, and the output side of which is connected to the data output module. The post-processing module optimizes the labeled target object or target area, detects and removes redundant information obtained by the recognition and labeling module, and corrects erroneous labeling.

[0007] Optionally, the system further comprises a user interaction module, the user interaction module being connected to the data output module and configured to receive and view the annotated image data and manually edit the annotations in the image data; The user interaction module is also connected to the recognition and annotation module, and the manually edited image data is fed back to the classification model in the recognition and annotation module for training and optimizing the classification model.

[0008] Optionally, the large-scale pre-trained model is a Transformer model with an attention mechanism or a large language model.

[0009] In particular, the present application also includes a large model-based image automated reasoning and recognition method, which includes the following working steps: Acquire an image to be processed and preprocess the image to improve the quality of the image; Performing global feature extraction on the optimized image through a large-scale pre-trained model and generating a global mask of the image; Each region in the image is automatically identified and labeled according to the global mask of the image, the target object or target region in the image is identified and extracted and classified through the classification model, and each target object or target region is labeled and output according to the classification result.

[0010] Optionally, the process of preprocessing the image includes but is not limited to image enhancement, brightness enhancement, contrast enhancement, denoising and size normalization.

[0011] Optionally, the process of extracting global features from the optimized image through a large-scale pre-trained model and generating a global mask of the image includes the following processing steps: Extract the feature areas corresponding to each level from low to high in the image hierarchically from the preprocessed image; Capture the correlation between each feature area through the attention mechanism to form a global feature; Based on the global features, the associated feature areas are taken as a target area, and a mask is formed for each target area. The masks corresponding to all target areas constitute the global mask. The global mask corresponds to the boundaries of each target area, and each target area is an independent area or there is overlap depending on the specific situation.

[0012] Optionally, the process of automatically identifying and labeling each region in the image according to the global mask of the image comprises the following processing steps: After generating the global mask, the generated global mask is applied to the input original image, and the original image is divided into a number of marked areas corresponding to each of the target areas. In each marked area, the corresponding object category or event type is identified and classified by a preset classification model, and each marked area is automatically labeled based on the classification result. The labeled data corresponds to the classification result and is used to display the object category or event type corresponding to the marked area.

[0013] Optionally, the annotated image is further post-processed to detect redundant information in the image during the annotation process and merge or remove it. At the same time, the classification results corresponding to each target area are verified again through a learning algorithm and the erroneous annotations are corrected.

[0014] Optionally, for the output annotated image, the user manually edits the annotations in the image data to correct or improve the annotated data, and after editing, the image data is fed back to the classification model in the recognition and annotation step for online learning and optimization of the classification model.

[0015] Compared with the prior art, the advantages of the patent application of the present invention are: The large-model-based automatic image reasoning and recognition system and method of the present application generates a global mask for the image through the large model, integrates the various related areas in the original image based on the various areas in the image information, and then realizes the accurate division of each target area, and then applies the global mask to the original image to realize the definition of the marked area corresponding to the target area, and then classifies and recognizes the marked area, effectively improving the efficiency and accuracy of recognition, and more reliably analyzing the specific event relationship, which helps to promote the development of intelligent systems.

[0016] Furthermore, in the fields of construction site safety monitoring, drone inspection, high and medium voltage line maintenance, etc., the technology provided by this application can realize the automatic recognition and tracking of objects in the environment, thereby providing key information support for intelligent decision-making. At the same time, with the popularization of technologies such as 5G and the Internet of Things, the image global mask automatic reasoning annotation and recognition technology based on large models will play an important role in more fields and promote the intelligent process of society. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Hereinafter, some specific embodiments of the present invention will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings: Figure 1 It is a principle block diagram of an image automatic reasoning and recognition system based on a large model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0020] Existing technologies such as convolutional neural networks still face some challenges and shortcomings when processing complex image scenes, which are specifically manifested in the following aspects: 1. Strong dependence on local features Traditional convolutional neural networks (CNNs) rely on local receptive fields to extract image features. Although multiple layers of convolution can gradually expand the receptive field, this method still tends to capture local information of the image. For complex scenes or tasks that require global understanding, this local feature extraction method may not accurately reflect the global structure and semantic relationship of the image.

[0021] 1.1 Limitations of Local Information Although local feature extraction performs well in processing simple, single-target image tasks, its performance may drop significantly when facing images with multiple targets or complex backgrounds. For example, multiple objects in an image may have similar local features, which may make it difficult for the model to distinguish between different objects or incorrectly label background information as foreground objects.

[0022] 2. High computing requirements for large-scale data Deep learning models usually need to be trained on large-scale labeled data sets to achieve good performance. This places high demands on computing resources. In particular, as the model becomes deeper and the number of parameters increases, the computing resource consumption during training and inference also increases.

[0023] 2.1 Long training time The training process of large-scale models is usually very time-consuming, especially when using high-resolution images or processing multiple tasks (such as image segmentation and recognition at the same time), the required training time may be extremely long. In addition, even in the deployment stage, real-time reasoning requires high-performance hardware support, which may be difficult to achieve in scenarios with limited resources.

[0024] 2.2 Cost of Data Annotation The performance of deep learning models depends on large-scale annotated data sets, which usually require manual annotation, which is not only time-consuming but also increases the cost of the project. For some professional fields (such as drone imagery), annotation work even requires the participation of experts, further increasing the difficulty and cost of data preparation.

[0025] 3. Limitations of generalization ability Although deep learning models perform well on training data, they may not be able to generalize well to new data or scenes that they have never seen before. Existing models tend to overfit to the training data, which may cause them to perform worse than expected when processing images that are different from the training data.

[0026] 3.1 Problems caused by data distribution differences In practical applications, there may be differences in the distribution of data in training data and actual applications. For example, the training data may mainly come from images taken during the day, while in practical applications, images taken at night need to be recognized. This difference in data distribution may lead to a significant decrease in model performance.

[0027] 3.2 Unstable performance in complex scenes In complex scenes (e.g., multiple objects, dynamic backgrounds, etc.), existing models may have difficulty accurately understanding all the information in the image, resulting in labeling errors or inaccurate recognition. This problem is particularly evident in tasks that require precise recognition or segmentation, such as pedestrian detection in autonomous driving, abnormal area or fault area segmentation in drone images, etc.

[0028] 4. Lack of comprehensive understanding of global information Existing technologies often lack the ability to comprehensively understand the global information of an image in image processing tasks. Although some models try to solve this problem by fusing multi-level features, there are still limitations. Especially when dealing with tasks that require comprehensive consideration of the relationships between multiple objects, such as scene understanding and multi-object interaction, the performance of existing technologies is still unsatisfactory.

[0029] 4.1 Information Integration Problem in Multi-Target Scenarios In multi-target scenarios, existing models are usually unable to effectively integrate the relationships between multiple targets. For example, in complex urban traffic scenarios, the relationships between multiple targets such as vehicles, pedestrians, and traffic signs are dynamic and complex. It is difficult to correctly understand these relationships by simply relying on local features, which affects the final recognition and decision-making results.

[0030] 4.2 Lack of spatiotemporal information When processing continuous image sequences (such as videos), existing technologies often only consider spatial information and ignore changes in the temporal dimension. This results in the model being unable to capture the dynamic features of objects when processing dynamic scenes, which in turn affects the correct understanding and prediction of the scene.

[0031] 5. Model interpretability is insufficient Existing deep learning models are generally considered to be "black box" models, that is, their decision-making process is not explainable to users. This unexplainability is particularly serious in some high-risk fields (such as drones and autonomous driving), because users cannot understand why the model reaches a certain conclusion, and thus find it difficult to trust the model's judgment.

[0032] 5.1 Uninterpretability of results Due to the complexity of the model, existing technologies have difficulty providing a clear explanation of the decision-making process. This not only limits the application of the model, but also makes it difficult to locate the problem when an error occurs. For example, in drone image analysis, if the model incorrectly marks an abnormal area or a faulty area, workers who conduct inspections based on drone images may not understand how the error occurred, nor can they effectively correct the model.

[0033] 5.2 Challenges of Model Tuning The tuning process of existing models usually relies on trial and error and experience rather than a deep understanding of the internal mechanisms of the model. This increases the difficulty and uncertainty of tuning, especially in applications that require high accuracy, where the model tuning process can become very complex and difficult to control.

[0034] 6. Sensitivity to noise and anomalies Existing image recognition models are often very sensitive to noise and abnormal data in the input images. This sensitivity may cause the model to make recognition errors or labeling biases when processing poor quality images.

[0035] 6.1 Noise Interference In practical applications, image data may contain various types of noise, such as sensor noise, motion blur, etc. These noises may seriously affect the performance of the model, especially in tasks that require high accuracy, such as drone image analysis.

[0036] 6.2 Processing of abnormal data Existing technologies often perform unstably when processing abnormal data (such as images taken under extreme lighting conditions). The model may not be able to correctly identify or label objects in these abnormal images, resulting in a decrease in recognition accuracy.

[0037] In summary, the existing technology has obvious shortcomings in capturing global image information, consumption of computing resources, generalization ability, model interpretability and robustness to noise. Example

[0038] This application provides an image automatic reasoning recognition system based on a large model, such as Figure 1 As shown, it includes: A data preprocessing module, connected to an external input unit, for acquiring an image to be processed and preprocessing the image, for optimizing and improving the quality of the image; A large model inference module, which extracts global features of the optimized image output by the data preprocessing module through a large-scale pre-trained model and generates a global mask of the image. The large-scale pre-trained model may adopt a Transformer model with an attention mechanism or a large language model; An identification and annotation module, which automatically identifies and annotates each region in the image according to the global mask of the image, is used to identify and extract the target object or target region in the image and classify it, and annotate each target object or target region according to the classification result; The data output module is used to output the labeled image data.

[0039] Optionally, the system includes a post-processing module, the input side of which is connected to the recognition and labeling module, and the output side of which is connected to the data output module. The post-processing module optimizes the labeled target object or target area, detects and removes redundant information obtained by the recognition and labeling module, and corrects erroneous labeling.

[0040] Optionally, the system also includes a user interaction module, which is connected to the data output module and is used to receive and view the annotated image data and manually edit the annotations in the image data; the user interaction module is also connected to the recognition and annotation module, and the image data after manual editing is fed back to the classification model in the recognition and annotation module for training and optimizing the classification model.

[0041] That is to say, the large model-based image automatic reasoning and recognition system of this embodiment mainly includes the following key modules: 1. Data preprocessing module: preprocess the original image data, including image enhancement, noise removal, size normalization and other operations to improve the quality of input data.

[0042] 2. Large model inference module: Use large-scale pre-trained models (such as Transformer or GPT, etc.) to extract global features of the input image and generate a global mask of the image.

[0043] 3. Recognition and annotation module: Based on the global mask generated by the large model, the image is automatically annotated and identified to extract the target object or area in the image.

[0044] 4. Post-processing module: Optimize the labeling and recognition results, including removing redundant information, correcting mislabeled areas, etc., to ensure the accuracy and consistency of the results.

[0045] 5. User interaction module: Users can view and edit annotation results through this module and apply them to actual scenarios to achieve human-computer collaboration.

[0046] The following is a detailed description of the functions implemented by each key module and the technical means used.

[0047] 1. Data Preprocessing Module Main functions: Aims to improve the quality of image data through efficient preprocessing operations, reduce the interference of noise and abnormal data on the model, and improve the overall reasoning accuracy of the model.

[0048] This module is responsible for performing necessary preprocessing operations on the input raw image data. The preprocessing steps include but are not limited to the following: Image enhancement: Enhance image quality by adjusting brightness, contrast, color saturation, etc., making it easier for the model to extract effective features.

[0049] Noise removal: Use filters or other denoising algorithms to remove random noise in the image and improve image clarity.

[0050] Size normalization: Scale images of different sizes to a specified size to ensure the consistency of input data and facilitate subsequent processing.

[0051] 2. Large Model Inference Module Main functions: Innovatively use large models to extract global features of images, breaking through the limitations of local feature extraction of traditional models, achieving a more comprehensive understanding of image content, and generating accurate global masks.

[0052] Use a large-scale pre-trained model to extract global features from images. The core of this module is: Global feature extraction: Large-scale pre-trained models (referred to as large models), such as models based on the Transformer architecture, can capture the global information of the image through the self-attention mechanism, rather than relying solely on local features. This can better understand multiple objects in the image and their relationships in complex scenes.

[0053] Global mask generation: The large model generates a global mask of the image through reasoning. The global mask identifies the boundaries or regions of each target object in the image, providing a basis for subsequent annotation and recognition.

[0054] 3. Recognition and Annotation Module Main functions: By combining the global reasoning capabilities of large models and automatic labeling technology, it is possible to efficiently complete image labeling and recognition in complex scenes, reduce manual intervention, and improve work efficiency.

[0055] According to the global mask generated by the large model, the image is automatically annotated and recognized. The technical core of this module includes: Automatic labeling: The system automatically labels the target objects in the image based on the global mask and identifies the object’s category, location, shape and other information.

[0056] Accurate identification: Use the model to further identify and classify the annotation results to ensure the accuracy of the results.

[0057] 4. Post-processing module Main function: By optimizing the preliminary annotation results, the quality of the final output results is further improved to ensure that they can be directly used in practical applications.

[0058] This module performs post-processing on the labeling and recognition results to improve the accuracy and consistency of the results. The specific operations include: Redundant information removal: Automatically remove duplicate or redundant annotation information that may appear during the annotation process.

[0059] Mislabeling correction: Use rules or machine learning algorithms to detect and correct possible mislabeled areas to ensure the reliability of the labeling results.

[0060] 5. User Interaction Module Main functions: Through interaction with users, it not only provides highly automated labeling and recognition capabilities, but also has flexible human-computer collaboration features, and can adjust and optimize results according to actual needs.

[0061] This module provides a human-computer interaction interface, where users can view and edit the annotation results generated by the system. At the same time, user feedback can also be used for online learning and optimization of the model. The functions that can be realized include: Visualization of annotation results: Users can intuitively view the annotation and recognition results of the image through the interface.

[0062] User feedback and correction: Allow users to manually adjust the annotation results and use these adjustments as feedback information to optimize future reasoning of the classification model in the recognition and annotation module.

[0063] In summary, the advantages of the large-model-based image automatic reasoning and recognition system of this embodiment are summarized as follows: Global mask reasoning and generation: This application is the first to use a large model to automatically generate global masks for images, effectively improving the accuracy of image processing in complex scenes; Automatic labeling and recognition: Through global feature extraction and mask generation, automatic labeling and recognition of images are achieved, reducing the workload of manual labeling.

[0064] Modular design and flexible application: The modules of the system have clear division of labor and complement each other. They have high scalability and flexibility and are suitable for a variety of image processing scenarios.

[0065] Human-computer interaction and online optimization: The introduction of user interaction mechanism enables the system to not only have automated processing capabilities, but also to continuously optimize and evolve through user feedback to adapt to the needs of different application scenarios. Example

[0066] This embodiment describes an image automatic reasoning and recognition method based on a large model, and its overall process is summarized as follows: 1. Data preprocessing: Standardize the input image to ensure data quality.

[0067] 2. Large model reasoning: Use a pre-trained large-scale model to extract global features of the image and generate a global mask.

[0068] 3. Automatic labeling and recognition: Automatically label images and identify targets based on global masks.

[0069] 4. Post-processing: Correct and optimize the preliminary annotation results.

[0070] 5. User interaction and feedback: Users review and provide feedback on annotation results, and the system optimizes itself based on the feedback.

[0071] Specifically, the large model-based image automatic reasoning and recognition method of this embodiment includes the following working steps: Acquire an image to be processed and preprocess the image to improve the quality of the image; Performing global feature extraction on the optimized image through a large-scale pre-trained model and generating a global mask of the image; Automatically identify and label each region in the image according to the global mask of the image, extract and classify the target object or target region in the image through the classification model, and label each target object or target region according to the classification result. At this point, the target object or target region can be directly output, or a post-processing step can be performed; The annotated images are post-processed to detect redundant information in the image during the annotation process and merge or remove it. At the same time, the classification results corresponding to each target area are verified again through the learning algorithm and the wrong annotations are corrected. For the output annotated image, the user manually edits the annotations in the image data, and outputs or displays it after correcting or improving the annotated data. After editing, the image data is fed back to the classification model in the recognition and annotation step for online learning and optimization of the classification model.

[0072] Optionally, the process of preprocessing the image includes but is not limited to image enhancement, brightness enhancement, contrast enhancement, denoising and size normalization.

[0073] Optionally, the process of extracting global features from the optimized image through a large-scale pre-trained model and generating a global mask of the image includes the following processing steps: Extract the feature areas corresponding to each level from low to high in the image hierarchically from the preprocessed image; Capture the correlation between each feature area through the attention mechanism to form a global feature; Based on the global features, the associated feature areas are taken as a target area, and a mask is formed for each target area. The masks corresponding to all target areas constitute the global mask. The global mask corresponds to the boundaries of each target area, and each target area is an independent area or there is overlap depending on the specific situation.

[0074] Optionally, the process of automatically identifying and labeling each region in the image according to the global mask of the image comprises the following processing steps: After generating the global mask, the generated global mask is applied to the input original image, and the original image is divided into a number of marked areas corresponding to each of the target areas. In each marked area, the corresponding object category or event type is identified and classified by a preset classification model, and each marked area is automatically labeled based on the classification result. The labeled data corresponds to the classification result and is used to display the object category or event type corresponding to the marked area.

[0075] The specific processing flow of the image automatic reasoning and recognition method of this embodiment is described in detail as follows.

[0076] 1. Data Preprocessing Overall idea: The main goal of preprocessing is to improve the quality of input data so that it is suitable for subsequent reasoning and annotation steps. This step includes but is not limited to operations such as image enhancement, denoising, and size normalization.

[0077] The pre-processing process includes: 1. Image enhancement: Enhance the visual effect of an image by adjusting image parameters such as brightness and contrast. For example, for low-light images, increase brightness and contrast to highlight the target object.

[0078] 2. Denoising: Use Gaussian filtering or median filtering to remove random noise in the image and reduce interference. For example, in drone image processing, denoising can make areas with abnormalities or faults clearer.

[0079] 3. Size normalization: Scale images of different resolutions to a uniform size to ensure input consistency. This can avoid model instability caused by size differences during inference.

[0080] 2. Large Model Reasoning The overall idea is to use a pre-trained large model (such as the Transformer model or other models based on the self-attention mechanism) to extract global features of the image. This module generates a global mask of the image through the reasoning ability of the model, providing a basis for subsequent annotation and recognition.

[0081] The large model reasoning process includes: 1. Input processing: The preprocessed image is input into the large model, and the model extracts high-level features of the image through a series of hierarchical structures.

[0082] 2. Self-attention mechanism: The model captures the global information in the image through the self-attention mechanism. This mechanism can effectively integrate the associations between different regions in the image to generate a more comprehensive feature expression. For example, in complex scenes, the model can simultaneously consider the relationship between multiple characters in the image.

[0083] 3. Global mask generation: Based on the extracted global features, the model generates a global mask of the image. The mask divides the image into different regions, each of which corresponds to an object or background region in the image.

[0084] In specific application scenarios, the reasoning logic of large models needs to be refined. For example, in drone inspection scenarios, the model needs to pay special attention to the relationship between towers, aerial work vehicles, and insulators. To this end, scenario prior knowledge can be introduced into the model to improve the accuracy of reasoning.

[0085] Algorithm optimization: By adjusting the self-attention mechanism of the large model, it can pay more attention to dynamic objects in the scene, such as the movement trajectory of aerial work vehicles.

[0086] Process optimization: During the reasoning process, multi-frame image information is introduced and combined with changes in the time dimension to improve the ability to understand dynamic scenes.

[0087] 3. Automatic recognition and labeling The overall idea is to use the mask information to automatically complete image annotation and object recognition after generating the global mask. The core of this step is to combine the global mask with the image content of the original image to automatically extract and annotate the key information in the image.

[0088] The identification and annotation process includes: 1. Mask application: The system first applies the generated global mask to the original image to divide the image into several regions.

[0089] 2. Target recognition: In each mask area, the classification model is used to identify the corresponding object category. For example, the system can identify different object categories such as towers, aerial work vehicles, insulators, etc.

[0090] 3. Automatic labeling: Based on the identified object category, the system automatically labels the image and records the location, size and category information of the object. For example, in a drone inspection scenario, the system can label the tower, aerial work vehicle, and insulator in front and display their exact location.

[0091] 4. Post-processing Overall idea: The post-processing module aims to optimize the preliminary results of labeling and recognition to ensure the accuracy and consistency of the results. This module includes operations such as redundant information removal and mislabeling correction.

[0092] The post-processing process includes: 1. Redundancy removal: Detect and remove redundant information during the annotation process. For example, when the same object is annotated multiple times, the system will merge these annotations to avoid duplication.

[0093] 2. Mislabeling correction: Detect and correct possible mislabeled areas through rules or learning algorithms. For example, in drone images, if the system mistakenly labels normal equipment facilities as abnormal equipment, the post-processing module will correct it.

[0094] The performance of the post-processing module in different scenarios can be optimized through algorithm adjustment. For example, in drone inspection images, the mislabeling correction algorithm can be customized based on domain knowledge.

[0095] Logical improvement: By introducing the knowledge of on-site maintenance experts, a special set of rules is designed to correct mislabeled areas. For example, the morphological characteristics of abnormal power equipment areas are combined to adjust the labeling boundaries.

[0096] Algorithm optimization: Use adversarial training technology in deep learning to improve the model's ability to detect and correct abnormal annotations and reduce the mislabeling rate.

[0097] 5. User Interaction and Feedback Overall idea: The implementation of user interaction allows users to view, edit and provide feedback on annotation results, which is different from the traditional "black box" model processing flow. User feedback will be used for online learning and optimization of the model to improve the accuracy and practicality of the system.

[0098] The user interaction process includes: 1. Result display: The system displays the annotation and recognition results to the user in a visual form. The user can view the specific information of each annotated area on the interface.

[0099] 2. Manual Editing: Users can manually edit the annotation results, such as adjusting the annotation boundaries or changing the object category.

[0100] 3. Feedback mechanism: The user's editing operations will be used as feedback input into the system, and the model will be adjusted based on this feedback to optimize subsequent reasoning and labeling results. For example, if the user corrects the labeling of a certain type of object multiple times, the system will retrain the model to improve the recognition accuracy of that type of object.

[0101] User feedback plays a vital role in optimizing system performance. In practical applications, specific feedback mechanisms can be designed to enable the system to better learn user preferences and needs.

[0102] Process optimization: Dynamically adjust the model’s learning rate and parameter settings based on the frequency and type of user feedback to accelerate model convergence.

[0103] Logic improvement: Introduce guidance prompts in the user interaction interface to help users edit annotations more accurately and reduce the burden of user operations.

[0104] At present, the large model-based image automatic reasoning and recognition system and method of the present application have been widely used in inspections and daily work in the power industry. The following is an explanation with several examples from the power industry for easy understanding.

[0105] Application scenario 1: Image monitoring and fault detection of power transmission lines Scenario description: Transmission lines are widely distributed and exposed to complex natural environments for a long time, and are easily affected by external forces, climate influences, aging, etc. The technology of the present invention can collect images of transmission lines through drones or fixed cameras, and automatically identify and mark abnormal conditions on the lines, such as broken wires, damaged insulators, and foreign objects entangled.

[0106] Implementation process: 1. Data preprocessing: Image data collected by drones or fixed cameras is first pre-processed, including noise removal and contrast enhancement, to better capture details on the line. Especially in strong or low light conditions, the system automatically adjusts image brightness and contrast.

[0107] 2. Large model reasoning: The preprocessed image is fed into the large model, which extracts the global features of the transmission line through a self-attention mechanism and generates a global mask. The mask helps the system distinguish between normal lines and possible fault areas, such as broken wires or foreign objects.

[0108] 3.Automated labeling and recognition: The system uses global masks to automatically mark and identify fault areas in the image. For example, it can identify the location of broken wires, damaged insulators, or foreign objects entangled, and accurately locate them to generate a fault report.

[0109] 4. Post-processing: The post-processing module combines historical data and line geographic information to further optimize the annotation results to ensure the accuracy and completeness of fault identification. The system also generates maintenance suggestions based on the identification results to guide the work of on-site maintenance personnel.

[0110] 5. User interaction and feedback: Power operation and maintenance personnel can view the recognition results through the interactive interface and correct the questionable annotations. The correction results will be fed back to the system to optimize the model and improve the accuracy of subsequent recognition.

[0111] Application effect: Through the technology of the present invention, power companies can more quickly and accurately discover potential faults on transmission lines, reduce the workload of manual inspections, and improve the operational safety and reliability of the lines.

[0112] Application scenario 2: Intelligent detection and maintenance of substation equipment Scenario description: There are many types of equipment in the substation, and daily inspection and maintenance are the key to ensure its safe operation. The technology of the present invention can automatically identify the equipment status, detect equipment damage, aging or abnormal conditions, and generate maintenance suggestions by regularly collecting image data of substation equipment.

[0113] Implementation process: 1. Data preprocessing: The acquired device images are first preprocessed, such as removing reflections and noise, and the images are aligned and standardized to ensure comparability between images from different devices.

[0114] 2. Large model reasoning: The preprocessed images are input into the large model, which extracts the global features of the equipment through deep learning and generates a global mask. The mask can accurately identify abnormal conditions such as cracks, rust, and deformation on the surface of the equipment.

[0115] 3.Automated labeling and recognition: The system automatically labels and identifies equipment using global masks, for example, identifying overheating of switchgear, oil leakage in transformers, and other physical damage to equipment, and generates detailed reports.

[0116] 4. Post-processing: The post-processing module combines the historical maintenance records of the equipment to optimize the annotation results and generate accurate diagnostic reports. The system can also automatically dispatch maintenance personnel for on-site maintenance based on the recognition results.

[0117] 5. User interaction and feedback: Maintenance personnel can view the recognition and diagnosis results through the system interface and manually correct inaccurate recognition results. This feedback information will be used for continuous optimization of the model to improve recognition accuracy.

[0118] Application effect: Through the technology of the present invention, the equipment inspection and maintenance work of the substation has been greatly optimized. The system can timely detect abnormal conditions of the equipment, reduce the probability of failure, and improve the operational stability of the substation.

[0119] Application scenario 3: Tree barrier monitoring and management of power lines Scenario description: The distance between trees and power lines is an important factor affecting the safe operation of power transmission lines. Especially in extreme weather such as typhoons or blizzards, fallen trees may cause line short circuits or disconnection accidents. The technology of the present invention can automatically monitor the growth of trees around power lines and warn of potential safety hazards in advance.

[0120] Implementation process: 1. Data preprocessing: Images around power lines are regularly collected by drones or surveillance cameras. The system pre-processes these images to remove environmental noise and highlights the boundaries between trees and lines through edge enhancement technology.

[0121] 2. Large model reasoning: The preprocessed image is fed into the large model, which extracts the global features between trees and power lines through the self-attention mechanism and generates a global mask. The mask can accurately separate trees and power lines and calculate the distance between them.

[0122] 3.Automated labeling and recognition: The system uses global masks to automatically mark and identify tree obstacles in the image. For example, it identifies dangerous areas where trees are close to the line, marks potential risk points, and generates early warning reports.

[0123] 4. Post-processing: The post-processing module combines terrain data and weather forecasts to further optimize the recognition results and generate a detailed tree obstacle monitoring report. The system can also recommend tree pruning or other preventive measures based on the recognition results.

[0124] 5. User interaction and feedback: Power line managers can view tree barrier monitoring results through the interface and manually confirm and adjust the identified dangerous areas. The confirmed information will be used to further optimize the system and improve the accuracy of monitoring.

[0125] Application effect: Through the technology of the present invention, power companies can effectively monitor the growth of trees around power lines, discover and eliminate potential safety hazards in advance, reduce the risk of line failures caused by tree obstacles, and ensure the safe operation of transmission lines.

[0126] Application scenario 4: Blade inspection and maintenance of wind turbines Scenario description: The blades of wind turbines are exposed to harsh environments such as wind, sand, rain and snow for a long time, and are easily affected by problems such as wear, cracks and ice accumulation. Through the technology of the present invention, regular image detection can be performed on the blades of wind turbines, the damage of the blades can be automatically identified, and maintenance suggestions can be generated.

[0127] Implementation process: 1. Data preprocessing: Using the images of wind turbine blades taken by drones, the system pre-processes the images to remove noise such as wind, sand, light and shadow, and aligns the images for subsequent analysis.

[0128] 2. Large model reasoning: The preprocessed blade image is input into the large model, which extracts the global features of the blade surface through deep learning and generates a global mask. The mask helps the system identify problems such as cracks, wear, and ice accumulation on the blade.

[0129] 3.Automated labeling and recognition: The system uses global masks to automatically mark and identify blade damage, for example, identifying the length and location of cracks, the degree of wear, and the extent of ice accumulation, and generates a maintenance report.

[0130] 4. Post-processing: The post-processing module combines the historical data and operation records of the blades to optimize the annotation results and generate detailed maintenance recommendations. The system can also optimize the operating parameters of the generator set based on the identification results to reduce the risk of further damage.

[0131] 5. User interaction and feedback: Maintenance personnel can view the inspection results of the blades through the system interface and can modify or confirm the marked results. This feedback information will be used for continuous optimization of the model to improve the accuracy and efficiency of subsequent inspections.

[0132] Application effect: Through the technology of the present invention, the blade detection of wind turbine generator sets becomes more efficient and accurate. The system can detect the damage of blades in time, reduce the downtime for maintenance, and improve the operating efficiency and power generation of wind farms.

[0133] In summary, through the above application cases in the power industry, the present invention demonstrates its wide application potential in multiple fields such as transmission lines, substations, wind power generation, etc. This technology not only improves the detection efficiency of power equipment, but also provides strong technical support for the intelligent operation and maintenance of the power industry, ensuring the safe and stable operation of the power system.

[0134] With the continuous advancement of science and technology, the power industry is gradually transforming towards intelligence and automation. In this process, the introduction of large-scale model-based image automated reasoning and recognition systems and methods has brought unprecedented changes and improvements to the power industry. This application not only improves the monitoring and maintenance efficiency of power equipment, but also enhances the overall stability and safety of the power system. Its specific beneficial effects are reflected in many aspects.

[0135] 1. Improve equipment monitoring accuracy and efficiency In the power system, various types of equipment such as transformers, transmission lines, and switch stations are in operation for a long time, and their health status directly affects the stability and safety of the entire system. Traditional equipment monitoring methods often rely on manual inspections, which are not only time-consuming and labor-intensive, but also difficult to achieve full coverage and real-time monitoring. The image global mask automatic reasoning annotation and recognition technology based on large models can efficiently process the images of power equipment and automatically identify abnormal conditions on the surface of the equipment, such as overheating, corrosion, cracks, etc., thereby achieving accurate monitoring. This process does not require human intervention, which greatly improves the efficiency and accuracy of monitoring.

[0136] 2. Realize fault warning and rapid response Through real-time monitoring and automatic labeling of power equipment, large-model-based image recognition technology can promptly detect potential fault signs and issue early warning signals. This helps power companies to quickly take countermeasures to avoid the expansion of faults, reduce power outage time and economic losses. At the same time, the technology can also provide maintenance personnel with detailed information on the location and nature of the fault, making it easier to quickly locate the problem and repair it, thereby improving maintenance efficiency and accuracy.

[0137] 3. Reduce operation and maintenance costs and improve economic benefits Traditional manual inspection and troubleshooting methods require a lot of human resources and are inefficient. However, the automated reasoning labeling and recognition technology based on large models can significantly reduce the number and frequency of manual inspections and reduce operation and maintenance costs. At the same time, by improving monitoring accuracy and efficiency, this technology can also help to promptly discover and solve potential problems and avoid greater losses caused by the expansion of faults. These advantages work together to improve the economic benefits of power companies and bring more considerable profit margins to companies.

[0138] 4. Enhance the overall stability and security of the power system The stability and safety of the power system are directly related to the normal operation of social production and people's lives. The large-scale model-based image global mask automatic reasoning annotation and recognition technology provides a strong guarantee for the stable operation of the power system through real-time monitoring and accurate recognition. This technology can timely discover and deal with potential safety hazards, reduce the risk of system failures, and improve the overall stability and reliability of the system. This is of great significance for ensuring social electricity demand and maintaining public safety.

[0139] 5. Promote intelligent upgrading of the power industry With the development of smart grids and energy internet, the power industry has an increasingly urgent need for intelligent technologies. As an important part of intelligent technologies, large-scale model-based image global mask automatic reasoning annotation and recognition technology is gradually being widely used in the power industry. The introduction of this technology not only improves the monitoring and maintenance level of power equipment, but also promotes the intelligent upgrading process of the entire power industry. Through the integration and application of other intelligent technologies, the power industry will achieve a more efficient, more accurate and more sustainable development model.

[0140] The application of large-scale model-based image automatic reasoning and recognition system and method in the power industry has brought significant beneficial effects. It not only improves the accuracy and efficiency of equipment monitoring, realizes fault warning and rapid response, but also reduces operation and maintenance costs and improves economic benefits. At the same time, this technology also enhances the overall stability and security of the power system and promotes the intelligent upgrading process of the power industry. With the continuous development and improvement of technology, we have reason to believe that this technology will play a more important role in the power industry.

[0141] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with this technology to understand the contents of the present invention and implement them accordingly. They cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. An image automatic reasoning and recognition system based on a large model, characterized in that: include: A data preprocessing module, connected to an external input unit, for acquiring an image to be processed and preprocessing the image, for optimizing and improving the quality of the image; A large model inference module performs global feature extraction on the optimized image output by the data preprocessing module through a large-scale pre-trained model, and generates a global mask of the image; An identification and annotation module, which automatically identifies and annotates each region in the image according to the global mask of the image, is used to identify and extract the target object or target region in the image and classify it, and annotate each target object or target region according to the classification result; The data output module is used to output the labeled image data.

2. The large model-based image automated reasoning and recognition system according to claim 1, characterized in that: The system includes a post-processing module, the input side of which is connected to the recognition and labeling module, and the output side of which is connected to the data output module. The post-processing module optimizes the labeled target object or target area, detects and removes redundant information obtained by the recognition and labeling module, and corrects erroneous labeling.

3. The large model-based image automated reasoning and recognition system according to claim 1, characterized in that: A user interaction module is included, the user interaction module is connected to the data output module, and is used to receive and view the annotated image data, and manually edit the annotations in the image data; The user interaction module is also connected to the recognition and annotation module, and the manually edited image data is fed back to the classification model in the recognition and annotation module for training and optimizing the classification model.

4. The large model-based image automated reasoning and recognition system according to any one of claims 1 to 3, characterized in that: The large-scale pre-training model is a Transformer model with an attention mechanism or a large language model.

5. A large-model-based image automated reasoning and recognition method, characterized in that: The following steps are included: Acquire an image to be processed and preprocess the image to improve the quality of the image; Performing global feature extraction on the optimized image through a large-scale pre-trained model and generating a global mask of the image; Each region in the image is automatically identified and labeled according to the global mask of the image, the target object or target region in the image is identified and extracted and classified through the classification model, and each target object or target region is labeled and output according to the classification result.

6. The large model-based image automated reasoning and recognition method according to claim 5, characterized in that: The process of preprocessing the image includes but is not limited to image enhancement, brightness enhancement, contrast enhancement, denoising and size normalization.

7. The large model-based image automated reasoning and recognition method according to claim 5, characterized in that: The process of extracting global features from the optimized image through a large-scale pre-trained model and generating a global mask of the image includes the following processing steps: Extract the feature areas corresponding to each level from low to high in the image hierarchically from the preprocessed image; Capture the correlation between each feature area through the attention mechanism to form a global feature; Based on the global features, the associated feature areas are taken as a target area, and a mask is formed for each target area. The masks corresponding to all target areas constitute the global mask. The global mask corresponds to the boundaries of each target area, and each target area is an independent area or there is overlap depending on the specific situation.

8. The large model-based image automated reasoning and recognition method according to claim 7, characterized in that: The process of automatically identifying and labeling each region in the image according to the global mask of the image comprises the following processing steps: After generating the global mask, the generated global mask is applied to the input original image, and the original image is divided into a number of marked areas corresponding to each of the target areas. In each marked area, the corresponding object category or event type is identified and classified by a preset classification model, and each marked area is automatically labeled based on the classification result. The labeled data corresponds to the classification result and is used to display the object category or event type corresponding to the marked area.

9. The large model-based automatic image reasoning and recognition method according to any one of claims 5 to 8, characterized in that: The annotated images are further post-processed to detect redundant information in the image during the annotation process and merge or remove it. At the same time, the classification results corresponding to each target area are verified again through a learning algorithm and the incorrect annotations are corrected.

10. The large model-based image automated reasoning and recognition method according to claim 9, characterized in that: For the output annotated image, the user manually edits the annotations in the image data to correct or improve the annotated data, and after editing, the image data is fed back to the classification model in the recognition and annotation step for online learning and optimization of the classification model.

Images