Indicator lamp on-off identification method and system based on large and small model collaboration, and medium

Abstract

The invention discloses an indicator light on-off identification method and system based on coordination of large and small models and a medium, and the technical scheme comprises the steps: firstly, respectively training an indicator light target detection model (small model) used for accurately positioning an indicator light and a multi-modal large model (large model) used for finely analyzing a state; during identification application, firstly, a trained target detection model is utilized to quickly position and cut out image blocks of all indicator lamps from an input whole image; and then, combining the image blocks with preset text prompt words, inputting the image blocks and the preset text prompt words into a multi-mode large model subjected to fine adjustment, and analyzing and outputting the on-off state of each indicator lamp by the large model. Through the cooperation strategy of small model positioning and large model analysis, effective balance of calculation efficiency and identification precision is realized, and the method has the advantages of high identification accuracy, strong robustness, high automation degree and strong application generalization ability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence and computer vision, and particularly relates to a method and system for identifying the on-off state of an indicator light based on a large-small model collaboration, and a medium. BACKGROUND

[0002] In the operation and maintenance management of industrial automation, power systems, data centers and various machine rooms, the equipment status indicator light is an important window for monitoring the running status of the equipment. For example, in the control room of a power plant and a substation, there are hundreds or even thousands of indicator lights, the color and on-off state (constant, off, flickering) of which directly reflects the key information of the corresponding equipment, such as running, standby, alarm, etc.

[0003] Traditional indicator light state monitoring mainly relies on manual inspection. This method has many disadvantages: first, the workload is huge, and the inspection personnel need to spend a lot of time and effort to check each indicator light, which is inefficient; second, the frequency of manual inspection is limited, and it cannot achieve 24-hour uninterrupted monitoring, so it may not be able to discover the instantaneous alarm or state change in time, which may miss the best processing opportunity and even cause safety accidents; third, manual interpretation is easily affected by subjective factors and fatigue state, and there is a risk of misjudgment and omission.

[0004] With the development of artificial intelligence technology, automatic inspection using computer vision has become a research hotspot. Some traditional methods use image processing techniques such as color threshold segmentation, edge detection, brightness analysis, etc. to determine the state of the indicator light. However, these methods have poor robustness in complex scenes and are easily disturbed by factors such as changes in environmental lighting, shooting angles, indicator light reflections, and the diversity of indicator lights (shape, color, brightness, etc.), making it difficult to ensure recognition accuracy.

[0005] In recent years, deep learning methods, especially convolutional neural networks (CNN), have achieved great success in image recognition tasks. However, if a large end-to-end model is used directly to analyze the entire monitoring image containing numerous indicator lights, two major challenges will be faced: first, the consumption of computing resources is huge, and the inference speed is slow, making it difficult to meet real-time requirements; second, the model's attention is easily distracted by complex background information, resulting in low recognition accuracy for individual, small-sized indicator lights. While using a small target detection model alone can locate the indicator light, its classification ability is limited in the state judgment task, especially when dealing with difficult samples such as dim light and severe reflections.

[0006] Therefore, there is an urgent need for an automatic indicator light on-off recognition method that can balance recognition efficiency and accuracy, and has good adaptability to complex environments, to address the shortcomings of existing technology. SUMMARY

[0007] Invention purposes: The purpose of the present application is to overcome the defects of the prior art, provide a kind of indicator light on-off identification method, system and medium based on size model cooperation.The method aims to solve the problems of large workload, low efficiency and easy to make mistakes in traditional manual inspection, and the problems that the recognition accuracy and efficiency of existing automated methods are difficult to balance in complex scenes, so as to realize the efficient, accurate and automatic intelligent inspection of indicator light state.

[0008] Technical scheme: To achieve the above purpose, the present application provides the following technical scheme: A kind of indicator light on-off identification method based on size model cooperation, comprising the following steps: a) train an indicator light target detection model, the model is used to receive input image and output the position information of indicator light in image; b) fine-tune a multimodal large model, the model is used to receive the combination input of indicator light image block and text prompt word, and analyze the indicator light image block according to the requirement of the text prompt word to output the state information of indicator light; c) when identifying application, the image to be identified is input into the indicator light target detection model to obtain the position information of one or more indicator lights; d) according to the position information, the corresponding indicator light image block is cropped from the image to be identified; e) the indicator light image block is combined with the preset text prompt word and input into the fine-tuned multimodal large model, and the judgment result is output by the multimodal large model, so as to determine the on-off state of the indicator light.

[0009] Further, the step a) in training indicator light target detection model specifically includes: Collect image data containing indicator light, and label the position of indicator light in image to construct indicator light target detection data set; The preselected target detection algorithm is trained using the indicator light target detection data set until the model converges.

[0010] Further, the target detection algorithm is one of RCNN, YOLO series, EfficientDet or RetinaNet.

[0011] Further, the step b) in fine-tuning multimodal large model specifically includes: Collect image data of indicator light, and label the text describing the on-off state of each image data to construct image-text aligned indicator light multimodal data set; The pre-trained multimodal large model is parameter-efficiently fine-tuned using the indicator light multimodal data set.

[0012] Further, the parameter efficient fine-tuning method is LoRA or BitFit fine-tuning.

[0013] Further, the text prompt in step e) is used to instruct the multi-modal large model to determine the on-off state of the indicator light, and the format of the output result can be specified.

[0014] Further, the position information output in step c) is the bounding box coordinates of the indicator light in the image.

[0015] Further, the judgment result output by the multi-modal large model in step e) is text or structured data representing the "on" or "off" state.

[0016] Specifically, an indicator light on-off recognition method based on the cooperation of large and small models includes the following stages: Model training stage: Step 1: Build and train an indicator light target detection model. Collect image data containing various types of indicator lights, and label the position of each indicator light in the image to form a target detection dataset; then, use the dataset to train an indicator light target detection model, which is used to accurately locate the indicator light in the image in the subsequent recognition process.

[0017] Step 2: Build and fine-tune a multi-modal large model for indicator light state analysis. Collect image data of indicator lights and their corresponding state description texts, and build a multi-modal dataset with aligned images and texts; then, fine-tune a pre-trained multi-modal large model using the multi-modal dataset, so that the model has the ability to accurately determine the on-off state of the indicator light according to the indicator light image block and the specific task prompt.

[0018] Recognition application stage: Step 3: Positioning and extraction. Input the real-time or historical image to be recognized into the trained indicator light target detection model, and the model processes the image to output the position coordinate information of one or more indicator lights, and according to the coordinate information, the corresponding indicator light image block is cropped from the original image.

[0019] Step 4: Analysis and judgment. For each cropped indicator light image block, configure a text prompt for guiding state analysis; then, input the indicator light image block and the corresponding text prompt into the fine-tuned multi-modal large model; the large model analyzes the input information comprehensively, and outputs structured judgment results according to the requirements of the prompt, thereby determining the on-off state of the indicator light.

[0020] A system for implementing any of the above indicator light on-off recognition methods based on the cooperation of large and small models, comprising: a traffic light target detection module configured to perform steps c) and d) for locating and extracting traffic light image blocks; a traffic light state analysis module configured to perform step e) for receiving the traffic light image blocks and prompt words and determining their on-off state using the fine-tuned multi-modal large model.

[0021] A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement any of the above traffic light on-off recognition methods based on the cooperation of small and large models Compared with the prior art, the beneficial effects of the present application are: (1) The present application adopts the cooperative strategy of "small model positioning and large model analysis", and the lightweight target detection model (small model) quickly and accurately filters out the target area (traffic light image block) from the complex large scene image, greatly reducing the data amount of subsequent processing. Only small-sized image blocks are sent to the computationally intensive multi-modal large model (large model) for fine analysis, effectively avoiding the huge computational overhead brought by processing the entire high-resolution image, realizing the rational allocation of computing resources, and improving the efficiency of the overall recognition process.

[0022] (2) The present application decouples the tasks of traffic light positioning and state determination, and the target detection model focuses on accurate positioning, while the fine-tuned multi-modal large model focuses on fine-grained state analysis of individual traffic light image blocks without background interference. This focused processing method, combined with the powerful visual understanding and reasoning ability of the large model, can effectively overcome the interference of factors such as light changes, reflections, and poor shooting angles, making the judgment of on-off state more accurate and robust.

[0023] (3) The present application can learn the deep semantic features of traffic light states through fine-tuning on multi-modal data sets, rather than just the surface color or brightness. This allows the method to easily adapt to new traffic lights of different shapes, colors, and models, with strong generalization ability. At the same time, by adjusting the prompt words, the task can be flexibly expanded, such as judging the color of the traffic light or identifying the flashing state, without the need to redesign the model structure.

[0024] (4) The present application provides a complete end-to-end automated solution, from image input to state result output, without human intervention. It realizes automatic inspection of traffic light states, reduces labor costs, improves inspection frequency and response speed, and provides a strong guarantee for the safe and stable operation of equipment. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flowchart of a traffic light on-off recognition method based on the cooperation of small and large models provided by the present application. DETAILED DESCRIPTION

[0026] The present application will be described in detail below with reference to the embodiments shown in the drawings, but it should be noted that these embodiments are not limiting to the present application, and equivalent transformations or substitutions of function, method, or structure made by those skilled in the art according to these embodiments are within the scope of protection of the present application.

[0027] Referring to Figure 1 The indicator light on-off recognition method based on size model cooperation provided by the embodiment of the present application specifically comprises the following steps: S101: Constructing an indicator light target detection data set.

[0028] The purpose of this step is to prepare data for training a "small model" that can accurately locate the indicator light.

[0029] First, image data acquisition is performed. The data sources can be diversified, including but not limited to: in actual scenes such as power plants, substations, machine rooms, etc., using fixed cameras or cameras on inspection robots to shoot a large number of images containing indicator light equipment under different time periods (day, night), different lighting conditions, and different shooting angles; crawling related public pictures from the Internet; using open source industrial equipment data sets.

[0030] Next, data labeling is performed. Using professional image labeling tools (such as LabelImg, CVAT, etc.), the collected images are manually labeled. When labeling, a rectangular detection box (bounding box) is used to completely frame each visible indicator light in the image. The key is that the detection box should not be too much or too little, and should just surround the entire physical outline of the indicator light to ensure the integrity of the information of the subsequently cropped image blocks. Assign a class label to each detection box, for example, uniformly marked as "indicator_light".

[0031] Finally, the data set is constructed. All labeled data is randomly divided into training set, validation set and test set according to the preset ratio. A typical division ratio is 7:2:1, that is, 70% of the data is used for model training, 20% is used to verify the model performance and adjust the parameters during the training process, and 10% is used to finally evaluate the generalization ability of the model after the training is completed.

[0032] S102: Training an indicator light target detection model.

[0033] This step uses the data set constructed in S101 to train an efficient and lightweight indicator light target detection model (i.e. "small model").

[0034] The current mainstream and high-performance target detection algorithm can be selected, such as RCNN, YOLO (You Only Look Once) series (such as YOLOv5, YOLOv8), EfficientDet or RetinaNet, etc. These models have achieved a good balance between speed and accuracy.

[0035] Among them, the optimizer used in network back propagation includes but is not limited to SGD, ASGD, Adagrad, Adam, etc.

[0036] The training process follows the standard deep learning training paradigm: the training set data is divided into multiple mini-batches. In each training iteration, a mini-batch of data is input into the model, forward propagation is performed, and after a series of convolution, activation, etc. The model network calculates and outputs the prediction result of the indicator light position. Then, the difference (loss value) between the prediction result and the real label is calculated by the loss function (such as CIoULoss, Focal Loss, etc.). Then, the back propagation algorithm is started, and the gradient of each layer parameter in the network is calculated according to the loss value. Finally, the optimizer (such as Adam, SGD, etc.) is used to update the network parameters according to the calculated gradient, and the goal is to minimize the loss value.

[0037] This process (forward propagation, loss calculation, back propagation, parameter update) will be repeated for multiple training cycles (epochs), and each cycle will traverse all training data. During training, the model performance (such as mAP, i.e. mean average precision) on the validation set will be periodically evaluated to monitor whether the model is overfitting, and the learning rate and other hyperparameters will be adjusted accordingly. When the performance of the model no longer improves on the validation set or reaches the preset number of training cycles, the training is completed, and the model weight file with the best performance is saved.

[0038] S103: Constructing an indicator light multi-modal dataset.

[0039] This step prepares data for fine-tuning a "large model" that can understand images and make logical judgments.

[0040] Data collection is similar to S101, but it focuses more on collecting images that can clearly show the "on" or "off" state of the indicator light.

[0041] The key is data cleaning, labeling, and alignment. Here, labeling is no longer drawing a detection box, but rather providing an accurate and objective text description for each indicator light image (which can be a single indicator light image that has been cropped). (1) For a picture of a green indicator light that is on, the corresponding text can be: "In the picture, there is a green round indicator light that is on." (2) For a picture of an extinguished red indicator light, the corresponding text can be: "This is a red square indicator light in an extinguished state." This "image-text pair" forms the core of the multi-modal dataset. Similarly, the constructed dataset is also divided into training, validation, and test sets.

[0042] S104: Fine-tuning the multi-modal large model.

[0043] The purpose of this step is to adapt a general pre-trained multi-modal large model (i.e., "large model") to the specific task of "judging whether the indicator light is on or off."

[0044] Select a suitable base model, such as OpenAI's GPT-4V, or open-source multi-modal large models with visual understanding capabilities such as Qwen-VL, Llama, Janus-Pro, Qwen, Llama, etc. These models have been pre-trained on a large amount of general data and have strong basic capabilities.

[0045] Use the fine-tuning (Fine-tuning) technique. First, load the pre-trained model weights. To save computing resources and prevent catastrophic forgetting, usually use the parameter-efficient fine-tuning (Parameter-Efficient Fine-Tuning, PEFT) method, such as BitFit or LoRA (Low-Rank Adaptation). LoRA adds a small, low-rank "bypass matrix" to the original large weight matrix of the model for training. During training, only a small number of new parameters are updated, while the majority of the original parameters are frozen.

[0046] During training, input the multi-modal data constructed in S103 into the model in batches. The model receives both images and corresponding text descriptions, and its training goal is to learn the deep connection between image visual features and text semantics. Through the loss function (such as cross-entropy loss) and backpropagation, the parameters that can be fine-tuned are iteratively updated. After multiple rounds of training, until the model's performance on the validation set meets expectations, the fine-tuning process is complete. The fine-tuned model will be very good at understanding indicator light images and can perform judgment tasks according to text instructions.

[0047] S105: Configure the prompt words for the analysis task.

[0048] This step sets clear instructions for the automated analysis in the application stage. The prompt (Prompt) is an interface for interacting with the large model. One or more standardized text prompts need to be designed according to specific requirements.

[0049] For example, a simple and direct prompt word can be: "Please determine the status of the indicator light in the following image. Only answer 'on' or 'off'." A prompt word that requires structured output can be: "Analyze the indicator light in the following image and return its status in JSON format. The format is: {'status': 'on' / 'off'}." Well-designed prompt words will be called in the recognition application stage.

[0050] S106: Process the image through the target detection model to obtain the indicator light image block.

[0051] This is the first step in the recognition application stage. A live image to be analyzed (for example, a panoramic image of the control panel taken by a surveillance camera) is input into the indicator light target detection model trained in S102.

[0052] The model quickly scans the entire image and outputs the location information of all the indicator lights it detects, usually as a series of bounding box coordinates (such as the top-left x, y coordinates, and the width w and height h of the box).

[0053] According to these coordinate information, the program accurately crops the area corresponding to each indicator light from the original input image, generating a series of small images containing only a single indicator light.

[0054] S107: Input the prompt word and image block into the multi-modal large model to determine the on / off status.

[0055] This is the last step in the recognition application stage, and is also the core judgment step.

[0056] For each indicator light image block obtained in S106, the program takes the prompt word configured in S105 and packages the image block and prompt word together as input to the multi-modal large model fine-tuned in S104.

[0057] After receiving the combined input of "image block + prompt word", the large model uses its powerful visual understanding and command-following ability to analyze it. It will "observe" the indicator light features (brightness, halo, color saturation, etc.) in the image block and combine them with the requirements of the prompt word to generate the corresponding text output.

[0058] For example, if the input is an image block with a bright light and the prompt word "Please determine the status of the indicator light in the following image. Only answer 'on' or 'off'", the model will output the text "on" after analysis. The program receives this output and completes the status judgment of the indicator light. Repeat this process for all cropped image blocks to achieve comprehensive and automated recognition of the status of all indicator lights in the entire image.

[0059] According to the above method, the application further provides a system of an indicator light on-off recognition method based on size model cooperation, comprising: an indicator light target detection module for positioning and extracting an indicator light image block; an indicator light state analysis module for receiving the indicator light image block and a prompt word, and judging the on-off state thereof by using the fine-tuned multi-modal large model.

[0060] In addition, according to the indicator light on-off recognition method based on size model cooperation, the application provides a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement any one of the above indicator light on-off recognition methods based on size model cooperation.

[0061] To sum up, by intelligent cooperation of the size model, the application divides a complex recognition task into two stages of "fast positioning" and "accurate analysis", and each stage performs its own function, so that the automatic recognition of the on-off state of the indicator light is completed in an efficient, accurate and robust manner, and has a very high practical application value.

[0062] In addition, although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description manner of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be properly combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for identifying the on-off of an indicator light based on size model cooperation, characterized in that, The method comprises the following steps: a) training an indicator light target detection model, which is configured to receive an input image and output position information of an indicator light in the image; b) fine-tuning a multi-modal large model, which is configured to receive a combined input of an indicator light image block and a text prompt word, and analyze the indicator light image block according to the requirement of the text prompt word to output state information of the indicator light; c) when an application is identified, inputting an image to be identified into the indicator light target detection model to obtain position information of one or more indicator lights; d) according to the position information, cropping a corresponding indicator light image block from the image to be identified; e) combining the indicator light image block with a preset text prompt word, and inputting the combination into the fine-tuned multi-modal large model to output a judgment result, so as to determine the on-off state of the indicator light.

2. The method of claim 1, wherein, The step a) of training the indicator light target detection model specifically comprises: collecting image data containing indicator lights, and labeling the positions of the indicator lights in the images to construct an indicator light target detection data set; training a preselected target detection algorithm using the indicator light target detection data set until the model converges.

3. The method of claim 2, wherein, The target detection algorithm is one of RCNN, YOLO series, EfficientDet or RetinaNet.

4. The method of claim 1, wherein, The step b) of fine-tuning the multi-modal large model specifically comprises: collecting image data of indicator lights, and labeling a text describing the on-off state of each image data to construct an indicator light multi-modal data set with aligned images and texts; using the indicator light multi-modal data set to perform parameter-efficient fine-tuning on a pre-trained multi-modal large model.

5. The method of claim 4, wherein, The parameter-efficient fine-tuning method is RCNN or LoRA fine-tuning.

6. The method of claim 1, wherein, The text prompt word in the step e) is used to instruct the multi-modal large model to judge the on-off state of the indicator light, and can specify the format of the output result.

7. The method of claim 1, wherein, The position information output in the step c) is the bounding box coordinates of the indicator light in the image.

8. The method of claim 1, wherein, The judgment result output by the multi-modal large model in the step e) is a text or structured data representing the "on" or "off" state.

9. A system for implementing the method of any one of claims 1 to 8, characterized by The method comprises: an indicator light target detection module configured to perform steps c) and d), for locating and extracting an indicator light image block; an indicator light state analysis module configured to perform step e), for receiving the indicator light image block and a prompt word, and judging the on-off state of the indicator light by using the fine-tuned multi-modal large model.

10. A computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method of any one of claims 1 to 8.

Images