Indicator light state identification method, device and electronic equipment

Abstract

The application provides a method and device for identifying the state of an indicator light and an electronic device, comprising: extracting the region image of each indicator light from a target image to be identified; generating an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence according to the region image of each indicator light, the sorting strategy of each indicator light and a preset identification quantity threshold; extracting the global context feature information of the target image according to the target image; extracting a local feature sequence according to the indicator light image sequence and the effective target mask sequence; constructing a target feature sequence according to the global context feature information and the local feature sequence; and identifying and generating the identification result of each indicator light in the target image according to the target feature sequence and the effective target mask sequence. This method can effectively combine the state of other indicator lights around the indicator light, the device layout and the overall image information to make a state judgment, thereby improving the accuracy and stability of the identification result.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a method, apparatus, and electronic device for recognizing indicator light status. Background Technology

[0002] In scenarios such as industrial equipment operation monitoring, water equipment monitoring, computer room inspection, pump station inspection, and power distribution cabinet status detection, indicator light status is typically used to reflect the equipment's operating, standby, alarm, fault, communication, and power supply status. Therefore, accurately identifying the on / off status of indicator lights is a crucial step in achieving automatic equipment status monitoring and anomaly early warning.

[0003] In existing technologies, each indicator light image is typically cropped from an image, and then each indicator light image is individually input into a classification model to determine whether it is in an on or off state.

[0004] The above method ignores the joint state recognition of multiple indicator lights in the same image, resulting in poor recognition accuracy. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a method, apparatus, and electronic device for identifying indicator light status, so as to improve the accuracy of indicator light identification results.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a method for identifying the status of an indicator light, including: Extract the region images of each indicator light from the target image to be identified; Based on the regional images of each indicator light, the sorting strategy of each indicator light, and the preset recognition quantity threshold, an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence are generated; each effective target mask sequence contains multiple mask values ​​arranged in order, and the mask values ​​are used to indicate whether the image at the corresponding position in the indicator light image sequence is an indicator light regional image; Based on the target image, extract the global contextual feature information of the target image; Based on the indicator light image sequence and the effective target mask sequence, extract local feature sequences; Based on the global context feature information and the local feature sequence, a target feature sequence is constructed; Based on the target feature sequence and the effective target mask sequence, the identification results of each indicator light in the target image are identified and generated, and the identification results include at least the on / off state of the indicator light.

[0007] Optionally, generating the indicator light image sequence corresponding to the target image and the effective target mask sequence corresponding to the indicator light image sequence based on the regional images of each indicator light, the sorting strategy of each indicator light, and a preset recognition quantity threshold includes: Based on the spatial arrangement information of each indicator light in the target image, determine the arrangement order of the regional images of each indicator light; The sequence length of the initial indicator light image sequence is determined based on the preset recognition quantity threshold. According to the arrangement order of the area images of each indicator light, the area images of each indicator light are added to the initial indicator light image sequence in sequence, and blank images are filled in the empty positions in the initial indicator light image sequence to obtain the indicator light image sequence; The effective target mask sequence is generated based on the image information at each position in the indicator light image sequence.

[0008] Optionally, extracting the global contextual feature information of the target image based on the target image includes: The target image is input into a global visual encoder, which then identifies and extracts the initial contextual feature information of the target image. The initial context feature information is projected to map it to the target dimension to obtain the global context feature information. The global context feature information includes at least: the illumination information of the target image, the device panel layout information of the target image, the exposure information of the target image, the shooting angle information of the target image, the background information of the target image, and the context association information between the indicator lights in the target image.

[0009] Optionally, the step of extracting local feature sequences based on the indicator light image sequence and the effective target mask sequence includes: The indicator light image sequence and the effective target mask sequence are input into a local visual encoder. The local visual encoder determines the region image of each effective indicator light and the invalid blank image in the indicator light image sequence according to the effective target mask sequence. The local visual encoder is obtained by tuning the pre-trained visual model based on preset prompt words. The prompt words are used to provide additional attention information when recognizing the region image of the indicator light. Based on the area images of each valid indicator light, identify and extract the local feature information of each indicator light; Based on each invalid blank image, determine the occupancy feature information of each invalid blank image; The local feature sequence is generated based on the local feature information of each indicator light and the occupancy feature information of each invalid blank image; the local feature information includes at least the color, brightness and edge information of the indicator light.

[0010] Optionally, constructing the target feature sequence based on the global context feature information and the local feature sequence includes: The global context feature information and the feature information in the local feature sequence are concatenated in a preset order to generate an initial feature sequence; Positional encoding information is added to the information at each position in the initial feature sequence to obtain the target feature sequence.

[0011] Optionally, the step of identifying and generating the recognition results of each indicator light in the target image based on the target feature sequence and the effective target mask sequence includes: The target feature sequence and the effective target mask sequence are input into a relation encoder. The relation encoder determines a first association between the global context feature information and the local feature information of each indicator light in the target feature sequence, and a second association between the local feature information of each indicator light. Based on the first association and the second association, the enhanced local feature information of each indicator light is determined. The enhanced local feature information of the indicator light incorporates the global context feature information and the local feature information of other indicator lights besides the indicator light itself. The enhanced local feature information of each indicator light is input into a pre-trained state prediction model, which then predicts and generates the recognition results for each indicator light.

[0012] Optionally, during the training process, the relation encoder calculates loss information based on the prediction results of the valid sample indicator lights and the actual state labels of the valid sample indicator lights.

[0013] Optionally, it also includes: If the recognition results of the target indicator light in multiple consecutive frames of the target image all indicate an abnormality and the confidence level of the target indicator light's status recognition is greater than a preset threshold, then an alarm message is generated and output.

[0014] Secondly, embodiments of this application also provide an indicator light status recognition device, including: an extraction module, a generation module, a construction module, and a recognition module; The extraction module is used to extract the area images of each indicator light from the target image to be identified; The generation module is used to generate an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence based on the regional images of each indicator light, the sorting strategy of each indicator light, and a preset recognition quantity threshold; each effective target mask sequence contains multiple mask values ​​arranged in order, and the mask values ​​are used to indicate whether the image at the corresponding position in the indicator light image sequence is an indicator light region image; The extraction module is used to extract global contextual feature information of the target image based on the target image; The extraction module is used to extract local feature sequences based on the indicator light image sequence and the effective target mask sequence; The construction module is used to construct a target feature sequence based on the global context feature information and the local feature sequence; The recognition module is used to identify and generate the recognition results of each indicator light in the target image based on the target feature sequence and the effective target mask sequence. The recognition results include at least the on / off state of the indicator light.

[0015] Optionally, the generation module is specifically used to determine the arrangement order of the regional images of each indicator light based on the spatial arrangement information of each indicator light in the target image; The sequence length of the initial indicator light image sequence is determined based on the preset recognition quantity threshold. According to the arrangement order of the area images of each indicator light, the area images of each indicator light are added to the initial indicator light image sequence in sequence, and blank images are filled in the empty positions in the initial indicator light image sequence to obtain the indicator light image sequence; The effective target mask sequence is generated based on the image information at each position in the indicator light image sequence.

[0016] Optionally, the extraction module is specifically used to: input the target image into a global visual encoder, and have the global visual encoder identify and extract the initial contextual feature information of the target image; The initial context feature information is projected to map it to the target dimension to obtain the global context feature information. The global context feature information includes at least: the illumination information of the target image, the device panel layout information of the target image, the exposure information of the target image, the shooting angle information of the target image, the background information of the target image, and the context association information between the indicator lights in the target image.

[0017] Optionally, the extraction module is specifically used to: input the indicator light image sequence and the effective target mask sequence into a local visual encoder, whereby the local visual encoder determines the region images of each effective indicator light and each invalid blank image in the indicator light image sequence based on the effective target mask sequence; the local visual encoder optimizes the pre-trained visual model based on preset prompt words; the prompt words are used to provide additional attention information when recognizing the region images of the indicator lights; Based on the area images of each valid indicator light, identify and extract the local feature information of each indicator light; Based on each invalid blank image, determine the occupancy feature information of each invalid blank image; The local feature sequence is generated based on the local feature information of each indicator light and the occupancy feature information of each invalid blank image; the local feature information includes at least the color, brightness and edge information of the indicator light.

[0018] Optionally, the construction module is specifically used to: concatenate the global context feature information and each feature information in the local feature sequence in a preset order to generate an initial feature sequence; Positional encoding information is added to the information at each position in the initial feature sequence to obtain the target feature sequence.

[0019] Optionally, the identification module is specifically configured to: input the target feature sequence and the effective target mask sequence into a relation encoder, wherein the relation encoder determines a first association relationship between the global context feature information and the local feature information of each indicator light in the target feature sequence, and a second association relationship between the local feature information of each indicator light; and determines the enhanced local feature information of each indicator light based on the first association relationship and the second association relationship; wherein the enhanced local feature information of the indicator light incorporates the global context feature information and the local feature information of other indicator lights besides the indicator light itself. The enhanced local feature information of each indicator light is input into a pre-trained state prediction model, which then predicts and generates the recognition results for each indicator light.

[0020] Optionally, a training module may also be included; The training module is used by the relation encoder to calculate loss information based on the prediction results of the effective sample indicator lights and the actual state labels of the effective sample indicator lights during the training process.

[0021] Optionally, the generation module is further configured to: generate and output alarm prompt information if the recognition results of the target indicator light in the recognition results of multiple consecutive frames of the target image all indicate an abnormality and the confidence level of the state recognition of the target indicator light is greater than a preset threshold.

[0022] Thirdly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to implement the indicator light status recognition method provided in the first aspect.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the indicator light status recognition method as provided in the first aspect.

[0024] The beneficial effects of this application are: This application provides a method, apparatus, and electronic device for identifying indicator light status, comprising: extracting region images of each indicator light from a target image to be identified; generating an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence based on the region images of each indicator light, the sorting strategy of each indicator light, and a preset identification quantity threshold; extracting global contextual feature information of the target image based on the target image; extracting local feature sequences based on the indicator light image sequence and the effective target mask sequence; constructing a target feature sequence based on the global contextual feature information and the local feature sequence; and identifying and generating the identification result of each indicator light in the target image based on the target feature sequence and the effective target mask sequence. This method extracts global contextual features and local features of each indicator light in the target image, constructing target feature information. Based on this target feature information, it can jointly analyze the relationships between indicator lights and the contextual information of each indicator light in the target image, generating recognition results for each indicator light. This reduces misjudgments caused by reflections, exposure changes, occlusion, background interference, or insufficient cropping information when relying solely on the local image information of a single indicator light. It effectively combines the status of other indicator lights around the indicator light, device layout, and overall image information, thus significantly improving the accuracy and stability of the recognition results. Furthermore, by uniformly setting a preset recognition threshold for the length of the generated indicator light image sequence, it addresses the technical challenge of batch training and inference for variable-length multi-target samples when the number of indicator lights in a single image is not fixed. This effectively supports batch processing of multiple images with varying numbers of indicator lights.

[0025] Secondly, by utilizing pre-trained visual encoders and visual cue tuning mechanisms, task transfer can be performed with the visual backbone parameters frozen or partially frozen, reducing the amount of data and training costs required to train the model from scratch. This improves the transfer efficiency of local and global visual encoders in small-sample industrial scenarios while ensuring the accuracy of feature extraction.

[0026] In addition, by using a relational encoder to enhance the local feature information of each indicator light, the layout relationship and status association between multiple indicator lights on the same device panel or in the same image, as well as between each indicator light and the entire image, are calculated. This allows the enhanced feature information of each indicator light to be effectively integrated with global information and information from other indicator lights, thereby improving the accuracy of indicator light status recognition.

[0027] By encapsulating the indicator light recognition results and the original information into a structured result, it is easy to link the monitoring platform, alarm system, operation and maintenance system or manual review system. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 1 ; Figure 2 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 2 ; Figure 3 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 3 ; Figure 4 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 4 ; Figure 5 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 5 ; Figure 6 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 6 ; Figure 7 This is a schematic diagram of the overall process of the indicator light status recognition method provided in the embodiments of this application; Figure 8A schematic diagram of an indicator light status recognition device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0031] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0032] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0033] In scenarios such as industrial equipment operation monitoring, water equipment monitoring, computer room inspection, pump station inspection, and power distribution cabinet status detection, indicator light status is typically used to reflect the equipment's operating, standby, alarm, fault, communication, and power supply status. Therefore, accurately identifying the on / off status of indicator lights is a crucial step in achieving automatic equipment status monitoring and anomaly early warning.

[0034] In the existing technology, the following types of solutions are usually used for indicator light status recognition.

[0035] The first approach involves manual inspection or viewing of monitoring footage. Inspectors determine the status of indicator lights on the equipment panel through on-site observation or remote monitoring and record the equipment's operational status. This method is simple to implement, but it heavily relies on human experience, has low recognition efficiency, and struggles to perform real-time, stable, and batch analysis on a large number of devices and continuous video streams.

[0036] The second approach involves cropping individual indicator lights and then performing binary classification. This method typically involves first obtaining the indicator light region through manual annotation or an object detection model, then cropping each indicator light region into a separate image, and inputting it into a classification model to determine whether it is on or off. This method offers a higher degree of automation compared to manual inspection, but it usually treats each indicator light as an independent sample, ignoring the spatial layout, adjacency relationships, state associations, and overall contextual information of the device panel among multiple indicator lights in the same image.

[0037] The third type of approach involves directly identifying the indicator light using a general object detection model or a general classification model. For example, an object detection model can be used to locate the indicator light area, and then a classification model can be used to determine its on / off state; alternatively, a general image classification model can be used to perform binary classification of the cropped indicator light area to determine whether it is on or off. This type of method may achieve certain results on ordinary samples, but it is prone to false positives or false negatives in complex scenarios such as reflections, abnormal exposure, low resolution, occlusion, small light size, unclear on / off boundaries, and class imbalance.

[0038] The fourth approach involves training the deep learning model from scratch or fine-tuning it with all data. This approach requires a large number of labeled samples and incurs high training costs. However, indicator light data in industrial settings typically suffers from limited sample size, unbalanced class distribution, and significant variations in the acquisition environment. Training from scratch is prone to overfitting, while fine-tuning large-scale visual models with all data can lead to problems such as a large number of training parameters, high deployment costs, and unstable generalization.

[0039] In summary, the existing solutions have the following main shortcomings: 1. The independent classification method for single indicator lights is prone to losing contextual information in the complete image and cannot make full use of the device panel layout, ambient lighting, overall exposure and the relationship between adjacent indicator lights.

[0040] 2. The indicator light image area is usually small in size. The cropped individual indicator light image is easily affected by reflection, low brightness, overexposure, underexposure, occlusion, blur, low resolution and background interference, resulting in unstable status judgment.

[0041] 3. Multiple indicator lights in the same screen often have a fixed arrangement and status association. Existing single indicator light recognition methods do not model the relationship between multiple indicator lights.

[0042] 4. The number of indicator lights may vary in different images, and the traditional fixed input structure is difficult to adapt naturally to the problem of changes in the number of multiple targets in a single image.

[0043] 5. In industrial scenarios with small sample sizes, training a model from scratch or fine-tuning a large model with all samples is costly, prone to overfitting, and not conducive to rapid deployment and transfer.

[0044] 6. Existing solutions typically only output a single classification result, lacking the structured results, confidence levels, and anomaly markers required for integration with alarm systems, operation and maintenance systems, monitoring platforms, or manual review systems.

[0045] Based on this, the indicator light status recognition method provided in this solution first acquires the image to be recognized and crops the region images of each indicator light from the image to be recognized. Both the image to be recognized and the region images of each indicator light are normalized in size. Then, the image to be recognized is sent to a global visual encoder to extract global contextual feature information, and the region images of each indicator light are sent to a local visual encoder that incorporates visual cueing parameters to extract local feature information of each indicator light. Next, the global contextual feature information and the local feature information of each indicator light are used to construct a target feature sequence. The target feature sequence is then input into a relational encoder to obtain the complete image and each indicator light, as well as the spatial layout and status association between each indicator light. Finally, the status prediction model outputs the structured recognition result for each indicator light. The recognition result includes the on / off state, confidence level, location, and anomaly marker, etc., and the structured recognition result is synchronized to the business system for early warning, manual review, etc.

[0046] Figure 1 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 1 The subject executing this method can be a computer device, such as... Figure 1 As shown, the method includes: S101. Extract the area images of each indicator light from the target image to be identified.

[0047] The image to be identified can be any frame extracted from a video stream captured in real time by a specified device. For any frame in the video stream, the method and steps provided in this solution can be used to identify the status of the device's indicator lights. Furthermore, by sequentially identifying each frame in the video stream of the specified device, continuous monitoring of the device's status can be achieved.

[0048] Video streams from designated devices can be acquired using fixed surveillance cameras, industrial cameras, mobile acquisition devices, cameras mounted on inspection equipment, cameras mounted on robots, cameras mounted on unmanned vehicles, or other image acquisition devices.

[0049] In some embodiments, for video stream scenarios, the image to be identified can be extracted from the video stream according to a preset frame rate or event triggering method; for static image scenarios, the captured image can be directly read as the image to be identified; for batch inspection scenarios, the images can be archived and processed according to the device number, area number, or task number.

[0050] This solution can be applied to image recognition scenarios that include one or more indicator lights, status lights, alarm lights, and equipment panel lights, including but not limited to: water equipment monitoring scenarios; industrial control cabinet status recognition scenarios; power distribution cabinet, instrument cabinet, and computer room equipment status detection scenarios; remote inspection scenarios for pump stations, substations, energy equipment, and manufacturing production lines; equipment panel status recognition scenarios in surveillance videos; and intelligent operation and maintenance scenarios that require batch recognition of the status of multiple indicator lights.

[0051] Based on this, the target image to be identified may include one or more indicator lights. The region image of each indicator light can be extracted from the target image to be identified by the image detection method, that is, the region of interest (ROI) image of each indicator light can be extracted.

[0052] S102. Based on the regional images of each indicator light, the sorting strategy of each indicator light, and the preset recognition quantity threshold, generate the indicator light image sequence corresponding to the target image and the effective target mask sequence corresponding to the indicator light image sequence.

[0053] The effective target mask sequence contains multiple mask values ​​arranged in order. The mask values ​​are used to indicate whether the image at the corresponding position in the indicator image sequence is an indicator area image.

[0054] The sorting strategy is used to determine the arrangement order of the regional images of each indicator light in the indicator light image sequence, so as to encode the spatial proximity relationship between each indicator light; the preset recognition number threshold is used to configure the length of the indicator light image sequence to be processed. The preset recognition number threshold can ensure that the images to be recognized containing different numbers of indicator lights can generate indicator light image sequences of the same length, so that the images to be recognized containing different numbers of indicator lights can be processed in batches.

[0055] The number of mask values ​​contained in the effective target mask sequence is consistent with the number of images contained in the generated indicator light image sequence. Since the region images of the indicator lights extracted from some images to be identified may be insufficient to meet the preset recognition threshold, invalid images will be filled in the corresponding positions in the indicator light image sequence to unify the length of the indicator light image sequence. The generated effective target mask sequence can clearly indicate which positions in the indicator light image sequence are the real indicator light region images and which positions are the filled invalid images.

[0056] S103. Extract the global contextual feature information of the target image based on the target image.

[0057] By recognizing and processing the target image, global contextual feature information such as overall exposure, background environment, device panel layout, shooting angle, and overall lighting conditions can be extracted to describe the target image.

[0058] S104. Extract local feature sequences based on the indicator light image sequence and the effective target mask sequence.

[0059] Based on the effective target mask sequence, the effective indicator area images in the indicator image sequence can be determined. By performing recognition processing on the area images of each indicator, the local feature information of each indicator can be extracted.

[0060] For invalid images in the indicator light image sequence, local feature analysis is not performed; instead, invalid feature information is directly generated to serve only as a placeholder.

[0061] Finally, a local feature sequence is constructed from the local feature information of each indicator light and the invalid feature information of each invalid image.

[0062] S105. Construct the target feature sequence based on global context feature information and local feature sequences.

[0063] In some embodiments, global contextual feature information and features in local feature sequences can be fused together to construct a unified target feature sequence.

[0064] S106. Based on the target feature sequence and the effective target mask sequence, identify and generate the recognition results of each indicator light in the target image.

[0065] The recognition result should include at least the on / off status of the indicator light.

[0066] By identifying and analyzing the feature information in the target feature sequence, the identification results of each indicator light in the target image can be generated. The effective target mask sequence is used to filter out invalid feature information filled at invalid filling positions during the identification and analysis process, so as to avoid errors in the analysis results.

[0067] Because the target feature sequence effectively integrates the global contextual feature information of the target image and the local feature information of each indicator light, by recognizing and analyzing the target feature sequence, the interrelationship between each indicator light and the relationship between each indicator light and the context of the target image can be effectively analyzed. This makes the recognition results of each indicator light more accurate and avoids the problems of poor recognition accuracy and insufficient robustness caused by analyzing a single indicator light separately.

[0068] The recognition result includes at least the on / off state of the indicator light, that is, whether the indicator light is on or off.

[0069] In summary, the indicator light status recognition method provided in this embodiment includes: extracting the region images of each indicator light from the target image to be recognized; generating an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence based on the region images of each indicator light, the sorting strategy of each indicator light, and a preset recognition quantity threshold; extracting global contextual feature information of the target image based on the target image; extracting local feature sequences based on the indicator light image sequence and the effective target mask sequence; constructing a target feature sequence based on the global contextual feature information and the local feature sequence; and recognizing and generating the recognition results of each indicator light in the target image based on the target feature sequence and the effective target mask sequence. This method extracts the global contextual feature information and the local feature information of each indicator light in the target image and constructs target feature information, thereby enabling joint analysis of the correlation between each indicator light in the target image and the contextual information between each indicator light and the target image based on the target feature information, generating the recognition results of each indicator light in the target image, effectively improving the accuracy and stability of the recognition results. By standardizing the length of the generated indicator light image sequence by setting a preset recognition threshold, the technical problem that the model has difficulty in batch training and batch inference of variable-length multi-target samples when the number of indicator lights in a single image is not fixed can be solved, thus effectively supporting batch processing of multiple images with different numbers of indicator lights.

[0070] This embodiment briefly explains the steps for extracting the region image of the indicator light from the target image: Assume the target image is:

[0071] in, Indicates the height of the target image. This indicates the width of the target image, and 3 indicates the RGB three channels.

[0072] By using image detection algorithms to detect and identify indicator lights in a target image, the detection bounding boxes for each indicator light can be obtained. Alternatively, image segmentation models, keypoint detection models, or template matching algorithms can also be used to detect the indicator light bounding boxes.

[0073] Set target image Includes There are 1 indicator light, and the target box for each indicator light is represented as follows:

[0074] in, , They represent the first Normalized x and y coordinates of the center point of the indicator light target frame , These represent the normalized width and height of the target bounding box, respectively.

[0075] Since the target bounding box is usually represented using normalized coordinates, we only know the relative position of the indicator light in the target image. However, when actually cropping the image, pixel coordinates are needed. Therefore, it is necessary to convert the target bounding box coordinates to pixel coordinates. The pixel coordinates of the target bounding box of each indicator light can be represented as:

[0076]

[0077]

[0078]

[0079] Based on the transformed pixel coordinates, the target image can be... The middle cropping yields the first Image of the area for each indicator light:

[0080] in, Indicates the first Image of the area of ​​each indicator light.

[0081] To adapt to the input requirements of subsequent processing, the size of the indicator light area image can be normalized:

[0082] in, The uniform input size represents the area image of the indicator light.

[0083] The target image is also normalized in size:

[0084] in, This represents the uniform input size of the target image.

[0085] In one embodiment, one or more of the following preprocessing operations may be performed on the target image and the extracted regional images of each indicator light: image denoising, brightness enhancement, contrast enhancement, distortion correction, color normalization, outlier filtering, target box boundary correction, image cropping region expansion, etc., to improve image quality and thus improve the stability of subsequent recognition.

[0086] Figure 2 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 2 Optionally, in step S102, based on the region image of each indicator light, the sorting strategy of each indicator light, and a preset recognition quantity threshold, an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence are generated, including: S201. Based on the spatial arrangement information of each indicator light in the target image, determine the arrangement order of the regional images of each indicator light.

[0087] In one implementation, the arrangement order of the regional images of each indicator light can be determined according to the spatial distribution information of each indicator light in the target image. For example, the indicator lights can be sorted from left to right and from top to bottom in the target image. In this way, the indicator lights at the beginning of the generated indicator light image sequence correspond to the indicator lights on the left or top of the target image, which itself encodes a spatial proximity relationship between the indicator lights.

[0088] In another implementation, the arrangement order of the indicator area images can be determined according to the preset sorting of the device's indicator lights. For example, if the indicator lights of a certain control cabinet have fixed numbers: power light 1, running light 2, alarm light 3, then the area images of each indicator light can be sorted according to this preset business order.

[0089] Of course, the indicator lights can also be sorted according to the order in which they were extracted.

[0090] By sorting the regional images of each indicator light in the above manner, the spatial and logical relationships between the indicator lights can be effectively analyzed during subsequent identification and analysis.

[0091] S202. Determine the sequence length of the initial indicator light image sequence based on the preset recognition quantity threshold.

[0092] In some embodiments, the maximum number of indicator lights that a single image can process can be preset, which is the preset recognition threshold, assuming it is N, and N can be 20 for example.

[0093] Based on a preset recognition threshold, the initial indicator light image sequence length can be determined to be N. When the number of actual indicator lights extracted from the target image is less than N, blank images are filled in the remaining positions to make the length of the generated indicator light image sequence reach N. When the number of actual indicator lights extracted from the target image is greater than N, the first N indicator lights can be selected according to the target confidence level of each indicator light, device area priority, or the device's preset indicator light sorting. Alternatively, multiple indicator light image sequences can be generated in batches from the indicator lights in the target image. This embodiment mainly uses the generation of one indicator light image sequence from one target image for illustration.

[0094] S203. According to the arrangement order of the area images of each indicator light, add the area images of each indicator light to the initial indicator light image sequence in sequence, and fill the empty positions in the initial indicator light image sequence with blank images to obtain the indicator light image sequence.

[0095] Assuming the initial indicator light image sequence has a length of 20, and the target image contains 10 indicator light region images, these 10 indicator light region images are first written into the first ten positions of the initial indicator light image sequence according to the sorting method determined above. The remaining 10 positions are then filled with blank images, resulting in the indicator light image sequence. In practical applications, the filling is not limited to blank images; any zero tensor that only serves as a placeholder and does not participate in the actual calculation is acceptable.

[0096] By setting a preset threshold for the number of recognitions, the resulting sequence of indicator light images from target images containing different numbers of indicator lights all have the same sequence length. Subsequently, batch processing of indicator light status recognition can be performed on different target images.

[0097] S204. Generate an effective target mask sequence based on the image information at each position in the indicator light image sequence.

[0098] In some embodiments, by generating a corresponding mask value for the image at each position in the indicator light image sequence, an effective target mask sequence is obtained. The effective target mask sequence can clearly indicate whether the image at each position is a real and valid indicator light area image or a blank image that serves as a placeholder. This allows for flexible processing of different numbers of indicator lights during subsequent indicator light status recognition, improving computational efficiency, and avoiding interference from invalid data in subsequent processing.

[0099] Assume the effective target mask sequence is represented as { The actual number of area images for the indicator lights is M, and the preset recognition threshold is [value missing]. Then the first one in the indicator light image sequence The mask value of an image It can be represented as:

[0100] It is worth noting that the length of the effective target mask sequence is the same as the length of the indicator light image sequence. Assuming that the first 10 images in the indicator light image sequence are all region images of the indicator lights, the mask value at the first 10 positions in the effective target mask sequence can be 1, indicating that the images at the corresponding positions in the indicator light image sequence are all real and valid region images of the indicator lights. The last 10 images in the indicator light image sequence are all blank images, so the mask value at the last 10 positions in the effective target mask sequence can be 0, indicating that the images at the corresponding positions in the indicator light image sequence are all invalid blank images.

[0101] Figure 3 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 3 Optionally, in step S103, global contextual feature information of the target image is extracted based on the target image, including: S301. Input the target image into the global visual encoder, and the global visual encoder will identify and extract the initial contextual feature information of the target image.

[0102] Optionally, the size-normalized target image can be input into a global visual encoder, which is implemented by a pre-trained visual encoder with frozen or partially frozen parameters. The global visual encoder recognizes the target image and extracts initial contextual feature information that can reflect the overall information of the target image.

[0103] By freezing or partially freezing parameters, the pre-trained visual encoder already possesses basic visual capabilities such as recognizing textures, edges, colors, and object contours. These capabilities are not retrained. Building upon this foundation, other parameters are fine-tuned to adapt the pre-trained visual encoder to the current recognition scenario, making it more sensitive to subtle changes in indicator light brightness and glass reflection patterns specific to industrial environments. This approach is more cost-effective than full-scale fine-tuning.

[0104] S302. Project the initial context feature information to map it to the target dimension to obtain global context feature information.

[0105] The global contextual feature information includes at least: the illumination information of the target image, the device panel layout information of the target image, the exposure information of the target image, the shooting angle information of the target image, the background information of the target image, and the contextual association information between the indicator lights in the target image.

[0106] Optionally, in order to meet the input dimension requirements of the subsequent relation encoder, a global feature projection function can be used to project the initial context feature information to the target dimension, such as 512 dimensions, to obtain the global context feature information.

[0107] For example, global contextual feature information can describe, for instance: the overall lighting of the target image—whether it's too bright or too dark; whether the target image was taken during the day or at night, and whether there's any glare; what the device panel looks like; whether the camera is shooting from the front or at an angle; and whether the background of the target image is dark sheet metal or a light-colored wall. While global contextual feature information doesn't directly tell us which indicator light is on, it can provide some analytical information. For example, if the entire image is very bright, then an indicator light that appears bright may not actually be on.

[0108] By extracting global contextual feature information, the subsequent indicator light status recognition processing is no longer based solely on the information expressed in the regional image of a single indicator light, but can incorporate global information, thereby improving the robustness of the recognition results.

[0109] In practice, the normalized target image can be used. Input global visual encoder This yields the initial contextual feature information of the target image:

[0110] in:

[0111] Represents feature dimension, initial context feature information Used to represent the overall exposure of the target image, background environment, device panel layout, image shooting angle, lighting environment, and overall contextual information between multiple indicator lights.

[0112] Initial context feature information Project:

[0113] in, The global feature projection function can be composed of one or more of the following: linear layers, normalization layers, activation functions, and dropout layers.

[0114] In some embodiments, the normalized target image is input into a pre-trained visual encoder with frozen or partially frozen parameters. Taking a Transformer model or a Contrastive Language-Image Pre-training (CLIP) visual encoder as an example, the visual encoder first divides the target image into multiple image patch tokens through patch embedding or convolutional projection, and adds classification tokens and positional encodings. Then, it extracts global semantic features through multiple layers of visual encoding blocks. Finally, the output corresponding to the classification tokens or the pooled image features are taken as the initial contextual feature information of the target image.

[0115] Figure 4 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 4 Optionally, in step S104, local feature sequences are extracted based on the indicator light image sequence and the effective target mask sequence, including: S401. Input the indicator light image sequence and the valid target mask sequence into the local visual encoder. The local visual encoder determines the region image of each valid indicator light and the invalid blank image in the indicator light image sequence according to the valid target mask sequence.

[0116] The local visual encoder is obtained by tuning the pre-trained visual model based on preset prompts; the prompts are used to provide additional information of interest when recognizing the region image of the indicator light.

[0117] In some embodiments, the indicator light image sequence and the valid target mask sequence can be simultaneously input into the local visual encoder. The local visual encoder can first determine the region image of the real and valid indicator light and the invalid blank images filled in from the indicator light image sequence based on the valid target mask sequence, thereby clarifying which images need to be truly extracted for local feature information.

[0118] In other embodiments, the region images of the real and valid indicator lights in the indicator light image sequence can be determined first based on the effective target mask sequence, and then the region images of each real and valid indicator light can be sequentially input into the local visual encoder.

[0119] The local visual encoder used in this embodiment introduces preset prompt words on the basis of a pre-trained visual model in order to adapt to the task of recognizing the on / off state of indicator lights in small-sample industrial scenarios.

[0120] For example, the prompts could be: "You are looking at the indicator light now. Pay attention to the brightness changes, be careful of glare, and don't be fooled by the color." These prompts are additional, learnable parameters inserted before the input data to guide the encoder to focus more on features related to the indicator light's on / off state recognition, such as brightness, color, lampshade edges, glare, and weak areas in the regional image.

[0121] The pre-trained visual model used can be a visual transformer, a convolutional neural network, a visual encoder in a visual language model, or other image feature extraction models.

[0122] The prompts can be implemented using shallow prompts, deep prompts, prefix prompts, adapter modules, low-rank adaptation, partial layer fine-tuning, category head fine-tuning, or other efficient parameter migration methods.

[0123] In some embodiments, a Transformer is used as a local visual encoder. During the training phase, for the input region image of an indicator light, the local visual encoder first generates a word sequence including classification words and multiple image patch words; then, several learnable cue words are additionally inserted into the word sequence. These cue words participate in attention calculation as task adaptation parameters, enabling the local visual encoder to form an attention bias oriented towards indicator light status recognition during feature extraction. During the inference phase, the cue words and the region image of the indicator light are jointly input into the local visual encoder to extract local feature information of each indicator light.

[0124] S402. Based on the area images of each valid indicator light, identify and extract the local feature information of each indicator light.

[0125] S403. Based on each invalid blank image, determine the occupancy feature information of each invalid blank image.

[0126] For each indicator light's regional image, the local visual encoder can identify and extract the local feature information of each indicator light; however, for each invalid blank image, the local visual encoder does not perform any actual processing, but directly uses the placeholder feature information as the feature information of the invalid blank image. The placeholder feature information can remain a zero tensor or be an invalid feature.

[0127] S404. Generate a local feature sequence based on the local feature information of each indicator light and the occupancy feature information of each invalid blank image.

[0128] Similarly, the local feature information of each indicator light can be further processed using the local feature projection function to map the local feature information of each indicator light to the target dimension, which can also be 512 dimensions.

[0129] The local feature information of each indicator light and the occupancy feature information of each invalid blank image are arranged according to the arrangement order of the regional images of each indicator light and the invalid blank images in the indicator light image sequence to obtain the local feature sequence.

[0130] Local feature information includes at least the color, brightness, and edge information of the indicator light.

[0131] Of course, it can also include the lampshade condition, lampshade reflective characteristics, lampshade texture characteristics, etc.

[0132] Optionally, color information describes the color of the indicator light, such as whether it is a bright green or a faint red; brightness information describes whether the indicator light is continuously lit or flashing; edge information describes whether the edge of the indicator light is blurred or whether there is any obstruction; lampshade status describes the geometric integrity and light transmittance of the lampshade; reflective features describe whether there is any reflection in the lampshade; and texture features describe the surface material of the lampshade.

[0133] Some characteristics of the lampshade can affect the determination of the indicator light's status.

[0134] In specific implementation, for the normalized first... An image of the area of ​​each indicator light is input into a local visual encoder. Local feature information is obtained:

[0135] in:

[0136] Representing feature dimension, Used to indicate the first The indicator light displays the color, brightness, edge information, lampshade status, lampshade reflective features, and lampshade texture features.

[0137] Further projection of local feature information:

[0138] in, This represents the local feature projection function.

[0139] For the Image of the area of ​​each indicator light Let the original input word sequence be:

[0140] in, As a categorized word group, For image block terms.

[0141] Introduce learnable cue lexical units :

[0142] in, Indicates the number of prompt words.

[0143] In the shallow suggestion optimization method, suggestion words are inserted into the input layer:

[0144] Then input the local visual encoder:

[0145] In the deep suggestion optimization method, in the first Insert or update cue words before each visual coding layer:

[0146] in, Indicates the first A visual encoding layer, Indicates the first The learnable cue words corresponding to each layer.

[0147] Figure 5 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 5 Optionally, in step S105, a target feature sequence is constructed based on global contextual feature information and local feature sequences, including: S501. Concatenate the global context feature information and the feature information in the local feature sequence in a preset order to generate the initial feature sequence.

[0148] In some embodiments, global contextual feature information can be used as a global token, each feature information in the local feature sequence can be used as a local token, and the global token and each local token can be concatenated together in a preset order to generate an initial feature sequence.

[0149] Optionally, the global context feature information is... As a global token, it combines the local feature information of each indicator light and the placeholder feature information of each invalid blank image. The initial feature sequence constructed using local tokens is represented as follows:

[0150] in,

[0151] S502. Add positional encoding information to the information at each position in the initial feature sequence to obtain the target feature sequence.

[0152] To introduce the positional relationship between the indicator lights, positional encoding can be added to the initial feature sequence. Position encoding is used to describe which indicator light corresponds to the feature information recorded at each position in the initial feature sequence. For example, the first position in the initial feature sequence records global context feature information, the second position records local feature information of indicator light 1, the third position records local feature information of indicator light 2, and so on.

[0153] After adding positional encoding information, the resulting target feature sequence is:

[0154] in,

[0155] Location encoding representing global contextual feature information Indicates the first The location code of each indicator light.

[0156] Position coding can be implemented using fixed position coding, two-dimensional spatial position coding, position coding based on target box coordinates, relative position coding, or prior coding of device panel structure, etc.

[0157] The generated target feature sequence can be, for example, as follows: {The first word is global contextual feature information, described as a power distribution cabinet with strong light and the camera slightly looking down; the second word is local feature information of the first indicator light, described as a small red light with clear edges and high brightness in the center; the third word is local feature information of the second indicator light, described as a small green light that is relatively dark overall and appears to be off; ...}.

[0158] Target feature sequence The corresponding effective target mask sequence is adjusted as follows:

[0159] Global tokens are always valid, so the first position is 1.

[0160] Figure 6 A flowchart illustrating the indicator light status recognition method provided in this application embodiment. Figure 6 Optionally, in step S106, based on the target feature sequence and the effective target mask sequence, the recognition results of each indicator light in the target image are identified and generated, including: S601. Input the target feature sequence and the effective target mask sequence into the relation encoder. The relation encoder determines the first association relationship between the global context feature information and the local feature information of each indicator light in the target feature sequence, and the second association relationship between the local feature information of each indicator light. Based on the first association relationship and the second association relationship, determine the enhanced local feature information of each indicator light.

[0161] The enhanced local feature information of the indicator light incorporates global contextual feature information as well as local feature information of other indicator lights besides the indicator light itself.

[0162] In some embodiments, the target feature sequence and the effective target mask sequence constructed above can be input together into the relational encoder. The relational encoder calculates the correlation between different feature information in the target feature sequence through a self-attention mechanism. By calculating the correlation between global contextual feature information and the local feature information of each indicator light, the overall exposure, background environment, overall lighting conditions, camera shooting angle, and device panel background information of the target image can be transmitted to each indicator light. By calculating the correlation between the local feature information of each indicator light, the arrangement relationship, adjacency relationship, and state association of the indicator lights within the same panel can be determined. For example, when the indicator lights in adjacent positions have fixed business meanings or the overall exposure causes the area images of multiple indicator lights to be brighter simultaneously, the relational encoder can use the local feature information of other indicator lights and the global contextual feature information to correct the unstable judgment of a single indicator light. The relational encoder can determine which positions of feature information in the target feature sequence do not need to participate in the calculation based on the effective target mask sequence, thereby avoiding interference from invalid information on the calculation results.

[0163] After the target feature sequence is input into the relation encoder, the relation encoder interprets and analyzes the target feature sequence through a self-attention mechanism. It determines that when analyzing the state of the second indicator light, it can refer to the state of the first indicator light and the brightness of the third indicator light, while also taking into account the overall illumination of the target image.

[0164] If you find that normally the yellow light is off when the green light is on, but now both are on, it will raise suspicion; if one indicator light looks bright, but the other indicator lights are dim, and the overall exposure of the target image is not high, it may be a reflection.

[0165] The relation encoder can also be replaced by graph neural networks, attention pooling networks, recurrent neural networks, cross-attention networks, multilayer perceptron mixers, or other multi-objective relation modeling structures.

[0166] In practical implementation, the target feature sequence can be... and effective target mask sequence Input Relational Encoder The enhanced feature sequence is obtained as follows:

[0167] in:

[0168] This represents the enhanced global contextual feature information. Indicates the first Enhanced local feature information of each indicator light.

[0169] When the relational encoder uses a self-attention structure, its core computation can be represented as:

[0170]

[0171]

[0172] in, , , These represent the linear transformation matrices for the query, key, and value, respectively.

[0173] The self-attention weights are:

[0174] Where T represents the length of the target feature sequence input to the relation encoder; in this embodiment, T can be equal to... , This represents the preset threshold for the number of recognitions. d represents the feature scaling dimension, which can be the key vector dimension or word feature dimension for each attention head. This is used to scale the dot product of the query vector and the key vector to prevent the attention weights from becoming overly concentrated due to the large dimension.

[0175] Combine target feature sequences The corresponding effective target mask sequence can be represented as:

[0176] in, This represents the masking matrix generated from the valid target mask sequence, with a minimum value assigned to invalid padding positions so that they do not participate in the attention calculation.

[0177] The enhanced feature sequence output by the relation encoder is:

[0178] S602. Input the enhanced local feature information of each indicator light into the pre-trained state prediction model, and the state prediction model predicts and generates the recognition result of each indicator light.

[0179] In some embodiments, the enhanced local feature information of each indicator light can be input into the state prediction model. The state prediction model sequentially performs linear mapping, normalization, nonlinear activation, deactivation, and binary classification linear mapping to obtain the state probability of each indicator light being off or on. The category with the highest state probability is then used as the state recognition result of the indicator light, and this highest probability is used as the state recognition confidence level.

[0180] When the predicted category is "light on" and the confidence level reaches a preset threshold, it can be marked as "light on". When the predicted category is "light off" or "light on", but the confidence level is insufficient, it can be marked as: "light off, low confidence"; or "light on, low confidence"; or "pending verification". For multi-state extended scenarios, the dimension of the state prediction model output can be expanded to multiple categories. For example, in addition to "light off" and "light on", it can also be other categories such as fault, flashing, standby, abnormal, alarm, etc.

[0181] In some embodiments, the state prediction model filters out invalid blank images filled with valid target mask sequences, generating results only for real indicator lights. The generated recognition results and the original data can be encapsulated into structured results, which include, but are not limited to: the on / off state of the indicator light, state confidence, the target bounding box coordinates of the indicator light, the indicator light's serial number, the target image's number in the video stream, the device number, the camera code, and the recognition time.

[0182] In alarm scenarios, structured results may also include: abnormal indicator markings, alarm levels, number of consecutive frame acknowledgments, etc.

[0183] In practical implementation, the state prediction model can be implemented using the classification head of a relational encoder, which will include indicator lights. Enhanced local feature information Classification head of input relation encoder This yields the classification score logit (which is the raw score the model assigns to each category):

[0184] in:

[0185] Indicates the first The unnormalized scores for each indicator light, indicating whether it is in an off or on state.

[0186] The state probability is calculated using the Softmax function:

[0187] in:

[0188] The final prediction category is:

[0189] At the same time, the highest probability can be used as the confidence level for identification:

[0190] Therefore, the first The structured recognition result of each indicator light can be represented as:

[0191] in, Indicates the position of the indicator light. Indicates the predicted state. Indicates the confidence level. Indicates the recognition time. Indicates the target image, device, or camera number.

[0192] Therefore, the indicator light in this embodiment The transfer function of the recognition result can be expressed as:

[0193] in: , The number of actual indicator lights in the target image. For the target image, This is an image of the indicator light area. For global vision encoder, For local visual encoders, For relational encoders, This is the classification header for the relation encoder.

[0194] Optionally, during the training process, the state prediction model calculates loss information based on the prediction results of the valid sample indicator lights and the actual state labels of the valid sample indicator lights.

[0195] This embodiment constrains the model to correctly classify multiple valid indicator lights simultaneously during the training phase of the relational encoder. The training sample data consists of the classification score, true state label, and mask value of the region image for each valid indicator light. Through iterative training, the state of the valid indicator lights is predicted, loss information is calculated, and the model is optimized inversely using this loss information.

[0196] Specifically, when calculating the loss information, only the data at the positions with a mask value of 1 are calculated, while the data at the padding positions with a mask value of 0 are not calculated. This avoids incorrect supervision of the model due to the default labels at the padding positions when the number of indicator lights varies across different images, thus supporting unified batch training of variable-length multi-target samples.

[0197] In one implementation, label smoothing cross-entropy can be used to calculate loss information. Label smoothing can reduce the model's overconfidence in a single class label, alleviate overfitting problems in industrial scenarios with small sample sizes, class imbalance, and a large number of boundary samples, and enable the model to output a smoother and more stable probability distribution under scenarios with reflective light, low brightness, occlusion, and exposure changes.

[0198] Of course, in actual implementation, loss information can also be calculated using methods such as ordinary cross-entropy, weighted cross-entropy, focus loss, class balance loss, contrast loss, distillation loss, or multi-task joint loss.

[0199] The calculation of the aforementioned loss information enables batch training of sample images containing different numbers of indicator lights; it shields the interference of invalid padding samples on training; it reduces the model's sensitivity to noise labels and boundary samples; it improves the calibrability of the on / off state probability; and it enhances the training stability and generalization ability of the state prediction model in complex industrial scenarios.

[0200] For multiple indicator lights in a single image, the loss is calculated only for the actual locations of the indicator lights. Let the... The true status label of each indicator light is The predicted probability is The mask value is Then the loss function for a single image can be expressed as:

[0201] in, This represents the cross-entropy loss.

[0202] When label smoothing is used, the true state label Can be converted to a smooth label distribution:

[0203] in, The label smoothing coefficient. The number of categories. In the light-to-light binary classification task, c represents the state category number or category index, with a value range of c=1,2,... .

[0204] The label smoothing cross-entropy loss is:

[0205] Therefore, the training loss function for state recognition of multiple indicator lights in a single image can be expressed as:

[0206] For the entire training batch, the overall loss function is:

[0207] in, This indicates the number of sample images in the training samples. Indicates the first Sample images.

[0208] To improve training stability, this embodiment can prune the classification score logit:

[0209] in, This is the preset clipping threshold.

[0210] To avoid gradient explosion, the gradient norm is clipped:

[0211] in, This is the gradient clipping threshold.

[0212] If weight decay is used, the optimization objective can be expressed as:

[0213] in, This represents the weight decay coefficient.

[0214] The training parameter update can be represented as:

[0215] in, Indicates the first Round learning rate, This represents the optimizer update function.

[0216] In addition, abnormal value detection, learning rate scheduling, normalization layer processing, missing data handling, and parameter initialization can be performed during training to further improve the stability of model training.

[0217] Optionally, the method further includes: if the recognition results of the target indicator light in the recognition results of multiple consecutive frames of the target image all indicate abnormality and the confidence level of the target indicator light status recognition is greater than a preset threshold, then an alarm prompt message is generated and output.

[0218] In some embodiments, an alarm can be triggered by determining whether a certain indicator light is in an abnormal state based on a preset anomaly detection strategy.

[0219] The preset anomaly detection strategy is as follows: if an indicator light is determined to be lit in multiple consecutive frames of images, and the confidence level of the lit light is greater than a preset threshold, then the indicator light is considered to be abnormal, and an alarm message can be output to indicate that the indicator light has malfunctioned.

[0220] The reason why the recognition results of multiple consecutive frames need to meet the conditions is to avoid misjudgment caused by a single recognition result, so as to improve the accuracy of alarms.

[0221] Optionally, alarm notifications can be uploaded to the maintenance system for timely repairs. Alarm notifications can also be synchronized to the monitoring platform, alarm system, equipment status management platform, and manual verification system for further processing.

[0222] Figure 7 This is a schematic diagram illustrating the overall process of the indicator light status recognition method provided in this application embodiment. Figure 7 As shown, the core steps of this solution include: S701, Acquire the target image to be identified or extract the target image to be identified from the video stream; S702. Extract the coordinate information of the target detection box of each indicator light from the target image; S703. Convert the coordinate information of the target detection box of each indicator light into pixel information, and perform region cropping to obtain the region image of each indicator light; S704. Based on the regional images of each indicator light, the sorting strategy of each indicator light, and the preset recognition quantity threshold, generate an indicator light image sequence and generate an effective target mask sequence. S705. Extract global contextual feature information from the target image; S706. Extract local feature sequences from the indicator light image sequence; S707. Construct the target feature sequence based on global contextual feature information and local feature sequences; S708. The target feature sequence is processed by a relational encoder to obtain the enhanced local feature information of each indicator light.

[0223] S709. The enhanced local feature information of each indicator light is identified using a state prediction model, and the identification result of each indicator light is predicted. S710. Determine whether the recognition result passes the verification; if it passes the verification, proceed to step S711; otherwise, proceed to step S712. S711. Send the recognition results to an external system for business linkage; S712. Review the identification results that fail the verification, and return to step S702 if necessary.

[0224] In summary, the indicator light status recognition method provided in this embodiment includes: extracting the region images of each indicator light from the target image to be recognized; generating an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence based on the region images of each indicator light, the sorting strategy of each indicator light, and a preset recognition quantity threshold; extracting global contextual feature information of the target image based on the target image; extracting local feature sequences based on the indicator light image sequence and the effective target mask sequence; constructing a target feature sequence based on the global contextual feature information and the local feature sequence; and recognizing and generating the recognition results of each indicator light in the target image based on the target feature sequence and the effective target mask sequence. This method extracts global contextual features and local features of each indicator light in the target image, constructing target feature information. Based on this target feature information, it can jointly analyze the relationships between indicator lights and the contextual information of each indicator light in the target image, generating recognition results for each indicator light. This reduces misjudgments caused by reflections, exposure changes, occlusion, background interference, or insufficient cropping information when relying solely on the local image information of a single indicator light. It effectively combines the status of other indicator lights around the indicator light, device layout, and overall image information, thus significantly improving the accuracy and stability of the recognition results. Furthermore, by uniformly setting a preset recognition threshold for the length of the generated indicator light image sequence, it addresses the technical challenge of batch training and inference for variable-length multi-target samples when the number of indicator lights in a single image is not fixed. This effectively supports batch processing of multiple images with varying numbers of indicator lights.

[0225] Secondly, by utilizing pre-trained visual encoders and visual cue tuning mechanisms, task transfer can be performed with the visual backbone parameters frozen or partially frozen, reducing the amount of data and training costs required to train the model from scratch. This improves the transfer efficiency of local and global visual encoders in small-sample industrial scenarios while ensuring the accuracy of feature extraction.

[0226] In addition, by using a relational encoder to enhance the local feature information of each indicator light, the layout relationship and status association between multiple indicator lights on the same device panel or in the same image, as well as between each indicator light and the entire image, are calculated. This allows the enhanced feature information of each indicator light to be effectively integrated with global information and information from other indicator lights, thereby improving the accuracy of indicator light status recognition.

[0227] By encapsulating the indicator light recognition results and the original information into a structured result, it is easy to link the monitoring platform, alarm system, operation and maintenance system or manual review system.

[0228] The following describes the apparatus, device, and storage medium used to implement the indicator light status recognition method provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0229] Figure 8 This is a schematic diagram of an indicator light status recognition device provided in an embodiment of this application. The function implemented by this indicator light status recognition device corresponds to the steps performed by the above-described method. This device can be understood as the aforementioned server, or the server's processor, or it can be understood as a component that implements the functions of this application under the control of the server, independent of the aforementioned server or processor. Figure 8 As shown, the device includes: an extraction module 100, a generation module 200, a construction module 300, and an identification module 400; Extraction module 100 is used to extract the region images of each indicator light from the target image to be identified; The generation module 200 is used to generate an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence based on the regional images of each indicator light, the sorting strategy of each indicator light, and a preset recognition quantity threshold. The effective target mask sequence contains multiple mask values ​​arranged in order, and the mask values ​​are used to indicate whether the image at the corresponding position in the indicator light image sequence is an indicator light region image. The extraction module 100 is used to extract global contextual feature information of the target image based on the target image; Extraction module 100 is used to extract local feature sequences based on the indicator light image sequence and the effective target mask sequence; Module 300 is used to construct the target feature sequence based on global context feature information and local feature sequences; The recognition module 400 is used to identify and generate the recognition results of each indicator light in the target image based on the target feature sequence and the effective target mask sequence. The recognition results include at least the on / off state of the indicator light.

[0230] Optionally, the generation module 200 is specifically used to determine the arrangement order of the regional images of each indicator light based on the spatial arrangement information of each indicator light in the target image; The sequence length of the initial indicator light image sequence is determined based on a preset recognition threshold. According to the arrangement order of the area images of each indicator light, the area images of each indicator light are added to the initial indicator light image sequence in sequence, and blank images are filled in the empty positions in the initial indicator light image sequence to obtain the indicator light image sequence; Based on the image information at each position in the indicator light image sequence, a valid target mask sequence is generated.

[0231] Optionally, the extraction module 100 is specifically used to: input the target image into the global visual encoder, and have the global visual encoder identify and extract the initial contextual feature information of the target image; The initial context feature information is projected to map the initial context feature information to the target dimension to obtain global context feature information. The global context feature information includes at least: the illumination information of the target image, the device panel layout information of the target image, the exposure information of the target image, the shooting angle information of the target image, the background information of the target image, and the context association information between the indicator lights in the target image.

[0232] Optionally, the extraction module 100 is specifically used to: input the indicator light image sequence and the effective target mask sequence into the local visual encoder, and the local visual encoder determines the region image of each effective indicator light and each invalid blank image in the indicator light image sequence according to the effective target mask sequence; the local visual encoder optimizes the pre-trained visual model based on preset prompt words; the prompt words are used to prompt additional attention information when recognizing the region image of the indicator light; Based on the area images of each valid indicator light, identify and extract the local feature information of each indicator light; Based on each invalid blank image, determine the occupancy feature information of each invalid blank image; A local feature sequence is generated based on the local feature information of each indicator light and the occupancy feature information of each invalid blank image; the local feature information includes at least the color, brightness and edge information of the indicator light.

[0233] Optionally, the construction module 300 is specifically used to: concatenate the global context feature information and the feature information in the local feature sequence in a preset order to generate an initial feature sequence; Positional encoding information is added to the information at each position in the initial feature sequence to obtain the target feature sequence.

[0234] Optionally, the recognition module 400 is specifically used to: input the target feature sequence and the effective target mask sequence into the relation encoder, and the relation encoder determines the first association relationship between the global context feature information and the local feature information of each indicator light in the target feature sequence, and the second association relationship between the local feature information of each indicator light, and determines the enhanced local feature information of each indicator light based on the first association relationship and the second association relationship; the enhanced local feature information of the indicator light incorporates the global context feature information and the local feature information of other indicator lights besides the indicator light itself; The enhanced local feature information of each indicator light is input into a pre-trained state prediction model, which then predicts and generates the recognition results for each indicator light.

[0235] Optionally, a training module may also be included; The training module is used by the relation encoder to calculate loss information based on the prediction results of the valid sample indicator lights and the actual state labels of the valid sample indicator lights during the training process.

[0236] Optionally, the generation module 200 is further configured to: generate and output alarm prompt information if the recognition results of the target indicator light in the recognition results of multiple consecutive frames of the target image all indicate abnormality and the confidence level of the target indicator light status recognition is greater than a preset threshold.

[0237] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0238] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0239] The modules described above can be connected or communicate with each other via wired or wireless connections. Wired connections may include metal cables, optical fibers, hybrid cables, or any combination thereof. Wireless connections may include connections via LAN, WAN, Bluetooth, ZigBee, or NFC, or any combination thereof. Two or more units can be combined into a single module, and any module can be divided into two or more units. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here.

[0240] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The device may be a computing device with data processing capabilities.

[0241] The device includes a processor 801, a storage medium 802, and a bus 803. The storage medium 802 stores program instructions that can be executed by the processor 801. When the electronic device is running, the processor 801 communicates with the storage medium 802 through the bus 803. The processor 801 executes the program instructions to implement the indicator light status recognition method as described in the embodiment.

[0242] The storage medium 802 stores program code, which, when executed by the processor 801, causes the processor 801 to perform various steps in the indicator light status recognition method according to various exemplary embodiments of this application as described in the "Exemplary Methods" section above.

[0243] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0244] Storage medium 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The storage medium can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type storage medium, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage medium, magnetic disk, optical disk, etc. The storage medium is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, storage medium 802 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0245] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs the above-described method embodiments.

[0246] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0247] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0248] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0249] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for recognizing the status of an indicator light, characterized in that, include: Extract the region images of each indicator light from the target image to be identified; Based on the regional images of each indicator light, the sorting strategy of each indicator light, and the preset recognition quantity threshold, an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence are generated; the effective target mask sequence contains multiple mask values ​​arranged in order, and each mask value is used to indicate whether the image at the corresponding position in the indicator light image sequence is an indicator light regional image; Based on the target image, extract the global contextual feature information of the target image; Based on the indicator light image sequence and the effective target mask sequence, extract local feature sequences; Based on the global context feature information and the local feature sequence, a target feature sequence is constructed; Based on the target feature sequence and the effective target mask sequence, the identification results of each indicator light in the target image are identified and generated, and the identification results include at least the on / off state of the indicator light.

2. The method according to claim 1, characterized in that, The step of generating an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence based on the regional images of each indicator light, the sorting strategy of each indicator light, and a preset recognition quantity threshold includes: Based on the spatial arrangement information of each indicator light in the target image, determine the arrangement order of the regional images of each indicator light; The sequence length of the initial indicator light image sequence is determined based on the preset recognition quantity threshold. According to the arrangement order of the area images of each indicator light, the area images of each indicator light are added to the initial indicator light image sequence in sequence, and blank images are filled in the empty positions in the initial indicator light image sequence to obtain the indicator light image sequence; The effective target mask sequence is generated based on the image information at each position in the indicator light image sequence.

3. The method according to claim 1, characterized in that, The step of extracting global contextual feature information of the target image based on the target image includes: The target image is input into a global visual encoder, which then identifies and extracts the initial contextual feature information of the target image. The initial context feature information is projected to map it to the target dimension to obtain the global context feature information. The global context feature information includes at least: the illumination information of the target image, the device panel layout information of the target image, the exposure information of the target image, the shooting angle information of the target image, the background information of the target image, and the context association information between the indicator lights in the target image.

4. The method according to claim 1, characterized in that, The step of extracting local feature sequences based on the indicator light image sequence and the effective target mask sequence includes: The indicator light image sequence and the effective target mask sequence are input into a local visual encoder. The local visual encoder determines the region image of each effective indicator light and the invalid blank image in the indicator light image sequence according to the effective target mask sequence. The local visual encoder is obtained by tuning the pre-trained visual model based on preset prompt words. The prompt words are used to provide additional attention information when recognizing the region image of the indicator light. Based on the area images of each valid indicator light, identify and extract the local feature information of each indicator light; Based on each invalid blank image, determine the occupancy feature information of each invalid blank image; The local feature sequence is generated based on the local feature information of each indicator light and the occupancy feature information of each invalid blank image; the local feature information includes at least the color, brightness and edge information of the indicator light.

5. The method according to claim 1, characterized in that, The step of constructing the target feature sequence based on the global context feature information and the local feature sequence includes: The global context feature information and the feature information in the local feature sequence are concatenated in a preset order to generate an initial feature sequence; Positional encoding information is added to the information at each position in the initial feature sequence to obtain the target feature sequence.

6. The method according to claim 1, characterized in that, The step of identifying and generating the recognition results of each indicator light in the target image based on the target feature sequence and the effective target mask sequence includes: The target feature sequence and the effective target mask sequence are input into a relation encoder. The relation encoder determines a first association between the global context feature information and the local feature information of each indicator light in the target feature sequence, and a second association between the local feature information of each indicator light. Based on the first association and the second association, the enhanced local feature information of each indicator light is determined. The enhanced local feature information of the indicator light incorporates the global context feature information and the local feature information of other indicator lights besides the indicator light itself. The enhanced local feature information of each indicator light is input into the state prediction model, and the state prediction model predicts and generates the recognition result of each indicator light.

7. The method according to claim 6, characterized in that, During training, the relation encoder calculates loss information based on the prediction results of the valid sample indicator lights and the actual state labels of the valid sample indicator lights.

8. The method according to claim 1, characterized in that, Also includes: If the recognition results of the target indicator light in multiple consecutive frames of the target image all indicate an abnormality and the confidence level of the target indicator light's status recognition is greater than a preset threshold, then an alarm message is generated and output.

9. A device for recognizing the status of an indicator light, characterized in that, include: Extraction module, generation module, construction module, and recognition module; The extraction module is used to extract the area images of each indicator light from the target image to be identified; The generation module is used to generate an indicator light image sequence corresponding to the target image and an effective target mask sequence corresponding to the indicator light image sequence based on the regional images of each indicator light, the sorting strategy of each indicator light, and a preset recognition quantity threshold; the effective target mask sequence contains multiple mask values ​​arranged in order, and each mask value is used to indicate whether the image at the corresponding position in the indicator light image sequence is an indicator light region image; The extraction module is used to extract global contextual feature information of the target image based on the target image; The extraction module is used to extract local feature sequences based on the indicator light image sequence and the effective target mask sequence; The construction module is used to construct a target feature sequence based on the global context feature information and the local feature sequence; The recognition module is used to identify and generate the recognition results of each indicator light in the target image based on the target feature sequence and the effective target mask sequence. The recognition results include at least the on / off state of the indicator light.

10. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to implement the method as described in any one of claims 1 to 8.

Images