Warning lamp identification method, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of artificial intelligence, and provides a warning lamp recognition method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target shot image and a target thermal imaging image of a warning lamp; inputting the target shooting image and the target thermal imaging image into a trained caution light identification model; the method comprises the following steps: positioning according to a target shooting image to obtain a target area image comprising a warning lamp; correcting the target area image according to the target thermal imaging image to obtain a corrected target area image; and based on the corrected target area image, the recognition result of the warning lamp is determined, and the accuracy of warning lamp recognition is improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and particularly relates to a warning light recognition method, an electronic device, and a storage medium. Background Art

[0002] A warning light (or an indicator light) is a key indicating device that can intuitively reflect the operating state of a device, and is of great significance for timely detecting and handling faults and ensuring the safe operation of the device. However, traditional warning light recognition methods mainly rely on manual inspections, which have problems such as low efficiency, easy errors, and poor real-time performance.

[0003] With the rapid development of artificial intelligence technology, intelligent recognition methods based on image processing and computer vision technology have gradually been applied to the field of warning light recognition. Common methods include: constructing a warning light recognition detection model for identifying the features of the warning light, where the features of the warning light include the position feature and the shape feature of the warning light; using a camera device to capture an image containing the device to be detected; using the warning light recognition detection model to recognize the position and features of the warning light of the device to be detected in the captured image, and extracting the image part containing only the warning light of the device to be detected from the image containing the device to be detected; performing image processing on the image part containing only the warning light of the device to be detected to identify the state of the warning light of the device to be detected. This method only uses the image containing the device to be detected captured by the camera device to recognize the position and features of the warning light of the device to be detected in this image, and extracts the image part containing only the warning light of the device to be detected. However, when the image containing the device to be detected captured by the camera device is unclear or there are obstacles, it will cause errors in the position and features of the warning light of the device to be detected in the recognized image, thereby making it impossible to accurately extract the image part of the warning light of the device to be detected, reducing the accuracy of warning light recognition.

[0004] Therefore, how to improve the accuracy of warning light recognition has become an urgent problem to be solved. Summary of the Invention

[0005] Embodiments of this application provide a warning light recognition method, an electronic device, and a storage medium, aiming to improve the accuracy of warning light recognition.

[0006] In a first aspect, embodiments of this application provide a warning light recognition method, and the method includes:

[0007] Obtain a target captured image and a target thermal imaging image of the warning light;

[0008] Input the target captured image and the target thermal imaging image into the trained warning light recognition model; first, based on the target captured image, locate the target region image including the warning light; then, based on the target thermal imaging image, correct the target region image to obtain the corrected target region image;

[0009] Based on the corrected target region image, determine the recognition result of the warning light.

[0010] In a possible implementation, the warning light recognition model includes a model input module, a target detection module, a semantic segmentation module, and a model output module;

[0011] The model input module is used to input the target captured image and the target thermal imaging image;

[0012] The target detection module is connected to the model input module and is used to extract the first target region image including the warning light according to the target captured image;

[0013] The semantic segmentation module is connected to the model input module and the target detection module and is used to extract the second target region image including the warning light as the corrected target region image according to the first target region image and the target thermal imaging image; the area of the second target region image is smaller than the area of the first target region image;

[0014] The model output module is connected to the output of the semantic segmentation module and is used to output the corrected target region image.

[0015] In a possible implementation, the training process of the warning light recognition model includes:

[0016] Obtain a set of warning light captured images and a set of warning light thermal imaging images;

[0017] Based on the set of warning light captured images, construct and train a convolutional neural network model to obtain the target detection module; based on the set of warning light captured images and the set of warning light thermal imaging images, construct and train a UNet model to obtain the semantic segmentation module;

[0018] Based on the target detection module and the semantic segmentation module, construct the warning light recognition model.

[0019] In a possible implementation, the constructing and training a convolutional neural network model based on the set of warning light captured images to obtain the target detection module; constructing and training a UNet model based on the set of warning light captured images and the set of warning light thermal imaging images to obtain the semantic segmentation module includes:

[0020] Label the warning light areas in each warning light captured image in the warning light captured image set based on a preset area reference value and / or a preset aspect ratio reference value to obtain a first labeled information set;

[0021] Label the areas in each warning light captured image in the warning light captured image set that only contain warning lights, and label the areas corresponding to the warning lights in each warning light thermal imaging map in the warning light thermal imaging map set to obtain a second labeled information set;

[0022] Construct a first sample data set based on the warning light captured image set and the first labeled information set; construct a second sample data set based on the warning light captured image set, the warning light thermal imaging map set, and the second labeled information set;

[0023] Construct and train a convolutional neural network model based on the first sample data set to obtain the target detection module;

[0024] Construct and train a UNet model based on the second sample data set to obtain the semantic segmentation module.

[0025] In a possible implementation, the semantic segmentation module includes: two feature extraction units arranged in parallel; and a feature fusion unit;

[0026] The feature extraction unit includes an input layer, a number of convolutional blocks, and a number of convolutional layers connected in sequence, and is respectively used to extract the features of the first target area image and the target thermal imaging image;

[0027] The feature fusion unit includes a number of upsampling layers with the same number of levels as the feature extraction unit. Each upsampling layer takes the outputs of the same level in the two feature extraction units as inputs; and the output of each upsampling layer is used as the input of the upper upsampling layer from bottom to top until the output of the uppermost upsampling layer is connected to the output layer to output the total fused feature.

[0028] In a possible implementation, the semantic segmentation module includes:

[0029] The first feature extraction unit includes a first input layer, a first convolutional block, a second convolutional block, a first channel convolutional layer, a first pointwise convolutional layer, and a first dilated convolutional layer connected in sequence, and is used to extract the features of the first target area image;

[0030] The second feature extraction unit includes a second input layer, a third convolutional block, a fourth convolutional block, a second channel convolutional layer, a second pointwise convolutional layer, and a second dilated convolutional layer connected in sequence, and is used to extract the features of the target thermal imaging image;

[0031] A feature fusion unit, the outputs of the first dilated convolutional layer and the second dilated convolutional layer are connected to the input of the first upsampling layer; the outputs of the first pointwise convolutional layer, the second pointwise convolutional layer, and the first upsampling layer are connected to the input of the second upsampling layer; the outputs of the first channel convolutional layer, the second channel convolutional layer, and the second upsampling layer are connected to the input of the third upsampling layer, the outputs of the second convolutional block, the fourth convolutional block, and the third upsampling layer are connected to the input of the fourth upsampling layer, and the outputs of the first convolutional block, the third convolutional block, and the fourth upsampling layer are connected to the output layer for outputting the overall fused features.

[0032] In a possible implementation, the recognition result includes the on / off state of the warning light; determining the recognition result of the warning light based on the corrected target region image includes:

[0033] Based on the corrected target region image and the target camera device, determining the brightness value corresponding to the corrected target region image, where the target camera device is the camera device that captures the target captured image;

[0034] Based on the brightness value corresponding to the corrected target region image and a preset brightness threshold, determining the on / off state of the warning light, where the on / off state includes a bright state and an off state.

[0035] In a possible implementation, the recognition result further includes the color of the warning light; when determining that the on / off state of the warning light is the bright state, determining the recognition result of the warning light based on the corrected target region image further includes:

[0036] Based on the target camera device, determining a target color display mode, and based on the target color display mode, determining a target color display value corresponding to the corrected target region image, where the target color display mode is any one of the RGB mode, the HSV mode, and the CMYK mode;

[0037] Based on the target color display value and a preset color threshold corresponding to the target color display mode, determining the color of the warning light.

[0038] In a possible implementation, after determining the recognition result of the warning light based on the corrected target region image, the method further includes:

[0039] Based on the recognition result of the warning light, determining the state of the device where the warning light is located.

[0040] Second aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in the first aspect or any one of its implementation manners is implemented.

[0041] Third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method described in the first aspect or any one of its implementation manners is implemented.

[0042] Fourth aspect, an embodiment of the present application provides a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect or any one of its implementation manners are implemented.

[0043] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: The target captured image and the target thermal imaging image of the warning light are input into the trained warning light recognition model. First, based on the target captured image, the target region image including the warning light is located. Then, based on the target thermal imaging image, the target region image is corrected to obtain the corrected target region image. Based on the corrected target region image, the recognition result of the warning light is determined, that is, on the basis of the traditional method of only locating the target region image according to the target captured image, it is corrected by the target thermal imaging image. Since the target thermal imaging image can capture the infrared radiation emitted by the device where the warning light is located and provide the temperature distribution information of the device, the temperature of the warning light is higher than other regions due to its emission of light, so its color in the target thermal imaging image is significantly different from other regions, and its contour can be clearly shown through the color difference, which helps to more accurately segment the boundary of the warning light. In addition, the thermal imaging map is not affected by factors such as changes in light, occlusion, and background interference, and can provide reliable semantic segmentation results in various environments, improving the accuracy of determining the target region image, and thus improving the accuracy of subsequent warning light recognition. It should be noted that the key of the present application lies in how to correct and obtain a target region image with higher accuracy, and how to perform recognition through the determined target region image can adopt any method in the prior art.

[0044] It can be understood that the electronic device, computer-readable storage medium, and computer program product provided by the embodiments of the present application have the same beneficial effects as the above-mentioned warning light recognition method, and will not be elaborated here. Description of the Drawings

[0045] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for use in the embodiments or the description of the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0046] Figure 1 A flowchart of a warning light recognition method provided by an embodiment of the present application;

[0047] Figure 2 A structural diagram of a warning light recognition model provided by an embodiment of the present application;

[0048] Figure 3 A structural diagram of a semantic segmentation module provided by an embodiment of the present application;

[0049] Figure 4 A structural diagram of a convolutional block provided by an embodiment of the present application;

[0050] Figure 5 Another structural diagram of a convolutional block provided by an embodiment of the present application;

[0051] Figure 6 A structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0052] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0053] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0054] It should also be understood that the term " / and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0055] As used in the specification of this application and the appended claims, the term "if" may be construed contextually as "when", "once", "in response to determining", or "in response to detecting". Similarly, the phrases "if determined" or "if [the described condition or event] is detected" may be construed contextually to mean "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".

[0056] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used for differential description and cannot be construed as indicating or implying relative importance.

[0057] Reference to "one embodiment" or "some embodiments" etc. described in the specification of this application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0058] For ease of understanding, the technical solutions of this application will be introduced in detail below in conjunction with the accompanying drawings.

[0059] Figure 1 The figure is a schematic flowchart of a warning light recognition method provided for an embodiment of this application, which is optionally but not limited to being used for the recognition of many warning lights in a power distribution room. For ease of explanation, only the parts related to this embodiment are shown. The method provided in this embodiment includes the following steps:

[0060] S110, obtain a target captured image and a target thermal imaging image of the warning light.

[0061] Specifically, obtain a target captured image and a target thermal imaging image of the warning light on the target switch cabinet to be recognized in the power distribution room. Among them, the target switch cabinet is any switch cabinet in the power distribution room.

[0062] In a specific implementation, optionally, a target captured image and a target thermal imaging image of a target switch cabinet in a power distribution room are obtained through a power distribution room inspection robot. The power distribution room inspection robot is equipped with a target imaging device, an infrared sensor, etc. When the power distribution room inspection robot travels according to a preset inspection path of the current power distribution room, the target captured image and the target thermal imaging image are respectively collected at a preset capture position through the loaded target imaging device and infrared sensor, so as to comprehensively capture information on the environment around the target switch cabinet under different lighting conditions and obstacle occlusion situations.

[0063] S120, input the target captured image and the target thermal imaging image into the trained warning light recognition model; first, based on the target captured image, locate the target area image including the warning light; then, based on the target thermal imaging image, correct the target area image to obtain the corrected target area image.

[0064] Specifically, optionally, a number of target captured images and target thermal imaging images are collected, the warning light area is marked, input into a pre-constructed warning light recognition model, and it is trained to adjust various parameters to obtain the trained warning light recognition model; then, the currently obtained target captured image and target thermal imaging image are input into the trained warning light recognition model to obtain the target area image including the warning light.

[0065] Specifically, in a possible implementation manner, the structure of the warning light recognition model may optionally include:

[0066] A model input module for inputting the target captured image and the target thermal imaging image;

[0067] A target detection module connected to the model input module for extracting the first target area image including the warning light according to the target captured image;

[0068] A semantic segmentation module connected to the model input module and the target detection module for extracting the second target area image including the warning light as the corrected target area image according to the first target area image and the target thermal imaging image; the area of the second target area image is smaller than the area of the first target area image;

[0069] A model output module connected to the output of the semantic segmentation module for outputting the corrected target area image.

[0070] More specifically, the target detection module may optionally be a convolutional neural network model; the semantic segmentation module may optionally be a UNet model.

[0071] More specifically, in a possible implementation manner, the training process of the warning light recognition model includes: obtaining a set of warning light captured images and a set of warning light thermal images; constructing and training a convolutional neural network model based on the set of warning light captured images to obtain a target detection module; constructing and training a UNet model based on the set of warning light captured images and the set of warning light thermal images to obtain a semantic segmentation module; and constructing a warning light recognition model based on the target detection module and the semantic segmentation module.

[0072] In specific implementation, a large number of various warning light images captured by different camera devices at different shooting angles, lighting conditions, and backgrounds are collected. After denoising the collected warning light images, a set of warning light captured images is constructed; at the same time, a large number of thermal images collected at different positions in front of each device equipped with a warning light are collected to construct a set of warning light thermal images.

[0073] As an example, the steps of constructing and training a convolutional neural network model based on the set of warning light captured images to obtain a target detection module include: annotating the warning light regions in each warning light captured image in the set of warning light captured images based on a preset area reference value and / or a preset aspect ratio reference value to obtain a first set of annotation information; constructing a first set of sample data based on the set of warning light captured images and the first set of annotation information; and constructing and training a convolutional neural network model based on the first set of sample data to obtain a target detection module.

[0074] Preferably, a rectangular box is usually used to annotate the warning light regions in each warning light captured image in the set of warning light captured images to obtain a first set of annotation information. Among them, the preset area reference value and the preset aspect ratio reference value corresponding to the rectangular box used for annotation can be determined according to the size of the warning lights in each warning light captured image. For example, the preset area reference value is greater than or equal to the area of the minimum circumscribed rectangle corresponding to the largest warning light in each warning light captured image, and the aspect ratio of the minimum circumscribed rectangle is used as the preset aspect ratio reference value; the set of warning light captured images and the first set of annotation information are constructed into a first set of sample data; the first set of sample data is divided into a test set and a validation set according to a preset ratio. The convolutional neural network model is trained using the test set, and the performance of the model, including indicators such as accuracy and recall rate, is evaluated using the validation set. The trained convolutional neural network model is used as the target detection module.

[0075] In a possible implementation manner, after denoising and other processing of the target captured image, it is input into the target detection module in the trained warning light recognition model to obtain a first target region image.

[0076] As an example, the first target region image is an image obtained by annotating the region where the warning light is located in the target captured image using the rectangular box used during the training of the target detection module.

[0077] Exemplarily, in the first target area image, the area corresponding to the rectangular frame where the warning light is located is marked in the form of a mask.

[0078] In a possible implementation manner, based on the warning light image set and the warning light thermal imaging map set, the steps of constructing and training a UNet model to obtain a semantic segmentation module include: marking the areas containing only the warning lights in each warning light image in the warning light image set, and marking the areas corresponding to the warning lights in each warning light thermal imaging map in the warning light thermal imaging map set, to obtain a second annotation information set; constructing a second sample data set based on the warning light image set, the warning light thermal imaging map set and the second annotation information set; constructing and training a UNet model based on the second sample data set to obtain a semantic segmentation module.

[0079] Specifically, the UNet model includes an encoder (downsampling path), a decoder (upsampling path) and skip connections. The encoder (downsampling path) is used to extract the features of the input image, gradually reduce the spatial dimension and increase the number of feature channels; the decoder (upsampling path) is used to restore the spatial resolution of the image and generate the final segmentation result; the skip connections connect the output of each layer in the encoder to the input of the corresponding layer in the decoder to ensure that the model retains more detailed information when restoring the image resolution. This connection allows the network to refer to the features in the encoder part when performing upsampling, so as to better restore the high-resolution features.

[0080] In a specific implementation, the mask technology is used to mark the areas containing only the warning lights in each warning light image in the warning light image set, and mark the areas corresponding to the warning lights in each warning light thermal imaging map in the warning light thermal imaging map set, to obtain a second annotation information set. The second annotation information set is divided into a training set, a validation set and a test set according to a preset ratio. The training set is used for model training, the validation set is used for model evaluation and parameter tuning, and the test set is used for the final model performance evaluation; set model training parameters such as batch size, learning rate, number of training epochs, etc., input the training set into the model to obtain the model prediction result, use the loss function to calculate the difference between the model prediction result and the true label, calculate the gradient according to the loss value, use the optimizer to update the model parameters. After each training epoch ends, use the validation set to evaluate the model performance, adjust the training parameters or the model structure according to the evaluation result. When the model reaches the expected performance, the model training is completed to obtain a semantic segmentation module.

[0081] Specifically, the semantic segmentation module may optionally include: two feature extraction units arranged in parallel; and a feature fusion unit;

[0082] The feature extraction unit includes an input layer, a number of convolutional blocks, and a number of convolutional layers connected in sequence, which are respectively used to extract the features of the first target region image and the target thermal imaging image;

[0083] The feature fusion unit includes a number of upsampling layers with the same number of levels as the feature extraction unit. Each upsampling layer takes the outputs of the same level in the two feature extraction units as inputs; and the output of each upsampling layer is used as the input of the upper upsampling layer from bottom to top until the output of the topmost upsampling layer is connected to the output layer to output the total fused features.

[0084] As an example, such as Figure 3 shown, the semantic segmentation module may optionally include a first input layer, a first convolutional block, a second convolutional block, a first channel convolutional layer, a first pointwise convolutional layer, and a first atrous convolutional layer connected in sequence, which are used to extract the features of the first target region image; it also includes a second input layer, a third convolutional block, a fourth convolutional block, a second channel convolutional layer, a second pointwise convolutional layer, and a second atrous convolutional layer connected in sequence, which are used to extract the features of the target thermal imaging image; the outputs of the first atrous convolutional layer and the second atrous convolutional layer are connected to the input of the first upsampling layer; the outputs of the first pointwise convolutional layer, the second pointwise convolutional layer, and the first upsampling layer are connected to the input of the second upsampling layer; the outputs of the first channel convolutional layer, the second channel convolutional layer, and the second upsampling layer are connected to the input of the third upsampling layer, the outputs of the second convolutional block, the fourth convolutional block, and the third upsampling layer are connected to the input of the fourth upsampling layer, and the outputs of the first convolutional block, the third convolutional block, and the fourth upsampling layer are connected to the output layer.

[0085] As an example, the structures of the first convolutional block and the third convolutional block are as Figure 4 shown, including a first depthwise separable convolutional layer and a second depthwise separable convolutional layer; the structures of the second convolutional block and the fourth convolutional block are as Figure 5 shown, including a third atrous convolutional layer and a third depthwise separable convolutional layer.

[0086] Exemplarily, the depthwise separable convolutional layer decouples the spatial convolution of the convolutional kernel from the convolution between channels. The basic process of depthwise separable convolution includes: 1. Applying convolution operations to each input channel separately through depthwise convolution; 2. Combining the outputs of different channels through pointwise convolution (usually 1x1 convolution) to enhance the feature representation ability. This decomposition significantly reduces the number of model parameters and computational complexity while maintaining or even improving the model's accuracy.

[0087] Exemplarily, the above 5 upsampling layers usually use methods such as bilinear interpolation to increase the resolution of the feature map to the size of the original input image. The purpose of upsampling is to restore the lower-resolution encoder output to a higher resolution for further fusion and processing.

[0088] In a possible implementation, the first target region image of the rectangular frame area where the warning light is located in the target captured image and the target thermal image are input into the semantic segmentation module in the warning light recognition model to obtain a second target region image that only contains the warning light area, that is, the corrected target region image.

[0089] As an example, the second target region image is a circular area that only contains the warning light and is segmented from the first target region image.

[0090] S130. Based on the corrected target region image, determine the recognition result of the warning light.

[0091] In a possible implementation, the recognition result includes the on / off state of the warning light. Based on the corrected target region image and the target imaging device, determine the brightness value corresponding to the corrected target region image. The target imaging device is the imaging device that captures the target captured image. Based on the brightness value corresponding to the corrected target region image and a preset brightness threshold, determine the on / off state of the warning light. The on / off state includes the bright state and the off state.

[0092] As an example, directly or indirectly determine the brightness value of the corrected target region image through the device parameters of the target imaging device. When the brightness value is greater than or equal to the preset brightness threshold, determine that the on / off state of the warning light is the bright state. When the brightness value is less than the preset brightness threshold, determine that the on / off state of the warning light is the off state.

[0093] As another example, convert the corrected target region image into a grayscale image, and calculate the average brightness value of the grayscale image as the brightness value of the corrected target region image.

[0094] In a possible implementation, the recognition result also includes the color of the warning light. When it is determined that the on / off state of the warning light is the bright state, then determine the target color display mode based on the target imaging device, and determine the target color display value corresponding to the corrected target region image based on the target color display mode. The target color display mode is any one of the RGB mode, the HSV mode, and the CMYK mode. Based on the target color display value and the preset color threshold corresponding to the target color display mode, determine the color of the warning light.

[0095] In a specific implementation, the circular region in the segmentation map output by the warning light recognition model is converted to 1, and other regions are 0 as a mask, which is multiplied by the target captured image, and then its state is determined according to its pixel values, etc. The color display modes of different imaging devices are different. The color display mode of the target captured image is determined according to the device parameters of the target imaging device. The target color display value corresponding to the corrected target region image is determined based on the target color display mode, and then the color of the warning light is determined based on the target color display value and the preset color threshold corresponding to the target color display mode.

[0096] Exemplarily, when the color display mode is the RGB mode, the preset color threshold for red has an RGB value of (255, 0, 0), and the hexadecimal code (Hexadecimal Encoding, HEX) is #FF0000; the RGB value for green is (0, 255, 0), and the HEX code is #00FF00.

[0097] Exemplarily, when the color display mode is the HSV mode, the preset color threshold for red has a hue value of 0° or 360°; the hue value for green is 120°.

[0098] Exemplarily, when the color display mode is the CMYK mode, the preset color threshold for red has CMYK values where the values of cyan (C), magenta (M), and yellow (Y) are (0%, 100%, 100%), and the value of black (K) is 0%; the CMYK value for green has a value of cyan (C) of 100%, and the values of magenta (M) and yellow (Y) are 0%, and the value of black (K) is 0%.

[0099] In this embodiment, the error caused by the conversion between different color display modes is avoided, and the accuracy of determining the color of the warning light is improved.

[0100] In a possible implementation manner, after determining the recognition result of the warning light based on the corrected target region image and the target imaging device, the state of the device where the warning light is located is determined based on the recognition result of the warning light.

[0101] In a specific implementation, when it is determined that the on-off state of the warning light is the off state, it is determined that the device where the warning light is located is in an abnormal state; when it is determined that the on-off state of the warning light is the bright state, the color of the warning light is further determined. When the color of the warning light is green, it indicates that the state of the device where the warning light is located is normal; when the color of the warning light is red, it indicates that the state of the device where the warning light is located is abnormal.

[0102] As an example, in a power distribution room, based on the recognition result of the warning light, the inspection robot in the power distribution room can make corresponding decisions immediately, such as pressing the correct switch or taking appropriate obstacle avoidance actions. With the help of the path planning algorithm, the inspection robot in the power distribution room can shuttle freely in a complex space and perform various tasks. In addition, in cooperation with the object tracking algorithm, even in a dynamically changing scene, it can continuously track the warning light. Moreover, the inspection robot in the power distribution room can establish a communication connection with the cloud platform, and upload the recognition result of the warning light to the cloud in real time, which is convenient for engineers to remotely monitor and diagnose. At the same time, the cloud can also provide the robot with the latest instructions and policy updates to achieve intelligent collaborative operation. When the switchgear is in the normal / abnormal working state, the warning light will change. During the inspection process, the robot judges whether the switchgear is working normally by detecting the warning light and combining the indicator light image recognition technology, which shortens the time to discover equipment failures in the power distribution room and improves the efficiency of fault handling.

[0103] For the technical solution provided by this application, the target captured image and the target thermal imaging image of the warning light are input into the trained warning light recognition model. First, based on the target captured image, the target region image including the warning light is located. Then, according to the target thermal imaging image, the target region image is corrected to obtain the corrected target region image. Based on the corrected target region image, the recognition result of the warning light is determined, that is, on the basis of the traditional method of locating the target region image according to the target captured image, it is corrected by the target thermal imaging image. Since the target thermal imaging image can capture the infrared radiation emitted by the equipment where the warning light is located and provide the temperature distribution information of the equipment, the temperature of the warning light is higher than other regions due to its emission of light, so its color in the target thermal imaging map is significantly different from other regions, and its contour can be clearly displayed through the color difference, which helps to more accurately segment the boundary of the warning light. In addition, the thermal imaging map is not affected by factors such as changes in illumination, occlusion, and background interference, and can provide reliable semantic segmentation results in various environments, improving the accuracy of determining the target region image, and thus improving the accuracy of subsequent warning light recognition.

[0104] It should be noted that the key of this application lies in how to correct and obtain a target region image with higher accuracy, and as for how to perform recognition through the determined target region image, any method in the prior art can be adopted, and this application will not elaborate on it.

[0105] In summary, introducing the warning light recognition method provided by this application into the power distribution room has significant benefits. First, by automatically recognizing the status of warning lights through a robot, real-time monitoring of the operation of power distribution room equipment can be achieved, abnormal signals can be detected in a timely manner, the frequency and work intensity of manual inspections can be reduced, and the operation and maintenance efficiency can be improved. Second, the accuracy and stability of the robot recognition technology are relatively high, which can effectively avoid potential safety hazards caused by manual judgment errors and ensure the safe operation of the power distribution room. In addition, this application can also be combined with the intelligent management system of the power distribution room to achieve automatic alarm and fault location, providing strong support for quickly responding to and handling faults, further improving the intelligent management level of the power distribution room, reducing operation and maintenance costs, and providing a reliable guarantee for the stable operation of the power system. It not only greatly improves the autonomous operation ability of the robot in the industrial field, but also provides strong technical support for the transformation of factory automation and intelligence, helping to improve production efficiency and operation safety.

[0106] Figure 6 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 6 shown, the electronic device 6 of this embodiment includes: at least one processor 60 ( Figure 6 only one is shown in the figure), a memory 61, and a computer program 62 stored in the memory 61 and executable on at least one processor 60. When the processor 60 executes the computer program 62, the steps in the above Figure 1 method embodiment are implemented.

[0107] The electronic device 6 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The electronic device 6 may include, but is not limited to, the processor 60 and the memory 61. Those skilled in the art can understand that Figure 6 merely examples of the electronic device 6, which do not constitute a limitation on the electronic device 6. It may include more or fewer components than those shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0108] The processor 60 may be a central processing unit (CPU), and the processor 60 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0109] In some embodiments, the memory 61 may be an internal storage unit of the electronic device 6, such as the hard disk or memory of the electronic device 6. In other embodiments, the memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk equipped on the electronic device 6, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 61 may also include both an internal storage unit and an external storage device of the electronic device 6. The memory 61 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of a computer program. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0110] The embodiments of the present application also provide a computer-readable storage medium storing a computer program, which when executed by a processor can implement the steps in the above-mentioned method embodiments.

[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned method embodiments of the present application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code to the electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc.

[0112] The computer-readable storage medium provided by the embodiments of the present application has the same beneficial effects as the above-mentioned warning light recognition method.

[0113] The embodiments of the present application provide a computer program product including a computer program, which when executed by a processor can implement the steps in the above-mentioned method embodiments.

[0114] A computer program product provided by an embodiment of the present application has the same beneficial effects as the above-mentioned warning light recognition method.

[0115] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not described or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0116] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present application.

[0117] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0118] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0119] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A warning light recognition method, characterized in that, The method includes: Obtaining a target captured image and a target thermal imaging image of a warning light; Inputting the target captured image and the target thermal imaging image into a trained warning light recognition model; first, based on the target captured image, locating a target region image including the warning light; then, based on the target thermal imaging image, correcting the target region image to obtain a corrected target region image; Determining an identification result of the warning light based on the corrected target region image.

2. The method according to claim 1, wherein The warning light recognition model includes a model input module, a target detection module, a semantic segmentation module, and a model output module; The model input module is used to input the target captured image and the target thermal imaging image; The target detection module is connected to the model input module and is used to extract a first target region image including the warning light according to the target captured image; The semantic segmentation module is connected to the model input module and the target detection module and is used to extract a second target region image including the warning light as the corrected target region image according to the first target region image and the target thermal imaging image; the area of the second target region image is smaller than the area of the first target region image; The model output module is connected to the output of the semantic segmentation module and is used to output the corrected target region image.

3. The method according to claim 2, wherein The training process of the warning light recognition model includes: Obtaining a warning light captured image set and a warning light thermal imaging image set; Based on the warning light captured image set, constructing and training a convolutional neural network model to obtain the target detection module; based on the warning light captured image set and the warning light thermal imaging image set, constructing and training a UNet model to obtain the semantic segmentation module; Constructing the warning light recognition model based on the target detection module and the semantic segmentation module.

4. The method according to claim 3, wherein The constructing and training a convolutional neural network model based on the warning light captured image set to obtain the target detection module; and constructing and training a UNet model based on the warning light captured image set and the warning light thermal imaging image set to obtain the semantic segmentation module includes: Labeling the warning light regions in each warning light captured image in the warning light captured image set based on a preset area reference value and / or a preset aspect ratio reference value to obtain a first labeling information set; Labeling the regions in each warning light captured image in the warning light captured image set that only contain the warning light, and labeling the regions corresponding to the warning light in each warning light thermal imaging image in the warning light thermal imaging image set to obtain a second labeling information set; Constructing a first sample data set based on the warning light captured image set and the first labeling information set; constructing a second sample data set based on the warning light captured image set, the warning light thermal imaging image set, and the second labeling information set; Constructing and training a convolutional neural network model based on the first sample data set to obtain the target detection module; Based on the second sample data set, a UNet model is constructed and trained to obtain the semantic segmentation module.

5. The method according to claim 2, wherein The semantic segmentation module includes: two feature extraction units arranged in parallel; and a feature fusion unit; The feature extraction unit includes an input layer, a plurality of convolutional blocks, and a plurality of convolutional layers connected in sequence, which are respectively used to extract the features of the first target region image and the target thermal imaging image; The feature fusion unit includes a plurality of upsampling layers with the same number of levels as the feature extraction unit. Each upsampling layer takes the outputs of the same level in the two feature extraction units as inputs; and the output of each upsampling layer is used as the input of the upper upsampling layer from bottom to top until the output of the uppermost upsampling layer is connected to the output layer to output the total fused feature.

6. The method according to claim 5, characterized in that, The semantic segmentation module includes: The first feature extraction unit includes a first input layer, a first convolutional block, a second convolutional block, a first channel convolutional layer, a first pointwise convolutional layer, and a first dilated convolutional layer connected in sequence, which is used to extract the features of the first target region image; The second feature extraction unit includes a second input layer, a third convolutional block, a fourth convolutional block, a second channel convolutional layer, a second pointwise convolutional layer, and a second dilated convolutional layer connected in sequence, which is used to extract the features of the target thermal imaging image; The feature fusion unit, the outputs of the first dilated convolutional layer and the second dilated convolutional layer are connected to the input of the first upsampling layer; the outputs of the first pointwise convolutional layer, the second pointwise convolutional layer, and the first upsampling layer are connected to the input of the second upsampling layer; the outputs of the first channel convolutional layer, the second channel convolutional layer, and the second upsampling layer are connected to the input of the third upsampling layer, the outputs of the second convolutional block, the fourth convolutional block, and the third upsampling layer are connected to the input of the fourth upsampling layer, and the outputs of the first convolutional block, the third convolutional block, and the fourth upsampling layer are connected to the output layer, which is used to output the total fused feature.

7. The method according to any one of claims 1 to 6, characterized in that The recognition result includes the on-off state of the warning light; Determining the recognition result of the warning light based on the corrected target region image includes: Based on the corrected target region image and the target camera device, determining the brightness value corresponding to the corrected target region image, where the target camera device is the camera device that captures the target captured image; Based on the brightness value corresponding to the corrected target region image and a preset brightness threshold, determining the on-off state of the warning light, where the on-off state includes a bright state and an off state.

8. The method according to claim 7, wherein The recognition result further includes the color of the warning light; When it is determined that the on-off state of the warning light is the bright state, determining the recognition result of the warning light based on the corrected target region image further includes: Determining a target color display mode based on the target camera device, and determining a target color display value corresponding to the corrected target region image based on the target color display mode, where the target color display mode is any one of the RGB mode, the HSV mode, and the CMYK mode; Determine the color of the warning light based on the preset color threshold corresponding to the target color display value and the target color display mode.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that When the processor executes the computer program, the method described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method described in any one of claims 1 to 8 is implemented.

Images