A method and device for identifying the state of an indicator light, a terminal device, and a storage medium

Abstract

The application belongs to the technical field of indicator light recognition, and provides a state recognition method and device of an indicator light, a terminal device and a storage medium. The method comprises: acquiring a to-be-recognized image; performing indicator light state detection processing on the to-be-recognized image through a trained target detection network to obtain an indicator light state detection result of the to-be-recognized image; determining an indicator light state review result of the to-be-recognized image according to the image features of the to-be-recognized image and the indicator light state detection result; and outputting the indicator light state review result. The state recognition method of the indicator light in the application can further obtain an indicator light state review result according to the image features of the to-be-recognized image and the obtained indicator light state detection result on the basis of the conventional indicator light state detection result obtained through the target detection network, can reduce the influence of various factors on the accuracy of the state recognition result of the indicator light, and improves the accuracy of the state recognition method of the indicator light.

Description

Technical Field

[0001] This application belongs to the field of indicator light recognition technology, and particularly relates to an indicator light status recognition method, device, terminal equipment and storage medium. Background Technology

[0002] In current indicator light status recognition methods, a trained object detection network is typically used to process the acquired image to be recognized to detect the indicator light status, thereby obtaining the indicator light status detection result.

[0003] However, in practical applications, the following factors can affect the accuracy of the status detection results: reduced indicator light brightness or dirty lampshades leading to decreased distinction between indicator light color and on / off state; the influence of ambient light on the acquired image to be identified; and non-indicator devices such as buttons and screws being mistakenly identified as indicator lights. These factors result in low accuracy for current indicator light status recognition methods. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, terminal device, and storage medium for identifying the status of an indicator light, in order to solve the technical problem of low accuracy in existing indicator light status identification methods.

[0005] In a first aspect, embodiments of this application provide a method for identifying the status of an indicator light, including:

[0006] Acquire the image to be recognized;

[0007] The indicator light status detection process is performed on the image to be identified using a trained target detection network to obtain the indicator light status detection result of the image to be identified.

[0008] Based on the image features of the image to be identified and the indicator light status detection results, the indicator light status verification result of the image to be identified is determined;

[0009] Output the status verification result of the indicator light.

[0010] Optionally, the indicator light status verification result includes the indicator light color verification result; determining the indicator light status verification result of the image to be identified based on the image features of the image to be identified and the indicator light status detection result includes:

[0011] Based on the image features of the image to be identified, calculate the target hue center and target saturation center of the indicator lights contained in the image to be identified;

[0012] After performing HSV clustering on the sample images, obtain the reference hue center and reference saturation center for each different color;

[0013] Based on the target hue center, the target saturation center, the reference hue center and the reference saturation center corresponding to each different color, the Euclidean distance between the indicator light and the cluster center corresponding to each different color is calculated respectively.

[0014] The color corresponding to the cluster center with the smallest Euclidean distance to the indicator light among the cluster centers corresponding to each different color is determined as the indicator light color verification result.

[0015] Optionally, the indicator light status verification result includes the on / off status verification result; determining the indicator light status verification result of the image to be identified based on the image features of the image to be identified and the indicator light status detection result includes:

[0016] Using the indicator light detection box in the image to be identified as the center, at least one background box is constructed around the center;

[0017] Based on the image features of the image to be identified, calculate the average brightness of each background frame and the average brightness of the indicator light detection frame;

[0018] Select the maximum brightness value from the average brightness values ​​of each of the background frames;

[0019] The on / off state verification result is determined based on the average brightness value of the indicator light detection frame and the maximum brightness value.

[0020] Optionally, the indicator light status detection result includes the on / off status detection result; determining the on / off status verification result based on the average brightness of the indicator light detection frame and the maximum brightness value includes:

[0021] The confidence level of the first on / off state is calculated based on the average brightness of the indicator light detection frame, the maximum brightness value, and the preset threshold value.

[0022] Obtain the second brightness / duration confidence level corresponding to the brightness / duration state detection result;

[0023] The target brightness / duration confidence level is calculated based on the confidence levels of the first brightness / duration state and the second brightness / duration state.

[0024] The verification result of the on / off state is determined based on the confidence level of the target on / off state.

[0025] Optionally, after determining the verification result of the on / off state based on the confidence level of the target on / off state, the method further includes:

[0026] Based on the average brightness of each background frame, the average brightness of the indicator light detection frame, and the confidence level of the second on / off state, the target on / off state feature is constructed.

[0027] The target on / off state features are matched with each on / off state feature stored in the false alarm database; wherein, the false alarm database stores the on / off state features of each input image in the case of false alarms of indicator light status;

[0028] If the target on / off state feature matches any of the on / off state features stored in the false alarm database, then the on / off state verification result is determined to be a false alarm, and the on / off state verification result is corrected using a preset method.

[0029] Optionally, the indicator light status detection result includes the indicator light target detection result, the indicator light status verification result includes the indicator light target verification result, and the false alarm database also stores the depth features of each input image in the case of false alarms of the indicator light target; determining the indicator light status verification result of the image to be identified based on the image features of the image to be identified and the indicator light status detection result includes:

[0030] Based on the indicator light target detection results, depth feature extraction processing is performed on the indicator light region in the image to be identified to obtain the target depth features;

[0031] The target depth features are matched with each depth feature stored in the false positive database;

[0032] If the target depth feature matches any of the depth features stored in the false alarm database, the indicator light target detection result is determined to be a false alarm, and the indicator light target verification result is determined by a preset method.

[0033] Optionally, after matching the target on / off state feature with each on / off state feature stored in the false alarm database, the method further includes:

[0034] If the target on / off state feature fails to match all on / off state features stored in the false alarm database, then the on / off state verification result is determined to be a false alarm by a preset method.

[0035] If the verification result of the on / off state is a false alarm, then the target on / off state feature is stored in the false alarm database;

[0036] After matching the target depth features with the various depth features stored in the false positive database, the method further includes:

[0037] If the target depth feature fails to match all depth features stored in the false alarm database, then the target detection result of the indicator light is determined to be a false alarm by a preset method.

[0038] If the target detection result of the indicator light is a false alarm, the target depth feature is stored in the false alarm database.

[0039] Secondly, embodiments of this application provide a status recognition device for an indicator light, comprising:

[0040] An image acquisition unit is used to acquire the image to be recognized.

[0041] The state detection unit is used to perform indicator light state detection processing on the image to be identified through a trained target detection network to obtain the indicator light state detection result of the image to be identified.

[0042] The status verification unit is used to determine the status verification result of the indicator light of the image to be identified based on the image features of the image to be identified and the status detection result of the indicator light.

[0043] The result output unit is used to output the status verification result of the indicator light.

[0044] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the test case generation method as described in any of the first aspects above.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the indicator light status recognition method as described in any of the first aspects above.

[0046] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the steps of the indicator light status recognition method as described in any of the first aspects above.

[0047] The indicator light status recognition method, apparatus, terminal device, and storage medium provided in this application have the following beneficial effects:

[0048] The indicator light state recognition method provided in this application acquires an image to be recognized and performs indicator light state detection processing on the image using a trained object detection network to obtain the indicator light state detection result of the image to be recognized. Then, based on the image features of the image to be recognized and the indicator light state detection result, the indicator light state verification result of the image to be recognized is determined, and finally, the indicator light state verification result is output. Using the indicator light state recognition method of this application, based on the traditional indicator light state detection result obtained through an object detection network, a further verification result of the indicator light state is obtained based on the image features of the image to be recognized and the obtained indicator light state detection result. This reduces the impact of various factors on the accuracy of the indicator light state recognition result and improves the accuracy of the indicator light state recognition method. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating the implementation of an indicator light status recognition method provided in this application embodiment;

[0051] Figure 2 A flowchart illustrating the process of determining the verification result of the indicator light color of an image to be identified, provided in an embodiment of this application;

[0052] Figure 3 A flowchart illustrating the verification result of the indicator light on / off state of an image to be identified, provided in an embodiment of this application;

[0053] Figure 4 A flowchart illustrating the process of determining the verification result of an indicator light target in an image to be identified, provided in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the structure of an indicator light status recognition device provided in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0056] It should be noted that the terminology used in the embodiments of this application is only for explaining specific embodiments of this application and is not intended to limit this application. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, "at least one" or "one or more" means one, two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0057] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0058] The indicator light status recognition method provided in this application can be executed by a terminal device. Terminal devices can include electronic devices such as mobile phones, tablets, laptops, and desktop computers.

[0059] The indicator light status recognition method provided in this application can be applied to identify the status of various indicator lights. Specifically, when a user wants to obtain the status of a target indicator light, they can execute the various steps of the indicator light status recognition method provided in this application through a terminal device, thereby obtaining the status of the target indicator light. For example, the target indicator light can be an indicator light in a substation, or it can be other types of indicator lights; the type of target indicator light is not specifically limited here.

[0060] Please see Figure 1 , Figure 1 This application provides a flowchart of an indicator light state recognition method, which may include steps S101 to S104, as detailed below:

[0061] In S101, the image to be recognized is acquired.

[0062] In this embodiment of the application, the image to be identified may include one or more indicator lights indicating the state to be identified. The image to be identified may be one or more images or videos (multiple consecutive images) captured by a camera carried by a robot, a camera carried by a drone, or a fixed camera.

[0063] The indicator light status that can be identified through images can include the indicator light color, the indicator light on / off status, and the indicator light target detection result.

[0064] The status of an indicator light that can be identified through video can include the flashing status of the indicator light.

[0065] Terminal devices can obtain the image to be identified by receiving one or more images or videos captured by preset devices (such as cameras carried by robots, cameras carried by drones, or fixed cameras).

[0066] In S102, the trained target detection network is used to perform indicator light status detection processing on the image to be recognized, and the indicator light status detection result of the image to be recognized is obtained.

[0067] In this embodiment of the application, after the terminal device acquires the image to be identified, it can first perform the traditional indicator light detection method, that is, through the trained target detection network, perform indicator light status detection processing on the image to be identified, and obtain the indicator light status detection result of the image to be identified.

[0068] The object detection network can be a lightweight object detection network. A trained object detection network can be obtained by: identifying and labeling the mapping relationship between image information contained in sample images and indicator light states, and adding non-indicator light devices (such as buttons, screws, etc.) that are easily mistaken for indicator lights as negative samples. The object detection network is then trained and optimized using the identified and labeled sample images, and the trained and optimized object detection network is determined as the trained object detection network.

[0069] In this embodiment of the application, the indicator light status detection result is a preliminary indicator light recognition result. Since the indicator light status detection result is easily affected by various factors in practical applications and is not accurate, it is necessary to review the indicator light status detection result according to the image features of the image to be recognized in order to improve the accuracy of the indicator light recognition result.

[0070] The types of indicator light status detection results can include indicator light position, indicator light color detection results, indicator light on / off status detection results, and indicator light target detection results. The indicator light position can be used to describe the position of the indicator light in the image to be recognized; the position of each indicator light can be represented by (x... i ,y i) indicates, specifically, (x i ,y i ) represents the coordinates of the center point of the i-th indicator area.

[0071] For example, the indicator light color may include red, green and yellow, the indicator light on / off state may include on and off, and the indicator light target detection result may include indicator light device and non-indicator light device.

[0072] In S103, the indicator light status verification result of the image to be identified is determined based on the image features of the image to be identified and the indicator light status detection result.

[0073] In this embodiment of the application, after obtaining the indicator light status detection result of the image to be identified, the terminal device can determine the indicator light status verification result of the image to be identified based on the image features of the image to be identified and the indicator light status detection result.

[0074] The type of indicator light status verification result can correspond to the type of indicator light status detection result. Based on this, the types of indicator light status verification results can include indicator light color verification result, indicator light on / off status verification result, and indicator light target verification result.

[0075] In one possible implementation, it can be achieved through, for example... Figure 2 The steps S201 to S204 shown indicate the verification results of the indicator light colors of the image to be recognized. Figure 2 A flowchart illustrating the process of determining the verification result of the indicator light color of an image to be identified, provided in an embodiment of this application, is described in detail below:

[0076] In S201, based on the image features of the image to be identified, the target hue center and target saturation center of the indicator lights contained in the image to be identified are calculated.

[0077] In this implementation, the terminal device can calculate the target hue center (H center) and target saturation center (S center) of the indicator light contained in the image to be identified based on the image features of the image to be identified using a preset method.

[0078] In S202, the reference hue center and reference saturation center corresponding to each different color are obtained after HSV clustering processing of the sample image.

[0079] In this implementation, the terminal device can perform HSV(Hue,Saturation,Value) clustering on the sample image to obtain the reference hue center and reference saturation center for each color.

[0080] For example, the reference hue center and reference saturation center corresponding to red, the reference hue center and reference saturation center corresponding to green, and the reference hue center and reference saturation center corresponding to yellow can be obtained respectively.

[0081] In S203, based on the target hue center, target saturation center, and the reference hue center and reference saturation center corresponding to each different color, the Euclidean distance between the indicator light and the cluster center corresponding to each different color is calculated.

[0082] In this implementation, after obtaining the target hue center, target saturation center, and reference hue center and reference saturation center corresponding to each different color of the indicator light, the terminal device can calculate the Euclidean distance between the indicator light and the cluster center corresponding to each different color using a preset method based on the obtained target hue center, target saturation center, and reference hue center and reference saturation center corresponding to each different color.

[0083] In S204, the color corresponding to the cluster center with the smallest Euclidean distance from the indicator light among the cluster centers corresponding to each different color is determined as the indicator light color verification result.

[0084] In this implementation, after calculating the Euclidean distance between the indicator light and the cluster centers corresponding to each different color, the terminal device can determine the color corresponding to the cluster center with the smallest Euclidean distance as the indicator light color verification result.

[0085] In practical applications, since the accuracy of the indicator light color determined by the methods shown in S201 to S204 is significantly higher than that of the indicator light color obtained by the trained target detection network, the indicator light color determined by the methods shown in S201 to S204 can be directly used as the indicator light color verification result.

[0086] In one possible implementation, it can be achieved through, for example... Figure 3 The S301-S304 steps shown indicate the verification results of the indicator light on / off status of the image to be recognized. Figure 3 A flowchart illustrating the verification result of the indicator light on / off state of an image to be identified, provided in an embodiment of this application, is described in detail below:

[0087] In S301, at least one background frame is constructed around the indicator light detection frame in the image to be identified.

[0088] In this implementation, the indicator light status detection result of the image to be identified can include the position of the indicator light in the image to be identified. The position of the indicator light in the image to be identified can be represented by coordinates. It should be noted that since the indicator light target has not been verified in this step, the indicator light to be identified may be an indicator light device or a non-indicator light device that is mistakenly identified as an indicator light device.

[0089] Based on this, the indicator light detection frame can be determined according to the position of the indicator light in the image to be identified.

[0090] After determining the indicator light detection frame, the terminal device can construct at least one background frame around the indicator light detection frame in the image to be recognized.

[0091] Preferably, eight background frames identical to the indicator light detection frame can be constructed around the center of the indicator light detection frame in the image to be identified.

[0092] In S302, based on the image features of the image to be identified, the average brightness of each background frame and the average brightness of the indicator light detection frame are calculated.

[0093] In this implementation, after constructing the background frame, the terminal device can calculate the average brightness of each background frame and the average brightness of the indicator light detection frame based on the image features of the image to be recognized.

[0094] In S303, the maximum brightness value is selected from the average brightness values ​​of each background frame.

[0095] In this implementation, after calculating the average brightness of each background frame, the terminal device can select the maximum brightness value from the average brightness values ​​of each background frame.

[0096] In S304, the on / off state verification result is determined based on the average and maximum brightness values ​​of the indicator light detection frame.

[0097] In this implementation, steps a to d can be used to determine the on / off state verification result based on the average and maximum brightness values ​​of the indicator light detection frame, as detailed below:

[0098] In step a, the confidence level of the first on / off state is calculated based on the average brightness value, maximum brightness value, and preset threshold value of the indicator light detection frame.

[0099] In this implementation, the confidence level of the first on / off state can be calculated using the following formula:

[0100] t=V / (V+k*V max )

[0101] Where t is the confidence level of the first on / off state, V is the average brightness of the indicator light detection frame, and k is the preset threshold value. max This is the maximum brightness value.

[0102] In step b, the second brightness / duration confidence level corresponding to the brightness / duration state detection result is obtained.

[0103] In addition to the on / off state detection result, the indicator light status detection result may also include the second on / off state confidence level corresponding to the on / off state detection result.

[0104] Based on this, the terminal device can obtain the second on / off state confidence level corresponding to the on / off state detection result from the indicator light state detection result.

[0105] In step c, the confidence level of the target bright / dark state is calculated based on the confidence levels of the first and second bright / dark states.

[0106] In this implementation, the confidence level of the target's on / off state can be calculated using the following formula:

[0107] S=d*t+(1-d)*s

[0108] Where S is the confidence level of the target on / off state, d is a preset parameter, t is the confidence level of the first on / off state, and s is the confidence level of the second on / off state.

[0109] In step d, the verification result of the on / off state is determined based on the confidence level of the target on / off state.

[0110] After obtaining the confidence level of the target on / off state, the terminal device can determine the verification result of the on / off state based on this confidence level. For example, when the confidence level of the target on / off state is greater than a preset confidence threshold, the verification result can be determined as "on"; when the confidence level is less than the preset confidence threshold, the verification result can be determined as "off". In practical applications, the preset confidence threshold can be set to 0.5, meaning that when the confidence level of the target on / off state is greater than 0.5, the verification result is determined as "on", and when the confidence level is less than 0.5, the verification result is determined as "off".

[0111] In this embodiment of the application, since false alarms regarding the on / off status of indicator lights may occur in practical applications due to various factors, after determining the verification result of the on / off status, steps e to g can be used to determine whether the verification result of the on / off status is a false alarm, as detailed below:

[0112] In step e, the target on / off state features are constructed based on the average brightness of each background frame, the average brightness of the indicator light detection frame, and the confidence level of the second on / off state.

[0113] In this implementation, the first feature vector can be constructed from the average brightness of each background frame, the average brightness of the indicator light detection frame, and the confidence level of the second on / off state:

[0114] {V0, V1, V2, V3, V4, V5, V6, V7, V8, s}

[0115] Where V0 to V8 correspond to the average brightness of each background frame and the average brightness of the indicator light detection frame, respectively, and s is the confidence level of the second on / off state.

[0116] It should be noted that the order of the first feature vector can be set according to actual needs, but the order of the first feature vector must remain unchanged. The first feature vector can be a one-dimensional feature vector.

[0117] The first feature vector obtained from this construction can be determined as the target's on / off state feature.

[0118] In step f, the target on / off state feature is matched with each on / off state feature stored in the false alarm database.

[0119] Among them, the false alarm library stores the on / off state characteristics of each input image in the case of false alarms of indicator light status;

[0120] When matching the target's on / off state features with those stored in the false alarm database, for any on / off state feature stored in the false alarm database, the first cosine of the angle between the first feature vector corresponding to the target's on / off state feature and the first feature vector corresponding to that same on / off state feature stored in the false alarm database can be calculated in vector space. The match between the target's on / off state feature and the first feature vector stored in the false alarm database is then determined based on this first cosine value. Here, the first feature vector can be any first feature vector.

[0121] The first cosine value can range from [-1, 1]. The larger the first cosine value, the more the target's on / off state feature matches any one of the on / off state features. The smaller the first cosine value, the less the target's on / off state feature matches any one of the on / off state features. Based on this, when the first cosine value is greater than the preset cosine threshold, it can be considered that the target's on / off state feature matches any one of the on / off state features. When the first cosine value is less than the preset cosine threshold, it can be considered that the target's on / off state feature does not match any one of the on / off state features.

[0122] In step g, if the target on / off state feature matches any of the on / off state features stored in the false alarm database, the on / off state verification result is determined to be a false alarm, and the on / off state verification result is corrected using a preset method.

[0123] The terminal device can use the method described in step f to determine whether the target on / off state feature matches any of the on / off state features stored in the false alarm database. If the target on / off state feature matches any of the on / off state features stored in the false alarm database, the on / off state verification result is determined to be a false alarm, and the on / off state verification result is corrected using a preset method.

[0124] For example, in practical applications, after determining that the verification result of the on / off state is a false alarm, the verification result of the on / off state can be corrected by manual verification.

[0125] In one alternative implementation, step h may be included after step g, as detailed below:

[0126] In step h, if the target on / off state feature fails to match all on / off state features stored in the false alarm database, the result of the on / off state verification can be determined by a preset method to determine whether the result is a false alarm.

[0127] In practical applications, in order to efficiently obtain the verification results of the on / off state, it is possible to choose not to execute the step in step h that determines whether the verification result of the on / off state is a false alarm through a preset method. That is, if the target on / off state feature fails to match all the on / off state features stored in the false alarm database, it is determined that the verification result of the on / off state is not a false alarm, and there is no need to determine whether the verification result of the on / off state is a false alarm through a preset method (such as manual verification).

[0128] In order to expand the various on / off state features stored in the false alarm database and make the output on / off state verification results more accurate, it is possible to select a method (such as manual verification) to determine whether the on / off state verification result is a false alarm after the target on / off state feature fails to match all the on / off state features stored in the false alarm database. If the on / off state verification result is determined to be a false alarm, the target on / off state feature is stored in the false alarm database.

[0129] In one possible implementation, it can be achieved through, for example... Figure 4 S401 to S403, as shown, determine the verification results of the indicator light target in the image to be identified. Figure 4 A flowchart illustrating the process of determining the verification result of an indicator light target in an image to be identified, as provided in this application embodiment, is described in detail below:

[0130] In S401, based on the indicator light target detection results, depth feature extraction processing is performed on the indicator light area in the image to be identified to obtain the target depth features.

[0131] In this embodiment of the application, the indicator light target detection result may also include the indicator light region in the image to be identified, which can be represented by a coordinate set.

[0132] Based on this, the terminal device can perform depth feature extraction processing on the indicator light region in the image to be recognized according to the indicator light target detection results, thereby obtaining the target depth features. These target depth features are used to determine whether the indicator light target detection result is a false alarm. The specific methods and steps for performing depth feature extraction processing on the indicator light region in the image to be recognized can be set according to actual needs and are not limited here.

[0133] In S402, the target depth features are matched with each depth feature stored in the false alarm database.

[0134] The false alarm database also stores the depth features of each input image in the event of a false alarm for an indicator light target.

[0135] In this embodiment, since there are many non-indicator devices similar to indicator lights, such as buttons and screws, around the indicator light, the indicator light status detection result obtained by the trained object detection network may misidentify non-indicator devices as indicator lights. Furthermore, the depth features corresponding to non-indicator devices are different from the depth features corresponding to indicator lights. Therefore, the terminal device can match the target depth features with the various depth features stored in the false alarm database.

[0136] When matching the target depth feature with each depth feature stored in the false alarm database, for any depth feature stored in the false alarm database, the second cosine value of the angle between the second feature vector corresponding to the target depth feature and the second feature vector corresponding to the same depth feature stored in the false alarm database in the vector space can be calculated, and the target depth feature can be determined to match the same depth feature based on the second cosine value.

[0137] The second cosine value can range from [-1, 1]. The larger the second cosine value, the more the target depth feature matches any one of the depth features. The smaller the second cosine value, the less the target depth feature matches any one of the depth features. Based on this, when the second cosine value is greater than the preset cosine threshold, it can be considered that the target depth feature matches any one of the depth features. When the second cosine value is less than the preset cosine threshold, it can be considered that the target depth feature does not match any one of the depth features.

[0138] In S403, if the target depth feature matches any depth feature stored in the false alarm database, the indicator light target detection result is determined to be a false alarm, and the indicator light target verification result is determined by a preset method.

[0139] In this embodiment of the application, the terminal device can determine the target depth feature by matching it with each depth feature stored in the false alarm library through the method described in S402. If the target depth status feature matches any depth feature stored in the false alarm library, the indicator light target detection result is determined to be a false alarm, and the indicator light target detection result is corrected by a preset method, thereby determining the indicator light target verification result.

[0140] For example, in practical applications, after determining that the indicator light target detection result is a false alarm, the indicator light target detection result can be corrected by manual verification, thereby determining the indicator light target verification result.

[0141] In one alternative implementation, step i may be included after S403, as detailed below:

[0142] In step i, if the target depth feature fails to match all depth features stored in the false alarm database, the indicator light target detection result is determined to be a false alarm by a preset method.

[0143] In practical applications, in order to efficiently obtain the indicator light target verification result, the step of determining whether the indicator light target detection result is a false alarm by a preset method in step i can be skipped. That is, if the target depth feature fails to match all depth features stored in the false alarm library, the indicator light target detection result is determined to be a false alarm, and the indicator light target verification result is determined to be the indicator light target detection result.

[0144] In order to expand the depth features stored in the false alarm database and make the verification results of indicator lights more accurate in the future, it is possible to select a preset method (such as manual verification) to determine whether the target detection result of the indicator light is a false alarm after the target depth feature fails to match all depth features stored in the false alarm database. After determining that the target detection result of the indicator light is a false alarm, the target depth feature is stored at the false alarm location.

[0145] In S104, the status verification result of the indicator light is output.

[0146] In this embodiment of the application, after the terminal device determines the indicator light status verification result, it can output the indicator light status verification result, thereby enabling the identification of the indicator light status.

[0147] The indicator light status verification results include, for example: Figure 2 The steps shown yielded the indicator light color verification results, which were obtained through methods such as... Figure 3 The steps shown yielded the verification results of the on / off state and as follows: Figure 4 The steps shown yielded the target verification results for the indicator lights.

[0148] As can be seen from the above, the indicator light state recognition method provided in this application acquires an image to be recognized, performs indicator light state detection processing on the image to be recognized using a trained target detection network, obtains the indicator light state detection result of the image to be recognized, then determines the indicator light state verification result of the image to be recognized based on the image features of the image to be recognized and the indicator light state detection result, and finally outputs the indicator light state verification result. Using the indicator light state recognition method of this application, based on the traditional indicator light state detection result obtained through a target detection network, a further verification result of the indicator light state can be obtained based on the image features of the image to be recognized and the obtained indicator light state detection result. This reduces the impact of various factors on the accuracy of the indicator light state recognition result and improves the accuracy of the indicator light state recognition method.

[0149] Based on the indicator light status recognition method provided in the above embodiments, this application further provides an indicator light status recognition device that implements the above method embodiments. Please refer to [link to relevant documentation]. Figure 5 , Figure 5 This is a schematic diagram of the structure of an indicator light status recognition device provided in an embodiment of this application. Figure 5 As shown, the indicator light status recognition device 50 may include an image acquisition unit 51, a status detection unit 52, a status verification unit 53, and a result output unit 54. Wherein:

[0150] The image acquisition unit 51 is used to acquire the image to be recognized.

[0151] The state detection unit 52 is used to perform indicator light state detection processing on the image to be recognized through a trained target detection network, and obtain the indicator light state detection result of the image to be recognized.

[0152] The status verification unit 53 is used to determine the status verification result of the indicator lights of the image to be identified based on the image features of the image to be identified and the status detection result of the indicator lights.

[0153] The result output unit 54 is used to output the indicator light status verification result.

[0154] Optionally, the indicator light status verification result includes the indicator light color verification result. The status verification unit 53 may include a first calculation unit, a first acquisition unit, a second calculation unit, and a first determination unit, wherein:

[0155] The first calculation unit is used to calculate the target hue center and target saturation center of the indicator lights contained in the image to be identified, based on the image features of the image to be identified.

[0156] The first acquisition unit is used to acquire the reference hue center and reference saturation center corresponding to each different color after HSV clustering processing of the sample image.

[0157] The second calculation unit is used to calculate the Euclidean distance between the indicator light and the cluster center corresponding to each different color, based on the target hue center, the target saturation center, the reference hue center and the reference saturation center corresponding to each different color.

[0158] The first determining unit is used to determine the color corresponding to the cluster center with the smallest Euclidean distance from the indicator light among the cluster centers corresponding to each different color as the indicator light color verification result.

[0159] Optionally, the state verification unit 53 may further include a first construction unit, a third calculation unit, a selection unit, and a second determination unit, wherein:

[0160] The first building unit is used to construct at least one background box around the indicator light detection box in the image to be recognized.

[0161] The third calculation unit is used to calculate the average brightness of each background frame and the average brightness of the indicator light detection frame based on the image features of the image to be identified.

[0162] The selection cell is used to select the maximum brightness value from the average brightness values ​​of each background frame.

[0163] The second determining unit is used to determine the on / off state verification result based on the average and maximum brightness values ​​of the indicator light detection frame.

[0164] Optionally, the second determining unit may include a fourth calculation unit, a second acquisition unit, a fifth calculation unit, and a third determining unit, wherein:

[0165] The fourth calculation unit is used to calculate the confidence level of the first on / off state based on the average brightness value, maximum brightness value and preset threshold value of the indicator light detection frame.

[0166] The second acquisition unit is used to acquire the second brightness / duration confidence level corresponding to the brightness / duration state detection result.

[0167] The fifth calculation unit is used to calculate the target's brightness / duration confidence level based on the confidence levels of the first and second brightness / duration states.

[0168] The third determining unit is used to determine the verification result of the target's on / off state based on the confidence level of the target's on / off state.

[0169] Optionally, the status recognition device 50 of the indicator light may further include a second building unit, a first matching unit, and a fourth determining unit, wherein:

[0170] The second construction unit is used to construct the target on / off state features based on the average brightness of each background frame, the average brightness of the indicator light detection frame, and the confidence level of the second on / off state.

[0171] The first matching unit is used to match the target on / off state features with each on / off state feature stored in the false alarm database; wherein, the false alarm database stores the on / off state features of each input image in the case of false alarms of indicator light status.

[0172] The fourth determining unit is used to determine that the brightness status verification result is a false alarm if the target brightness status feature matches any brightness status feature stored in the false alarm database, and to correct the brightness status verification result through a preset method.

[0173] Optionally, the indicator light status detection result includes the indicator light target detection result, the indicator light status verification result includes the indicator light target verification result, and the false alarm database also stores the depth features of each input image in the case of false alarms of the indicator light target; the status verification unit 53 may also include a depth feature extraction unit, a second matching unit, and a fifth determination unit, wherein:

[0174] The depth feature extraction unit is used to perform depth feature extraction processing on the indicator light area in the image to be identified based on the indicator light target detection results, so as to obtain the target depth features.

[0175] The second matching unit is used to match the target depth features with each depth feature stored in the false positive database.

[0176] The fifth determining unit is used to determine that the indicator light target detection result is a false alarm if the target depth feature successfully matches any depth feature stored in the false alarm database, and to determine the indicator light status verification result through a preset method.

[0177] Optionally, the status recognition device 50 of the indicator light may further include a sixth determining unit, a first storage unit, a seventh determining unit, and a second storage unit, wherein:

[0178] The sixth determining unit is used to determine whether the result of the brightness status verification is a false alarm if the target brightness status feature fails to match all brightness status features stored in the false alarm database.

[0179] The first storage unit is used to store the target's on / off state characteristics into the false alarm database if the verification result of the on / off state is a false alarm.

[0180] The seventh determination unit is used to determine whether the target detection result of the indicator light is a false alarm if the target depth feature fails to match all depth features stored in the false alarm database.

[0181] The second storage unit is used to store the target depth features into the false alarm library if the indicator light target detection result is a false alarm.

[0182] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of this application. Their specific functions and technical effects can be referred to the method embodiments section, and will not be repeated here.

[0183] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 6 As shown, the terminal device 6 provided in this embodiment may include: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. For example, a program corresponding to an indicator light status recognition method. When the processor 60 executes the computer program 62, it implements the steps described above in the embodiment of the indicator light status recognition method, for example... Figure 1 S101 to S104 shown Figure 2 S201 to S204 are shown. Figure 3 S301 to S304 and Figure 4 S401 to S403 in the above. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the embodiment corresponding to the terminal device 6, for example... Figure 5 The functions of units 61 to 64 shown.

[0184] For example, computer program 62 can be divided into one or more modules / units, one or more of which are stored in memory 61 and executed by processor 60 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 62 in terminal device 6. For example, computer program 62 can be divided into an image acquisition unit 51, a status detection unit 52, a status verification unit 53, and a result output unit 54. For the specific functions of each unit, please refer to [link to relevant documentation]. Figure 5 The relevant descriptions in the corresponding embodiments are not repeated here.

[0185] Those skilled in the art will understand that Figure 6 This is merely an example of terminal device 6 and does not constitute a limitation on terminal device 6. It may include more or fewer components than shown, or combine certain components, or use different components.

[0186] The processor 60 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0187] The memory 61 can be an internal storage unit of the terminal device 6, such as a hard disk or RAM of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card equipped on the terminal device 6. Furthermore, the memory 61 can include both internal and external storage units of the terminal device 6. The memory 61 is used to store computer programs and other programs and data required by the terminal device. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0188] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units is merely an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the indicator light status recognition device can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0189] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0190] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0191] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

[0192] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0193] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for identifying the status of an indicator light, characterized in that, include: Acquire the image to be recognized; The indicator light status detection process is performed on the image to be identified using a trained target detection network to obtain the indicator light status detection result of the image to be identified. Based on the image features of the image to be identified and the indicator light status detection results, the indicator light status verification result of the image to be identified is determined; Output the status verification result of the indicator light; The indicator light status verification result includes the on / off status verification result; The step of determining the indicator light status verification result of the image to be identified based on the image features of the image to be identified and the indicator light status detection result includes: Using the indicator light detection box in the image to be identified as the center, at least one background box is constructed around the center; Based on the image features of the image to be identified, calculate the average brightness of each background frame and the average brightness of the indicator light detection frame; Select the maximum brightness value from the average brightness values ​​of each of the background frames; The on / off state verification result is determined based on the average brightness value of the indicator light detection frame and the maximum brightness value. The indicator light status detection result includes the on / off status detection result; determining the on / off status verification result based on the average brightness of the indicator light detection frame and the maximum brightness value includes: The confidence level of the first on / off state is calculated based on the average brightness of the indicator light detection frame, the maximum brightness value, and the preset threshold value. Obtain the second brightness / duration confidence level corresponding to the brightness / duration state detection result; The target brightness / duration confidence level is calculated based on the confidence levels of the first brightness / duration state and the second brightness / duration state. The verification result of the on / off state is determined based on the confidence level of the target on / off state.

2. The method according to claim 1, characterized in that, The indicator light status verification result includes the indicator light color verification result; determining the indicator light status verification result of the image to be identified based on the image features of the image to be identified and the indicator light status detection result includes: Based on the image features of the image to be identified, calculate the target hue center and target saturation center of the indicator lights contained in the image to be identified; After performing HSV clustering on the sample images, obtain the reference hue center and reference saturation center for each different color; Based on the target hue center, the target saturation center, the reference hue center and the reference saturation center corresponding to each different color, the Euclidean distance between the indicator light and the cluster center corresponding to each different color is calculated respectively. The color corresponding to the cluster center with the smallest Euclidean distance to the indicator light among the cluster centers corresponding to each different color is determined as the indicator light color verification result.

3. The method according to claim 1, characterized in that, After determining the verification result of the brightness state based on the confidence level of the target brightness state, the process further includes: Based on the average brightness of each background frame, the average brightness of the indicator light detection frame, and the confidence level of the second on / off state, the target on / off state feature is constructed. The target on / off state features are matched with each on / off state feature stored in the false alarm database; wherein, the false alarm database stores the on / off state features of each input image in the case of false alarms of indicator light status; If the target on / off state feature matches any of the on / off state features stored in the false alarm database, then the on / off state verification result is determined to be a false alarm, and the on / off state verification result is corrected using a preset method.

4. The method according to claim 3, characterized in that, The indicator light status detection result includes the indicator light target detection result, the indicator light status verification result includes the indicator light target verification result, and the false alarm database also stores the depth features of each input image in the case of false indicator light target alarms; determining the indicator light status verification result of the image to be identified based on the image features of the image to be identified and the indicator light status detection result includes: Based on the indicator light target detection results, depth feature extraction processing is performed on the indicator light region in the image to be identified to obtain the target depth features; The target depth features are matched with each depth feature stored in the false positive database; If the target depth feature matches any of the depth features stored in the false alarm database, the indicator light target detection result is determined to be a false alarm, and the indicator light target verification result is determined by a preset method.

5. The method according to claim 4, characterized in that, After matching the target on / off state features with each on / off state feature stored in the false alarm database, the process further includes: If the target on / off state feature fails to match all on / off state features stored in the false alarm database, then the on / off state verification result is determined to be a false alarm by a preset method. If the verification result of the on / off state is a false alarm, then the target on / off state feature is stored in the false alarm database; After matching the target depth features with the various depth features stored in the false positive database, the method further includes: If the target depth feature fails to match all depth features stored in the false alarm database, then the target detection result of the indicator light is determined to be a false alarm by a preset method. If the target detection result of the indicator light is a false alarm, the target depth feature is stored in the false alarm database.

6. A status recognition device for an indicator light, characterized in that, include: An image acquisition unit is used to acquire the image to be recognized. The state detection unit is used to perform indicator light state detection processing on the image to be identified through a trained target detection network to obtain the indicator light state detection result of the image to be identified. The status verification unit is used to determine the status verification result of the indicator light of the image to be identified based on the image features of the image to be identified and the status detection result of the indicator light. The result output unit is used to output the status verification result of the indicator light; The indicator light status verification result includes an on / off status verification result; the status verification unit is specifically used for: Using the indicator light detection box in the image to be identified as the center, at least one background box is constructed around the center; Based on the image features of the image to be identified, calculate the average brightness of each background frame and the average brightness of the indicator light detection frame; Select the maximum brightness value from the average brightness values ​​of each of the background frames; The on / off state verification result is determined based on the average brightness value of the indicator light detection frame and the maximum brightness value. The indicator light status detection results include on / off status detection results; The step of determining the on / off state verification result based on the average brightness value and the maximum brightness value of the indicator light detection frame includes: The confidence level of the first on / off state is calculated based on the average brightness of the indicator light detection frame, the maximum brightness value, and the preset threshold value. Obtain the second brightness / duration confidence level corresponding to the brightness / duration state detection result; The target brightness / duration confidence level is calculated based on the confidence levels of the first brightness / duration state and the second brightness / duration state. The verification result of the on / off state is determined based on the confidence level of the target on / off state.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements each step of the indicator light status recognition method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements each step of the indicator light status recognition method as described in any one of claims 1 to 5.

Images