A fully automatic method and system for identifying the status of a fish-eye type on-off indicator

Abstract

The present invention belongs to the field of image recognition, and provides a fully automatic method and system for recognizing the state of a fish-eye closing and opening indicator, including obtaining a monitoring image of the indicator; performing preprocessing based on the monitoring image of the indicator, and according to the preprocessed indicator image, extracting the contour information of each connected domain to determine the area of the elliptical panel to be detected; according to the principle of the minimum within-class standard deviation, selecting the segmentation method with the smallest within-class standard deviation, dividing the area of the elliptical panel to be detected into four fan-shaped regions, and then by comparing the color vividness of each region, selecting the region with the most vivid color as the observation window; judging the state of the indicator according to the color information of the pixel points within the observation region window; without the need for prior annotation information, the present invention determines the area of the elliptical panel to be detected through the contour information of each connected domain of the monitoring image of the indicator, automatically segments the area of the observation window therefrom, and thus judges the state of the indicator based on the color information of the pixel points within the observation region window.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and particularly relates to a method and system for automatically recognizing the state of a fish-eye switch-on / off indicator. Background Art

[0002] The statements in this part only provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] The existing detection and recognition methods for substation disconnectors are based on the Faster RCNN model. The deep neural network is used to extract and analyze the features of the disconnector in the detection image, which can effectively improve the recognition accuracy and detection speed of the disconnector detection and recognition, and has strong robustness. However, the substation disconnector detection and recognition method based on Faster RCNN requires a large amount of pre-annotation and training in advance, and the calculation amount in the reasoning process is huge, making it difficult to meet the real-time detection requirements on edge devices.

[0004] The existing method for recognizing the switch-on / off state of a fish-eye type in a substation vertically acquires the indicator image and analyzes the indicator image to obtain an angle at which the line connecting the center of the circle and the end point deviates from the boundary of the observation window, with the center of the circle being the intersection of the two boundary lines connecting the two observation windows and the end point being the position of the characters within the observation window. This angle is used to recognize the switch-on / off state of the switch, realizing the quantitative analysis of the switch-on / off in-place degree through the angle. And this method uses the yolov3 network to detect the observation window of the fish-eye switch-on / off, and also requires a large amount of annotation and training. Moreover, when detecting characters, the template matching method is used, and template images need to be prepared in advance for "switch-on / off identification characters" of different types, fonts, and angles.

[0005] In addition, there is another method that segments the corresponding non-observation window area of the image of the fish-eye switch-on / off collected by the image acquisition device, obtains the mask image corresponding to the character area through the exclusive OR operation of the segmented image, extracts the original image corresponding to the mask image of the character area, segments the original image corresponding to the character area on the S channel to obtain the character image on the S channel; connects the character images on the S channel to obtain the center point position of the connected character image; determines the offset angle of the character image in the character area according to the center point position; segments the original image corresponding to the mask image of the character area on the H channel to obtain the character image on the H channel, performs pixel histogram statistics on the character image on the H channel, and determines that the fish-eye switch-on / off is in the open state when the pixel value of the character image is within the first preset range, otherwise it is in the closed state. When detecting, a plurality of manually marked feature points and seed points need to be input in advance, and re-annotation is required every time a new device is deployed or the position of the shooting device changes, which has certain complexity in large-scale applications. Summary of the Invention

[0006] To solve the above problems, the present invention proposes a fully automatic method and system for identifying the state of a fish-eye type on-off indicator. Without prior annotation information, the present invention determines the area of the elliptical panel to be detected based on the contour information of each connected domain in the monitoring image of the indicator, and then automatically segments the area of the observation window therefrom, thereby judging the state of the indicator based on the color information of the pixel points within the observation area window; this method can adapt to various colors and styles of fish-eye type on-off indicators and has good robustness in the case of medium and small range of reflection.

[0007] According to some embodiments, the first solution of the present invention provides a fully automatic method for identifying the state of a fish-eye type on-off indicator, adopting the following technical solution:

[0008] A fully automatic method for identifying the state of a fish-eye type on-off indicator includes:

[0009] Obtain the monitoring image of the indicator;

[0010] Perform preprocessing on the monitoring image of the indicator, and based on the preprocessed indicator image, extract the contour information of each connected domain to determine the area of the elliptical panel to be detected;

[0011] According to the principle of the minimum within-class standard deviation, select the segmentation method with the minimum within-class standard deviation, divide the area of the elliptical panel to be detected into four fan-shaped areas, and then select the area with the most distinct color as the observation window by comparing the color distinctness of each area;

[0012] Judge the state of the indicator according to the color information of the pixel points within the observation area window.

[0013] Further, the preprocessing performed on the monitoring image of the indicator includes:

[0014] Perform grayscale conversion on the monitoring image of the indicator;

[0015] Use the sauvola algorithm to perform binary segmentation on the grayscale-converted monitoring image of the indicator;

[0016] Invert the binary result;

[0017] Use a 5*5 sized kernel to perform mathematical morphological opening operation to eliminate noise points, and obtain the preprocessed monitoring image of the indicator.

[0018] Further, the extracting the contour information of each connected domain based on the preprocessed indicator image to determine the area of the elliptical panel to be detected includes:

[0019] Detect the contour of the indicator monitoring image after preprocessing, and store the contour result in a non-compressed manner, that is, store each point of the contour in sequence to obtain all the contours of the indicator monitoring image after preprocessing.

[0020] Traverse each contour to be screened, select a contour with the highest similarity to an ellipse, and use the ellipse fitted by it as the ellipse panel area to be detected.

[0021] Further, the step of traversing each contour to be screened, selecting a contour with the highest similarity to an ellipse, and using the ellipse fitted by it as the ellipse panel area to be detected includes:

[0022] Denote each contour as Contours i , where i represents the contour number;

[0023] Determine the convex hull of each contour and denote it as ConvexHull i ;

[0024] Determine the ellipse fitted by each contour to obtain the ellipse contour denoted as Ellipse i ;

[0025] Calculate the shape similarity between all Contours i and ConvexHull i and denote it as Sim1 i ;

[0026] Calculate the shape similarity between all ConvexHull i and Ellipse i and denote it as Sim2 i ;

[0027] Take Sim i =(Sim1 i +0.0001)*(Sim2 i +0.0001) as an evaluation index. The smaller this value is, the higher the similarity between the contour, convex hull and the fitted ellipse;

[0028] Select the ellipse fitted by the contour with the smallest Sim i as the ellipse panel area to be detected, denoted as Ellipse. The mask of this area is denoted as Mask, and the center point is denoted as Center.

[0029] Further, according to the principle of the smallest within-class standard deviation, select the segmentation method with the smallest within-class standard deviation, divide the ellipse panel area to be detected into four fan-shaped areas, and then select the area with the most distinct color as the observation window by comparing the color vividness of each area, including:

[0030] The traversal indicator monitors each point in the area of the elliptical panel to be detected, calculates the angle of the point relative to the center point of the area, rounds it to the nearest integer and records it as d. If d is equal to 360, then set d equal to 0, and the integer d ∈ [0, 359];

[0031] Establish 360 sets, denoted as Points d , and according to the angle d, classify each point into the set Points corresponding to its angle d d ;

[0032] Establish three arrays Avg_R, Avg_G, and Avg_B. Traverse the integer angle d within [0, 359], calculate the average values of the points in each Points d in the red, green, and blue channels, and store them into Avg_R[d], Avg_G[d], and Avg_B[d] in sequence, to obtain the average values of the point sets at each angle in each red, green, and blue component;

[0033] Based on the average values of the point sets at each angle in each red, green, and blue component, and taking the principle of the minimum within-class standard deviation as the segmentation basis, divide the complete 360-degree elliptical panel area into 4 continuous and non-overlapping elliptical fan-shaped areas, denoted as A n , n ∈ {1, 2, 3, 4}, the central angles of A1 and A3 face each other, and the central angles of A2 and A4 face each other;

[0034] For each pixel point (x, y) in the elliptical area of the original image, calculate

[0035] MAXRGB(x,y) = max(R(x,y), G(x,y), B(x,y))

[0036] MINRGB(x,y) = min(R(x,y), G(x,y), B(x,y))

[0037] COLORFUL(x,y) = (MAXRGB - MINRGB) * MAXRGB

[0038] where R(x,y), G(x,y), B(x,y) represent the red, green, and blue component values of the pixel point (x, y) respectively, and COLORFUL(x,y) represents the color vividness of the pixel point (x, y);

[0039] Compare the average values of COLORFUL(x,y) of the pixels in the 4 fan-shaped areas, and take the fan-shaped area with the largest average value as the finally determined observation window.

[0040] Furthermore, based on the mean values of the point sets at each angle on each red, green, and blue component, and taking the principle of the minimum within-class standard deviation as the segmentation basis, the complete 360-degree elliptical panel area is segmented into 4 continuous and non-overlapping elliptical sector areas. The specific steps are as follows:

[0041] For any central angle size of A1, denoted as SIZE_A1, set SIZE_A1 ∈ [30, 90] by prior knowledge, and SIZE_A1 is an integer. Then, it can be known that the central angle size of A2, SIZE_A2 = 180 - SIZE_A1, the central angle size of A3, SIZE_A3 = A1_SIZE, and the central angle size of A4, SIZE_A4 = 180 - SIZE_A1;

[0042] For any starting angle of A1, denoted as START_A1, START_A1 ∈ [0, 359], and START_A1 is an integer. Then, it can be known that the start and end interval of the angle corresponding to A1 is [START_A1, START_A1 + SIZE_A1). By analogy, the start and end intervals of the angles corresponding to the A2, A3, and A4 regions can be obtained;

[0043] Taking the red component as an example, calculate the standard deviation between the mean values of each angle within the A n region, that is, calculate the standard deviation of the values in the array Avg_R at the subscripts [START_A n , START_A n + SIZE_A n ), denoted as Similarly, the standard deviation under the green component is obtained as The standard deviation under the blue component is

[0044] Calculate the within-class standard deviation after color fusion for each A n region, as follows:

[0045]

[0046] For any determined START_A1 and SIZE_A1, regarded as a certain segmentation method, calculate the weighted sum of the within-class standard deviations of all regions under the current segmentation method:

[0047]

[0048] Traverse all segmentation methods, that is, traverse all combinations of START_A1 and SIZE_A1, and find the START_A1 and SIZE_A1 that make the smallest as the optimal result, and determine the final 4 continuous and non-overlapping elliptical sector areas based on the optimal result.

[0049] Further, judging the state of the indicator according to the color information of the pixel points within the observation area window includes:

[0050] Convert the indicator monitoring image into the HSV color space, calculate the average saturation of all pixels within the observation window, and denote it as S_AVG;

[0051] For any pixel point (x, y) within the observation window, judge whether it is red or green according to its color information. The formula is as follows:

[0052] MAXRGB(x,y)=max(R(x,y),G(x,y),B(x,y))

[0053]

[0054]

[0055] Where R(x,y), G(x,y), and B(x,y) respectively represent the red, green, and blue component values of the pixel point (x, y), S(x,y) represents the saturation of the pixel point (x, y). If Red(x,y) = 1, it means the pixel point (x, y) is red, and if Green(x,y) = 1, it means the pixel point (x, y) is green. Both may be 0 or both may be equal to 1 (such as pure yellow), which does not affect the detection result;

[0056] Count the number of all pixels within the observation window that are judged to be red and green, and denote them as RedCount and GreenCount. The formula is as follows:

[0057]

[0058]

[0059] Finally, judge the current state of the fish-eye type closing and opening. If RedCount > GreenCount + 100, it is judged as the "closed state"; if GreenCount > RedCount + 100, it is judged as the "open state"; otherwise, it is judged as the "abnormal state".

[0060] According to some embodiments, the second solution of the present invention provides a full-automatic fish-eye closing and opening indicator state recognition system, which adopts the following technical solution:

[0061] A full-automatic fish-eye closing and opening indicator state recognition system includes:

[0062] An image acquisition module, configured to obtain an indicator monitoring image;

[0063] The elliptical panel area determination module is configured to preprocess based on the indicator monitoring image, and extract the contour information of each connected domain according to the preprocessed indicator image to determine the elliptical panel area to be detected;

[0064] The observation window area determination module is configured to select the segmentation method with the smallest intra-class standard deviation according to the principle of the smallest intra-class standard deviation, divide the elliptical panel area to be detected into four fan-shaped areas, and then select the area with the most distinct color as the observation window by comparing the color distinctness of each area;

[0065] The indicator state determination module is configured to judge the state of the indicator according to the color information of the pixel points in the observation area window.

[0066] According to some embodiments, the third aspect of the present invention provides a computer-readable storage medium.

[0067] A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the steps in a fully automatic fish-eye closing and opening indicator state recognition method as described in the first aspect above.

[0068] According to some embodiments, the fourth aspect of the present invention provides a computer device.

[0069] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in a fully automatic fish-eye closing and opening indicator state recognition method as described in the first aspect above.

[0070] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0071] Based on no prior annotation information, the present invention realizes the recognition, positioning and segmentation of the elliptical panel area; at the same time, it can automatically segment the area of the observation window without annotation or template; this method gradually improves the pixel-level features from two levels of angle and area, can adapt to various colors and styles of fish-eye closing and opening indicators, has good robustness in the case of medium and small-range reflection, and can effectively identify the state of fish-eye closing and opening indicators without annotation and template.

[0072] The present invention proposes a method for segmenting the observation window of a fish-eye type switch-on and switch-off indicator based on the principle of minimum within-class standard deviation, which can automatically segment the area of the observation window without annotation or template. This method gradually enhances the pixel-level features from two levels: angle and region, and can adapt to various colors and styles of fish-eye type switch-on and switch-off indicators, and has good robustness in the case of medium and small range of reflection. The principle of minimum within-class standard deviation is to minimize the within-class standard deviation of each segmented region, which is equivalent to the maximum between-class standard deviation, and has good self-adaptability for such "two-peak segmentation problems with obvious foreground and background differences" (no need to select a threshold).

[0073] The present invention proposes a method for detecting the elliptical panel of a fish-eye type switch-on and switch-off indicator based on contour information, which realizes the recognition, positioning and segmentation of the elliptical panel area without prior annotation information. The main recognition basis of this method is the contour of the connected domain, and by calculating the similarity between the convex hull of the contour, the fitted ellipse and the contour itself, the target closest to the ellipse is selected as the detected elliptical panel area. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

[0075] Figure 1 is a flowchart of a method for automatically identifying the state of a fish-eye switch-on and switch-off indicator according to an embodiment of the present invention;

[0076] Figure 2 is the detection result and segmentation result of the elliptical panel area of the switch-on and switch-off in a method for automatically identifying the state of a fish-eye switch-on and switch-off indicator according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0077] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0078] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0079] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0080] Without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other.

[0081] Embodiment 1

[0082] As Figure 1 shown, this embodiment provides a method for automatically identifying the state of a fish-eye type closing and opening indicator. This embodiment takes the application of this method to a server as an example. It can be understood that this method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, web servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here. In this embodiment, the method includes the following steps:

[0083] Obtain the indicator monitoring image;

[0084] Based on the indicator monitoring image, perform preprocessing, and according to the preprocessed indicator image, extract the contour information of each connected domain to determine the elliptical panel area to be detected;

[0085] According to the principle of the minimum within-class standard deviation, select the segmentation method with the minimum within-class standard deviation, divide the elliptical panel area to be detected into four sector areas, and then by comparing the color vividness of each area, select the area with the most vivid color as the observation window;

[0086] Judge the state of the indicator according to the color information of the pixel points in the observation area window.

[0087] Specifically, the method described in this embodiment includes the following steps:

[0088] Step 1 Detect the elliptical panel area, that is, the fish-eye type closing and opening ellipse panel detection method based on contour information

[0089] Brief description: After preprocessing the indicator monitoring image, extract the contour information of each connected domain, and by calculating the similarity between the convex hull of the contour, the fitted ellipse and the contour itself, select the contour closest to the ellipse, and use the ellipse fitted by it as the detected elliptical panel area.

[0090] Detailed steps:

[0091] 1.1 Image grayscale processing.

[0092] 1.2 Perform binary segmentation on the image using the Sauvola algorithm. In this implementation step, the neighborhood size for calculating the mean variance is 75*75, the parameter R for calculating the local threshold T is taken as 128, and k is taken as 0.2.

[0093] 1.3 Invert the binary result of step 1.2.

[0094] 1.4 Use a 5*5 kernel for mathematical morphological opening operation to eliminate noise points.

[0095] 1.5 Detect the contours and store the contour results in a non-compressed manner, that is, store each point of the contour in sequence.

[0096] 1.6 After excluding the inner contours of the hole structure and the contours with too small area, traverse each contour to be screened, select a contour that is closest to an ellipse, and use the ellipse fitted by it as the detected panel area.

[0097] The specific steps are as follows:

[0098] 1.6.1 Denote each contour as Contours i , where i represents the contour number, the same below;

[0099] 1.6.2 Calculate the convex hull of each contour and denote it as ConvexHull i ;

[0100] 1.6.3 Calculate the ellipse fitted by each contour to obtain the elliptical contour denoted as Ellipse i ;

[0101] 1.6.4 Calculate the shape similarity between all Contours i and ConvexHull i and denote it as Sim1 i ;

[0102] 1.6.5 Calculate the shape similarity between all ConvexHull i and Ellipse i and denote it as Sim2 i ;

[0103] 1.6.6 Take Sim i =(Sim1 i +0.0001)*(Sim2 i +0.0001) as the evaluation index. The smaller this value is, the higher the similarity (coincidence degree) between the contour, convex hull and the fitted ellipse. Adding 0.0001 in the operation is to prevent a certain multiplier from being too small (such as 1e -6 ) from causing weight imbalance;

[0104] 1.6.7 Select Sim i The fitted ellipse of the smallest contour is used as the panel area, denoted as Ellipse. The mask of this area is denoted as Mask, and the center point is denoted as Center; The result example is as Figure 2 shown. The white ellipse in the figure is the result of the detected elliptical panel area.

[0105] Step 2 Determine the observation window area, a fisheye opening and closing observation window segmentation method based on the principle of minimum within-class standard deviation;

[0106] Brief description: The elliptical area obtained in Step 1 is divided into 4 fan-shaped areas. (Traverse all possible segmentation methods) According to the principle of minimum within-class standard deviation, select the segmentation method with the smallest within-class standard deviation as the final segmentation result. Then, by comparing the color vividness of each area, select the area with the most vivid color as the observation window.

[0107] Detailed steps:

[0108] 2.1 Traverse each point in the Mask area of the original image, calculate its angle relative to Center and round it to an integer, denoted as d. If d is equal to 360, then make d equal to 0. Obviously, the integer d ∈ [0, 359].

[0109] 2.2 Establish 360 sets, denoted as Points d , and according to the angle calculated in Step 2.1, classify each point into the set Points d corresponding to its angle d.

[0110] 2.3 Establish three arrays Avg_R, Avg_G, and Avg_B. Traverse the integer angle d in [0, 359], calculate the mean values of the points in each Points d in the red, green, and blue channels, and store them into Avg_R[d], Avg_G[d], and Avg_B[d] in sequence. Thus, we obtain the mean values of the point sets at each angle in each red, green, and blue component. At this time, the size of each array is 360. To meet the needs of subsequent calculations, expand its size to 720. For any integer d ∈ [0, 359], Avg_R[d + 360] = Avg_R[d], Avg_G[d + 360] = Avg_G[d], and Avg_B[d + 360] = Avg_B[d].

[0111] 2.4 Based on the mean values obtained in Step 2.3 as the basic data and taking the principle of minimum within-class standard deviation as the segmentation basis, divide the complete 360-degree panel area into 4 continuous and non-overlapping elliptical fan-shaped areas, denoted as A n, where \(n\in\{1,2,3,4\}\) (all subscripts \(n\) in this and subsequent texts refer to the region numbers), it is required that the central angles of \(A1\) and \(A3\) face each other, and the central angles of \(A2\) and \(A4\) face each other. The specific steps are as follows:

[0112] 2.4.1 Denote the size of the central angle of any \(A1\) as \(SIZE\_A1\). Set \(SIZE\_A1\in[30,90]\) based on prior knowledge, and \(SIZE\_A1\) is an integer. Then, it can be known that the size of the central angle of \(A2\), \(SIZE\_A2 = 180 - SIZE\_A1\), the size of the central angle of \(A3\), \(SIZE\_A3 = A1\_SIZE\), and the size of the central angle of \(A4\), \(SIZE\_A4 = 180 - SIZE\_A1\).

[0113] 2.4.2 Denote the starting angle of any \(A1\) as \(START\_A1\), \(START\_A1\in[0,359]\), and \(START\_A1\) is an integer. Then, it can be known that the starting and ending intervals of the angles corresponding to \(A1\) are \([START\_A1, START\_A1 + SIZE\_A1)\). By analogy, the starting and ending intervals of the angles corresponding to regions \(A2\), \(A3\), and \(A4\) can be obtained.

[0114] 2.4.3 Based on steps 2.4.1 and 2.4.2, taking the red component as an example, calculate the standard deviation between the means of each angle within the \(A\) n region, that is, calculate the standard deviation of the values within the subscript range \([START\_A\) n , \(START\_A\) n + SIZE\_A\) n ), denoted as Similarly, the standard deviation under the green component is The standard deviation under the blue component is

[0115] 2.4.4 Calculate the within-class standard deviation after color fusion for each \(A\) n region

[0116]

[0117] 2.4.5 For any determined \(START\_A1\) and \(SIZE\_A1\), regarded as a certain segmentation method, calculate the weighted sum of the within-class standard deviations of all regions under the current segmentation method:

[0118]

[0119] 2.4.6 Traverse all segmentation methods, that is, traverse all combinations of \(START\_A1\) and \(SIZE\_A1\), and find the \(START\_A1\) and \(SIZE\_A1\) that minimize as the optimal result. At this time, the ranges of the 4 regions are all determined, such as Figure 2As shown, the optimal segmentation result is four sectors separated by four white "lines" within the ellipse range.

[0120] 2.5 For each pixel point (x, y) within the elliptical region of the original image, calculate

[0121] MAXRGB(x,y) = max(R(x,y), G(x,y), B(x,y)) (3)

[0122] MINRGB(x,y) = min(R(x,y), G(x,y), B(x,y)) (4)

[0123] COLORFUL(x,y) = (MAXRGB - MINRGB) * MAXRGB (5)

[0124] Among them, R(x,y), G(x,y), B(x,y) represent the red, green, and blue component values of the pixel point (x, y) respectively, and COLORFUL(x,y) represents the color vividness of the pixel point (x, y) (similar to the saturation in the HSV color space).

[0125] 2.6 Compare the average values of COLORFUL(x,y) of the pixels within the four sector regions, and take the sector region with the largest average value as the finally determined observation window.

[0126] Step 3 Determine the current opening and closing state of the fish-eye type switch

[0127] 3.1 Convert the image to the HSV color space, and calculate the average saturation of all pixels within the observation window obtained in step 2, denoted as S_AVG.

[0128] 3.2 For any pixel point (x, y) within the observation window, judge whether it is red or green according to its color information. The formula is as follows:

[0129] MAXRGB(x,y) = max(R(x,y), G(x,y), B(x,y)) (3)

[0130]

[0131]

[0132] Among them, R(x,y), G(x,y), B(x,y) represent the red, green, and blue component values of the pixel point (x, y) respectively, S(x,y) represents the saturation of the pixel point (x, y). If Red(x,y) = 1, it means the pixel point (x, y) is red, and Green(x,y) = 1 means the pixel point (x, y) is green. Both may be 0, or both may be equal to 1 (such as pure yellow), which does not affect the detection result.

[0133] 3.3 Count the number of pixels determined to be red and green in step 3.2 within the statistical observation window, denoted as RedCount and GreenCount, and the formula is as follows:

[0134] RedCount = ∑ (x,y) Red(x,y) (8)

[0135] GreenCount = ∑ (x,y) Green(x,y) (9)

[0136] 3.4 Finally, judge the current state of the fish-eye type switch. If RedCount > GreenCount + 100, it is judged as the "closed state"; if GreenCount > RedCount + 100, it is judged as the "open state"; in other cases, it is judged as the "abnormal state".

[0137] Embodiment 2

[0138] This embodiment provides a full-automatic fish-eye switch indicator state recognition system, including:

[0139] An image acquisition module configured to obtain an indicator monitoring image;

[0140] An elliptical panel area determination module configured to perform preprocessing based on the indicator monitoring image, and extract the contour information of each connected domain according to the preprocessed indicator image to determine the elliptical panel area to be detected;

[0141] An observation window area determination module configured to select the segmentation method with the smallest within-class standard deviation according to the principle of the smallest within-class standard deviation, divide the elliptical panel area to be detected into four fan-shaped areas, and then select the area with the most distinct color as the observation window by comparing the color vividness of each area;

[0142] An indicator state determination module configured to judge the state of the indicator according to the color information of the pixel points within the observation area window.

[0143] The examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0144] In the above embodiments, the descriptions of each embodiment have their own focuses. For parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0145] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above-mentioned modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0146] Embodiment III

[0147] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps in a fully automatic fish-eye switch-on / off indicator state recognition method as described in Embodiment I above.

[0148] Embodiment IV

[0149] This embodiment provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in a fully automatic fish-eye switch-on / off indicator state recognition method as described in Embodiment I above.

[0150] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can adopt the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0151] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the specified functions in Figure 1 one or more flows or multiple flows and / or blocks Figure 1 one or more blocks or multiple blocks.

[0152] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the specified functions in the flow Figure 1One or more processes and / or blocks Figure 1 The functions specified in one or more blocks.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 One or more processes and / or blocks Figure 1 The steps of the functions specified in one or more blocks.

[0154] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0155] Although the specific implementation manners of the present invention have been described above in conjunction with the accompanying drawings, it is not a limitation on the protection scope of the present invention. Those skilled in the art should understand that based on the technical solutions of the present invention, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of the present invention.

Claims

1. A method for automatically identifying the state of a fish-eye type closing and opening indicator, characterized in that, Including: Obtain the indicator monitoring image; Perform preprocessing based on the indicator monitoring image. According to the preprocessed indicator image, extract the contour information of each connected component to determine the area of the elliptical panel to be detected, including: Detect the contour of the preprocessed indicator monitoring image, and store the contour result in a non-compressed manner, that is, store each point of the contour in sequence to obtain all the contours of the preprocessed indicator monitoring image; Traverse each contour to be screened, select a contour with the highest similarity to the ellipse, and use the ellipse fitted by it as the area of the elliptical panel to be detected, including: For each contour denoted as Contours i , where i represents the contour number; Determine the convex hull of each contour, denoted as ConvexHull i ; Determine the ellipse fitted to each contour, and denote the elliptical contour as Ellipse i ; Calculate all Contours i and the ConvexHull i The shape similarity is denoted as Sim1 i ; Calculate all ConvexHulls i and Ellipse i The shape similarity is denoted as Sim2 i ; Take As an evaluation index, the smaller this value is, the higher the similarity between the contour, convex hull and the fitted ellipse is; Select Sim i The fitted ellipse of the smallest contour is used as the ellipse panel area to be detected, denoted as Ellipse, the mask of this area is denoted as Mask, and the center point is denoted as Center; According to the principle of the minimum within-class standard deviation, select the segmentation method with the smallest within-class standard deviation, divide the area of the elliptical panel to be detected into four fan-shaped areas, and then select the area with the most distinct color as the observation window by comparing the color vividness of each area; Judge the state of the indicator according to the color information of the pixel points in the observation area window.

2. The fully automatic state recognition method for the fish-eye type opening / closing indicator according to claim 1, characterized in that The preprocessing based on the indicator monitoring image includes: Grayscale the indicator monitoring image; Use the Sauvola algorithm to perform binary segmentation on the grayscaled indicator monitoring image; Invert the binary result; Perform a morphological opening operation using a 5*5 kernel to eliminate noise points to obtain the preprocessed indicator monitoring image.

3. The fully automatic state recognition method for fish-eye type opening and closing indicators according to claim 1, wherein, According to the principle of the minimum within-class standard deviation, select the segmentation method with the smallest within-class standard deviation as the final segmentation result, and then select the area with the most distinct color as the observation window by comparing the color vividness of each area. The specific steps are as follows: Traverse each point in the area of the elliptical panel to be detected in the indicator monitoring image, calculate its angle relative to the center point of the area and round it to an integer and record it as d. If d is equal to 360, then make d equal to 0, and the integer d ∈ [0, 359]; Establish 360 sets, denoted as Points d , and according to the angle d, classify each point into the set Points corresponding to its angle d d ; Create three arrays Avg_R, Avg_G, and Avg_B. Traverse the integer angle d within [0, 359], calculate the average value of the points in each Points d in the red, green, and blue channels, and store them in Avg_R[d], Avg_G[d], and Avg_B[d] in sequence, obtaining the average value of the point set at each angle on each red, green, and blue component; Based on the mean values of the point sets at each angle on each red, green, and blue component, and taking the principle of the minimum within-class standard deviation as the segmentation criterion, the complete 360-degree elliptical panel area is segmented into 4 continuous and non-overlapping elliptical fan-shaped areas, denoted as A n , n ∈ {1, 2, 3, 4}, the central angles of A1 and A3 are opposite to each other, and the central angles of A2 and A4 are opposite to each other; For each pixel point (x, y) in the elliptical area of the original image, calculate Among them, respectively represent the red, green, and blue component values of the pixel point (x, y), represents the color vividness of the pixel point (x, y); Compare the means of the pixels in the four fan-shaped regions, and take the fan-shaped region with the largest mean as the finally determined observation window.

4. The fully automatic state recognition method for the fish-eye type closing and opening indicator according to claim 3, wherein Based on the mean value of each point set at each angle on each red, green, and blue component, and taking the principle of the minimum within-class standard deviation as the segmentation basis, divide the complete 360-degree elliptical panel area into 4 continuous and non-overlapping elliptical fan-shaped areas. The specific steps are as follows: For any central angle size of A1, denoted as SIZE_A1, set SIZE_A1 ∈ [30, 90] by prior knowledge, and SIZE_A1 is an integer. Then it can be known that the central angle size of A2, SIZE_A2 = 180 - SIZE_A1, the central angle size of A3, SIZE_A3 = A1_SIZE, and the central angle size of A4, SIZE_A4 = 180 - SIZE_A1; For any starting angle of A1, denoted as START_A1, START_A1 ∈ [0, 359], and START_A1 is an integer. Then it can be known that the start and end interval of the angle corresponding to A1 is [START_A1, START_A1 + SIZE_A1). By analogy, the start and end intervals of the angles corresponding to A2, A3, and A4 areas can be obtained; Taking the red component as an example, calculate A n The standard deviation between the means of each angle within the region, that is, calculate the standard deviation of the values in the array Avg_R at the subscript [START_A n , START_A n + SIZE_A n ), denoted as ; Similarly, the standard deviation under the green component is ; The standard deviation under the blue component is ; Calculate each A n The within-class standard deviation after regional color fusion is as follows: ； For any determined START_A1 and SIZE_A1, regarded as a determined segmentation method, calculate the weighted sum of the within-class standard deviations of all areas under the current segmentation method: Traverse all segmentation methods, that is, traverse all combinations of START_A1 and SIZE_A1, and obtain the START_A1 and SIZE_A1 that minimize as the optimal result, and determine the final 4 consecutive and non-overlapping elliptical sector regions based on the optimal result.

5. The fully automatic state recognition method for the fish-eye type opening and closing indicator according to claim 1, characterized in that Judging the state of the indicator according to the color information of the pixel points within the observation region window includes: Converting the indicator monitoring image into the HSV color space, calculating the average saturation of all pixels within the observation window, and denoting it as S_AVG; For any pixel point (x, y) within the observation window, judging whether it is red or green according to its color information, and the formula is as follows: wherein respectively represent the red, green, and blue component values of the pixel point (x, y), represents the saturation of the pixel point (x, y). If then it means the pixel point (x, y) is red, represents that the pixel point (x, y) is green; Counting the number of pixels determined to be red and green within the observation window, and denoting them as RedCount and GreenCount, and the formula is as follows: Finally, judging the current state of the fish-eye type closing and opening. If RedCount > GreenCount + 100, it is judged as the "closed state"; if GreenCount > RedCount + 100, it is judged as the "open state"; in other cases, it is judged as the "abnormal state".

6. A full-automatic state recognition system for fish-eye opening and closing indicators, characterized in that, Including: An image acquisition module configured to acquire the indicator monitoring image; An elliptical panel region determination module configured to perform preprocessing based on the indicator monitoring image, and extract the contour information of each connected domain according to the preprocessed indicator image to determine the elliptical panel region to be detected, including: Detecting the contour of the preprocessed indicator monitoring image, storing the contour result in a non-compressed manner, that is, storing each point of the contour in sequence to obtain all the contours of the preprocessed indicator monitoring image; Traversing each contour to be screened, selecting a contour with the highest similarity to an ellipse, and using the ellipse fitted by it as the elliptical panel region to be detected, including: For each contour denoted as Contours i , where i represents the contour number; Determine the convex hull of each contour, denoted as ConvexHull i ; Determine the ellipse fitted to each contour, and record the elliptical contour as Ellipse i ; Calculate all Contours i and ConvexHull i The shape similarity is denoted as Sim1 i ; Calculate all ConvexHulls i and Ellipse i The shape similarity is denoted as Sim2 i ; Take As an evaluation index, the smaller this value is, the higher the similarity between the contour, convex hull and the fitted ellipse is; Select Sim i The fitted ellipse of the smallest contour is used as the ellipse panel area to be detected, denoted as Ellipse. The mask of this area is denoted as Mask, and the center point is denoted as Center; An observation window region determination module configured to select the segmentation method with the smallest intra-class standard deviation according to the principle of the smallest intra-class standard deviation, divide the elliptical panel region to be detected into four fan-shaped regions, and then select the region with the most distinct color as the observation window by comparing the color vividness of each region; An indicator state determination module configured to judge the state of the indicator according to the color information of the pixel points within the observation region window.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in a full-automatic fish-eye closing and opening indicator state recognition method as described in any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in a full-automatic fish-eye closing and opening indicator state recognition method as described in any one of claims 1-5.

Images