An image-based electrical cabinet with lamp button state recognition and positioning method

Abstract

The application provides an image-based electrical cabinet with lamp button state recognition and positioning method, and belongs to the field of industrial image processing. Mainly solve the problem that the traditional image processing method is not accurate in recognizing and positioning the electrical cabinet with lamp button, judging the color and on-off state of the button in the industrial automation production process. The method comprises the following steps: first, shooting the image and taking a screenshot, then based on the color channel, extracting the image, subtracting the image, filtering, binarizing and extracting the contour, then screening the minimum circumscribed rectangle of the image contour to obtain the target circumscribed rectangle, and calculating the pixel coordinates of the button; then, the target circumscribed rectangle is used to cut the image, and the color and on-off state of the button are judged according to the image mean and image standard deviation of the image. The application can realize the recognition and positioning of the electrical cabinet with lamp button, and judge the color and on-off state of the button based on the visual information of the camera, with high recognition accuracy and accurate positioning.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of industrial image processing, and particularly relates to a kind of based on image's electrical cabinet with light button state recognition and positioning method, the purpose is to the position of button in electrical cabinet is positioned and judges button color and bright and dark state. BACKGROUND

[0002] The era of robots is coming, and robot technology is deeply changing the way people produce and live. With the continuous development and maturity of its technology, the robot industry is gradually shifting from manufacturing to life, but the environment in real life is complex and variable. To make robots serve people's life and work in various situations, many problems still need to be solved.

[0003] The substation is a place for transforming, concentrating and distributing voltage and current of electric energy. Due to the factors such as small size of indoor equipment and narrow movement space of substation, most of the operations of equipment in the station rely on manual operation. The operator not only wastes time and effort, but also faces great labor risks. In order to realize the unmanned management of the substation, many substations use mobile mechanical arms to replace operators to operate the equipment in the substation, such as giving instructions to the button switch on the electrical cabinet. However, in the actual application process, the robot is difficult to accurately identify and locate the target button, and often causes greater safety problems because the robot cannot correctly complete the operation process. Therefore, in order to solve the problem of accurately identifying and locating the target button switch and accurately pressing the button switch to complete the operation process, the present application proposes a kind of based on image's electrical cabinet with light button state recognition and positioning method, which uses reliable image taking device to develop effective identification algorithm to achieve the purpose of replacing manual operation with robot technology to accurately operate the equipment.

[0004] The Chinese invention patent document (CN201710308741.X) disclosed on May 28, 2021, "An elevator button recognition method based on autonomous tilt correction and projection histogram", uses the method of generating elevator button image template by training and calculating the projection histogram of elevator button image in horizontal and vertical directions to train and recognize the elevator button. Although this method can calculate the correlation using the projection histogram in the elevator button image template generated during training, and according to the correlation, the accurate elevator button area is calculated, and then the position of each elevator button is determined and the corresponding elevator button image template is called to recognize the elevator button, the training process is complex and time-consuming, and it does not have the generality of recognition, the use range is relatively limited, and it is difficult to adapt to the recognition and positioning of buttons in different scenarios.

[0005] In the robot operation, the accuracy of image recognition positioning directly affects whether the robot can accurately complete the operation. The research on the image-based electrical cabinet button state recognition and positioning method still has the following technical problems:

[0006] 1. When identifying and positioning the button of the photographed image, it is easy to be affected by other types of switches and other interference objects on the electrical cabinet panel, thereby causing misjudgment of the target button identification and positioning.

[0007] 2. When judging the color and on-off state of the target button, it is easy to be affected by the light condition, thereby incorrectly judging the color and on-off state of the target button. SUMMARY

[0008] In view of the deficiencies of the prior art, the present application provides an image-based electrical cabinet button state recognition and positioning method, which can enable the robot to accurately recognize and position the electrical cabinet button, and then accurately press the button.

[0009] The purpose of the present application is achieved, and the present application provides an image-based electrical cabinet button state recognition and positioning method, which involves a system including a robot and an electrical cabinet. The operating panel of the electrical cabinet is provided with four switch buttons, which are respectively denoted as button 1, button 2, button 3 and button 4. Each button switch has two states of on and off. The color of button 1 is green, the color of button 2 is red, the color of button 3 is green, and the color of button 4 is red. The center points of button 1, button 2, button 3 and button 4 are respectively denoted as point C1, point C2, point C3 and C4. The straight line connecting point C1 and C2 is a horizontal line 1, and the straight line connecting point C3 and C4 is a horizontal line 2, and the horizontal line 2 is located below and to the right of the horizontal line 1.

[0010] The robot includes an AGV trolley, a 6-degree-of-freedom mechanical arm, an end effector and a depth camera. The 6-degree-of-freedom mechanical arm includes a 6-rotary-joint and a mechanical arm base. The 6-rotary-joint includes rotary joint one, rotary joint two, rotary joint three, rotary joint four, rotary joint five and rotary joint six in sequence from the mechanical arm base. The 6-rotary-joint rotates around the first joint axis, the second joint axis, the third joint axis, the fourth joint axis, the fifth joint axis and the sixth joint axis in sequence. The fourth joint axis and the fifth joint axis are perpendicular to each other, and the fifth joint axis and the sixth joint axis are perpendicular to each other. The end effector and the depth camera are both installed on the end of the 6-degree-of-freedom mechanical arm, and the optical axis of the depth camera is parallel to the sixth joint axis. The depth camera moves with the 6-degree-of-freedom mechanical arm.

[0011] The recognition and positioning method includes the following steps:

[0012] Step 1, define the pixel coordinate system: the pixel coordinate system is a rectangular coordinate system established with the upper left corner of the image as the origin and in pixel units, including mutually perpendicular pixel U axis and pixel V axis, the horizontal coordinate on the pixel U axis is the column number of the pixel in its image, and the vertical coordinate on the pixel V axis is the row number of the pixel in its image;

[0013] Step 2, the robot moves in front of the button area of the electrical cabinet, drives the 6-DOF mechanical arm, and keeps the depth camera optical axis perpendicular to the front of the electrical cabinet and the depth camera plane parallel to the electrical cabinet plane, then uses the depth camera to shoot the electrical cabinet button area, and records the image obtained by shooting as image M, which is an RGB three-channel image with a width of 1280 pixels, a height of 720 pixels and a pixel depth of 24 bits; record the pixel point D0 at the upper left corner of the image M, and the pixel coordinate of the pixel point D0 in the pixel coordinate system of the image M is (0, 0);

[0014] The image M is intercepted so that there is only one complete button in the intercepted image, and the button in the image M0 is recorded as the target button, and the intercepted image is recorded as image M0, the pixel point D1 at the upper left corner of the image M0 is recorded as pixel point D1, and the pixel coordinate of the pixel point D1 in the pixel coordinate system of the image M is (u1, v1);

[0015] The image M0 is subjected to RGB three-channel segmentation to obtain image L0 representing R channel, image L1 representing G channel and image L2 representing B channel respectively;

[0016] Step 3, the button color of the target button is red, and the image subtraction is performed on the image L0 and the image L2 obtained in step 2 to obtain an image M1 which has the same pixel coordinate system as the image M0 and the same pixel coordinate origin, M1=L0-L2;

[0017] Step 4, the image M1 is subjected to mean filtering to obtain a more smooth image M2 which has the same pixel coordinate system as the image M1 and the same pixel coordinate origin; it is assumed that the image M2 includes H pixel points, and any one pixel point in the image M2 is recorded as pixel point D1 η , η = 1, 2, … H, the pixel coordinate of the pixel point D1 η in the pixel coordinate system of the image M2 is (u1 η , v1 η ), and the gray value of the pixel point D1 η is I1 η ;

[0018] The image M2 is subjected to binarization processing, specifically, first, a first constant I min = 0, a second constant I max = 255 and a binarization threshold T are given, and then each pixel value I1η The following assignments are made:

[0019] If I1 η <T, let I1 η = I min ;

[0020] If I1 η ≥ T, let I1 η = I max ;

[0021] Let the re-assigned image M2 be denoted as image M3;

[0022] Step 5, assuming there are n boundary disjoint pixel regions in image M3, contour extraction is performed on any one of the regions, and is denoted as contour O k , k = 1, 2,... n, n is a positive integer;

[0023] For each O k , the minimum circumscribed rectangle is obtained, and is denoted as the minimum circumscribed rectangle R k of the region, and n minimum circumscribed rectangles R k of the regions are obtained, and a set R is generated, R = {R1, R2,..., R k ,..., R n};

[0024] Step 6, given an aspect ratio α, all minimum circumscribed rectangles R k in set R that satisfy the aspect ratio condition 0.8 ≤ α ≤ 1.2 are extracted, and a minimum circumscribed rectangle set that satisfies the aspect ratio condition is formed, and is denoted as set R', R' = {R1, R2,..., R m}, m is a positive integer, m ≤ n; the minimum circumscribed rectangle with the largest area in set R' is found, and is denoted as target circumscribed rectangle R' max ;

[0025] Step 7, assuming the width of target circumscribed rectangle R' max is w pixels and the height is h pixels, the upper left corner pixel point of target circumscribed rectangle R' max is denoted as pixel point D2, and the geometric center of target circumscribed rectangle R' max is denoted as pixel point P, then the pixel coordinates of pixel point D2 in the pixel coordinate system of image M3 are (u2, v2), and the pixel coordinates of pixel point P in the pixel coordinate system of image M3 are

[0026] Step 8, a new image Mc is cut from image M1, with pixel point D2 as the starting point, and the width and height are w pixels and h pixels, respectively;

[0027] Let image Mc contain Γ pixels, and denote any pixel in image Mc as pixel D2. δ δ = 1, 2, ..., Γ, pixel D2 δ The pixel coordinates in the pixel coordinate system of image Mc are (u2) δ v2 δ ), pixel D2 δ The grayscale value is I2 δ ;

[0028] The mean and standard deviation (stdDev) of image Mc are calculated using the following formulas:

[0029]

[0030]

[0031] Step 9, choose between the following two options:

[0032] The first method: In step 3, set the target button color to red, and then proceed to step 9, and then to step 10.

[0033] The second scenario involves confirming in step 10 that the target button is green, and after performing the specified operation in step 10, returning to step 4 and proceeding to step 9, then proceeding to step 11.

[0034] Step 10: Given the color judgment threshold STDDEV, perform the following judgment:

[0035] If stdDev ≥ STDDEV, confirm that the target button color is red and proceed to step 11;

[0036] If stdDev < STDDEV, confirm that the target button color is green; then, subtract the images L0 and L1 obtained in step 2 to obtain an image M1′ with the same pixel coordinate system and pixel coordinate origin as image M0, M1′ = L1 - L0; replace image M1 in step 4 with image M1′ and proceed to step 4.

[0037] Step 11: Given the on / off state judgment threshold MEAN, perform the following judgment:

[0038] If mean ≥ MEAN, confirm that the target button is on.

[0039] If mean < MEAN, confirm that the target button is off.

[0040] Step 12, denote the pixel coordinates of pixel point P in the pixel coordinate system of image M as (u α v α), u α The value is the x-coordinate of pixel D0, translated along the pixel U-axis of image M. The value after units, v α The value is the ordinate of pixel D0, translated along the pixel V-axis of image M. The values ​​after a certain number of units are calculated using the following formulas:

[0041]

[0042]

[0043] The pixel coordinates (u) of pixel point P in the pixel coordinate system of image M α v α This indicates the position of the target button in image M.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows;

[0045] 1. This invention adopts an image-based method for recognizing and locating the status of illuminated buttons in electrical cabinets. First, the button foreground and background are segmented according to the button's color. Then, the button image is extracted using a binarization method. Finally, the button is accurately identified using contour extraction and minimum bounding rectangle methods. Compared with the traditional button location method based on Hough circle detection, this method greatly improves the recognition speed and accuracy.

[0046] 2. This invention determines the button color and on / off state by calculating the image mean and standard deviation of the identified image region, which is fast and efficient.

[0047] 3. The present invention adopts an image-based method for identifying and locating the status of illuminated buttons in electrical cabinets. It has the characteristics of good versatility and high accuracy, even when the button captured by the camera is distorted or the button itself is irregular. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the robot and electrical cabinet status in an embodiment of the present invention;

[0049] Figure 2 This is a front view of the electrical cabinet operation panel in an embodiment of the present invention;

[0050] Figure 3 This is a flowchart of the identification and positioning method of the present invention;

[0051] Figure 4 This is a schematic diagram showing the pixel coordinates of the target button in the captured image;

[0052] Figure 5 This is a simplified flowchart of the identification and positioning method of the present invention. DETAILED DESCRIPTION

[0053] The application will be described in detail below through examples.

[0054] Figure 1 The robot and the electrical cabinet state diagram in the embodiment of the application, Figure 2 The electrical cabinet operation panel front view in the embodiment of the application is shown in Figure 1. Figure 1 、 Figure 2 It can be seen that the application provides an image-based electrical cabinet lamp button state recognition and positioning method, and the system involved in the recognition and positioning method comprises a robot and an electrical cabinet. The operation panel of the electrical cabinet is provided with four switch buttons, which are respectively denoted as button 1, button 2, button 3 and button 4. Each button switch has two states of on and off. The color of button 1 is green, the color of button 2 is red, the color of button 3 is green, and the color of button 4 is red. The center points of button 1, button 2, button 3 and button 4 are respectively denoted as point C1, point C2, point C3 and C4. The straight line connecting point C1 and C2 is a horizontal line 1, the straight line connecting point C3 and C4 is a horizontal line 2, and the horizontal line 2 is located at the lower right of the horizontal line 1.

[0055] The robot comprises an AGV trolley, a 6-degree-of-freedom mechanical arm, an end effector and a depth camera. The 6-degree-of-freedom mechanical arm comprises a 6-degree-of-freedom mechanical arm base and six rotating joints. The six rotating joints are sequentially rotating joint one, rotating joint two, rotating joint three, rotating joint four, rotating joint five and rotating joint six from the mechanical arm base. The six rotating joints are sequentially rotated around the first joint axis, the second joint axis, the third joint axis, the fourth joint axis, the fifth joint axis and the sixth joint axis. The fourth joint axis and the fifth joint axis are perpendicular to each other, and the fifth joint axis and the sixth joint axis are perpendicular to each other. The end effector and the depth camera are both installed on the end of the 6-degree-of-freedom mechanical arm, and the optical axis of the depth camera is parallel to the sixth joint axis. The depth camera moves with the 6-degree-of-freedom mechanical arm.

[0056] Figure 3 The flowchart of the recognition and positioning method of the application, Figure 4 The target button position diagram, Figure 5 The flowchart of the recognition and positioning method of the application is shown in Figure 2. Figures 3-5 It can be seen that the recognition and positioning method in the application comprises the following steps:

[0057] Step 1, define the pixel coordinate system: the pixel coordinate system is a rectangular coordinate system established with the upper left corner of the image as the origin and with pixels as the unit. The pixel U axis and the pixel V axis are perpendicular to each other. The horizontal coordinate of the pixel U axis is the column number of the pixel in the image, and the vertical coordinate of the pixel V axis is the row number of the pixel in the image.

[0058] Step 2, the robot moves in front of the button area of the electrical cabinet, drives the 6-DOF mechanical arm, and keeps the depth camera optical axis vertical to the front of the electrical cabinet and the depth camera plane parallel to the plane of the electrical cabinet, then uses the depth camera to shoot the button area of the electrical cabinet, and records the image obtained by shooting as image M, which is an RGB three-channel image with a width of 1280 pixels, a height of 720 pixels, and a pixel depth of 24 bits; record the pixel point D0 at the upper left corner of the image M, and the pixel coordinates of the pixel point D0 in the pixel coordinate system of the image M are (0, 0);

[0059] The image M is intercepted so that there is only one complete button in the intercepted image, and the button in the image M0 is recorded as the target button, and the intercepted image is recorded as image M0, and the pixel point D1 at the upper left corner of the image M0 is recorded as pixel point D1, and the pixel coordinates of the pixel point D1 in the pixel coordinate system of the image M are (u1, v1);

[0060] The image M0 is subjected to RGB three-channel segmentation to obtain an image L0 representing the R channel, an image L1 representing the G channel, and an image L2 representing the B channel.

[0061] Step 3, the button color of the target button is red, the image L0 and the image L2 obtained in step 2 are subjected to image subtraction to obtain an image M1 which is the same as the pixel coordinate system of the image M0 and has the same pixel coordinate origin, M1 = L0-L2.

[0062] Step 4, the image M1 is subjected to mean filtering to obtain a more smooth image M2 which is the same as the pixel coordinate system of the image M1 and has the same pixel coordinate origin; it is assumed that the image M2 includes H pixel points, and any one pixel point in the image M2 is recorded as pixel point D1 η , η = 1, 2,... H, the pixel coordinates of the pixel point D1 η in the pixel coordinate system of the image M2 are (u1 η , v1 η ), and the gray value of the pixel point D1 η is I1 η ;

[0063] The image M2 is subjected to binarization processing, specifically, a first constant I min = 0, a second constant I max = 255, and a binarization threshold T are given, and then each pixel value I1 η in the image M2 is assigned as follows:

[0064] If I1 η < T, I1 η = I min ;

[0065] If I1 η≥ T, let I1 η = I max ;

[0066] Let the re-assigned image M2 be denoted as image M3.

[0067] Step 5, assuming that there are n boundary disjoint pixel regions in image M3, contour extraction is performed on any one of the regions, and is denoted as contour O k , k = 1, 2,... n, n is a positive integer;

[0068] For each O k , a minimum circumscribed rectangle is obtained, and is denoted as the minimum circumscribed rectangle R k of the region, and n minimum circumscribed rectangles R k of the regions are obtained, and a set R is generated, R = {R1, R2,..., R k ,..., R n}.

[0069] Step 6, given an aspect ratio α, all minimum circumscribed rectangles Rk in set R that satisfy the aspect ratio condition 0.8 ≤ α ≤ 1.2 are extracted, to form a minimum circumscribed rectangle set that satisfies the aspect ratio condition, and is denoted as set R', R' = {R1, R2,..., R m}, m is a positive integer, m ≤ n; the minimum circumscribed rectangle with the largest area in set R' is found, and is denoted as the target circumscribed rectangle R' max .

[0070] Step 7, assuming that the width of target circumscribed rectangle R' max is w pixels and the height is h pixels, the upper left corner pixel point of target circumscribed rectangle R' max is denoted as pixel point D2, and the geometric center of target circumscribed rectangle R' max is denoted as pixel point P, then the pixel coordinates of pixel point D2 in the pixel coordinate system of image M3 are (u2, v2), and the pixel coordinates of pixel point P in the pixel coordinate system of image M3 are

[0071] Step 8, a new image Mc is cut from image M1, with pixel point D2 as the starting point, a width of w pixels and a height of h pixels;

[0072] Assuming that image Mc includes Γ pixel points, any one pixel point in image Mc is denoted as pixel point D2 δ , δ = 1, 2,... Γ, the pixel coordinates of pixel point D2 δ in the pixel coordinate system of image Mc are (u2 δ , v2 δ ), and the gray value of pixel point D2 δ is I2 δ;

[0073] Calculate the image mean mean and the image standard deviation stdDev of the image M0, whose calculation formulae are as follows:

[0074]

[0075]

[0076] Step 9, select the following two cases:

[0077] The first case: in step 3, the target button color is red, and on the basis of which, step 9 is entered, step 10 is entered.

[0078] The second case: in step 10, the target button color is confirmed to be green, and after the specified operation in step 10 is performed and returned to step 4, and on the basis of which, step 9 is entered, step 11 is entered.

[0079] Step 10, give a color judgment threshold STDDEV, and make the following judgment:

[0080] If stdDev≥STDDEV, confirm that the target button color is red, and enter step 11;

[0081] If stdDev<STDDEV, confirm that the target button color is green; then, perform image subtraction on the image L0 and the image L1 obtained in step 2 to obtain an image M1', which has the same pixel coordinate system and the same pixel coordinate origin as the image M0, and i M1'=L1-L0; replace the image M1 in step 4 with the image M1' and enter step 4.

[0082] Step 11, give a bright-dark state judgment threshold MEAN, and make the following judgment:

[0083] If mean≥MEAN, confirm that the bright-dark state of the target button is bright;

[0084] If mean<MEAN, confirm that the bright-dark state of the target button is dark.

[0085] Step 12, record the pixel coordinates of the pixel point P in the pixel coordinate system of the image M as (u α , v α ), the value of u α is the value of the horizontal coordinate of the pixel point D0 after being translated along the pixel U axis of the image M by units, and the value of v α is the value of the vertical coordinate of the pixel point D0 after being translated along the pixel V axis of the image M by units, whose calculation formulae are as follows:

[0086]

[0087]

[0088] The pixel coordinate (u α , v α ) of the pixel point P in the pixel coordinate system of the image M is the position of the target button in the image M.

[0089] In this embodiment, the color judgment threshold value STDDEV = 50, the bright-dark state judgment threshold value MEAN = 100, and the binarization threshold value T = 30.

Claims

1. An image-based electrical cabinet with lamp button state recognition and positioning method, characterized in that, The system involved in the identification and positioning method comprises a robot and an electrical cabinet; four switch buttons are arranged on the operation panel of the electrical cabinet, and are respectively recorded as button 1, button 2, button 3 and button 4; each button switch has two states of being on and off, the color of button 1 is green, the color of button 2 is red, the color of button 3 is green, and the color of button 4 is red; the center points of button 1, button 2, button 3 and button 4 are recorded as point C1, point C2, point C3 and C4 respectively, wherein the straight line connecting point C1 and C2 is a horizontal line 1, the straight line connecting point C3 and C4 is a horizontal line 2, and the horizontal line 2 is located at the lower right of the horizontal line 1; The robot comprises an AGV trolley, a 6-degree-of-freedom mechanical arm, an end effector and a depth camera, the 6-degree-of-freedom mechanical arm comprises a 6-degree-of-freedom mechanical arm base and six rotating joints, the six rotating joints are sequentially rotating joint one, rotating joint two, rotating joint three, rotating joint four, rotating joint five and rotating joint six from the mechanical arm base, and the six rotating joints are sequentially rotated around a first joint axis, a second joint axis, a third joint axis, a fourth joint axis, a fifth joint axis and a sixth joint axis, wherein the fourth joint axis and the fifth joint axis are perpendicular to each other, and the fifth joint axis and the sixth joint axis are perpendicular to each other; the end effector and the depth camera are both installed on the end of the 6-degree-of-freedom mechanical arm, the optical axis of the depth camera is parallel to the sixth joint axis, and the depth camera moves with the 6-degree-of-freedom mechanical arm; The identification and positioning method comprises the following steps: Step 1, defining a pixel coordinate system: the pixel coordinate system is a straight line coordinate system established with the upper left corner of the image as the origin and with pixels as the unit, which comprises mutually perpendicular pixel U axis and pixel V axis, the horizontal coordinate of the pixel U axis is the column number of the pixel in the image, and the vertical coordinate of the pixel V axis is the row number of the pixel in the image; Step 2, the robot moves to the front of the button area of the electrical cabinet, drives the 6-degree-of-freedom mechanical arm, and keeps the optical axis of the depth camera perpendicular to the front of the electrical cabinet and the plane of the depth camera parallel to the plane of the electrical cabinet, then uses the depth camera to shoot the button area of the electrical cabinet, records the image obtained by shooting as image M, image M is a RGB three-channel image with a width of 1280 pixels, a height of 720 pixels and a pixel depth of 24 bits; record the pixel point D0 at the upper left corner of the image M, the pixel coordinate of the pixel point D0 in the pixel coordinate system of the image M is (0, 0); The image M is intercepted so that there is only one complete button in the intercepted image, the button in the image M0 is recorded as the target button, and the intercepted image is recorded as image M0, the pixel point D1 at the upper left corner of the image M0 is recorded as the pixel point D1, and the pixel coordinate of the pixel point D1 in the pixel coordinate system of the image M is (u1, v1); The image M0 is subjected to RGB three-channel segmentation to obtain image L0 representing R channel, image L1 representing G channel and image L2 representing B channel respectively; Step 3, make the button color of the target button red, and subtract the image L2 from the image L0 to obtain an image M1, which has the same pixel coordinate system and the same pixel coordinate origin as the image M0; Step 4, mean filter is performed on the image M1 to obtain a more smooth image M2 which has the same pixel coordinate system and the same pixel coordinate origin as the image M1; it is assumed that the image M2 includes H pixel points, and any one pixel point in the image M2 is denoted as pixel point D1 η , η = 1, 2, …H, the pixel coordinate of the pixel point D1 η in the pixel coordinate system of the image M2 is (u1 η , v1 η ), and the gray value of the pixel point D1 η is I1 η ; The image M2 is subjected to a binarization process. Specifically, first, a first constant I min = 0, a second constant I max = 255, and a binarization threshold T are given, and then each pixel value I1 η in the image M2 is subjected to the following assignment: if I1 η <T, let I1 η = I min ; If I1 η ≥ T, let I1 η = I max ; Let the re-assigned image M2 be an image M3; Step 5, assuming there are n boundary disjoint pixel regions in the image M3, contour extraction is performed on any one of the regions, and the contour is recorded as contour O k k = 1, 2,... n, n is a positive integer; For each O k The minimum circumscribed rectangle is made and recorded as the minimum circumscribed rectangle R of the region k The minimum circumscribed rectangle is made and recorded as the minimum circumscribed rectangle R of the region k The minimum circumscribed rectangle is made and recorded as the minimum circumscribed rectangle R of the region k The minimum circumscribed rectangle is made and recorded as the minimum circumscribed rectangle R of the region n} Step 6, given the aspect ratio a, and all the minimum bounding rectangles R in the set R that satisfy the aspect ratio 0.8≤a≤1.2 k are extracted, and a set of minimum bounding rectangles that satisfy the aspect ratio condition is formed, and is recorded as set R', R' = {R1, R2,..., R m}, m is a positive integer, m≤n; find the minimum bounding rectangle with the largest area in set R', and record it as the target bounding rectangle R' max ; Step 7, set the target circumscribed rectangle R' max with a width of w pixels and a height of h pixels, and let the top-left pixel point of the target circumscribed rectangle R' max be pixel point D2, the geometric center of the target circumscribed rectangle R' max be pixel point P, then the pixel coordinates of pixel point D2 in the pixel coordinate system of image M3 are (u2, v2), and the pixel coordinates of pixel point P in the pixel coordinate system of image M3 are Step 8, cut a new image Mc from the image M1, which has the same pixel coordinate system and the same pixel coordinate origin as the image M0, and has a starting point of the pixel point D2, a width of w pixels and a height of h pixels; Let image Mc contain Γ pixels, and denote any pixel in image Mc as pixel D2. δ δ = 1, 2, ..., Γ, pixel D2 δ The pixel coordinates in the pixel coordinate system of image Mc are (u2) δ v2 δ ), pixel D2 δ The grayscale value is I2 δ ; Calculate the image mean mean and the image standard deviation stdDev of the image Mc, and the calculation formulas are as follows: Step 9, select the following two cases: The first kind: in step 3, the color of the target button is red, and on this basis, step 9 enters step 10; The second kind: after confirming that the color of the target button is green in step 10, and after performing the specified operation in step 10, return to step 4, and on this basis, step 9 enters step 11; Step 10, give a color judgment threshold STDDEV, and make the following judgments: If stdDev≥STDDEV, confirm that the color of the target button is red, and enter step 11; If stdDev<STDDEV, confirm that the color of the target button is green; then, subtract the image L1 from the image L0 to obtain an image M1', which has the same pixel coordinate system and the same pixel coordinate origin as the image M0; replace the image M1 in step 4 with the image M1' and enter step 4; Step 11, give a bright-dark state judgment threshold MEAN, and make the following judgments: If mean≥MEAN, confirm that the bright-dark state of the target button is bright; If mean<MEAN, confirm that the bright-dark state of the target button is dark; Step 12, the pixel coordinate of the pixel point P in the pixel coordinate system of the image M is (u α , v α ), the value of u α is the value of the horizontal coordinate of the pixel point D0 after being translated along the pixel U axis of the image M by 1 unit, and the value of v α is the value of the vertical coordinate of the pixel point D0 after being translated along the pixel V axis of the image M by 1 unit, and the calculation formula is as follows respectively: The pixel coordinates (u) of pixel point P in the pixel coordinate system of image M α v α This indicates the position of the target button in image M.

Images