Power monitoring invisible automatic checking method based on RPA and image recognition

Abstract

The invention relates to the technical field of power system emergency monitoring video picture abnormity inspection, and discloses a power monitoring invisible automatic checking method based on RPA and image recognition. According to the method, firstly, a robot process automation technology is utilized to click and access a camera picture, and a screenshot is named and stored; creating a gray average value calculation module based on an image recognition technology, configuring a resource path of the gray average value calculation module, and calculating and storing a gray average value; judging whether the picture is abnormal or not according to an image gray average value threshold value, and if yes, executing recording operation; and finally, the cycle is repeated until the expected transformer substation or power transmission line is inspected. Two technologies are combined, unmanned, non-intrusive and full-automatic troubleshooting is achieved, human resources are saved, the work satisfaction degree of workers is improved, the influence of human factors is avoided, the emergency on-duty quality and effect are improved, and powerful support is provided for reasonable personnel allocation and cost saving of enterprises.

Description

Technical Field

[0001] The present invention relates to the field of invisible troubleshooting of emergency professional monitoring video images of power systems, and in particular to a method for automatically troubleshooting invisible power monitoring video images by combining robotic process automation (RPA) technology and image recognition technology. Background Art

[0002] RPA (Robotic Process Automation) uses specific "robot software" to simulate human operations on computers and automatically execute process tasks according to rules.

[0003] Image recognition refers to the technology of using computers to process, analyze and understand images in order to identify targets and objects of various different patterns.

[0004] However, traditional RPA has certain limitations: it can only perform simple tasks based on fixed rules and cannot recognize and deeply process images; the deep integration of RPA and image recognition overcomes the limitations of traditional RPA and accurately enables the upgrading and development of traditional businesses.

[0005] At present, the substations and transmission lines of the power system are in the "centralized control station + unmanned" mode. In this mode, surveillance video is the only way for emergency duty personnel to quickly grasp the actual disaster situation of key links such as various converter stations, substations, transmission lines, etc. in an emergency; however, large areas of black screen or no signal of surveillance video caused by factors such as surveillance camera failure, network failure, disk storage failure, etc. may cause emergency duty personnel to misjudge the current situation of areas that are not visible to the surveillance video, resulting in the risk and safety hazards of missing emergency situations such as floods, geological disasters, and real fires.

[0006] In order to ensure that each surveillance camera can effectively monitor the real-time status of key power system equipment in an emergency state, the existing technology requires emergency duty personnel to regularly conduct manual inspections of all surveillance videos for invisible situations, and to make a list of invisible cameras and urge relevant units to make timely rectifications; however, the power system substations are large in area and large in number, the transmission channels span long distances, and there are many power equipment and facilities. The number of surveillance cameras in municipal power supply units alone is as high as thousands, and it often takes several days to manually check all of them; as a result, a lot of time of emergency duty personnel is wasted on low-end and repetitive video invisible inspection business, which seriously wastes the company's human resources and hinders the advancement of other businesses. Summary of the invention

[0007] In view of the above deficiencies in the prior art, the purpose of the present invention is to provide a method for automatically troubleshooting invisible power monitoring video images based on RPA and image recognition, aiming to solve the problem of time-consuming and labor-intensive manual troubleshooting of invisible power system monitoring video images.

[0008] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0009] A method for automatic troubleshooting of invisible power monitoring based on RPA and image recognition, comprising the following steps:

[0010] S1. Use RPA technology to automatically open the monitoring video inspection platform, enter the system account and password, and log in to the power monitoring video platform;

[0011] S2. Using RPA technology, automatically input the name of the place to be searched in the search box of the power monitoring video platform and search, capture all the camera names in the place in the search results and store them as an array; search and access the camera names in the captured array, open the specified video screen and wait for a fixed time, and ensure that the video camera screen can be opened normally within the predetermined waiting time under normal network speed.

[0012] S3. After the waiting time in S2 is over, use RPA technology to take a screenshot of the real-time image of the power monitoring video, name the captured image with the name of the camera, and save it to a specified folder.

[0013] S4. Using image recognition technology, read the surveillance video screenshots saved in S3 and convert them from color images to grayscale images, calculate the grayscale average of the grayscale images and store them.

[0014] S5. Based on the grayscale average value calculated in S4, perform threshold judgment. Since the image is visible because of diffuse reflection of light, the overall image is brighter and the image grayscale average value is higher. The invisible camera image is completely black, and the image grayscale average value is extremely low. There is a clear difference in the grayscale average values ​​between the two. Therefore, the grayscale average value can be used to diagnose the invisible camera. If the image grayscale average value is greater than the threshold, the image is normal, and continue to determine whether all places have been checked. If they have been checked, the program terminates. If not, start executing S2; if the grayscale average value is less than the threshold (black screen or no signal, etc., invisible camera images), it is determined that the surveillance video screen is invisible. After recording the camera name, determine whether all places have been checked. If they have been checked, the program terminates. Otherwise, start executing S2.

[0015] S6. Record the name of the invisible camera detected, use the RPA technology "Copy File" component to copy the image to the invisible image folder, and use the RPA technology "Write to Excel Cell" component to write the invisible file name (camera name) of the screen into the invisible camera list table.

[0016] Further, the S4 includes:

[0017] S41. Use the os.listdir function of the Python os standard library to obtain the image names in the folder where the images captured by the RPA technology are located and store them in an array.

[0018] S42. Read the color image and convert it into a grayscale image. The resulting grayscale image contains the brightness information of the image but removes the color information. Brightness is the only feature of the grayscale image.

[0019] S43. Traverse the grayscale value of each pixel of the grayscale image, calculate the sum of the grayscale values ​​of all pixels of the grayscale image, and find the grayscale average value of all pixels of the image.

[0020] Furthermore, in S41, since the image reading function cv2.imread of the OpenCV library may encounter encoding problems when reading Chinese paths, or even fail to read images, the Numpy library is used to read the image file using the np.fromfile function and convert it into a uint8 (unsigned 8-bit integer) data type, and then the cv2.imdecode function of the OpenCV library is used to parse the path, thereby correctly reading the image file.

[0021] Furthermore, in S42, the method for calculating the conversion of a color image to a grayscale image includes: using the cv2.cvtColor function of the python OpenCV library to convert the color image into a grayscale image; a grayscale image is a single-channel image, and the value of each pixel represents the brightness value of the pixel, but does not contain color information; the computer will process the grayscale into 256 grayscale levels, represented by a numerical interval [0, 255]. The grayscale image is generally stored as a matrix representation in which the values ​​of each row and column are between [0, 255]. The calculation formula for the grayscale value of each pixel is:

[0022] Gray=0.299×R+0.578+0114×B;

[0023] Among them, Gray represents the calculated gray value of the pixel, R represents the red component value of the pixel in the color image, and B represents the blue component value of the pixel in the color image.

[0024] Further, in the S43, the method for calculating the grayscale average value of the grayscale image includes: for OpenCVFor Python, the grayscale image is a two-dimensional array or matrix, and the np.mean function of the python Numpy library is used to calculate the grayscale average value of the grayscale image pixel matrix; essentially, the grayscale value of all pixels in the grayscale image is accumulated and divided by the number of pixels, that is, the grayscale average value is expressed as;

[0025] Average = ∑ (all pixels) / (m*n);

[0026] In the above formula, Average represents the grayscale average value of the grayscale image, m is the number of pixels in the length direction of the grayscale image, that is, the number of rows of the image, and n represents the number of pixels in the width direction of the grayscale image, that is, the number of columns of the image; ∑(all pixels) means to sum the grayscale values ​​of all pixels in the grayscale image, traverse each pixel in the grayscale image, add up their grayscale values, and get the sum of the grayscale values ​​of all pixels.

[0027] Further, the S5 includes:

[0028] S51. Set the grayscale average threshold to between [10-30], and use the RPA technology logic judgment component to judge the calculated grayscale average and the threshold; if the grayscale average of the image is greater than the threshold, it indicates that the source image is bright and the picture is visible; if the grayscale average of the image is less than the threshold, it indicates that the source image is dark, and the video camera screen is black or has no signal and is not visible.

[0029] In summary, due to the adoption of the above technical solution, the beneficial technical effects of the invention are:

[0030] The present invention innovatively combines robotic process automation (RPA) and image recognition technology to realize the automatic inspection of invisible services in the power system monitoring video screen. First, RPA technology is used to obtain the real-time screen image of the video camera to be checked in an automated, non-invasive (legal) manner. Then, image recognition technology is used to perform grayscale conversion and grayscale average calculation on the image obtained by RPA. Combined with threshold judgment, the invisible screen camera is diagnosed and the name is recorded. The combination of the two realizes the automatic inspection of invisible power system monitoring video screens.

[0031] In summary, the invention realizes the unmanned, non-intrusive and automated inspection of invisible images in the power system monitoring video, saving a lot of human resources for the enterprise; effectively avoids the long-term execution of simple repetitive processes like assembly lines by employees, and improves the job satisfaction of employees; the automated inspection process eliminates the influence of uncontrollable factors caused by human beings (such as random inspections, etc.) due to human participation, and effectively avoids the risk of invisible cameras not being discovered in time and affecting the emergency support capabilities at critical moments; enterprises can reasonably simplify personnel allocation according to the workload saved by employees, saving manpower and economic costs; enterprises can arrange business promotion and upgrading, new business exploration, etc. for emergency duty personnel, and employees will have more energy than before to provide basic support, further promoting the quality and effectiveness of emergency duty. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flow chart of an automatic inspection method for an electric power system monitoring video screen that is not visible;

[0033] Figure 2 is the grayscale image grayscale level and the corresponding grayscale value;

[0034] Figure 3 Principle sample diagram of the relationship between grayscale color and value;

[0035] Figure 4 Sample pictures of power monitoring video;

[0036] Figure 5 Grayscale image of power monitoring video screen and sample image of grayscale average value. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in combination with the embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and are not used to limit the invention.

[0038] RPA technology can operate computer software (such as file managers, browsers, etc.) in a non-invasive manner (without the risk of illegal intrusion), and image recognition technology can analyze and process images. Therefore, this solution uses RPA technology to automatically capture real-time video surveillance images and complete simple automated operations; image recognition technology is used to automatically analyze and judge video surveillance images to find invisible cameras in the surveillance video.

[0039] like Figure 1 As shown, a method for automatic troubleshooting of invisible power monitoring based on RPA and image recognition includes the following steps:

[0040] S1. Use RPA technology to automatically open the monitoring video platform, enter the system account and password, and log in to the power monitoring video platform.

[0041] S2. Using RPA technology, automatically enter the name of the place to be searched in the search box of the power monitoring video platform and search, capture all the camera names in the place in the search results and store them as an array. Search and access the camera names in the captured array, open the specified video screen and wait for a fixed time, and ensure that the video camera screen can be opened normally within the predetermined waiting time under normal network speed.

[0042] S3. Delay waiting for 10 seconds to ensure that the video is loaded, apply the "Element Screenshot" component to take a screenshot of the video screen, store the image name as the full name of the surveillance camera and save it to the specified file path.

[0043] S4. Create a grayscale average value calculation module based on image recognition technology in PRA, and the resource path of the grayscale average value calculation module is configured as the path used to save screenshots in S3. The average value calculation module is used to process the image captured in the previous step, calculate its grayscale average value and store it.

[0044] S5. Based on the grayscale average value calculated in S4, a threshold judgment is performed. If the image grayscale average value is greater than the threshold, it is determined whether all places have been checked. If so, the program is terminated; otherwise, S2 is started if the check is not completed. If the grayscale average value is less than the threshold (black screen or no signal, etc., invisible camera images), it is determined that the surveillance video image is invisible. After recording the camera name, it is determined whether all places have been checked. If so, the program is terminated; otherwise, S2 is started if the check is not completed.

[0045] S6. At this time, use the RPA technology "Copy File" component to copy the image to the invisible image folder. At the same time, the RPA technology "Write to Excel Cell" component writes the invisible file name (camera name) of the screen to the invisible camera list table.

[0046] S7. Use the RPA technology judgment component to determine whether all places have been checked. If all have been checked, the program terminates. If not, jump to S2 to continue execution until all the places expected to be inspected are inspected and the program terminates.

[0047] The above S1, S2, S3, S5, and S6 are all completed by RPA (Robotic Process Automation), which automatically realizes functions such as logging into the monitoring video platform, reviewing videos, waiting for video loading, taking and saving camera screen screenshots, etc., and uses RPA technology to achieve manual replacement of simple, high-frequency and repetitive business, thereby liberating labor.

[0048] The S2 includes:

[0049] S21. Capturing the names of all cameras in a designated location includes: reading the pre-set string array of the desired patrol locations, applying the "For count loop component", and carrying out the first set of loops with the element index as the independent variable. The loop content includes: applying the "click" component, configuring the designated element to the "search scene" mode, and automatically performing a single-click operation. Applying the "input text" component, configuring the target element to the "search box" coordinate, automatically inputting the substation or transmission line name corresponding to the current array index and clicking the search icon, opening the video camera list of the current substation, applying the "anchor search element" component, setting the substation name as the anchor coordinate, obtaining the names of all surveillance cameras at the relative position and storing them as string variables.

[0050] S22. Automatic review of power monitoring videos includes: applying the "For count loop component" inside the first set of loops, using the subscript of the camera name array within the jurisdiction of a specific substation or transmission line as the independent variable, and starting the second set of loops until all monitoring cameras under the menu of the substation or transmission line have been visited. The loop includes: applying the "click" component, anchoring the search type to "search device", switching the search type, and finding specific camera information; applying the "input text" component, anchoring the target element to the "search box", and entering the camera name corresponding to the current array subscript. Apply the "click" component, anchor the absolute coordinates of the area where the search results appear, perform the click operation after a delay, and open the specific camera screen.

[0051] The S4 includes:

[0052] S41. Use the os.listdir function of the Python os standard library to obtain the file name in the folder where the image captured by the RPA technology is located and store it in an array; because the image reading function cv2.imread of the OpenCV library may encounter encoding problems when reading Chinese paths, or even fail to read images, the Numpy library is used to use the np.fromfile function to read the image file and convert it to the uint8 (unsigned 8-bit integer) data type, and then the cv2.imdecode function of the OpenCV library is used to parse the path, so as to correctly read the image file;

[0053] S42. Read the color image and convert it into a grayscale image. The final grayscale image contains the brightness information of the image but removes the color information. Brightness is the only feature of the grayscale image.

[0054] like Figure 2As shown, in S42, the method for converting a color image into a grayscale image includes: using the cv2.cvtColor function of the python OpenCV library to convert the color image into a grayscale image; a grayscale image is a single-channel image, and the value of each pixel represents the brightness value of the pixel, but does not contain color information; the computer will process the grayscale into 256 grayscale levels, represented by a value range of [0,255], and the detailed effect is as follows Figure 2 As shown in the figure; the value "0" represents pure black, the value "255" represents pure black, and the remaining values ​​represent different levels of gray from pure white to pure black;

[0055] Grayscale images are generally stored as a matrix representation with values ​​in each row and column between [0, 255]; the calculation formula for the grayscale value of each pixel is:

[0056] Gray=0.299×R+0.578+0114×B;

[0057] Among them, Gray represents the calculated gray value of the pixel, R represents the red component value of the pixel in the color image, and B represents the blue component value of the pixel in the color image.

[0058] S43. Traverse the grayscale value of each pixel of the grayscale image, calculate the sum of the grayscale values ​​of all pixels of the grayscale image, and find the grayscale average value of all pixels of the image.

[0059] like Figure 3 As shown, in S43, the method for calculating the grayscale average includes: using the np.mean function of the python Numpy library to calculate the grayscale average of the grayscale image pixel matrix. In essence, it is obtained by accumulating the grayscale values ​​of all pixels in the grayscale image and dividing it by the number of pixels. Figure 5 The data in the table is a demonstration of the grayscale average value;

[0060] Average = Σ(all pixels) / (m*n);

[0061] In the above formula, Average represents the grayscale average value of the grayscale image, m is the number of pixels in the length direction of the grayscale image, that is, the number of rows of the image, and n represents the number of pixels in the width direction of the grayscale image, that is, the number of columns of the image; ∑(all pixels) means to sum the grayscale values ​​of all pixels in the grayscale image, traverse each pixel in the grayscale image, add up their grayscale values, and get the sum of the grayscale values ​​of all pixels.

[0062] like Figure 4 and Figure 5 As shown, the S5 includes:

[0063] After image recognition technology converts a color image into a grayscale image, each pixel on the image is a value between 0 and 255. The closer it is to 0, the darker the pixel is, and the closer it is to 255, the whiter the pixel is. The principle demonstration effect is as follows: Figure 3 As shown; the higher the grayscale average value, the brighter the image, and if the image is invisible (black screen or no signal), the grayscale average value is closer to 0.

[0064] Through case studies, it is concluded that the screen is not visible. The video camera screen is mostly displayed as a black screen or no signal, or the picture quality is black and blurry, making it difficult to perform video surveillance tasks. In this case, the average grayscale value is low, generally below 10, and can be adjusted appropriately, such as Figure 4 and Figure 5 shown.

[0065] Therefore, the grayscale average value can be used for judgment. Set the threshold value to be less than or equal to 10, and use the RPA technology logic judgment component to judge the calculated grayscale average value and the threshold value; if the grayscale average value of the image is greater than the threshold, it means that the source image is bright and the picture is visible; if the grayscale average value of the image is less than the threshold, it means that the source image is dark, and the video camera screen is black or there is no signal and it is not visible.

[0066] The above description is a preferred embodiment of the invention and is not intended to limit the invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the invention should be included in the protection scope of the invention.

Claims

1. A method for automatic troubleshooting of invisible power monitoring based on RPA and image recognition, characterized in that: The following steps are involved: S1. Use RPA technology to automatically open the monitoring video platform, enter the system account and password, and log in to the power monitoring video platform. S2. Using RPA technology, automatically input the name of the place to be searched in the search box of the power monitoring video platform and search, and capture the names of all cameras in the place in the search results and store them as an array; Search and access the camera names in the captured array, open the specified video screen and wait for a fixed time, to ensure that the video camera screen can be opened normally within the predetermined waiting time under normal network speed; S3. After the waiting time in S2 is over, use the RPA technology to take a screenshot of the real-time screen of the power monitoring video, name the captured image with the name of the camera, and save it to a specified folder; S4. Using image recognition technology, read the surveillance video screenshot saved in S3 and convert it from a color image to a grayscale image, calculate the grayscale average of the grayscale image and store it; S5. Based on the grayscale average value calculated in S4, a threshold judgment is performed. If the grayscale average value of the image is greater than the threshold value, the image is diagnosed as normal, and the judgment is continued as to whether all places have been checked. If they have been checked, the program is terminated. If they have not been checked, the program returns to S2 for execution. If the grayscale average value is less than the threshold value, the surveillance video image is diagnosed as invisible, and the judgment is continued as to whether all places have been checked. If they have been checked, the program is terminated. If they have not been checked, the program returns to S2 for execution. The program is terminated after the inspection of all the places expected to be inspected is completed. S6. Record the name of the invisible camera detected and copy the image to the invisible image folder.

2. According to claim 1, a method for automatic troubleshooting of power monitoring based on RPA and image recognition, characterized in that: The S4 includes: S41. Use the os.listdir function of the Python os standard library to obtain the image names in the folder where the images captured by the RPA technology are located and store them in an array. S42. Read the color image and convert it into a grayscale image. The resulting grayscale image contains the brightness information of the image but removes the color information. Brightness is the only feature of the grayscale image. S43. Traverse the grayscale value of each pixel of the grayscale image, calculate the sum of the grayscale values ​​of all pixels of the grayscale image, and find the grayscale average value of all pixels of the image.

3. According to claim 1, a method for automatic troubleshooting of power monitoring based on RPA and image recognition, characterized in that: In S41, since the image reading function cv2.imread of the OpenCV library may encounter encoding problems when reading Chinese paths, or even fail to read images, the Numpy library is used to read the image file using the np.fromfile function and convert it into a uint8 (unsigned 8-bit integer) data type, and then the cv2.imdecode function of the OpenCV library is used to parse the path, thereby correctly reading the image file.

4. According to claim 1, a method for automatic troubleshooting of power monitoring based on RPA and image recognition, characterized in that: In the S42, the method for calculating the conversion of the color image to the grayscale image includes: using the cv2.cvtColor function of the python OpenCV library to convert the color image into a grayscale image; the grayscale image is a single-channel image, and the value of each pixel represents the brightness value of the pixel, but does not contain color information; the computer will process the grayscale into 256 grayscale levels, represented by a numerical interval [0, 255]. The grayscale image is generally stored as a matrix representation in which the values ​​of each row and column are between [0, 255]. The calculation formula for the grayscale value of each pixel is: Gray=0.299×R+0.578+0114×B; Among them, Gray represents the calculated gray value of the pixel, R represents the red component value of the pixel in the color image, and B represents the blue component value of the pixel in the color image.

5. According to claim 1, a method for automatic troubleshooting of power monitoring based on RPA and image recognition, characterized in that: In the S43, the method for calculating the grayscale average value of the grayscale image includes: for OpenCV For Python, the grayscale image is a two-dimensional array or matrix, and the np.mean function of the python Numpy library is used to calculate the grayscale average value of the grayscale image pixel matrix; essentially, the grayscale value of all pixels in the grayscale image is accumulated and divided by the number of pixels, that is, the grayscale average value is expressed as; Average = ∑ (all pixels) / (m*n); In the above formula, Average represents the grayscale average value of the grayscale image, m is the number of pixels in the length direction of the grayscale image, that is, the number of rows of the image, and n represents the number of pixels in the width direction of the grayscale image, that is, the number of columns of the image; ∑(all pixels) means to sum the grayscale values ​​of all pixels in the grayscale image, traverse each pixel in the grayscale image, add up their grayscale values, and get the sum of the grayscale values ​​of all pixels.

6. According to claim 1, a method for automatic troubleshooting of power monitoring based on RPA and image recognition, characterized in that: The threshold of the grayscale average value in S5 is between [10-30]. In the process of judging the calculated grayscale average value and the threshold using the RPA technology logic judgment component, if the grayscale average value of the image is greater than the threshold, it indicates that the source image is bright and the picture is visible; if the grayscale average value of the image is less than the threshold, it indicates that the source image is dark, and the video camera screen is black or has no signal and is not visible.

7. According to claim 1, a method for automatic troubleshooting of power monitoring based on RPA and image recognition, characterized in that: The method for recording the name of the invisible camera in the screen by using the PRA technology in S6 includes: Use the RPA technology "Copy File" component to copy the image to the invisible image folder. At the same time, use the RPA technology "Write to Excel Cell" component to write the invisible file name of the screen into the invisible camera list table.

Images