Method and device for automatic detection and identification of metering boxes based on rgb-d images

Abstract

The application provides a kind of based on RGB-D image's metering box automatic detection and identification method and device, it is related to image analysis and processing field.Determine the ROI region of metering box RGB-D image after being measured size, according to different position type is classified, respectively detects the real area and real endpoint of target area and calculates real size.The ROI region detection segmentation of different position metering box outer panel and inner panel is called different area detection and endpoint detection scheme in the application, the identification and segmentation result is accurate, and the applicability is high;While being able to characterize target area, the size calculation efficiency of target area is improved, and the type of metering box is further effectively identified, suitable for the measurement and detection of various metering boxes, can effectively monitor and identify the state type of metering box, help real-time monitoring the real-time situation of electrical metering box, improve the management efficiency of metering box, the scheme is efficient and practical, objective and accurate, the whole scheme has high feasibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image analysis and processing, and in particular to a method and device for automatic detection and recognition of metering boxes based on RGB-D images. BACKGROUND

[0002] Today, with the increasing demand for energy and the continuous development of the power industry, the use of electrical metering boxes is becoming more and more widespread. The size of the electrical metering box not only affects its own installation and use, but also affects the performance and safety of the power system. In modern power systems, accurate measurement of the size of the electrical metering box has become a basic task. Not only that, in the power industry, there are certain standards and specifications for the size of various electrical metering boxes, such as national standards and industry standards. Therefore, for technical personnel engaged in power engineering, it is very important to understand and master the methods and techniques of measuring the size of the electrical metering box. Only through accurate measurement can the quality and safety of the installation and use of the electrical metering box be guaranteed, providing a solid guarantee for the stable operation of the power system.

[0003] With the continuous development of automation technology, automatic measurement of the size of the electrical metering box has become a more and more popular choice in the power industry. Automatic measurement of the size of the electrical metering box can greatly improve the measurement efficiency, reduce the measurement error, and make the measurement result more accurate and reliable. At the same time, the automatic measurement equipment can better adapt to different types of electrical metering boxes and complex measurement environments, making the measurement work more convenient and efficient. SUMMARY

[0004] In order to solve the above technical problems, the present application proposes a method and device for automatic detection and recognition of the size of the metering box based on RGB-D images, which combines deep learning technology and feature engineering method to realize target size measurement and detection recognition, and has broad application prospect and market prospect.

[0005] The technical solution adopted by the present application is as follows:

[0006] A method for automatic detection and recognition of the size of the metering box based on RGB-D images, comprising the following steps:

[0007] S01, determining the ROI region ROI to be measured for the collected metering box RGB-D image RGB-D , the metering box RGB-D image includes a depth image and an RGB image;

[0008] S02, classifying the ROI region ROI to be measured for the metering box RGB-D image RGB-D according to different position types, into an initial door upper ROI region and an initial door inner ROI region;

[0009] S03, detecting the real area and real end point of the ROI region on the meter door based on the initial ROI region on the meter door;

[0010] S04, detecting the real area and real end point of the ROI region inside the meter based on the initial ROI region inside the meter;

[0011] S05, detecting the real size of the ROI region on the meter door and the ROI region inside the meter based on the depth data and the real area.

[0012] Further, the method for automatic detection and recognition of the target meter can further include the step S06 of detecting the number of terminal ends of the electric meter inside the meter based on the real area of the ROI region inside the meter, and the step S07 of judging and recognizing the type of the meter based on the detection result. Finally, the size parameter, the number parameter and the type parameter of the target ROI region of the meter are outputted to realize the automatic detection and recognition of the meter.

[0013] On the other hand, the application also provides a device for automatic detection and recognition of a meter based on an RGB-D image. The device is composed of module units corresponding to the steps of any of the above-mentioned methods for automatic detection and recognition of a meter, and is used for automatic detection and recognition of a meter.

[0014] As described above, the application has the following advantages:

[0015] The method and device for automatic detection and recognition of a meter based on an RGB-D image provided by the application have high feasibility. Different area detection and end point detection schemes are called for ROI region detection and segmentation of the external panel and the internal panel of the meter at different positions. The recognition and segmentation results are accurate and have high applicability. The size calculation efficiency of the target region is improved while the target region is characterized. The type of the meter is further effectively recognized. The error between the final detection and recognition result and the reference result of artificial detection is small. The method and device are suitable for measurement and detection of various meters, can effectively monitor and recognize the state type of the meter, are helpful for real-time monitoring of the real-time situation of the electric meter, and improve the management efficiency of the meter. The method and device are efficient, practical, objective and accurate. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the method for automatic detection and recognition of a meter based on an RGB-D image provided by the embodiment of the application.

[0017] Figure 2 is a schematic diagram of the external panel and the internal panel of a meter. (a) is the external panel of the meter, and (b) is the internal panel of the meter.

[0018] Figure 3The inner and outer templates are provided by the embodiment of the present application, (a) is an outer template; (b) is an inner template.

[0019] Figure 4 The foreground region extraction flowchart is provided by the embodiment of the present application.

[0020] Figure 5 The circumscribed rectangle diagram of the initial door ROI region is provided by the embodiment of the present application.

[0021] Figure 6 The schematic diagram of the first operator template detecting an endpoint is provided by the embodiment of the present application.

[0022] Figure 7 The schematic diagram of the door ROI real endpoint detection is provided by the embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the person skilled in the art better understand the technical solutions of the present application, the technical solutions of the present application will be described clearly and completely below in combination with the drawings of the present application, and other similar embodiments obtained by the person skilled in the art without creative labor on the basis of the embodiments in the present application shall all belong to the protection scope of the present application.

[0024] In the following embodiments, the electrical metering box will be selected as the target object, so that the specific scheme is described in detail, and in other embodiments, the target object can also be other types of boxes or other electrical equipment in similar scenes, which are only the analysis objects of the technical solutions of the present application, and the specific application of the present application scheme and solving the corresponding technical problems are used as the criterion.

[0025] Embodiment 1

[0026] As shown in the figure, the present embodiment is a metering box automatic detection and recognition method based on RGB-D image, comprising the following steps, Figure 1

[0027] S01, determining the ROI region to be measured for the collected metering box RGB-D image.

[0028] In the embodiment of the present application, the Intel Real Sense depth camera is used to simultaneously collect the target metering box RGB-D image, which contains metering box RGB image data and metering box depth image data, the metering box RGB image data is represented by RP, and the metering box depth image data is represented by DP.

[0029] ​In this embodiment, RGB images and depth images are acquired for the inside and outside of the metering box at the same time. In order to more accurately measure the target ROI region of the metering box, the ROI region of interest can be detected for the RGB images and the depth images respectively, and the final target ROI region is determined based on the ROI regions obtained from the two types of images respectively. The specific process includes:

[0030] In step S101, the initial ROI region of the metering box RGB image data RP is detected based on the deep learning network mask-RCNN.

[0031] First, the RGB image data RP of the metering box is detected by the deep learning network mask-RCNN to detect all ROI regions of the metering box inside and outside to be measured in size, and the initial ROI region R0 of the RGB image RP of the metering box is obtained.

[0032] In step S102, the initial ROI region R0 is edge corrected based on the edge correction method of straight line fitting and distance screening, and the ROI region to be measured in size corresponding to the RGB image is obtained.

[0033] Due to the influence of factors such as angle, light, and shooting time when the Intel Real Sense depth camera is shooting, the edge of the ROI region R0 obtained by the mask-RCNN rough segmentation may have errors. In order to correct the edge of R0 to reduce the influence on the subsequent size measurement accuracy, the edge correction method based on straight line fitting and distance screening is further proposed in this embodiment to correct the edge of the initial ROI region R0, which specifically includes:

[0034] In S1021, the initial ROI region R0 is first edge detected by using the canny operator to obtain the initial edge of the initial ROI region, denoted as c0.

[0035] In S1022, the four edges of the initial edge c0 are respectively straight line fitted by using the Hough transform, and the distance d i , i = 1, 2, …, n, n is the number of edge points of the current edge.

[0036] In S1023, the median d middle in d i is taken as the starting point, d middle and d i are added in turn, and after each addition, the average value m i of all points added at the current time is calculated. Before the next addition, the difference between the point to be added d c+1 and m c is calculated, and c represents d ithe cth point in the first group of points; if the difference is greater than 5, the (c+1)th point is deleted.

[0037] S1024, the deleted point is replaced by a point on the fitting straight line at its corresponding position.

[0038] The region after edge correction is the ROI region of the measured size corresponding to the RGB image, denoted as R RP .

[0039] Step S103, determining the ROI region of the measured size corresponding to the depth image data DP according to the ROI region of the measured size of the metering tank RGB image data RP.

[0040] Since the RGB image and the depth image collected by the Intel Real Sense depth camera are aligned with each other, the ROI region of the measured size corresponding to the RGB image obtained in the previous step R RP , the corresponding ROI region of the measured size in the depth image can be directly obtained, denoted as R DP .

[0041] Denote the ROI region of the measured size in the final metering tank RGB-D image as ROI RGB-D , the range of the ROI RGB-D region is the same as that of R RP / R DP .

[0042] Step S02, classifying the ROI region of the measured size ROI RGB-D in the metering tank RGB-D image according to different position types.

[0043] As shown in Figure 2 , it is a physical diagram of the electrical metering tank which is the research object of the present application. It can be seen that the size detection target region of the electrical metering tank contains Figure 2 two line frames on the door in (a) and Figure 2 four line frames inside the door in (b) respectively corresponding to the door ROI region and the inside door ROI region. Since the shapes of the frame bodies of the two types of regions are different, different calculation methods need to be used according to their characteristics to detect the end points of the two types of regions respectively, so as to ensure the accuracy of the size measurement. In order to realize the automatic classification of the door ROI region and the inside door ROI region before endpoint detection, the embodiment of the present application proposes a region classification method based on the depth image data DP of the metering tank and the inside-out classification template, so as to divide the region of interest detected in the previous step into two types of initial door and inside door.

[0044] The inside-out classification template proposed in the embodiment of the present application is shown in Figure 3 , wherein Figure 3 (a) is an outer template maskout , Figure 3 (b) is an inner mask in . Wherein the mask out is the same size and shape as the edge size of the to-be-measured dimension ROI region ROI RGB-D of the metering box, and the values of the outermost two layers of elements are set to 1, and the remaining values are set to 0; the inner mask mask in is the same size as the part of the outer mask element that is 0, that is, the shape is consistent, and the element values are all 1. By multiplying the above-mentioned inner and outer masks with the to-be-measured dimension ROI region ROI RGB-D , the internal and edge pixel value conditions of the to-be-measured dimension ROI region ROI RGB-D of the metering box can be reflected.

[0045] In this embodiment, the to-be-measured dimension ROI region ROI RGB-D of the metering box is multiplied by the inner and outer masks respectively and the difference is calculated, so as to obtain the edge and internal distance difference of the to-be-measured dimension ROI region ROI RGB-D of the metering box and the depth camera, and then it can be judged whether there is a depression in the to-be-measured dimension ROI region ROI RGB-D , and the depression can be distinguished according to the depression.

[0046] The calculation formula of the difference is as follows:

[0047] diff = |R RGB-D ·mask out -R RGB-D ·mask in | (1)

[0048] Wherein, diff is the difference obtained by multiplying the to-be-measured dimension ROI region ROI RGB-D with the inner and outer masks respectively.

[0049] The diff is taken as a feature parameter to divide the to-be-measured dimension ROI region ROI RGB-D of the metering box into two categories by unsupervised clustering. Since the ROI region inside the door has a depression feature, it is considered that the category with a larger average value of diff corresponds to N1 ROI regions ROI RGB-D , that is, N1 initial door inside ROI regions inside the metering box, and the category with a smaller average value of diff corresponds to N2 ROI regions ROI RGB-D , that is, N2 initial door on ROI regions outside the metering box.

[0050] Step S03, detecting the real end point of the door on ROI region of the metering box outside panel based on the initial door on ROI region.

[0051] The detection of the ROI area on the door of the external panel of the metering box is an important part of the detection of the power protection system, and the size of the detected ROI area on the door can help technicians to judge the state of the external panel of the metering box, which is of great significance to the protection work of the power system. In order to detect the appearance and structure of the ROI on the door in the external panel of the metering box and further automatically calculate the size of the ROI area on the door, it is necessary to further accurately detect the real area and real endpoint of the ROI area on the door in the external panel of the metering box. The embodiment of the present application proposes a method for detecting the real area and real endpoint of the ROI area on the door of the external panel of the metering box, and the flowchart of the method is as shown in Figure 4 The specific steps include:

[0052] Step S301, extracting the foreground of the initial ROI area on the door of the external panel of the metering box.

[0053] In this embodiment, the RGB image of the external panel of the metering box collected by the Intel Real Sense depth camera is first subjected to foreground extraction processing, which specifically includes:

[0054] S3011, first performing preliminary gray scale conversion on the metering box external panel image LRP ROI (I out ,……). In an embodiment, the resolution of the image LRP ROI (I out ,……) collected by the camera is 1280*720, so it is converted into a gray scale image LRP ROIGray (I out ,……) with the same resolution of 1280*720, so as to better perform the next image processing.

[0055] S3012, since the color characteristics of the metering box itself and the working environment are relatively fixed, a band-pass filter (band-pass interval is (130, 170)) can be used to perform foreground extraction on the gray scale image LRP ROIGray (I out ,……) to remove background information. After removing the background information, the external panel gray scale image LRP ROIGayROI (I out ,……) of the metering box is obtained.

[0056] S3013, finally using a Canny edge detection operator to perform edge positioning and detection coarse segmentation on the external panel gray scale image LRP ROIGrayROI (I out ,……) of the metering box, to obtain the coarse segmented foreground door ROI area LRP ROIGrayROIEdge (I out ,……).

[0057] Step S302: Extract the actual ROI area and actual endpoints on the door of the metering box's external panel.

[0058] This embodiment proposes a precise gate-based ROI edge detection method based on digitized binary images, which can accurately detect image edges. First, the foreground gate-based ROI regions (LRPs) obtained in the previous step are coarsely segmented. ROIGrayROIEdge (I out ...) Perform a top-to-bottom and left-to-right scan to find the boundary starting point. The criterion for determining the boundary starting point is S. i The 8-neighborhood is connected and there exists a pixel with a value of 0. After completely scanning the ROI region on the coarsely segmented foreground gate, all boundary start points S1, S2, ..., S1 in the image are obtained. n .

[0059] After obtaining all boundary starting points, boundary tracing begins. Starting from each boundary point, boundary checks are performed counter-clockwise, based on whether the pixel is 4-neighborhood connected and has a non-zero pixel value. This process is repeated to traverse the neighborhood and find boundary points, with a traversal range of P. ix ±3*P iy ±3, where P ix P represents the x-axis coordinate of the current boundary point. iy Let S be the y-coordinate of the boundary point. When no points around the boundary satisfy this condition, the boundary scan from the starting point is completed, yielding S. i Corresponding boundary i The set of all boundaries constitutes the real area of ​​the ROI region on the door of the metering box's outer panel, and the four corresponding endpoints are the real endpoints of the ROI region on the door.

[0060] Step S04: Based on the initial ROI region inside the door, detect the real area and real endpoint of the ROI region inside the metering box.

[0061] Due to the influence of the camera's shooting angle, the ROI region inside the door of the measuring box, representing the dimension to be measured, exhibits significant indentation, resulting in image distortion and appearing as an irregular parallelogram. The ROI region of the dimension to be measured, segmented in step S01... RGB-DThe door ROI region is rectangular, so in order to find the real end points of the door ROI region inside the metering box to be measured, a new method for detecting the real end points of the door ROI region is proposed, which is suitable for detecting images taken from different angles inside the metering box. For the detection of the size of the ROI region inside the metering box door, the shooting angle does not need to be limited to 90° vertically, and images taken at various angles can be automatically detected, greatly improving the efficiency of automatic detection of the size of the ROI region of the metering box. At the same time, this end point detection method is also suitable for various scenarios of parallelogram real end point detection. The specific steps are as follows:

[0062] S401, based on the segmented initial door ROI region, find its circumscribed rectangle.

[0063] Based on the concept of circumscribed rectangle, two of the four end points of the circumscribed rectangle are the end points of the real parallelogram. As shown in Figure 5 , end points ABCD represent the initial door ROI region inside the metering box segmented by step S01, AB1CD1 is the circumscribed rectangle of the initial door ROI region ABCD, and the pixel point value inside the initial door ROI region ABCD is set to 0 by using the region filling method.

[0064] S402, construct a first operator template, and use the first operator template to detect the product sum of the end points A, B1, C, D1 of the circumscribed rectangle.

[0065] In order to detect the end points of the circumscribed rectangle in this embodiment, two operator templates with a size of 4X4 are constructed, which are the first operator template and the second operator template, as shown in the following table. The first operator template is

[0066] 1 1 0 0 1 1 0 0 0 0 -1 -1 0 0 -1 -1

[0067] The second operator template is

[0068] 0 0 1 1 0 0 1 1 -1 -1 0 0 -1 -1 0 0 .

[0069] The center points of the first operator template and / or the second operator template are placed on the four end points A, B1, C, D1 of the circumscribed rectangle AB1CD1, preferably the center points of the first operator template are placed on A and C, and the center points of the second operator template are placed on B1 and D1. Based on the position distribution characteristics of the circumscribed rectangle AB1CD1 and the initial door ROI region ABCD, the pixel points around the end points A, C and the other two end points B1, D1 of the circumscribed rectangle are different, and the difference can be shown in Figure 6 , so the first operator template can be used to detect the sum of the pixels around the four end points.

[0070] The product sum of each point A, B1, C, D1 and the first template operator of 4X4 is respectively sumA, sumB1, sumC, sumD1, which can be found from Figure 6 It can be found from the formula (1) that |sumA|>|sumB1|>=|sumD1|, |sumC|>|sumB1|>=|sumD1|.

[0071] S403, two maximum values |sumA| and |sumC| in the four product sums of the product of the four end points A, B1, C, D1 of the circumscribed rectangle calculated in step S402 and the first operator template are screened out, that is, two points corresponding to the two maximum values |sumA| and |sumC| are the two real end points of the real parallelogram, and the real end points are recorded as A' and C' at this time.

[0072] S404, the other two real end points B' and D' of the real parallelogram are solved according to the real end points A' and C'.

[0073] First, the real end points A' and C' are connected to obtain a line segment L A′C′ , and the perpendicular line of L A′C′ is obtained to obtain a straight line L. The straight line L is traversed along the direction of L A′C′ , and the traversed straight line numbers are respectively L1, L2, L3,..., L n , L1, L2, L3,..., L n form n intersection points with the line segments ADC and ABC, respectively. The vector of the previous intersection point L i pointing to the next intersection point L i+1 is recorded as as shown in the formula (2). Figure 7

[0074] The vector remains unchanged before encountering the end points B' and D'. When the straight line L traverses the line segment D'C', the direction of the vector x will change, and the position where the change occurs is the position of the real end points B' and D', so that the end points B' and D' are determined. Thus, the ROI area correction inside the door of the metering box is completed, and the real parallelogram A'B'C'D' after correction is the real area of the ROI area inside the door, and the four end points A', B', C', D' of the quadrilateral ROI area are the real end points of the ROI area inside the door.

[0075] S05, the real size of the ROI area on the door and the ROI area inside the door of the metering box is determined based on the depth data and the real end points.

[0076] After the real area and the real end points of the ROI area on the door and the ROI area inside the door of the metering box detected and segmented by the foregoing steps, the real area of the ROI area on the door and the ROI area inside the door is set as the first to-be-measured ROI area, which is recorded as ROI​depth Since the real end points of the first to-be-measured ROI region ROI depth have been determined in the foregoing steps S03-S04, the real end point information can be used to measure the real size of the first to-be-measured ROI region ROI depth .

[0077] Since the Intel Real Sense depth camera projects modulated light in the near-infrared (NIR) spectrum onto the metering box when collecting image information, the near-infrared records the indirect time measurement of light propagation from the camera to the scene and back to the camera. By processing these measurements, the real three-dimensional coordinate points of each pixel point in the depth image are obtained. Through calibration and other methods, the real camera parameters can only be approximated as much as possible, and the camera parameters always have an absolute error. The depth image of the region of interest obtained finally has some depth pixel points that do not belong to the actual to-be-measured ROI region of the metering box within a certain probability. Therefore, before calculating the real size using the real end points of the first to-be-measured ROI region ROI depth , the outlier filtering needs to be performed, and the specific steps are as follows:

[0078] Step S501, convert the first to-be-measured ROI region ROI depth of the metering box (including the real region of the ROI region on the door and the real region of the ROI region inside the door) into a first point cloud ROI cloud . Then, the points that do not belong to the first to-be-measured ROI region will exist in the form of outliers in the first point cloud ROI cloud , and therefore the outliers need to be removed. The outlier removal of the first point cloud ROI cloud mainly has a radius method and a statistical method. In this embodiment, the statistical method is adopted, and specifically, the average distance of Q points in the neighborhood of a point P is calculated, and then the standard deviation is calculated. If the value is greater than a preset value, the point P is determined as an outlier. In a preferred embodiment, Q is 20, and the to-be-measured ROI region of the metering box after removing the outliers is denoted as a second point cloud ROI' cloud .

[0079] Step S502, re-project the second point cloud ROI' cloud onto the two-dimensional plane of the depth image to obtain a second to-be-measured ROI region ROI' depth . Re-perform rectangular frame fitting on the second to-be-measured ROI region ROI' depth to obtain four vertices A'', B'', C'', and D'' of the second to-be-measured ROI region ROI' depth .

[0080] Step S503, based on the real three-dimensional coordinate points measured by the depth camera, the second to-be-measured ROI region ROI' depth The four end points A", B", C", and D" of the region are converted into corresponding real three-dimensional coordinate points, denoted as: A"(x a ,y a ,z a ), B"(x b ,y b ,z b ), C"(x c ,y c, z c ), and D"(x d ,y d ,z d ).

[0081] Step S504, according to the coordinate values of the real three-dimensional coordinate points, the first to-be-measured dimension ROI region ROI depth of the metering box is calculated, and the calculation formula is shown in equations (2) and (3):

[0082]

[0083] where Height i represents the height of the i-th first to-be-measured dimension ROI region ROI depth , Width i represents the width of the i-th first to-be-measured dimension ROI region ROI depth , and i = 1, 2, …, N.

[0084] Step S505, error correction is performed in combination with the calculated values of the multiple real heights and real widths.

[0085] Suppose that N (N = N1 or N2) first to-be-measured dimension ROI regions ROI depth of the metering box are detected in the foregoing steps, for example Figure 2 (a) the metering box outside includes a total of 2 door ROI regions, then N = N2 = 2; and Figure 2 (b) the metering box inside includes a total of 4 door ROI regions, then N = N1 = 4. Since the first to-be-measured dimension ROI region ROI depth of the metering box is N regions with compatible height and width, the real height and real width of the N first to-be-measured dimension ROI regions ROI depth are solved respectively by using the above steps, and then the average value is obtained, which is used to represent the real dimension of the first to-be-measured ROI region of the metering box, so as to reduce the error of dimension measurement, and the calculation formula is shown in equations (4) and (5):

[0086]

[0087]

[0088] Height i and Width i are the real height and real width of the first ROI region of the metering box to be measured respectively, that is, the real size of the ROI region on the metering box door and the ROI region inside the door, thus completing the automatic calculation of the size of the target region on the metering box panel.

[0089] S06, calculating the number of terminal ends of the electric meter inside the metering box based on the real region of the ROI region inside the door.

[0090] Since the terminal ends of the electric meter inside the metering box are located below the ROI region inside the door, the real region of the ROI region inside the door detected in the foregoing step is detected and the number of corresponding terminal ends is identified in the RGB image, and the specific detection method includes:

[0091] S601, extracting the color component of the real region of the ROI region inside the door and segmenting the terminal end region therein.

[0092] Since the color of the terminal end region in the real region of the ROI region inside the door is different from that of the non-terminal end region, the terminal end region is darker than the non-terminal end region, and therefore the RGB image in which the real region of the ROI region inside the door is located is converted to grayscale, a grayscale threshold G is set, and the grayscale of the pixel points in the region with G<100 is reset to 1, so that the terminal end region is segmented into multiple white small region blocks.

[0093] S602, calculating the number of white region blocks and determining the number of terminal ends.

[0094] The white region blocks are counted by using the statistical method of the number of connected domains, and the number of connected domains represents the number of terminal ends, and the number of terminal ends below the electric meter in each ROI region inside the door is recorded as M.

[0095] Step S07, judging and identifying the type of the metering box based on the detection result.

[0096] The foregoing steps can detect and calculate a series of parameters of the target metering box, including the number of ROI regions on the door, the real end point, the real region, the real size, and the number of ROI regions inside the door, the real end point, the real region, the real size, and the number of terminal ends. Based on the parameters obtained above, the type of the metering box can be further judged. In the embodiment of the present application, two types of attributes of the metering box are mainly judged and identified: the phase of the metering box and the metering box position.

[0097] Step S701, according to the number of metering box internal electric meter terminal determines the phase type of metering box;

[0098] According to the technical specification and industry standard of metering box, the existing metering box is usually divided into single-phase metering box and three-phase metering box. The single-phase metering box is connected by one fire line and one zero line. The single electric meter in the single-phase metering box needs 2 terminals for the input and output of the fire line and 2 terminals for the input and output of the zero line. Therefore, the single electric meter in the single-phase metering box has 4 terminals. The three-phase metering box is connected by two fire lines and one zero line. The single electric meter in the three-phase metering box needs 6 terminals for the input and output of the fire line and 1 terminal for the input and output of the zero line. Therefore, the single electric meter in the three-phase metering box has 7 terminals.

[0099] Therefore, the phase of the metering box is determined by the number of terminals in step S06. When M=4, it indicates that the target metering box is single-phase. When M=7, it indicates that the metering box is three-phase.

[0100] Step S702, determine the metering box table position;

[0101] According to the technical specification and industry standard of metering box, the table position of the metering box is determined by the number of electric meters inside the metering box. From the foregoing steps, it can be detected that the number of ROI regions inside the metering box door is N1, which corresponds to the number of electric meters inside the metering box, i.e. the table position parameter of the metering box.

[0102] Embodiment 2

[0103] The embodiment is a device for automatic detection and identification of metering boxes based on RGB-D images. The device is composed of module units corresponding to the method steps of automatic detection and identification in any of the preceding embodiments, for automatic detection and identification of metering boxes.

[0104] The present application proposes a method and device for automatic detection and identification of metering boxes based on RGB-D images for the target regions of the external panel and internal panel of the electrical metering box. The whole scheme has high feasibility. Different region detection and endpoint detection schemes are called for ROI region detection and segmentation of the external panel and internal panel of the metering box in different positions. The identification and segmentation results are accurate and highly applicable. The size calculation efficiency of the target region is improved while the target region is characterized. The type of the metering box is further effectively identified. The error between the final detection and identification results and the reference results of artificial detection is small. The scheme is applicable to the measurement and detection of various metering boxes, can effectively monitor and identify the state type of the metering box, helps to monitor the real-time situation of the electrical metering box in real time, improves the management efficiency of the metering box, and is efficient, practical, objective and accurate.

[0105] All of the features disclosed in this specification, and all of the steps of any methods or processes described herein, can be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive.

[0106] The application is not restricted to the details of the foregoing embodiment. The application extends to any novel one, or any novel combination, of the features disclosed in this specification, and to any novel one, or any novel combination, of the steps of any of the methods or processes disclosed.

Claims

1. A method for meter automatic detection and recognition based on RGB-D images, characterized in that, The method comprises the following steps: Step S01, determining a to-be-measured size ROI region ROI from the collected metering box RGB-D image RGB-D , the metering box RGB-D image comprising a depth image and an RGB image; Step S02, measuring the size of the ROI region ROI of the metering box RGB-D image RGB-D According to different position types, it is divided into initial door-on ROI region and initial door-in ROI region; Step S03, detecting the real area and real endpoints of the ROI area on the meter door based on the initial ROI area on the meter door; Step S04, detecting the real area and real endpoints of the ROI area in the meter door based on the initial ROI area in the meter door; Step S05, detecting the real size of the ROI area on the meter door and the ROI area in the meter door based on the depth data and the real area; The step S03 comprises: Step S301, extracting the foreground of the initial ROI area on the meter door outside the meter by using the gray information and the edge detection operator; Step S302, extracting the real area and real endpoints of the ROI on the meter door outside the meter based on the foreground; The step S04 specifically comprises: Step S401, finding the circumscribed rectangle corresponding to the segmented initial ROI area in the meter door; Step S402, constructing a first operator template and a second operator template, and using the first operator template and the second operator template to detect the product sum of the four endpoints of the circumscribed rectangle respectively; Step S403, screening out two points corresponding to the two maximum values in the four product sums, which are the two real endpoints of the ROI area in the meter door; Step S404, solving the other two real endpoints according to the two determined real endpoints, and obtaining the real area and real endpoints of the ROI area in the meter door; The step S05 specifically comprises: Step S501, converting the real area as a first to-be-measured ROI area into a first point cloud, and removing outliers in the first point cloud to obtain a second point cloud after removing outliers; Step S502, projecting the second point cloud onto a two-dimensional plane of the depth image to obtain a second to-be-measured ROI area, and fitting a rectangular frame to the second to-be-measured ROI area to obtain four vertices A'', B'', C'', and D'' of the second to-be-measured ROI area; Step S503, based on the real three-dimensional coordinate points measured by the depth camera, the four end points A", B", C", and D" of the second ROI region to be measured are converted into real three-dimensional coordinate points, which are denoted as: A"(x a , y a , z a ), B"(x b , y b , z b ), C"(x c , y c , z c ), and D"(x d , y d , z d ). Step S504, calculating the first to-be-measured dimension ROI region ROI of the metering box according to the coordinate value of the real three-dimensional coordinate point depth Side length; Step S505, error correcting the calculated values of the real height and the real width of the ROI area of the same position type to obtain the real size of the ROI area on the meter door and / or the ROI area in the meter door of the target meter.

2. The method for automatic detection and recognition of metering boxes based on RGB-D images according to claim 1, characterized in that, Further comprising: step S06, detecting the number of meter terminal ends inside the meter based on the real area of the ROI area in the meter door; Step S07, judging and identifying the type of the meter based on the detection result.

3. The method for meter automatic detection and recognition based on RGB-D image according to claim 1, characterized in that, The step S01 comprises: Step S101, detecting the initial ROI area of the meter RGB image based on a deep learning network; Step S102, performing edge correction on the initial ROI area based on a straight line fitting and distance screening edge correction method to obtain a to-be-measured size ROI area corresponding to the RGB image; Step S103, determining the to-be-measured size ROI region of the depth image according to the to-be-measured size ROI region of the metering tank RGB image RGB-D .

4. The method for meter automatic detection and recognition based on RGB-D image according to claim 3, characterized in that, The step S02 comprises: Respectively construct inner template and outer template, and measure the size of the ROI region ROI RGB-D Dot product with inner template and outer template respectively, and calculate the ROI region ROI to be measured RGB-D The difference after dot product with inner and outer templates, according to the difference to judge the ROI region ROI to be measured RGB-D Whether there is a recess inside, and according to the recess to measure the size of the ROI region ROI RGB-D Divided into two categories: initial door on ROI region and initial door inside ROI region.

5. The method for meter automatic detection and recognition based on RGB-D images according to claim 1, characterized in that, The step S302 of extracting the real area and real endpoints of the ROI on the meter door outside the meter based on the foreground specifically comprises: Scan the foreground region from top to bottom and from left to right to find the boundary starting point, and the determination condition of the boundary starting point is S i 8-order neighborhood of S is connected and there is a pixel point with a pixel value of 0, to obtain all the boundary starting points S1, S2, …, S n n contained in the image; The boundary tracking is started from the boundary point start point, and the boundary judgment is performed, the boundary point is searched in the neighborhood of the boundary start point, when there is no point around the boundary start point satisfying the judgment condition, the boundary scanning of the boundary start point is completed, and the boundary start point S is obtained i The corresponding boundary Border i The set of boundaries corresponding to all boundary start points constitutes the real area of the ROI region on the door outside the metering box, and the four end points corresponding to the real area are the real end points of the ROI region on the door.

6. The method for meter automatic detection and recognition based on RGB-D images according to claim 1, wherein, The step S404 specifically comprises: Connect the two real endpoints A', C' that have been determined to obtain a line segment L A′C′ , draw a perpendicular line L of L A′C′ to obtain a straight line L, and traverse the straight line L in the direction of L A′C′ . The traversed straight line numbers are {L1, L2, L3,..., L n} and {L1, L2, L3,..., L n} respectively. The n intersections between the initial in-door ROI region and {L1, L2, L3,..., L i} and {L1, L2, L3,..., L i+1} respectively are denoted as L The position where the direction of the vector L changes during the traversal of the straight line L is the position of the other two real endpoints B' and D'. The quadrilateral A'B'C'D' is the real region of the in-door ROI region, and the endpoints A', B', C', and D' are the real endpoints of the in-door ROI region.

7. An apparatus for automatic detection and recognition of meter boxes based on RGB-D images, characterized in that, The device is composed of module units corresponding to the method steps of the automatic detection and identification of the meter according to any one of claims 1-6, and is used for automatically detecting and identifying the size of the meter.

Images