Photovoltaic panel posture recognition method, device, and installation robot

Abstract

The present invention provides a photovoltaic panel pose recognition method, device, and installation robot. The method comprises: controlling a depth camera to move and illuminate a photovoltaic panel to be clamped, collecting RGB image information and depth point cloud information; segmenting the RGB image information to obtain a grayscale image of the photovoltaic panel to be clamped; performing edge contour detection and polygon fitting on the grayscale image to obtain a fitted quadrilateral image of the photovoltaic panel to be clamped and the coordinates of the quadrilateral corner points; performing a perspective transformation on the fitted quadrilateral corner coordinates of the photovoltaic panel to be clamped and the coordinates of the four corner points of a preset photovoltaic panel image template to obtain a perspective matrix of the photovoltaic panel to be clamped; and calculating the six-degree-of-freedom pose of the photovoltaic panel to be clamped using the depth point cloud information of the photovoltaic panel to be clamped, the fitted quadrilateral image, the perspective matrix of the photovoltaic panel to be clamped, and the depth point cloud information of the preset photovoltaic panel image template. This solution combines image information and depth point cloud information to accurately identify the six-degree-of-freedom pose of the photovoltaic panel.

Description

Technical Field

[0001] The present invention relates to the field of solar photovoltaic power generation, and in particular to a method, device, and installation robot for recognizing the posture of a photovoltaic panel. Background Art

[0002] When the photovoltaic panel automatic installation robot automatically grasps the photovoltaic panel, it needs to move its gripper in a direction perpendicular to the plane of the photovoltaic panel with its claws parallel to the plane of the photovoltaic panel. However, the placement and posture of the photovoltaic panel are not fixed, and the sunlight environment on site is also changing. Accurately identifying the posture of the photovoltaic panel and thus accurately grasping the photovoltaic panel has become a technical challenge. Summary of the Invention

[0003] The purpose of the present invention is to provide a photovoltaic panel posture recognition method, device, and installation robot to accurately obtain the six-degree-of-freedom posture of the photovoltaic panel in space.

[0004] Specifically, the technical solutions provided by the present invention are as follows:

[0005] The present invention provides a photovoltaic panel posture recognition method, comprising:

[0006] Controlling the depth camera to move and illuminate the photovoltaic panel to be clamped, and collecting RGB image information and depth point cloud information in the field of view of the depth camera;

[0007] Segment the RGB image information using deep learning instance segmentation technology to obtain a grayscale image of the photovoltaic panel to be clamped;

[0008] Performing edge contour detection and polygon fitting on the grayscale image to obtain a fitted quadrilateral image and quadrilateral corner point coordinates of the photovoltaic panel to be clamped;

[0009] Performing perspective transformation on the coordinates of the fitted quadrilateral corner points of the photovoltaic panel to be clamped and the coordinates of the four corner points of the preset photovoltaic panel image template to obtain a perspective matrix of the photovoltaic panel to be clamped;

[0010] The six-degree-of-freedom posture of the photovoltaic panel to be clamped is calculated using the depth point cloud information of the photovoltaic panel to be clamped, the fitted quadrilateral image, the perspective matrix of the photovoltaic panel to be clamped, and the depth point cloud information of the preset photovoltaic panel image template.

[0011] In the above embodiment, the photovoltaic panel to be clamped is illuminated by a depth camera to obtain RGB image information and depth point cloud information, and the deep learning instance segmentation technology is used to segment the RGB image information to obtain a grayscale image and remove the influence of the photovoltaic panel placement position and lighting. The grayscale image is subjected to edge contour detection and quadrilateral fitting, and the unstable points on the edge of the photovoltaic panel are discarded to make the recognition result more stable. Finally, the six degrees of freedom posture data of the photovoltaic panel to be clamped are calculated by the singular value decomposition method. The image information and depth point cloud information are combined to output the spatial X, Y, Z position information and the three angle information of pitch angle, yaw angle and roll angle to accurately identify the posture of the photovoltaic panel.

[0012] In some embodiments, performing edge contour detection and quadrilateral fitting on the grayscale image includes:

[0013] Performing edge filtering on the grayscale image of the photovoltaic panel to be clamped, and using the maximum edge contour filtered out as the edge contour of the photovoltaic panel to be clamped;

[0014] Performing polygon fitting on the edge contour and removing the non-smooth error after segmentation to obtain a polygonal image;

[0015] The edges and points of the polygonal image are filtered out, and the adjacent edges of the polygonal image are extended and intersected to obtain the quadrilateral image of the photovoltaic panel to be clamped and the coordinates of the quadrilateral corner points.

[0016] In the above embodiment, the edge contour of the photovoltaic panel in the segmented mark image is obtained by image edge filtering technology; the edge contour of the previous step is fitted by polygon fitting technology to obtain polygons and polygon corner points; the polygon edges and corner points at the edge of the image are filtered out, and the corner points and edges are obtained again by intersecting adjacent lines, and fitted into a quadrilateral, thereby eliminating the unstable points at the edge of the photovoltaic panel, and the identified photovoltaic panel posture is more stable.

[0017] In some embodiments, the method of calculating the six-degree-of-freedom pose of the photovoltaic panel to be clamped by using the depth point cloud information of the photovoltaic panel to be clamped, the fitted quadrilateral image, the perspective matrix of the photovoltaic panel to be clamped, and the depth point cloud information of the preset photovoltaic panel image template comprises the steps of:

[0018] Filter valid point clouds in the depth point cloud information of the photovoltaic panel to be clamped through the perspective matrix of the photovoltaic panel to be clamped, and obtain a valid depth point cloud set of the photovoltaic panel to be clamped.

[0019] In some embodiments, further comprising:

[0020] Obtaining sampling points of the preset photovoltaic panel image template;

[0021] Obtaining image points corresponding to the sampling points in the fitted quadrilateral image using the perspective matrix;

[0022] Searching for point cloud data corresponding to the image point in the depth point cloud information, eliminating invalid point cloud data in the point cloud data, and obtaining valid depth point cloud data of the photovoltaic panel to be clamped;

[0023] A matrix equation is established based on the effective depth point cloud data and the point cloud data of the preset photovoltaic panel image template, and six-degree-of-freedom posture data in the space of the photovoltaic panel to be clamped is obtained through singular value decomposition.

[0024] In some embodiments, the step of eliminating invalid point cloud data from the point cloud data to obtain valid depth point cloud data of the photovoltaic panel to be clamped comprises the following steps:

[0025] Determining whether there are invalid depth points in the depth point cloud data;

[0026] If there are invalid depth points, their serial numbers are recorded, and in the point cloud data of the preset photovoltaic panel image template and the point cloud data of the photovoltaic panel to be clamped, the invalid point cloud data corresponding to the invalid depth points are eliminated to obtain valid point cloud data of the preset photovoltaic panel image template and the point cloud data of the photovoltaic panel to be clamped.

[0027] The present invention also provides a photovoltaic panel posture recognition device, comprising:

[0028] An acquisition module is used to control the depth camera to move and illuminate the photovoltaic panel to be clamped, and to acquire RGB image information and depth point cloud information in the field of view of the depth camera;

[0029] a segmentation module, configured to segment the RGB image information using a deep learning instance segmentation technique to obtain a grayscale image of the photovoltaic panel to be clamped;

[0030] A fitting module is used to perform edge contour detection and polygon fitting on the grayscale image to obtain a fitted quadrilateral image and quadrilateral corner point coordinates of the photovoltaic panel to be clamped;

[0031] An acquisition module is used to perform perspective transformation on the coordinates of the corner points of the quadrilateral fitted by the photovoltaic panel to be clamped and the coordinates of the four corner points of the preset photovoltaic panel image template to obtain the perspective matrix of the photovoltaic panel to be clamped;

[0032] A calculation module is used to calculate the six-degree-of-freedom posture of the photovoltaic panel to be clamped through the depth point cloud information of the photovoltaic panel to be clamped, the fitted quadrilateral image, the perspective matrix of the photovoltaic panel to be clamped and the depth point cloud information of the preset photovoltaic panel image template.

[0033] In some embodiments, the fitting module comprises:

[0034] a filtering and screening unit, configured to perform edge filtering on the grayscale image of the photovoltaic panel to be clamped, and use the maximum edge contour filtered out as the edge contour of the photovoltaic panel to be clamped;

[0035] A fitting and segmentation unit is used to perform polygon fitting on the edge contour and remove the non-smooth error after segmentation to obtain a polygonal image;

[0036] The filtering and fitting unit is used to filter out the edges and points of the polygonal image, and extend and intersect adjacent edges of the polygonal image to obtain the quadrilateral image of the photovoltaic panel to be clamped and the coordinates of the quadrilateral corner points.

[0037] In some embodiments, the computing module comprises:

[0038] The acquisition unit is used to screen the valid point clouds in the depth point cloud information of the photovoltaic panel to be clamped through the perspective matrix of the photovoltaic panel to be clamped, and acquire the valid depth point cloud set of the photovoltaic panel to be clamped.

[0039] In some embodiments, the computing module further includes:

[0040] The acquisition unit is further configured to acquire sampling points of the preset photovoltaic panel image template;

[0041] a sampling unit, configured to obtain image points corresponding to the sampling points in the fitted quadrilateral image using the perspective matrix;

[0042] a search unit, configured to search the depth point cloud information for point cloud data corresponding to the image point, remove invalid point cloud data from the point cloud data, and obtain valid depth point cloud data of the photovoltaic panel to be clamped;

[0043] The decomposition acquisition unit is used to establish a matrix equation based on the effective depth point cloud data and the point cloud data of the preset photovoltaic panel image template, and obtain the six-degree-of-freedom posture data in the space of the photovoltaic panel to be clamped through singular value decomposition.

[0044] The present invention also provides a photovoltaic panel installation robot, comprising a depth camera installed at the end of the robot's robotic arm, a processor, a memory connected to the processor, and program instructions stored on the memory. When the processor executes the program instructions, it implements the photovoltaic panel posture recognition method described in any one of the above items.

[0045] The present invention also provides a storage medium for photovoltaic panel posture recognition, wherein the storage medium stores at least one instruction, which is loaded and executed by a processor to implement the operations performed by the photovoltaic panel posture recognition method as described in any one of the above items.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The photovoltaic panel posture recognition method, device, installation robot, and storage medium provided by the present invention combine image information and depth point cloud information to accurately identify the six-degree-of-freedom posture of the photovoltaic panel;

[0048] 2. This invention uses deep learning to segment RGB image information to obtain grayscale images and remove the influence of photovoltaic panel placement posture and lighting, discarding the unstable point cloud at the edge of the photovoltaic panel, and the identified photovoltaic panel posture is more stable;

[0049] 3. The present invention outputs the calculated six-degree-of-freedom posture data to the photovoltaic panel automatic installation robot, and the photovoltaic panel is automatically grasped by the robotic arm, thereby realizing automatic intelligent installation of the photovoltaic panel installation robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The following will explain the preferred implementation scheme in a clear and understandable manner with reference to the accompanying drawings, and further illustrate the above-mentioned characteristics, technical features, advantages and implementation methods of a photovoltaic panel posture recognition method, device, installation robot and storage medium.

[0051] Figure 1 This is a flow chart of an embodiment of a method for recognizing the posture of a photovoltaic panel according to the present invention;

[0052] Figure 2 is a flow chart of another embodiment of a photovoltaic panel posture recognition method of the present invention;

[0053] Figure 3 It is a structural schematic diagram of an embodiment of a photovoltaic panel posture recognition device of the present invention.

[0054] Description of Figure Numbers:

[0055] Acquisition module 10, segmentation module 20, fitting module 30, acquisition module 40, calculation module 50, filtering and screening unit 31, fitting and segmentation unit 32, filtering and fitting unit 33, acquisition unit 51, sampling unit 52, search unit 53, decomposition and acquisition unit 54. DETAILED DESCRIPTION

[0056] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details.

[0057] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections.

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific embodiments of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings and other embodiments can be obtained based on these drawings without inventive work.

[0059] To simplify the drawings, only portions relevant to the invention are schematically depicted in each figure; they do not represent the actual structure of the product. Furthermore, to simplify the drawings and facilitate understanding, in some figures, only one component with the same structure or function is schematically depicted or labeled. In this document, "one" not only means "only one" but also "more than one."

[0060] In addition, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0061] In one embodiment, referring to the accompanying drawings Figure 1 , the present application provides a photovoltaic panel posture recognition method, comprising:

[0062] S101 controls the depth camera to move and illuminate the photovoltaic panel to be clamped, and collects RGB image information and depth point cloud information in the depth camera's field of view;

[0063] Specifically, the depth camera is installed at the end of the robotic arm to collect RGB color images and depth point cloud information. Through the data obtained by the depth camera, we can accurately know the distance of each point in the image from the camera. In this way, by adding the (x, y) coordinates of the point in the two-dimensional image, we can obtain the three-dimensional spatial coordinates of each point in the image. The entire system only uses the depth camera as a key sensor, which makes system integration easy.

[0064] S102 segments the RGB image information using deep learning instance segmentation technology to obtain a grayscale image of the photovoltaic panel to be clamped;

[0065] Specifically, the placement and orientation of photovoltaic panels are not fixed, and ambient lighting can also vary. Deep learning instance segmentation technology removes the influence of the placement and orientation of photovoltaic panels and lighting, making recognition more stable. Deep learning instance segmentation is used to process the RGB color image information, resulting in a segmented grayscale image of the photovoltaic panel. Each pixel in a grayscale image can be represented by a grayscale value from 0 to 255, specifically, there are 255 intermediate gray values between completely black and completely white.

[0066] S103 performs edge contour detection and polygon fitting on the grayscale image and obtains the fitted quadrilateral image and quadrilateral corner coordinates of the photovoltaic panel to be clamped;

[0067] Specifically, the grayscale image of the photovoltaic panel segmented by deep learning is edge filtered to screen out the largest edge contour, which is considered to be the edge contour of the photovoltaic panel; the edge contour of the previous step is then fitted using polygon fitting technology to obtain polygons and polygon corner points, thereby removing the non-smooth segmentation error; finally, the polygon edges and corner points at the edge of the image are filtered out, and the corner points and edges are re-obtained by intersecting adjacent lines, and fitted into a quadrilateral, thus discarding the unstable points at the edge of the photovoltaic panel. The identified photovoltaic panel posture is more stable, thereby obtaining the fitted quadrilateral image and quadrilateral corner coordinates of the photovoltaic panel to be clamped.

[0068] Filtering can be understood as a filter that traverses an image from top to bottom and left to right, calculating the filter value and the corresponding pixel value, performing numerical calculations based on the filtering purpose, and returning the value to the current pixel. Filtering is used to extract image features and simplify the image information for subsequent image processing. To meet the needs of image processing, filtering is used to eliminate noise introduced during image digitization. Polygon fitting is a common image processing technique that can fit a set of discrete points into a polygon, thereby better describing and analyzing the shape and structure of the image. Polygon fitting is widely used in fields such as computer vision, robotics, and automated control. The basic idea of polygon fitting is to treat a set of discrete points as the vertices of a polygon and then use a specific algorithm to determine the polygon's boundary and vertex positions.

[0069] S104 performs perspective transformation on the coordinates of the fitted quadrilateral corner points of the photovoltaic panel to be clamped and the coordinates of the four corner points of the predefined photovoltaic panel image template to obtain a perspective matrix of the photovoltaic panel to be clamped.

[0070] Specifically, the pixel coordinates of the four corner points are calculated by fitting the four sides of the photovoltaic panel to be clamped, and then the coordinates of the four corner points of the actual photovoltaic panel are calculated in combination with the fixed length and width of the actual photovoltaic panel. Then, the perspective matrix of the photovoltaic panel to be clamped is calculated through two groups of four corresponding corner point coordinates, wherein the preset photovoltaic panel image template is the actual photovoltaic panel, that is, the relevant information of the preset photovoltaic panel image template is known.

[0071] S105 calculates the six-degree-of-freedom posture of the photovoltaic panel to be clamped through the depth point cloud information of the photovoltaic panel to be clamped, the fitted quadrilateral image, the perspective matrix of the photovoltaic panel to be clamped, and the depth point cloud information of the preset photovoltaic image template.

[0072] Specifically, 49 points are uniformly sampled from 7 rows and 7 columns inside the actual photovoltaic panel. The 49 image points in the fitted quadrilateral image of the photovoltaic panel to be clamped are calculated using the perspective matrix. Based on the row and column coordinates corresponding to the 49 image points in the fitted quadrilateral image of the photovoltaic panel to be clamped, the corresponding point cloud is found in the depth point cloud data, thus obtaining the 49 spatial depth point information. Based on the 49 spatial depth points on the fitted photovoltaic panel to be clamped captured by the depth camera, the 49 spatial depth points of the preset photovoltaic image template are also known, as well as the fitted quadrilateral image of the photovoltaic panel to be clamped and the perspective matrix of the photovoltaic panel to be clamped, an equation is jointly established, and the 6-degree-of-freedom position information of the photovoltaic panel to be clamped in space is calculated using the singular value decomposition method.

[0073] In this embodiment, the photovoltaic panel to be clamped is illuminated by a depth camera to obtain RGB image information and depth point cloud information, and the computer segments and marks the photovoltaic panel in the RGB image through deep learning instance segmentation technology; through image edge filtering technology, the edge contour of the photovoltaic panel in the segmented mark image is obtained and quadrilateral fitting is performed, and the unstable points on the edge of the photovoltaic panel are discarded. The identified photovoltaic panel posture is more stable and has stronger adaptability to the scene. Finally, the image information and depth point cloud information are combined, and the six degrees of freedom posture data of the photovoltaic panel to be clamped are calculated through the singular value decomposition method.

[0074] In one embodiment, referring to the accompanying drawings Figure 1 and Figure 2 , the present application provides a photovoltaic panel posture recognition method, comprising:

[0075] S101 controls the depth camera to move and illuminate the photovoltaic panel to be clamped, and collects RGB image information and depth point cloud information in the depth camera's field of view;

[0076] S102 segments the RGB image information using deep learning instance segmentation technology to obtain a grayscale image of the photovoltaic panel to be clamped;

[0077] S201 performs edge filtering on the grayscale image of the photovoltaic panel to be clamped, and uses the maximum edge contour filtered out as the edge contour of the photovoltaic panel to be clamped;

[0078] Specifically, the grayscale image of the photovoltaic panel segmented by deep learning is edge filtered through image filtering technology to screen out the largest edge contour, which is considered to be the edge contour of the photovoltaic panel. This can eliminate the noise mixed in the image and simplify the information carried by the image for subsequent other image processing.

[0079] S202 performs polygon fitting on the edge contour and removes the non-smooth error after segmentation to obtain a polygonal image;

[0080] Specifically, the discrete points that make up the edge contour are regarded as the vertices of the polygon, and the polygons and polygon corners are determined through certain algorithms to better describe and analyze the shape and structure in the image.

[0081] S203 filters out the edges and points of the polygonal image, and extends and intersects adjacent edges of the polygonal image to obtain a quadrilateral image of the photovoltaic panel to be clamped and the coordinates of the quadrilateral corner points.

[0082] Specifically, the photovoltaic panel to be clamped is a quadrilateral model. Due to factors such as the camera's field of view and shooting angle, the resulting polygon may not be a quadrilateral. By setting a threshold, polygon edges and points close to the edge of the image are deleted, and adjacent edges are extended and intersected. The resulting quadrilateral is considered to be the photovoltaic panel's image in the camera, thus obtaining the quadrilateral image and coordinates of the quadrilateral's corner points of the photovoltaic panel to be clamped.

[0083] S105 calculates the six-degree-of-freedom posture of the photovoltaic panel to be clamped through the depth point cloud information of the photovoltaic panel to be clamped, the fitted quadrilateral image, the perspective matrix of the photovoltaic panel to be clamped, and the depth point cloud information of the preset photovoltaic panel image template.

[0084] In this embodiment, deep learning instance segmentation technology is used to segment RGB image information to obtain grayscale images and remove the influence of photovoltaic panel placement position posture and lighting, so that the recognition results are more stable and the system integration is convenient. The key sensor only requires a depth camera; edge contour detection and quadrilateral fitting are performed on the grayscale image to discard unstable points on the edge of the photovoltaic panel, which can also make the posture recognition of the photovoltaic panel stable and improve the accuracy of photovoltaic panel posture recognition.

[0085] In one embodiment, the present application provides a photovoltaic panel pose recognition method. Based on the above embodiment, the six-degree-of-freedom pose of the photovoltaic panel to be clamped is calculated by using the depth point cloud information of the photovoltaic panel to be clamped, the fitted quadrilateral image, the perspective matrix of the photovoltaic panel to be clamped, and the depth point cloud information of the preset photovoltaic panel image template, including:

[0086] Through the perspective matrix of the photovoltaic panel to be clamped, valid point clouds in the depth point cloud information of the photovoltaic panel to be clamped are screened to obtain a valid depth point cloud set of the photovoltaic panel to be clamped.

[0087] Specifically, by performing perspective transformation on the coordinates of the fitted quadrilateral corner points of the photovoltaic panel to be clamped and the coordinates of the four corner points of the preset photovoltaic panel image template, the perspective matrix of the photovoltaic panel to be clamped is obtained, and the valid point cloud in the depth point cloud information of the photovoltaic panel to be clamped is screened through the perspective matrix of the photovoltaic panel to be clamped to obtain the valid depth point cloud set of the photovoltaic panel to be clamped.

[0088] In one embodiment, the present application provides a photovoltaic panel posture recognition method, which, based on the above embodiment, further includes:

[0089] Get the sampling points of the preset photovoltaic image template;

[0090] Specifically, 49 points uniformly arranged in 7 rows and 7 columns inside the preset photovoltaic image template are obtained.

[0091] Obtain the image points corresponding to the sampling points in the fitted quadrilateral image through the perspective matrix;

[0092] Specifically, 49 image points corresponding to 49 sampling points in the fitted quadrilateral image are obtained through the perspective matrix.

[0093] Searching for point cloud data corresponding to the image points in the depth point cloud information, eliminating invalid point cloud data in the point cloud data, and obtaining valid depth point cloud data of the photovoltaic panel to be clamped;

[0094] According to the row and column coordinates corresponding to the 49 image points in the fitted four-sided imaging, the corresponding point cloud is found in the depth point cloud data, and the invalid point cloud data in the point cloud data is eliminated, thereby obtaining the valid depth point cloud data of the photovoltaic panel to be clamped.

[0095] According to the effective depth point cloud data and the point cloud data of the preset photovoltaic panel image template, a matrix equation is established, and the six-degree-of-freedom pose data of the photovoltaic panel to be clamped in space is obtained through singular value decomposition.

[0096] In this embodiment, the perspective transformation matrix of the quadrilateral image and the actual photovoltaic panel to be clamped is calculated by fitting the pixel coordinates of the four corner points of the quadrilateral image and the coordinates of the four corner points of the photovoltaic panel to be clamped, and then the corresponding image point coordinates are calculated according to the sampling point coordinates of the photovoltaic panel to be clamped, and then the corresponding spatial depth point information is searched through the depth point cloud information, and finally the six degrees of freedom posture data of the photovoltaic panel to be clamped are calculated by the singular value decomposition method to accurately identify the posture of the photovoltaic panel.

[0097] In one embodiment, the present application provides a method for identifying a photovoltaic panel posture, comprising:

[0098] Control the depth camera to move and illuminate the photovoltaic panel to be clamped, and collect RGB image information and depth point cloud information in the depth camera's field of view;

[0099] Specifically, the depth camera is installed at the end of the robotic arm to collect RGB color images and depth point cloud information. Through the data obtained by the depth camera, we can accurately know the distance of each point in the image from the camera. In this way, by adding the (x, y) coordinates of the point in the two-dimensional image, we can obtain the three-dimensional spatial coordinates of each point in the image. The entire system only uses the depth camera as a key sensor, which makes system integration easy.

[0100] The RGB image information is segmented using deep learning instance segmentation technology to obtain a grayscale image of the photovoltaic panel to be clamped;

[0101] Specifically, the placement and posture of photovoltaic panels are not fixed, and the ambient light will also change. By adopting deep learning instance segmentation technology, the influence of the placement and posture of photovoltaic panels and lighting is removed, making the recognition results more stable. The RGB color image information is processed through deep learning instance segmentation to obtain the segmented grayscale image of the photovoltaic panel.

[0102] Perform edge filtering on the grayscale image of the photovoltaic panel to be clamped, and use the maximum edge contour filtered out as the edge contour of the photovoltaic panel to be clamped;

[0103] Specifically, the grayscale image of the photovoltaic panel segmented by deep learning is edge filtered through image filtering technology to screen out the largest edge contour, which is considered to be the edge contour of the photovoltaic panel. This can eliminate the noise mixed in the image and simplify the information carried by the image for subsequent other image processing.

[0104] Perform polygon fitting on the edge contour and remove the non-smooth error after segmentation to obtain a polygonal image;

[0105] Specifically, the discrete points that make up the edge contour are regarded as the vertices of the polygon, and the polygons and polygon corners are determined through certain algorithms to better describe and analyze the shape and structure in the image.

[0106] Filter out the edges and points of the polygonal image, extend and intersect the adjacent edges of the polygonal image, and fit it into a quadrilateral image;

[0107] Specifically, photovoltaic panels are quadrilateral models. Due to factors such as the camera's field of view and shooting angle, the resulting polygons may not be quadrilaterals. By setting a threshold, polygon edges and points close to the edge of the image are deleted, and adjacent edges are extended and intersected. The resulting quadrilateral is considered to be the photovoltaic panel's image in the camera.

[0108] Calculate the pixel coordinates of the four corner points of the quadrilateral image, and calculate the coordinates of the four corner points of the actual photovoltaic panel based on the length and width of the photovoltaic panel;

[0109] The perspective matrix of the photovoltaic panel to be clamped is calculated using two sets of four corner point coordinates;

[0110] Specifically, the pixel coordinates of the four corner points of the quadrilateral image are calculated based on the fitted quadrilateral image, and then the coordinates of the four corner points of the actual photovoltaic panel are calculated based on the fixed length and width dimensions of the photovoltaic panel. The perspective matrix is calculated by comparing the two sets of coordinates.

[0111] Obtain the sampling points of the preset photovoltaic panel image template, and use the perspective matrix to obtain the image points corresponding to the sampling points in the fitted quadrilateral image;

[0112] Specifically, 49 points of 7 rows and 7 columns uniformly arranged inside the preset photovoltaic image template are obtained, and 49 image points corresponding to the 49 sampling points in the fitted quadrilateral image are obtained through the perspective matrix.

[0113] Searching for point cloud data corresponding to the image point in the depth point cloud information, eliminating invalid point cloud data in the point cloud data, and obtaining valid depth point cloud data of the photovoltaic panel to be clamped;

[0114] A matrix equation is established based on the effective depth point cloud data and the point cloud data of the preset photovoltaic panel image template, and six-degree-of-freedom posture data in the space of the photovoltaic panel to be clamped is obtained through singular value decomposition.

[0115] In this embodiment, the depth camera is first moved to illuminate the photovoltaic panel, and then the depth camera sends the RGB image information and point cloud depth information in the field of view to the computer. The computer segments and marks the photovoltaic panel in the RGB image through deep learning instance segmentation technology; the edge contour of the photovoltaic panel in the segmentation mark image is obtained through image edge filtering technology; the edge contour of the previous step is fitted by polygon fitting technology to obtain polygons and polygon corner points; the polygon edges and corner points at the edge of the image are filtered out, and the corner points and edges are re-obtained by intersecting adjacent lines to fit into a quadrilateral; then the perspective transformation matrix is obtained by comparing the actual length and width of the photovoltaic panel with the size of the fitted quadrilateral; the image point corresponding to the sampling point in the middle of the image quadrilateral is obtained through the perspective transformation matrix; the corresponding point cloud data in the point cloud depth information is searched through the image point; based on the effective depth point cloud data and the point cloud data of the preset photovoltaic panel image template, the six-degree-of-freedom posture data of the photovoltaic panel space is calculated through the singular value decomposition method.

[0116] The entire system only requires one key sensor, a depth camera, which makes system integration easy. Edge filtering technology and polygon fitting technology are used when processing images to eliminate unstable points at the edge of the photovoltaic panel, making the identified photovoltaic panel posture more stable. Image information and depth point cloud information are combined, and the six spatial degrees of freedom of the photovoltaic panel to be clamped are specifically calculated through the singular value decomposition method. After the automatic installation robot obtains the degree of freedom information, it can automatically grasp the photovoltaic panel through the robotic arm, ultimately realizing automatic and intelligent installation of the photovoltaic panel installation robot.

[0117] In one embodiment, the present application provides a photovoltaic panel posture recognition method. Based on the above embodiment, the method removes invalid point cloud data from the point cloud data to obtain valid depth point cloud data of the photovoltaic panel to be clamped, including the steps of:

[0118] Determine whether there are invalid depth points in the depth point cloud data;

[0119] If there are invalid depth points, their serial numbers are recorded, and the invalid point cloud data corresponding to the invalid depth points are eliminated from the point cloud data of the preset photovoltaic panel image template and the point cloud data of the photovoltaic panel to be clamped, so as to obtain the valid point cloud data of the preset photovoltaic panel image template and the point cloud data of the photovoltaic panel to be clamped.

[0120] In one embodiment, referring to the accompanying drawings Figure 3 The present application also provides a photovoltaic panel posture recognition device, including an acquisition module 10, a segmentation module 20, a fitting module 30, an acquisition module 40, and a calculation module 50, wherein:

[0121] An acquisition module 10 is used to control the depth camera to move and illuminate the photovoltaic panel to be clamped, and to acquire RGB image information and depth point cloud information in the field of view of the depth camera;

[0122] Specifically, the acquisition module 10 collects RGB color images and depth point cloud information, and the depth camera can detect the depth of field distance of the shooting space. Through the data obtained by the depth camera, we can accurately know the distance of each point in the image from the camera. In this way, by adding the (x, y) coordinates of the point in the two-dimensional image, we can obtain the three-dimensional spatial coordinates of each point in the image.

[0123] A segmentation module 20 is configured to segment the RGB image information using a deep learning instance segmentation technique to obtain a grayscale image of the photovoltaic panel to be clamped;

[0124] Specifically, the placement position and posture of the photovoltaic panels are not fixed, and the ambient light will also change. By using the segmentation module 20 to remove the influence of the placement position and posture of the photovoltaic panels and the lighting, the recognition results are made more stable. The RGB color image information is processed through deep learning instance segmentation to obtain a segmented grayscale image of the photovoltaic panel.

[0125] A fitting module 30 is used to perform edge contour detection and polygon fitting on the grayscale image and obtain the fitted quadrilateral image and quadrilateral corner point coordinates of the photovoltaic panel to be clamped;

[0126] Specifically, the fitting module 30 uses image edge filtering technology to obtain the edge contour of the photovoltaic panel in the segmented mark image; then the polygon fitting technology is used to fit the edge contour of the previous step to obtain polygons and polygon corner points; finally, the polygon edges and corner points at the edge of the image are filtered out, and the corner points and edges are obtained again by intersecting adjacent lines, and fitted into a quadrilateral, thereby eliminating the unstable points at the edge of the photovoltaic panel, and the identified photovoltaic panel posture is more stable.

[0127] The acquisition module 40 is used to perform perspective transformation on the coordinates of the fitted quadrilateral corner points of the photovoltaic panel to be clamped and the coordinates of the four corner points of the preset photovoltaic panel image template to obtain the perspective matrix of the photovoltaic panel to be clamped.

[0128] Specifically, the pixel coordinates of the four corner points are calculated by fitting the four sides of the photovoltaic panel to be clamped, and then the coordinates of the four corner points of the actual photovoltaic panel are calculated in combination with the fixed length and width of the actual photovoltaic panel. Then, the perspective matrix of the photovoltaic panel to be clamped is calculated through two groups of four corresponding corner point coordinates, wherein the preset photovoltaic panel image template is the actual photovoltaic panel, that is, the relevant information of the preset photovoltaic panel image template is known.

[0129] The calculation module 50 is used to calculate the six-degree-of-freedom posture of the photovoltaic panel to be clamped through the depth point cloud information of the photovoltaic panel to be clamped, the fitted quadrilateral image, the perspective matrix of the photovoltaic panel to be clamped and the depth point cloud information of the preset photovoltaic panel image template.

[0130] Specifically, 49 points of 7 rows and 7 columns are uniformly sampled inside the actual photovoltaic panel, and the 49 image points in the fitted quadrilateral imaging of the photovoltaic panel to be clamped are calculated through the perspective matrix. According to the row and column coordinates corresponding to the 49 image points in the fitted quadrilateral imaging of the photovoltaic panel to be clamped, the corresponding point cloud is found in the depth point cloud data, and the 49 spatial depth point information is obtained. Based on the 49 spatial depth points on the fitted photovoltaic panel to be clamped captured by the depth camera, the 49 spatial depth points of the preset photovoltaic image template are also known, as well as the fitted quadrilateral image of the photovoltaic panel to be clamped and the perspective matrix of the photovoltaic panel to be clamped, an equation is jointly established, and the calculation module 50 calculates the 6-degree-of-freedom posture information of the photovoltaic panel to be clamped in space through the singular value decomposition method.

[0131] The photovoltaic panel posture recognition method used in this embodiment has been described in detail in the above embodiments and will not be repeated here.

[0132] In one embodiment, referring to the accompanying drawings Figure 3 Based on the above embodiment, the fitting module 30 includes:

[0133] The filtering and screening unit 31 is used to perform edge filtering on the grayscale image of the photovoltaic panel to be clamped, and use the maximum edge contour filtered out as the edge contour of the photovoltaic panel to be clamped;

[0134] The fitting and segmentation unit 32 is used to perform polygon fitting on the edge contour and remove the non-smooth error after segmentation to obtain a polygonal image;

[0135] The filtering and fitting unit 33 is used to filter out the edges and points of the polygonal image, and extend and intersect adjacent edges of the polygonal image to obtain the quadrilateral image of the photovoltaic panel to be clamped and the coordinates of the quadrilateral corner points.

[0136] In this embodiment, the edge contour of the photovoltaic panel in the segmented mark image is obtained by the filtering and screening unit 31 using image edge filtering technology; the edge contour of the previous step is fitted by the fitting and segmentation unit 32 using polygon fitting technology to obtain polygons and polygon corner points; finally, the polygon edges and corner points at the edge of the image are filtered out by the filtering and fitting unit 33, and the corner points and edges are obtained again by intersecting adjacent lines, and fitted into a quadrilateral, thereby eliminating the unstable points at the edge of the photovoltaic panel, and the identified photovoltaic panel posture is more stable.

[0137] In one embodiment, referring to the accompanying drawings Figure 3 Based on the above embodiment, the calculation module 50 includes:

[0138] An acquisition unit 51 is configured to filter valid point clouds in the depth point cloud information of the photovoltaic panel to be clamped through the perspective matrix of the photovoltaic panel to be clamped, and acquire a valid depth point cloud set of the photovoltaic panel to be clamped;

[0139] The acquisition unit 51 is further used to acquire sampling points of a preset photovoltaic panel image template;

[0140] A sampling unit 52 is configured to obtain image points corresponding to the sampling points in the fitted quadrilateral image using the perspective matrix;

[0141] A search unit 53 is configured to search the depth point cloud information for point cloud data corresponding to the image point, remove invalid point cloud data from the point cloud data, and obtain valid depth point cloud data of the photovoltaic panel to be clamped;

[0142] The decomposition acquisition unit 54 is used to establish a matrix equation based on the effective depth point cloud data and the point cloud data of the preset photovoltaic panel image template, and obtain the six-degree-of-freedom posture data in the space of the photovoltaic panel to be clamped through singular value decomposition.

[0143] The photovoltaic panel posture recognition method used in this embodiment has been described in detail in the above embodiments and will not be repeated here.

[0144] In one embodiment, the present application further provides a photovoltaic panel installation robot comprising a depth camera, a processor, and a memory mounted at the end of the robot's mechanical arm. The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the photovoltaic panel posture recognition method described in the above method embodiment.

[0145] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0146] The memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. The memory may also be an external storage device of the terminal device, such as a plug-in hard disk equipped with the terminal device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory may include both an internal storage unit of the terminal device and an external storage device. The memory is used to store the computer program and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is to be output.

[0147] A communication bus is a circuit that connects the elements described and enables transmission between them. For example, a processor receives commands from other elements via a communication bus, decrypts the received commands, and performs calculations or data processing based on the decrypted commands. A memory may include program modules, such as a kernel, middleware, an application programming interface (API), and applications. These program modules may be composed of software, firmware, hardware, or at least two of these. An input / output interface forwards commands or data entered by a user through an input / output interface (such as a sensor, keyboard, or touchscreen). A communication interface connects the terminal device to other network devices, user devices, and networks. For example, a communication interface can connect to a network via a wired or wireless connection to connect to other external network devices or user devices. Wireless communication may include at least one of the following: Wi-Fi, Bluetooth (BT), Near Field Communication (NFC), Global Positioning System (GPS), and cellular communication. Wired communication may include at least one of the following: Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), RS-232, and the like. A network may include a telecommunications network or a communications network. The communication network may be a computer network, the Internet, the Internet of Things, or a telephone network. A terminal device may connect to the network via a communication interface, and the protocol used by the terminal device to communicate with other network devices may be supported by at least one of an application, an application programming interface (API), middleware, a kernel, and a communication interface.

[0148] In one embodiment, the present application further provides a storage medium for photovoltaic panel posture recognition, wherein the storage medium stores at least one instruction, which is loaded and executed by a processor to implement the operations performed by the corresponding embodiment of the image display method in the above-mentioned design scenario. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a read-only compact disc (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0149] They can be implemented using program code executable by a computing device, and thus, can be stored in a storage device and executed by the computing device, or can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0150] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0151] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0153] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0154] It should be noted that the above embodiments can be freely combined as needed. The above are only preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A photovoltaic panel posture recognition method, characterized in that: Including steps: Controlling the depth camera to move and illuminate the photovoltaic panel to be clamped, and collecting RGB image information and depth point cloud information in the field of view of the depth camera; Segment the RGB image information using deep learning instance segmentation technology to obtain a grayscale image of the photovoltaic panel to be clamped; Performing edge contour detection and polygon fitting on the grayscale image to obtain a fitted quadrilateral image and quadrilateral corner point coordinates of the photovoltaic panel to be clamped; Performing perspective transformation on the coordinates of the corner points of the quadrilateral fitted to the photovoltaic panel to be clamped and the coordinates of the four corner points of the preset photovoltaic panel image template to obtain a perspective matrix of the photovoltaic panel to be clamped; Calculating the six-degree-of-freedom pose of the photovoltaic panel to be clamped by using the depth point cloud information of the photovoltaic panel to be clamped, the fitted quadrilateral image, the perspective matrix of the photovoltaic panel to be clamped, and the depth point cloud information of the preset photovoltaic panel image template, specifically comprising the steps of: screening the valid point clouds in the depth point cloud information of the photovoltaic panel to be clamped by using the perspective matrix of the photovoltaic panel to be clamped, and obtaining the valid depth point cloud set of the photovoltaic panel to be clamped; Obtaining sampling points of the preset photovoltaic panel image template; Obtaining image points corresponding to the sampling points in the fitted quadrilateral image using the perspective matrix; searching for point cloud data corresponding to the image points in the depth point cloud information, eliminating invalid point cloud data in the point cloud data, and obtaining valid depth point cloud data of the photovoltaic panel to be clamped; Establishing a matrix equation based on the effective depth point cloud data and the point cloud data of the preset photovoltaic panel image template, and obtaining six-degree-of-freedom pose data in the space of the photovoltaic panel to be clamped by singular value decomposition; The method of eliminating invalid point cloud data in the point cloud data to obtain valid depth point cloud data of the photovoltaic panel to be clamped includes the steps of: judging whether there are invalid depth points in the depth point cloud data; if there are invalid depth points, recording their serial numbers, and eliminating invalid point cloud data corresponding to the invalid depth points in the point cloud data of the preset photovoltaic panel image template and the point cloud data of the photovoltaic panel to be clamped to obtain valid point cloud data of the preset photovoltaic panel image template and the point cloud data of the photovoltaic panel to be clamped.

2. The photovoltaic panel posture recognition method according to claim 1, characterized in that: The edge contour detection and polygon fitting of the grayscale image includes the following steps: Performing edge filtering on the grayscale image of the photovoltaic panel to be clamped, and using the maximum edge contour filtered out as the edge contour of the photovoltaic panel to be clamped; Performing polygon fitting on the edge contour and removing the non-smooth error after segmentation to obtain a polygonal image; The edges and points of the polygonal image are filtered out, and the adjacent edges of the polygonal image are extended and intersected to obtain the quadrilateral image of the photovoltaic panel to be clamped and the coordinates of the quadrilateral corner points.

3. A photovoltaic panel posture recognition device, characterized in that: include: An acquisition module is used to control the depth camera to move and illuminate the photovoltaic panel to be clamped, and to acquire RGB image information and depth point cloud information in the field of view of the depth camera; a segmentation module, configured to segment the RGB image information using a deep learning instance segmentation technique to obtain a grayscale image of the photovoltaic panel to be clamped; A fitting module is used to perform edge contour detection and polygon fitting on the grayscale image to obtain a fitted quadrilateral image and quadrilateral corner point coordinates of the photovoltaic panel to be clamped; An acquisition module is used to perform perspective transformation on the coordinates of the corner points of the quadrilateral fitted by the photovoltaic panel to be clamped and the coordinates of the four corner points of the preset photovoltaic panel image template to obtain the perspective matrix of the photovoltaic panel to be clamped; a calculation module, configured to calculate the six-degree-of-freedom pose of the photovoltaic panel to be clamped by using the depth point cloud information of the photovoltaic panel to be clamped, the fitted quadrilateral image, the perspective matrix of the photovoltaic panel to be clamped, and the depth point cloud information of the preset photovoltaic panel image template; The computing module includes: an acquisition unit, configured to screen valid point clouds in the depth point cloud information of the photovoltaic panel to be clamped through the perspective matrix of the photovoltaic panel to be clamped, and acquire a valid depth point cloud set of the photovoltaic panel to be clamped; the acquisition unit is further configured to acquire sampling points of the preset photovoltaic panel image template; a sampling unit, configured to obtain image points corresponding to the sampling points in the fitted quadrilateral image using the perspective matrix; a search unit, configured to search the depth point cloud information for point cloud data corresponding to the image point, and eliminate invalid point cloud data from the point cloud data to obtain valid depth point cloud data of the photovoltaic panel to be clamped; a decomposition and acquisition unit, configured to establish a matrix equation based on the effective depth point cloud data and the point cloud data of the preset photovoltaic panel image template, and obtain six-degree-of-freedom pose data in the space of the photovoltaic panel to be clamped by singular value decomposition; The search unit eliminates invalid point cloud data in the point cloud data to obtain valid depth point cloud data of the photovoltaic panel to be clamped, including the steps of: judging whether there are invalid depth points in the depth point cloud data; if there are invalid depth points, recording their serial numbers, and eliminating invalid point cloud data corresponding to the invalid depth points in the point cloud data of the preset photovoltaic panel image template and the point cloud data of the photovoltaic panel to be clamped, to obtain valid point cloud data of the preset photovoltaic panel image template and the point cloud data of the photovoltaic panel to be clamped.

4. The photovoltaic panel posture recognition device according to claim 3, characterized in that: The fitting module includes: a filtering and screening unit, configured to perform edge filtering on the grayscale image of the photovoltaic panel to be clamped, and use the maximum edge contour filtered out as the edge contour of the photovoltaic panel to be clamped; A fitting and segmentation unit is used to perform polygon fitting on the edge contour and remove the non-smooth error after segmentation to obtain a polygonal image; The filtering and fitting unit is used to filter out the edges and points of the polygonal image, and extend and intersect adjacent edges of the polygonal image to obtain the quadrilateral image of the photovoltaic panel to be clamped and the coordinates of the quadrilateral corner points.

5. A photovoltaic panel installation robot, characterized in that: include: A depth camera is mounted at the end of the robot's mechanical arm. A processor, wherein when executing program instructions, the processor implements the photovoltaic panel posture recognition method according to any one of claims 1 to 3.

Images