Method and device for acquiring normal vector of photovoltaic panel, medium and screw locking robot

Abstract

The invention relates to a normal vector acquisition method and device for photovoltaic panels, a medium and a screw locking robot, and the method comprises the steps: obtaining an array point cloud and an array image of a photovoltaic array which are displayed in a same coordinate system in an overlapping manner, and the array point cloud and the array image comprise a plurality of photovoltaic panels; carrying out image identification on the array image, and obtaining boundary features of the plurality of photovoltaic panels; based on the boundary features, obtaining centroid coordinates of corresponding boundary point clouds; acquiring panel point clouds corresponding to the photovoltaic panels in the array point clouds based on the centroid coordinates of the boundary point clouds and the pre-acquired sizes of the photovoltaic panels; and performing plane fitting on each panel point cloud to obtain a normal vector corresponding to each photovoltaic panel, thereby solving the problem of relatively low installation quality of a photovoltaic array caused by incorrect locking direction of a screw.

Description

Technical Field

[0001] The present application relates to the technical field of photovoltaic equipment, and in particular to a method, device, medium and screw locking robot for obtaining a normal vector of a photovoltaic panel. Background Art

[0002] During the construction of photovoltaic power stations, a large number of photovoltaic arrays need to be fixed to pre-designed positions with screws. Currently, the photovoltaic modules are generally installed manually and screwed to tighten the photovoltaic modules. However, the process of screwing is highly repetitive, manual fatigue is easy to occur, and the problem of non-standard tightening process is prone to occur. At present, large-scale photovoltaic power stations have been trying to use photovoltaic installation robotic arms or mobile photovoltaic power station construction robots to assist in the installation of photovoltaic arrays through visual positioning technology. During installation, the normal vector of the photovoltaic panel is usually obtained through visual positioning, and the screw locking device is controlled to tighten the screws in the direction of the normal vector. However, due to certain errors in visual positioning, when the robot tightens the screws in the direction of the identified normal vector, there may be deviations in the tightening direction of the screws, resulting in the inability to complete the installation, or the photovoltaic panel fixation quality is poor, or even damage to the photovoltaic panel, affecting the installation quality and efficiency of the photovoltaic array.

[0003] With regard to the problem in the related art that incorrect tightening direction of screws leads to low installation quality of photovoltaic arrays, no effective solution has been proposed so far. Summary of the invention

[0004] In this embodiment, a method, device, medium and screw locking robot for obtaining a normal vector of a photovoltaic panel are provided to solve the problem in the related art that incorrect screw locking direction leads to low installation quality of the photovoltaic array.

[0005] In a first aspect, a method for obtaining a normal vector of a photovoltaic panel is provided in this embodiment. A plurality of photovoltaic panels are arranged in sequence to form a photovoltaic array. The method includes:

[0006] Acquire an array point cloud and an array image of the photovoltaic array that are superimposed and displayed in the same coordinate system, wherein the array point cloud and the array image include a plurality of the photovoltaic panels;

[0007] Performing image recognition on the array image to obtain boundary features of the plurality of photovoltaic panels;

[0008] Based on the boundary features, obtaining the centroid coordinates of the corresponding boundary point cloud;

[0009] Based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel, acquiring a panel point cloud corresponding to each photovoltaic panel in the array point cloud;

[0010] Plane fitting is performed on each of the panel point clouds to obtain a normal vector corresponding to each of the photovoltaic panels.

[0011] In some embodiments, performing image recognition on the array image to obtain boundary features of the plurality of photovoltaic panels includes:

[0012] Acquire a grayscale image of the array image and perform image segmentation to obtain gap areas between the plurality of photovoltaic panels;

[0013] Acquire a connected domain of the segmented grayscale image to obtain regions corresponding to the plurality of photovoltaic panels;

[0014] Based on the areas corresponding to the plurality of photovoltaic panels and the gap areas, boundary lines of the plurality of photovoltaic panels are generated in the array image.

[0015] In some embodiments, acquiring a grayscale image of the array image and performing image segmentation to obtain the gap area between the plurality of photovoltaic panels comprises:

[0016] Counting pixel values ​​of the grayscale image to obtain a corresponding grayscale histogram;

[0017] Based on the maximum pixel value in the grayscale histogram, the grayscale image is segmented to obtain the gap areas between the multiple photovoltaic panels.

[0018] In some embodiments, obtaining the centroid coordinates of the corresponding boundary point cloud based on the boundary feature includes:

[0019] Coloring the point cloud data having the boundary features in the array point cloud to obtain the boundary point cloud;

[0020] Based on the coordinates of each point cloud data in the boundary point cloud, the centroid coordinates of the boundary point cloud are calculated.

[0021] In some of the embodiments, obtaining the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel includes:

[0022] Based on the centroid coordinates and a first preset threshold, filtering out point cloud data in the array point cloud that is located outside the first preset threshold on the plane where the photovoltaic panel is located;

[0023] Based on the width of the photovoltaic panel, point cloud data corresponding to photovoltaic panels outside a second preset threshold value adjacent to the photovoltaic panel in the array point cloud are filtered out to obtain a panel point cloud corresponding to the photovoltaic panel.

[0024] In some embodiments, performing plane fitting on each of the panel point clouds to obtain a normal vector corresponding to each of the photovoltaic panels includes:

[0025] Perform multiple plane fitting based on the point cloud data in the panel point cloud to obtain corresponding multiple candidate plane equations;

[0026] Obtaining the number of point cloud data satisfying each of the candidate plane equations in the panel point cloud;

[0027] Based on the candidate plane equation corresponding to the maximum value of the point cloud data quantity, the plane equation of the photovoltaic panel is determined, and then the normal vector corresponding to the photovoltaic panel is obtained.

[0028] In some embodiments, the method is applied to a screw locking robot, the screw locking robot includes a mechanical arm and a laser radar and a camera mounted on the mechanical arm, and the acquisition of the array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system includes:

[0029] Acquire the array point cloud based on the laser radar, and acquire the array image based on the camera;

[0030] Based on the coordinate conversion equation between the laser radar and the camera, the array point cloud is converted to the camera coordinate system corresponding to the camera and displayed superimposed with the array image.

[0031] In a second aspect, a normal vector acquisition device for a photovoltaic panel is provided in this embodiment, wherein a plurality of photovoltaic panels are arranged in sequence to form a photovoltaic array, and the device comprises:

[0032] A first acquisition module is used to acquire an array point cloud and an array image of the photovoltaic array superimposed and displayed in the same coordinate system, wherein the array point cloud and the array image include a plurality of the photovoltaic panels;

[0033] A recognition module, used to perform image recognition on the array image to obtain boundary features of the plurality of photovoltaic panels;

[0034] A second acquisition module, used to acquire the centroid coordinates of the corresponding boundary point cloud based on the boundary feature;

[0035] A third acquisition module is used to acquire a panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel;

[0036] The fitting module is used to perform plane fitting on each of the panel point clouds to obtain a normal vector corresponding to each of the photovoltaic panels.

[0037] According to a third aspect, a readable storage medium is provided in this embodiment, on which a program is stored, and when the program is executed by a processor, the steps of the method for obtaining the normal vector of the photovoltaic panel described in the first aspect are implemented.

[0038] In a fourth aspect, a screw locking robot is provided in this embodiment, the screw locking robot comprising a controller, a mechanical arm, and a screw locking device, a laser radar, and a camera installed on the mechanical arm.

[0039] The controller obtains the normal vector of the photovoltaic panel based on the normal vector acquisition method of the photovoltaic panel described in any one of the first aspect to the second aspect, and controls the screw locking device to perform a screw locking operation on the photovoltaic panel based on the normal vector.

[0040] Compared with the related art, the normal vector acquisition method of the photovoltaic panel provided in the present embodiment converts the point cloud data generated by the laser radar and the image data generated by the camera into the same coordinate system by acquiring the array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system, so as to facilitate the subsequent data processing based on the position coordinates of each feature point on the photovoltaic panel; by performing image recognition on the array image, the boundary features of multiple photovoltaic panels are obtained, and necessary reference data is provided for the identification of the photovoltaic panels; by obtaining the centroid coordinates of the corresponding boundary point cloud based on the boundary features, the image features are converted into point cloud features; by obtaining the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the size of the photovoltaic panel obtained in advance, the point cloud data of each photovoltaic panel is identified from the array point cloud; by performing plane fitting on each panel point cloud, the normal vector corresponding to each photovoltaic panel is obtained, so as to ensure the correctness of the screw tightening direction and solve the problem of low installation quality of the photovoltaic array due to incorrect screw tightening direction.

[0041] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 is a computer hardware structure block diagram of a method for obtaining a normal vector of a photovoltaic panel in some embodiments of the present application;

[0044] Figure 2 is a flow chart of a method for obtaining a normal vector of a photovoltaic panel in some embodiments of the present application;

[0045] Figure 3 is a flow chart for obtaining boundary features of a photovoltaic panel in some embodiments of the present application;

[0046] Figure 4 is a flowchart of obtaining the gap area of ​​a photovoltaic panel in some embodiments of the present application;

[0047] Figure 5 is a flowchart for obtaining the centroid coordinates of the boundary point cloud in some embodiments of the present application;

[0048] Figure 6 is a flow chart for obtaining panel point clouds in some embodiments of the present application;

[0049] Figure 7 is a flow chart for obtaining the normal vector of a photovoltaic panel in some embodiments of the present application;

[0050] Figure 8 is a flowchart of array point cloud and array image overlay display in some embodiments of the present application;

[0051] Fig. 9 is a flow chart of a method for obtaining a normal vector of a photovoltaic panel in some preferred embodiments of the present application;

[0052] Fig.10 This is a structural block diagram of a normal vector acquisition device for a photovoltaic panel in some embodiments of the present application. DETAILED DESCRIPTION

[0053] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0055] The method for obtaining the normal vector of a photovoltaic panel provided in the embodiment of the present application can be executed in a processor of a server, a computer, a terminal or a similar computing device. Figure 1 1 is a block diagram of the computer hardware structure of the method for obtaining the normal vector of a photovoltaic panel in some embodiments of the present application. Figure 1 As shown, the computer may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a CPU, a microprocessor MCU or a programmable logic device FPGA. The above computer may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer. Figure 1 More or fewer arrays as shown, or with Figure 1 Different configurations shown.

[0056] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the normal vector acquisition method of the photovoltaic panel in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the normal vector acquisition method of the photovoltaic panel described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some embodiments, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0057] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by a communication provider. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0058] In this embodiment, a method for obtaining a normal vector of a photovoltaic panel is provided. Figure 2 is a flow chart of a method for obtaining a normal vector of a photovoltaic panel in some embodiments of the present application. Figure 2 As shown, the process includes the following steps:

[0059] Step S201 : acquiring an array point cloud and an array image of a photovoltaic array superimposed and displayed in the same coordinate system, wherein the array point cloud and the array image include a plurality of the photovoltaic panels.

[0060] The photovoltaic array includes a plurality of photovoltaic panels arranged in sequence, and the installation direction and tilt angle of each photovoltaic panel can be the same or different. The array point cloud refers to the point cloud data set in the three-dimensional space corresponding to the photovoltaic array. The array point cloud is obtained by scanning the photovoltaic array through a laser radar. The array image refers to the two-dimensional image data corresponding to the photovoltaic array. The array image can be obtained by photographing the photovoltaic array through an image acquisition device such as a camera.

[0061] In some embodiments, the array point cloud and the array image are generated based on different coordinate systems, for example, the array point cloud is generated based on the laser radar coordinate system, and the array image is generated based on the camera coordinate system. The array point cloud and the array image can be unified into the same coordinate system through the coordinate conversion equation between the two coordinate systems. The array point cloud and the array image after the coordinate system is unified can be superimposed and displayed on the display device, and the position coordinates of the same photovoltaic panel in the photovoltaic array are the same in the array point cloud and the array image.

[0062] Step S202: performing image recognition on the array image to obtain boundary features of multiple photovoltaic panels.

[0063] By image recognition, the boundary features of each photovoltaic panel in the array image are obtained. The boundary features can be the boundary area, boundary points or boundary lines of the photovoltaic panel. There are many ways to obtain image features by image recognition, which are not limited in this embodiment.

[0064] Step S203: based on the boundary feature, obtain the centroid coordinates of the corresponding boundary point cloud.

[0065] According to the boundary features of each photovoltaic panel in the array image, a boundary point cloud corresponding to the boundary feature is obtained, and the centroid coordinates of the boundary point cloud are further obtained. The boundary point cloud can be composed of point cloud data that coincides with the position of the boundary feature. The centroid coordinates can be calculated based on the position coordinates of all point cloud data in the boundary point cloud.

[0066] Step S204 , based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel, obtain the panel point cloud corresponding to each photovoltaic panel in the array point cloud.

[0067] Since the point cloud data in the boundary point cloud coincides with the boundary feature position of the photovoltaic panel, the centroid coordinates of the boundary point cloud can be used as the boundary point coordinates of the photovoltaic panel. Then, based on the pre-acquired size of the photovoltaic panel, the coordinate range of the photovoltaic panel is determined, and the point cloud data within the coordinate range is included in the panel point cloud corresponding to the photovoltaic panel. Other point cloud data that do not belong to the coordinate range are filtered out, thereby obtaining the panel point cloud corresponding to each photovoltaic panel.

[0068] Step S205 , performing plane fitting on each panel point cloud to obtain a normal vector corresponding to each photovoltaic panel.

[0069] A plane fitting is performed on the panel point cloud corresponding to each photovoltaic panel. The specific fitting method is not limited. The plane equation corresponding to each panel point cloud is obtained, and the normal vector corresponding to the photovoltaic panel is calculated according to the plane equation.

[0070] Through steps S201 to S205, by acquiring the array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system, the point cloud data generated by the laser radar and the image data generated by the camera are converted into the same coordinate system, so as to facilitate subsequent data processing based on the position coordinates of each feature point on the photovoltaic panel; by performing image recognition on the array image, the boundary features of multiple photovoltaic panels are acquired to provide necessary reference data for the identification of the photovoltaic panels; by acquiring the centroid coordinates of the corresponding boundary point cloud based on the boundary feature, the image features are converted into point cloud features; by acquiring the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the size of the photovoltaic panel acquired in advance, the point cloud data of each photovoltaic panel is identified from the array point cloud; by performing plane fitting on each panel point cloud, the normal vector corresponding to each photovoltaic panel is obtained, so as to ensure the correctness of the screw locking direction and solve the problem of low installation quality of the photovoltaic array due to incorrect screw locking direction.

[0071] In some embodiments, Figure 3 is a flowchart for obtaining boundary features of photovoltaic panels in some embodiments of the present application, such as Figure 3 As shown, the process includes the following steps:

[0072] Step S301 , obtaining a grayscale image of an array image and performing image segmentation to obtain gap regions between a plurality of photovoltaic panels.

[0073] The array image can be an RGB format image, and the array image is grayed to obtain a gray image. Since the brightness of the photovoltaic panel gap area is different from the brightness of the photovoltaic panel area, the gray image can be segmented according to the pixel value of each pixel in the gray image to obtain the gap area between each photovoltaic panel.

[0074] Step S302, obtaining a connected domain of the segmented grayscale image to obtain regions corresponding to a plurality of photovoltaic panels.

[0075] According to the location of the gap area in the grayscale image, a connected domain analysis is performed on the grayscale image. Specifically, the grayscale image can be binarized to distinguish the gap area from the non-gap area, and the binarized image can be subjected to a connected domain analysis to obtain the image area corresponding to each photovoltaic panel. The algorithm for the connected domain analysis is not described in detail in this embodiment.

[0076] Furthermore, before performing the connected domain analysis, irrelevant noise points in the image may be eliminated by using a morphological opening operation.

[0077] Step S303 : generating boundary lines of the plurality of photovoltaic panels in the array image based on the regions corresponding to the plurality of photovoltaic panels and the gap region.

[0078] According to the image areas corresponding to the photovoltaic panels and the gap areas, the boundary lines of each photovoltaic panel are generated in the array image. Specifically, the boundary of the gap area, ie, the boundary line of the photovoltaic panel, can be drawn in the array image using the Hough line detection algorithm.

[0079] Through steps S301 to S303, by acquiring a grayscale image of the array image and performing image segmentation, the gap area between the multiple photovoltaic panels is obtained, and the position of the gap between the photovoltaic panels in the array image is obtained; by acquiring the connected domain of the segmented grayscale image, the area corresponding to the multiple photovoltaic panels is obtained, and the position of the photovoltaic panels in the array image is obtained; by generating the boundary lines of the multiple photovoltaic panels in the array image based on the area corresponding to the multiple photovoltaic panels and the gap area, the boundary features of each photovoltaic panel in the array image are obtained, and the necessary reference data is obtained for obtaining the point cloud corresponding to each photovoltaic panel.

[0080] In some embodiments, Figure 4 is a flow chart for obtaining the gap area of ​​a photovoltaic panel in some embodiments of the present application, such as Figure 4 As shown, the process includes the following steps:

[0081] Step S401, counting the pixel values ​​of the grayscale image to obtain a corresponding grayscale histogram.

[0082] Specifically, the horizontal axis of the grayscale histogram is the pixel value in the grayscale image, and the vertical axis of the grayscale histogram is the number of occurrences corresponding to the pixel value.

[0083] Step S402 : performing image segmentation on the grayscale image based on the maximum pixel value in the grayscale histogram to obtain gap regions between the plurality of photovoltaic panels.

[0084] Since light enters the gaps between photovoltaic panels, the gap area can be identified by the maximum pixel value in the grayscale image. Specifically, the peak with the largest pixel value in the histogram peak can be found through Scipy's find_peaks function, and the pixel value corresponding to the peak is used as the input value of the image threshold segmentation algorithm to segment the brightest gap area between each photovoltaic panel.

[0085] Through steps S401 to S402, the corresponding grayscale histogram is obtained by counting the pixel values ​​of the grayscale image, and based on the maximum pixel value in the grayscale histogram, the grayscale image is segmented to obtain the gap area between multiple photovoltaic panels, thereby obtaining the position of the gap between the photovoltaic panels in the array image, providing an accurate position coordinate reference for subsequent identification of the photovoltaic panels.

[0086] In some embodiments, Figure 5 is a flowchart for obtaining the centroid coordinates of the boundary point cloud in some embodiments of the present application, such as Figure 5As shown, the process includes the following steps:

[0087] Step S501 , coloring the point cloud data with boundary features in the array point cloud to obtain a boundary point cloud.

[0088] Specifically, the boundary feature may be a boundary area, boundary point or boundary line of the photovoltaic panel. In one embodiment, the boundary feature is a boundary line of the photovoltaic panel. The point cloud data that coincides with the position coordinates of the boundary line is colored to distinguish it from other point cloud data to obtain a boundary point cloud.

[0089] Step S502: based on the coordinates of each point cloud data in the boundary point cloud, the centroid coordinates of the boundary point cloud are calculated.

[0090] According to the coordinates of each point cloud data in the boundary point cloud, the centroid coordinates of the boundary point cloud are calculated. Specifically, the average coordinates of each point cloud data in the X, Y, and Z directions can be calculated respectively as the centroid coordinates (x, y, z) of the boundary point cloud.

[0091] Through steps S501 to S502, the point cloud data with boundary features in the array point cloud is colored to obtain the boundary point cloud. Based on the coordinates of each point cloud data in the boundary point cloud, the centroid coordinates of the boundary point cloud are calculated to obtain the accurate coordinates of the boundary points of the photovoltaic panel, thereby improving the position accuracy of the subsequent acquisition of the panel point cloud corresponding to the photovoltaic panel.

[0092] In some embodiments, Figure 6 is a flow chart of obtaining panel point cloud in some embodiments of the present application, such as Figure 6 As shown, the process includes the following steps:

[0093] Step S601 : based on the centroid coordinates and the first preset threshold, filter out the point cloud data in the array point cloud that is located outside the first preset threshold on the plane where the photovoltaic panel is located.

[0094] Since the thickness of the photovoltaic panel is relatively small, and the point clouds of the upper and lower surfaces are separated from the point clouds of the environment, a straight-through filtering algorithm can be used to filter out the point cloud data outside the first preset threshold of the thickness of the plane where the photovoltaic panel is located according to the preset threshold of the thickness of the photovoltaic panel. In a specific embodiment, the Z-axis direction in the three-dimensional coordinate system is set as the thickness direction of the photovoltaic panel, and the point cloud data outside the range of [za, z+a] is filtered out. Where z is the coordinate of the Z-axis centroid, and a is the first preset threshold.

[0095] Step S602 , based on the width of the photovoltaic panel, filter out point cloud data corresponding to the photovoltaic panel outside a second preset threshold value adjacent to the photovoltaic panel in the array point cloud to obtain a panel point cloud corresponding to the photovoltaic panel.

[0096] When the array point cloud includes multiple photovoltaic panels arranged in rows, the point cloud data corresponding to other photovoltaic panels adjacent to the photovoltaic panel can be filtered out by a straight-through filtering algorithm based on the boundary line of the photovoltaic panel in the width direction and the pre-acquired photovoltaic panel width value. In a specific embodiment, assuming that the boundary line where the boundary point cloud is located is the left boundary line of the photovoltaic panel, the point cloud data outside the range of [x-b1, x-b2] is filtered out according to the centroid coordinates of the boundary point cloud and the width of the photovoltaic panel, where x is the X-axis centroid coordinate, and b1 and b2 are the second preset thresholds in the preset width direction.

[0097] Similarly, when the array point cloud includes a plurality of photovoltaic panels arranged in columns, the point cloud data corresponding to other photovoltaic panels adjacent to the photovoltaic panel may be filtered out based on the boundary line of the photovoltaic panel in the length direction and the pre-acquired photovoltaic panel length value.

[0098] After filtering the point clouds in different directions as described above, the panel point cloud corresponding to the photovoltaic panel can be obtained.

[0099] Through steps S601 to S602, point cloud data in the array point cloud that are located outside the first preset threshold on the plane where the photovoltaic panel is located are filtered out based on the centroid coordinates and the first preset threshold; point cloud data corresponding to photovoltaic panels in the array point cloud that are located outside the second preset threshold adjacent to the photovoltaic panel are filtered out based on the width of the photovoltaic panel, and the panel point cloud corresponding to the photovoltaic panel is obtained, and the point cloud data of each photovoltaic panel is identified from the array point cloud and used for the calculation of the normal vector corresponding to each photovoltaic panel, thereby improving the accuracy of obtaining the normal vector of each photovoltaic panel.

[0100] In some embodiments, Figure 7 is a flowchart for obtaining the normal vector of a photovoltaic panel in some embodiments of the present application, such as Figure 7 As shown, the process includes the following steps:

[0101] Step S701, performing multiple plane fittings based on the point cloud data in the panel point cloud to obtain corresponding multiple candidate plane equations.

[0102] Specifically, the point cloud planes of each photovoltaic panel can be fitted separately through the point cloud segmentation function pcd.segment_plane() of Open3D. This function can randomly collect point clouds with given parameters within the set range parameters for plane fitting. The number of fittings is also given as a parameter, for example, randomly taking 3 points each time to fit the plane within a certain range. In this way, multiple candidate plane equations can be obtained by multiple fittings.

[0103] Step S702, obtaining the number of point cloud data satisfying each candidate plane equation in the panel point cloud.

[0104] After obtaining the candidate plane equation, the number of point cloud data that satisfies the candidate plane equation in the panel point cloud is calculated, that is, the number of point cloud data located in the plane corresponding to the candidate plane equation.

[0105] Step S703, determining the plane equation of the photovoltaic panel based on the candidate plane equation corresponding to the maximum value of the point cloud data quantity, and then obtaining the normal vector corresponding to the photovoltaic panel.

[0106] The candidate plane equation with the largest number of point cloud data is taken as the final selected plane equation, and the corresponding normal vector is calculated according to the plane equation.

[0107] Through steps S701 to S703, multiple plane fittings are performed based on the point cloud data in the panel point cloud to obtain multiple corresponding candidate plane equations, and the number of point cloud data in the panel point cloud that satisfies each candidate plane equation is obtained. Based on the candidate plane equation corresponding to the maximum number of point cloud data, the plane equation of the photovoltaic panel is determined, and then the normal vector corresponding to the photovoltaic panel is obtained, thereby improving the accuracy of the normal vector calculation of the photovoltaic panel.

[0108] In some embodiments, Figure 8 is a flowchart of array point cloud and array image overlay display in some embodiments of the present application, such as Figure 8 As shown, the process includes the following steps:

[0109] Step S801, acquiring array point cloud based on the laser radar and acquiring array image based on the camera.

[0110] The laser radar and camera in this embodiment can be installed in an environment near the photovoltaic panel, or can be installed on the mechanical arm of the screw locking robot. The laser radar and camera respectively obtain the array point cloud and array image of the photovoltaic array based on different coordinate systems.

[0111] Step S802, based on the coordinate conversion equation between the laser radar and the camera, the array point cloud is converted to the camera coordinate system corresponding to the camera, and is superimposed and displayed with the array image.

[0112] Before acquiring the array point cloud and array image, the coordinate conversion equation between the camera and the laser radar has been obtained by joint calibration of the camera and the laser radar. The steps of the joint calibration are not repeated in this embodiment. In a specific embodiment, the relationship between the camera coordinate system and the laser radar coordinate system can be obtained by calling the corresponding function, and then the conversion matrix between the two coordinate systems is calculated. According to the conversion matrix, the array point cloud is converted to the camera coordinate system and displayed superimposed with the array image.

[0113] Through steps S801 to S802, the array point cloud is obtained based on the laser radar, and the array image is obtained based on the camera, and different forms of position data of the same photovoltaic array are collected; through the coordinate conversion equation between the laser radar and the camera, the array point cloud is converted to the camera coordinate system corresponding to the camera, and superimposed with the array image for display, providing accurate reference data for subsequent data processing of the array image and array point cloud.

[0114] The present embodiment is described and illustrated below through preferred embodiments. Fig. 9 is a flow chart of a method for obtaining a normal vector of a photovoltaic panel in some preferred embodiments of the present application. Fig. 9 As shown, the process includes the following steps:

[0115] Step S901, using an industrial camera to obtain an RGB array image of the photovoltaic array, and reading the array image into an image processing program of an industrial computer;

[0116] Step S902, graying the array image and obtaining a grayscale histogram of the image by statistics;

[0117] Step S903, using the find_peaks function of Scipy to find the peak with the largest pixel value in the histogram peaks, and using the pixel value corresponding to the peak as the input of the image threshold segmentation algorithm to segment the brightest gap area between the photovoltaic panels;

[0118] Step S904, using morphological opening operation to eliminate irrelevant noise points in the image, and calculating the connected domain in the image, retaining the pixel region with the largest connected domain area in the image;

[0119] Step S905, using the Hough line detection algorithm to draw the boundary of the gap in the array image, that is, the boundary features between components;

[0120] Step S906, the relationship between the camera coordinate system and the lidar coordinate system can be obtained through the tf monitoring function of the tf2_ros library, and the transformation matrix between the two coordinate systems can be calculated;

[0121] Step S907, converting the array point cloud acquired by the laser radar into the camera coordinate system according to the conversion matrix, and superimposing it with the array image for display;

[0122] The order of steps S901 to S905 and steps S906 to S907 can be swapped.

[0123] Step S908, coloring the point cloud data with boundary features in the array point cloud to obtain a boundary point cloud;

[0124] Step S909, based on the coordinates of each point cloud data in the boundary point cloud, calculate the centroid coordinates of the boundary point cloud;

[0125] Step S910, based on the centroid coordinates and the first preset threshold, filtering out the point cloud data in the array point cloud that is outside the thickness of the plane where the photovoltaic panel is located;

[0126] Step S911, based on the width of the photovoltaic panel, filtering out point cloud data corresponding to the photovoltaic panel beyond a second preset threshold value adjacent to the photovoltaic panel in the array point cloud, to obtain a panel point cloud corresponding to the photovoltaic panel;

[0127] Step S912, using the point cloud segmentation function pcd.segment_plane() of Open3D to fit the point cloud planes of each photovoltaic panel respectively, and obtain corresponding multiple candidate plane equations;

[0128] Step S913, obtaining the number of point cloud data satisfying each candidate plane equation in the panel point cloud;

[0129] Step S914, determining the plane equation of the photovoltaic panel based on the candidate plane equation corresponding to the maximum value of the point cloud data quantity, and then obtaining the normal vector corresponding to the photovoltaic panel.

[0130] Through steps S901 to S914, the array image of the photovoltaic array is grayscaled, and the maximum pixel value in the grayscale image is used as the input of the image threshold segmentation algorithm to segment the brightest gap area between the photovoltaic panels, and the boundary features of the photovoltaic panels are obtained through connected domain analysis to provide necessary reference data for the identification of the photovoltaic panels; by converting the point cloud data generated by the lidar and the image data generated by the camera into the same coordinate system, it is convenient for subsequent data processing based on the position coordinates of each feature point on the photovoltaic panel; the specific position of the boundary features of the photovoltaic panel is obtained through the centroid coordinate calculation, and the panel point cloud corresponding to the photovoltaic panel is obtained through the straight-through filtering algorithm and the size parameters of the photovoltaic panel in the length, width and thickness directions and the preset threshold, so as to accurately obtain the position data of the photovoltaic panel surface; by plane fitting of each panel point cloud, the normal vector corresponding to each photovoltaic panel is obtained, so as to ensure the correctness of the screw locking direction and solve the problem of low installation quality of the photovoltaic array caused by incorrect screw locking direction.

[0131] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0132] In some embodiments, the present application further provides a photovoltaic panel normal vector acquisition device, which is used to implement the above embodiments and preferred implementations, and will not be repeated for those that have been described. The terms "module", "unit", "subunit", etc. used below may be a combination of software and / or hardware that implements a predetermined function. In some embodiments, Fig.10 is a structural block diagram of the normal vector acquisition device of the photovoltaic panel of this embodiment, such as Fig.10 As shown, the device comprises:

[0133] A first acquisition module 1001 is used to acquire an array point cloud and an array image of a photovoltaic array superimposed and displayed in the same coordinate system, wherein the array point cloud and the array image include a plurality of photovoltaic panels;

[0134] The recognition module 1002 is used to perform image recognition on the array image to obtain boundary features of multiple photovoltaic panels;

[0135] The second acquisition module 1003 is used to acquire the centroid coordinates of the corresponding boundary point cloud based on the boundary features;

[0136] The third acquisition module 1004 is used to acquire the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the size of the photovoltaic panel acquired in advance;

[0137] The fitting module 1005 is used to perform plane fitting on each panel point cloud to obtain the normal vector corresponding to each photovoltaic panel.

[0138] The normal vector acquisition device of the photovoltaic panel of this embodiment acquires the array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system through the first acquisition module 1001, and converts the point cloud data generated by the laser radar and the image data generated by the camera into the same coordinate system, so as to facilitate the subsequent data processing based on the position coordinates of each feature point on the photovoltaic panel; the recognition module 1002 performs image recognition on the array image to obtain the boundary features of multiple photovoltaic panels, and provides necessary reference data for the recognition of the photovoltaic panels; the second acquisition module 1003 acquires the centroid coordinates of the corresponding boundary point cloud based on the boundary features, and converts the image features into point cloud features; the third acquisition module 1004 acquires the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the size of the photovoltaic panel acquired in advance, and identifies the point cloud data of each photovoltaic panel from the array point cloud; the fitting module 1005 performs plane fitting on each panel point cloud to obtain the normal vector corresponding to each photovoltaic panel, thereby ensuring the correctness of the screw tightening direction and solving the problem of low installation quality of the photovoltaic array caused by incorrect screw tightening direction.

[0139] In addition, in combination with the photovoltaic panel normal vector acquisition method provided in the above embodiments, a readable storage medium can also be provided in this embodiment to implement the method. The readable storage medium stores a program; when the program is executed by the processor, any photovoltaic panel normal vector acquisition method in the above embodiments is implemented.

[0140] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0141] In some embodiments, the present application also provides a screw locking robot, which includes a controller, a robotic arm, a screw locking device, a laser radar and a camera installed on the robotic arm. The controller obtains the normal vector of the photovoltaic panel based on the normal vector acquisition method of the photovoltaic panel in the above-mentioned embodiment, and controls the screw locking device to perform a screw locking operation on the photovoltaic panel based on the normal vector.

[0142] The screw locking robot of this embodiment obtains the array point cloud and array image corresponding to the photovoltaic panel through laser radar and camera respectively, and collects different forms of position data of the same photovoltaic array; the normal vector of the photovoltaic panel is obtained through the controller, which ensures the correctness of the screw tightening direction and improves the installation quality and efficiency of the photovoltaic panel.

[0143] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0144] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0145] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0146] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0147] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A method for obtaining a normal vector of a photovoltaic panel, wherein a plurality of photovoltaic panels are arranged in sequence to form a photovoltaic array, characterized in that: The method comprises: Acquire an array point cloud and an array image of the photovoltaic array that are superimposed and displayed in the same coordinate system, wherein the array point cloud and the array image include a plurality of the photovoltaic panels; Performing image recognition on the array image to obtain boundary features of the plurality of photovoltaic panels; Based on the boundary features, obtaining the centroid coordinates of the corresponding boundary point cloud; Based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel, acquiring a panel point cloud corresponding to each photovoltaic panel in the array point cloud; Plane fitting is performed on each of the panel point clouds to obtain a normal vector corresponding to each of the photovoltaic panels.

2. The method according to claim 1, characterized in that The performing image recognition on the array image to obtain boundary features of the plurality of photovoltaic panels includes: Acquire a grayscale image of the array image and perform image segmentation to obtain gap areas between the plurality of photovoltaic panels; Acquire a connected domain of the segmented grayscale image to obtain regions corresponding to the plurality of photovoltaic panels; Based on the areas corresponding to the plurality of photovoltaic panels and the gap area, boundary lines of the plurality of photovoltaic panels are generated in the array image.

3. The method according to claim 2, characterized in that The step of acquiring a grayscale image of the array image and performing image segmentation to obtain gap areas between the plurality of photovoltaic panels comprises: Counting pixel values ​​of the grayscale image to obtain a corresponding grayscale histogram; Based on the maximum pixel value in the grayscale histogram, the grayscale image is segmented to obtain the gap areas between the multiple photovoltaic panels.

4. The method according to claim 1, characterized in that: The acquiring the centroid coordinates of the corresponding boundary point cloud based on the boundary feature includes: Coloring the point cloud data having the boundary features in the array point cloud to obtain the boundary point cloud; Based on the coordinates of each point cloud data in the boundary point cloud, the centroid coordinates of the boundary point cloud are calculated.

5. The method according to claim 1, characterized in that The step of obtaining the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel comprises: Based on the centroid coordinates and a first preset threshold, filtering out point cloud data in the array point cloud that is located outside the first preset threshold on the plane where the photovoltaic panel is located; Based on the width of the photovoltaic panel, point cloud data corresponding to a photovoltaic panel adjacent to the photovoltaic panel within a second preset threshold in the array point cloud is filtered out to obtain a panel point cloud corresponding to the photovoltaic panel.

6. The method according to claim 1, characterized in that The performing plane fitting on each of the panel point clouds to obtain the normal vector corresponding to each of the photovoltaic panels comprises: Perform multiple plane fitting based on the point cloud data in the panel point cloud to obtain corresponding multiple candidate plane equations; Obtaining the number of point cloud data satisfying each of the candidate plane equations in the panel point cloud; Based on the candidate plane equation corresponding to the maximum value of the point cloud data quantity, the plane equation of the photovoltaic panel is determined, and then the normal vector corresponding to the photovoltaic panel is obtained.

7. The method according to claim 1, characterized in that The method is applied to a screw locking robot, the screw locking robot includes a mechanical arm and a laser radar and a camera installed on the mechanical arm, and the acquisition of the array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system includes: Acquire the array point cloud based on the laser radar, and acquire the array image based on the camera; Based on the coordinate conversion equation between the laser radar and the camera, the array point cloud is converted to the camera coordinate system corresponding to the camera and displayed superimposed with the array image.

8. A normal vector acquisition device for a photovoltaic panel, wherein a plurality of photovoltaic panels are arranged in sequence to form a photovoltaic array, characterized in that: The device comprises: A first acquisition module is used to acquire an array point cloud and an array image of the photovoltaic array superimposed and displayed in the same coordinate system, wherein the array point cloud and the array image include a plurality of the photovoltaic panels; A recognition module, used to perform image recognition on the array image to obtain boundary features of the plurality of photovoltaic panels; A second acquisition module, used to acquire the centroid coordinates of the corresponding boundary point cloud based on the boundary feature; A third acquisition module is used to acquire a panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel; The fitting module is used to perform plane fitting on each of the panel point clouds to obtain a normal vector corresponding to each of the photovoltaic panels.

9. A readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for obtaining the normal vector of a photovoltaic panel described in any one of claims 1 to 7 are implemented.

10. A screw locking robot, characterized in that: The screw locking robot includes a controller, a mechanical arm, and a screw locking device, a laser radar and a camera installed on the mechanical arm. The controller obtains the normal vector of the photovoltaic panel based on the normal vector acquisition method of the photovoltaic panel described in any one of claims 1 to 8, and controls the screw locking device to perform a screw locking operation on the photovoltaic panel based on the normal vector.

Images