Normal vector acquisition method and device for photovoltaic panel, medium and screw locking robot

Abstract

The application relates to a normal vector acquisition method and device of a photovoltaic panel, a medium and a screw locking robot, the method comprising the following steps: acquiring array point clouds and array images of the photovoltaic array which are superimposed and displayed in the same coordinate system, wherein the array point clouds and the array images comprise a plurality of photovoltaic panels; performing image recognition on the array images to acquire boundary features of the plurality of photovoltaic panels; acquiring the centroid coordinates of corresponding boundary point clouds based on the boundary features; acquiring panel point clouds corresponding to each photovoltaic panel in the array point clouds based on the centroid coordinates of the boundary point clouds and the size of the photovoltaic panel acquired in advance; and performing plane fitting on each panel point cloud to obtain the normal vector corresponding to each photovoltaic panel, thereby solving the problem that the locking direction of the screw is incorrect, resulting in low photovoltaic array installation quality.

Description

Technical Field

[0001] This application relates to the field of photovoltaic equipment technology, and in particular to a method, device, medium, and screw-locking robot for obtaining the normal vector of a photovoltaic panel. Background Technology

[0002] During the construction of photovoltaic power plants, a large number of photovoltaic arrays need to be fixed to pre-designed positions using screws. Currently, this is generally done manually by installing photovoltaic modules and tightening screws. However, the screw-tightening process is highly repetitive, leading to human fatigue and potential for improper tightening. Large-scale photovoltaic power plants are now piloting the use of robotic arms or mobile photovoltaic power plant construction robots that utilize visual positioning technology to assist in the installation of photovoltaic arrays. During installation, the normal vector of the photovoltaic panel is typically obtained through visual positioning, and the screw-tightening device is controlled to tighten the screws in the direction of this normal vector. However, due to the inherent errors in visual positioning, when the robot tightens the screws in the direction of the identified normal vector, deviations in the screw-tightening direction may occur, leading to installation failure, poor panel fixing quality, or even damage to the photovoltaic panels. These issues affect the installation quality and efficiency of the photovoltaic array.

[0003] There is currently no effective solution to the problem of low installation quality of photovoltaic arrays caused by incorrect screw tightening direction in related technologies. Summary of the Invention

[0004] This embodiment provides a method, device, medium, and screw-tightening robot for obtaining the normal vector of a photovoltaic panel, in order to solve the problem of low installation quality of photovoltaic arrays caused by incorrect screw tightening direction in related technologies.

[0005] Firstly, this embodiment provides a method for obtaining the normal vector of a photovoltaic panel, wherein multiple photovoltaic panels are arranged sequentially to form a photovoltaic array, and the method includes:

[0006] Obtain the array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system, wherein the array point cloud and the array image include multiple photovoltaic panels;

[0007] Image recognition is performed on the array image to obtain the boundary features of the multiple photovoltaic panels;

[0008] Based on the boundary features, the centroid coordinates of the corresponding boundary point cloud are obtained;

[0009] Based on the centroid coordinates of the boundary point cloud and the pre-acquired dimensions of the photovoltaic panels, obtain the panel point cloud corresponding to each photovoltaic panel in the array point cloud;

[0010] Planar fitting is performed on the point cloud of each panel to obtain the normal vector corresponding to each photovoltaic panel.

[0011] In some embodiments, performing image recognition on the array image to obtain the boundary features of the plurality of photovoltaic panels includes:

[0012] The grayscale image of the array image is acquired and the image is segmented to obtain the gap region between the multiple photovoltaic panels;

[0013] Obtain the connected components of the segmented grayscale image to obtain the regions corresponding to the multiple photovoltaic panels;

[0014] Based on the regions corresponding to the plurality of photovoltaic panels and the gap regions, the boundary lines of the plurality of photovoltaic panels are generated in the array image.

[0015] In some embodiments, obtaining a grayscale image of the array image and performing image segmentation to obtain the gap region between the plurality of photovoltaic panels includes:

[0016] The pixel values ​​of the grayscale image are counted to obtain the corresponding grayscale histogram;

[0017] Based on the maximum pixel value in the grayscale histogram, the grayscale image is segmented to obtain the gap region between the multiple photovoltaic panels.

[0018] In some embodiments, obtaining the centroid coordinates of the corresponding boundary point cloud based on the boundary features includes:

[0019] The point cloud data with the boundary features in the array point cloud are colored to obtain the boundary point cloud;

[0020] Based on the coordinates of each point cloud data in the boundary point cloud, the centroid coordinates of the boundary point cloud are calculated.

[0021] In some embodiments, obtaining the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired dimensions of the photovoltaic panels includes:

[0022] Based on the centroid coordinates and the first preset threshold, filter out point cloud data in the array point cloud that are located in the plane where the photovoltaic panel is located but are outside the first preset threshold.

[0023] Based on the width of the photovoltaic panel, the point cloud data corresponding to the photovoltaic panel that is not adjacent to the photovoltaic panel in the array point cloud is filtered out to obtain the panel point cloud corresponding to the photovoltaic panel.

[0024] In some embodiments, the step of performing planar fitting on the point clouds of each panel to obtain the normal vector corresponding to each photovoltaic panel includes:

[0025] Multiple plane fitting operations are performed based on the point cloud data in the panel point cloud to obtain multiple candidate plane equations.

[0026] Obtain the number of point cloud data in the panel point cloud that satisfy each of the candidate plane equations;

[0027] Based on the candidate plane equation corresponding to the maximum number of point cloud data, the plane equation of the photovoltaic panel is determined, and then the normal vector corresponding to the photovoltaic panel is obtained.

[0028] In some embodiments, the method is applied to a screw-locking robot, which includes a robotic arm and a lidar and camera mounted on the robotic arm. Acquiring the array point cloud and array image of the photovoltaic array superimposed in the same coordinate system includes:

[0029] The array point cloud is acquired based on the lidar, and the array image is acquired based on the camera;

[0030] Based on the coordinate transformation equation between the lidar and the camera, the array point cloud is transformed to the camera coordinate system corresponding to the camera, and then superimposed on the array image for display.

[0031] Secondly, this embodiment provides a device for obtaining the normal vector of a photovoltaic panel, wherein multiple photovoltaic panels are arranged sequentially to form a photovoltaic array, and the device includes:

[0032] The first acquisition module is used to acquire the array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system, wherein the array point cloud and array image include multiple photovoltaic panels;

[0033] The recognition module is used to perform image recognition on the array image and obtain the boundary features of the multiple photovoltaic panels;

[0034] The second acquisition module is used to acquire the centroid coordinates of the corresponding boundary point cloud based on the boundary features;

[0035] The third acquisition module is used to acquire the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel;

[0036] The fitting module is used to perform planar fitting on the point cloud of each panel to obtain the normal vector corresponding to each photovoltaic panel.

[0037] Thirdly, this embodiment provides a readable storage medium storing a program that, when executed by a processor, implements the steps of the photovoltaic panel normal vector acquisition method described in the first aspect.

[0038] Fourthly, this embodiment provides a screw-locking robot, which includes a controller, a robotic arm, and a screw-locking device, a lidar, and a camera mounted on the robotic arm.

[0039] The controller obtains the normal vector of the photovoltaic panel based on the normal vector acquisition method of the photovoltaic panel described in either the first or second aspect, and controls the screw tightening device to perform screw tightening operation on the photovoltaic panel based on the normal vector.

[0040] Compared with related technologies, the photovoltaic panel normal vector acquisition method provided in this embodiment acquires the array point cloud and array image of the photovoltaic array superimposed in the same coordinate system, converting the point cloud data generated by the lidar and the image data generated by the camera into the same coordinate system, which facilitates subsequent data processing based on the position coordinates of each feature point on the photovoltaic panel; by performing image recognition on the array image, the boundary features of multiple photovoltaic panels are obtained, providing necessary reference data for the identification of photovoltaic panels; by obtaining the centroid coordinates of the corresponding boundary point cloud based on the boundary feature, the image features are converted into point cloud features; by obtaining the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel, the point cloud data of each photovoltaic panel is identified from the array point cloud; by performing plane fitting on each panel point cloud, the normal vector corresponding to each photovoltaic panel is obtained, ensuring the correctness of the screw tightening direction and solving the problem of low installation quality of photovoltaic array caused by incorrect screw tightening direction.

[0041] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0043] Figure 1 This is a computer hardware structure block diagram of a method for obtaining the normal vector of a photovoltaic panel according to some embodiments of this application;

[0044] Figure 2 This is a flowchart of a method for obtaining the normal vector of a photovoltaic panel according to some embodiments of this application;

[0045] Figure 3 This is a flowchart illustrating the process of obtaining the boundary features of a photovoltaic panel according to some embodiments of this application;

[0046] Figure 4 This is a flowchart illustrating the process of obtaining the gap region of a photovoltaic panel according to some embodiments of this application;

[0047] Figure 5 This is a flowchart illustrating the process of obtaining the centroid coordinates of boundary point clouds in some embodiments of this application.

[0048] Figure 6 This is a flowchart of the panel point cloud acquisition process in some embodiments of this application;

[0049] Figure 7 This is a flowchart illustrating the process of obtaining the normal vector of a photovoltaic panel according to some embodiments of this application;

[0050] Figure 8 This is a flowchart illustrating the overlay display of array point clouds and array images according to some embodiments of this application;

[0051] Figure 9 This is a flowchart of a method for obtaining the normal vector of a photovoltaic panel according to some preferred embodiments of this application;

[0052] Figure 10 This is a structural block diagram of a photovoltaic panel normal vector acquisition device according to some embodiments of this application. Detailed Implementation

[0053] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0055] The method for obtaining the normal vector of a photovoltaic panel provided in this application embodiment can be executed in the processor of a server, computer, terminal or similar computing device. Figure 1 This is a computer hardware structure block diagram of a photovoltaic panel normal vector acquisition method according to some embodiments of this application. For example... Figure 1 As shown, a computer may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a CPU, a microprocessor (MCU), or a programmable logic device (FPGA). The computer may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer described above. For example, the computer may also include components that are larger than... Figure 1 The array shown is more or less, or has the same as Figure 1 The different configurations shown are illustrated.

[0056] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the photovoltaic panel normal vector acquisition method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the aforementioned photovoltaic panel normal vector acquisition method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some embodiments, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0057] The transmission device 106 is used to receive or send data via a network. This network includes wireless networks provided by a telecommunications provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0058] This embodiment provides a method for obtaining the normal vector of a photovoltaic panel. Figure 2 This is a flowchart of a method for obtaining the normal vector of a photovoltaic panel according to some embodiments of this application, such as... Figure 2 As shown, the process includes the following steps:

[0059] Step S201: Obtain the array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system, wherein the array point cloud and the array image include multiple photovoltaic panels.

[0060] A photovoltaic (PV) array consists of multiple photovoltaic panels arranged sequentially, each with its own installation orientation and tilt angle, which may be the same or different. An array point cloud refers to the set of point cloud data in three-dimensional space corresponding to the PV array. Methods for acquiring the array point cloud include, but are not limited to, scanning the PV array using LiDAR. An array image refers to the two-dimensional image data corresponding to the PV array, which can be obtained by capturing images of the PV array using image acquisition devices such as cameras.

[0061] In some embodiments, the array point cloud and the array image are generated based on different coordinate systems. For example, the array point cloud is generated based on the lidar coordinate system, while the array image is generated based on the camera coordinate system. The array point cloud and the array image can be unified into the same coordinate system through coordinate transformation equations between the two coordinate systems. The array point cloud and the array image after coordinate system unification can be superimposed on a display device, and the position coordinates of the same photovoltaic panel in the photovoltaic array are the same in both the array point cloud and the array image.

[0062] Step S202: Perform image recognition on the array image to obtain the boundary features of multiple photovoltaic panels.

[0063] Image recognition is used to obtain the boundary features of each photovoltaic panel in the array image. Boundary features can be boundary regions, boundary points, or boundary lines of the photovoltaic panels. There are many ways to obtain image features through image recognition, and this embodiment does not impose any limitations.

[0064] Step S203: Based on the boundary feature, obtain the centroid coordinates of the corresponding boundary point cloud.

[0065] Based on the boundary features of each photovoltaic panel in the array image, the boundary point cloud corresponding to the boundary feature is obtained, and the centroid coordinates of the boundary point cloud are further obtained. The boundary point cloud can be composed of point cloud data coinciding with the location of the boundary feature. The centroid coordinates can be calculated based on the position coordinates of all point cloud data in the boundary point cloud.

[0066] Step S204: Based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel, obtain the panel point cloud corresponding to each photovoltaic panel in the array point cloud.

[0067] Since the point cloud data in the boundary point cloud coincides with the boundary feature position of the photovoltaic panel, the centroid coordinates of the boundary point cloud can be used as the boundary point coordinates of the photovoltaic panel. Then, based on the pre-obtained size of the photovoltaic panel, the coordinate range of the photovoltaic panel is determined. The point cloud data within the coordinate range is assigned to the panel point cloud corresponding to the photovoltaic panel, and other point cloud data that do not belong to the coordinate range are filtered out, thereby obtaining the panel point cloud corresponding to each photovoltaic panel.

[0068] Step S205: Perform plane fitting on the point cloud of each panel to obtain the normal vector corresponding to each photovoltaic panel.

[0069] For each photovoltaic panel, a plane fit is performed on the panel point cloud. The specific fitting method is not limited. The plane equation corresponding to each panel point cloud is obtained. The normal vector corresponding to the photovoltaic panel is calculated based on the plane equation.

[0070] Through steps S201 to S205, the array point cloud and array image of the photovoltaic array, which are superimposed and displayed in the same coordinate system, are acquired. The point cloud data generated by the lidar and the image data generated by the camera are converted into the same coordinate system, which facilitates subsequent data processing based on the position coordinates of each feature point on the photovoltaic panel. By performing image recognition on the array image, the boundary features of multiple photovoltaic panels are obtained, providing necessary reference data for the identification of photovoltaic panels. Based on the boundary features, the centroid coordinates of the corresponding boundary point cloud are obtained, and the image features are converted into point cloud features. Based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel, the panel point cloud corresponding to each photovoltaic panel in the array point cloud is obtained, and the point cloud data of each photovoltaic panel is identified from the array point cloud. By performing plane fitting on each panel point cloud, the normal vector corresponding to each photovoltaic panel is obtained, ensuring the correctness of the screw tightening direction and solving the problem of low installation quality of photovoltaic array caused by incorrect screw tightening direction.

[0071] In some embodiments, Figure 3 This is a flowchart illustrating the process of obtaining boundary features of a photovoltaic panel according to some embodiments of this application, such as... Figure 3 As shown, the process includes the following steps:

[0072] Step S301: Obtain the grayscale image of the array image and perform image segmentation to obtain the gap area between multiple photovoltaic panels.

[0073] The array image can be an RGB format image. Grayscale processing is performed on the array image to obtain a grayscale image. Since the brightness of the gap area between the photovoltaic panels differs from the brightness of the photovoltaic panel area, the grayscale image can be segmented based on the pixel values ​​of each pixel to obtain the gap area between each photovoltaic panel.

[0074] Step S302: Obtain the connected components of the segmented grayscale image to obtain the regions corresponding to multiple photovoltaic panels.

[0075] Based on the location of the gap regions in the grayscale image, connected component analysis is performed on the grayscale image. Specifically, the grayscale image can be binarized to distinguish between gap regions and non-gap regions, and connected component analysis is performed on the binarized image to obtain the image region corresponding to each photovoltaic panel. The algorithm for connected component analysis will not be described in detail in this embodiment.

[0076] Furthermore, before performing connected component analysis, irrelevant noise in the image can be eliminated through morphological opening operations.

[0077] Step S303: Based on the regions corresponding to the multiple photovoltaic panels and the gap region, generate the boundary lines of the multiple photovoltaic panels in the array image.

[0078] Based on the image regions corresponding to the photovoltaic panels and the gap areas, the boundary lines of each photovoltaic panel are generated in the array image. Specifically, the Hough line detection algorithm can be used to draw the boundaries of the gap areas, i.e., the boundary lines of the photovoltaic panels, in the array image.

[0079] Through steps S301 to S303, by acquiring the grayscale image of the array image and performing image segmentation, the gap region between multiple photovoltaic panels is obtained, and the position of the gap between the photovoltaic panels in the array image is obtained; by acquiring the connected components of the segmented grayscale image, the region corresponding to the multiple photovoltaic panels is obtained, and the position of the photovoltaic panels in the array image is obtained; by generating the boundary lines of multiple photovoltaic panels in the array image based on the region corresponding to the multiple photovoltaic panels and the gap region, the boundary features of each photovoltaic panel in the array image are obtained, and necessary reference data is obtained for obtaining the point cloud corresponding to each photovoltaic panel.

[0080] In some embodiments, Figure 4 This is a flowchart illustrating the process of obtaining the gap region of a photovoltaic panel according to some embodiments of this application, such as... Figure 4 As shown, the process includes the following steps:

[0081] Step S401: Count the pixel values ​​of the grayscale image to obtain the corresponding grayscale histogram.

[0082] Specifically, the horizontal axis of the grayscale histogram represents the pixel value in the grayscale image, and the vertical axis represents the number of times that pixel value appears.

[0083] Step S402: Based on the maximum pixel value in the grayscale histogram, perform image segmentation on the grayscale image to obtain the gap region between multiple photovoltaic panels.

[0084] Since light enters through the gaps between photovoltaic panels, the gap regions can be identified by the maximum pixel value in the grayscale image. Specifically, the `find_peaks` function in SciPy can be used to find the peak with the largest pixel value in the histogram peaks, and the pixel value corresponding to this peak can be used as the input value for an image thresholding algorithm to segment the brightest gap regions between the photovoltaic panels.

[0085] Through steps S401 to S402, the corresponding grayscale histogram is obtained by statistically analyzing the pixel values ​​of the grayscale image. Based on the maximum pixel value in the grayscale histogram, the grayscale image is segmented to obtain the gap region between multiple photovoltaic panels. This allows the position of the gap between the photovoltaic panels in the array image to be obtained, providing an accurate position coordinate reference for the subsequent identification of the photovoltaic panels.

[0086] In some embodiments, Figure 5 This is a flowchart illustrating the process of obtaining the centroid coordinates of boundary point clouds in some embodiments of this application, such as... Figure 5As shown, the process includes the following steps:

[0087] Step S501: Color the point cloud data with boundary features in the array point cloud to obtain the boundary point cloud.

[0088] Specifically, the boundary feature can be the boundary region, boundary point, or boundary line of the photovoltaic panel. In one embodiment, the boundary feature is the boundary line of the photovoltaic panel. The point cloud data that coincides with the position coordinates of this boundary line is colored to distinguish it from other point cloud data, thus obtaining the boundary point cloud.

[0089] Step S502: Calculate the centroid coordinates of the boundary point cloud based on the coordinates of each point cloud data in the boundary point cloud.

[0090] The centroid coordinates of the boundary point cloud are calculated based on the coordinates of each point cloud data point. Specifically, the average coordinates of each point cloud data point in the X, Y, and Z directions can be calculated as the centroid coordinates (x, y, z) of the boundary point cloud.

[0091] Through steps S501 to S502, the boundary point cloud is obtained by coloring the point cloud data with boundary features in the array point cloud. Based on the coordinates of each point cloud data in the boundary point cloud, the centroid coordinates of the boundary point cloud are calculated, and the accurate coordinates of the boundary points of the photovoltaic panel are obtained, which improves the accuracy of obtaining the position of the panel point cloud corresponding to the photovoltaic panel in the subsequent process.

[0092] In some embodiments, Figure 6 This is a flowchart of the panel point cloud acquisition process in some embodiments of this application, such as... Figure 6 As shown, the process includes the following steps:

[0093] Step S601: Based on the centroid coordinates and the first preset threshold, filter out point cloud data in the array point cloud that are outside the first preset threshold located on the plane where the photovoltaic panel is located.

[0094] Because the photovoltaic panel is relatively thin, and the point clouds on its upper and lower surfaces are separated from the point clouds of the environment, a direct-pass filtering algorithm can be used to filter out point cloud data outside the first preset threshold of the thickness of the plane containing the photovoltaic panel, based on a pre-set thickness threshold of the photovoltaic panel. In a specific embodiment, the Z-axis direction in the three-dimensional coordinate system is set as the thickness direction of the photovoltaic panel, and point cloud data outside the range [za, z+a] are filtered out. Here, z is the centroid coordinate of the Z-axis, and a is the first preset threshold.

[0095] Step S602: Based on the width of the photovoltaic panel, filter out the point cloud data corresponding to the photovoltaic panel that is not adjacent to the photovoltaic panel in the array point cloud, except for the second preset threshold, to obtain the panel point cloud corresponding to the photovoltaic panel.

[0096] When the array point cloud contains multiple photovoltaic panels arranged in rows, a pass-through filtering algorithm can be used to filter out point cloud data corresponding to other photovoltaic panels adjacent to the photovoltaic panel, based on the boundary line of the photovoltaic panel in the width direction and the pre-acquired width value of the photovoltaic panel. In a specific embodiment, assuming that the boundary line of the boundary point cloud is the left boundary line of the photovoltaic panel, then point cloud data outside the range [x-b1, x-b2] is filtered out according to the centroid coordinates of the boundary point cloud and the width of the photovoltaic panel, where x is the centroid coordinate of the X-axis and b1 and b2 are preset second preset thresholds in the width direction.

[0097] Similarly, when the array point cloud contains multiple photovoltaic panels arranged in columns, the point cloud data corresponding to other photovoltaic panels adjacent to the photovoltaic panel can be filtered out based on the boundary line of the photovoltaic panel in the length direction and the pre-acquired length value of the photovoltaic panel.

[0098] After filtering the point clouds in different directions as described above, the panel point cloud corresponding to the photovoltaic panel can be obtained.

[0099] Through steps S601 to S602, point cloud data located outside the plane of the photovoltaic panel in the array point cloud are filtered out based on the centroid coordinates and a first preset threshold. Point cloud data corresponding to the photovoltaic panel that is adjacent to the photovoltaic panel but outside the second preset threshold in the array point cloud are filtered out based on the width of the photovoltaic panel. The panel point cloud corresponding to the photovoltaic panel is obtained. The point cloud data of each photovoltaic panel is identified from the array point cloud and used for the calculation of the normal vector corresponding to each photovoltaic panel, thereby improving the accuracy of obtaining the normal vector of each photovoltaic panel.

[0100] In some embodiments, Figure 7 This is a flowchart illustrating the process of obtaining the normal vector of a photovoltaic panel according to some embodiments of this application, such as... Figure 7 As shown, the process includes the following steps:

[0101] Step S701: Perform multiple plane fitting operations based on the point cloud data in the panel point cloud to obtain multiple candidate plane equations.

[0102] Specifically, the point cloud plane of each photovoltaic panel can be fitted using the Open3D point cloud segmentation function `pcd.segment_plane()`. This function can randomly collect point clouds with given parameters within a set range and perform plane fitting. The number of fitting iterations is also given as a parameter; for example, three points can be randomly selected each time to fit the plane within a certain range. In this way, multiple fitting iterations can yield multiple candidate plane equations.

[0103] Step S702: Obtain the number of point cloud data that satisfy the equations of each candidate plane in the point cloud of the panel.

[0104] After obtaining the candidate plane equation, calculate the number of point cloud data in the panel point cloud that satisfies the candidate plane equation, that is, the number of point cloud data located on the plane corresponding to the candidate plane equation.

[0105] Step S703: Based on the candidate plane equation corresponding to the maximum number of point cloud data, determine the plane equation of the photovoltaic panel, and then obtain the normal vector corresponding to the photovoltaic panel.

[0106] The candidate plane equation with the largest number of point cloud data is selected as the final plane equation, and the corresponding normal vector is calculated based on the plane equation.

[0107] Through steps S701 to S703, multiple plane fittings are performed based on the point cloud data in the panel point cloud to obtain multiple candidate plane equations. The number of point cloud data in the panel point cloud that satisfies each candidate plane equation is obtained. Based on the candidate plane equation corresponding to the maximum number of point cloud data, the plane equation of the photovoltaic panel is determined, and then the normal vector corresponding to the photovoltaic panel is obtained, which improves the accuracy of the photovoltaic panel normal vector calculation.

[0108] In some embodiments, Figure 8 This is a flowchart illustrating the overlay display of array point clouds and array images according to some embodiments of this application, such as... Figure 8 As shown, the process includes the following steps:

[0109] Step S801: Obtain array point cloud based on lidar and array image based on camera.

[0110] In this embodiment, the lidar and camera can be installed in the environment near the photovoltaic panel or on the robotic arm of a screw-locking robot. The lidar and camera acquire the array point cloud and array image of the photovoltaic array based on different coordinate systems.

[0111] Step S802: Based on the coordinate transformation equation between the lidar and the camera, the array point cloud is transformed to the camera coordinate system corresponding to the camera, and then superimposed on the array image for display.

[0112] Before acquiring the array point cloud and array image, the coordinate transformation equation between the camera and the LiDAR has been obtained through joint calibration of the camera and LiDAR. This embodiment will not elaborate on the joint calibration steps. In one specific embodiment, the relationship between the camera coordinate system and the LiDAR coordinate system can be obtained by calling the corresponding function, and then the transformation matrix between the two coordinate systems can be calculated. Based on this transformation matrix, the array point cloud is transformed to the camera coordinate system and overlaid with the array image for display.

[0113] Through steps S801 to S802, array point clouds are acquired based on lidar, array images are acquired based on cameras, and position data of different forms of the same photovoltaic array are collected. The array point clouds are transformed into the camera coordinate system corresponding to the camera based on the coordinate transformation equation between lidar and camera, and then superimposed on the array image for display, providing accurate reference data for subsequent data processing of array images and array point clouds.

[0114] The present embodiment will now be described and illustrated through preferred embodiments. Figure 9 This is a flowchart of a method for obtaining the normal vector of a photovoltaic panel according to some preferred embodiments of this application, such as... Figure 9 As shown, the process includes the following steps:

[0115] Step S901: Use an industrial camera to acquire an RGB array image of the photovoltaic array and read the array image into the image processing program of the industrial control computer;

[0116] Step S902: Convert the array image to grayscale and obtain the grayscale histogram of the image.

[0117] Step S903: Use SciPy's find_peaks function to find the peak with the largest pixel value in the histogram peaks, and use the pixel value corresponding to the peak as the input of the image thresholding algorithm to segment the brightest gap area between photovoltaic panels.

[0118] Step S904: Morphological opening operation is used to eliminate irrelevant noise in the image, and connected components in the image are calculated to retain the pixel region with the largest connected component area in the image.

[0119] Step S905: Use the Hough line detection algorithm to draw the boundary of the gap in the array image, that is, the boundary features between components;

[0120] Step S906: The relationship between the camera coordinate system and the lidar coordinate system can be obtained through the tf listening function of the tf2_ros library, and the transformation matrix between the two coordinate systems can be calculated.

[0121] Step S907: Convert the array point cloud acquired by the lidar to the camera coordinate system according to the transformation matrix, and overlay it with the array image for display.

[0122] The order of steps S901 to S905 and steps S906 to S907 can be interchanged.

[0123] Step S908: Color the point cloud data with boundary features in the array point cloud to obtain the boundary point cloud;

[0124] Step S909: Calculate the centroid coordinates of the boundary point cloud based on the coordinates of each point cloud data in the boundary point cloud;

[0125] Step S910: Based on the centroid coordinates and the first preset threshold, filter out point cloud data located outside the thickness of the plane where the photovoltaic panel is located in the array point cloud;

[0126] Step S911: Based on the width of the photovoltaic panel, filter out the point cloud data corresponding to the photovoltaic panel that is not adjacent to the second preset threshold in the array point cloud to obtain the panel point cloud corresponding to the photovoltaic panel.

[0127] Step S912: The point cloud segmentation function pcd.segment_plane() of Open3D is used to fit the point cloud plane of each photovoltaic panel to obtain the corresponding multiple candidate plane equations.

[0128] Step S913: Obtain the number of point cloud data that satisfy the equations of each candidate plane in the panel point cloud;

[0129] Step S914: Based on the candidate plane equation corresponding to the maximum number of point cloud data, determine the plane equation of the photovoltaic panel, and then obtain the normal vector corresponding to the photovoltaic panel.

[0130] In steps S901-S914, the array image of the photovoltaic array is converted to grayscale. The maximum pixel value in the grayscale image is used as input to the image thresholding algorithm to segment the brightest gaps between photovoltaic panels. Boundary features of the photovoltaic panels are obtained through connected component analysis, providing necessary reference data for photovoltaic panel identification. By converting the point cloud data generated by the LiDAR and the image data generated by the camera into the same coordinate system, subsequent data processing based on the position coordinates of each feature point on the photovoltaic panel is facilitated. The specific location of the boundary features of the photovoltaic panel is obtained by calculating the centroid coordinates. Through the pass-through filtering algorithm and the dimensional parameters of the photovoltaic panel in the length, width, and thickness directions, as well as the preset threshold, the panel point cloud corresponding to the photovoltaic panel is obtained, accurately acquiring the position data of the photovoltaic panel surface. By performing plane fitting on each panel point cloud, the normal vector corresponding to each photovoltaic panel is obtained, ensuring the correctness of the screw tightening direction and solving the problem of low installation quality of the photovoltaic array caused by incorrect screw tightening direction.

[0131] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0132] In some embodiments, this application also provides a photovoltaic panel normal vector acquisition device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated for details already described. The terms "module," "unit," "subunit," etc., used below can refer to a combination of software and / or hardware that implements a predetermined function. In some embodiments, Figure 10 This is a structural block diagram of the photovoltaic panel normal vector acquisition device in this embodiment, as shown below. Figure 10 As shown, the device includes:

[0133] The first acquisition module 1001 is used to acquire the array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system, wherein the array point cloud and the array image include multiple photovoltaic panels.

[0134] The recognition module 1002 is used to perform image recognition on the array image and obtain the boundary features of multiple photovoltaic panels;

[0135] The second acquisition module 1003 is used to acquire the centroid coordinates of the corresponding boundary point cloud based on the boundary features;

[0136] The third acquisition module 1004 is used to acquire the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel.

[0137] The fitting module 1005 is used to perform planar fitting on the point cloud of each panel to obtain the normal vector corresponding to each photovoltaic panel.

[0138] The photovoltaic panel normal vector acquisition device of this embodiment acquires the array point cloud and array image of the photovoltaic array superimposed in the same coordinate system through the first acquisition module 1001, converting the point cloud data generated by the lidar and the image data generated by the camera into the same coordinate system, which facilitates subsequent data processing based on the position coordinates of each feature point on the photovoltaic panel; the recognition module 1002 performs image recognition on the array image to acquire the boundary features of multiple photovoltaic panels, providing necessary reference data for the recognition of photovoltaic panels; the second acquisition module 1003 acquires the centroid coordinates of the corresponding boundary point cloud based on the boundary features, converting the image features into point cloud features; the third acquisition module 1004 acquires the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel, identifying the point cloud data of each photovoltaic panel from the array point cloud; the fitting module 1005 performs planar fitting on each panel point cloud to obtain the normal vector corresponding to each photovoltaic panel, ensuring the correctness of the screw tightening direction and solving the problem of low installation quality of photovoltaic array caused by incorrect screw tightening direction.

[0139] Furthermore, in conjunction with the photovoltaic panel normal vector acquisition method provided in the above embodiments, this embodiment can also provide a readable storage medium for implementation. This readable storage medium stores a program; when executed by a processor, the program implements any of the photovoltaic panel normal vector acquisition methods described in the above embodiments.

[0140] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0141] In some embodiments, this application also provides a screw-locking robot, which includes a controller, a robotic arm, and a screw-locking device, a lidar, and a camera mounted on the robotic arm. The controller obtains the normal vector of the photovoltaic panel based on the photovoltaic panel normal vector acquisition method of the above embodiments, and controls the screw-locking device to perform screw-locking operations on the photovoltaic panel based on the normal vector.

[0142] The screw-tightening robot in this embodiment acquires the array point cloud and array image corresponding to the photovoltaic panel through LiDAR and camera, respectively, and collects position data of different forms of the same photovoltaic array; the controller obtains the normal vector of the photovoltaic panel, ensuring the correctness of the screw tightening direction and improving the installation quality and efficiency of the photovoltaic panel.

[0143] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0144] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0145] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0146] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0147] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for obtaining the normal vector of a photovoltaic panel, wherein multiple photovoltaic panels are arranged sequentially to form a photovoltaic array, characterized in that, The method includes: Acquire array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system, wherein the array point cloud and the array image include multiple photovoltaic panels; Image recognition is performed on the array image to obtain the boundary features of the multiple photovoltaic panels; Based on the boundary features, the centroid coordinates of the corresponding boundary point cloud are obtained; Based on the centroid coordinates of the boundary point cloud and the pre-acquired dimensions of the photovoltaic panels, obtain the panel point cloud corresponding to each photovoltaic panel in the array point cloud; Planar fitting is performed on the point cloud of each panel to obtain the normal vector corresponding to each photovoltaic panel.

2. The method according to claim 1, characterized in that, The step of performing image recognition on the array image to obtain the boundary features of the multiple photovoltaic panels includes: The grayscale image of the array image is acquired and the image is segmented to obtain the gap region between the multiple photovoltaic panels; Obtain the connected components of the segmented grayscale image to obtain the regions corresponding to the multiple photovoltaic panels; Based on the regions corresponding to the plurality of photovoltaic panels and the gap regions, the boundary lines of the plurality of photovoltaic panels are generated in the array image.

3. The method according to claim 2, characterized in that, The step of acquiring the grayscale image of the array image and performing image segmentation to obtain the gap region between the multiple photovoltaic panels includes: The pixel values ​​of the grayscale image are counted to obtain the corresponding grayscale histogram; Based on the maximum pixel value in the grayscale histogram, the grayscale image is segmented to obtain the gap region between the multiple photovoltaic panels.

4. The method according to claim 1, characterized in that, The step of obtaining the centroid coordinates of the corresponding boundary point cloud based on the boundary features includes: The point cloud data with the boundary features in the array point cloud are colored to obtain the boundary point cloud; Based on the coordinates of each point cloud data in the boundary point cloud, the centroid coordinates of the boundary point cloud are calculated.

5. The method according to claim 1, characterized in that, The step of obtaining the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired dimensions of the photovoltaic panels includes: Based on the centroid coordinates and the first preset threshold, filter out point cloud data in the array point cloud that are located in the plane where the photovoltaic panel is located but are outside the first preset threshold. Based on the width of the photovoltaic panel, the point cloud data corresponding to the photovoltaic panel that is not adjacent to the photovoltaic panel in the array point cloud is filtered out to obtain the panel point cloud corresponding to the photovoltaic panel.

6. The method according to claim 1, characterized in that, The step of performing planar fitting on the point clouds of each panel to obtain the normal vector corresponding to each photovoltaic panel includes: Multiple plane fitting operations are performed based on the point cloud data in the panel point cloud to obtain multiple candidate plane equations. Obtain the number of point cloud data in the panel point cloud that satisfy each of the candidate plane equations; Based on the candidate plane equation corresponding to the maximum number of point cloud data, the plane equation of the photovoltaic panel is determined, and then the normal vector corresponding to the photovoltaic panel is obtained.

7. The method according to claim 1, characterized in that, The method is applied to a screw-locking robot, which includes a robotic arm and a lidar and camera mounted on the robotic arm. The acquisition of the array point cloud and array image of the photovoltaic array, superimposed and displayed in the same coordinate system, includes: The array point cloud is acquired based on the lidar, and the array image is acquired based on the camera; Based on the coordinate transformation equation between the lidar and the camera, the array point cloud is transformed to the camera coordinate system corresponding to the camera, and then superimposed on the array image for display.

8. A device for obtaining the normal vector of a photovoltaic panel, wherein multiple photovoltaic panels are arranged sequentially to form a photovoltaic array, characterized in that, The device includes: The first acquisition module is used to acquire the array point cloud and array image of the photovoltaic array superimposed and displayed in the same coordinate system, wherein the array point cloud and array image include multiple photovoltaic panels. The recognition module is used to perform image recognition on the array image and obtain the boundary features of the multiple photovoltaic panels; The second acquisition module is used to acquire the centroid coordinates of the corresponding boundary point cloud based on the boundary features; The third acquisition module is used to acquire the panel point cloud corresponding to each photovoltaic panel in the array point cloud based on the centroid coordinates of the boundary point cloud and the pre-acquired size of the photovoltaic panel; The fitting module is used to perform planar fitting on the point cloud of each panel to obtain the normal vector corresponding to each photovoltaic panel.

9. A readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method for obtaining the normal vector of the photovoltaic panel as described in any one of claims 1 to 7.

10. A screw-locking robot, characterized in that, The screw-locking robot includes a controller, a robotic arm, and a screw-locking device, a lidar, and a camera mounted on the robotic arm. The controller obtains the normal vector of the photovoltaic panel based on the normal vector acquisition method of the photovoltaic panel according to any one of claims 1 to 7, and controls the screw tightening device to perform screw tightening operation on the photovoltaic panel based on the normal vector.

Images