Pose detection method and device of photovoltaic module, robot and storage medium
By extracting features and modeling context from binocular images, and combining them with an attention mechanism for weighted fusion, a high-precision disparity map is generated and point cloud data is constructed. This solves the problems of low disparity estimation accuracy and high cost in photovoltaic module pose detection, and achieves efficient pose detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUNPURE TECH CO LTD
- Filing Date
- 2025-12-22
- Publication Date
- 2026-05-29
Smart Images
Figure CN122115553A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of photovoltaic module testing, and in particular relates to a method, device, robot and storage medium for photovoltaic module pose detection. Background Technology
[0002] In photovoltaic (PV) module installation pose detection scenarios, relevant technologies mainly employ Aruco codes, targets, or other manually placed guides to establish the geometric relationship between the module and the camera. Some research has also attempted to directly acquire point clouds using 3D cameras to obtain module surface information. While these two methods can achieve a certain level of pose detection accuracy under ideal conditions, they suffer from high labor and equipment costs and are limited by lighting conditions, PV module reflection characteristics, and acquisition range, making them difficult to promote and apply in large-scale PV power plant scenarios. Among related technologies, the geometric matching method based on binocular cameras reduces the accuracy of parallax estimation when the PV module surface has sparse texture, strong reflection, or truncated boundaries, resulting in poor point cloud reconstruction quality and affecting the accuracy of pose detection results. Summary of the Invention
[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method, apparatus, robot, and storage medium for photovoltaic module pose detection, which reduces local mismatches in disparity estimation, improves the integrity and accuracy of the photovoltaic module point cloud representation, and thus improves the accuracy of photovoltaic module pose detection.
[0004] In a first aspect, this application provides a method for detecting the pose of a photovoltaic module, the method comprising: Feature extraction is performed on the acquired binocular images to obtain the target feature vector and the initial disparity map; Context modeling is performed on the first image in the binocular images to obtain the context information corresponding to the first image; The target feature vector, the initial disparity map, and the context information are weighted and fused to obtain the first disparity map; Based on the camera parameters corresponding to the first disparity map and the binocular image, the point cloud data corresponding to the photovoltaic module is obtained; Based on the point cloud data, the pose information of the photovoltaic module is determined.
[0005] According to the photovoltaic module pose detection method of this application, feature extraction is performed on binocular images to obtain target feature vectors and initial disparity maps. Context modeling is performed on the first image to extract global correlation information, enhancing the adaptability of disparity estimation to complex imaging conditions such as weak texture and strong reflection, and improving the reconstruction quality of photovoltaic module edges and low-texture areas. Based on this, the target feature vector, initial disparity map, and context information are weighted and fused, which can combine local features, initial geometric relationships, and context information in disparity estimation, simultaneously preserving global context and local details, and improving disparity accuracy. Through the above deep learning model, disparity estimation of photovoltaic modules and subsequent construction of point cloud data can be completed based on binocular images, reducing the hardware cost requirements of image devices, reducing local mismatches in disparity estimation, and improving the integrity and accuracy of photovoltaic module point cloud representation, thereby improving the detection accuracy of photovoltaic module pose.
[0006] According to one embodiment of this application, the step of extracting features from the acquired binocular images to obtain a target feature vector and an initial disparity map includes: Feature extraction is performed on the first image and the second image in the binocular images respectively to obtain the first feature vector corresponding to the first image and the second feature vector corresponding to the second image; Deep feature extraction is performed on the first feature vector and the second feature vector to obtain the target feature vector corresponding to the first feature vector and the target feature vector corresponding to the second feature vector; The initial disparity map is obtained based on the target feature vector.
[0007] According to one embodiment of this application, the step of weightedly fusing the target feature vector, the initial disparity map, and the context information to obtain a first disparity map includes: An attention mechanism algorithm is used to process the target feature vector, the initial disparity map, and the context information to determine the weight sequence; The target feature vector, the initial disparity map, and the context information are weighted and fused based on the weight sequence to obtain the first disparity map.
[0008] According to one embodiment of this application, determining the pose information of the photovoltaic module based on the point cloud data includes: Based on the point cloud data, obtain the normal vector of the photovoltaic module plane; The point cloud data is projected onto the first image in the binocular image to obtain the first direction vector and the second direction vector of the photovoltaic module plane; The pose of the photovoltaic module is determined based on the first direction vector, the second direction vector, and the normal vector.
[0009] According to one embodiment of this application, the step of projecting the point cloud data onto a first image in the binocular image to obtain a first direction vector and a second direction vector of the photovoltaic module plane includes: Based on the projection of the point cloud data onto the first image in the binocular image, obtain the first point cloud data and the second point cloud data corresponding to the first image; The first point cloud data and the second point cloud data are filtered for interior points to obtain the first straight line and the second straight line; Based on the angular relationship between the first straight line and the second straight line and the reference axis of the photovoltaic module plane, the first direction vector and the second direction vector corresponding to the photovoltaic module plane are determined.
[0010] According to one embodiment of this application, obtaining first point cloud data and second point cloud data corresponding to the first image based on the projection of the point cloud data onto the first image in the binocular image includes: The point cloud data is projected onto the first image to obtain first projection information; Based on the first projection information and the point cloud data, the boundary distance between the image edge of the first image and the point cloud data is obtained; If the boundary distance is less than a preset threshold, re-acquire binocular images; If the boundary distance is greater than or equal to the preset threshold, the first point cloud data and the second point cloud data corresponding to the first image are determined.
[0011] Secondly, this application provides a pose detection device for a photovoltaic module, the device comprising: The first processing module is used to extract features from the acquired binocular images to obtain the target feature vector and the initial disparity map; The second processing module is used to perform context modeling on the first image in the binocular image and obtain the context information corresponding to the first image. The third processing module is used to perform weighted fusion of the target feature vector, the initial disparity map and the context information to obtain the first disparity map; The fourth processing module is used to obtain point cloud data corresponding to the photovoltaic module based on the camera parameters corresponding to the first disparity map and the binocular image; The fifth processing module is used to determine the pose information of the photovoltaic module based on the point cloud data.
[0012] According to the photovoltaic module pose detection device of this application, feature extraction is performed on binocular images to obtain target feature vectors and initial disparity maps. Context modeling is performed on the first image to extract global correlation information, enhancing the adaptability of disparity estimation to complex imaging conditions such as weak texture and strong reflection, and improving the reconstruction quality of photovoltaic module edges and low-texture areas. Based on this, the target feature vector, initial disparity map, and context information are weighted and fused, which can combine local features, initial geometric relationships, and context information in disparity estimation, simultaneously preserving global context and local details, and improving disparity accuracy. Through the above deep learning model, disparity estimation of photovoltaic modules and subsequent construction of point cloud data can be completed based on binocular images, reducing the hardware cost requirements of image devices, reducing local mismatches in disparity estimation, and improving the integrity and accuracy of photovoltaic module point cloud representation, thereby improving the detection accuracy of photovoltaic module pose.
[0013] Thirdly, this application provides a robot, such as the photovoltaic module pose detection device described in the second aspect above.
[0014] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pose detection method for photovoltaic modules as described in the first aspect above.
[0015] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the photovoltaic module pose detection method as described in the first aspect above.
[0016] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects: By extracting features from binocular images to obtain target feature vectors and initial disparity maps, and performing context modeling on the first image to extract global correlation information, the adaptability of disparity estimation to complex imaging conditions such as weak texture and strong reflection is enhanced, and the reconstruction quality of photovoltaic module edges and low-texture areas is improved. On this basis, the target feature vector, initial disparity map, and context information are weighted and fused, which can combine local features, initial geometric relationships, and context information in disparity estimation, and can simultaneously preserve global context and local details, thereby improving disparity accuracy. Through the above deep learning model, disparity estimation of photovoltaic modules and subsequent construction of point cloud data can be completed based on binocular images, reducing the hardware cost requirements of image devices, reducing local mismatches in disparity estimation, and improving the integrity and accuracy of photovoltaic module point cloud representation, thereby improving the detection accuracy of photovoltaic module pose.
[0017] Furthermore, by projecting the extracted component planar point cloud back into the image and extracting the edge point cloud of the photovoltaic module from the image, the computational complexity of the edge extraction process is reduced. Grouping the projected point cloud and taking the median value as the representative pixel reduces the interference of outliers and improves the stability of edge extraction. By judging the distance of the pixel from the image boundary, the robotic arm can be guided to retake the image when it is incomplete, reducing the need for manual intervention. The combination of the above edge extraction and automatic retake mechanism reduces the probability of incomplete photovoltaic module edge point cloud, improves the completeness and accuracy of photovoltaic module point cloud representation, and thus improves the detection accuracy of photovoltaic module pose.
[0018] Furthermore, by introducing angle and distance constraints, the left and bottom edges of the photovoltaic module can be automatically distinguished in the point cloud data. This ensures that the two edge lines extracted from the photovoltaic module plane maintain consistency with the module plane, improving the extraction accuracy of the edge direction vectors and thus obtaining a robust local coordinate system. Further, based on the angle relationship between the first and second straight lines and the reference axis of the photovoltaic module plane, the first and second direction vectors corresponding to the photovoltaic module plane are determined. This method can more stably extract mutually orthogonal edge lines in actual installation environments with complex noise and significant structural interference, resulting in a more accurate local coordinate system for the constructed module and thus improving the detection accuracy of the photovoltaic module pose.
[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts illustrating the pose detection method for photovoltaic modules provided in the embodiments of this application; Figure 2 This is a second schematic flowchart of the photovoltaic module pose detection method provided in the embodiments of this application; Figure 3 This is a stereoscopic image diagram of the pose detection method for photovoltaic modules provided in the embodiments of this application; Figure 4 This is a monocular image schematic diagram of the pose detection method for photovoltaic modules provided in the embodiments of this application; Figure 5 This is the third flowchart illustrating the pose detection method for photovoltaic modules provided in this application embodiment; Figure 6 This is a schematic diagram of the point cloud generated by the pose detection method of the photovoltaic module provided in the embodiments of this application under different viewpoints; Figure 7 This is one of the photovoltaic module point cloud schematic diagrams of the photovoltaic module pose detection method provided in the embodiments of this application; Figure 8 This is the second point cloud diagram of a photovoltaic module for the pose detection method of the photovoltaic module provided in this application embodiment; Figure 9 This is a schematic diagram of a photovoltaic module for the photovoltaic module pose detection method provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of the photovoltaic module pose detection device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0021] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0022] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0023] The photovoltaic module pose detection method, photovoltaic module pose detection device, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.
[0024] The pose detection method for photovoltaic modules can be applied to the terminal, specifically executed by the hardware or software in the terminal.
[0025] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets. It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer.
[0026] It should be understood that a terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.
[0027] The photovoltaic module pose detection method provided in this application embodiment can be executed by an electronic device or a functional module or functional entity in an electronic device that can implement the photovoltaic module pose detection method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The following uses an electronic device as the execution subject to illustrate the photovoltaic module pose detection method provided in this application embodiment.
[0028] like Figure 1 As shown, the pose detection method for the photovoltaic module includes steps 110, 120, 130, 140, and 150.
[0029] Step 110: Extract features from the acquired binocular images to obtain the target feature vector and the initial disparity map; In this step, the binocular image is an image of the installed photovoltaic module; the target feature vector is a feature vector used to generate the disparity map, which is obtained by feature extraction from the binocular image; the initial disparity map is generated based on the disparity cost tensor constructed from the target feature vector.
[0030] In some embodiments, the binocular images are paired left and right view images with a stereo matching relationship, which can be acquired by a binocular camera or by two independent monocular cameras.
[0031] In some embodiments, acquiring binocular images may include: Control the mobile actuator to move to the shooting position; Based on the image acquisition device mounted on the mobile actuator, binocular images of the installed photovoltaic modules are acquired.
[0032] In this embodiment, the mobile actuator may include a robotic arm, a robot, or a drone; the image acquisition device is a binocular camera, and the shooting position corresponds to the installed photovoltaic module. The following explanation uses a robotic arm as an example to illustrate the acquisition of binocular images.
[0033] In actual implementation, refer to Figure 2 The robotic arm moves to the camera position, and the binocular camera captures binocular images; among them, the binocular images are as follows: Figure 3 As shown, including left and right eye images, an example of how the installed photovoltaic modules appear in the field of view of a binocular camera.
[0034] In actual implementation, refer to Figure 4 , Figure 4Monocular image from a binocular image of an installed photovoltaic module captured by a binocular camera.
[0035] The following section uses a deep learning model structure example to illustrate how feature extraction is performed on the left and right images to generate a disparity map for reconstructing the point cloud.
[0036] In some embodiments, step 110 includes: Feature extraction is performed on the first image and the second image in the binocular images respectively to obtain the first feature vector corresponding to the first image and the second feature vector corresponding to the second image; Deep feature extraction is performed on the first feature vector and the second feature vector to obtain the target feature vector corresponding to the first feature vector and the target feature vector corresponding to the second feature vector. An initial disparity map is obtained based on the target feature vector.
[0037] In this step, the first image and the second image correspond to the left and right eye images in the binocular images; the target feature vector is a feature vector used to characterize deep semantic features that are more suitable for disparity estimation; wherein, the deep semantic features may include: structural features such as the edges and corners of the photovoltaic module, the surface texture distribution and regional consistency features of the photovoltaic module, or other geometric semantic features that are beneficial to pixel matching and disparity inference between the left and right eye images.
[0038] In some embodiments, reference Figure 5 Based on the Depth Anything V2 depth estimation model, features can be extracted from the first and second images in the stereo image to obtain the first feature vector corresponding to the first image and the second feature vector corresponding to the second image.
[0039] In this embodiment, the Depth Anything V2 depth estimation model is a pre-trained model trained on a public dataset. It can utilize the geometric and semantic prior knowledge learned during pre-training to encode features of the input image.
[0040] Continue to refer to Figure 5 During the feature extraction process of the left and right eye images, Depth AnythingV2 shares weights.
[0041] It should be noted that the weights of Depth Anything V2 are kept frozen during the training process of the deep learning model.
[0042] In some embodiments, deep feature extraction can be performed on the first feature vector and the second feature vector based on a 2D convolutional network to obtain the target feature vector corresponding to the first feature vector and the target feature vector corresponding to the second feature vector.
[0043] In this embodiment, reference Figure 5 During the deep feature extraction process of the first and second feature vectors, the 2D convolutional network shares weights.
[0044] The target feature vector is obtained by further extracting the first and second feature vectors through a 2D convolutional network, making the target feature vector more suitable for disparity estimation of photovoltaic modules.
[0045] In some embodiments, reference Figure 5 Based on the target feature vector, an initial disparity map is obtained, including: The target feature vector is processed based on the pixel matching relationship of epipolar constraints to construct the disparity cost tensor; The cost value is estimated along the disparity dimension to determine the disparity value corresponding to each pixel in the disparity cost tensor, and an initial disparity map is obtained based on the disparity value.
[0046] In this embodiment, the pixel matching relationship based on epipolar constraints can be regarded as a specialized application of the optical flow method under the condition of binocular stereo correction.
[0047] The following section explains the process of processing the target feature vector based on epipolar constraints using pixel matching relationships to construct the disparity cost tensor.
[0048] In actual execution, taking pixel p1(x,y) in the left-eye image as an example, the corresponding point in the right-eye image is p2(x+u,y+v). Here, x and y represent the pixel coordinates of point p1 in the left-eye image, and u and v represent the displacement of the pixel in the x and y directions, respectively.
[0049] After stereo correction, the matching point p2 of the binocular camera is only shifted in the horizontal direction, so it can be specialized to p2(x d, y), where d represents the disparity displacement of the pixel in the horizontal direction.
[0050] In the stereo-corrected binocular images, matching points can be searched in the horizontal direction based on epipolar constraints within the disparity search range in the horizontal direction, referencing... Figure 5 The target feature vector corresponding to the first feature vector is compared with the target feature vector corresponding to the second feature vector along the epipolar line, and a disparity cost tensor of size H×W×D is constructed. Here, H and W are the height and width of the image, respectively, and D is the maximum range of quantized disparity.
[0051] The process of estimating the cost value in the disparity dimension to determine the disparity value of each pixel and generating an initial disparity map accordingly is described below.
[0052] In actual execution, after obtaining the disparity cost tensor, the cost value can be estimated in the disparity dimension to determine the disparity value corresponding to each pixel. Based on the determined disparity value, the distribution of disparity in the image space is reconstructed, and finally the initial disparity map corresponding to the left eye image is obtained.
[0053] In some embodiments, reference Figure 5 We can perform weighted regression on the cost value of the disparity dimension based on a regression function (such as soft-argmin), so that the disparity hypothesis with a smaller cost value (i.e. a higher degree of feature matching) gets a higher weight, thereby obtaining the disparity estimate of each pixel; and then reconstruct the disparity estimate according to the pixel position to obtain the initial disparity map.
[0054] Step 120: Perform context modeling on the first image in the binocular images to obtain the context information corresponding to the first image; In this step, contextual information is used to characterize global structural features and spatial relationships between pixels in the first image.
[0055] In actual implementation, refer to Figure 5 The context information of the photovoltaic module image in the left eye image can be extracted through the context network feature extraction network. This information can be used to better estimate the disparity of the photovoltaic module in the acquired binocular images based on its weak texture and high reflectivity.
[0056] According to the photovoltaic module pose detection method provided in the embodiments of this application, by performing multi-scale information aggregation and context-dependent modeling on the first image, the ability of disparity estimation to adapt to complex imaging conditions such as weak texture areas and reflection areas in the photovoltaic module image can be improved, thereby improving the overall stability and accuracy of the initial disparity map.
[0057] Step 130: Perform weighted fusion of the target feature vector, the initial disparity map, and the contextual information to obtain the first disparity map; In this step, the target feature vector and the initial disparity map are obtained by feature extraction from the binocular images; the first disparity map is used to characterize the disparity information after fusion processing, which can further improve the precision and stability of disparity estimation based on the initial disparity map.
[0058] During the research and development process, the inventors discovered that in photovoltaic module installation pose detection scenarios, related technologies mainly use Aruco codes, targets, or other manually placed guides to establish the geometric relationship between the module and the camera. Some studies have also attempted to directly acquire point clouds using 3D cameras to obtain module surface information. While these two methods can achieve a certain level of pose detection accuracy under ideal conditions, they suffer from high labor and equipment costs and are limited by lighting conditions, photovoltaic module reflection characteristics, and acquisition range, making them difficult to promote and apply in large-scale photovoltaic power plant scenarios. Furthermore, due to issues such as sparse texture, strong reflection, or truncated boundaries on the surface of photovoltaic modules, geometrically constrained matching algorithms reduce the accuracy of disparity estimation when generating disparity maps, resulting in poor point cloud reconstruction quality and affecting the accuracy of pose detection results.
[0059] According to the photovoltaic module pose detection method provided in this application embodiment, feature extraction is performed on binocular images to obtain target feature vectors and initial disparity maps. Context modeling is performed on the first image to extract global correlation information, enhancing the adaptability of disparity estimation to complex imaging conditions such as weak texture and strong reflection, and improving the reconstruction quality of photovoltaic module edges and low-texture areas. Based on this, the target feature vector, initial disparity map, and context information are weighted and fused, which can combine local features, initial geometric relationships, and context information in disparity estimation, simultaneously preserving global context and local details, and improving disparity accuracy. Through the above deep learning model, disparity estimation of photovoltaic modules and subsequent construction of point cloud data can be completed based on binocular images, reducing the hardware cost requirements of image devices, reducing local mismatches in disparity estimation, and improving the integrity and accuracy of photovoltaic module point cloud representation, thereby improving the detection accuracy of photovoltaic module pose.
[0060] In some embodiments, step 130 includes: An attention mechanism algorithm is used to process the target feature vector, initial disparity map, and contextual information to determine the weight sequence; The target feature vector, the initial disparity map, and the context information are weighted and fused based on the weight sequence to obtain the first disparity map.
[0061] In this embodiment, the attention mechanism is used to dynamically allocate weights based on the feature importance of different information sources, thereby enhancing the response to effective parallax information and suppressing the influence of noise features such as weak texture regions and reflective regions.
[0062] In some embodiments, the attention mechanism algorithm may employ SE attention mechanism, CBAM attention mechanism, and ECA attention mechanism, etc.
[0063] The following explanation uses a 3D convolutional network with added SE attention mechanism as an example.
[0064] In actual implementation, refer to Figure 5 The context information, initial disparity map, and target feature vector are concatenated into a tensor, and the tensor is then input into a 3D convolutional network for processing.
[0065] An SE attention mechanism is incorporated into the 3D convolutional processing to adjust the weights of information from different feature sources, adaptively strengthening the feature components that are more effective in the current disparity estimation and weakening the influence of noisy or unstable features. Through this fusion and adaptive weight adjustment process, the initial disparity can be updated and refined, ultimately yielding a more accurate first disparity map.
[0066] In some embodiments, after step 130, the disparity estimation model can also be trained based on the acquired binocular image data.
[0067] In actual implementation, point cloud data and corresponding disparity maps can be collected under different lighting conditions (such as low light) using image acquisition devices (such as binocular cameras, 3D laser cameras, etc.). While keeping the camera pose unchanged, left and right eye images of the camera can be collected under different lighting conditions to construct a training dataset for model training.
[0068] It should be noted that, in some embodiments, point cloud data and corresponding parallax maps can be obtained by scanning with a 3D laser camera under low light conditions.
[0069] During the model training phase, the encoding weights of Depth Anything V2 can be kept unchanged, the parameters of ContextNetwork can be fine-tuned, and the disparity estimation model can be trained using the aforementioned image data to improve the stability and accuracy of disparity prediction.
[0070] According to the photovoltaic module pose detection method provided in the embodiments of this application, by introducing an attention mechanism, dynamic weight allocation is performed on the tensor spliced by context information, initial disparity map and target feature vector, so that the model can adaptively focus on key features of different regions. In the disparity estimation process, local details, initial geometric relationships and global context information are effectively combined, which enhances the feature expression ability of weak texture and reflective areas and improves the accuracy and stability of disparity estimation.
[0071] Step 140: Based on the camera parameters corresponding to the first disparity map and the binocular image, obtain the point cloud data corresponding to the photovoltaic module; In this step, the camera parameters corresponding to the stereo image include at least the camera's intrinsic parameters and the baseline distance between the stereo cameras.
[0072] In actual implementation, binocular images can be as follows: Figure 3As shown, the binocular images are input into the trained deep learning model to obtain the first disparity map. Combined with the intrinsic parameters of the binocular cameras and the baseline distance between the left and right cameras, the point cloud data of the photovoltaic module is obtained. The schematic diagrams of the photovoltaic module's point cloud data from different viewpoints can be provided. Figure 6 As shown.
[0073] In some embodiments, reference Figure 2 Following step 140, the following is also included: Clustering algorithms are used to segment the point cloud data corresponding to photovoltaic modules, and ground point cloud data is removed by distance determination to obtain the first sub-point cloud data; The photovoltaic module plane is extracted from the first sub-point cloud data, and the purlin point cloud is removed to update the point cloud data corresponding to the photovoltaic module.
[0074] In this embodiment, by segmenting and extracting the planar data from the point cloud, interference from non-target structure point clouds can be reduced, and effective point clouds of the photovoltaic module can be obtained for subsequent pose calculation.
[0075] In some embodiments, the clustering algorithm may be a density-based spatial clustering algorithm (DBSCAN), a K-means clustering algorithm (K-Means), or a point cloud clustering method based on region growing, and this application does not limit it.
[0076] In some embodiments, the photovoltaic module plane can be extracted using least-squares plane fitting or random sampling consistency variant algorithms.
[0077] refer to Figure 2 The following section explains how to remove ground point cloud data, extract photovoltaic module planar data, and remove purlin point cloud data.
[0078] Step 1: Parameter Setting. Based on factors such as point cloud density and scene scale, select... Clustering parameters such as neighborhood radius and MinPts (minimum number of points).
[0079] Step 2, Initialization. Select an unvisited point from the point cloud data as the starting point for clustering.
[0080] Step 3: Determine the core object. Search centered on the starting point. If the number of points in the neighborhood is not less than MinPts, then the point is marked as the core point.
[0081] Step 4: Cluster Expansion. Using the core point as the center, gradually add all reachable core points and boundary points to the current cluster until the cluster no longer expands.
[0082] Step 5: Repeat clustering. Repeat core point determination and cluster expansion for the remaining unvisited points until all point clouds have been assigned or labeled.
[0083] Step 6: Ground Point Cloud Removal. Due to the significant height difference between the ground point cloud and the photovoltaic modules, ground point clouds can be identified and removed based on the centroid height or distance threshold to obtain the first sub-point cloud data. A schematic diagram of the first sub-point cloud data after removing the ground point cloud can be found here. Figure 7 .
[0084] Step 7: Photovoltaic module plane extraction. Perform random sample consensus algorithm (RANSAC) on the first sub-point cloud to perform plane fitting to extract the main plane of the photovoltaic module, and remove the purlin point cloud located on the back structure of the module according to the fitting residual or distance threshold.
[0085] Step 8, Point Cloud Update. The point cloud after removing the ground and purlin point clouds is used as the final valid point cloud data for the photovoltaic module, and is used for subsequent pose calculations. A schematic diagram of the point cloud after removing the ground and purlin point clouds can be found in [reference needed]. Figure 8 .
[0086] Step 150: Determine the pose information of the photovoltaic module based on point cloud data.
[0087] In this step, the point cloud data is the point cloud data corresponding to the photovoltaic module after the ground point cloud and purlin point cloud are extracted in step 140; the pose information of the photovoltaic module is used to characterize the spatial position and orientation of the photovoltaic module in three-dimensional space.
[0088] In some embodiments, the pose information of the photovoltaic module includes, but is not limited to, directional parameters describing the plane orientation of the photovoltaic module and displacement parameters characterizing the specific position of the photovoltaic module in space. For example, orientation information determined based on the plane normal vector of the photovoltaic module and spatial position coordinates determined by point cloud feature points.
[0089] In some embodiments, reference Figure 2 Step 150 includes: Based on point cloud data, obtain the normal vector of the photovoltaic module plane; The point cloud data is projected onto the first image in the binocular image to obtain the first and second direction vectors of the photovoltaic module plane; The pose of the photovoltaic module is determined based on the first direction vector, the second direction vector, and the normal vector.
[0090] In this embodiment, the first image is the left-eye image from a binocular image. The first direction vector and the second direction vector are the normal vectors of the photovoltaic module edge, corresponding to the vectors in the x-axis and y-axis directions, respectively.
[0091] In some embodiments, the left eye image used for projection may be preprocessed, such as distortion correction, brightness equalization, black and white conversion, noise suppression, or edge enhancement, to improve image quality and the stability of subsequent point cloud projection, but is not limited to the above methods.
[0092] In some embodiments, projecting point cloud data onto the left eye image in a binocular image can be represented by the following formula:
[0093] Where x, y, and z are the three-dimensional coordinates of the photovoltaic module's planar point cloud in the camera coordinate system, respectively. Here, u and v represent the camera intrinsic parameters corresponding to the camera, respectively, and represent the pixel coordinates on the left eye image plane calculated based on the 3D point (x, y, z) and the camera intrinsic parameter K. and These are the focal lengths of the camera along the x and y axes, respectively. and These are the coordinates of the principal point.
[0094] In actual implementation, the normal vector of the photovoltaic module plane can be obtained based on point cloud data through the following steps.
[0095] Step 1: Perform Principal Component Analysis (PCA) on the point cloud data to fit it and obtain the initial normal vector of the point cloud.
[0096] Step 2: In each iteration, randomly select 3 non-collinear points as the minimum sample set, and use these to solve for the parameters of the plane equation ax + by + cz + d = 0. Here, a, b, and c are normal vectors, and d is the bias term.
[0097] Step 3: Calculate the distance from all point cloud data to the current plane, and mark points whose distance is less than a preset threshold as interior points. The preset threshold can be based on experimental data or user-defined criteria.
[0098] Step 4: Repeat the above steps until the iteration condition is met, and record the plane with the most interior points as the current optimization result.
[0099] Step 5: Perform PCA fitting again based on all interior points obtained in Step 4 to obtain the final normal vector of the photovoltaic module plane. .
[0100] In some embodiments, projecting point cloud data onto a first image in a binocular image to obtain a first direction vector and a second direction vector of the photovoltaic module plane includes: Based on the projection of point cloud data onto the first image in the binocular image, obtain the first point cloud data and the second point cloud data corresponding to the first image; By filtering the interior points of the first and second point cloud data, the first and second straight lines are obtained. Based on the angular relationship between the first and second straight lines and the reference axis of the photovoltaic module plane, the first direction vector and the second direction vector corresponding to the photovoltaic module plane are determined.
[0101] In this embodiment, the reference axes of the photovoltaic module plane are the x-axis and y-axis, and the first direction vector and the second direction vector are the normal vectors of the edge of the photovoltaic module, corresponding to the vectors in the x-axis and y-axis directions.
[0102] The following explains how to obtain the first and second straight lines based on constrained RANSAC, and how to determine the first and second direction vectors corresponding to the photovoltaic module plane.
[0103] Step 1: Based on the first point cloud data and the second point cloud data projected onto the first image, randomly select two pairs of points in each point cloud and construct candidate straight lines corresponding to the two pairs of points respectively. and And calculate the candidate direction vectors corresponding to the two lines. and .in, and Let these be the normal vectors corresponding to the edges of the photovoltaic module, and let these vectors be the normal vectors. and Normal vector to the plane of the photovoltaic module It is a vertical relationship.
[0104] Step 2, use the candidate direction vectors obtained in Step 1. and Normal vector to the plane of the aforementioned photovoltaic module Perform an angle determination; if the angle between any direction vector and the normal vector is greater than or equal to the angle threshold, then reselect two pairs of points to construct a straight line and repeat step 1. The angle threshold can be determined based on user-defined parameters or experimental data, for example, set to 90°±0.5°.
[0105] Step 3: Calculate the distance from each point in the point cloud to the straight line. and Points within a distance of less than a certain threshold are marked as interior points of the corresponding line. This distance threshold can be determined user-defined or based on experimental data, for example, set to 3mm.
[0106] Step 4: Repeat steps 1 to 3, and record the two lines with the most interior points as the temporary optimal solutions for the first and second lines.
[0107] Step 5: Perform least-squares fitting on the two temporary optimal lines based on their respective interior points to determine the final first line. Second straight line The closest point between the two lines is used as the translation reference point. ;refer to Figure 9 , Figure 9 The image shows the pose information of the installed photovoltaic modules. The red dashed line represents the first straight line. The blue dashed line is the second straight line. The green dot represents the lower left corner of an installed photovoltaic module. Based on the determined first straight line... Second straight line Based on the angles between the x-axis and the reference axis of the photovoltaic module plane, the direction corresponding to the line with the smaller angle to the reference x-axis is determined as the first direction vector (i.e., ...). The direction corresponding to the other straight line is defined as the second direction vector (i.e. ).
[0108] According to the photovoltaic module pose detection method provided in this application, by introducing angle and distance constraint mechanisms, the left and bottom edges of the photovoltaic module can be automatically distinguished in point cloud data. This ensures that the two edge lines extracted from the photovoltaic module plane maintain consistency with the module plane, improving the extraction accuracy of edge direction vectors and thus obtaining a robust local coordinate system. Furthermore, based on the angle relationship between the first and second straight lines and the reference axis of the photovoltaic module plane, the first and second direction vectors corresponding to the photovoltaic module plane are determined. This method can more stably extract mutually orthogonal edge lines in actual installation environments with complex noise and significant structural interference, making the constructed local coordinate system of the module more accurate and thus improving the pose detection accuracy of the photovoltaic module.
[0109] In some embodiments, obtaining first point cloud data and second point cloud data corresponding to the first image based on the projection of point cloud data onto the first image in a binocular image includes: The point cloud data is projected onto the first image to obtain the first projection information; Based on the first projection information and point cloud data, obtain the boundary distance between the image edge of the first image and the point cloud data. If the boundary distance is less than a preset threshold, re-acquire binocular images; If the boundary distance is greater than or equal to a preset threshold, determine the first point cloud data and the second point cloud data corresponding to the first image.
[0110] In this embodiment, the first image is the left eye image from the binocular images. The preset threshold can be determined based on user-defined criteria or experimental data.
[0111] It is understood that the trigger condition for re-acquiring binocular images can be determined based on the projection integrity of a preset reference area in the first image, and this application does not limit this. The reference area can be selected according to the specific application scenario. For example, in an implementation using the lower left corner of the photovoltaic module as a reference for pose detection, when incomplete point cloud projection is detected in the lower left corner area, the image acquisition device can re-acquire binocular images by controlling the movement of the motion actuator; for an implementation using the lower right corner of the photovoltaic module as a reference, re-acquisition can be triggered when incomplete projection is detected in the lower right corner area.
[0112] In actual execution, binocular images can be re-acquired based on mobile actuators, such as robotic arms, robots, or drones; the image acquisition device for acquiring binocular images is a binocular camera, and the shooting position corresponds to the installed photovoltaic modules.
[0113] The following explains how to obtain the first point cloud data and the second point cloud data corresponding to the first image.
[0114] refer to Figure 2 After projecting the point cloud data onto the first image, it can be segmented along the row direction based on the pixel coordinates of the point cloud, for example, divided into groups of 10 rows each. Within each group, the leftmost projected pixel of each row is obtained, forming a corresponding pixel set. After sorting the set by its horizontal coordinate, the median value of the sorted set is selected as the representative pixel of the group. Similarly, the projected point cloud can be processed along the column direction to obtain representative pixels representing the bottom edge of the image.
[0115] Continue to refer to Figure 2 The distance between the representative pixel and the boundary of the first image can be calculated based on the aforementioned representative pixel. If the distance is less than a preset threshold, the moving actuator can be controlled to move, causing the image acquisition device to re-acquire the binocular image to ensure the integrity of the projected point cloud in the image. If the distance is greater than or equal to the preset threshold, the point cloud data associated with the corresponding representative pixel can be determined as the first point cloud data and the second point cloud data corresponding to the first image.
[0116] During the research and development process, the inventors discovered that in related technologies, the image acquisition of photovoltaic modules often lacks an automatic detection and reshoot mechanism. When the image boundary is missing, it is easy to cause the subsequent edge extraction to fail. If the reshoot is carried out by manually adjusting the shooting position, it will increase the complexity of the operation process.
[0117] According to the photovoltaic module pose detection method provided in this application, by projecting the extracted module planar point cloud back into the image and extracting the edge point cloud of the photovoltaic module in the image, the computational complexity of the edge extraction process is reduced. Grouping the projected point cloud and taking the median value as the representative pixel reduces the interference of outliers and improves the stability of edge extraction. By judging the distance of the pixel from the image boundary, the robotic arm can be guided to retake the image if the image is incomplete, reducing the need for manual intervention. The combination of the above edge extraction and automatic retake mechanism reduces the probability of incomplete photovoltaic module edge point cloud, improves the completeness and accuracy of the photovoltaic module point cloud representation, and thus improves the accuracy of photovoltaic module pose detection.
[0118] The following section explains how to determine the pose of a photovoltaic module based on the first direction vector, the second direction vector, and the normal vector.
[0119] Construct the direction vector matrix M= ;in, The normal vector of the photovoltaic module plane obtained above , Let be the first direction vector. This is the second direction vector.
[0120] Perform singular value decomposition (SVD) on matrix M and orthogonalize it to obtain the corrected rotation matrix R. Combine this with the translation reference point obtained in step 5 above. The final photovoltaic module pose is obtained, expressed by the formula:
[0121] Where T is the pose transformation matrix describing the photovoltaic module in three-dimensional space, used to characterize the rotation and translation relationship of the photovoltaic module in the reference coordinate system; R is the rotation matrix, used to describe the orientation direction of the photovoltaic module plane relative to the reference coordinate system; t is the translation reference point, used to represent the spatial position of the photovoltaic module in the reference coordinate system.
[0122] In some embodiments, after step 150, the method further includes: Based on the pose of the photovoltaic module determined in step 150, the module is translated along the reference axis to obtain the pose of the photovoltaic module to be installed.
[0123] In this embodiment, reference Figure 9 The determined pose can be the position information of the lower left corner of the installed photovoltaic module. The reference axis can be the x-axis and y-axis, or a reference axis determined by the pose information of the installed photovoltaic module.
[0124] In actual execution, the determined pose can be translated in the x and y directions according to the arrangement of the installed photovoltaic modules, the installation spacing requirements, and the spatial layout relationship between the modules, so as to obtain the target pose of the photovoltaic modules to be installed, and then the photovoltaic modules can be moved and installed by controlling the moving actuator.
[0125] In some embodiments, the mobile actuator may include a robotic arm, a robot, or a drone.
[0126] The photovoltaic module pose detection method provided in this application can be executed by a photovoltaic module pose detection device. This application uses the photovoltaic module pose detection device executing the photovoltaic module pose detection method as an example to illustrate the photovoltaic module pose detection device provided in this application.
[0127] This application also provides a photovoltaic module pose detection device.
[0128] like Figure 10 As shown, the photovoltaic module pose detection device includes: a first processing module 1010, a second processing module 1020, a third processing module 1030, a fourth processing module 1040, and a fifth processing module 1050.
[0129] The first processing module 1010 is used to extract features from the acquired binocular images to obtain target feature vectors and initial disparity maps; The second processing module 1020 is used to perform context modeling on the first image in the binocular image and obtain the context information corresponding to the first image. The third processing module 1030 is used to perform weighted fusion of the target feature vector, the initial disparity map and the context information to obtain the first disparity map; The fourth processing module 1040 is used to obtain point cloud data corresponding to the photovoltaic module based on the camera parameters corresponding to the first disparity map and the binocular image; The fifth processing module 1050 is used to determine the pose information of the photovoltaic module based on point cloud data.
[0130] In some embodiments, the first processing module 1010 may also be used for: Feature extraction is performed on the first image and the second image in the binocular images respectively to obtain the first feature vector corresponding to the first image and the second feature vector corresponding to the second image; Deep feature extraction is performed on the first feature vector and the second feature vector to obtain the target feature vector corresponding to the first feature vector and the target feature vector corresponding to the second feature vector. The base target feature vector is used to obtain the initial disparity map.
[0131] In some embodiments, the third processing module 1030 can also be used for: An attention mechanism algorithm is used to process the target feature vector, initial disparity map, and contextual information to determine the weight sequence; The target feature vector, the initial disparity map, and the context information are weighted and fused based on the weight sequence to obtain the first disparity map.
[0132] In some embodiments, the fifth processing module 1050 can also be used for: Based on point cloud data, obtain the normal vector of the photovoltaic module plane; The point cloud data is projected onto the first image in the binocular image to obtain the first and second direction vectors of the photovoltaic module plane; The pose of the photovoltaic module is determined based on the first direction vector, the second direction vector, and the normal vector.
[0133] In some embodiments, the fifth processing module 1050 can also be used for: Based on the projection of point cloud data onto the first image in the binocular image, obtain the first point cloud data and the second point cloud data corresponding to the first image; By filtering the interior points of the first and second point cloud data, the first and second straight lines are obtained. Based on the angular relationship between the first and second straight lines and the reference axis of the photovoltaic module plane, the first direction vector and the second direction vector corresponding to the photovoltaic module plane are determined.
[0134] In some embodiments, the fifth processing module 1050 can also be used for: The point cloud data is projected onto the first image to obtain the first projection information; Based on the first projection information and point cloud data, obtain the boundary distance between the image edge of the first image and the point cloud data. If the boundary distance is less than a preset threshold, re-acquire binocular images; If the boundary distance is greater than or equal to a preset threshold, determine the first point cloud data and the second point cloud data corresponding to the first image.
[0135] The photovoltaic module pose detection device provided in this application extracts features from binocular images to obtain target feature vectors and initial disparity maps, and performs context modeling on the first image to extract global correlation information. This enhances the adaptability of disparity estimation to complex imaging conditions such as weak texture and strong reflection, and improves the reconstruction quality of photovoltaic module edges and low-texture areas. Furthermore, by weighted fusion of the target feature vector, initial disparity map, and context information, local features, initial geometric relationships, and context information can be combined in disparity estimation, simultaneously preserving global context and local details, thus improving disparity accuracy. Through the aforementioned deep learning model, disparity estimation of photovoltaic modules and subsequent point cloud data construction can be completed based on binocular images, reducing the hardware cost requirements of the image processing device, minimizing local mismatches in disparity estimation, and improving the completeness and accuracy of the photovoltaic module point cloud representation, thereby improving the detection accuracy of photovoltaic module pose.
[0136] The pose detection device for photovoltaic modules in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific device.
[0137] The photovoltaic module pose detection device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.
[0138] The photovoltaic module pose detection device provided in this application embodiment can achieve Figures 1 to 9 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.
[0139] In some embodiments, such as Figure 11 As shown, this application embodiment also provides an electronic device 1100, including a processor 1101, a memory 1102, and a computer program stored in the memory 1102 and executable on the processor 1101. When the program is executed by the processor 1101, it implements the various processes of the above-described photovoltaic module pose detection method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0140] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.
[0141] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described photovoltaic module pose detection method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.
[0142] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0143] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described photovoltaic module pose detection method.
[0144] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0145] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described photovoltaic module pose detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0146] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0147] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0149] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
[0150] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0151] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.
Claims
1. A method for detecting the pose of a photovoltaic module, characterized in that, The method includes: Feature extraction is performed on the acquired binocular images to obtain the target feature vector and the initial disparity map; Context modeling is performed on the first image in the binocular images to obtain the context information corresponding to the first image; The target feature vector, the initial disparity map, and the context information are weighted and fused to obtain the first disparity map; Based on the camera parameters corresponding to the first disparity map and the binocular image, the point cloud data corresponding to the photovoltaic module is obtained; Based on the point cloud data, the pose information of the photovoltaic module is determined.
2. The photovoltaic module pose detection method according to claim 1, characterized in that, The step of extracting features from the acquired binocular images to obtain target feature vectors and an initial disparity map includes: Feature extraction is performed on the first image and the second image in the binocular images respectively to obtain the first feature vector corresponding to the first image and the second feature vector corresponding to the second image; Deep feature extraction is performed on the first feature vector and the second feature vector to obtain the target feature vector corresponding to the first feature vector and the target feature vector corresponding to the second feature vector; The initial disparity map is obtained based on the target feature vector.
3. The photovoltaic module pose detection method according to claim 1, characterized in that, The step of weightedly fusing the target feature vector, the initial disparity map, and the context information to obtain the first disparity map includes: An attention mechanism algorithm is used to process the target feature vector, the initial disparity map, and the context information to determine the weight sequence; The target feature vector, the initial disparity map, and the context information are weighted and fused based on the weight sequence to obtain the first disparity map.
4. The method for detecting the pose of a photovoltaic module according to any one of claims 1-3, characterized in that, Determining the pose information of the photovoltaic module based on the point cloud data includes: Based on the point cloud data, obtain the normal vector of the photovoltaic module plane; The point cloud data is projected onto the first image in the binocular image to obtain the first direction vector and the second direction vector of the photovoltaic module plane; The pose of the photovoltaic module is determined based on the first direction vector, the second direction vector, and the normal vector.
5. The photovoltaic module pose detection method according to claim 4, characterized in that, The step of projecting the point cloud data onto the first image in the binocular image to obtain the first direction vector and the second direction vector of the photovoltaic module plane includes: Based on the projection of the point cloud data onto the first image in the binocular image, obtain the first point cloud data and the second point cloud data corresponding to the first image; The first point cloud data and the second point cloud data are filtered for interior points to obtain the first straight line and the second straight line; Based on the angular relationship between the first straight line and the second straight line and the reference axis of the photovoltaic module plane, the first direction vector and the second direction vector corresponding to the photovoltaic module plane are determined.
6. The photovoltaic module pose detection method according to claim 5, characterized in that, The step of obtaining the first point cloud data and the second point cloud data corresponding to the first image based on the projection of the point cloud data onto the first image in the binocular image includes: The point cloud data is projected onto the first image to obtain first projection information; Based on the first projection information and the point cloud data, the boundary distance between the image edge of the first image and the point cloud data is obtained; If the boundary distance is less than a preset threshold, re-acquire binocular images; If the boundary distance is greater than or equal to the preset threshold, the first point cloud data and the second point cloud data corresponding to the first image are determined.
7. A pose detection device for a photovoltaic module, characterized in that, The device includes: The first processing module is used to extract features from the acquired binocular images to obtain the target feature vector and the initial disparity map; The second processing module is used to perform context modeling on the first image in the binocular image and obtain the context information corresponding to the first image. The third processing module is used to perform weighted fusion of the target feature vector, the initial disparity map and the context information to obtain the first disparity map; The fourth processing module is used to obtain point cloud data corresponding to the photovoltaic module based on the camera parameters corresponding to the first disparity map and the binocular image; The fifth processing module is used to determine the pose information of the photovoltaic module based on the point cloud data.
8. A robot, characterized in that, Including the pose detection device for photovoltaic modules as described in claim 7.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the pose detection method for photovoltaic modules as described in any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the pose detection method for photovoltaic modules as described in any one of claims 1-6.