Photovoltaic panel target pose estimation visual recognition method based on Gen6D strategy

Abstract

The present invention discloses a photovoltaic panel target pose estimation visual recognition method based on the Gen6D strategy. The pose estimation is divided into three parts: photovoltaic panel detection, pose rough estimation, and pose refinement. Each part includes a neural network. The photovoltaic panel detection part inputs the picture obtained by the camera into the convolutional neural network 1, and finally outputs the central pixel coordinates and scaling value of the photovoltaic panel. The pose of the photovoltaic panel is roughly estimated by the convolutional network 2. After the calculation of the convolutional network 3, the refined photovoltaic panel pose is finally obtained. In order to accurately distinguish the photovoltaic panel during the photovoltaic panel detection process, the present invention designs a similarity discrimination method combining the color distribution histogram and the hash value in the HSV color space. In order to reduce the jitter problem of the bounding box, an adaptive weighted filtering algorithm is designed to filter the pixel coordinate values ​​of the corner points of the bounding box.

Description

Technical Field

[0001] The present invention relates to the technical field of target pose estimation, and in particular to a photovoltaic panel target pose estimation visual recognition method based on a Gen6D strategy. Background Art

[0002] Solar photovoltaic panels are stacked on the chassis of the installation robot. As the robot chassis transports the photovoltaic panels from the initial position to the installation position, the photovoltaic panels will move due to the vibration of the vehicle. Some even move significantly, which affects the grasping of the robot arm. How to adjust the robot arm to ensure that the position of the photovoltaic panel is the same each time it is grasped, thereby ensuring the grasping accuracy of the photovoltaic panel. Therefore, how to identify the target pose of the photovoltaic panel will be the key to accurately grasping and installing the photovoltaic panel. Designing a visual recognition method for photovoltaic panel target pose estimation will improve the recognition accuracy of photovoltaic panels during the installation process, and ultimately increase the work efficiency of the installation robot in installing photovoltaic panels. Summary of the invention

[0003] The existing photovoltaic panels undergo significant changes in position during transportation on the installation robot chassis. In most cases, the photovoltaic panels cannot be arranged neatly. Figure 1 As shown, 1, 2...n represent photovoltaic panels in a disordered state, so it will be difficult for the robotic arm grasping mechanism to identify the accurate posture of the photovoltaic panels. The present invention designs a photovoltaic panel target posture estimation visual recognition method based on the Gen6D strategy, so as to obtain the accurate posture of the photovoltaic panels and ensure that the robotic arm can accurately complete the photovoltaic panel grasping task.

[0004] The technical solution of the present invention is as follows:

[0005] A photovoltaic panel target pose estimation visual recognition method based on the Gen6D strategy includes three steps: photovoltaic panel detection, rough pose estimation, and pose refinement. Each step contains a neural network, such as Figure 2 shown.

[0006] Step 1) Photovoltaic panel detection: Input the image obtained by the camera into the convolutional neural network 1, and calculate the similarity score map, coordinate offset map and scaling ratio based on the reference image generated in advance. Figure 3 The group feature map is constructed, and the point with the highest score in the similarity score map is taken as the center of the photovoltaic panel. According to the position of this point, the corresponding offset and scaling ratio values ​​are selected on the coordinate offset map and scaling ratio map. Then, the coordinates of this point in the similarity score map and the offset are added to obtain the final photovoltaic panel center pixel coordinates, and finally the photovoltaic panel center pixel coordinates and scaling ratio values ​​are output.

[0007] Step 2) Rough pose estimation: According to the center pixel coordinates and scaling value of the photovoltaic panel, a square image of the target photovoltaic panel is cut out from the original image with a box and scaled to 128×128 size. It and all reference images are input into the convolutional neural network 2 of the rough pose estimation part. Convolutional neural network 2 calculates the similarity between the cropped image and each reference image, and selects a reference image with the highest similarity. The pose of the photovoltaic panel is roughly estimated according to this reference image;

[0008] Step 3) Pose refinement: The rough pose is sent to the convolutional neural network 3 of the pose refinement part, and multiple reference images are selected and compared with the original image. Figure 1 The image is sent to the convolutional neural network 3. After calculation by the convolutional neural network 3, the refined photovoltaic panel posture is finally obtained.

[0009] Step 4) Target tracking: After obtaining the position and posture of the photovoltaic panel, its 3D bounding box is displayed in the image. Finally, the photovoltaic panel is tracked as a single target and continuously tracked and identified.

[0010] Furthermore, in order to ensure the accuracy of photovoltaic panel recognition, after obtaining the reference image with the highest similarity from convolutional neural network 2, a secondary similarity judgment is performed between the reference image and the cropped image of the original image. When the similarity is low, the cropped image is considered to be an incorrectly recognized image and is discarded.

[0011] Image similarity secondary discrimination method: The present invention designs a similarity discrimination method combining color distribution histogram and hash value in HSV color space, and its flow chart is as follows Figure 3 shown.

[0012] First, convert the two images to the HSV color space, divide the 0-180 hue values ​​into six groups under the H channel, and reassign the hue values ​​in each group to 15, 45, 75, 105, 135, and 165 to generate new H channel data. Then set the contrast area based on the position of the object in the image. Because the original image and the reference image are both centered on the photovoltaic panel, and the background images in the four corners are interference information, the mask1 area is set to a circular area with a radius of 60 with the center of the image as the center, and the mask2 area is set to a circular area with a radius of 30-45 with the center of the image as the center.

[0013] The mask1 area mainly contains the overall image of the object, and the mask2 area mainly contains the edge image of the object. Then, the color distribution histogram similarity comparison is performed under the H channel data in the mask1 area, and the hash similarity comparison is performed under the H channel data in the mask2 area. When both sets of similarity results meet the threshold, the query image is considered to be the correctly identified object, otherwise the query image is discarded. This similarity comparison method takes into account both the color similarity and the shape similarity of the objects in the image, and has higher accuracy.

[0014] Furthermore, in the target tracking stage, after obtaining the 3D bounding box of the photovoltaic panel and mapping it to the image, in order to reduce the jitter problem of the bounding box, the pixel coordinate values ​​of the corner points of the bounding box need to be filtered.

[0015] The weighted average operation is performed on the previous four data and the current data to obtain the final coordinate value. The weight calculation method is as follows: first set the initial weights 0, 1, 2, 3, 4, and then divide each by the standard deviation 1.414 of 0, 1, 2, 3, 4 respectively, to obtain 0, 0.707, 1.414, 2.12, 2.83, then perform square operation and take the opposite number, and finally perform exponential operation with e as the base to obtain the final values ​​1, 0.61, 0.14, 0.01, 0.001, which are the weights of the current time to the previous 4 times from the beginning to the end;

[0016] At the same time, an adaptive parameter is added , when the data changes, calculate the amount of change in the data , according to formula (2) to calculate the adaptive parameters ,

[0017] ;

[0018] in To adjust the parameters.

[0019] The adaptive weighted filtering algorithm formula is shown in formula (3):

[0020] ;

[0021] When the data is basically stable, the range of change is small, and the change speed is slow, the adaptive parameters When it is close to 1, the weight of the previous four data is relatively large, and the latest coordinate data is obtained based on the current and previous four data. When the camera or object moves quickly, the data changes greatly, and the adaptive parameters It decreases rapidly, thereby reducing the weight of the previous four data. At this time, the latest data is mainly determined by the current data. This improves the dynamic response performance of the system while maintaining the stability of the static data, which not only ensures the stability of the system but also increases the flexibility of the system.

[0022] Design ideas:

[0023] The present invention designs a photovoltaic panel target pose estimation visual recognition method based on the Gen6D strategy, so as to obtain the accurate pose of the photovoltaic panel and ensure that the robotic arm can accurately complete the grabbing task of the photovoltaic panel. The pose estimation is divided into three parts: photovoltaic panel detection, pose rough estimation, and pose refinement, each of which contains a convolutional neural network. The photovoltaic panel detection part inputs the picture obtained by the camera into the convolutional neural network 1, and finally outputs the central pixel coordinates and scaling value of the photovoltaic panel; the pose of the photovoltaic panel is roughly estimated by the convolutional neural network 2; after the calculation of the convolutional neural network 3, the refined photovoltaic panel pose is finally obtained. In order to accurately distinguish the photovoltaic panel during the photovoltaic panel detection process, the present invention designs a similarity discrimination method combining the color distribution histogram and the hash value in the HSV color space. In order to reduce the jitter problem of the bounding box, an adaptive weighted filtering algorithm is designed to filter the pixel coordinate values ​​of the corner points of the bounding box.

[0024] The beneficial effects of the present invention are as follows:

[0025] 1) The present invention adopts the Gen6D strategy, preloads weight parameters and information of the photovoltaic panel itself through the neural network, detects the posture information of the photovoltaic panel in the image, and then performs a weighted average operation on this posture and the posture data of the previous four times to obtain the final posture.

[0026] 2) In order to increase the accuracy of photovoltaic panel identification during photovoltaic panel detection, the present invention increases the secondary discrimination of image similarity between the cropped photovoltaic panel image and the reference image output by the convolutional neural network 2. At the same time, in order to reduce the judgment error, a similarity discrimination method combining the color distribution histogram and the hash value in the HSV color space is designed.

[0027] 3) In order to reduce the jitter problem of the bounding box, the present invention designs an adaptive weighted filtering algorithm to filter the pixel coordinate values ​​of the corner points of the bounding box. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the posture of the photovoltaic panel in the transportation state;

[0029] Figure 2 This is a block diagram of the photovoltaic panel pose estimation algorithm based on the Gen6D strategy of the present invention;

[0030] Figure 3 It is a flow chart of the image similarity secondary discrimination algorithm of the present invention;

[0031] Figure 4 It is a schematic diagram of the photovoltaic panel grabbing system platform of the present invention;

[0032] Figure 5 Bounding box corner pixel coordinate change map;

[0033] Figure 6 Coordinate change diagram of the bounding box corner points (static part);

[0034] Figure 7 Bounding box corner coordinate change diagram (dynamic part). DETAILED DESCRIPTION

[0035] In order to make the technical problems, solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific examples. Figure 4 As shown, it includes a photovoltaic panel installation robot 11, a robot gripping device 12, a photovoltaic panel 13, a bracket 14, a visual camera 1 15, and a visual camera 2 16. The robot gripping device 12 is arranged at the front end of the photovoltaic panel installation robot 11, and the visual camera 1 15 and the visual camera 2 16 are respectively arranged at the positions on both sides of the robot gripping device 12. The photovoltaic panel 13 on the chassis of the installation robot needs to be installed on the bracket 14.

[0036] In the embodiment, the data of the photovoltaic panel is first collected, and the photovoltaic panel is rotated and photographed with a camera to collect video, and the video is sent to the data preprocessing program to decompose it into reference pictures and save them, and then pose estimation and 3D model reconstruction are performed on all reference pictures. Then, the reconstructed 3D model is trimmed using CloudCompare software, and the redundant background parts are removed and marked. x Axis and Y The direction vector of the axis is saved in a txt file, and the trimmed model file is saved. Then the size information of the photovoltaic panel is manually measured and saved in a txt file. Finally, these data information are uniformly placed in the corresponding directory of the program to complete the data collection.

[0037] After collecting the data information, the Gen6D visual recognition algorithm is used to identify and estimate the poses of these photovoltaic panels respectively. Through secondary discrimination of image similarity and target pose estimation and tracking, the accurate poses of all the disorganized photovoltaic panels can be accurately detected and captured.

[0038] like Figure 2 As shown in the figure, a photovoltaic panel target pose estimation visual recognition method based on the Gen6D strategy is shown in the figure. The specific steps are as follows:

[0039] Step 1) Photovoltaic panel detection: Input the image obtained by the camera into the VGG-16 network, and calculate the similarity score map, coordinate offset map and scaling ratio based on the reference image generated in advance. Figure 3The group feature map is constructed, and the point with the highest score in the similarity score map is taken as the center of the photovoltaic panel. According to the position of this point, the corresponding offset and scaling ratio values ​​are selected on the coordinate offset map and scaling ratio map. Then, the coordinates of this point in the similarity score map and the offset are added to obtain the final photovoltaic panel center pixel coordinates, and finally the photovoltaic panel center pixel coordinates and scaling ratio values ​​are output.

[0040] Step 2) Rough pose estimation: According to the center pixel coordinates and scaling value of the photovoltaic panel, a square image of the target photovoltaic panel is cropped from the original image with a box and scaled to 128×128 size. It and all reference images are input into the VGG-16 network for rough pose estimation. The VGG-16 network calculates the similarity between the cropped image and each reference image, and selects the reference image with the highest similarity. The pose of the photovoltaic panel is roughly estimated based on this reference image.

[0041] In order to ensure the accuracy of photovoltaic panel recognition, after obtaining the reference image with the highest similarity from the VGG-16 network, a secondary similarity judgment was performed between the reference image and the cropped image of the original image. When the similarity is low, the cropped image is considered to be an incorrectly recognized image and is discarded.

[0042] Image similarity secondary discrimination method: The present invention designs a similarity discrimination method combining color distribution histogram and hash value in HSV color space, and its flow chart is as follows Figure 3 shown.

[0043] First, convert the two images to the HSV color space, divide the 0-180 hue values ​​into six groups under the H channel, and reassign the hue values ​​in each group to 15, 45, 75, 105, 135, and 165 to generate new H channel data. Then set the contrast area based on the position of the object in the image. Because the original image and the reference image are both centered on the photovoltaic panel, and the background images in the four corners are interference information, the mask1 area is set to a circular area with a radius of 60 with the center of the image as the center, and the mask2 area is set to a circular area with a radius of 30-45 with the center of the image as the center.

[0044] The mask1 area mainly contains the overall image of the object, and the mask2 area mainly contains the edge image of the object. Then, the color distribution histogram similarity comparison is performed under the H channel data in the mask1 area, and the hash similarity comparison is performed under the H channel data in the mask2 area. When both sets of similarity results meet the threshold, the query image is considered to be the correctly identified object, otherwise the query image is discarded. This similarity comparison method takes into account both the color similarity and the shape similarity of the objects in the image, and has higher accuracy.

[0045] Step 3) Pose refinement: The rough pose is sent to the pose refinement part of the network 3, and multiple reference images are selected and compared with the original image. Figure 1 The image is sent to the VGG-16 network, and after calculation by the VGG-16 network, the refined photovoltaic panel posture is finally obtained.

[0046] Step 4) Target tracking: After obtaining the position and posture of the photovoltaic panel, its 3D bounding box is displayed in the image. Finally, the photovoltaic panel is tracked as a single target and continuously tracked and identified.

[0047] In the target tracking stage, after obtaining the 3D bounding box of the photovoltaic panel and mapping it to the image, in order to reduce the jitter problem of the bounding box, it is necessary to filter the pixel coordinate values ​​of the corner points of the bounding box. The weighted average operation is performed on the previous four data and the current data to obtain the final coordinate value. The weight calculation method is as follows. First, set the initial weights 0, 1, 2, 3, and 4, and then divide each item by 2.5 to obtain 0, 0.4, 0.8, 1.2, and 1.6, then perform square operation and take the opposite number to obtain 0, -0.16, -0.64, -1.44, and -2.56. Finally, perform exponential operation with e as the base to obtain the final values ​​1, 0.85, 0.53, 0.23, and 0.08, which are the weights from the current time to the previous four times from the front to the back. The original filtering formula is shown in formula (1). is the result after filtering, , , , , is the real data collected for the current nth time and the previous times. Although this filtering method effectively reduces the jitter problem of the bounding box, it has a greater impact on dynamic data. When the camera or object moves faster, the bounding box cannot follow the object in time due to the larger weight values ​​of the previous four times, causing lag problems.

[0048] ;

[0049] In view of this deficiency, the present invention designs an adaptive weighted filtering algorithm. First, the weight selection rule is improved. The initial weights are set to 0, 1, 2, 3, and 4. Then, each item is divided by the standard deviation 1.414 of 0, 1, 2, 3, and 4 respectively to obtain 0, 0.707, 1.414, 2.12, and 2.83. Then, a square operation is performed and the opposite number is taken. Finally, an exponential operation with e as the base is performed to obtain the final values ​​1, 0.61, 0.14, 0.01, and 0.001, which are the weights of the current time to the previous 4 times from the front to the back respectively.

[0050] The previous divisor 2.5 is changed to 0, 1, 2, 3, 4 with a standard deviation of 1.414, resulting in new weights of 1, 0.61, 0.14, 0.01, 0.001. At the same time, an adaptive parameter is added , when the data changes, calculate the amount of change in the data , according to formula (2) to calculate the adaptive parameters ,in To adjust the parameter, this paper takes the value of 1.5. The adaptive weighted filtering algorithm formula is shown in formula (3). When the data is basically stable, the change range is small, and the change speed is slow, the adaptive parameter When it is close to 1, the weight of the previous four data is relatively large, and the latest coordinate data is obtained based on the current and previous four data. When the camera or object moves quickly, the data changes greatly, and the adaptive parameters It decreases rapidly, thereby reducing the weight of the previous four data. At this time, the latest data is mainly determined by the current data. This improves the dynamic response performance of the system while maintaining the stability of the static data, which not only ensures the stability of the system but also increases the flexibility of the system.

[0051] ;

[0052] ;

[0053] In the embodiment, a comparison is also made with the filtering algorithm. The present invention recognizes the position of the photovoltaic panel and collects the pixel coordinate sequence of the corner points of the photovoltaic panel bounding box that has not been filtered for a period of time (based on the pixel coordinate sequence of a corner point). u The original filtering algorithm and the improved adaptive filtering algorithm are applied to this coordinate sequence to compare the stability performance of the two filtering algorithms for static data and the following performance for dynamic data.

[0054] like Figure 5 The figure shows the coordinate change of the corner points of the bounding box. The object starts to move at about 9 seconds. The gray solid line is the original data without filtering, the red solid line is the data after filtering by the original filtering algorithm, and the black dotted line is the data after filtering by the improved adaptive filtering algorithm. In order to facilitate the observation of the results, the static part and the dynamic part are plotted separately for comparison.

[0055] like Figure 6The figure shows the coordinate change diagram of the static part. It can be seen from the figure that even if the object does not move, the original pixel coordinate data still fluctuates, with a maximum fluctuation of 0.79 pixels. After the original filtering algorithm is used, the data fluctuation is significantly improved, the curve becomes stable, and the maximum fluctuation is 0.56 pixels. After the data is filtered by the improved adaptive filtering algorithm, the maximum fluctuation is 0.61 pixels. The result is close to the result of the original filtering algorithm, and still has a good filtering effect.

[0056] like Figure 7 The coordinate change diagram of the dynamic part is shown. It can be seen from the figure that when the photovoltaic panel moves, the original data begins to change with time. The curve filtered by the original filtering algorithm lags behind the original data curve significantly. The maximum difference in pixel coordinates can reach 16.43 pixels, and the average error is 4.87 pixels. The dynamic following performance of the original filtering algorithm is poor. The curve filtered by the improved adaptive filtering algorithm is basically the same as the original data curve, with a maximum error of 3.88 pixels and an average error of 0.78 pixels, which is a small error. Therefore, compared with the original algorithm, the improved adaptive filtering algorithm has better dynamic following performance.

[0057] It can be seen that the photovoltaic panel pose estimation algorithm based on the Gen6D strategy takes into account both static and dynamic data through the improved adaptive filtering method, and has good static stability performance and dynamic following performance.

[0058] The above embodiment only represents one implementation of the present invention and cannot be understood as limiting the scope of the present invention. It should be pointed out that a person skilled in the art can make several variations without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. A photovoltaic panel target pose estimation visual recognition method based on Gen6D strategy, characterized in that: The steps include: 1) Photovoltaic panel detection: The picture taken by the camera, i.e. the original picture, is input into the convolutional neural network 1, and three sets of feature maps, namely the similarity score map, the coordinate offset map and the scaling ratio map, are calculated in combination with the reference picture generated in advance. The point with the highest score in the similarity score map is taken as the center of the photovoltaic panel, and the corresponding offset and scaling ratio value are selected on the coordinate offset map and the scaling ratio map according to the position of this point. Then, the coordinates of this point in the similarity score map and the offset are added to obtain the final pixel coordinates of the center of the photovoltaic panel, and finally the pixel coordinates and scaling ratio value of the center of the photovoltaic panel are output; 2) Rough estimation of pose: According to the center pixel coordinates and scaling value of the photovoltaic panel, a square image of the target photovoltaic panel is cropped out of the original image with a box and scaled to 128×128 size, and it and all the reference images are input into the convolutional neural network 2 of the rough pose estimation part; the convolutional neural network 2 calculates the similarity between the cropped image and each reference image, and selects a reference image with the highest similarity, and roughly estimates the pose of the photovoltaic panel according to this reference image; 3) Pose refinement: The rough pose is sent to the convolutional neural network 3, and multiple reference images are selected and sent to the convolutional neural network 3 together with the original image. After calculation by the convolutional neural network 3, the refined pose of the photovoltaic panel is finally obtained; 4) Target tracking: After obtaining the position and posture of the photovoltaic panel, a 3D bounding box of the photovoltaic panel is drawn through line segments based on the known size and position information of the photovoltaic panel and displayed in the image. Finally, the photovoltaic panel is tracked as a single target and continuously tracked and identified. In the step 2), in order to ensure the accuracy of photovoltaic panel recognition, after obtaining the reference image with the highest similarity from the convolutional neural network 2, the reference image and the cropped image of the original image are subjected to secondary similarity discrimination. The steps of the secondary discrimination method for image similarity are as follows: First, convert the two images to the HSV color space, divide the 0-180 hue values ​​into six groups under the H channel, and reassign the hue values ​​in each group to 15, 45, 75, 105, 135, and 165 respectively to generate new H channel data; Then, the contrast area is set based on the position of the object in the image. In the image, the mask1 area is set as a circular area with a radius of 60 and a center of the image, and the mask2 area is set as a circular area with a radius of 30-45 and a center of the image. The mask1 area is the overall image of the object, and the mask2 area is the edge image of the object. Then, a color distribution histogram similarity comparison is performed under the H channel data in the mask1 area, and a hash similarity comparison is performed under the H channel data in the mask2 area. When both sets of similarity results meet the threshold, the image is considered to be a correctly identified object, otherwise the image is discarded.

2. According to a photovoltaic panel target pose estimation visual recognition method based on Gen6D strategy according to claim 1, it is characterized in that: In the target tracking stage of step 4, after obtaining the 3D bounding box of the photovoltaic panel and mapping it to the image, in order to reduce the jitter problem of the bounding box, it is necessary to filter the pixel coordinate values ​​of the corner points of the bounding box; Perform weighted average calculation on the previous four data and the current data to obtain the final coordinate value; The weight calculation method is as follows: first set the initial weights to 0, 1, 2, 3, and 4, then divide each by the standard deviation of 1.414 of 0, 1, 2, 3, and 4, to get 0, 0.707, 1.414, 2.12, and 2.83, then perform square operations and take the opposite, and finally perform exponential operations with e as the base to get the final values ​​1, 0.61, 0.14, 0.01, and 0.001, which are the weights of the current time to the previous 4 times from the beginning to the end; At the same time, an adaptive parameter A is added d When the data changes, the change of data ΔC is calculated, and the adaptive parameter A is calculated according to formula (2) d ; Where P d To adjust the parameters; The adaptive weighted filtering algorithm formula is shown in formula (3): C f is the result after filtering, C n , C n-1 , C n-2 , C n-3 , C n-4 It is the real data collected for the current nth time and the previous times; When the data changes slowly, the adaptive parameter A d Close to 1, the weight of the previous four data is relatively large, and the latest coordinate data is obtained based on the current and previous four data; When the camera or object moves quickly and the data changes rapidly, the adaptive parameter A d becomes smaller, thereby reducing the weight of the previous four data. At this time, the latest data is determined by the current data.

3. The photovoltaic panel target pose estimation visual recognition method based on the Gen6D strategy according to claim 1 is characterized in that: The convolutional neural network 1, the convolutional neural network 2 and the convolutional neural network 3 all adopt the VGG-16 network; the VGG-16 network structure includes a convolutional layer, a nonlinear activation function and a pooling layer.

Images