A robot autonomous operation method, system, storage medium and electronic device

Abstract

The present application provides a robot autonomous operation method, system, storage medium and electronic device, which relates to the field of industrial control. The method includes: identifying a target object, determining an object area and a grasping posture corresponding to the target object; grasping the target object by applying the grasping posture in the object area; acquiring environmental information through a camera, generating a panoramic image, and determining the rough positioning of the target object in the panoramic image; acquiring surface information of the target object through point cloud data or a laser rangefinder, and determining the precise positioning of the installation point corresponding to the target object according to the rough positioning and the surface information; determining an operation reference coordinate system based on the precise positioning, and applying a robot to perform the installation operation of the target object based on the operation reference coordinate system. The present application can optimize the existing process without manual handling, thereby improving construction efficiency, improving installation quality, and enhancing construction safety.

Description

Technical Field

[0001] The present application relates to the field of industrial control, and in particular to a robot autonomous operation method, system, storage medium and electronic device. Background Art

[0002] Traditional strip installation mainly relies on manual labor. Construction workers move the strips to the designated position by installing auxiliary devices. One worker stands on the frame-type vertical strip machine to adjust the top position of the erected strips, and another worker stands on the ground to adjust the bottom position of the strips, thereby completing the positioning and installation of the partition board. However, manual installation is inefficient, and the workers have high labor intensity and high risk. In actual projects, the weight of a building strip is often more than 200kg. Building strips are large in size and weight. With the help of auxiliary facilities, it takes multiple people to cooperate to complete the installation manually. Manual assisted installation increases the difficulty of construction and the labor intensity of workers, and is prone to accidents. At the same time, with the development of the times, the emergence of high-rise buildings has led to the continuous increase in the height of interior decoration. The higher the height of the installed strips, the greater the labor intensity of the workers, and the worse the construction safety. At the same time, the construction quality is poor, the installation accuracy is low, and the strips are easily damaged during installation, affecting the construction progress. Summary of the invention

[0003] The purpose of this application is to provide a robot autonomous operation method, system, storage medium and electronic device, which can optimize the existing process by identifying the target object and using the robot autonomous operation, thereby improving construction efficiency, improving installation quality and enhancing construction safety.

[0004] In order to solve the above technical problems, this application provides a robot autonomous operation method, and the specific technical solution is as follows:

[0005] Identify the target object, and determine the object area and grasping posture corresponding to the target object;

[0006] Grasping the target object by applying the grasping gesture in the object area;

[0007] Acquire environmental information through a camera, generate a panoramic image, and determine a rough location of the target object in the panoramic image;

[0008] Acquire surface information of the target object through point cloud data or a laser rangefinder, and determine the precise positioning of the corresponding installation point of the target object according to the rough positioning and the surface information;

[0009] Based on the precise positioning, an operation reference coordinate system is determined, and based on the operation reference coordinate system, a robot is used to perform installation operations on the target object.

[0010] Optionally, before identifying the target object, the method further includes:

[0011] Calibrate the camera to determine the mapping relationship between the image and the actual object;

[0012] The external parameter matrix and the internal parameter matrix of the camera are determined according to the mapping relationship; the external parameter matrix is ​​used to represent the transformation relationship between the corresponding points in the world coordinate system and the camera coordinate system; the internal parameter matrix is ​​used to represent the transformation relationship between the corresponding points in the camera coordinate system and the image coordinate system.

[0013] Optionally, determining the object area and grasping posture corresponding to the target object includes:

[0014] Determine the object region to which the target object belongs using a convolutional neural network;

[0015] The grasping posture of the target object is determined using a generative grasping convolutional neural network.

[0016] Optionally, the determining the object region to which the target object belongs by using a convolutional neural network includes:

[0017] Input the collected image data, extract a feature map of the image data using a shared convolutional layer, and input the feature map into a region proposal network component;

[0018] In the region proposal network component, a convolution kernel is used to perform convolution sliding on the input image to generate a sliding window;

[0019] Generate a plurality of anchor boxes at the center point of each sliding window; the anchor boxes include different scales and different aspect ratios;

[0020] The feature map of the anchor box is passed to the fully connected layer to generate a corresponding feature vector, and the classification and positioning of the feature vector are determined by a normalized exponential function and a linear regression classifier;

[0021] Calculating the intersection-over-union ratio between the anchor point box and the manually calibrated true value;

[0022] Determine whether the detection target is included according to the intersection-and-union ratio; if the intersection-and-union ratio is less than a first intersection-and-union ratio threshold, determine it as a background area; if the intersection-and-union ratio is greater than a second intersection-and-union ratio threshold, it indicates that the detection target is effectively included;

[0023] If the IoU is not less than the first IoU threshold and not greater than the second IoU threshold, inputting linear regression to make the boundary value output by the anchor box approach the actual value, and taking the anchor box with a branch greater than the preset value as a suggestion area;

[0024] Through the region of interest pooling layer, feature extraction is performed on the selected area of ​​the anchor box and the feature map output by the region proposal network component, and after feature extraction, it is passed to the fully connected layer to generate a feature vector;

[0025] The feature vector is subjected to a normalized exponential function and linear regression to obtain a classification score of the region and an object region after regression adjustment.

[0026] Optionally, the determining the grasping posture of the target object by generating a grasping convolutional neural network includes:

[0027] Before grasping, a depth image without any objects is stored as a reference.

[0028] Encode the index of each bounding box of the object area and map the pixels in the bounding box to the same position in the blank depth image;

[0029] Cropping the depth image into a region image containing only the object of interest, and setting the region image to the same size as the input image of the generative grasping convolutional neural network;

[0030] The trained generative grasp convolutional neural network is used to generate the grasping posture with the highest quality score.

[0031] Optionally, acquiring environmental information through a camera, generating a panoramic image, and determining a rough position of the target object in the panoramic image includes:

[0032] Obtain fisheye images from the panoramic fisheye camera platform;

[0033] Using a generative network to perform distortion correction on the fisheye image to obtain a corrected image;

[0034] Converting the corrected image into a panoramic image;

[0035] Performing straight line detection on the panoramic image using Hough transform, and calculating vertex coordinates of the target object using the detected straight lines;

[0036] Coordinate transformation is performed on the vertex coordinates to obtain the rough positioning coordinates of the target object in the panoramic image.

[0037] Optionally, after converting the corrected image into a panoramic image, the method further includes:

[0038] The panoramic image is subjected to image preprocessing; the image preprocessing includes grayscale processing, binarization processing, connected region extraction and morphological processing.

[0039] Optionally, determining an operation reference coordinate system based on the precise positioning, and applying a robot to perform the installation operation of the target object based on the operation reference coordinate system includes:

[0040] Adjusting the verticality of the target object so that the bottom of the target object remains parallel to the ground;

[0041] Moving the target object to the precise position based on the coordinate origin in the operation reference coordinate system;

[0042] Performing verticality detection and flatness detection on the target object;

[0043] If the verticality detection and the flatness detection of the target object are both in compliance, the robot end effector is regulated by using a proportional integral differential algorithm to perform the installation operation of the target object.

[0044] The present application also provides a robot autonomous operation system, comprising:

[0045] An object recognition module is used to recognize a target object and determine an object area and a grasping posture corresponding to the target object;

[0046] An object grasping module, used for grasping the target object by applying the grasping gesture in the object area;

[0047] A coarse positioning module, used to obtain environmental information through a camera, generate a panoramic image, and determine the coarse positioning of the target object in the panoramic image;

[0048] A precise positioning module, used to obtain surface information of the target object through point cloud data or a laser rangefinder, and determine the precise positioning of the corresponding installation point of the target object according to the rough positioning and the surface information;

[0049] The robot autonomous operation module is used to determine the operation reference coordinate system based on the precise positioning, and to use the robot to perform the installation operation of the target object based on the operation reference coordinate system.

[0050] The present application also provides an electronic device, including:

[0051] Memory for storing computer programs;

[0052] A processor is used to implement the steps of the robot autonomous operation method as described in any one of the above items when executing the computer program.

[0053] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the steps of the robot autonomous operation method as described in any one of the above items are implemented.

[0054] The present application provides a robot autonomous operation method, including: identifying a target object, determining an object area and a grasping posture corresponding to the target object; grasping the target object by applying the grasping posture in the object area; acquiring environmental information through a camera, generating a panoramic image, and determining a rough positioning of the target object in the panoramic image; acquiring surface information of the target object through point cloud data or a laser rangefinder, and determining a precise positioning of an installation point corresponding to the target object according to the rough positioning and the surface information; determining an operation reference coordinate system based on the precise positioning, and applying a robot to perform installation operations on the target object based on the operation reference coordinate system.

[0055] This application identifies and grabs the target object, and generates a panoramic image based on environmental information to perform rough guidance of the robot's movement, thereby achieving accurate perception of the robot's position and posture detection, so that the robot can transport the target object to the installation point. Further use point cloud data or laser rangefinders to detect the target object, achieve precise positioning of the installation point, and perform the installation of the target object. It can be seen that this application can optimize the existing process without manual handling, thereby improving construction efficiency, improving installation quality, and enhancing construction safety.

[0056] The present application also provides a robot autonomous operation system, storage medium and electronic device, which have the above-mentioned beneficial effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0058] Figure 1 A flowchart of a robot autonomous operation method provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of the coordinate conversion principle provided in the embodiment of the present application;

[0060] Figure 3 A schematic diagram of the Faster RCNN model architecture provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of a target object recognition network architecture based on Faster RCNN provided in an embodiment of the present application;

[0062] Figure 5 A network architecture diagram for generating a grasping convolutional neural network provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of the FRCNN-GGCNN architecture provided in an embodiment of the present application;

[0064] Figure 7 This is a diagram of the overall architecture of the vision correction network provided in the embodiment of the present application;

[0065] Figure 8 A network architecture diagram is generated according to an embodiment of the present application;

[0066] Fig. 9 A diagram of the prediction network architecture provided in the embodiment of the present application;

[0067] Fig.10 A diagram of the discriminant network structure provided in an embodiment of the present application;

[0068] Fig.11 A schematic diagram of panoramic calibration provided in an embodiment of the present application;

[0069] Fig.12 A schematic diagram of the relative relationship between the working robot and the marker provided in the embodiment of the present application;

[0070] Fig.13 A schematic diagram of a weighted fusion region provided in an embodiment of the present application;

[0071] Fig.14 Schematic diagram of the PID control system provided in the embodiment of the present application;

[0072] Fig.15 A schematic diagram of vertical adjustment provided in an embodiment of the present application;

[0073] Fig.16 A schematic diagram of horizontal adjustment provided in an embodiment of the present application;

[0074] Fig.17 A schematic diagram of flatness adjustment provided in an embodiment of the present application;

[0075] Fig.18 A schematic diagram of the structure of a robot autonomous operation system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] Please refer to Figure 1 , Figure 1 A flowchart of a robot autonomous operation method provided in an embodiment of the present application, the method comprising:

[0078] S101: Identify a target object, and determine an object area and a grasping posture corresponding to the target object;

[0079] S102: Grasping the target object by applying the grasping gesture in the object area;

[0080] S103: Acquire environmental information through a camera, generate a panoramic image, and determine a rough location of the target object in the panoramic image;

[0081] S104: Acquire surface information of the target object through point cloud data or a laser rangefinder, and determine a precise positioning of a corresponding installation point of the target object according to the rough positioning and the surface information;

[0082] S105: Determine an operation reference coordinate system based on the precise positioning, and use a robot to perform installation work on the target object based on the operation reference coordinate system.

[0083] This embodiment can be divided into three main processes as a whole. The first part is to automatically grasp the target object, the second part is to guide the robot to move, and the third part is to install the target object. It should be noted that the target object is usually a regular structure, such as various prefabricated components, so that identification and grasping can be performed.

[0084] In the first part, the object detector can be used to identify and locate the object to be grasped, and a grasping plan can be formulated according to the object area selected by the detector to achieve end-to-end grasping posture generation based on deep learning, and complete the grasping or placement and then grasping of the target strip.

[0085] In the second part, image distortion can be corrected based on the generative adversarial network. The pixel units in the image can be mapped to the actual distance through panoramic scene calibration to generate a bird's-eye view and complete the panoramic stitching. The image can also be grayscaled, enhanced, binarized, morphologically processed and Hough transformed to extract laser lines, calculate the coordinates of the installation points, and drive the robot to achieve coarse-guided installation of the target object.

[0086] In the third part, the position and posture data of the target object is obtained based on the coarse positioning. After the collected point cloud or laser ranging data is analyzed and processed, the geometric parameters such as angle and distance are calculated for precise positioning calibration; the error signal is monitored and analyzed in real time through the proportional integral differential algorithm to adjust the output of the controller to ensure the rapid convergence of the structural installation error.

[0087] The following is a description of each part:

[0088] First, the recognition of the target object needs to be performed, which relies on the camera. However, during the operation of the camera, the assembly errors and distortions of components such as the lens will cause the image to not completely correspond to the real object. Therefore, the camera needs to be calibrated to correct the errors and lens distortion. Specifically, the camera can be calibrated to determine the mapping relationship between the image and the actual object, and then the intrinsic parameter matrix and internal parameter matrix of the camera are determined based on the mapping relationship. The external parameter matrix is ​​used to represent the conversion relationship between the corresponding points in the world coordinate system and the camera coordinate system; the internal parameter matrix is ​​used to represent the conversion relationship between the corresponding points in the camera coordinate system and the image coordinate system.

[0089] In this way, the camera is calibrated through mathematical methods, the mapping relationship between the imaging and the actual object is derived, and the camera's internal and external parameter matrix is ​​calculated.

[0090] This embodiment uses Zhang Zhengyou's calibration method as an example for calibration. Because the OpenCV library provides a program template for camera calibration, this application calibrates the camera by calling the calibration program to obtain the internal parameter matrix and distortion coefficient of each camera. Use a chessboard as the calibration plate, take 16 calibration plate pictures from different angles, ensure that the calibration plate can occupy most of the space in the camera's field of view, and calculate the camera matrix, distortion coefficient, rotation vector, displacement vector and other parameters. For the conversion relationship between the world coordinate system, camera coordinate system, and image physical coordinate system, see Figure 2 , Figure 2 A schematic diagram of the coordinate transformation principle provided in an embodiment of the present application.

[0091] Figure 2 In, O W Represents the origin of the world coordinate system, O C represents the origin of the camera coordinate system, O represents the origin of the image coordinate system. P1, P2, and P3 represent three non-collinear landmarks in the world coordinate system, and P1, P2, and P3 are their corresponding imaging points in the image coordinate system.

[0092] Assume that the camera coordinate system relative to the world coordinate system is described as , the image coordinate system relative to the camera coordinate system is described as ,but:

[0093] , formula 1;

[0094] , formula 2;

[0095] In formula 1 and formula 2, Represents the rotation matrix, which describes the rotation relationship between coordinate systems. It converts the direction of one coordinate system into the direction of another coordinate system, such as Represents the world coordinate system Relative camera coordinate system The rotation matrix transforms the direction in the world coordinate system to the camera coordinate system. Represents the camera coordinate system c relative to the image coordinate system The rotation matrix that transforms the orientation in the camera coordinate system into the image coordinate system.

[0096] and Represents the translation vector, which describes the displacement relationship between the origins of the coordinate system. Represents the translation of a point in the world coordinate system relative to the camera coordinate system. Represents the translation of a point in the camera coordinate system relative to the image coordinate system.

[0097] The primary task of the intelligent panel installation robot during construction operations is to obtain the basic distance and orientation information of the panels. In order to achieve target recognition and positioning of the panels, image target detection has become a key technology. In the field of target detection, the popularity of convolutional neural networks (CNN) has promoted the continuous development of target feature extraction and learning methods. Among them, the RCNN (Region-based Convolutional Neural Networks) series has become a representative method in target detection algorithms. The RCNN method has undergone continuous evolution, including RCNN, FastRCNN and FasterRCNN. This application may adopt the FasterRCNN algorithm that takes into account performance, speed and accuracy to achieve efficient recognition of target strips. FasterRCNN such as Figure 3 As shown in the figure, it consists of 5 convolutional layers, 3 pooling layers, 2 fully connected layers and 1 normalized exponential function, namely Softmax classifier. The convolutional layers use a 11*11, a 5*5 and three 3*3 convolution kernels respectively. The pooling layer uses the maximum value of 2×2 pooling to achieve 8 times down sampling and generate the corresponding feature map. The framework diagram of the strip recognition network based on FasterRCNN in this application is shown in Figure 4 As shown, Figure 4 Schematic diagram of the target object recognition network architecture based on FasterRCNN provided in an embodiment of the present application. Figure 4 In the process, the collected image needs to be processed through a shared convolution layer to obtain a feature map, and then output to the ROI pooling layer and the convolution layer.

[0098] The specific steps are as follows:

[0099] The first step is to input the collected image data, use the shared convolutional layer to extract the feature map of the image data, and then input the extracted feature map into the region proposal network component (RPN, Region Proposal Network).

[0100] The second step is to achieve multi-scale target detection. Figure 4 In the convolution layer shown, a 3*3 convolution kernel is used to convolve the input image to generate a sliding window, and 9 anchor boxes are generated at the center point of each sliding window. These anchor boxes cover three different scales (128*128, 256*256, 512*512) and three different aspect ratios (1:1, 1:2, 2:1).

[0101] The third step is to pass the feature map of the anchor box to the fully connected layer to generate the corresponding feature vector, and then classify and locate the target through the Softmax classifier and linear regression classifier. Then the intersection over union (IoU) between the anchor box and the manually calibrated true value is calculated to determine whether the detected target is included.

[0102] If the intersection-over-union ratio is less than the first intersection-over-union ratio threshold, it is determined to be a background area; if the intersection-over-union ratio is greater than the second intersection-over-union ratio threshold, it indicates that the detection target is effectively contained;

[0103] If the IoU is not less than the first IoU threshold and not greater than the second IoU threshold, a linear regression is input to make the boundary value output by the anchor box approach the actual value, and the anchor box with a branch greater than the preset value is used as a suggested region. The first IoU threshold and the second IoU threshold are not limited here, and can be set to 0.3 and 0.7 respectively, for example.

[0104] At this time, if IoU is less than 0.3, it indicates that the part is the background area; if IoU is greater than 0.7, it means that the detected target is effectively contained. If its attributes cannot be determined, that is, not less than 0.3 and not greater than 0.7, input linear regression can make the boundary value output by the anchor box approach the actual value of manual annotation. In order to simplify the positioning box, the positioning box with a higher score can be selected as the recommended area.

[0105] Step 4: Through the ROI (Region of Interest Pooling) pooling layer, feature extraction is performed on the selected area and the feature map output by the RPN model, and then it is passed to the fully connected layer to generate a feature vector. Finally, through the Softmax classifier and linear regressor, the classification result of the region and the target frame area after regression adjustment, that is, the detection result, are obtained. The ROI pooling layer is a key component in the target detection network, which is used to extract fixed-size feature regions from the convolutional feature map.

[0106] In the object detection task, different input images may cause the object detection model to have inconsistent predictions about the existence of the object box. At the same time, the area in the true value box may or may not contain the object to be detected. Therefore, when calculating the recognition accuracy, you can perform recognition accuracy detection, and at this time, you need to count the data of the following four situations:

[0107] TruePositive (TP): The number of times the object detection model generates a prediction box and there is an object to be detected at that location.

[0108] FalsePositive (FP): The number of times the object detection model generates a prediction box, but there is no object to be detected at that location.

[0109] TrueNegative (TN): The number of times the target detection model does not generate a prediction box and there is actually no target to be detected at that location.

[0110] FalseNegative (FN): The number of times the object detection model does not generate a prediction box, but there is an object to be detected at that location.

[0111] Based on the above four data, two key indicators of the target detection algorithm, namely the accuracy P and the recall R, can be calculated. The calculation method is shown in Formula 3 and Formula 4:

[0112] , formula 3;

[0113] , formula 4;

[0114] In the above formula, the accuracy P is used to indicate the probability that the predicted box successfully selects the target during the target detection process, and the recall R is used to indicate the probability that the target to be detected is successfully detected during the target detection process.

[0115] In target detection, the same object may be identified as different labels by the model. Usually, the model's classifier will assign different classification confidences to the object under different labels, indicating the possibility that the object belongs to a certain label. The value of the classification confidence is between 0 and 1. In order to determine whether to predict the object, all classification confidences are sorted and a threshold is set. If the highest classification confidence is higher than the threshold, it means that the object needs to be predicted, and the predicted target label is the label corresponding to the classification confidence. Conversely, if the highest classification confidence is lower than the threshold, no prediction is required.

[0116] The embodiments of the present application can use a Generative Grasping Convolutional Neural Network (GGCNN) to directly generate grasping postures at the pixel level through depth images without sampling and classifying individual grasping candidates first. This makes the GGCNN very flexible in grasping general tasks. The network architecture of the GGCNN is as follows: Figure 5 As shown, Figure 5 The network architecture diagram of the generative grasping convolutional neural network provided in the embodiment of the present application, taking a 300*300 depth map as an example, through 3*3, 5*5 and 9*9 convolutions, and 3*3 transposed convolutions, the grasping quality, grasping width and grasping angle can be output respectively. However, when responding to specific needs in real environments, the generative grasping convolutional neural network may be subject to some limitations. For example, when the target object appears in the camera's field of view at the same time as other objects in the environment, it is difficult for the generative grasping convolutional neural network to select predefined objects in a targeted manner according to actual needs. FasterRCNN can identify and locate objects of interest in complex scenes, making up for the shortcomings of the GGCNN algorithm.

[0117] Therefore, the embodiment of the present application can combine GGCNN with FasterRCNN to construct a two-step cascade structure, namely FRCNN-GGCNN. In this structure, FasterRCNN is used to identify the target strip to be tested, generate a bounding box to remove irrelevant areas in the image, and generate areas of interest for the generative grasping convolutional neural network. Subsequently, the depth images of these areas of interest are input into the generative grasping convolutional neural network to identify the optimal grasping position of the target object. The schematic diagram of FRCNN-GGCNN is shown in Figure 6 As shown, Figure 6 This is a schematic diagram of the FRCNN-GGCNN architecture provided in the embodiment of the present application, where FasterRCNN is responsible for target recognition and positioning, while the generative grasping convolutional neural network is responsible for generating the optimal object grasping posture. In order to realize the grasping process, the specific implementation steps of the proposed FRCNN-GGCNN detection structure are as follows:

[0118] Step 1: Before grabbing, store a depth image that does not contain any objects as a reference, that is, a blank depth image.

[0119] Step 2: Encode the index of each bounding box in the FasterRCNN output image and map the pixels within its box to the same position in the blank depth image.

[0120] Step 3. Crop the depth image to contain only the object region of interest and ensure that it has the same size as the input image for the generated grasping convolutional neural network.

[0121] Step 4: Use the trained generative grasp convolutional neural network to generate the grasping posture with the highest quality score.

[0122] After that, the coordinate system can be transformed to obtain the grasping plan in the robot coordinate system, and the automatic grasping of the target object can be completed according to the grasping plan.

[0123] The following is an explanation of the mobile guidance of the execution robot. First, a panoramic fisheye camera platform needs to be built. In this application, four fisheye cameras of the same specifications can be fixed on the body of the autonomous working robot at four positions (for example, 45°, 135°, 225°, 315°), and the orientation of each fisheye camera is perpendicular to the plane on which it is installed, and each camera has an independent operation, shooting and storage environment. To ensure that the four cameras can be synchronized when shooting the same scene, each camera is equipped with a wireless control device, and the four wireless control devices are connected to the same remote control. The operation on the remote control can be transmitted to the four cameras in real time, so that the four cameras can complete the 360-degree no-dead-angle image acquisition of the current scene at almost the same time. It should be noted that in other embodiments of the present application, other numbers of fisheye cameras can also be used, and partially different or completely different installation orientations can be used, which should also be within the scope of protection of the present application.

[0124] Traditional pinhole cameras have limited field of view (generally about 40-50 degrees), which makes it difficult to meet the needs of all-round information collection. The field of view of fisheye cameras can reach more than 180 degrees, and can obtain information far beyond the field of view of traditional pinhole cameras. However, the images taken by fisheye cameras do not conform to the ideal assumptions of pinhole cameras, and usually show obvious radial distortion. Current work on fisheye image correction usually requires calibrating the internal and external parameters of the camera by collecting images from multiple perspectives. However, it may be difficult to collect multi-perspective images of a moving scene.

[0125] This application improves the correction effect and quality of the model for fisheye images through the generation network, parameter prediction network and discriminant network, and realizes end-to-end image correction. The network structure is as follows Figure 7 As shown, Figure 7 The overall architecture diagram of the vision correction network provided in the embodiment of the present application shows that the fisheye image is generated through the generation network and parameter prediction to obtain the generated image, and then combined with the real image to obtain the discriminant network.

[0126] The generated network is improved based on the U-NET network, such as Figure 8 As shown, Figure 8This is a diagram of the generated network architecture provided by an embodiment of the present application. The size of the input image is 256×256×3. After entering the network, the first layer uses a 3×3 convolution kernel to map the three-dimensional features to sixty-four dimensions, the step size, i.e., stride, is 1, and the padding is 1. Next, the high-dimensional features obtained in the first layer are processed by a three-layer residual network, and then downsampled by a convolutional network. The size of the convolution kernel is 2×2, the step size is 2, and the output dimension is 128. Subsequently, the ReLU activation function is used as the input for the next downsampling, and downsampling is performed again to convert the output dimension to 256, and then the output dimension is increased to 512 after another downsampling. At the same time, a part of the features are extracted during the downsampling process as a connection with the corresponding dimension of the decoder to form a jump connection in the network. Compared with the original U-Net network, this application replaces the maximum pooling layer with a convolution layer with a step size of not one, because the maximum pooling layer is irreversible, which may cause the spatial relationship between the target pixel and its surrounding pixels to be lost.

[0127] The parameter prediction network is embedded in the Generative Adversarial Networks (GAN), which takes the high-dimensional feature map output by the generative network as input to estimate the distortion parameter γ of the image captured by the fisheye camera, and then passes γ to the distortion parameter attention module of the generative network decoder to predict the distortion parameter. The network structure is as follows Fig. 9 As shown, Fig. 9 This is a diagram of the prediction network architecture provided by the embodiment of the present application. After the fisheye image enters the generation network, parameter prediction is performed to obtain the generated image. The generated image and the real image are then sent to the discriminant network.

[0128] The main task of the discriminant network is to distinguish the corrected fisheye image generated by the generative network and the real background image in the dataset. The network structure is as follows: Fig.10 As shown, Fig.10 This is a diagram of the structure of the discrimination network provided in an embodiment of the present application.

[0129] The discriminant network receives the image generated by the generative network and the real image, where the label of the real image is 1 and the label of the generated image is 0. This step performs binary classification on the input image to determine the authenticity of the input image.

[0130] When judging the authenticity of the input fisheye image, the corresponding loss can be calculated. The prediction network selects the loss value between the predicted parameters and the true parameters of the L1 loss function, aiming to reduce the penalty term for outliers. Where N represents the number of samples drawn from the training sample, Indicates samples, P pre represents the prediction parameter, P gt represents the real parameter, L Prepresents the prediction loss, as shown in Formula 5.

[0131] , Formula 5;

[0132] The L1 loss is selected as the loss function of the generative network, as shown in Formula 6 below, where Represents the reconstructed image, that is, the corrected image. Represents the background image, which is the real image used as a comparison benchmark. Represents the reconstruction loss, which is the difference between the reconstructed image and the real image. The reconstruction effect of the generated network is measured by averaging the L1 loss (absolute value difference) of each pixel. This loss function calculates the L1 loss for each pixel in the channel for the corrected image and the background image.

[0133] , Formula 6;

[0134] The Sigmod function is used as the activation function:

[0135] , Formula 7;

[0136] In Formula 7, K represents the number of categories. The Softmax function maps an input vector to a dimensional probability distribution, where each value represents the probability that the input belongs to a certain category. Refers to the index, which is used to traverse all categories. The value range is 1 to , which indicates the exponential calculation of each category when performing Softmax normalization.

[0137] For the binary classification problem, when the output value y is 1, the probability formula is shown in Formula 8; when the output value y is 0, the probability formula is shown in Formula 9:

[0138] , formula 8;

[0139] , formula 9;

[0140] In formula 8 and formula 9, is a linear combination, represents the parameter vector of the logistic regression model, including the weight corresponding to each feature, x represents the input feature vector, that is, Formula 8 represents the weight after the model The result after linear combination.

[0141] Hyperparameters for the total loss function in the generator network , which is kept at 0.001 during the entire training process. The adaptive moment estimation (Adam) optimizer is used during the training process. This optimizer has the characteristics of fast convergence and can quickly find the minimum value of the loss function. The learning rate in this application is set to , and a strategy of dynamically adjusting the learning rate is adopted during the training process. The training rounds (epochs) of the model are set to 50, because the model basically achieves convergence after 30 epochs and the loss has stabilized. The batch size (batch-size) in each epoch is 16, that is, the number of images input into the network each time is 16.

[0142] After distortion correction is completed, the corrected image is converted into a bird's-eye view, which includes three steps: panoramic scene calibration, bird's-eye view generation, and panoramic image stitching.

[0143] When calibrating the panoramic scene, four calibration plates are placed in front of, behind, to the left and to the right of the vehicle. The calibration plates are parallel to the vehicle and are 10 to 50 cm away from the vehicle. The distance between the two calibration plates in front and behind the vehicle is taken as the vehicle length, and the distance between the calibration plates on the left and right sides is taken as the vehicle width. Fig.11 As shown, Fig.11 The panoramic calibration schematic diagram provided in the embodiment of the present application can calculate the inverse perspective transformation matrix according to formula 10.

[0144] , formula 10;

[0145] Among them, (T x ,T y ) is the pixel coordinate of the image after distortion correction; (T u ,T v ) are the pixel coordinates of the image after inverse perspective transformation; U, V are the width and height of the target bird's-eye view; W, H are the actual width and height corresponding to the bird's-eye view.

[0146] After obtaining the inverse perspective transformation matrix, the distortion-corrected images are converted into their respective corresponding bird's-eye view images according to Formula 11 to generate a bird's-eye view.

[0147] , Formula 11;

[0148] , Formula 12;

[0149] in, Represents the elements of the inverse perspective transformation matrix, which defines how to map the pixels of the source image to the target image through perspective transformation. (x src , y src ) is the pixel coordinate in the distortion-corrected image; (x dst , y dst) is the pixel coordinate in the bird's-eye view image; T is the inverse perspective transformation matrix.

[0150] After obtaining the bird's-eye view corresponding to the fisheye images of the four perspectives, the four bird's-eye view images need to be stitched together to form a panoramic image, and then the stitched images are fused to achieve a more balanced effect in terms of brightness, saturation, and contrast. This application uses a marker-based stitching algorithm to stitch images. The specific process is as follows:

[0151] The first step is to determine a number of fisheye images containing different perspectives;

[0152] The second step is to determine two landmark points of each common area in the fisheye image;

[0153] Step 3: Use the straight line formed by the two marking points as a stitching line;

[0154] Step 4: stitching the fisheye images of different viewing angles based on the stitching lines to obtain a panoramic image.

[0155] There are some overlapping areas in the images taken by the fisheye camera. In this step, markers are set in these four common areas, and each camera image is mapped to the position in the panorama to achieve image stitching. Two markers are set in each common area, and these two markers form a straight line. Since the straight line still maintains its straight line properties after distortion correction and top view transformation, it can be used as the stitching line in the image stitching operation. In the vehicle body coordinate system, the relative coordinates of the 8 markers A1, A2, B1, B2, C1, C2, D1 and D2 are as follows: Fig.12 As shown, Fig.12 This is a schematic diagram of the relative relationship between the working robot and the marker provided in the embodiment of the present application.

[0156] The stitched panoramic image has obvious stitching seams, which need to be fused. This application adopts a distance-based weighted average fusion algorithm. In the overlapping area, the grayscale value of the fused image is obtained by weighted average of the grayscale values ​​of the two images. Fig.13 As shown, Fig.13 This is a schematic diagram of the weighted fusion area provided in an embodiment of the present application. In the left front common area, the shaded portion is the weighted fusion area, that is, the overlapping area.

[0157] According to formula 13, the gray value of the weighted fusion area of ​​the fused image in the left front common area is calculated:

[0158] , Formula 13;

[0159] Among them, I(x,y) represents the gray value of the pixel at the coordinate (x,y) of the fused image, and I l(x,y) represents the grayscale value of the pixel at the coordinate (x,y) of the left view image. r (x, y) represents the grayscale value of the pixel at the coordinate (x, y) of the front view image, λ l and λ r are the weight coefficients of the left view image and the front view image respectively, and satisfy Formula 14. Weight coefficient λ l and λ r The weight coefficient of the pixel point can be dynamically adjusted as the pixel point changes, and the weight coefficient of the pixel point can be determined by calculating the Euclidean distance.

[0160] , Formula 14;

[0161] After a series of steps including fisheye camera calibration, distortion correction, and image stitching and fusion, a 360-degree panoramic bird's-eye view of the robot's surroundings was successfully obtained, i.e., a panoramic image was generated.

[0162] Thereafter, the obtained image can be further preprocessed by graying, binarizing, connecting region extraction, morphological processing and other operations to ensure accurate extraction of the installation positioning laser line.

[0163] First, grayscale the image. In an RGB image, the three components R, G, and B have different importance in the grayscale image. Different weights are used to perform weighted average processing on these three components, and the obtained value is used as the grayscale value of the grayscale image, as shown in Formula 15. Let W r , W g , W b are the weight values ​​of R, G, and B, respectively, satisfying:

[0164] , Formula 15;

[0165] , Formula 16;

[0166] Generally speaking, the human eye is most sensitive to green, followed by red, and least sensitive to blue. Therefore, the best grayscale image can be obtained by calculating formula 17: .

[0167] , Formula 17;

[0168] Image enhancement is then performed. Since there are many uncertain factors when the operating robot is working at the construction site, this will cause various types of noise to be generated during the image transmission process, affecting the imaging effect. Therefore, it is necessary to perform noise reduction processing on the image to facilitate the subsequent extraction of target features. Considering that the images collected by this application lack complicated details, an improved adaptive median filter is used for image processing. This filtering method reduces noise while causing less blurring of the original image, thereby maintaining the integrity of the edges. The specific steps are as follows:

[0169] (1) Establish the neighborhood area. Select a 3*3 square area around a certain pixel point as the center. Based on the statistical sorting principle, arrange the 9 pixel values ​​of the 8 neighborhoods centered on the point (x, y) from small to large and calculate the median value.

[0170] (2) Sort the grayscale values ​​of each pixel in the region. Select the middle value after sorting as the new grayscale value of the center pixel, and apply the calculated pixel median to the median filter response at the point (x, y).

[0171] Finally, binarization is performed. In order to perform installation positioning, the target information (laser line used for installation positioning) needs to be distinguished from other background information so that only the target information can be processed to reduce the amount of calculation. According to the difference in grayscale characteristics between the laser line and the background information, they can be segmented through image binarization. The process of image binarization is to set the grayscale value of pixels below the threshold to 0 and the grayscale value of pixels above the threshold to 255 according to the preset threshold, so as to generate a black and white contrast image. The mathematical expression of image binarization is formula 18, where T represents the threshold.

[0172] , Formula 18;

[0173] Morphological processing includes two operations, dilation and erosion, which are mainly aimed at the highlight part in the image, that is, the white area after the laser line image is binarized. This application first performs dilation and then erosion, with the purpose of closing narrow gaps and eliminating smaller holes.

[0174] The dilation operation aims to expand the boundaries of the bright parts of the image, making the lines wider, and also has the effect of connecting adjacent bright areas with shorter distances. Use a 3×3 convolution kernel B to convolve the processing area A. The value of the pixel to be processed is the maximum value of the pixel in the area covered by the convolution kernel B. The highlight area in the image is enlarged. The algorithm can be expressed by formula 19:

[0175] , Formula 19;

[0176] Among them, A represents the area to be processed, ⊕ represents the dilation operator, and B is the convolution kernel of the dilation operation.

[0177] In contrast to the dilation operation, the erosion operation reduces the boundaries of the highlighted parts of the image and can eliminate some smaller highlighted areas. The erosion operation uses a 3×3 convolution kernel B to perform convolution with the area to be processed A. The value of the pixel to be processed is the minimum value of the pixel in the area covered by the convolution kernel B. The highlighted area in the image is reduced. The algorithm can be expressed by Formula 20.

[0178] , formula 20;

[0179] Where A represents the area to be processed, represents the erosion operator, and B is the convolution kernel of the erosion operation.

[0180] The following uses a strip board as an exemplary target object to illustrate the installation coordinate coarse positioning process based on vehicle drive.

[0181] In this step, the Hough transform is used to detect straight lines in the preprocessed image, and the coordinates of the installation target vertices are calculated through the detected straight lines. Then, the final positioning coordinates of the vehicle in the panoramic image are calculated through coordinate transformation to achieve the rough positioning of the strip installation.

[0182] After acquiring the laser line, the present application uses an improved Hough transform to detect the straight line and roughly locate the intersection point. The Hough transform transforms the straight line detection problem into a point detection problem in a polar coordinate system, which is expressed by using the polar radius r and the polar angle θ, as shown in Formula 21.

[0183] , Formula 21;

[0184] Since the cumulative probability Hough transform (HoughLinesP) has a small amount of calculation, high execution efficiency, and can obtain the coordinates of the two endpoints of the straight line segment, the cumulative probability Hough transform is chosen to detect the parking angle straight line segment in this application.

[0185] After that, the laser point coordinate calculation is performed. The straight line segments detected by the cumulative probability Hough transform are screened and expressed in vector form, and the longest straight line segment is determined as a valid straight line segment. Subsequently, the sequence of arranged straight line segments is traversed in order, and the angles between them and the valid straight line segments are calculated until the angle is between 80° and 100°. Finally, two valid straight line segments and their endpoint coordinates are obtained, so that the intersection coordinates can be calculated.

[0186] For the cumulative probability Hough transform detection, all line segments are tested. The length of each line segment is calculated and sorted from largest to smallest. The longest straight line segment is determined to be a valid straight line segment. Subsequently, the sequence of straight line segments is traversed in order, and their angles with the valid straight line segments are calculated until the angle is between 80° and 100°, which can be calculated by formula 23. The straight line segments determined in this way are also considered to be valid straight line segments. Finally, two valid straight line segments and their endpoint coordinates are obtained, and the intersection coordinates can be calculated. The two laser lines are represented in the form of vectors. Suppose the endpoint coordinates of the two valid straight line segments are (x s1 ,y s1 )、(x t1 ,y t1 ) and (x s2 ,y s2 )、(x t2 ,y t2 ), which is shown in Formula 22:

[0187] , formula 22;

[0188] The angle is calculated by converting radians into degrees using formula 23:

[0189] , formula 23;

[0190] In step S104, surface information of the target object is obtained through point cloud data or a laser rangefinder, and a precise positioning of a corresponding installation point of the target object is determined according to the rough positioning and the surface information.

[0191] If a laser rangefinder is used, a system can be built using multiple laser rangefinders to simulate the process of measuring flatness with a ruler to detect small height differences in the plate. These laser rangefinders are fixed at different positions of the actuator to measure the distance of the plate (the rangefinders must be located in the same plane) and transmit the distance data to the central processing unit via Bluetooth. The system analyzes the data to evaluate the flatness of the plate surface and generates a corresponding analysis report. The report is compared with the specification to determine whether the plate installation meets the requirements.

[0192] If laser point cloud is used, a laser scanner or 3D camera is used to collect the point cloud data of the strip. Point cloud data is unstructured points, and it needs to be segmented first to identify the target strip and remove the background points. The key algorithms related to point cloud data processing are detailed below:

[0193] One is the region growing algorithm. The region growing algorithm is based on the proximity between points and the similarity of local surface features, such as flatness or roughness, calculated by normal vectors and curvature. All points are sorted by curvature value, starting with the point with the smallest curvature value as the seed point. The seed point is usually located in a flat area, and growing around it can reduce the amount of calculation. Finally, points with similar properties to the seed point are classified into the same cluster.

[0194] When fitting and classifying the plane, the RANSAC algorithm is used to estimate the optimal plane parameters for each segment. The RANSAC algorithm can also fit in the presence of a high proportion of outliers (>50%), and its required parameters include the minimum number of points, distance threshold, and number of iterations. In this study, the minimum number of points is the minimum number of points for effective plane clustering determined by the region growing algorithm; the distance threshold corresponds to the accuracy of the sensor to obtain the point cloud data; the number of iterations is set to 500 to ensure the robustness and efficiency of the fitting results. After RANSAC, the mathematical expression of the fitted plane, the unit normal vector, and the points contained are obtained. Subsequently, the ground and the slats are classified based on the characteristics of the normal vector.

[0195] In open-loop control, the output of the controller is not affected by the system output, but only depends on the pre-set input signal. This makes the open-loop control system very sensitive to external disturbances and system errors. Correspondingly, in a closed-loop control system, the system output is measured and compared with the desired reference signal, and then the system input is adjusted based on the comparison result so that the system output reaches the desired target. The closed-loop control system controls and regulates the system behavior by constantly monitoring and adjusting the system state, making the system stable and robust.

[0196] The embodiment of the present application uses the proportional integral differential algorithm, that is, the PID (Proportional Integral Differential) control algorithm as an example to control the robot end effector. By adjusting the three parameters of proportion, integration and differentiation, a satisfactory control effect can be achieved. In specific control, different parameter combinations are selected according to different control objects. See Fig.14 , Fig.14 This is a schematic diagram of the PID control system provided in an embodiment of the present application. Fig.14 Where r(t) is the set value of the actuator movement, e(t) is the deviation between the actual value and the set value, P, I, and D are the coefficients of each link, corresponding to the proportional link, integral link, and differential link, respectively.

[0197] As can be seen from the above, this application realizes the automatic grasping and placement of strips based on deep learning. The process involves steps such as strip recognition, grasping posture planning, grasping execution, transportation and placement.

[0198] In this process, if the two-step cascade algorithm (FRCNN-GGCNN) proposed in this application is applied, end-to-end autonomous recognition and grasping of target objects can be achieved. The target object is identified through the Faster R-CNN model, the image of the target object is input, the bounding box information of the target object is output, and the basic distance and spatial positioning of the target object are provided. Secondly, the GGCNN model is introduced to generate grasping postures at the pixel level based on the depth image output by Faster R-CNN, so as to achieve high-precision grasping operations and improve the system's adaptability to complex environments.

[0199] Vehicle control is easily disturbed by a variety of internal (mechanical errors, sensor errors, electronic control errors, communication delays, etc.) and external factors (environmental factors), resulting in an increase in cumulative errors; and it is difficult to collect multi-view images in mobile scenes to achieve effective correction of fisheye images. To solve this problem, this application adopts a real-time correction solution for fisheye images based on a generative adversarial network (GAN), using coarse positioning sensors to achieve accurate perception and pose estimation of the vehicle position, thereby effectively transporting the strips to the installation target point. Images with higher correction quality are generated through the generative network; the discriminative network identifies the generated correction images, thereby prompting the generative network to generate more realistic correction results; the parameter prediction network is responsible for predicting the key parameters of the correction model, which enhances the universality of the system and enables it to adapt to a variety of mobile scenes.

[0200] Fisheye cameras are particularly suitable for crowded and narrow spaces, such as construction sites, because of their wide-angle field of view, which can provide a wider monitoring range. Their powerful monitoring capabilities enable them to fully observe the surroundings of the vehicle, including blind spots. In such a complex environment, fisheye cameras can not only detect obstacles more easily, but also play a key role in the identification and positioning of laser lines. In the present invention, fisheye cameras are preferred because pinhole cameras are more expensive, have a narrow field of view, and are larger in size. At the same time, the use of multiple pinhole cameras can also generate a bird's-eye view, which should also be within the scope of protection of this application.

[0201] Based on the rough positioning, this application uses point cloud or laser ranging data for analysis and processing, and uses the reference strip as the reference to calculate geometric parameters such as angle and distance for accurate positioning calibration, and drives the actuator to achieve precise positioning of the strip installation. Finally, the proportional integral differential algorithm can be used to adjust the output parameter feedback actuator to achieve high-precision installation control.

[0202] In the actual application of this application, in order to achieve precise adjustment of the target object, two solutions can be adopted for the layout of point cloud sensors (such as structured light cameras, line laser scanners, etc.): one is to install the sensor outside the vehicle, and the other is to install the sensor on the vehicle. After coarse positioning to the installation target point, the clamp drives the plate hydraulically or electrically to achieve precise adjustment. The position arrangement of the sensor is crucial to the accuracy and stability of the adjustment process. When the sensor is installed outside the vehicle, it can sense the position and posture of the plate more directly, but its positioning efficiency may be reduced by the stability of the sensor layout; when the sensor is installed on the vehicle, it can track the position and posture of the vehicle more accurately, but it may be affected by the external environment and reduce the positioning accuracy.

[0203] According to the advantages and disadvantages of the two solutions, when applying this application, the most suitable solution can be selected for a specific application scenario to ensure the accuracy and stability of the plate adjustment process.

[0204] A system built with multiple laser distance meters can simulate the process of measuring flatness with a ruler and achieve similar results. These laser distance meters can be fixed at different positions of the actuator to measure the distance of the plate, and by comparing the distance data at different positions, the slight height difference on the surface of the plate can be detected to determine its flatness.

[0205] In the process of applying the robot to perform the installation operation of the target object based on the operation reference coordinate system, the present application can further adopt an error control algorithm to improve the control accuracy and stability of the end effector to ensure the rapid convergence of the installation error. Through closed-loop control, the output of the system is compared with the expected reference signal, and the input of the system is adjusted according to the comparison result to achieve control and regulation of the system behavior, so that the system has stability and robustness. Taking the PID algorithm as an example, the present invention realizes the closed-loop control of the current error through proportion (P), realizes the cumulative error control within the past period of time Tn through integration (I), and realizes the predicted cumulative error control within a future period of time Tv through differentiation (D). In addition, other related control algorithms such as fuzzy control and adaptive control are also covered in the protection scope of the present invention.

[0206] In particular, in the actual application process of this application, network cable communication can be used as the main communication method to achieve stable and high-speed data transmission. Network cable communication has high-speed and stable data transmission and relatively low hardware cost, which is suitable for application scenarios with high communication rate requirements and limited budget, but is affected by wiring distance limitations and wiring complexity. At the same time, single-chip microcomputer communication, ROS system communication and other common communication methods can also be used to provide more flexible communication options. Single-chip microcomputer communication has powerful hardware control capabilities and stability, which is suitable for applications with high real-time requirements, but the development cycle is long and the function is limited. ROS system communication has the characteristics of strong openness and high flexibility, which is suitable for a variety of robot applications, but requires higher computing and memory resources. The selection of a suitable communication method should be determined according to the specific application requirements and hardware environment.

[0207] Based on the above embodiment, as a preferred embodiment, if the target object is a plate, in order to ensure that the plate can be accurately and stably installed at the target position, it is crucial to specify the degree of freedom calibration sequence when the plate is installed. The following is an explanation of the target object detection and correction sequence process taking the plate as an example:

[0208] First, adjust the verticality of the plate. Adjust Rx and Ry to make the bottom of the plate parallel to the ground. Rx is the direction of the bottom of the plate parallel to the wall, and Ry is the direction of the bottom of the plate perpendicular to the wall. Next, adjust Dx and Dy based on the coordinate origin O, and then adjust Dz up and down along the Rz direction, that is, the direction perpendicular to the ground, and move the plate to the coordinates of the installation point to complete the verticality calibration. Then, adjust Rz based on the coordinate origin O to make the target plate parallel to the reference plate to complete the flatness calibration.

[0209] The vector angle can be used to indicate the closeness between different vectors. This application uses basic trigonometric principles to detect the verticality during installation. After the RANSAC algorithm, the normal vector representation of the strip surface and the ground is obtained. Fig.15 As shown, Fig.15 A schematic diagram of vertical adjustment provided in an embodiment of the present application, where n0 is the ground normal vector, and n i is the target strip plane fitting normal vector, is the verticality deviation of the target strip; α i represents the vertical angle deviation of the target strip, β i For v0 and v i The angle between.

[0210] According to and complete angle α i and vertical deviation The adjustment of Rx can be completed after calculation:

[0211] , Formula 24;

[0212] , formula 25;

[0213] , Formula 26;

[0214] In the above formula, Ry is used to ensure that the slats are parallel to the ground. The detection is focused on the bottom edge of the target slats. The key task is to obtain the position of the bottom edge of the target slats, such as Fig.16 As shown, Fig.16 A schematic diagram of horizontal adjustment provided in an embodiment of the present application.

[0215] The ground deviation angle θ1 can be calculated according to formula 27, where v i is the vector representing the bottom edge of the target slat, and v0 is the direction vector of the ground:

[0216] , formula 27;

[0217] The adjustment of Ry can be completed according to formula 27.

[0218] Subsequently, Dx, Dy and Dz are adjusted with the coordinate origin O as the reference datum, and the plate is precisely positioned at the installation coordinate to complete the verticality detection.

[0219] Flatness detection requires that the strips are flush with each other, and Rz needs to be adjusted. Fig.17 , Fig.17 This is a schematic diagram of flatness adjustment provided in an embodiment of the present application. The flatness adjustment is achieved specifically through the above method, and no repeated description is given here.

[0220] Similarly, the slat deviation angle θ2 is used to ensure that the target slat is parallel to the reference slat and can be calculated according to formula 28, where v0 is the vector of the bottom edge of the target slat, v j is the vector of the bottom edge of the reference strip. See Table 1, which summarizes all the properties to be tested in this application.

[0221] , formula 28;

[0222] Table 1 Detection properties and uses

[0223]

[0224] The degree of freedom installation control calibration sequence is crucial to ensure accurate and stable installation of the plate. This process involves strict calibration and adjustment of the various degrees of freedom of the robot or assembly system to ensure accuracy and stability during the installation process. Following the predetermined calibration sequence can improve installation efficiency, reduce error rate, and ensure the quality and reliability of the final product. Through this embodiment, the installation degree of freedom calibration sequence of the target object is realized to ensure accurate and stable installation of the plate at the target position.

[0225] The following is an introduction to the robot autonomous operation system provided in an embodiment of the present application. The robot autonomous operation system described below and the robot autonomous operation method described above can be referenced to each other.

[0226] See also Fig.18 , Fig.18 This is a schematic diagram of the structure of a robot autonomous operation system provided in an embodiment of the present application. The present application also provides a robot autonomous operation system, including:

[0227] An object recognition module is used to recognize a target object and determine an object area and a grasping posture corresponding to the target object;

[0228] An object grasping module, used for grasping the target object by applying the grasping gesture in the object area;

[0229] A coarse positioning module, used to obtain environmental information through a camera, generate a panoramic image, and determine the coarse positioning of the target object in the panoramic image;

[0230] A precise positioning module, used to obtain surface information of the target object through point cloud data or a laser rangefinder, and determine the precise positioning of the corresponding installation point of the target object according to the rough positioning and the surface information;

[0231] The robot autonomous operation module is used to determine the operation reference coordinate system based on the precise positioning, and to use the robot to perform the installation operation of the target object based on the operation reference coordinate system.

[0232] Based on the above embodiments, as a preferred embodiment, it also includes:

[0233] The camera calibration module is used to calibrate the camera and determine the mapping relationship between the image and the actual object; determine the external parameter matrix and the internal parameter matrix of the camera according to the mapping relationship; the external parameter matrix is ​​used to represent the transformation relationship between the corresponding points in the world coordinate system and the camera coordinate system; the internal parameter matrix is ​​used to represent the transformation relationship between the corresponding points in the camera coordinate system and the image coordinate system.

[0234] Based on the above embodiment, as a preferred embodiment, the object recognition module includes:

[0235] A region recognition unit, used for determining the object region to which the target object belongs using a convolutional neural network;

[0236] A posture determination unit is used to determine the grasping posture of the target object by using a generative grasping convolutional neural network.

[0237] Based on the above embodiment, as a preferred embodiment, the region identification unit is a unit for performing the following steps:

[0238] Input the collected image data, extract a feature map of the image data using a shared convolutional layer, and input the feature map into a region proposal network component;

[0239] In the region proposal network component, a convolution kernel is used to perform convolution sliding on the input image to generate a sliding window;

[0240] Generate a plurality of anchor boxes at the center point of each sliding window; the anchor boxes include different scales and different aspect ratios;

[0241] The feature map of the anchor box is passed to the fully connected layer to generate a corresponding feature vector, and the classification and positioning of the feature vector are determined by a Softmax classifier and a linear regression classifier;

[0242] Calculating the intersection-over-union ratio between the anchor point box and the manually calibrated true value;

[0243] Determine whether the detection target is included according to the intersection-and-union ratio; if the intersection-and-union ratio is less than a first intersection-and-union ratio threshold, determine it as a background area; if the intersection-and-union ratio is greater than a second intersection-and-union ratio threshold, it indicates that the detection target is effectively included;

[0244] If the IoU is not less than the first IoU threshold and not greater than the second IoU threshold, inputting linear regression to make the boundary value output by the anchor box approach the actual value, and taking the anchor box with a branch greater than the preset value as a suggestion area;

[0245] Through the region of interest pooling layer, feature extraction is performed on the selected area of ​​the anchor box and the feature map output by the region proposal network component, and after feature extraction, it is passed to the fully connected layer to generate a feature vector;

[0246] The feature vector is passed through a Softmax classifier and linear regression to obtain a classification score of the region and an object region after regression adjustment.

[0247] Based on the above embodiment, as a preferred embodiment, the posture determination unit is a unit for performing the following steps:

[0248] Before grasping, a depth image without any objects is stored as a reference.

[0249] Encode the index of each bounding box of the object area and map the pixels in the bounding box to the same position in the blank depth image;

[0250] Cropping the depth image into a region image containing only the object of interest, and setting the region image to the same size as the input image of the generative grasping convolutional neural network;

[0251] The trained generative grasp convolutional neural network is used to generate the grasping posture with the highest quality score.

[0252] Based on the above embodiment, as a preferred embodiment, the coarse positioning module is a module for performing the following steps:

[0253] Obtain fisheye images from the panoramic fisheye camera platform;

[0254] Using a generative network to perform distortion correction on the fisheye image to obtain a corrected image;

[0255] Converting the corrected image into a panoramic image;

[0256] Performing straight line detection on the panoramic image using Hough transform, and calculating vertex coordinates of the target object using the detected straight lines;

[0257] Coordinate transformation is performed on the vertex coordinates to obtain the rough positioning coordinates of the target object in the panoramic image.

[0258] Based on the above embodiments, as a preferred embodiment, it also includes:

[0259] The preprocessing module is used to perform image preprocessing on the panoramic image; the image preprocessing includes grayscale processing, binarization processing, connected area extraction and morphological processing.

[0260] Based on the above embodiments, as a preferred embodiment, the robot autonomous operation module is a module for performing the following steps:

[0261] Adjusting the verticality of the target object so that the bottom of the target object remains parallel to the ground;

[0262] Moving the target object to the precise position based on the coordinate origin in the operation reference coordinate system;

[0263] Performing verticality detection and flatness detection on the target object;

[0264] If the verticality detection and the flatness detection of the target object are both in compliance, the robot end effector is regulated by using a proportional integral differential algorithm to perform the installation operation of the target object.

[0265] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0266] The present application also provides an electronic device, which may include a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed, the steps provided in the above embodiment can also be implemented.

[0267] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system provided in the embodiment, since it corresponds to the method provided in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0268] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

[0269] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A robot autonomous operation method, characterized in that: include: Identify the target object, and determine the object area and grasping posture corresponding to the target object; Grasping the target object by applying the grasping gesture in the object area; Acquire environmental information through a camera, generate a panoramic image, and determine a rough location of an installation target of the target object in the panoramic image; Acquire surface information of the target object through point cloud data or a laser rangefinder, and determine the precise positioning of the corresponding installation point of the target object according to the rough positioning and the surface information; Determine an operation reference coordinate system based on the precise positioning, and use a robot to perform installation work on the target object based on the operation reference coordinate system; Wherein, determining the object area and grasping posture corresponding to the target object includes: Determine the object region to which the target object belongs using a convolutional neural network; Determining a grasping posture of the target object using a generative grasping convolutional neural network; Wherein, determining the object area to which the target object belongs by using a convolutional neural network includes: Input the collected image data, extract a feature map of the image data using a shared convolutional layer, and input the feature map into a region proposal network component; In the region proposal network component, a convolution kernel is used to perform convolution sliding on the input image to generate a sliding window; Generate a plurality of anchor boxes at the center point of each sliding window; the anchor boxes include different scales and different aspect ratios; The feature map of the anchor box is passed to the fully connected layer to generate a corresponding feature vector, and the classification and positioning of the feature vector are determined by a Softmax classifier and a linear regression classifier; Calculating the intersection-over-union ratio between the anchor point box and the manually calibrated true value; Determine whether the detection target is included according to the intersection-and-union ratio; if the intersection-and-union ratio is less than a first intersection-and-union ratio threshold, determine it as a background area; if the intersection-and-union ratio is greater than a second intersection-and-union ratio threshold, it indicates that the detection target is effectively included; If the IoU is not less than the first IoU threshold and not greater than the second IoU threshold, inputting linear regression to make the boundary value output by the anchor box approach the actual value, and taking the anchor box with a branch greater than the preset value as a suggestion area; Through the region of interest pooling layer, feature extraction is performed on the selected area of ​​the anchor box and the feature map output by the region proposal network component, and after feature extraction, it is passed to the fully connected layer to generate a feature vector; The feature vector is subjected to a normalized exponential function and linear regression to obtain a classification score of the region and an object region after regression adjustment.

2. The robot autonomous operation method according to claim 1, characterized in that: Before identifying the target object, it also includes: Calibrate the camera to determine the mapping relationship between the image and the actual object; The external parameter matrix and the internal parameter matrix of the camera are determined according to the mapping relationship; the external parameter matrix is ​​used to represent the transformation relationship between the corresponding points in the world coordinate system and the camera coordinate system; the internal parameter matrix is ​​used to represent the transformation relationship between the corresponding points in the camera coordinate system and the image coordinate system.

3. The robot autonomous operation method according to claim 1, characterized in that: The method of determining the grasping posture of the target object by generating a grasping convolutional neural network includes: Before grasping, a depth image without any objects is stored as a reference; Encode the index of each bounding box of the object area and map the pixels in the bounding box to the same position in the blank depth image; Cropping the depth image into a region image containing only the object of interest, and setting the region image to the same size as the input image of the generative grasping convolutional neural network; The trained generative grasp convolutional neural network is used to generate the grasping posture with the highest quality score.

4. The robot autonomous operation method according to claim 1, characterized in that: Acquiring environmental information through a camera, generating a panoramic image, and determining a rough location of an installation target of the target object in the panoramic image includes: Obtain fisheye images from the panoramic fisheye camera platform; Using a generative network to perform distortion correction on the fisheye image to obtain a corrected image; Converting the corrected image into a panoramic image; Performing straight line detection on the panoramic image using Hough transform, and calculating vertex coordinates of the installation target of the target object through the detected straight lines; Coordinate conversion is performed on the vertex coordinates to obtain the rough positioning coordinates of the installation target of the target object in the panoramic image.

5. The robot autonomous operation method according to claim 4, characterized in that: After converting the corrected image into a panorama, the method further comprises: The panoramic image is subjected to image preprocessing; the image preprocessing includes grayscale processing, binarization processing, connected region extraction and morphological processing.

6. The robot autonomous operation method according to claim 1, characterized in that: Determining an operation reference coordinate system based on the precise positioning, and applying a robot to perform the installation operation of the target object based on the operation reference coordinate system includes: Adjusting the verticality of the target object so that the bottom of the target object remains parallel to the ground; Moving the target object to the precise position based on the coordinate origin in the operation reference coordinate system; Performing verticality detection and flatness detection on the target object; If the verticality detection and the flatness detection of the target object are both in compliance, the robot end effector is regulated by using a proportional integral differential algorithm to perform the installation operation of the target object.

7. The robot autonomous operation method according to claim 4, characterized in that: Converting the rectified image into a panorama comprises: determining a number of fisheye images including different perspectives; Determine two landmark points of each common area in the fisheye image; The straight line formed by the two marking points is used as a splicing line; The fisheye images of different viewing angles are stitched together based on the stitching line to obtain a panoramic image.

8. A robot autonomous operation system, characterized in that: include: An object recognition module is used to recognize a target object and determine an object area and a grasping posture corresponding to the target object; An object grasping module, used for grasping the target object by applying the grasping gesture in the object area; A coarse positioning module, used to obtain environmental information through a camera, generate a panoramic image, and determine the coarse positioning of the installation target of the target object in the panoramic image; A precise positioning module, used to obtain surface information of the target object through point cloud data or a laser rangefinder, and determine the precise positioning of the corresponding installation point of the target object according to the rough positioning and the surface information; A robot autonomous operation module, used to determine an operation reference coordinate system based on the precise positioning, and to use a robot to perform installation operations on the target object based on the operation reference coordinate system; Wherein, the object recognition module includes: A region recognition unit, used for determining the object region to which the target object belongs using a convolutional neural network; A posture determination unit, used to determine the grasping posture of the target object by using a generative grasping convolutional neural network; The region identification unit is a unit for performing the following steps: Input the collected image data, extract a feature map of the image data using a shared convolutional layer, and input the feature map into a region proposal network component; In the region proposal network component, a convolution kernel is used to perform convolution sliding on the input image to generate a sliding window; Generate a plurality of anchor boxes at the center point of each sliding window; the anchor boxes include different scales and different aspect ratios; The feature map of the anchor box is passed to the fully connected layer to generate a corresponding feature vector, and the classification and positioning of the feature vector are determined by a Softmax classifier and a linear regression classifier; Calculating the intersection-over-union ratio between the anchor point box and the manually calibrated true value; Determine whether the detection target is included according to the intersection-and-union ratio; if the intersection-and-union ratio is less than a first intersection-and-union ratio threshold, determine it as a background area; if the intersection-and-union ratio is greater than a second intersection-and-union ratio threshold, it indicates that the detection target is effectively included; If the IoU is not less than the first IoU threshold and not greater than the second IoU threshold, inputting linear regression to make the boundary value output by the anchor box approach the actual value, and taking the anchor box with a branch greater than the preset value as a suggestion area; Through the region of interest pooling layer, feature extraction is performed on the selected area of ​​the anchor box and the feature map output by the region proposal network component, and after feature extraction, it is passed to the fully connected layer to generate a feature vector; The feature vector is passed through a Softmax classifier and linear regression to obtain a classification score of the region and an object region after regression adjustment.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the robot autonomous operation method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed, implements the steps of the robot autonomous operation method according to any one of claims 1 to 7.

Images