Valve identification pose estimation method and system based on FPPE network

Abstract

The invention discloses a valve recognition pose estimation method and system based on an FPPE network, and relates to the field of image recognition. The problems that an existing visual positioning and pose detection method for the valve is not high in pose recognition degree and too large in calculated amount are solved. The method comprises the following steps: collecting a multi-angle valve picture and valve point cloud data, and constructing a valve point cloud data set and template point cloud; the RT-DETR v2 target detection network is trained; an FPPE network is constructed and trained; according to the trained RT-DETR v2 network, carrying out target identification and plane target position detection on the valve image to obtain a central position coordinate of a valve area and an ROI boundary point pixel coordinate; points of all depths in the ROI space area are extracted to form a view cone point cloud; and inputting the view cone point cloud into the trained FPPE network to obtain a final valve pose estimation result. The method is applied to the technical field of object positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, and in particular to a valve recognition and pose estimation method based on an FPPE network. BACKGROUND

[0002] In the workshops of the oil, chemical, power generation, heating and other industries, valves, as key components, are widely used in the pipelines of various gas-liquid systems to regulate and control the opening and closing of the pipeline fluid passage. Since the design of the valve is relatively simple, it adopts a mechanical structure and has a long service life, therefore, in these industrial production scenes, the valve is usually controlled to open and close by manual operation. However, since the valve is usually located in the key link of the production process, and the control operation is usually carried out in a complex environment of high temperature, high pressure, etc., the reliability and safety of manual operation face great challenges. Therefore, how to improve the control efficiency and accuracy of the valve has become a problem that enterprises need to solve.

[0003] In recent years, with the rapid development of artificial intelligence, deep learning technology, machine vision and robot technology, it has gradually become a feasible solution to realize the opening and closing control of the valve through a special mechanical arm in the workshop. The mechanical arm can accurately identify the position and pose of the valve through image recognition and analysis, and complete the corresponding screwing operation, thereby effectively reducing the manual intervention. In this process, the application of machine vision technology is particularly critical, which can provide accurate operation guidance for the mechanical arm through real-time collection and analysis of image data in the workshop.

[0004] However, the particularity of valve operation poses certain challenges to this technology. In the valve screwing operation, the mechanical arm needs to bear a large torque and complex motion load, therefore, accurate pose recognition and positioning are the core requirements for completing accurate operation. The existing visual positioning and pose detection methods usually adopt pure image or pure point cloud technology, but these two schemes often have the following problems: on the one hand, the pose recognition accuracy of the pure image method may be affected by factors such as light, angle, reflection, etc., resulting in low recognition accuracy; on the other hand, although the pure point cloud scheme can provide relatively accurate spatial data, due to the large amount of calculation, the processing speed is slow, which may not meet the real-time requirements of the production line, especially in the high-speed production environment, the balance between accuracy and efficiency is particularly critical.

[0005] Therefore, how to improve the accuracy of pose recognition and reduce the amount of calculation to meet the dual requirements of efficiency and accuracy of modern production has become the key to current technical research and development. SUMMARY

[0006] The application aims at the problems of low pose recognition degree and large calculation amount in the existing visual positioning and pose detection methods for valves, and proposes a valve pose estimation method based on an FPPE network, which comprises the following steps:

[0007] Step S1: acquiring multi-angle valve pictures by using a color camera, acquiring valve point cloud data by using a depth camera to construct a valve point cloud data set, and constructing a template point cloud;

[0008] Step S2: training an RT-DETR v2 target detection network according to the acquired valve pictures;

[0009] Step S3: constructing an FPPE network and training the FPPE network according to the valve point cloud data set;

[0010] Step S4: performing target recognition and plane target position detection on the valve image according to the trained RT-DETR v2 network, obtaining the center position coordinates of the valve region and the boundary point pixel coordinates of the region of interest (ROI), and calculating the three-dimensional camera coordinates of the region boundary points by using the camera intrinsic matrix and the depth information;

[0011] Step S5: extracting all the depth points in the ROI space region according to the boundary point pixel coordinates of the ROI and the three-dimensional camera coordinates of the region boundary points, and composing a view frustum point cloud;

[0012] Step S6: inputting the view frustum point cloud into the trained FPPE network to obtain the final valve pose estimation result.

[0013] Further, an optimal mode is also proposed, wherein the step S2 further comprises preprocessing the valve pictures, including: screening the images to remove blurred images; and performing data enhancement on the screened images, including: horizontal flipping, vertical flipping, adding Gaussian noise and Gaussian filtering.

[0014] Further, an optimal mode is also proposed, wherein the FPPE network in the step S3 comprises: a point cloud segmenter and a pose estimation module.

[0015] The point cloud segmenter is an improved PCT network structure, comprising an encoder and a decoder, the encoder is composed of an Input Embedding layer and four attention layers, the Input embedding layer embeds the input point cloud coordinates into a new feature space, the attention outputs of each attention layer are concatenated through the feature dimension, and finally linear transformation is performed; after the encoder part is processed, the maximum pooling and the average pooling are connected and used to obtain the global features of the collected point cloud; the decoder fuses the global features before and after the pooling with the local features, uses one-dimensional convolution and batch normalization layer for decoding and uses ReLU function for activation; each point in the point cloud is output to predict the classification score through a one-dimensional convolution layer;

[0016] The point cloud pose estimation module is composed of three groups of MLPs containing 1024, 512 and 256 hidden layers, the template point cloud is fused with the segmented valve point cloud through the feature extraction part of the point cloud segmenter, and the pose between the valve point cloud and the template is estimated; finally, an output layer with a size of 6 and an EMD loss function are used to output the pose of the valve point cloud, representing the estimated transformation T of the collected point cloud to the template point cloud, the first three output values t ∈ R 3 represent the translation vector of the point, and the last three output values r ∈ R 3 represent the Euler angle rotation vector of the point.

[0017] Further, an optimal mode is further proposed, wherein the step S3 of training the FPPE network comprises calculating the distance between the template point cloud and the collected point cloud by using an EMD loss function.

[0018] Further, an optimal mode is further proposed, wherein the step S4 comprises:

[0019] Step S41: performing downsampling operation and data enhancement operation on the real-time collected valve image;

[0020] Step S42: inputting the image into RT-DETRv2 for target recognition, obtaining the pixel coordinates of the recognition frame center point coordinates and the recognition frame boundary points in the picture coordinate system, and converting them into camera coordinates in the camera coordinate system:

[0021]

[0022] wherein, X C , Y C , and Z C are the coordinate values of the point in the pixel coordinates in the camera coordinate system, Z C is obtained from the depth value of the pixel point measured by the depth camera; u and v are the pixel coordinates of the point, u0, v0, f x , and f y are four parameter values in the camera intrinsic matrix K.

[0023] Further, it is further proposed that step S5 comprises:

[0024] aligning the image captured by the color camera with the depth map captured by the depth camera;

[0025] coordinate transformation operation of converting the valve ROI boundary point coordinates in the color image into the depth camera coordinate system:

[0026] P depth = R rgb2depth * K rgb -1 * Z rgb * p rgb + T rgb2depth

[0027] wherein P depth is a point in the depth camera coordinate system, R rgb2depth is an external parameter rotation transformation matrix of the RGB camera to the depth camera, K rgb is an internal parameter matrix of the camera, Z rgb is the Z-axis coordinate of the point in the color camera coordinate system, p rgb is the homogeneous pixel coordinate of the point in the image, and T rgb2depth is an external parameter translation transformation matrix of the RGB camera to the depth camera;

[0028] Calculate the boundary point coordinates of the ROI in the depth coordinate system and the point cloud coordinate system, extract all points with x, y coordinates within the range and different depth values to form a view frustum point cloud set.

[0029] Further, it is further proposed that the color camera is used to capture pictures of the front, left side 15°, right side 15°, left side 30°, right side 30°, left side 45°, right side 45°, left side 60°, and right side 60° of the valve, respectively.

[0030] Based on the same inventive concept, the present application also proposes a valve recognition and pose estimation system based on an FPPE network, which comprises:

[0031] A data acquisition unit is configured to acquire multi-angle valve pictures using a color camera, acquire valve point cloud data using a depth camera, construct a valve point cloud data set, and construct a template point cloud.

[0032] A target detection network training unit is configured to train an RT-DETR v2 target detection network according to the acquired valve pictures.

[0033] An FPPE network construction unit is configured to construct an FPPE network and train the FPPE network according to the valve point cloud data set.

[0034] a target detection unit configured to perform target recognition and planar target position detection on the valve image according to the trained RT-DETR v2 network, to obtain the center position coordinates of the valve region and the pixel coordinates of the region of interest (ROI) boundary points, and to calculate the three-dimensional camera coordinates of the region boundary points by using the camera intrinsic matrix and the depth information;

[0035] a frustum point cloud extraction unit configured to extract all points in the ROI spatial region according to the pixel coordinates of the ROI boundary points and the three-dimensional camera coordinates of the region boundary points, to form a frustum point cloud;

[0036] a pose acquisition unit configured to input the frustum point cloud into the trained FPPE network to obtain a final valve pose estimation result.

[0037] Based on the same inventive concept, the application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the valve recognition pose estimation method based on the FPPE network according to any one of the above.

[0038] Based on the same inventive concept, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and when the computer program is run by a processor, the steps of the valve recognition pose estimation method based on the FPPE network according to any one of the above are executed.

[0039] The application has the following advantages:

[0040] The valve recognition pose estimation method based on the FPPE network provided by the application effectively fuses visual and depth information, overcomes the limitations of single image or point cloud methods, and improves the accuracy of pose recognition by combining point cloud data collected by a depth camera and image data collected by a color camera, constructing a mixed data set and a template point cloud, and using an FPPE network for pose estimation.

[0041] The valve recognition pose estimation method based on the FPPE network provided by the application can effectively reduce the amount of calculation by optimizing the network structure and combining the RT-DETR v2 target detection network and the FPPE network. Specifically, after identifying the valve position and performing target detection on the image data, only the region of interest is extracted and processed for depth information, thereby reducing the range and amount of calculation of processing point cloud data, improving the real-time performance of the system, and overcoming the problem of large amount of calculation in traditional point cloud data processing methods.

[0042] The valve recognition pose estimation method based on the FPPE network can effectively deal with valve operations in complex environments by combining multi-modal data (i.e., images and point clouds) and training deep learning technology. By combining RT-DETR v2 target detection and the FPPE network, the system can accurately locate the valve in a complex environment and estimate its pose. Even under the influence of factors such as light and occlusion, the system can ensure high recognition accuracy, overcoming the difficulty that traditional methods cannot stably and reliably perform accurate pose recognition and positioning under these complex conditions.

[0043] The valve recognition pose estimation method based on the FPPE network can effectively improve the accuracy of pose estimation by combining image data and point cloud data in a multi-modal fusion manner. Image data provides rich visual information, while point cloud data provides detailed three-dimensional spatial information. The combination of the two can significantly improve the accuracy of valve pose recognition, especially in complex environments, overcoming the shortcomings of a single data source.

[0044] Traditional point cloud processing methods have large computational loads and slow response speeds. By introducing target detection (RT-DETR v2) and the FPPE network, the method proposed in the present application can first determine the region of interest of the valve through image recognition, and then only process the point cloud data in this region, greatly reducing the computational load. Compared with traditional methods, the system can perform pose estimation more quickly, thereby meeting the needs of real-time production environments.

[0045] The valve recognition pose estimation method based on the FPPE network can not only adapt to valve positioning and operation in normal environments, but also perform stable operations in various complex environmental conditions by combining deep learning technology and machine vision technology. For example, in conditions of insufficient light, object occlusion, or reflection, the system can still stably perform pose estimation and recognition. This makes the method very suitable for valve control in industrial production, especially in harsh conditions such as high temperature and high pressure, ensuring efficient and accurate control.

[0046] The present application is applied to the field of object positioning technology. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 A valve recognition pose estimation method flowchart based on the FPPE network according to the first embodiment;

[0048] Figure 2 A structure diagram of the FPPE network according to the eleventh embodiment. DETAILED DESCRIPTION

[0049] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application.

[0050] Embodiment one, see Figure 1 This embodiment is described. The valve recognition pose estimation method based on the FPPE network described in this embodiment comprises the following steps:

[0051] Step S1: acquiring multi-angle valve pictures by using a color camera, acquiring valve point cloud data by using a depth camera to construct a valve point cloud data set, and constructing a template point cloud;

[0052] Step S2: training an RT-DETR v2 target detection network according to the acquired valve pictures;

[0053] Step S3: constructing an FPPE network and training the FPPE network according to the valve point cloud data set;

[0054] Step S4: performing target recognition and plane target position detection on the real-time acquired valve images according to the trained RT-DETR v2 network, obtaining the center position coordinates of the valve region and the region of interest ROI boundary point pixel coordinates, and calculating the three-dimensional camera coordinates of the region boundary points by using the camera intrinsic matrix and the depth information;

[0055] Step S5: extracting all the depth points in the ROI spatial region according to the region of interest ROI boundary point pixel coordinates and the three-dimensional camera coordinates of the region boundary points to form a to-be-processed view frustum point cloud;

[0056] Step S6: inputting the view frustum point cloud into the trained FPPE network to obtain the pose transformation vector of the final target valve point cloud and the template valve point cloud, that is, obtaining the final valve pose estimation result.

[0057] The valve recognition pose estimation method based on the FPPE network proposed in this embodiment effectively fuses visual and depth information by combining the point cloud data acquired by the depth camera and the image data acquired by the color camera, constructs a hybrid data set and a template point cloud, and performs pose estimation by using the FPPE network, thereby overcoming the limitations of single image or point cloud methods and improving the accuracy of pose recognition.

[0058] The method proposed in this embodiment can effectively reduce the amount of calculation by optimizing the network structure and combining the RT-DETR v2 target detection network and the FPPE network. Specifically, after identifying the valve position and performing target detection on the image data, only the region of interest is extracted and processed for depth information, thereby reducing the range and amount of calculation for processing point cloud data, improving the real-time performance of the system, and overcoming the problem that traditional point cloud data processing methods usually have a large amount of calculation.

[0059] The method proposed in this embodiment can effectively deal with valve operations in complex environments by combining multi-modal data (i.e., images and point clouds) and training deep learning techniques. By combining the RT-DETR v2 target detection and FPPE (Frustum Pointnet Pose Estimation) networks, the system can accurately locate and estimate the pose of the valve in complex environments, even under the influence of factors such as lighting and occlusion, ensuring high recognition accuracy and overcoming the problem that traditional methods cannot reliably and accurately identify and locate the pose under these complex conditions.

[0060] The method proposed in this embodiment can effectively improve the accuracy of pose estimation by combining image data and point cloud data using multi-modal fusion. Image data provides rich visual information, while point cloud data provides detailed three-dimensional spatial information, which can significantly improve the accuracy of valve pose recognition, especially in complex environments, overcoming the shortcomings of a single data source.

[0061] Traditional point cloud processing methods have a large amount of calculation and slow response speed. By introducing target detection (RT-DETR v2) and FPPE networks, this embodiment can first determine the region of interest of the valve through image recognition, and then only process the point cloud data in this region, significantly reducing the amount of calculation. Compared with traditional methods, the system can perform pose estimation faster, thereby meeting the needs of real-time production environments.

[0062] The method proposed in this embodiment combines deep learning techniques and machine vision techniques, not only adapting to valve positioning and operation in normal environments, but also performing stable operations in various complex environmental conditions. For example, in cases of insufficient lighting, object occlusion, or reflection, it can still stably perform pose estimation and recognition. This makes the method very suitable for valve control in industrial production, especially in harsh conditions such as high temperature and high pressure, ensuring efficient and accurate control.

[0063] Embodiment two, the embodiment is one kind based on the further limitation of the valve recognition pose estimation method of FPPE network of embodiment one, the step S2 further includes preprocessing valve photo, including;Image screening, eliminating blurred image;Data enhancement is carried out to the screened image, including: horizontal flip, vertical flip, add Gaussian noise and Gaussian filter.

[0064] The embodiment can effectively remove blurred images caused by shooting angle, uneven lighting or camera shake by screening images and eliminating blurred images. By data enhancement on the screened images, including horizontal flip, vertical flip, adding Gaussian noise and Gaussian filtering, the training data set can be effectively expanded. Adding Gaussian noise and Gaussian filtering also increases the robustness of the model.

[0065] Embodiment three, the embodiment is one kind based on the further limitation of the valve recognition pose estimation method of FPPE network of embodiment one, the FPPE network in step S3 includes: point cloud segmenter and pose estimation module;

[0066] The point cloud segmenter is an improved PCT network structure, including: encoder and decoder, the encoder is composed of an Input Embedding layer and four attention layers, the Input embedding layer embeds the input point cloud coordinates into a new feature space, the attention outputs of each attention layer are concatenated through feature dimension, and finally linear transformation is performed;After the encoder part is processed, the maximum pooling and average pooling are connected and used to obtain the global features of the collected point cloud;The decoder fuses the global features before and after pooling with local features, uses one-dimensional convolution and batch normalization layer for decoding and uses ReLU function for activation;Each point in the point cloud is outputted by a one-dimensional convolution layer to predict the classification score;

[0067] The point cloud pose estimation module is composed of three groups of MLPs containing 1024, 512 and 256 hidden layers, the feature extraction part of the template point cloud is fused with the segmented valve point cloud, and the pose between the valve point cloud and the template is estimated;Finally, an output layer with a size of 6 and an EMD loss function output the pose of the valve point cloud, representing the estimated transformation T of the collected point cloud to the template point cloud, the first three output values t in the output layer 3 represent the translation vector of the point, and the last three outputs r 3 represent the Euler angle rotation vector of the point.

[0068] The point cloud segmenter in this embodiment adopts an improved PCT network, which includes an encoder-decoder structure and improves the feature extraction capability through an enhanced attention mechanism. This enables the network to better focus on the key parts of the point cloud, thereby improving the accuracy of valve identification and pose estimation. The Input Embedding layer in the encoder embeds the point cloud coordinates into a new feature space, enhancing the representation capability of the point cloud and enabling the network to handle more complex spatial relationships. The addition of the attention layer helps the network to automatically identify and focus on important areas in the point cloud. The encoder obtains global features of the point cloud through the combination of max-pooling and average-pooling. This method ensures that the network can comprehensively understand the shape and structure information of the entire point cloud, rather than just focusing on local features. The decoder part further enhances the network's multi-scale feature learning capability in the valve identification process by fusing global and local features, enabling accurate handling of complex pose changes.

[0069] The point cloud pose estimation module performs pose prediction through three groups of MLPs with different numbers of hidden layers (1024, 512, 256). This structure enables the network to gradually learn and optimize the pose relationship between the valve and the template point cloud in multiple levels of feature space. The layer-by-layer processing of MLPs can improve the accuracy of pose estimation, especially for valves with complex geometric shapes or variable poses, the model can optimize and refine the results through multiple iterations. The output layer contains 6 nodes, representing the translation vector and Euler angle rotation vector of the point cloud. This enables the pose estimation to not only provide accurate spatial positioning (translation), but also accurately describe the rotation (Euler angle) of the object, thereby achieving high-precision three-dimensional pose estimation. Through the EMD (Earth Mover's Distance) loss function, the matching degree of the point cloud is further optimized, making the pose estimation between the valve point cloud and the template point cloud more accurate, reducing the sensitivity to noise and deformation.

[0070] Embodiment four, this embodiment is a further limitation of the valve identification and pose estimation method based on the FPPE network of embodiment three, wherein the step S3 of training the FPPE network includes calculating the distance between the template point cloud and the collected point cloud using the EMD loss function.

[0071] Embodiment five, this embodiment is a further limitation of the valve identification and pose estimation method based on the FPPE network of embodiment one, wherein the step S4 includes:

[0072] Step S41: performing downsampling and data enhancement operations on the real-time collected valve image;

[0073] Step S42: input the image into RT-DETRv2 for target recognition, obtain the pixel coordinates of the recognition box center point and the recognition box boundary point in the picture coordinate, and convert them into camera coordinates in the camera coordinate system:

[0074]

[0075] wherein X C , Y C , and Z C are the coordinate values of the pixel point in the camera coordinate system, Z C is obtained from the depth value of the pixel point measured by the depth camera; u and v are the pixel coordinates of the point, u0, v0, f x , and f y are four parameter values in the camera intrinsic matrix K.

[0076] The present embodiment can quickly complete valve recognition and pose estimation under lower computational overhead through image downsampling and data enhancement operations. RT-DETRv2 combined with a depth camera can achieve high-precision valve recognition and pose estimation through accurate target detection and depth information conversion, especially in complex environments. Data enhancement and downsampling operations enable the model to accurately judge valve images in different environments and angles, thereby improving the applicability and stability in actual deployment.

[0077] Further, the 2D pixel coordinates in the image can be converted to spatial positions in the 3D camera coordinate system through the depth value of the pixel point measured by the depth camera. This enables the method not only to recognize the position of the valve, but also to accurately estimate the pose of the valve, thereby providing accurate data support for subsequent motion planning and operation. Through the four parameter values in the camera intrinsic matrix, the points in the image coordinate system can be accurately mapped to the camera coordinate system, thereby further improving the accuracy of pose estimation. This mapping is particularly important for scenarios that require high-precision positioning in practical applications, such as valve operation in industrial automation.

[0078] Embodiment six, the present embodiment is a further limitation of the valve recognition and pose estimation method based on the FPPE network described in embodiment one, step S5 includes:

[0079] aligning the image captured by the color camera with the depth map captured by the depth camera;

[0080] coordinate transformation operation for converting the valve ROI boundary point coordinates in the color image to the depth camera coordinate system:

[0081] P depth = R rgb2depth *K rgb -1*Z rgb *p rgb +T rgb2depth

[0082] where P depth is the point in the depth camera coordinate system, R rgb2depth is the extrinsic rotation transformation matrix from the RGB camera to the depth camera, K rgb is the intrinsic camera matrix, Z rgb is the Z-axis coordinate of the point in the color camera coordinate system, p rgb is the homogeneous pixel coordinate of the point in the image, T rgb2depth is the extrinsic translation transformation matrix from the RGB camera to the depth camera.

[0083] The boundary point coordinates of the ROI in the depth coordinate system and the point cloud coordinate system are calculated, and all points with x, y coordinates within the range and different depth values are extracted to form a set of view frustum point clouds.

[0084] In this embodiment, by aligning the image captured by the color camera with the depth map captured by the depth camera, the advantages of the color image and the depth map can be fully utilized. The color image can provide more detailed information and texture, while the depth map can provide accurate spatial position information. Through data fusion, more comprehensive information can be obtained, which helps to improve the accuracy of valve recognition and pose estimation. Converting the ROI boundary point coordinates in the color image to the depth camera coordinate system and using coordinate transformation operation makes the matching of the camera coordinate system more accurate. This can avoid errors caused by mismatched coordinate systems, thereby ensuring the accuracy of valve boundary extraction and positioning. Especially in complex environments, the valve may be partially obscured or affected by external factors, and accurate coordinate transformation operation can ensure that the boundary points can be accurately extracted even in such cases. By calculating the ROI boundary point coordinates in the depth coordinate system and the point cloud coordinate system, and extracting points within a certain range and with different depth values to form a set of view frustum point clouds, high-quality three-dimensional data support can be provided for subsequent pose estimation.

[0085] Further, by extracting the set of view frustum point clouds in the ROI region, the area related to the valve can be focused on, the complexity of the calculation can be reduced, and the accuracy of the calculation can be ensured. The set of view frustum point clouds can provide geometric information of the valve, which is crucial for pose estimation, especially in dynamic or complex scenarios. Through accurate region extraction, the accuracy and robustness of valve recognition can be effectively improved.

[0086] The embodiment effectively improves the accuracy and robustness of valve recognition through the fusion of multi-modal data (color image and depth map), combined with accurate coordinate transformation and point cloud data processing. In complex environments, such as changes in lighting, object occlusion, etc., this method can supplement the deficiencies of color images through depth map data, ensuring that the system maintains high recognition and pose estimation accuracy under different conditions.

[0087] Embodiment seven, the embodiment is a further limitation of the valve recognition and pose estimation method based on the FPPE network according to any one of embodiments one to six, wherein the color camera is used to capture pictures of the valve from the front, left side 15°, right side 15°, left side 30°, right side 30°, left side 45°, right side 45°, left side 60°, and right side 60° of the valve.

[0088] In this embodiment, by capturing images from multiple different angles (front, left and right 15°, 30°, 45°, 60°, etc.), more comprehensive and rich information about the appearance of the valve can be obtained. The images at each angle provide different perspectives of the valve, which helps the FPPE network to more accurately learn and understand the three-dimensional structure, shape features and positional relationships of the valve. This enhances the robustness of the model in estimating the pose of the valve and reduces the recognition errors or local occlusion effects that may be caused by a single perspective.

[0089] At the same time, by processing and analyzing the images of the valve at different angles, information from multiple perspectives can be combined to improve the accuracy of valve pose estimation. In traditional pose estimation methods, if only a single perspective is relied upon, it may be affected by factors such as occlusion, reflection, or changes in lighting, resulting in errors. By capturing images from multiple angles, the actual position and pose of the valve can be more comprehensively perceived, thereby improving the final pose estimation accuracy.

[0090] Embodiment eight, the valve recognition and pose estimation system based on the FPPE network according to the embodiment, the system comprises:

[0091] The data acquisition unit is used to acquire multi-angle valve pictures using a color camera, acquire valve point cloud data using a depth camera, construct a valve point cloud dataset, and construct a template point cloud.

[0092] The target detection network training unit is used to train the RT-DETR v2 target detection network according to the acquired valve pictures.

[0093] The FPPE network construction unit is used to construct the FPPE network and train the FPPE network according to the valve point cloud dataset.

[0094] The target detection unit is used to perform target recognition and planar target position detection on the valve images collected in real time based on the trained RT-DETR v2 network, obtain the center position coordinates of the valve area and the pixel coordinates of the boundary points of the region of interest (ROI), and calculate the 3D camera coordinates of the region boundary points using the camera intrinsic parameter matrix and depth information;

[0095] A view cone point cloud extraction unit is used to extract points of all depths within the ROI spatial area according to the pixel coordinates of the boundary points of the region of interest (ROI) and the three-dimensional camera coordinates of the region boundary points to form a view cone point cloud to be processed;

[0096] The pose acquisition unit is used to input the view cone point cloud into the trained FPPE network to obtain the pose transformation vector between the final target valve point cloud and the template valve point cloud, that is, to obtain the final valve pose estimation result.

[0097] Implementation method nine: A computer device described in this implementation method includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a valve identification posture estimation method based on an FPPE network described in any one of implementation methods one to seven.

[0098] Implementation method ten, a computer-readable storage medium described in this implementation method, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a valve identification posture estimation method based on an FPPE network as described in any one of implementation methods one to seven are executed.

[0099] Implementation method 11, see Figure 2 This embodiment provides a specific example of the valve identification and posture estimation method based on the FPPE network described in the first embodiment, and is also used to explain the second to seventh embodiments. Specifically:

[0100] Step S1: Collect photos of the valve from multiple angles, use digital modeling tools to obtain point cloud data, and combine the valve point cloud data collected by the on-site depth camera to form a valve point cloud dataset. The point cloud data of the valve center coordinates aligned with the robot tool coordinate system is used as a template point cloud, specifically including:

[0101] Step S11: When collecting pictures, since the robot arm is on a moving chassis and may stop at different angles in front of the valve, a color camera is used to collect pictures of different valves in different workshops, respectively, in front of the valve, 15° to the left of the valve, 15° to the right of the valve, 30° to the left of the valve, 30° to the right of the valve, 45° to the left of the valve, 45° to the right of the valve, 60° to the left of the valve, and 60° to the right of the valve;

[0102] Step S12: record and save the point cloud information at the time of taking pictures using the depth camera at the time of taking pictures, one-to-one correspondence with the taken pictures, and use digital modeling software to obtain the standard template point cloud of different valves, and take the point cloud of the camera directly opposite the valve as the standard pose point cloud.

[0103] Step S2: offline part, training RT-DETR v2 target detection network using collected valve photos, including:

[0104] In the training stage of the RT-DETRv2 network, first, the images are screened to remove blurred and poor quality images; second, the valve images are data enhanced, through horizontal flipping, vertical flipping, adding Gaussian noise and Gaussian filtering, etc., to increase the data amount and improve the model recognition accuracy;

[0105] Step S3: construct the FPPE network, and train the FPPE network using the valve point cloud data set;

[0106] The FPPE point cloud pose estimation network includes a point cloud segmenter and a pose estimation module, as shown in Figure 2 The point cloud segmenter is a point cloud Transformer module based on the improvement of the PCT network, including an encoder and a decoder; the encoder is composed of an Input Embedding layer and four attention layers, and the Input embedding layer embeds the input point cloud coordinates into a new feature space. Each point in the point cloud can be represented as P∈R N×d , where N is the number of point cloud points, and d is the feature dimension of the point cloud. Since the point cloud is in a three-dimensional space, to reduce the calculation amount, only the point cloud coordinates can be taken, i.e. d takes 3. For each point P, the Input Embedding module first learns the feature f e ∈R N×d , and then concatenates the attention outputs of each attention layer through the feature dimension, and finally performs linear transformation, and its expression is:

[0107] f1=AT 1 (f e )

[0108] f i =AT i (f i-1 ),i=2,3,4

[0109] f o =concat(f1,f2,f3,f4)·W o

[0110] Where, AT is the attention layer, and W ois the weight of the one-dimensional convolutional layer. After the encoder completes processing, max pooling and average pooling are combined to obtain the global features of the collected point cloud. In the decoder, the global features before and after pooling are fused with the local features. Decoding is performed using a one-dimensional convolution and batch normalization layer, and activation is performed using the ReLU function. Finally, a one-dimensional convolutional layer is used to output a predicted classification score for each point in the point cloud. The point is assigned to the class with the highest score.

[0111] The pose estimation module is as follows Figure 2 As shown. The template point cloud extracted by the digital modeling tool is subjected to feature extraction using the same point cloud feature extraction part as the point cloud segmenter. The pose estimation module simultaneously receives the feature information of the real-time acquisition point cloud and the template point cloud segmented in step S5, and connects the features of the real-time acquisition point cloud and the template point cloud. In the MLP composed of 4 fully connected layers, the number of its 3 hidden layers is 1024, 512, and 256 respectively, and the final output layer size is 6. Its parameters represent the pose estimation of the final output, that is, the estimated transformation T from the acquisition point cloud to the template point cloud. Its first three output values ​​t∈R 3 Represents the translation vector of the point, followed by three outputs r∈R 3 A vector representing the Euler angle rotation of a point.

[0112] During the pose estimation network training process, since the final training result of the network needs to minimize the distance between the points in the template point cloud and the collected point cloud, and the template point cloud and the collected point cloud can be regarded as two distributions, the EMD loss function can better measure the distance between the two point cloud distributions. The specific expression of the EMD loss function is as follows:

[0113]

[0114] Among them, P S est is the point cloud feature vector after the pose estimation transformation matrix T calculated by the network, P T is the eigenvector of the template point cloud. This function seeks a bijective function that minimizes the sum of the distances between the points.

[0115] Step S4: For the online part, the RT-DETR v2 network is used to perform target recognition and planar target position detection on the valve photos collected in real time. The center position coordinates of the valve area and the pixel coordinates of the boundary points of the region of interest (ROI) are obtained. The 3D camera coordinates of the region boundary points are calculated using the camera intrinsic parameter matrix and depth information. Specifically, the following steps are performed:

[0116] Step S41: downsampling the real-time collected image to make the image size meet the RT-DETRv2 network input requirements, and performing data enhancement operations such as brightness adjustment and sharpening on the image;

[0117] Step S42: input the image into RT-DETRv2 for target recognition, obtain the pixel coordinates of the recognition box center point and the recognition box boundary point (upper left, lower right) in the picture coordinate, and convert them into camera coordinates in the camera coordinate system; if the target recognition is unsuccessful, reacquire the image;

[0118] In the above step S42, the pixel coordinates of the ROI boundary point can be represented as (u1, v1), (u2, v2), and when converting the pixel coordinates into camera coordinates in the camera coordinate system, the conversion formula is:

[0119]

[0120] wherein X C ,Y C ,Z C is the coordinate value of the pixel point in the camera coordinate system, Z C is obtained from the depth value of the pixel point measured by the depth camera; u, v are the pixel coordinates of the point, u0, v0, f x ,f y are four parameter values in the camera intrinsic matrix K, and the specific form is as follows:

[0121]

[0122] wherein f x ,f y are the focal lengths in pixel units, representing the focal lengths in the x and y directions respectively, u0, v0 are the positions of the principal point, i.e. the intersection coordinates of the optical axis and the image plane, which are usually close to the center of the image. According to the above formula, the three-dimensional coordinates of the ROI region boundary point in the camera coordinate system are calculated.

[0123] Step S5: according to the ROI boundary camera coordinates obtained in step S2, extract all points in the ROI space region, and form a view frustum point cloud to be processed, including:

[0124] Step S51: align the image collected by the color camera with the depth map collected by the depth camera. The pixel coordinate to camera coordinate conversion formula in step S3-2 can be transformed as follows:

[0125] Z rgb *p rgb =K rgb *P rgb

[0126] wherein Z rgb is the Z-axis coordinate of the point in the color camera coordinate system, K rgb is the camera intrinsic matrix, and p rgb is the homogeneous pixel coordinate of the point in the image, which has the form:

[0127] p rgb =[u rgb v rgb 1] T

[0128] P rgb is the coordinate of a point on the image in the color camera coordinate system, and its form is:

[0129] P rgb =[X rgb Y rgb Z rgb ] T

[0130] Similarly, in the depth camera coordinate system, we have:

[0131] Z depth *p depth =K depth *P depth

[0132] For the external parameters R of the color camera and depth camera rgb 、T rgb and R depth 、T depth There are the following transformation relationships:

[0133] R rgb2depth =R depth *R rgb -1

[0134] T rgb2depth =T depth -R rgb2depth *T rgb

[0135] For point P in color and depth camera coordinate systems rgb and P depth , there are the following transformation relationships:

[0136] P depth =R rgb2depth *P rgb +T rgb2depth

[0137] Combining all the above transformations, we can get:

[0138] Z depth *p depth =K depth *R rgb2depth *K rgb -1 *Z rgb *p rgb +Kdepth T rgb2depth

[0139] Again, the form is transformed, and finally:

[0140] P depth = R rgb2depth K rgb -1 Z rgb p rgb + T rgb2depth

[0141] According to the above formula, the coordinate transformation operation of converting the valve ROI boundary point coordinates in the color image into the depth camera coordinate system is completed.

[0142] Step S52: According to the boundary point coordinates of the ROI in the depth / point cloud coordinate system calculated in step S51, all points with different depth values and within the range of x, y coordinates are extracted to form a frustum point cloud set.

[0143] Step S6: Input the frustum point cloud obtained in S5 into the point cloud segmenter of the FPPE network. The segmenter predicts a label for each point of the input point cloud, and performs point cloud segmentation according to the predicted label.

[0144] Step S7: Input the segmented point cloud and the template point cloud into the pose estimation module of the FPPE network, and output the pose transformation vector of the final target valve point cloud and the template valve point cloud, that is, obtain the final pose estimation result.

[0145] Those skilled in the art will appreciate that embodiments of the present disclosure can be provided as methods, systems or computer program products. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, so that the instructions executed by the computer or other programmable data processing apparatus produce a device that implements the functions described in the flowcharts and / or block diagrams. Figure 1 one flow or multiple flows and / or blocks Figure 1apparatuses that implement the functions specified in the flowchart or flowcharts and / or blocks Figure 1 flowchart or flowcharts and / or blocks Figure 1 the function specified in the flowchart or flowcharts and / or blocks.

[0147] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart or flowcharts and / or blocks Figure 1 flowchart or flowcharts and / or blocks Figure 1 the function specified in the flowchart or flowcharts and / or blocks.

[0148] Finally, it should be noted that the above embodiments are merely used to illustrate the technical solutions of the present disclosure, rather than limit the scope of protection of the present disclosure. Although the present disclosure has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that, after reading the present disclosure, those skilled in the art can make various changes, modifications or equivalent replacements to the specific embodiments of the present disclosure. However, these changes, modifications or equivalent replacements are all within the scope of protection of the disclosed patent application.

Claims

1. A valve identification pose estimation method based on FPPE network, characterized in that: The method comprises: Step S1: Use a color camera to collect multi-angle valve images, use a depth camera to collect valve point cloud data to construct a valve point cloud dataset, and construct a template point cloud; Step S2: training the RT-DETR v2 object detection network based on the collected valve images; Step S3: Constructing the FPPE network and training the FPPE network based on the valve point cloud dataset; Step S4: Perform target recognition and planar target position detection on the valve image based on the trained RT-DETR v2 network, obtain the center position coordinates of the valve area and the pixel coordinates of the boundary points of the region of interest (ROI), and calculate the three-dimensional camera coordinates of the region boundary points using the camera intrinsic parameter matrix and depth information; Step S5: extracting points of all depths within the ROI spatial area according to the pixel coordinates of the boundary points of the ROI and the 3D camera coordinates of the boundary points of the region to form a view frustum point cloud; Step S6: Input the cone point cloud into the trained FPPE network to obtain the final valve pose estimation result.

2. A valve identification and posture estimation method based on FPPE network according to claim 1, characterized in that: The step S2 also includes pre-processing the valve photo, including: screening the image and removing the blurred image; Data enhancement is performed on the filtered images, including horizontal flipping, vertical flipping, adding Gaussian noise and Gaussian filtering.

3. The valve identification and posture estimation method based on FPPE network according to claim 1 is characterized in that: The FPPE network in step S3 includes: a point cloud segmenter and a pose estimation module; The point cloud segmenter is an improved PCT network structure, including: an encoder and a decoder. The encoder consists of an input embedding layer and four attention layers. The input embedding layer embeds the input point cloud coordinates into a new feature space, concatenates the attention outputs of each attention layer through the feature dimension, and finally performs a linear transformation. After the encoder part is processed, the maximum pooling and average pooling are connected to obtain the global features of the collected point cloud. The decoder fuses the global features and local features before and after pooling, uses a one-dimensional convolution and batch normalization layer for decoding, and uses the ReLU function for activation. A one-dimensional convolution layer outputs the predicted classification score for each point in the point cloud. The point cloud pose estimation module consists of three groups of MLPs with 1024, 512, and 256 hidden layers, respectively. The template point cloud is passed through the feature extraction part of the point cloud segmenter, fused with the segmented valve point cloud, and the pose between the valve point cloud and the template is estimated. Finally, an output layer of size 6 and the EMD loss function output the pose of the valve point cloud, which represents the estimated transformation T from the collected point cloud to the template point cloud.

4. The valve identification and posture estimation method based on FPPE network according to claim 1 is characterized in that: The training of the FPPE network in step S3 includes using the EMD loss function to calculate the distance between the template point cloud and the collected point cloud.

5. The valve identification and posture estimation method based on FPPE network according to claim 1 is characterized in that: The step S4 comprises: Step S41: performing downsampling and data enhancement operations on the valve image collected in real time; Step S42: Input the image into RT-DETRv2 for target recognition, obtain the pixel coordinates of the center point of the recognition frame and the boundary points of the recognition frame in the image coordinate system, and convert them into camera coordinates in the camera coordinate system: Among them, X C ,Y C ,Z C is the coordinate value of the point in pixel coordinates in the camera coordinate system, Z C The depth value of the pixel point measured by the depth camera is obtained; u, v are the pixel coordinates of the point, u0, v0, f x ,f y are the four parameter values ​​in the camera intrinsic parameter matrix K.

6. The valve identification and posture estimation method based on FPPE network according to claim 1 is characterized in that: Step S5 includes: Align the image captured by the color camera with the depth map captured by the depth camera; Coordinate transformation operation for converting the coordinates of the valve ROI boundary points in the color image to the depth camera coordinate system: P depth =R rgb2depth *K rgb -1 *Z rgb *p rgb +T rgb2depth Among them, P depth is the point in the depth camera coordinate system, R rgb2depth K is the external parameter rotation transformation matrix from RGB camera to depth camera, rgb is the camera intrinsic parameter matrix, Z rgb is the Z-axis coordinate of the point in the color camera coordinate system, p rgb is the homogeneous pixel coordinate of the image midpoint, T rgb2depth It is the external parameter translation transformation matrix from RGB camera to depth camera; Calculate the coordinates of the boundary points of the ROI in the depth coordinate system and the point cloud coordinate system, extract all points with different depth values ​​within the x, y coordinate range, and form a frustum point cloud set.

7. A valve identification and posture estimation method based on FPPE network according to any one of claims 1 to 6, characterized in that: The color camera is used to collect pictures at the positions directly in front of the valve, 15° to the left of the valve, 15° to the right of the valve, 30° to the left of the valve, 30° to the right of the valve, 45° to the left of the valve, 45° to the right of the valve, 60° to the left of the valve, and 60° to the right of the valve.

8. A valve identification and posture estimation system based on FPPE network, characterized in that: The system comprises: A data acquisition unit is used to use a color camera to collect multi-angle valve images, use a depth camera to collect valve point cloud data to construct a valve point cloud dataset, and construct a template point cloud; The object detection network training unit is used to train the RT-DETR v2 object detection network based on the collected valve images; An FPPE network construction unit is used to construct an FPPE network and train the FPPE network according to a valve point cloud dataset; The target detection unit is used to perform target recognition and planar target position detection on the valve image based on the trained RT-DETR v2 network, obtain the center position coordinates of the valve area and the pixel coordinates of the boundary points of the region of interest (ROI), and calculate the 3D camera coordinates of the region boundary points using the camera intrinsic parameter matrix and depth information; A view cone point cloud extraction unit is used to extract points of all depths within the ROI spatial area according to the pixel coordinates of the boundary points of the region of interest (ROI) and the three-dimensional camera coordinates of the region boundary points to form a view cone point cloud; The pose acquisition unit is used to input the cone point cloud into the trained FPPE network to obtain the final valve pose estimation result.

9. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a valve identification and posture estimation method based on an FPPE network according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of a valve identification and posture estimation method based on an FPPE network according to any one of claims 1 to 7.

Images