A cylinder filling port pose detection method and a detection system

Abstract

The application discloses a kind of steel bottle filling port pose detection methods, the image of different pose of steel bottle filling port obtained is preliminary processed;The image after pre-processing is carried out block segmentation, different blocks are encoded, the coordinate on object coordinate system is converted into the encoding coordinate of encoding image, and each dimension block number mask is obtained, then steel bottle filling port pose detection model is established;The image of real-time acquisition steel bottle filling port is preliminary processed, then block number mask is obtained by detection model;Real coordinate feature map is obtained by decoding;Two-dimensional coordinate and corresponding three-dimensional coordinate are extracted;The correspondence between two-dimensional coordinate and three-dimensional coordinate obtained by EPnP algorithm is converted into the pose information of filling port.The feature map of new partition encoding coordinate is used to predict deep neural network, the average error on all pixels of feature map can be reduced proportionally, and its accuracy is improved.

Description

Technical Field

[0001] This invention relates to automated refrigerant filling, specifically to a method and system for detecting the position of a cylinder filling port. Background Technology

[0002] The refrigerant production process is relatively simple, involves high repetitive labor, and poses a certain risk to workers due to material leakage. Therefore, refrigerant filling is one of the industrial sectors that has seen rapid automation. Currently, refrigerant filling production lines include semi-automated and fully automated lines, which improve production efficiency and reduce production costs. One of the key challenges is the disassembly and reassembly process of unscrewing the nut to fill the refrigerant and then reinstalling it. To achieve this, it is necessary to obtain the pose information of the one-way valve filling port through a vision sensor, that is, the transformation relationship between the object coordinate system at the filling port and the camera coordinate system. This is crucial for the vision-guided robot to align the screwdriver actuator with the one-way valve filling port.

[0003] Detecting the 6D pose of a filling port is one of the research hotspots in vision in recent years. Based on the principle of deep neural networks, it continuously learns various prior pose data so that in the application stage, it can read three-dimensional information from a single RGB image and estimate the pose of the target object. Generally, such a network structure is U-shaped and consists of an encoder and a decoder. The encoder analyzes the input RGB image and extracts information, while the decoder is responsible for generating the required output.

[0004] Generally speaking, methods for achieving target object pose detection can be divided into direct methods and indirect methods. Direct methods can directly regress the rotation matrix and translation vector information of the object. Indirect methods require first regressing various feature maps created based on the coordinate information in the object coordinate system, then analyzing the feature maps to obtain the correspondence between the two-dimensional coordinates of the pixels and the three-dimensional coordinates in the object coordinate system, and finally using algorithms such as EPnP to obtain the transformation relationship from the object coordinate system to the camera coordinate system.

[0005] One indirect method uses the feature map of the three-dimensional coordinates in the object coordinate system as the output. Represented by Pix2Pose and CDPN, it uses the depth map obtained by the depth camera and the pose information obtained by detecting ArUco corner points to obtain the object coordinates corresponding to each pixel of the image. These coordinates are normalized to the interval [-1,1] for prediction. Then, the correspondence between 2D points and 3D points is obtained by decoding for EPnP processing, but its accuracy is still lacking. Summary of the Invention

[0006] Purpose of the invention: To address the above-mentioned shortcomings, this invention provides a highly accurate method for detecting the position of a steel cylinder filling port.

[0007] The present invention also provides a cylinder filling port position detection system.

[0008] Technical solution: To solve the above problems, the present invention employs a method for detecting the position of a steel cylinder filling port, comprising the following steps:

[0009] (1) Obtain images of the cylinder filling port in different poses, perform preliminary processing on the obtained images to obtain preprocessed images, and at the same time obtain the pose information of the cylinder filling port.

[0010] (2) The preprocessed image is segmented into blocks, different blocks are encoded, and the coordinates on the object coordinate system are converted into the encoded coordinates of the encoded image to obtain the feature map of the partitioned encoded coordinates.

[0011] (3) Apply an image mask to the feature map of the partition coding coordinates to obtain the block number mask for each dimension;

[0012] (4) Using the preprocessed image as input, the feature map of the partitioned coding coordinates, the mask image, and the block number mask are obtained as outputs to establish a cylinder filling port pose detection model;

[0013] (5) Acquire images of the cylinder filling port in real time, perform preliminary processing on the acquired images to obtain preprocessed images, and obtain block number masks in each dimension based on the preprocessed images through the cylinder filling port pose detection model.

[0014] (6) Decode the block number mask to obtain the real coordinate feature map; extract the two-dimensional coordinates of the pixels in the image and the three-dimensional coordinates of the corresponding three-dimensional object coordinate system;

[0015] (7) The correspondence between the obtained two-dimensional coordinates and three-dimensional coordinates is converted into the pose information of the filling port by the EPNP algorithm.

[0016] Furthermore, in step (1), the target image data in the acquired image is obtained through a bounding box detection algorithm.

[0017] Furthermore, step (1) specifically includes:

[0018] (1.1) Obtain RGB images of the cylinder filling port in different poses;

[0019] (1.2) Convert the acquired RGB image to grayscale;

[0020] (1.3) Perform adaptive threshold binarization on the grayscale image;

[0021] (1.4) Perform line detection, quadrilateral detection and corner extraction on the processed image;

[0022] (1.5) Determine the analysis plane and calculate the perspective transformation matrix from the image plane to the analysis plane of the processed image;

[0023] (1.6) Determine the coordinates of the four vertices of the bounding box of the analysis plane based on the shape of the gas cylinder (a i ,b i (i∈{1,2,3,4});

[0024] (1.7) Determine the four vertices (a, b, c) in the analysis plane. i ,b i (u) is obtained by projecting it onto the image plane by right-multiplying it by the perspective transformation matrix. i ,v i );

[0025] (1.8) Obtain the pixel coordinates (c) of the center of the bounding box in the image plane. x ,c y ) is ([max(u i )+min(u i )] / 2,[max(v i )+min(v i )] / 2), the bounding box length and width (S x ,S y ) is ([max(u i )-min(u i ),max(v i )-min(v i )]);

[0026] (1.9) Obtain the image of interest within the bounding box; scale the image of interest to 256×256 using bilinear interpolation.

[0027] Furthermore, in step (2), the block segmentation method for the preprocessed image is to divide it equally in each of the xyz dimensions of the image, set the number of blocks to N, and use the Gray code encoding idea to convert the block number n∈[0,N) into a code composed of 0 and 1.

[0028] Furthermore, the functional relationship between the encoded coordinates ZCC and the corresponding original coordinates C in the object coordinate system is as follows:

[0029]

[0030] Where i∈{0,1,2} corresponds to the three dimensions x,y,z, and max i I is half the span of the cylinder filling port image in the i-th dimension. n The interval is

[0031] Furthermore, a ResNet-34 network structure is used as the backbone network to extract image information. The output layers are a 64×64×3 feature map with partitioned coded coordinates, a 64×64×1 mask image, and a 64×64×n feature map. N The block number mask for each dimension, n N The number of channels of the block number mask across all dimensions xyz.

[0032] Furthermore, in step (6), the decoding process involves reading the block number mask in each dimension to obtain the block number of the corresponding point in the object coordinate system, and then obtaining the three-dimensional point coordinates in the object coordinate system based on the inverse function of the encoding function of the partitioned encoding coordinates.

[0033] The present invention also employs a cylinder filling port pose detection system, including a camera for acquiring cylinder filling port pose images, an image preliminary processing module, an image reprocessing module, a model building module, and a calculation module;

[0034] The image preliminary processing module is used to perform preliminary processing on the images acquired by the camera to obtain preprocessed images;

[0035] The image reprocessing module is used to segment the image after preprocessing by the image preliminary processing module, encode different blocks, convert the coordinates on the object coordinate system into the encoded coordinates of the encoded image, and obtain the feature map of the partitioned encoded coordinates; and to perform image masking on the feature map of the partitioned encoded coordinates to obtain the block number mask of each dimension.

[0036] The model building module is used to take the preprocessed image as input and the feature map of the partitioned coding coordinates, the mask image, and the block number mask as output to build a cylinder filling port pose detection model.

[0037] The calculation module is used to obtain block number masks in each dimension from the preprocessed image through the cylinder filling port pose detection model; decode the block number masks to obtain the real coordinate feature map; extract the two-dimensional coordinates of the pixels in the image and the corresponding three-dimensional coordinates in the three-dimensional object coordinate system; and convert the correspondence between the obtained two-dimensional coordinates and three-dimensional coordinates into the pose information of the filling port through the EPnP algorithm.

[0038] Beneficial effects: Compared with the prior art, the significant advantage of this invention is that it uses a feature map of novel partitioned coded coordinates to predict the deep neural network. Since the decoding process of this encoding method can proportionally reduce the average error of all pixels in the feature map, its accuracy is significantly improved. It can better adapt to the needs of automated refrigerant production and has a wide range of applications that can be expanded. Attached Figure Description

[0039] Figure 1 The flowchart shown is a process for the detection method of the present invention.

[0040] Figure 2 The diagram shown is a schematic representation of the overall structure of the steel cylinder in this invention.

[0041] Figure 3 The diagram shown is a schematic of the filling port of the steel cylinder when it is placed inside a cardboard box in this invention.

[0042] Figure 4 The diagram shows a method for determining the surrounding frame of the cylinder filling port in this invention.

[0043] Figure 5 The figure shows the image block numbering relationship using Gray code segmentation in this invention;

[0044] Figure 6 The figure shows the functional relationship between the partitioned coding coordinates and the original coordinates when N is 2, 4, or 7 in this invention. Detailed Implementation

[0045] like Figure 1 As shown in the figure, the target object of the cylinder filling port pose detection method in this embodiment is as follows: Figure 2 The filling port on the one-way valve of the shown cylinder is used to set up a station in the production process to receive the cylinders, which are transported from the roller conveyor line and packaged in cartons. Figure 3 As shown, the cylinder is pushed to a specific position on the workstation. Figure (3) (a) is a side view of the cylinder in the carton, and Figure (3) (b) is a top view of the cylinder in the carton. The detection method of this embodiment includes the following steps:

[0046] (1) A one-way valve filling port model of a steel cylinder was created using a depth camera and ArUco codes. The depth camera acquired RGB images of the filling port in different poses, partitioned coded coordinate feature maps, mask images, block number masks, and the transformation relationship between the object coordinate system and the camera coordinate system set on the filling port, i.e., the poses, as training data. The trained network parameters were stored on a PC or input into the internal storage of the smart camera. The partitioned coded coordinate feature map was created by dividing the acquired filling port image into several blocks using a block segmentation method, and using different encoding methods for different blocks to convert the coordinates on the object coordinate system into coded coordinate values ​​for network prediction.

[0047] (2) During the production process, a station is set up where the steel cylinders, packed in cartons and transported from the roller conveyor, are placed at a specific position. Using an eye-to-hand camera configuration, a fixed 2D camera is used to capture RGB images. Then, a bounding box detection algorithm is used to obtain the bounding box data of the filling port. The bounding box detection algorithm is as follows: Figure 4As shown, this method is adopted based on the actual situation of the gas cylinder being packed in a cardboard box, i.e., the pose change of the gas cylinder filling port relative to the square opening of the cardboard box is relatively small. The general process is as follows: image capture, RGB image grayscale conversion, adaptive threshold binarization, line detection, quadrilateral detection and corner extraction, calculation of the perspective transformation matrix from the image plane to the analysis plane, and the determination of the four vertices (a, b, c) in the analysis plane. i ,b i (u) is obtained by projecting it onto the image plane by right-multiplying it by the perspective transformation matrix. i ,v i (i∈{1,2,3,4}), then set the pixel coordinates of the bounding box center (c x ,c y ) is ([max(u i )+min(u i )] / 2,[max(v i )+min(v i )] / 2), length and width (S x ,S y ) is ([max(u i )-min(u i ),max(v i )-min(v i )]), while (a i ,b i The coordinates of the image are selected based on the style of the cardboard box and the gas cylinder to obtain the image of interest within the bounding box. The image of interest is then extracted and uniformly scaled to a length and width of 256×256 using bilinear interpolation as input.

[0048] The method for segmenting the scaled image into blocks is to divide it equally along each of the x, y, and z dimensions, with the number of blocks set to N, such as... Figure 5 The idea of ​​Gray code encoding is used to convert the block number n∈[0,N) into a code composed of 0 and 1, and a block number mask is made based on this to predict the block where each coordinate value is located, so as to serve as a reference during decoding.

[0049] The partitioned coded coordinates are coded based on the block number n where the coordinate point on the object model is located, designated as ZCC. The functional relationship between the coded coordinates ZCC and the corresponding original coordinates C in the object coordinate system is as follows:

[0050]

[0051] Where i∈{0,1,2} corresponds to the three dimensions x,y,z, and max i Let I be half the span of the filling port model in the i-th dimension (taking the center of the filling port in the three dimensions as the origin of the object coordinate system). n The interval is Figure 6 The diagram shows the partitioned coding coordinates ZCC in each dimension when N is 2, 4, or 7. i With the original coordinates C i The functional relationship between them.

[0052] The ResNet-34 network structure is used as the backbone network to extract image information. The output layers are a 64×64×3 feature map with partitioned coded coordinates, a 64×64×1 mask image, and a 64×64×n feature map. N The block number mask for each dimension, n N The number of channels of the mask for the block numbering across all xyz dimensions is equal to... The decoding process yields the true coordinate feature map. The decoding process involves reading the block number mask in each dimension to obtain the block number of the corresponding point in the object coordinate system. The three-dimensional point coordinates in the object coordinate system are obtained according to the inverse function of the encoding function of the partitioned encoding coordinates. The two-dimensional coordinates of the pixels in the image and the corresponding three-dimensional coordinates in the three-dimensional object coordinate system are then extracted.

[0053] (3) The correspondence between the two-dimensional and three-dimensional coordinates obtained in (2) is converted into the pose information of the filling port through the EPNP algorithm. Based on this, the screw gun installed on the robotic arm is aligned with the filling port and the nut on it is unscrewed.

Claims

1. A method for detecting the position of a gas cylinder filling port, characterized in that, Includes the following steps: (1) Acquire images of the cylinder filling port in different poses, perform preliminary processing on the acquired images to obtain preprocessed images, and simultaneously acquire the pose information of the cylinder filling port; specifically including: (1.1) Obtain RGB images of the gas cylinder filling port in different poses; (1.2) Convert the acquired RGB image to grayscale; (1.3) Perform adaptive threshold binarization on the grayscale image; (1.4) Perform line detection, quadrilateral detection and corner extraction on the processed image; (1.5) Determine the analysis plane and calculate the perspective transformation matrix from the image plane to the analysis plane of the processed image; (1.6) Determine the coordinates of the four vertices of the bounding box of the analysis plane based on the shape of the gas cylinder. a i, b i () i ∈{1,2,3,4}); (1.7) Determine the four vertices in the analysis plane ( a i, b i ) is projected onto the image plane by right multiplying by the perspective transformation matrix to obtain ( u i , v i ); (1.8) Obtain the pixel coordinates of the center of the bounding box in the image plane. c x , c y ) is ([max( u i )+min( u i )] / 2, [max( v i )+min( v i )] / 2), bounding box length and width ( S x , S y ) is ([max( u i )-min( u i ), max( v i )-min( v i )]); (1.9) Obtain the image of interest within the bounding box; scale the image of interest to 256×256 using bilinear interpolation; (2) The preprocessed image is segmented into blocks. The idea of ​​Gray code encoding is used to convert the block number n∈[0, N) into a code composed of 0 and 1. The coordinates on the object coordinate system are converted into the coded coordinates of the coded image to obtain the feature map of the partitioned coded coordinates. (3) Obtain the mask image based on the feature map of the partition coding coordinates, and divide the mask image into blocks to obtain the block number mask of each dimension; (4) Using the preprocessed image as input, the feature map of the partitioned coding coordinates, the mask image, and the block number mask are obtained as outputs to establish a cylinder filling port pose detection model; (5) Acquire images of the cylinder filling port in real time, perform preliminary processing on the acquired images to obtain preprocessed images, and obtain block number masks in each dimension through the cylinder filling port pose detection model based on the preprocessed images. (6) Decode the block number mask to obtain the real coordinate feature map; extract the two-dimensional coordinates of the pixels in the image and the three-dimensional coordinates of the corresponding three-dimensional object coordinate system; (7) The correspondence between the obtained two-dimensional coordinates and three-dimensional coordinates is converted into the pose information of the filling port by the EPnP algorithm.

2. The method for detecting the position of the filling port of a gas cylinder according to claim 1, characterized in that, In step (1), the target image data in the acquired image is obtained through the bounding box detection algorithm.

3. The method for detecting the position of the filling port of a gas cylinder according to claim 2, characterized in that, In step (2), the preprocessed image is divided into blocks by dividing it equally in each of the x, y, and z dimensions of the image. The number of blocks is set to N, and the block number n∈[0, N) is converted into a code composed of 0 and 1 using the Gray code encoding idea.

4. The method for detecting the position of the filling port of a gas cylinder according to claim 3, characterized in that, The encoded coordinates ZCC Corresponding original coordinates in the object coordinate system C The functional relationship is as follows: ； in, i ∈{0, 1, 2} corresponds to xyz Three dimensions, max i Image of the cylinder filling port in the first i Half of the span in each dimension The interval is .

5. The method for detecting the position of the filling port of a gas cylinder according to claim 4, characterized in that, The ResNet-34 network structure is used as the backbone network to extract image information. The intermediate output layer consists of a 64×64×3 feature map with partitioned coded coordinates and a 64×64×1 mask image. The output layer is 64×64×... The block number mask for each dimension, for xyz The number of channels of the block number mask across all dimensions.

6. The method for detecting the position of the filling port of a gas cylinder according to claim 1, characterized in that, In step (6), the decoding process involves reading the block number mask in each dimension to obtain the block number of the corresponding point in the object coordinate system, and then obtaining the three-dimensional point coordinates in the object coordinate system based on the inverse function of the encoding function of the partitioned encoding coordinates.

7. A cylinder filling port position detection system, characterized in that, It includes a camera for acquiring images of the cylinder filling port pose, an image preprocessing module, an image reprocessing module, a model building module, and a calculation module; The image preliminary processing module is used to perform preliminary processing on the images acquired by the camera to obtain preprocessed images; specifically, it includes: (1.1) Obtain RGB images of the gas cylinder filling port in different poses; (1.2) Convert the acquired RGB image to grayscale; (1.3) Perform adaptive threshold binarization on the grayscale image; (1.4) Perform line detection, quadrilateral detection and corner extraction on the processed image; (1.5) Determine the analysis plane and calculate the perspective transformation matrix from the image plane to the analysis plane of the processed image; (1.6) Determine the coordinates of the four vertices of the bounding box of the analysis plane based on the shape of the gas cylinder. a i, b i () i ∈{1,2,3,4}); (1.7) Determine the four vertices in the analysis plane ( a i, b i ) is projected onto the image plane by right multiplying by the perspective transformation matrix to obtain ( u i , v i ); (1.8) Obtain the pixel coordinates of the center of the bounding box in the image plane. c x , c y ) is ([max( u i )+min( u i )] / 2, [max( v i )+min( v i )] / 2), bounding box length and width ( S x , S y ) is ([max( u i )-min( u i ), max( v i )-min( v i )]); (1.9) Obtain the image of interest within the bounding box; scale the image of interest to 256×256 using bilinear interpolation; The image reprocessing module is used to segment the image after preprocessing by the image preliminary processing module. It uses Gray code to convert the block number n∈[0, N) into a code composed of 0 and 1, and converts the coordinates on the object coordinate system into the coded coordinates of the coded image to obtain the feature map of the partitioned coded coordinates. Then, a mask image is obtained based on the feature map of the partitioned coded coordinates, and the mask image is divided into blocks to obtain the block number mask of each dimension. The model building module is used to take the preprocessed image as input and the feature map of the partitioned coding coordinates, the mask image, and the block number mask as output to build a cylinder filling port pose detection model. The calculation module is used to obtain block number masks in each dimension from the preprocessed image through the cylinder filling port pose detection model; decode the block number masks to obtain the real coordinate feature map; extract the two-dimensional coordinates of the pixels in the image and the corresponding three-dimensional coordinates in the three-dimensional object coordinate system; and convert the correspondence between the obtained two-dimensional coordinates and three-dimensional coordinates into the pose information of the filling port through the EPnP algorithm.

Images