Method for identifying coordinates of a vehicle board transformer based on a depth image

Abstract

The present invention relates to the application of image processing in the warehousing field, particularly to the application of depth image technology in the warehousing field, and more specifically to a method for identifying the coordinates of a carboard transformer based on a depth image. The depth image information of the transformer is output by a depth camera, the spatial coordinates of the optimal grasping point of the transformer are calculated according to each image feature point, and the actual site coordinates of the carboard transformer are accurately identified by using a transformation matrix, ensuring the safe fork entry and efficient loading and unloading of the transformer. By adopting the technical solution disclosed by the present invention, the depth camera and the machine vision recognition system can meet the requirement of accurately obtaining the image information of the carboard transformer in variable warehousing scenarios and application scenarios with diverse transformer specifications and types, laying a foundation for subsequent intelligent identification of the coordinates of the carboard transformer.

Description

Technical Field

[0001] The present invention relates to the application of image processing in the field of warehousing, particularly to the application of depth image technology in the field of warehousing, and more specifically to a method for identifying the coordinates of a transformer on a vehicle board based on a depth image. Background Art

[0002] A depth image, also known as a range image, is an image in which the distance (depth) from an image collector to each point in a scene is used as a pixel value, and it directly reflects the geometric shape of the visible surface of a scene. The depth image can be calculated as point cloud data through coordinate transformation, and the point cloud data with regular and necessary information can also be inversely calculated as depth image data.

[0003] Currently, methods for obtaining depth images include lidar depth imaging method, computer stereo vision imaging, coordinate measuring machine method, moiré fringe method, structured light method, and so on.

[0004] In the present invention, we use a depth camera to obtain a depth image, and based on the depth image, develop a method for identifying the coordinates of a transformer on a vehicle board, so as to provide accurate, precise, and reliable transformer position information for automated loading and unloading during the warehousing process of heavy materials in the power industry, and provide a basis for improving the accuracy of automatic forklift picking of transformers. Summary of the Invention

[0005] The main technical problems to be solved by the present invention are mainly problems such as high customization of relevant material equipment, diverse loading methods and forms, and difficulty in promoting automated loading and unloading during the warehousing process of heavy materials in the power industry.

[0006] To solve the above technical problems, the present invention discloses a method for identifying the coordinates of a transformer on a vehicle board based on a depth image, including the following steps:

[0007] Step 1: Use a depth camera to photograph the transformer on the vehicle board and output the depth image of the transformer on the vehicle board;

[0008] Step 2: Label and process the picture, and respectively confirm the independent area of each transformer;

[0009] Step 3: Extract and associate the image feature point information in each frame of the depth image, and combine with the transformer independent area information in Step 2 to calculate the coordinate information of the optimal grasping point of the transformer;

[0010] Step 4: Convert the depth image into point cloud data, and obtain the three-dimensional spatial coordinate value of the optimal grasping point according to the coordinate information of the optimal grasping point obtained in Step 2;

[0011] Step 5: Using the depth camera as the calibration point, convert the three-dimensional spatial coordinate values of the optimal grasping point obtained in Step 4 into the actual site coordinate values of the optimal grasping point.

[0012] As a preferred technical solution, it further includes Step 6: According to the communication protocol, the actual site coordinate values of the optimal grasping point obtained in Step 4 are transmitted to the on-site intelligent loading and unloading equipment, and the on-site intelligent loading and unloading equipment completes the precise fork entry operation according to the received actual site coordinate values of the grasping point.

[0013] In the present invention, the method of marking and processing in Step 2 is further disclosed as follows: The transformer, the edge of the transformer contour, and the background of the transformer are respectively marked and processed with different colors, so that the transformer, the edge of the transformer contour, and the background in the depth image information are respectively marked with different colors, thereby clarifying the independent area information of each transformer.

[0014] As a preferred technical solution, in the present invention, the coordinate information of the optimal grasping point of the transformer obtained by the operation of the machine vision system in Step 3 is further disclosed. The specific method is: Extract the independent operation area information of the transformer and match it with all the image feature points obtained by the depth camera. Mark the image feature points that match successfully with the edge of the independent area as the edge contour coordinates. Compare the remaining image feature point information with the image library to identify the grasping point coordinates (there may be multiple), and mark them as the grasping point coordinates in sequence. Calculate the spatial position and distance between each grasping point coordinate and the edge contour coordinate to form a set of position parameters. Compare the position parameter information of each grasping point coordinate with the data of various types of transformers in the data to determine the final reasonable grasping point.

[0015] In a preferred technical solution, the present invention further discloses that the conversion method in Step 5 is to convert the coordinate values through a coordinate conversion matrix. The specific method is: Select a certain point in the depth image as the reference point, and calculate the three-dimensional spatial coordinate value (camera coordinate system) of the reference point in the point cloud image and the coordinate value in the visual recognition system coordinate system, so as to obtain the conversion matrix between the three-dimensional spatial coordinate system and the visual recognition system coordinate system; then, using the depth camera as the calibration point, utilize the imaging principle of the depth camera to obtain the conversion matrix between the three-dimensional spatial coordinate value of the reference point and the actual site coordinate value; and use this conversion matrix to convert the three-dimensional spatial coordinate value of the optimal grasping point obtained in Step 4 into the actual site coordinate value of the optimal grasping point.

[0016] A further preferred technical solution is that before each fork entry of the on-site intelligent loading and unloading equipment, it further includes a calibration and deviation correction program.

[0017] Meanwhile, as a preferred technical solution, in the present invention, before each fork insertion operation, the transformer materials to be forked are all subjected to the operations of steps 3 to 5.

[0018] The present invention outputs the depth image information of the transformer through a depth camera, calculates the spatial coordinates of the optimal grasping point of the transformer according to each image feature point, and uses the transformation matrix to accurately identify the actual site coordinates of the transformer on the vehicle board, ensuring the safe fork insertion and efficient loading and unloading of the transformer.

[0019] Adopting the technical solution disclosed in the present invention, the use of a depth camera and a machine vision recognition system can meet the need to accurately obtain the image information of the transformer on the vehicle board in variable storage scenarios and application scenarios with diverse transformer specifications and types, laying a foundation for subsequent intelligent identification of the coordinates of the transformer on the vehicle board. Based on the technical solution disclosed in the present invention, the previous way of relying on manual operation for transformer loading and unloading can be changed, and the problems of large labor intensity and low operation efficiency in transformer loading and unloading work can be solved. It is also worth noting that the present invention fills the technical gap in the field of intelligent visual recognition of transformer materials, and can also provide reference for the automated storage of related heavy materials. Specific Embodiments

[0020] To better understand the present invention, the following will further elaborate on the present invention in combination with specific embodiments.

[0021] In this embodiment, a method for identifying the coordinates of the transformer on the vehicle board based on a depth image is disclosed, including the following steps:

[0022] Step 1: Use a depth camera to photograph the transformer on the vehicle board and output the depth image of the transformer on the vehicle board;

[0023] Step 2: Label and process the picture to respectively confirm the independent area of each transformer;

[0024] Preferably, in this embodiment, the method of labeling and processing is to respectively perform labeling processing on the transformer, the contour edge of the transformer, and the background of the transformer with different colors, so that the transformer, the contour edge of the transformer, and the background in the depth image information are respectively marked with different colors, thereby clarifying the independent area information of each transformer.

[0025] Step 3: Extract and associate the image feature point information in each frame of the depth image, and combine with the independent area information of the transformer in step 2 to calculate the coordinate information of the optimal grasping point of the transformer;

[0026] Preferably, in this embodiment, the coordinate information of the optimal grasping point of the transformer is obtained through the operation of a machine vision system.

[0027] Step 4: Convert the depth image into point cloud data, and obtain the three-dimensional spatial coordinate values of the optimal grasping points according to the optimal grasping point coordinate information obtained in Step 2;

[0028] Step 5: Using the depth camera as the calibration point, convert the three-dimensional spatial coordinate values of the optimal grasping points obtained in Step 4 into the actual site coordinate values of the optimal grasping points.

[0029] In this embodiment, we preferably use a coordinate transformation matrix to perform the coordinate value transformation in this step. The specific method is as follows: Select a certain point in the depth image as the reference point, and calculate the three-dimensional spatial coordinate values (camera coordinate system) of the reference point in the point cloud map and the coordinate values in the visual recognition system coordinate system, so as to obtain the transformation matrix between the three-dimensional spatial coordinate system and the visual recognition system coordinate system; Then, using the depth camera as the calibration point and the imaging principle of the depth camera, obtain the transformation matrix between the three-dimensional spatial coordinate values of the reference point and the actual site coordinate values; And use this transformation matrix to convert the three-dimensional spatial coordinate values of the optimal grasping points obtained in Step 4 into the actual site coordinate values of the optimal grasping points.

[0030] At the same time, in this embodiment, it is further preferably included that Step 6: According to the communication protocol, transmit the actual site coordinate values of the optimal grasping points obtained in Step 4 to the on-site intelligent loading and unloading equipment, and the on-site intelligent loading and unloading equipment completes the accurate fork entry according to the received actual site coordinate values of the grasping points.

[0031] In this embodiment, it is further preferably that before each fork entry of the on-site intelligent loading and unloading equipment, a calibration and deviation correction program is also included. Specifically, after each execution of a fork entry and reverse fork operation, before the next fork entry, the operations of Steps 3 to 5 are performed on the target transformer materials to be forked.

[0032] Next, we will further illustrate with a specific on-site operation as an example.

[0033] After the vehicle loading the transformer arrives at the designated area, the depth camera takes pictures of the vehicle and the transformer from different angles, and obtains multiple frames of depth image information at multiple angles. For example, if there are 2 transformers on the vehicle floor, multiple scene pictures containing 2 transformers at different angles are obtained through the shooting of the depth camera.

[0034] Label the pictures of the transformers on the vehicle floor, mark the transformers, the edges of the transformer contours, the background of the transformer areas, etc. with different colors to confirm the independent areas of the 2 transformers on the vehicle floor.

[0035] Select feature points, and extract these feature points separately in the depth images at different angles, and then complete the feature point matching between the images by fusing the feature points in different fields of view.

[0036] Select the first transformer, extract its independent operation area information, and match it with the image feature points of the first transformer obtained by the depth camera. Mark the image feature points that successfully match the edge of the independent area as the edge contour coordinates a1, a2, a3..a 16 , compare the remaining image feature point information with the image library to identify the grasping point coordinates c1, c2, c3, mark them as the grasping point coordinates, calculate the spatial positions and distances between each grasping point coordinate c1, c2, c3 and the edge contour coordinates to form three groups of position parameters d1, d2, d3, and compare the position parameter information of each grasping point coordinate with various types of transformer data in the data to determine the final reasonable grasping point Ct.

[0037] Convert the depth image of the transformer into point cloud data, obtain the three-dimensional spatial coordinates (camera coordinate system) of the grasping points of each transformer respectively, use the transformation matrix calculated by the reference points to convert the camera coordinates of the transformer into the actual site coordinates, and then perform data transmission according to the defined communication protocol to trigger the intelligent loading and unloading equipment to accurately fork in.

[0038] It should be particularly noted that when forking and unloading the transformer, the height of the vehicle board may change. For example, after unloading the first transformer, the position coordinates of the second transformer on the vehicle board may change, and these errors will directly affect the accuracy of the forking operation.

[0039] To avoid damage to the transformer, when forking and loading / unloading the second transformer, the visual recognition system will perform secondary identification, calibration, and correction of the transformer coordinates to ensure smooth, safe, and efficient loading and unloading of the transformer.

[0040] The above is the specific implementation manner of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A method for identifying the coordinates of a vehicle board transformer based on a depth image, characterized in that It includes the following steps: Step 1: Use a depth camera to photograph the vehicle board transformer and output the depth image of the vehicle board transformer; Step 2: Label and process the picture to confirm the independent area of each transformer respectively; Step 3: Extract and associate the image feature point information in each frame of the depth image, and combine with the transformer independent area information in Step 2 to calculate the optimal grasping point coordinate information of the transformer; Step 4: Convert the depth image into point cloud data, and obtain the three-dimensional spatial coordinate value of the optimal grasping point according to the optimal grasping point coordinate information obtained in Step 3; Step 5: Taking the depth camera as the calibration point, convert the three-dimensional spatial coordinate value of the optimal grasping point obtained in Step 4 into the actual site coordinate value of the optimal grasping point; In Step 3, the optimal grasping point coordinate information of the transformer is obtained through the operation of the machine vision system. The independent area information of the transformer is extracted and matched with all the image feature points obtained by the depth camera. The image feature points successfully matched with the edge of the independent area are marked as the edge contour coordinates. The remaining image feature point information is compared with the image library to identify the grasping point coordinates, which are all marked as the grasping point coordinates in sequence. Calculate the spatial position and distance between each grasping point coordinate and the edge contour coordinate to form a set of position parameters. Compare the position parameter information of each grasping point coordinate with the data of various types of transformers in the data to determine the final reasonable grasping point.

2. The method for identifying the coordinates of the vehicle board transformer based on the depth image according to claim 1, wherein: It also includes Step 6: According to the communication protocol, transmit the actual site coordinate value of the optimal grasping point obtained in Step 5 to the on-site intelligent loading and unloading equipment, and the on-site intelligent loading and unloading equipment completes the accurate fork-in according to the received actual site coordinate value of the grasping point.

3. The method for identifying the coordinates of the vehicle board transformer based on the depth image according to claim 1, wherein The specific method of labeling and processing in Step 2 is: Label and process the transformer, the transformer contour edge, and the transformer background with different colors respectively, so that the transformer, the transformer contour edge, and the transformer background in the depth image information are respectively marked with different colors, so as to clarify the independent area information of each transformer.

4. The method for identifying the coordinates of the vehicle board transformer based on the depth image according to claim 1, wherein: The conversion method in Step 5 is to convert the coordinate value through a coordinate conversion matrix.

5. The method for identifying the coordinates of the vehicle board transformer based on the depth image according to claim 4, wherein, The conversion method in Step 5 is: Select a certain point in the depth image as the reference point, and calculate the three-dimensional spatial coordinate value (camera coordinate system) of the reference point in the point cloud map and the coordinate value in the visual recognition system coordinate system, so as to obtain the conversion matrix between the three-dimensional spatial coordinate system and the visual recognition system coordinate system; Then, taking the depth camera as the calibration point, use the imaging principle of the depth camera to obtain the conversion matrix between the three-dimensional spatial coordinate value of the reference point and the actual site coordinate value; And use this conversion matrix to convert the three-dimensional spatial coordinate value of the optimal grasping point obtained in Step 4 into the actual site coordinate value of the optimal grasping point.

6. The method for identifying the coordinates of the vehicle board transformer based on the depth image according to claim 2, wherein: Before each fork-in of the on-site intelligent loading and unloading equipment, it also includes a calibration and deviation correction program.

7. The method for identifying the coordinates of the vehicle board transformer based on the depth image according to claim 6, characterized in that: Before each fork-in operation, perform the operations of Steps 3 to 5 on the transformer materials to be forked in, so as to complete calibration and deviation correction.