Automatic visual positioning method and system for industrial equipment

Abstract

The invention relates to the technical field of image processing, in particular to an industrial equipment automatic visual positioning method and system, and the method comprises the steps: obtaining a plurality of equipment images corresponding to different types of industrial equipment in a production process; extracting feature points in each equipment image, and respectively determining a pointing value of each feature point to each industrial equipment; distributing each feature point to the industrial equipment with the maximum pointing value to obtain optimal feature data corresponding to each industrial equipment; training and testing the machine learning model based on the preferable feature data corresponding to each industrial device to obtain an industrial device identification model; inputting to-be-detected feature data extracted from the to-be-detected image into the industrial equipment identification model to obtain an industrial equipment category output by the industrial equipment identification model; and based on the industrial equipment category, positioning the industrial equipment in the to-be-detected image. According to the scheme provided by the invention, the identification precision of the industrial equipment category is improved, and the positioning accuracy and reliability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to an industrial equipment automatic vision positioning method and system. Background Art

[0002] At present, industrial robots have been widely used in various production activities in the manufacturing industry, including production processes such as assembly, welding, spraying, and polishing. During the production process, industrial robots need to accurately grasp industrial equipment to ensure that target operations can be carried out accurately. In the process of an industrial robot grasping industrial equipment, it is first necessary to position the industrial equipment. In the positioning link, machine vision technology is usually used. Specifically, images are collected and a series of processes are performed to achieve the recognition of industrial equipment, and then positioning is achieved based on the recognition result.

[0003] In the related art, in the recognition link, since the industrial equipment is moving, the contour feature parameters will change, and the contour feature parameters of similar industrial equipment may be the same. Therefore, when there are many types of industrial equipment, the characterization ability of the contour feature parameters is weak, lacking the proprietary features of each industrial equipment, which easily leads to recognition errors and will affect the subsequent positioning accuracy.

[0004] It can be seen that the traditional industrial equipment vision positioning method has the problems of low recognition accuracy and insufficient positioning accuracy and reliability. Summary of the Invention

[0005] In order to solve the technical problems that the traditional industrial equipment vision positioning method has low recognition accuracy and insufficient positioning accuracy and reliability, the purpose of the present invention is to provide an industrial equipment automatic vision positioning method and system, and the specific technical solutions adopted are as follows:

[0006] An industrial equipment automatic vision positioning method, the method includes:

[0007] Obtain multiple device images corresponding to different types of industrial equipment during the production process;

[0008] Extract the feature points in each device image and respectively determine the pointing value of each feature point to each type of industrial equipment;

[0009] Assign each feature point to the industrial equipment with the largest pointing value to obtain the preferred feature data corresponding to each type of industrial equipment;

[0010] Based on the preferred feature data corresponding to each type of industrial equipment, train and test a pre-constructed machine learning model to obtain an industrial equipment recognition model;

[0011] Input the feature data to be measured extracted from the image to be measured into the industrial equipment recognition model, and obtain the industrial equipment category output by the industrial equipment recognition model;

[0012] Based on the industrial equipment category, locate the industrial equipment in the image to be measured.

[0013] According to an industrial equipment automatic vision positioning method provided by the present invention, extract feature points in each equipment image, including:

[0014] Determine the gray information and depth information of each pixel point in each equipment image;

[0015] According to the gray information of each pixel point in each equipment image, determine the object area where the industrial equipment is located in each equipment image;

[0016] Based on the depth information of each pixel point in the object area of each equipment image, establish a three-dimensional terrain model corresponding to the object area in each equipment image;

[0017] Perform regional division on the three-dimensional terrain model corresponding to the object area in each equipment image to obtain a plurality of pixel point sets;

[0018] Take the pixel point with the maximum or minimum depth information in each pixel point set as the feature point, and obtain the feature points in each equipment image.

[0019] According to an industrial equipment automatic vision positioning method provided by the present invention, based on the depth information of each pixel point in the object area of each equipment image, establish a three-dimensional terrain model corresponding to the object area in each equipment image, including:

[0020] Based on the pixel point with the maximum depth information in the object area of each equipment image, determine the horizon;

[0021] Take the depth distance between each pixel point in the object area of each equipment image and the horizon as the height of each pixel point;

[0022] Based on the heights of all pixel points in the object area of each equipment image, obtain a three-dimensional terrain model corresponding to the object area in each equipment image.

[0023] According to an industrial equipment automatic vision positioning method provided by the present invention, respectively determine the pointing value of each feature point to each industrial equipment, including:

[0024] Calculate the feature performance value corresponding to each pixel point set;

[0025] Take the pixel point set with the maximum feature performance value in the pixel point sets with the same feature points as the feature pixel point set;

[0026] Determine the matching region corresponding to each set of characteristic pixel points in each device image;

[0027] Calculate the matching degree between the feature points in each set of characteristic pixel points and each matching region;

[0028] Calculate the pointing value of each feature point to each industrial device based on the matching degree between the feature points in each set of characteristic pixel points and each matching region.

[0029] According to an industrial device automatic vision positioning method provided by the present invention, calculate the characteristic performance value corresponding to each pixel point set, including:

[0030] Determine the number of valley point sets and the number of watershed line point sets in the pixel point set obtained in each region division step respectively, and determine the number of pixel points in each pixel point set, the maximum and minimum values of the pixel point heights, and the height difference between any set of adjacent pixel points on each center connection line;

[0031] Calculate the quantity difference degree between valleys and watershed lines in each region division step according to the number of valley point sets and the number of watershed line point sets;

[0032] Calculate the average height distribution value of each pixel point set according to the number of pixel points and the maximum and minimum values of the pixel point heights in each pixel point set;

[0033] Calculate the sum of the height differences of adjacent pixel points according to the height differences of all sets of adjacent pixel points on each center connection line in each pixel point set;

[0034] Calculate the characteristic performance value corresponding to each pixel point set based on the quantity difference degree, the average height distribution value, and the sum of the height differences of adjacent pixel points.

[0035] According to an industrial device automatic vision positioning method provided by the present invention, calculate the matching degree between the feature points in each set of characteristic pixel points and the matching region, including:

[0036] Determine the shape feature and relative position feature of the feature points in each pixel point set respectively, and determine the shape feature and relative position feature of the matching points corresponding to the feature points in the matching region;

[0037] Calculate the edit distance between the shape feature of the feature points in each pixel point set and the shape feature of the corresponding matching points to obtain a first distance value;

[0038] Calculate the edit distance between the relative position feature of the feature points in each pixel point set and the relative position feature of the corresponding matching points to obtain a second distance value;

[0039] Based on the first distance value and the second distance value, calculate the matching degree between the feature points in each set of feature pixel points and the matching region.

[0040] According to an industrial equipment automated vision positioning method provided by the present invention, based on the matching degree between the feature points in each set of feature pixel points and each matching region, calculate the pointing value of each feature point to each type of industrial equipment, including:

[0041] Sum the matching degrees between the feature points in each set of feature pixel points and all the matching regions corresponding to each type of industrial equipment to obtain the matching sum value of each feature point to each type of industrial equipment;

[0042] Determine the maximum value among the matching degrees between each feature point and each matching region corresponding to each type of industrial equipment to obtain the first maximum matching value;

[0043] For any type of industrial equipment, determine the maximum value among the matching degrees between each feature point and each matching region corresponding to other types of industrial equipment to obtain the second maximum matching value;

[0044] Calculate the absolute value of the difference between the first maximum matching value and the second maximum matching value to obtain the maximum matching difference value;

[0045] Based on the maximum matching difference value and the matching sum value, calculate the pointing value of each feature point to each type of industrial equipment.

[0046] According to an industrial equipment automated vision positioning method provided by the present invention, based on the preferred feature data corresponding to each type of industrial equipment, train and test a pre-constructed machine learning model to obtain an industrial equipment recognition model, including:

[0047] Based on the preferred feature data corresponding to each type of industrial equipment and the type of industrial equipment corresponding to the preferred feature data, establish a sample data set;

[0048] Divide the sample data set into a training sample set and a test sample set according to a preset ratio;

[0049] Perform sample sampling on the training sample set by means of random sampling with replacement to generate multiple training subsets;

[0050] Train a pre-constructed machine learning model through the multiple training subsets, and test the trained machine learning model through the test sample set to obtain an industrial equipment recognition model.

[0051] According to an industrial equipment automated vision positioning method provided by the present invention, obtain multiple device images corresponding to different types of industrial equipment in the production process, including:

[0052] Receive multiple initial images corresponding to different industrial devices during the production process uploaded by a depth camera; among them, the multiple initial images are used to characterize the forms of industrial devices at multiple angles and in multiple directions;

[0053] Perform grayscale processing on each initial image to obtain multiple device images corresponding to different industrial devices during the production process.

[0054] On the other hand, the present invention also provides an industrial device automated vision positioning system, and the system includes:

[0055] An acquisition module for acquiring multiple device images corresponding to different industrial devices during the production process;

[0056] An extraction module for extracting feature points in each device image and respectively determining the pointing values of each feature point to each industrial device;

[0057] An allocation module for allocating each feature point to the industrial device with the largest pointing value to obtain preferred feature data corresponding to each industrial device;

[0058] A modeling module for training and testing a pre-constructed machine learning model based on the preferred feature data corresponding to each industrial device to obtain an industrial device recognition model;

[0059] An identification module for inputting the measured feature data extracted from the image to be measured into the industrial device recognition model to obtain the industrial device category output by the industrial device recognition model;

[0060] A positioning module for positioning the industrial device in the image to be measured based on the industrial device category.

[0061] The present invention has the following beneficial effects:

[0062] By obtaining multiple device images corresponding to different industrial devices in the production process, extracting feature points in each device image, and respectively determining the pointing values of each feature point to each industrial device, each feature point is assigned to the industrial device with the largest pointing value to obtain the preferred feature data corresponding to each industrial device. Based on the preferred feature data corresponding to each industrial device, the pre-constructed machine learning model is trained and tested to obtain an industrial device recognition model. The measured feature data extracted from the image to be measured is input into the industrial device recognition model to obtain the industrial device category output by the industrial device recognition model. Finally, based on the industrial device category, the industrial device in the image to be measured is located. Since the industrial device recognition model relied on in the recognition link is trained and tested based on the preferred feature data of each industrial device, and the preferred feature data is obtained based on the pointing values of each feature point to each industrial device, it can enable the preferred feature data to more accurately represent the characteristics of the corresponding category of industrial devices, with stronger feature representation ability. Furthermore, based on the industrial device recognition model, accurate recognition of the industrial device category can be achieved, improving the recognition accuracy, and thus enhancing the positioning accuracy and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following-described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0064] Figure 1 FIG. is a flowchart of a method for automatic visual positioning of industrial devices provided by an embodiment of the present invention;

[0065] Figure 2 is an image of a cube at an angle;

[0066] Figure 3 is an image of the cube at another angle;

[0067] Figure 4 is a schematic structural diagram of a three-dimensional terrain model corresponding to the object area;

[0068] Figure 5 is a schematic diagram of the mountain and valley forms that appear during the process of filling with water;

[0069] Figure 6 FIG. is a schematic structural diagram of a system for automatic visual positioning of industrial devices provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0070] To further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation manners, structures, features, and effects of an industrial equipment automated vision positioning method and system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0072] The following, in combination with the attached Figures 1 to 6 Specifically illustrates the specific solutions of an industrial equipment automated vision positioning method and system provided by the present invention.

[0073] Please refer to Figure 1 , which shows the method flowchart of an industrial equipment automated vision positioning method provided by an embodiment of the present invention. As Figure 1 shown, the above-mentioned industrial equipment automated vision positioning method specifically includes:

[0074] Step 110: Obtain multiple device images corresponding to different industrial equipment during the production process.

[0075] In this embodiment, the multiple device images corresponding to each industrial equipment can represent the forms of the industrial equipment at different angles and in different directions during the production process.

[0076] Step 120: Extract feature points from each device image and respectively determine the pointing value of each feature point to each industrial equipment.

[0077] In this embodiment, the pointing value of each feature point to each industrial equipment can represent the degree to which the feature point only points to that industrial equipment. The higher the pointing value, the stronger the pointing property of the feature point to that industrial equipment, and the more the feature point can uniquely represent the characteristics of that industrial equipment.

[0078] Step 130: Assign each feature point to the industrial equipment with the largest pointing value to obtain the preferred feature data corresponding to each industrial equipment.

[0079] It can be understood that the preferred feature data is a set of feature points that can best represent the characteristics of that industrial equipment after screening all feature points according to the pointing value. In this embodiment, the preferred feature data corresponding to each industrial equipment is obtained by calculating the pointing value. The representation ability of the feature data is stronger, which can provide more accurate data support for the generation of the subsequent industrial equipment recognition model, and thus can improve the accuracy and reliability of the recognition link.

[0080] Step 140: Based on the preferred feature data corresponding to each industrial device, train and test a pre-constructed machine learning model to obtain an industrial device recognition model.

[0081] In this embodiment, the construction of the industrial device recognition model can provide more convenient recognition conditions for the recognition link, and can achieve efficient and accurate recognition of industrial device categories.

[0082] Step 150: Input the measured feature data extracted from the image to be measured into the industrial device recognition model to obtain the industrial device category output by the industrial device recognition model.

[0083] It can be understood that since the input of the industrial device recognition model is feature data and the output is the industrial device category, before recognizing the image to be measured, it is necessary to first extract the measured feature data from the image to be measured, input the measured feature data into the industrial device recognition model, and then obtain the recognition result.

[0084] Step 160: Locate the industrial device in the image to be measured based on the industrial device category.

[0085] The solution provided in this embodiment improves the industrial device category recognition link, uses the pointing value to implement feature screening to obtain the preferred feature data of each industrial device, and then uses the preferred feature data to establish an industrial device recognition model to achieve accurate recognition of industrial device categories. Subsequently, based on the industrial device category, the industrial device in the image to be measured can be accurately located.

[0086] In one embodiment, obtaining multiple device images corresponding to different industrial devices during the production process specifically includes:

[0087] First, receive multiple initial images corresponding to different industrial devices during the production process uploaded by the depth camera; among them, the multiple initial images are used to represent the forms of the industrial devices at multiple angles and in multiple directions.

[0088] In practical applications, a depth camera can be installed at the end of the robotic arm of an industrial robot. The depth camera is arranged facing the industrial device. During the movement of the industrial robot, the industrial camera captures images at a fixed frame rate, and the frame rate can be set to 30fps to obtain multiple initial images corresponding to different industrial devices. The multiple initial images can cover the form information of the industrial devices at different angles and in different directions.

[0089] Then, perform grayscale processing on each initial image to obtain multiple device images corresponding to different industrial devices during the production process.

[0090] It is understandable that by grayscale processing the initial image, the grayscale information of any pixel can be obtained. At the same time, by shooting with a depth camera, the depth information of any pixel can be obtained, that is, the actual distance of the depth camera from the corresponding pixel on the industrial device.

[0091] In this embodiment, the actual coordinates of the industrial device relative to the depth camera can be directly obtained by using the depth camera, which can reduce the calibration process of the coordinate system and reduce the calculation error.

[0092] In one embodiment, the feature points in each device image are extracted, including:

[0093] First step, determine the grayscale information and depth information of each pixel point in each device image.

[0094] It is understandable that no matter how the shooting angle of the depth camera changes, the industrial device corresponding to the captured device image is unique. Therefore, it is necessary to select from the feature points that describe the essential features of the industrial device. The distance of the depth camera from each position point on the industrial device, that is, the depth information, can characterize the concave-convex structure on the surface of the industrial device, which is a relatively clear essential feature. Therefore, in this embodiment, the grayscale information and the depth information are combined to realize the feature extraction function.

[0095] Taking a cube as an example, each image can only capture a part of the cube. Figure 2 and Figure 3 are respectively two images of the cube at different angles. A1, A2, A3, and A4 are respectively the four front points of the cube. A ′ 1, A ′ 2, A ′ 3, A ′ 4 are respectively the four back points of the cube. The surfaces captured at the two angles are respectively the surface A1A2A4A3, the surface A ′ 1A ′ 2A2A1, the surface A2A ′ 2A ′ 4A4. Point A2 is connected to three sides. Then, from the change of the depth information of point A2 spreading to the edge along the contour, a part of the characteristics of the cube can be represented. Therefore, point A2 can be used as a feature point, and the spreading part can be used as the feature basis of this feature point.

[0096] Second step, according to the grayscale information of each pixel point in each device image, determine the object area where the industrial device is located in each device image.

[0097] In this embodiment, edge detection can be performed on industrial equipment in the device image using an edge detection algorithm to obtain the edge contour of the industrial equipment, and the area within the edge contour of the industrial equipment is the object area. In practical applications, edge detection can be implemented using an edge detection algorithm based on the Sobel operator or the Canny operator.

[0098] In the third step, according to the depth information of each pixel point in the object area of each device image, a three-dimensional terrain model corresponding to the object area in each device image is established.

[0099] Among the feature points on the mechanical structure surface, there are points that can express various structural features such as contours, edges, and vertices. However, when the observation angle of the industrial equipment changes, the structural features corresponding to some points will change due to the change in geometric shape in the device image. Although the contour of the industrial equipment will change, the relative positions of the various structures of the industrial equipment will not change. For example, Figure 2 and Figure 3 in which A2 represents a vertex of the cube, and this vertex will not become a concave point at any angle. Therefore, feature points can be screened according to the height of each position on the industrial equipment.

[0100] In this embodiment, in order to more accurately obtain the structural features that can effectively describe the industrial equipment presented in the device image, based on binocular vision, a three-dimensional terrain model of the corresponding industrial equipment is further established according to the horizontal and vertical coordinates and depth information of each pixel point in the device image. According to the different degrees of representation of different pixel points in the three-dimensional terrain model after diffusion, the three-dimensional terrain model is regarded as a topographic map. Valleys and peaks can represent an area better than plains. Therefore, areas similar to valleys and peaks can be selected from the three-dimensional terrain model to obtain feature information.

[0101] In a specific implementation, according to the depth information of each pixel point in the object area of each device image, a three-dimensional terrain model corresponding to the object area in each device image is established, which specifically includes:

[0102] First, according to the pixel point with the maximum depth information in the object area of each device image, the horizon is determined.

[0103] Then, the depth distance between each pixel point in the object area of each device image and the horizon is used as the height of each pixel point.

[0104] Finally, according to the heights of all pixel points in the object area of each device image, a three-dimensional terrain model corresponding to the object area in each device image is obtained.

[0105] Figure 4 An exemplary three-dimensional terrain model corresponding to the object area is shown. Figure 4210 in the figure represents the object area in the device image, and 220 represents one of the height change curves. The three-dimensional terrain model can reflect the height relationship of the surface of the industrial device. Since the concave and convex shapes do not change under different shooting angles, the concave and convex area structure can be used as feature information, which can reduce the recognition error caused by different shooting angles.

[0106] In this embodiment, the concave and convex structure is analogized to the valleys and peaks in the topographic map, and the characteristic points on its surface are extracted by using the change characteristics of the industrial device in the depth image. In addition, height change curves are formed between different pixel points in the depth image. In order to analyze the change characteristics of the height change curves, this embodiment is based on the watershed algorithm, and the concave and convex characteristics of the industrial device are analyzed by continuously filling water into the height change curves. As Figure 5 shown, during the process of filling water, mountains and valleys will appear. Therefore, the idea of the watershed algorithm can be referred to analyze the changes of mountains and valleys in the three-dimensional terrain model and determine the characteristic points. Figure 5 In the figure, 310 represents the horizontal plane, 320 represents the watershed line, and 330 represents the local minimum valley.

[0107] The fourth step is to divide the three-dimensional terrain model corresponding to the object area in each device image to obtain multiple pixel point sets.

[0108] In this embodiment, starting from the horizon, water is filled in units of 1 mm until the water fills to the maximum height, and several segmentation processes are obtained. For any segmentation process, the three-dimensional terrain model is divided into several isolated regions, and the process of filling water is also the process in which the isolated regions change and finally merge.

[0109] For any segmentation process, all isolated regions are divided into several valley pixel point sets, several watershed line pixel point sets and other multiple pixel point sets. Each pixel point set is a continuous region with similar depth on the industrial device. For any pixel point set, centered on the depth extreme point, it spreads to other pixel points. If the depth change is more obvious, the structural change at the corresponding position on the industrial device is more obvious, and the characteristic performance of this pixel point set is more obvious. Then this pixel point set can better characterize the structural characteristics of the industrial device.

[0110] The fifth step is to use the pixel point with the maximum or minimum depth information in each pixel point set as the characteristic point to obtain the characteristic points in each device image.

[0111] In one embodiment, the pointing values of each characteristic point to each industrial device are determined respectively, including:

[0112] The first step is to calculate the characteristic performance value corresponding to each pixel point set.

[0113] In a specific implementation, calculating the characteristic performance value corresponding to each pixel point set specifically includes:

[0114] First, respectively determine the number of valley point sets and the number of watershed line point sets in the pixel point sets obtained in each region division step, and determine the number of pixel points in each pixel point set, the maximum and minimum values of the pixel point heights, and the height difference between any set of adjacent pixel points on each center connection line.

[0115] Then, based on the number of valley point sets and the number of watershed line point sets, calculate the quantity difference degree between valleys and watershed lines in each region division step.

[0116] Subsequently, based on the number of pixel points and the maximum and minimum values of the pixel point heights in each pixel point set, calculate the average height distribution value of each pixel point set.

[0117] Next, based on the height differences between all sets of adjacent pixel points on each center connection line in each pixel point set, calculate the sum of the height differences of adjacent pixel points.

[0118] Finally, based on the quantity difference degree, the average height distribution value, and the sum of the height differences of adjacent pixel points, calculate the characteristic performance value corresponding to each pixel point set.

[0119] In this embodiment, the characteristic performance value of the j-th pixel point set in the i-th region division step can be calculated as follows:

[0120]

[0121] F i,j represents the characteristic performance value of the j-th pixel point set in the i-th region division step; n i,P and n i,L respectively represent the number of valley point sets and the number of watershed line point sets in the i-th region division step; represents the quantity difference degree between valleys and watershed lines in the i-th region division step. The quantity difference degree can characterize the current segmentation degree. Both too large and too small indicate that the current segmentation process is not sufficient, and most regions are regarded as valleys or peaks. Among them, there are some device structures with relatively gentle depth changes, so the characteristic performance of the pixel point sets therein is also small; n i,j represents the number of pixel points in the j-th pixel point set in the i-th region division step; d1 i,j and d2 i,j respectively represent the maximum height and the minimum height of the j-th pixel point set in the i-th region division step, that is, the maximum and minimum values of the pixel point heights; represents the average height distribution value of the j-th pixel point set in the i-th region division step. The fewer the number of pixel points corresponding to any height, the more obvious the height change of the device structure corresponding to the pixel point set, and thus the more obvious the feature manifestation; d i,j,l.e represents the height difference between the e-th group of adjacent pixel points on the l-th central connection line of the j-th pixel point set in the i-th region division step, ∑ l＝1 ∑ e＝1 |d i,j,l.e | represents the sum of the height differences of all groups of adjacent pixel points on all central connection lines, that is, the sum of the height differences of adjacent pixel points. The larger this value, the greater the overall height difference of the pixel point set, and thus the more obvious the feature manifestation of the device structure; exp() represents the exponential function with the natural constant as the base.

[0122] In the second step, the pixel point set with the largest feature manifestation value among the pixel point sets with the same feature points is used as the feature pixel point set.

[0123] During all the region division processes, there are some pixel point sets with the same center point. For example, for a concave structure with a depth of 50 mm, when filling water from 1 mm to 49 mm, it corresponds to the same valley. Just retain the pixel point set corresponding to the region division process with the largest feature manifestation value for the pixel point set of the same valley. Subsequently, the pixel point sets with the largest feature manifestation values corresponding to each center point can be obtained as the feature pixel point sets, and the center point can be recorded as the feature point.

[0124] In the third step, determine the matching region corresponding to each feature pixel point set in each device image.

[0125] In the fourth step, calculate the matching degree between the feature points in each feature pixel point set and each matching region.

[0126] In a specific implementation, calculating the matching degree between the feature points in each feature pixel point set and the matching region specifically includes:

[0127] First, respectively determine the shape feature and relative position feature of the feature point in each pixel point set, and determine the shape feature and relative position feature of the matching point corresponding to the feature point in the matching region.

[0128] It can be understood that the shape feature and relative position feature can extract the analysis of the industrial device from the production space and become the self-features of the local part of the industrial device, which can avoid analysis errors caused by external factors such as the shooting angle.

[0129] In this embodiment, the shape feature of the feature point can be represented by the height chain code of the edge pixel points of the corresponding pixel point set. Specifically, taking the position with the minimum height of the edge pixel points in the pixel point set as the starting point, the height values of each edge pixel point are recorded in the clockwise direction to obtain the height chain code of the edge pixel points of the pixel point set.

[0130] In practical applications, connect the feature point to any pixel point in the pixel point set to obtain a line segment of the feature point within the corresponding pixel point set, so that multiple line segments can be obtained. Obtain the length of each line segment. The length of each line segment can be represented by the number of pixel points on each line segment. Add the lengths of any two line segments to determine the line segment combination with the maximum sum of lengths and the line segment combination with the minimum sum of lengths. Taking the shortest line segment in the line segment combination with the minimum sum of lengths as the starting position, record the lengths of each line segment in the clockwise direction to obtain the line segment length chain code (h1, h2, h3, h4), and use the line segment length chain code as the relative position feature of the feature point.

[0131] Similarly, the shape feature of the matching point corresponding to the feature point in the matching area can be represented by the height chain code of the edge pixel points in the matching area, and the relative position feature of the matching point can be represented by the line segment length chain code in the matching area.

[0132] Then, calculate the edit distance between the shape feature of the feature point in each pixel point set and the shape feature of the corresponding matching point to obtain the first distance value.

[0133] Subsequently, calculate the edit distance between the relative position feature of the feature point in each pixel point set and the relative position feature of the corresponding matching point to obtain the second distance value.

[0134] It can be understood that the edit distance can be used to measure the degree of difference between two strings or feature sequences. In this embodiment, by calculating the edit distance of the shape features between the feature point and the matching point, the degree of difference in the shape features between the feature point and the matching point can be obtained. By calculating the edit distance of the relative position features between the feature point and the matching point, the degree of difference in the relative position features between the feature point and the matching point can be obtained.

[0135] Finally, based on the first distance value and the second distance value, calculate the matching degree between the feature point in each feature pixel point set and the matching area.

[0136] In this embodiment, the matching degree between the k-th feature point and the matching area can be expressed as follows:

[0137] P k =norm[DM k ×DH k

[0138] Among them, P k ​Indicates the matching degree between the k-th feature point and the matching region; DM k Indicates the edit distance between the shape feature of the k-th feature point and the shape feature of the corresponding matching point, that is, the first distance value; DH k Indicates the edit distance between the relative position feature of the k-th feature point and the relative position feature of the corresponding matching point, that is, the second distance value; norm[] represents the normalization function. In the embodiments of the present invention, the maximum-minimum normalization function can be specifically used, and there is no limitation on this.

[0139] Step 5: Calculate the pointing value of each feature point to each industrial device according to the matching degree between the feature points in each feature pixel point set and each matching region.

[0140] In one embodiment, calculating the pointing value of each feature point to each industrial device according to the matching degree between the feature points in each feature pixel point set and each matching region specifically includes:

[0141] First, sum the matching degrees between the feature points in each feature pixel point set and all the matching regions corresponding to each industrial device to obtain the matching sum value of each feature point to each industrial device.

[0142] Then, determine the maximum value among the matching degrees between each feature point and each matching region corresponding to each industrial device to obtain the first matching maximum value.

[0143] Subsequently, for any one industrial device, determine the maximum value among the matching degrees between each feature point and each matching region corresponding to other industrial devices to obtain the second matching maximum value.

[0144] Next, calculate the absolute value of the difference between the first matching maximum value and the second matching maximum value to obtain the maximum matching difference value.

[0145] Finally, calculate the pointing value of each feature point to each industrial device based on the maximum matching difference value and the matching sum value.

[0146] In this embodiment, the pointing value of the k-th feature point to the a-th industrial device can be specifically expressed as follows:

[0147]

[0148] Among them, F k-a Indicates the pointing value of the k-th feature point to the a-th industrial device, C represents the number of device images corresponding to the a-th industrial device, P a,k-c Indicates the matching degree between the k-th feature point and the matching region in the c-th device image of the a-th industrial device, max(P a ) represents the first matching maximum value, max(P ～a) represents the second matching maximum value, and norm[] represents the normalization function.

[0149] In one embodiment, based on the preferred feature data corresponding to each industrial device, the pre-constructed machine learning model is trained and tested to obtain an industrial device recognition model, which specifically includes:

[0150] The first step is to establish a sample data set based on the preferred feature data corresponding to each industrial device and the industrial device type corresponding to the preferred feature data.

[0151] It can be understood that by using the preferred feature data as data samples and the industrial device type corresponding to the preferred feature data as sample labels, a sample data set can be obtained.

[0152] The second step is to divide the sample data set into a training sample set and a test sample set according to a preset ratio.

[0153] In practical applications, the sample data set can be divided according to a ratio of 7:3. 70% of the data in the sample data set is used as the training sample set, and 30% of the data is used as the test sample set.

[0154] The third step is to perform sample sampling on the training sample set by means of random sampling with replacement to generate multiple training subsets.

[0155] In this embodiment, specifically, 100 training subsets can be generated by Bootstrap sampling. Each training subset constitutes a decision tree. The decision tree is constructed by randomly selecting feature points in the training sample set. The input is the feature data including the shape feature and relative position feature of the feature points, and the output is the industrial device category corresponding to the image.

[0156] Bootstrap sampling, also known as self-sampling, is a resampling technique widely used in the fields of statistics and machine learning. It performs random sampling with replacement from the training sample set. Each time a sample is drawn, it is recorded and then put back into the training sample set so that the sample still has a chance to be selected in the next draw. This sampling process is repeated multiple times (usually the number of sampling times is the same as the number of samples in the training sample set) to obtain multiple training subsets.

[0157] The fourth step is to train the pre-constructed machine learning model through multiple training subsets and test the trained machine learning model through the test sample set to obtain an industrial device recognition model.

[0158] It can be understood that as the number of recognition types increases, the number of preferred feature data also increases. In a supervised machine learning model, a decision tree is a model that can be used for rapid classification. Among the measured feature data extracted from the image to be measured, part of it is preferred feature data and part is interfering feature data. There is a certain error in the recognition of a single decision tree. Therefore, in this embodiment, a scheme of combining multiple decision trees is adopted, that is, the machine learning model uses a random forest model, and the category of industrial equipment is determined by voting and scoring, which improves the recognition accuracy and ensures the generalization ability of the model.

[0159] In the actual production process, first, a depth camera is used to collect the image to be measured, and the measured feature data of the image to be measured is obtained by matching with the feature point library through the SIFT algorithm. The measured feature data is input into the industrial equipment recognition model, and the category of industrial equipment in the image to be measured can be obtained.

[0160] After obtaining the category of industrial equipment, first, determine the key features for positioning of this category of industrial equipment. For example, for large numerically controlled machine tools, its fixed base corners, specific guide rail markings, etc. are key features for positioning; while for small detection instruments, obvious marks on its shell or unique interface shapes, etc. can be used as key features for positioning.

[0161] Then, according to the category of industrial equipment and the key features of the industrial equipment of the corresponding category, determine the corresponding positioning algorithm. For example, for industrial equipment categories with regular shapes and obvious features, a positioning algorithm based on geometric feature matching can be used, and the position is determined by calculating the matching degree between the key features and the preset template; for equipment with complex textures or appearances, an object detection algorithm in deep learning can be used to first detect the position of the industrial equipment in the image to be measured, and then convert it to the world coordinate system in combination with the camera calibration information. If it is a category with known installation methods and layout rules of the equipment, a positioning algorithm based on rule reasoning can also be used to determine its position through the relative position relationship between industrial equipment and the known installation drawings.

[0162] Subsequently, use the selected positioning algorithm to determine the preliminary position of the industrial equipment in the image to be measured. For example, use an edge detection algorithm to extract the contour edge of the industrial equipment, and then use a contour matching algorithm to determine the preliminary position coordinates of the industrial equipment in the image to be measured.

[0163] Finally, after obtaining the preliminary position of the industrial equipment in the image to be measured, the final position coordinates of the industrial equipment in the world coordinate system are obtained through the calibration of the camera coordinates and the world coordinates, completing the automated visual positioning of the industrial equipment. Subsequently, the industrial robot can perform production-related operations according to the final position coordinates of the industrial equipment.

[0164] In summary, the industrial equipment automated vision positioning method provided by the embodiments of the present invention analyzes the structural features of industrial equipment to be recognized, establishes relatively effective optimal feature data, and trains and tests a pre-established machine learning model with the optimal feature data. Subsequently, the industrial equipment recognition model is used to automatically recognize the industrial equipment in the image to be measured, thereby completing the automated vision positioning, improving the accuracy of industrial equipment automated vision recognition and the positioning reliability, and further enhancing the operation efficiency of industrial robots.

[0165] Based on the same general inventive concept, the present invention also protects an industrial equipment automated vision positioning system. The industrial equipment automated vision positioning system provided by the present invention will be described below. The industrial equipment automated vision positioning system described below can be mutually corresponding and referenced with the industrial equipment automated vision positioning method described above.

[0166] Please refer to Figure 6 , which shows the system structure diagram of an industrial equipment automated vision positioning system provided by an embodiment of the present invention. As Figure 6 shown, the above-mentioned industrial equipment automated vision positioning system specifically includes:

[0167] An acquisition module 410, configured to acquire multiple device images corresponding to different industrial equipment in the production process.

[0168] An extraction module 420, configured to extract feature points in each device image and respectively determine the pointing value of each feature point to each industrial equipment.

[0169] An allocation module 430, configured to allocate each feature point to the industrial equipment with the largest pointing value to obtain optimal feature data corresponding to each industrial equipment.

[0170] A modeling module 440, configured to train and test a pre-constructed machine learning model based on the optimal feature data corresponding to each industrial equipment to obtain an industrial equipment recognition model.

[0171] An identification module 450, configured to input the to-be-measured feature data extracted from the image to be measured into the industrial equipment recognition model to obtain the industrial equipment category output by the industrial equipment recognition model.

[0172] A positioning module 460, configured to position the industrial equipment in the image to be measured based on the industrial equipment category.

[0173] Regarding the system in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated in detail here.

[0174] The industrial equipment automation vision positioning system provided in this embodiment, because the industrial equipment recognition model on which the recognition link is based is trained and tested based on the optimal feature data of each industrial equipment, and the optimal feature data is obtained according to the pointing value of each feature point to each industrial equipment, can make the optimal feature data more accurately represent the characteristics of the corresponding category of industrial equipment. Furthermore, based on the industrial equipment recognition model, accurate recognition of the industrial equipment category can be achieved, the recognition accuracy is improved, and thus the positioning accuracy and reliability are improved.

[0175] It should be noted that the above sequence of the embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0176] Each embodiment in this specification is described in a progressive manner, and the same or similar parts among the embodiments can be referred to each other. The key point of each embodiment is to illustrate the differences from other embodiments.

Claims

1. An industrial equipment automatic vision positioning method, characterized in that, The method includes: Obtaining multiple device images corresponding to different industrial devices respectively during the production process; Extracting feature points in each device image and respectively determining the pointing value of each feature point to each industrial device; Assigning each feature point to the industrial device with the largest pointing value to obtain the preferred feature data corresponding to each industrial device; Based on the preferred feature data corresponding to each industrial device, training and testing a pre-constructed machine learning model to obtain an industrial device recognition model; Inputting the to-be-detected feature data extracted from the to-be-detected image into the industrial device recognition model to obtain the industrial device category output by the industrial device recognition model; Based on the industrial device category, positioning the industrial device in the to-be-detected image.

2. The industrial equipment automatic vision positioning method according to claim 1, characterized in that, Extracting feature points in each device image includes: Determining the gray information and depth information of each pixel point in each device image; According to the gray information of each pixel point in each device image, determining the object area where the industrial device is located in each device image; Based on the depth information of each pixel point in the object area of each device image, establishing a terrain three-dimensional model corresponding to the object area in each device image; Performing regional division on the terrain three-dimensional model corresponding to the object area in each device image to obtain multiple pixel point sets; Taking the pixel point with the largest or smallest depth information in each pixel point set as the feature point to obtain the feature points in each device image.

3. An industrial equipment automated vision positioning method according to claim 2, characterized in that, Based on the depth information of each pixel point in the object area of each device image, establishing a terrain three-dimensional model corresponding to the object area in each device image includes: Determining the horizon according to the pixel point with the largest depth information in the object area of each device image; Taking the depth distance between each pixel point in the object area of each device image and the horizon as the height of each pixel point; Based on the heights of all pixel points in the object area of each device image, obtaining the terrain three-dimensional model corresponding to the object area in each device image.

4. An industrial equipment automatic vision positioning method according to claim 2, characterized in that, Respectively determining the pointing value of each feature point to each industrial device includes: Calculating the feature performance value corresponding to each pixel point set; Taking the pixel point set with the largest feature performance value in the pixel point sets with the same feature points as the feature pixel point set; Determining the matching area corresponding to each feature pixel point set in each device image; Calculating the matching degree between the feature points in each feature pixel point set and each matching area; Based on the matching degree between the feature points in each feature pixel point set and each matching area, calculating to obtain the pointing value of each feature point to each industrial device.

5. The industrial equipment automatic vision positioning method according to claim 4, characterized in that, Calculating the feature performance value corresponding to each pixel point set includes: Respectively determining the number of valley point sets and the number of watershed line point sets in the pixel point sets obtained in each regional division link, and determining the number of pixel points, the maximum and minimum values of pixel point heights, and the height difference between any group of adjacent pixel points on each central connection line in each pixel point set; Based on the number of valley point sets and the number of watershed line point sets, calculating the number difference degree between valleys and watershed lines in each regional division link; Based on the number of pixel points and the maximum and minimum values of pixel point heights in each pixel point set, calculating the average height distribution value of each pixel point set; Calculate the sum of the height differences of adjacent pixel points based on the height differences of all groups of adjacent pixel points on the straight line connecting the centers in each pixel point set; Calculate the characteristic performance value corresponding to each pixel point set based on the quantity difference degree, the average height distribution value, and the sum of the height differences of adjacent pixel points.

6. An industrial equipment automated vision positioning method according to claim 4, characterized in that, Calculate the matching degree between the feature points in each feature pixel point set and the matching area, including: Determine the shape feature and relative position feature of the feature points in each pixel point set respectively, and determine the shape feature and relative position feature of the matching points corresponding to the feature points in the matching area; Calculate the edit distance between the shape feature of the feature points in each pixel point set and the shape feature of the corresponding matching points to obtain a first distance value; Calculate the edit distance between the relative position feature of the feature points in each pixel point set and the relative position feature of the corresponding matching points to obtain a second distance value; Calculate the matching degree between the feature points in each feature pixel point set and the matching area based on the first distance value and the second distance value.

7. A method for automated vision positioning of industrial equipment according to claim 4, characterized in that, Calculate the pointing value of each feature point to each industrial device based on the matching degree between the feature points in each feature pixel point set and each matching area, including: Sum up the matching degrees between the feature points in each feature pixel point set and all the matching areas corresponding to each industrial device to obtain the matching sum value of each feature point to each industrial device; Determine the maximum value among the matching degrees of each feature point with the respective matching areas corresponding to each industrial device to obtain a first maximum matching value; For any one industrial device, determine the maximum value among the matching degrees of each feature point with the respective matching areas corresponding to other industrial devices to obtain a second maximum matching value; Calculate the absolute value of the difference between the first maximum matching value and the second maximum matching value to obtain the maximum matching difference value; Calculate the pointing value of each feature point to each industrial device based on the maximum matching difference value and the matching sum value.

8. An industrial equipment automated vision positioning method according to claim 1, characterized in that, Train and test a pre-constructed machine learning model based on the preferred feature data corresponding to each industrial device to obtain an industrial device recognition model, including: Establish a sample data set based on the preferred feature data corresponding to each industrial device and the industrial device type corresponding to the preferred feature data; Divide the sample data set into a training sample set and a test sample set according to a preset ratio; Perform sample sampling on the training sample set by random sampling with replacement to generate multiple training subsets; Train the pre-constructed machine learning model through the multiple training subsets, and test the trained machine learning model through the test sample set to obtain an industrial device recognition model.

9. The industrial equipment automatic vision positioning method according to claim 1, characterized in that, Obtain multiple device images corresponding to different industrial devices in the production process, including: Receive multiple initial images corresponding to different industrial devices in the production process uploaded by a depth camera; wherein, the multiple initial images are used to represent the morphology of the industrial devices at multiple angles and in multiple directions; Perform grayscale processing on each initial image to obtain multiple device images corresponding to different industrial devices in the production process.

10. An industrial equipment automated vision positioning system, characterized in that, The system includes: An acquisition module for acquiring multiple device images corresponding to different industrial devices in the production process; An extraction module for extracting feature points in each device image and respectively determining the pointing values of each feature point to each industrial device; An allocation module for allocating each feature point to the industrial device with the largest pointing value to obtain the preferred feature data corresponding to each industrial device; A modeling module for training and testing a pre-constructed machine learning model based on the preferred feature data corresponding to each industrial device to obtain an industrial device recognition model; An identification module for inputting the measured feature data extracted from the image to be measured into the industrial device recognition model to obtain the industrial device category output by the industrial device recognition model; A positioning module for positioning the industrial device in the image to be measured based on the industrial device category.

Images