Industrial equipment automation visual positioning method and system

Abstract

The application relates to the technical field of image processing, in particular to an industrial equipment automatic visual positioning method and system, which comprises the following steps: acquiring a plurality of equipment images corresponding to different kinds of industrial equipment in a production process; extracting feature points in each equipment image, and determining the pointing value of each feature point to each kind of industrial equipment; distributing each feature point to the industrial equipment with the maximum pointing value to obtain preferred feature data corresponding to each kind of industrial equipment; training and testing a machine learning model based on the preferred feature data corresponding to each kind of industrial equipment to obtain an industrial equipment recognition model; inputting the extracted test feature data in a test image into the industrial equipment recognition model to obtain the industrial equipment category output by the industrial equipment recognition model; and positioning the industrial equipment in the test image based on the industrial equipment category. The scheme provided by the application improves the recognition accuracy of the industrial equipment category and improves the positioning accuracy and reliability.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an automated visual positioning method and system for industrial equipment. Background Technology

[0002] Currently, industrial robots are widely used in various production activities in the manufacturing industry, including assembly, welding, spraying, and polishing. During production, industrial robots need to precisely grasp industrial equipment to ensure accurate target operations. The first step in grasping industrial equipment is positioning it. This positioning is typically achieved using machine vision technology, specifically by acquiring images and performing a series of processing steps to identify the industrial equipment and then positioning it based on the identification results.

[0003] In related technologies, during the identification process, the contour feature parameters of industrial equipment change due to their movement. Furthermore, similar industrial equipment may have the same contour feature parameters. Therefore, when there are many types of industrial equipment, the representational ability of the contour feature parameters is weak, lacking the unique features of each type of industrial equipment, which can easily lead to identification errors and affect the subsequent positioning accuracy.

[0004] It is evident that traditional visual positioning methods for industrial equipment suffer from low recognition accuracy and insufficient positioning accuracy and reliability. Summary of the Invention

[0005] To address the technical problems of low recognition accuracy and insufficient positioning accuracy and reliability in traditional industrial equipment visual positioning methods, the present invention aims to provide an automated visual positioning method and system for industrial equipment. The specific technical solution adopted is as follows:

[0006] An automated visual positioning method for industrial equipment, the method comprising:

[0007] Acquire multiple images of different types of industrial equipment during the production process;

[0008] Extract feature points from each equipment image and determine the pointing value of each feature point for each type of industrial equipment;

[0009] Each feature point is assigned to the industrial equipment with the largest pointer value to obtain the preferred feature data for each type of industrial equipment.

[0010] Based on the preferred feature data corresponding to each type of industrial equipment, a pre-built machine learning model is trained and tested to obtain an industrial equipment identification model.

[0011] The feature data extracted from the image to be tested is input into the industrial equipment recognition model to obtain the industrial equipment category output by the industrial equipment recognition model;

[0012] Based on the industrial equipment category, the industrial equipment in the image to be tested is located.

[0013] According to the present invention, an automated visual positioning method for industrial equipment extracts feature points from each equipment image, including:

[0014] Determine the grayscale and depth information of each pixel in each device image;

[0015] Based on the grayscale information of each pixel in each equipment image, determine the object region where the industrial equipment is located in each equipment image;

[0016] Based on the depth information of each pixel within the object region in each device image, a 3D terrain model corresponding to the object region in each device image is established.

[0017] The terrain 3D model corresponding to the object area in each device image is divided into regions to obtain multiple pixel sets;

[0018] The feature points in each device image are obtained by taking the pixel with the largest or smallest depth information in the pixel set as the feature points.

[0019] According to the present invention, an automated visual positioning method for industrial equipment is provided, which establishes a three-dimensional terrain model corresponding to the object region in each equipment image based on the depth information of each pixel within the object region in each equipment image, including:

[0020] The horizon is determined based on the pixel with the largest depth information within the object region in each device image;

[0021] The depth distance between each pixel within the object region in each device image and the horizon is used as the height of each pixel.

[0022] Based on the height of all pixels within the object region in each device image, a 3D terrain model corresponding to the object region in each device image is obtained.

[0023] According to the present invention, an automated visual positioning method for industrial equipment is provided, which determines the pointing value of each feature point for each type of industrial equipment, including:

[0024] Calculate the feature representation value corresponding to each set of pixels;

[0025] The set of pixels with the largest feature expression value in the set of pixels with the same feature points is taken as the feature pixel set;

[0026] Determine the matching region corresponding to each feature pixel set in each device image;

[0027] Calculate the degree of matching between the feature points in each feature pixel set and each matching region;

[0028] Based on the degree of matching between the feature points in each feature pixel set and each matching region, the pointing value of each feature point to each type of industrial equipment is calculated.

[0029] According to the present invention, an automated visual positioning method for industrial equipment calculates the feature representation value corresponding to each pixel set, including:

[0030] Determine the number of valley point sets and watershed line point sets in the pixel set obtained in each region division step, and determine the number of pixels, the maximum and minimum pixel height, and the height difference of any pair of adjacent pixels on each center line for each pixel set.

[0031] Based on the number of valley point sets and the number of watershed line point sets, the degree of difference in the number of valleys and watershed lines in each region division stage is calculated;

[0032] Based on the number of pixels and the maximum and minimum pixel height of each pixel set, the average height distribution value of each pixel set is calculated;

[0033] Based on the height difference of all groups of adjacent pixels on the straight line connecting the centers of each pixel set, the sum of the height differences of adjacent pixels is calculated;

[0034] Based on the degree of difference in quantity, the average height distribution value, and the sum of the height differences between adjacent pixels, the feature performance value corresponding to each set of pixels is calculated.

[0035] According to the present invention, an automated visual positioning method for industrial equipment calculates the degree of matching between feature points in each feature pixel set and the matching region, including:

[0036] The shape features and relative position features of the feature points in each pixel set are determined respectively, and the shape features and relative position features of the matching points corresponding to the feature points in the matching region are determined;

[0037] Calculate the edit distance between the shape features of feature points in each pixel set and the shape features of the corresponding matching points to obtain the first distance value;

[0038] Calculate the edit distance between the relative position features of feature points in each pixel set and the relative position features of the corresponding matching points to obtain the second distance value;

[0039] Based on the first distance value and the second distance value, the matching degree between the feature points in each feature pixel set and the matching region is calculated.

[0040] According to the present invention, an automated visual positioning method for industrial equipment calculates the pointing value of each feature point for each type of industrial equipment based on the matching degree between feature points in each feature pixel set and each matching region, including:

[0041] The matching degree of each feature point in the feature pixel set is summed with all matching regions corresponding to each type of industrial equipment to obtain the matching sum value of each feature point for each type of industrial equipment.

[0042] Determine the maximum value among the matching degrees of each feature point and each matching region corresponding to each type of industrial equipment to obtain the first maximum matching value;

[0043] For any type of industrial equipment, determine the maximum value of the matching degree among the matching regions corresponding to each feature point and other types of industrial equipment to obtain the second maximum matching value;

[0044] Calculate the absolute value of the difference between the first maximum matching value and the second maximum matching value to obtain the maximum matching difference value;

[0045] Based on the maximum difference value of the matching and the sum of the matching values, the pointing value of each feature point to each type of industrial equipment is calculated.

[0046] According to the present invention, an automated visual positioning method for industrial equipment is provided, which trains and tests a pre-built machine learning model based on preferred feature data corresponding to each type of industrial equipment to obtain an industrial equipment recognition model, including:

[0047] A sample dataset is established based on the preferred feature data corresponding to each type of industrial equipment and the type of industrial equipment corresponding to the preferred feature data.

[0048] The sample dataset is divided into a training sample set and a test sample set according to a preset ratio;

[0049] The training sample set is sampled using a random sampling method with replacement to generate multiple training subsets;

[0050] The pre-built machine learning model is trained using the multiple training subsets, and the trained machine learning model is tested using the test sample set to obtain the industrial equipment recognition model.

[0051] According to the present invention, an automated visual positioning method for industrial equipment is provided, which acquires multiple equipment images corresponding to different types of industrial equipment during the production process, including:

[0052] The system receives multiple initial images of different types of industrial equipment uploaded by a depth camera during the production process; wherein, the multiple initial images are used to characterize the shape of the industrial equipment from multiple angles and directions;

[0053] Each initial image is converted to grayscale to obtain multiple equipment images corresponding to different types of industrial equipment in the production process.

[0054] On the other hand, the present invention also provides an automated visual positioning system for industrial equipment, the system comprising:

[0055] The acquisition module is used to acquire multiple equipment images corresponding to different types of industrial equipment during the production process.

[0056] The extraction module is used to extract feature points from each equipment image and determine the pointing value of each feature point for each type of industrial equipment.

[0057] The allocation module is used to assign each feature point to the industrial equipment with the largest pointer value, thereby obtaining the preferred feature data for each type of industrial equipment.

[0058] The modeling module is used to train and test a pre-built machine learning model based on the preferred feature data corresponding to each type of industrial equipment, so as to obtain an industrial equipment identification model.

[0059] The recognition module is used to input the feature data extracted from the image to be tested into the industrial equipment recognition model to obtain the industrial equipment category output by the industrial equipment recognition model.

[0060] The positioning module is used to locate the industrial equipment in the image to be tested based on the industrial equipment category.

[0061] The present invention has the following beneficial effects:

[0062] By acquiring multiple images of different types of industrial equipment during the production process, feature points are extracted from each image, and the pointing value of each feature point to each type of industrial equipment is determined. Each feature point is then assigned to the industrial equipment with the highest pointing value, resulting in optimized feature data for each type of industrial equipment. Based on this optimized feature data, a pre-built machine learning model is trained and tested to obtain an industrial equipment recognition model. The feature data extracted from the image to be tested is input into the industrial equipment recognition model, yielding the industrial equipment category output by the model. Finally, based on the industrial equipment category, the industrial equipment in the image to be tested is located. Since the industrial equipment recognition model used in the recognition process is trained and tested based on the optimized feature data for each type of industrial equipment, and this optimized feature data is obtained based on the pointing value of each feature point to each type of industrial equipment, it can more accurately represent the characteristics of the corresponding category of industrial equipment, resulting in stronger feature representation capabilities. Therefore, the industrial equipment recognition model can achieve accurate identification of industrial equipment categories, improving recognition accuracy and thus enhancing positioning accuracy and reliability. Attached Figure Description

[0063] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart of an automated visual positioning method for industrial equipment provided in one embodiment of the present invention;

[0065] Figure 2 An image of a cube at one angle;

[0066] Figure 3 Image of the cube from another angle;

[0067] Figure 4 A schematic diagram of the structure of the 3D terrain model corresponding to the object region;

[0068] Figure 5 A schematic diagram of the mountain and valley formations that appear during the process of filling with water;

[0069] Figure 6 This is a schematic diagram of the system structure of an automated visual positioning system for industrial equipment provided in one embodiment of the present invention. Detailed Implementation

[0070] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of an automated visual positioning method and system for industrial equipment proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0072] The following is in conjunction with the appendix Figures 1 to 6 This invention provides a specific solution for an automated visual positioning method and system for industrial equipment.

[0073] Please see Figure 1 It illustrates a flowchart of an automated visual positioning method for industrial equipment according to an embodiment of the present invention, as shown below. Figure 1 As shown, the above-mentioned automated visual positioning method for industrial equipment specifically includes:

[0074] Step 110: Obtain multiple images of different types of industrial equipment during the production process.

[0075] In this embodiment, multiple equipment images corresponding to each type of industrial equipment can represent the shape of the industrial equipment at different angles and directions during the production process.

[0076] Step 120: Extract feature points from each equipment image and determine the pointing value of each feature point for each type of industrial equipment.

[0077] In this embodiment, the pointing value of each feature point to each type of industrial equipment can characterize the degree to which the feature point points only to that type of industrial equipment. The higher the pointing value, the stronger the pointing of the feature point to that type of industrial equipment, and the more uniquely the feature point can characterize the features of that type of industrial equipment.

[0078] Step 130: Assign each feature point to the industrial equipment with the largest pointing value to obtain the preferred feature data for each type of industrial equipment.

[0079] It is understood that the preferred feature data is the set of feature points that best represent the characteristics of the industrial equipment after filtering all feature points based on the pointing value. In this embodiment, the preferred feature data corresponding to each type of industrial equipment is obtained by calculating the pointing value. The feature data has a stronger representational ability and can provide more accurate data support for the subsequent generation of industrial equipment identification models, thereby improving the accuracy and reliability of the identification process.

[0080] Step 140: Based on the preferred feature data corresponding to each type of industrial equipment, train and test the pre-built machine learning model to obtain the industrial equipment identification model.

[0081] In this embodiment, the construction of the industrial equipment identification model can provide more convenient identification conditions for the identification process, enabling efficient and accurate identification of industrial equipment categories.

[0082] Step 150: Input the feature data extracted from the image to be tested into the industrial equipment recognition model to obtain the industrial equipment category output by the industrial equipment recognition model.

[0083] Understandably, since the input of the industrial equipment recognition model is feature data and the output is the industrial equipment category, before recognizing the image to be tested, it is necessary to extract the feature data to be tested from the image to be tested, input the feature data to be tested into the industrial equipment recognition model, and then obtain the recognition result.

[0084] Step 160: Locate the industrial equipment in the image to be tested based on the industrial equipment category.

[0085] The solution provided in this embodiment improves the industrial equipment category identification process by using pointing values ​​to achieve feature filtering, obtaining preferred feature data for each type of industrial equipment, and then using the preferred feature data to establish an industrial equipment identification model, thereby achieving accurate identification of industrial equipment categories. Subsequently, based on the industrial equipment category, accurate positioning of industrial equipment in the image under test can be achieved.

[0086] In one embodiment, acquiring multiple equipment images corresponding to different types of industrial equipment during the production process specifically includes:

[0087] First, multiple initial images of different types of industrial equipment during the production process are received from the depth camera; these initial images are used to characterize the shape of the industrial equipment from multiple angles and directions.

[0088] In practical applications, a depth camera can be installed at the end of the robotic arm of an industrial robot. The depth camera is positioned facing the industrial equipment. During the movement of the industrial robot, the industrial camera acquires images at a fixed frame rate, which can be set to 30fps, to obtain multiple initial images corresponding to various industrial equipment. These multiple initial images can cover the morphological information of the industrial equipment at different angles and directions.

[0089] Then, each initial image is converted to grayscale to obtain multiple equipment images corresponding to different types of industrial equipment in the production process.

[0090] It is understandable that by converting the initial image to grayscale, grayscale information of any pixel can be obtained, and by taking a picture with a depth camera, depth information of any pixel can be obtained, which is the actual distance between the depth camera and the corresponding pixel on the industrial equipment.

[0091] In this embodiment, the actual coordinates of the industrial equipment relative to the depth camera can be directly obtained using a depth camera, which can reduce the coordinate system calibration process and reduce calculation errors.

[0092] In one embodiment, extracting feature points from each device image includes:

[0093] The first step is to determine the grayscale and depth information of each pixel in each device image.

[0094] It is understandable that regardless of the shooting angle of the depth camera, the industrial equipment corresponding to the captured image is unique. Therefore, it is necessary to select features from those that describe the essential characteristics of the industrial equipment. The distance between the depth camera and each point on the industrial equipment, i.e., the depth information, can characterize the surface texture of the industrial equipment and is a relatively clear essential feature. Therefore, this embodiment combines grayscale information and depth information to achieve the feature extraction function.

[0095] Taking a cube as an example, each image can only capture a portion of the cube. Figure 2 and Figure 3 These are two images of a cube viewed from different angles. A1, A2, A3, and A4 represent the four front points of the cube. ′ 1. A ′ 2. A ′ 3. A ′ 4 represents the four rear points of the cube. The faces captured from the two angles are faces A1, A2, A4, A3, and face A. ′ 1A ′ 2A2A1, surface A2A ′ 2A ′ 4A4, point A2 connects three faces. The depth information of point A2 as it spreads to the edge can represent part of the features of the cube. Therefore, point A2 can be regarded as a feature point, and the spread part can be regarded as the feature basis of this feature point.

[0096] The second step is to determine the object region where the industrial equipment is located in each equipment image based on the grayscale information of each pixel.

[0097] In this embodiment, an edge detection algorithm can be used to detect the edges of industrial equipment in the equipment image, obtaining the edge contour of the industrial equipment. The region within the edge contour of the industrial equipment is the object region. In practical applications, edge detection algorithms based on the Sobel operator or the Canny operator can be used to implement edge detection.

[0098] The third step is to establish a 3D terrain model corresponding to the object area in each device image based on the depth information of each pixel within the object area in each device image.

[0099] Among the feature points on the surface of a mechanical structure, there are points that can express various structural features such as contours, edges, and vertices. However, when the viewing angle of industrial equipment changes, the structural features corresponding to some points will change in the equipment image due to changes in geometry. While the contour of the industrial equipment may change, the relative positions of the various structures within the equipment will not. For example, Figure 2 and Figure 3 In the diagram, A2 represents a vertex of the cube. This vertex will not become a concave point at any angle, so feature points can be selected based on the height of each position on the industrial equipment.

[0100] In order to more accurately obtain the structural features that can effectively describe the industrial equipment presented in the equipment image, this embodiment uses binocular vision to further establish a corresponding three-dimensional terrain model of the industrial equipment based on the horizontal and vertical coordinates and depth information of each pixel in the equipment image. According to the different degrees of representation of the industrial equipment by different pixels in the terrain three-dimensional model, the terrain three-dimensional model is regarded as a topographic map. Valleys and peaks can represent a region better than plains. Therefore, regions similar to valleys and peaks can be selected from the terrain three-dimensional model to obtain feature information.

[0101] In one specific implementation, a 3D terrain model corresponding to the object region in each device image is established based on the depth information of each pixel within the object region in each device image. This specifically includes:

[0102] First, the horizon is determined based on the pixel with the largest depth information within the object region in each device image.

[0103] Then, the depth distance between each pixel within the object region in each device image and the horizon is used as the height of each pixel.

[0104] Finally, based on the height of all pixels within the object region in each device image, a 3D terrain model corresponding to the object region in each device image is obtained.

[0105] Figure 4 An exemplary 3D terrain model corresponding to the object region is shown. Figure 4In the image, 210 represents the object region in the equipment image, and 220 represents one of the height variation curves. The 3D terrain model can reflect the height relationship of the industrial equipment surface. Since the concave and convex shape does not change under different shooting angles, the concave and convex area structure can be used as feature information, which can reduce the recognition error caused by different shooting angles.

[0106] In this embodiment, the uneven structure is analogous to valleys and peaks in a topographic map. Feature points on the surface of the industrial equipment are extracted using the variation characteristics of the industrial equipment in the depth image. Furthermore, a height variation curve is formed between different pixels in the depth image. To analyze the variation characteristics of this height variation curve, this embodiment uses a watershed algorithm, continuously filling the height variation curve with water to analyze the uneven features of the industrial equipment. Figure 5 As shown, during the water filling process, peaks and valleys will appear. Therefore, the idea of ​​the watershed algorithm can be used to analyze the changes in peaks and valleys in the 3D terrain model and determine feature points. Figure 5 In the middle, 310 represents the horizontal plane, 320 represents the watershed line, and 330 represents the valley of the local minimum.

[0107] The fourth step is to divide the terrain 3D model corresponding to the object area in each device image into regions, resulting in multiple pixel sets.

[0108] In this embodiment, water is filled in units of 1 mm, starting from the horizon and continuing until the water reaches its maximum height, resulting in several segmentation processes. For any segmentation process, the 3D terrain model is divided into several isolated regions, and the process of filling with water is the process of these isolated regions continuously changing and eventually merging.

[0109] For any segmentation process, all isolated regions are divided into several sets of pixels, such as valley pixel sets and watershed pixel sets. Each pixel set represents a continuous region with similar depth on the industrial equipment. For any pixel set, the data expands outward from the point of maximum depth. The more significant the depth change, the more significant the structural change at the corresponding location on the industrial equipment, and the more pronounced the feature representation of this pixel set. Therefore, this pixel set better characterizes the structural features of the industrial equipment.

[0110] The fifth step is to take the pixel with the largest or smallest depth information in each pixel set as the feature point to obtain the feature points in each device image.

[0111] In one embodiment, determining the pointing value of each feature point to each type of industrial equipment includes:

[0112] The first step is to calculate the feature representation value corresponding to each set of pixels.

[0113] In a specific implementation, the feature representation value corresponding to each set of pixels is calculated, which specifically includes:

[0114] First, determine the number of valley point sets and watershed line point sets in the pixel set obtained from each region division step, and determine the number of pixels, the maximum and minimum pixel height, and the height difference of any pair of adjacent pixels on each center line for each pixel set.

[0115] Then, based on the number of valley point sets and the number of watershed point sets, the degree of difference in the number of valleys and watershed lines in each region division stage is calculated.

[0116] Subsequently, based on the number of pixels and the maximum and minimum pixel heights of each pixel set, the average height distribution value of each pixel set is calculated.

[0117] Next, based on the height difference of all groups of adjacent pixels on the straight line connecting the centers of each pixel set, the sum of the height differences of adjacent pixels is calculated.

[0118] Finally, based on the degree of difference in quantity, the average height distribution value, and the sum of the height differences between adjacent pixels, the feature performance value corresponding to each set of pixels is calculated.

[0119] In this embodiment, the feature representation value of the j-th pixel set in the i-th region segmentation step can be calculated as follows:

[0120]

[0121] F i,j This represents the feature representation value of the j-th pixel set in the i-th region segmentation step; n i,P n i,L These represent the number of valley point sets and the number of watershed line point sets in the i-th region partitioning stage, respectively. This represents the degree of difference in the number of valleys and watershed lines in the i-th region segmentation stage. The degree of difference in number can characterize the current segmentation level. Too large or too small a value indicates that the current segmentation process is insufficient, treating most regions as valleys or peaks, while some device structures with relatively gentle depth changes exist, thus the feature representation of the pixel set in these areas is also small; n i,j d1 represents the number of pixels in the j-th pixel set in the i-th region partitioning step; i,j d2 i,j Let these represent the maximum and minimum heights of the j-th pixel set in the i-th region partitioning step, respectively, i.e., the extreme values ​​of pixel height; d represents the average height distribution value of the j-th pixel set in the i-th region segmentation step. The fewer pixels corresponding to any height, the more obvious the height change of the device structure corresponding to that pixel set, and thus the more obvious the feature representation; i,j,l.e ∑ represents the height difference between the e-th group of adjacent pixels on the straight line connecting the l-th center of the j-th pixel set in the i-th region partitioning step. l＝1 ∑ e＝1 |d i,j,l.e | represents the sum of the height differences of all adjacent pixels on all centrally connected straight lines, i.e., the sum of the height differences of adjacent pixels. The larger this value is, the greater the overall height difference of the pixel set, and thus the more obvious the feature representation of the device structure; exp() represents an exponential function with the natural constant as the base.

[0122] The second step is to select the set of pixels with the largest feature expression value from the set of pixels with the same feature points as the feature pixel set.

[0123] During all region segmentation processes, there exist some sets of pixels with the same center point. For example, in a concave structure with a depth of 50mm, water filling from 1mm to 49mm corresponds to the same valley. For the pixel set of the same valley, we only need to retain the pixel set corresponding to the region segmentation process with the highest feature performance value. Subsequently, we can obtain the pixel set with the highest feature performance value corresponding to each center point as the feature pixel set, and the center point can be recorded as the feature point.

[0124] The third step is to determine the matching region corresponding to each feature pixel set in each device image.

[0125] The fourth step is to calculate the degree of matching between the feature points in each feature pixel set and each matching region.

[0126] In a specific implementation, the degree of matching between feature points in each feature pixel set and the matching region is calculated, specifically including:

[0127] First, determine the shape features and relative position features of the feature points in each pixel set, and then determine the shape features and relative position features of the matching points corresponding to the feature points in the matching region.

[0128] Understandably, shape and relative position features allow subsequent analysis of industrial equipment to be extracted from the production space and transformed into the local features of the industrial equipment itself, thus avoiding analysis errors caused by external factors such as shooting angle.

[0129] In this embodiment, the shape features of the feature points can be represented by the edge pixel height chain code of the corresponding pixel set. Specifically, starting from the position with the minimum height of the edge pixel in the pixel set, the height values ​​of each edge pixel are recorded in a clockwise direction to obtain the edge pixel height chain code of the pixel set.

[0130] In practical applications, a feature point is connected to any pixel in the pixel set to obtain a line segment of that feature point within the corresponding pixel set. This can result in multiple line segments. The length of each line segment is obtained, which can be represented by the number of pixels on that line segment. The lengths of any two line segments are added together to determine the combination of line segments with the largest sum and the combination of line segments with the smallest sum. Starting from the shortest line segment in the combination of line segments with the smallest sum, the lengths of each line segment are recorded clockwise to obtain the line segment length chain code (h1, h2, h3, h4). The line segment length chain code is used as the relative position feature of the feature point.

[0131] Similarly, the shape features of the matching points corresponding to the feature points in the matching region can be represented by the height chain code of the edge pixels in the matching region, and the relative position features of the matching points can be represented by the length chain code of the line segments in the matching region.

[0132] Then, the edit distance between the shape features of the feature points in each pixel set and the shape features of the corresponding matching points is calculated to obtain the first distance value.

[0133] Subsequently, the edit distance between the relative position features of the feature points in each pixel set and the relative position features of the corresponding matching points is calculated to obtain the second distance value.

[0134] Understandably, edit distance can be used to measure the degree of difference between two strings or feature sequences. In this embodiment, by calculating the edit distance of the shape features between the feature point and the matching point, the degree of difference in shape features between the feature point and the matching point can be obtained. By calculating the edit distance of the relative position features between the feature point and the matching point, the degree of difference in relative position features between the feature point and the matching point can be obtained.

[0135] Finally, based on the first distance value and the second distance value, the degree of matching between the feature points in each feature pixel set and the matching region is calculated.

[0136] In this embodiment, the degree of matching between the k-th feature point and the matching region can be expressed as follows:

[0137] P k =norm[DM k ×DH k ]

[0138] Among them, P kDM represents the degree of matching between the k-th feature point and the matching region. k This represents the edit distance between the shape feature of the k-th feature point and the shape feature of the corresponding matching point, i.e., the first distance value; DH k The edit distance between the relative position features of the k-th feature point and the relative position features of the corresponding matching point is represented by the second distance value; norm[] represents the normalization function. In this embodiment of the invention, a maximum / minimum value normalization function can be used, and there are no restrictions on this.

[0139] The fifth step is to calculate the pointing value of each feature point to each type of industrial equipment based on the matching degree between the feature points in each feature pixel set and each matching region.

[0140] In one embodiment, based on the matching degree between feature points in each feature pixel set and each matching region, the pointing value of each feature point for each type of industrial equipment is calculated, specifically including:

[0141] First, the matching degree between the feature points in each feature pixel set and all matching regions corresponding to each type of industrial equipment is summed to obtain the matching sum value of each feature point for each type of industrial equipment.

[0142] Then, the maximum value of the matching degree between each feature point and each matching region corresponding to each type of industrial equipment is determined to obtain the first matching maximum value.

[0143] Subsequently, for any type of industrial equipment, the maximum value of the matching degree of each feature point among the matching regions corresponding to other types of industrial equipment is determined, and the second matching maximum value is obtained.

[0144] Next, the absolute value of the difference between the first maximum matching value and the second maximum matching value is calculated to obtain the maximum difference value of the matching.

[0145] Finally, based on the maximum difference value and the sum of the matching values, the pointing value of each feature point to each type of industrial equipment is calculated.

[0146] In this embodiment, the pointing value of the kth feature point to the ath type of industrial equipment can be specifically represented as follows:

[0147]

[0148] Among them, F k-a P represents the pointing value of the k-th feature point to the a-th type of industrial equipment, C represents the number of equipment images corresponding to the a-th type of industrial equipment, and P represents the pointing value of the k-th feature point to the a-th type of industrial equipment. a,k-c This represents the degree of matching between the k-th feature point and the c-th equipment image of the a-th type of industrial equipment, where max(P) represents the matching region. a ) represents the first matching maximum value, max(P ～a) represents the second maximum matching value, and norm[] represents the normalization function.

[0149] In one embodiment, based on preferred feature data corresponding to each type of industrial equipment, a pre-built machine learning model is trained and tested to obtain an industrial equipment identification model, specifically including:

[0150] The first step is to establish a sample dataset based on the preferred feature data corresponding to each type of industrial equipment and the industrial equipment type corresponding to the preferred feature data.

[0151] It is understandable that by taking the preferred feature data as data samples and the industrial equipment type corresponding to the preferred feature data as sample labels, a sample dataset can be obtained.

[0152] The second step is to divide the sample dataset into a training sample set and a test sample set according to a preset ratio.

[0153] In practical applications, the sample dataset can be divided in a 7:3 ratio, with 70% of the data used as the training sample set and 30% used as the test sample set.

[0154] The third step is to sample the training sample set using random sampling with replacement to generate multiple training subsets.

[0155] In this embodiment, 100 training subsets can be generated through Bootstrap sampling. Each training subset constitutes a decision tree. The decision tree is constructed by randomly selecting feature points from the training sample set. The input is feature data containing the shape features and relative position features of the feature points, and the output is the industrial equipment category corresponding to the image.

[0156] Bootstrap sampling, also known as self-sampling, is a resampling technique widely used in statistics and machine learning. It involves randomly sampling with replacement from the training set. Each time a sample is drawn, it is recorded and then returned to the training set, ensuring it has a chance to be selected again in the next draw. This sampling process is repeated multiple times (usually the number of draws is the same as the number of samples in the training set), resulting in multiple training subsets.

[0157] The fourth step involves training the pre-built machine learning model using multiple training subsets and testing the trained machine learning model using a test sample set to obtain the industrial equipment recognition model.

[0158] Understandably, as the number of identification categories increases, the number of preferred feature data also increases. In supervised machine learning models, decision trees are a type of model that can be used for fast classification. The feature data extracted from the image to be tested contains both preferred feature data and some interference feature data. The identification of a single decision tree has a certain error. Therefore, this embodiment adopts a scheme of combining multiple decision trees. That is, the machine learning model adopts a random forest model and determines the category of industrial equipment through voting and scoring, which improves the identification accuracy and ensures the generalization ability of the model.

[0159] In actual production, the image to be tested is first acquired by a depth camera, and the SIFT algorithm is used to match the feature point library to obtain the feature data of the image to be tested. The feature data to be tested is then input into the industrial equipment recognition model to obtain the category of industrial equipment in the image to be tested.

[0160] After determining the category of industrial equipment, the first step is to identify the key features used for positioning within that category. For example, for large CNC machine tools, fixed base corners and specific guide rail markings are key features for positioning; while for small testing instruments, prominent markings on the casing or unique interface shapes can serve as key features for positioning.

[0161] Then, based on the industrial equipment category and its key features, a corresponding localization algorithm is determined. For example, for industrial equipment with regular shapes and obvious features, a localization algorithm based on geometric feature matching can be used to determine the location by calculating the degree of matching between key features and a preset template. For equipment with complex textures or appearances, object detection algorithms in deep learning can be used to first detect the location of the industrial equipment in the image to be tested, and then combine camera calibration information to transform it into the world coordinate system. If the installation method and layout rules of the equipment are known, a localization algorithm based on rule reasoning can also be used to determine its location through the relative positional relationships between industrial equipment and known installation drawings.

[0162] Subsequently, the selected localization algorithm is used to determine the preliminary position of the industrial equipment in the image under test. For example, an edge detection algorithm is used to extract the contour edges of the industrial equipment, and then a contour matching algorithm is used to determine the preliminary position coordinates of the industrial equipment in the image under test.

[0163] Finally, after obtaining the initial position of the industrial equipment in the image to be tested, the final position coordinates of the industrial equipment in the world coordinate system are obtained by calibrating the camera coordinates and world coordinates, thus completing the automated visual positioning of the industrial equipment. Subsequently, the industrial robot can perform production-related operations according to the final position coordinates of the industrial equipment.

[0164] In summary, the industrial equipment automated visual positioning method provided by the embodiments of the present invention analyzes the structural features of the industrial equipment to be identified, establishes more effective optimized feature data, and trains and tests a pre-established machine learning model using the optimized feature data. Subsequently, the industrial equipment recognition model is used to realize the automatic recognition of industrial equipment in the image under test, thereby completing automated visual positioning, improving the accuracy and reliability of automated visual recognition of industrial equipment, and thus improving the operating efficiency of industrial robots.

[0165] Based on the same general inventive concept, this invention also protects an automated visual positioning system for industrial equipment. The automated visual positioning system for industrial equipment provided by this invention will be described below. The automated visual positioning system for industrial equipment described below can be referred to in correspondence with the automated visual positioning method for industrial equipment described above.

[0166] Please see Figure 6 It shows a system structure diagram of an automated visual positioning system for industrial equipment according to an embodiment of the present invention, such as... Figure 6 As shown, the aforementioned automated visual positioning system for industrial equipment specifically includes:

[0167] The acquisition module 410 is used to acquire multiple equipment images corresponding to different types of industrial equipment during the production process.

[0168] The extraction module 420 is used to extract feature points from each equipment image and determine the pointing value of each feature point for each type of industrial equipment.

[0169] The allocation module 430 is used to assign each feature point to the industrial equipment with the largest pointing value, thereby obtaining the preferred feature data corresponding to each type of industrial equipment.

[0170] Modeling module 440 is used to train and test a pre-built machine learning model based on the preferred feature data corresponding to each type of industrial equipment, so as to obtain an industrial equipment recognition model.

[0171] The recognition module 450 is used to input the feature data extracted from the image to be tested into the industrial equipment recognition model to obtain the industrial equipment category output by the industrial equipment recognition model.

[0172] The positioning module 460 is used to locate industrial equipment in the image under test based on the category of industrial equipment.

[0173] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0174] The industrial equipment automated visual positioning system provided in this embodiment uses an industrial equipment identification model based on the industrial equipment identification process. This model is trained and tested using the preferred feature data of each type of industrial equipment. The preferred feature data is obtained based on the pointing value of each feature point to each type of industrial equipment. This allows the preferred feature data to more accurately represent the characteristics of the corresponding type of industrial equipment. As a result, the industrial equipment identification model can accurately identify the type of industrial equipment, thereby improving the identification accuracy and positioning accuracy and reliability.

[0175] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0176] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. An industrial equipment automation visual positioning method, characterized in that, The method comprises: acquiring a plurality of device images corresponding to different types of industrial equipment in a production process; extracting feature points in each device image and determining a pointing value of each feature point to each type of industrial equipment; assigning each feature point to the industrial equipment with the largest pointing value to obtain preferred feature data corresponding to each type of industrial equipment; training and testing a pre-constructed machine learning model based on the preferred feature data corresponding to each type of industrial equipment to obtain an industrial equipment recognition model; inputting test feature data extracted from a test image into the industrial equipment recognition model to obtain an industrial equipment category output by the industrial equipment recognition model; positioning the industrial equipment in the test image based on the industrial equipment category; extracting feature points in each device image, comprising: determining the gray information and depth information of each pixel point in each device image; determining an object region where the industrial equipment is located in each device image according to the gray information of each pixel point in each device image; establishing a terrain three-dimensional model corresponding to the object region in each device image according to the depth information of each pixel point in the object region in each device image; dividing the terrain three-dimensional model corresponding to the object region in each device image into a plurality of pixel point sets; taking the pixel point with the largest or smallest depth information in each pixel point set as a feature point to obtain the feature points in each device image; wherein the method for obtaining the terrain three-dimensional model corresponding to the object region in each device image comprises: determining a horizon according to the pixel point with the largest depth information in the object region in each device image; taking the depth distance between each pixel point in the object region in each device image and the horizon as the height of each pixel point; obtaining the terrain three-dimensional model corresponding to the object region in each device image according to the height of all pixel points in the object region in each device image; the method for determining the pointing value of each feature point to each type of industrial equipment comprises: calculating a feature expression value corresponding to each pixel point set; taking the pixel point set with the largest feature expression value in the pixel point set with the same feature point as a feature pixel point set; determining a matching region corresponding to each feature pixel point set in each device image; calculating the matching degree between the feature point in each feature pixel point set and each matching region; calculating the pointing value of each feature point to each type of industrial equipment according to the matching degree between the feature point in each feature pixel point set and each matching region; the method for calculating the feature expression value corresponding to each pixel point set comprises: determining the number of valley point sets and the number of watershed line point sets in each pixel point set obtained in each region division link, and determining the number of pixel points in each pixel point set, the maximum and minimum values of pixel point heights, and the height difference value of any one group of adjacent pixel points on each center connecting straight line; calculating the number difference degree of valleys and watershed lines in each region division link according to the number of valley point sets and the number of watershed line point sets; calculating the average height distribution value of each pixel point set according to the number of pixel points and the maximum and minimum values of pixel point heights of each pixel point set; The sum of height differences of adjacent pixel points on each center connecting line of each pixel point set is calculated; The feature value corresponding to each pixel point set is calculated based on the degree of difference, the average height distribution value, and the sum of height differences of adjacent pixel points; The calculation method of the matching degree of the feature point in each feature pixel point set and the matching region includes: The shape feature and relative position feature of the feature point in each pixel point set are determined, and the shape feature and relative position feature of the matching point corresponding to the feature point in the matching region are determined; The edit distance between the shape feature of the feature point in each pixel point set and the shape feature of the corresponding matching point is calculated to obtain a first distance value; The edit distance between the relative position feature of the feature point in each pixel point set and the relative position feature of the corresponding matching point is calculated to obtain a second distance value; The matching degree of the feature point in each feature pixel point set and the matching region is calculated based on the first distance value and the second distance value; The calculation method of the pointing value of each feature point to each industrial equipment includes: The sum of the matching degrees of the feature point in each feature pixel point set and all matching regions corresponding to each industrial equipment is calculated to obtain a matching sum value of each feature point to each industrial equipment; The maximum value of the matching degrees of each feature point and each matching region corresponding to each industrial equipment is determined to obtain a first matching maximum value; For any industrial equipment, the maximum value of the matching degrees of each feature point and each matching region corresponding to other industrial equipment is determined to obtain a second matching maximum value; The absolute value of the difference between the first matching maximum value and the second matching maximum value is calculated to obtain a matching maximum difference value; The pointing value of each feature point to each industrial equipment is calculated based on the matching maximum difference value and the matching sum value.

2. The method of claim 1, wherein, The industrial equipment recognition model is obtained by training and testing the pre-constructed machine learning model based on the preferred feature data corresponding to each industrial equipment, including: A sample data set is established based on the preferred feature data corresponding to each industrial equipment and the industrial equipment type corresponding to the preferred feature data; The sample data set is divided into a training sample set and a test sample set according to a preset ratio; The training sample set is sampled by a random sampling method with replacement to generate multiple training subsets; The pre-constructed machine learning model is trained through the multiple training subsets, and the trained machine learning model is tested through the test sample set to obtain the industrial equipment recognition model.

3. The method of claim 1, wherein, The multiple device images corresponding to different types of industrial equipment in the production process are obtained, including: Multiple initial images corresponding to different types of industrial equipment in the production process uploaded by a depth camera are received; wherein the multiple initial images are used to represent the morphology of the industrial equipment under multiple angles and multiple directions; Each initial image is subjected to grayscale processing to obtain multiple device images corresponding to different types of industrial equipment in the production process.

4. An industrial equipment automation vision positioning system, characterized by, The system includes: The acquisition module is configured to acquire a plurality of device images corresponding to different types of industrial equipment in a production process; The extraction module is configured to extract feature points in each device image and determine a pointing value of each feature point to each type of industrial equipment; The distribution module is configured to distribute each feature point to an industrial equipment with a maximum pointing value to obtain preferred feature data corresponding to each type of industrial equipment; The modeling module is configured to train and test a pre-constructed machine learning model based on the preferred feature data corresponding to each type of industrial equipment to obtain an industrial equipment recognition model; The recognition module is configured to input measured feature data extracted from a measured image into the industrial equipment recognition model to obtain an industrial equipment category output by the industrial equipment recognition model; The positioning module is configured to position an industrial equipment in the measured image based on the industrial equipment category. The extraction of the feature points in each device image includes: determining gray information and depth information of each pixel point in each device image; determining an object region in which the industrial equipment is located in each device image based on the gray information of each pixel point in each device image; establishing a terrain three-dimensional model corresponding to the object region in each device image according to the depth information of each pixel point in the object region in each device image; performing region division on the terrain three-dimensional model corresponding to the object region in each device image to obtain a plurality of pixel point sets; taking a pixel point with maximum or minimum depth information in each pixel point set as a feature point to obtain the feature points in each device image; The method for obtaining the terrain three-dimensional model corresponding to the object region in each device image includes: determining a horizon based on a pixel point with maximum depth information in the object region in each device image; taking a depth distance between each pixel point in the object region in each device image and the horizon as a height of the pixel point; obtaining the terrain three-dimensional model corresponding to the object region in each device image based on the heights of all pixel points in the object region in each device image; The method for determining the pointing value of each feature point to each type of industrial equipment includes: calculating a feature expression value corresponding to each pixel point set; taking a pixel point set with maximum feature expression value in a pixel point set with the same feature point as a feature pixel point set; determining a matching region corresponding to each feature pixel point set in each device image; calculating a matching degree between the feature point in each feature pixel point set and each matching region; calculating the pointing value of each feature point to each type of industrial equipment based on the matching degree between the feature point in each feature pixel point set and each matching region; The method for calculating the feature expression value corresponding to each pixel point set includes: determining a valley point set quantity and a watershed line point set quantity in each pixel point set obtained in each region division link, and determining a pixel point quantity, a maximum and minimum height of a pixel point, and a height difference value of any one group of adjacent pixel points on each center connecting straight line in each pixel point set; calculating a quantity difference degree of the valley and the watershed line in each region division link based on the valley point set quantity and the watershed line point set quantity. According to the number of pixels and the maximum height of the pixels in each pixel set, an average height distribution value of each pixel set is calculated; According to the height difference of all adjacent pixels on the center connecting line of each pixel set, a sum of height differences of adjacent pixels is calculated; Based on the number difference degree, the average height distribution value and the sum of height differences of adjacent pixels, a feature value corresponding to each pixel set is calculated; The calculation method of the matching degree of each feature pixel set and the matching region includes: The shape feature and the relative position feature of the feature points in each pixel set are determined, and the shape feature and the relative position feature of the matching points corresponding to the feature points in the matching region are determined; The edit distance between the shape feature of the feature points in each pixel set and the shape feature of the corresponding matching points is calculated to obtain a first distance value; The edit distance between the relative position feature of the feature points in each pixel set and the relative position feature of the corresponding matching points is calculated to obtain a second distance value; Based on the first distance value and the second distance value, the matching degree of the feature points in each feature pixel set and the matching region is calculated; The calculation method of the pointing value of each feature point to each industrial equipment includes: The matching degrees of the feature points in each feature pixel set and all matching regions corresponding to each industrial equipment are summed to obtain a matching sum value of each feature point to each industrial equipment; The maximum value in the matching degrees of each feature point and each matching region corresponding to each industrial equipment is determined to obtain a first matching maximum value; For any industrial equipment, the maximum value in the matching degrees of each feature point and each matching region corresponding to other industrial equipment is determined to obtain a second matching maximum value; The absolute value of the difference between the first matching maximum value and the second matching maximum value is calculated to obtain a matching maximum difference value; Based on the matching maximum difference value and the matching sum value, the pointing value of each feature point to each industrial equipment is calculated.

Images