An image feature extraction method, system, and storage medium based on stereo vision

Abstract

The present invention discloses an image feature extraction method based on stereo vision, including image acquisition: taking pictures of a workpiece through a stereo vision device to collect a point cloud map, separating the point cloud map to obtain a depth map A; dilation processing: performing dilation processing on the depth map A to obtain an image B, subtracting the dilated image B from the depth map A to obtain a gradient image C; hierarchical edge extraction: traversing the gradient image C, setting pixels with a height gradient value less than a set threshold θ in the gradient image C to 0 to obtain an image D with contour features; highest layer screening: performing connected region screening on the image D, extracting the largest connected region therein to obtain a point cloud image E; contour feature registration: performing contour extraction on the point cloud image E to obtain contour features, and calling the contour features for ICP registration with a saved feature template; capable of accurately identifying and positioning the features of stacked planar sheet metals or stamping parts.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, system and storage medium for extracting image features based on stereoscopic vision. Background Art

[0002] With the development of industrial technology, sheet metal processing, such as loading and unloading and steel plate welding, has gradually begun to use robots for batch processing. However, there are certain limitations in using robots for such repetitive operations. This is because different workpieces have different shapes or heights, and it is difficult for robots to locate, clamp and process the workpieces. Therefore, the existing technology usually uses the following methods for processing:

[0003] (1) The loading and unloading of workpieces is realized by directly processing the corresponding workpiece fixtures and cooperating with the robot depalletizing and stacking program. This solution is suitable for mass production. When small batches and diversification occur, the cost of making fixtures and the increase in design workload will cause problems, which will trouble customers to further expand production.

[0004] (2) Identify and locate the workpiece through three-dimensional vision. This solution directly solves the problem of repeated fixture production through visual positioning. However, this solution is only applicable to parts with curved surfaces or height differences in the imaging surface. When the entire workpiece is flat, there is no curved feature on the workpiece surface, and the normal vector features at any point are the same. Inaccurate or erroneous recognition may occur through point cloud feature point recognition technology, and point cloud feature point matching may lead to positioning deviation or error.

[0005] Therefore, there is an urgent need for a more accurate method for extracting features of planar stacked workpieces for positioning. Summary of the invention

[0006] In order to solve the above technical problems, the first aspect of the present invention discloses a method for extracting image features based on stereoscopic vision, the method comprising:

[0007] Image acquisition: photograph the workpiece through a stereoscopic vision device to collect a point cloud image, separate the point cloud image, and obtain a depth map A;

[0008] Dilation processing: dilating the depth map A to obtain an image B, and subtracting the depth map A from the dilated image B to obtain a gradient image C;

[0009] Hierarchical edge extraction: traverse the gradient image C, set the pixel points in the gradient image C whose height gradient value is less than the set threshold θ to 0, so as to obtain an image D with contour features;

[0010] Highest-level screening: Perform connected region screening on the image D, and extract the largest connected region therefrom to obtain a point cloud image E;

[0011] Contour feature registration: Perform contour extraction on the point cloud image E to obtain contour features, and call the contour features for ICP registration with the saved feature template.

[0012] Furthermore, in the contour feature registration, the saved feature template called is the point cloud image feature obtained by performing image acquisition, dilation processing, hierarchical edge extraction, and highest-level screening on the workpiece offline.

[0013] Furthermore, it also includes a kernel matrix K. When performing the dilation processing, the following steps are included:

[0014] Obtain the source matrix S of the depth map A, and the source matrix S includes the height value of the depth map;

[0015] Take the maximum value that is not 0 in the source matrix S as the kernel anchor point;

[0016] The kernel matrix K traverses from left to right and from top to bottom starting from the kernel anchor point, and updates the source matrix S to obtain the image matrix DD of the image B.

[0017] Furthermore, the following steps are also included in the hierarchical edge extraction:

[0018] Subtract the source matrix S from the image matrix DD to obtain the result matrix E of the gradient image C;

[0019] Traverse the pixel values of the result matrix E, and set to 0 the pixel value height values that are greater than the set threshold θ.

[0020] Furthermore, when performing the highest-level screening, the algorithm for performing connected region screening on the image D includes one of Two-Pass and Seed Filling.

[0021] Furthermore, when performing contour extraction on the point cloud image E, the boundary point cloud of the point cloud image E is extracted by an edge normal vector mutation algorithm, including the following steps:

[0022] Calculate the normal of the point cloud image E;

[0023] Based on the normal and data, calculate the boundary of the point cloud image E to obtain the boundary point cloud.

[0024] Furthermore, after obtaining the boundary point cloud, the burr points of the boundary point cloud are forcibly smoothed in the height direction by a smoothing algorithm.

[0025] The second aspect of the present invention discloses an image feature extraction system, which includes:

[0026] A stereo vision device for photographing a workpiece to collect a point cloud map, separating the point cloud map, and obtaining a depth map A;

[0027] An inflation processing module for performing inflation processing on the depth map A to obtain an image B, and subtracting the inflated image B from the depth map A to obtain a gradient image C;

[0028] A hierarchical edge extraction module for traversing the gradient image C and setting the pixel points with a height gradient value less than a set threshold θ in the gradient image C to 0 to obtain an image D with contour features;

[0029] A screening module for performing connected region screening on the image D and extracting the largest connected region therein to obtain a point cloud image E;

[0030] An extraction module for performing contour extraction on the point cloud image E to obtain contour features;

[0031] A storage module for storing a feature template;

[0032] A registration module for obtaining the feature template from the stored template and performing ICP registration with the contour features.

[0033] The third aspect of the present invention discloses a computer storage medium storing computer instructions, which are used to execute some or all of the steps in the image feature extraction method based on stereo vision disclosed in the first aspect of the present invention when the computer instructions are called.

[0034] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0035] In the embodiments of the present invention, first, a new workpiece is photographed by a stereo vision device to collect a point cloud map of the image. After data separation processing of the point cloud map, a depth map A is obtained. After dilation processing of the depth map A, the overall pixel values of the depth map A are expanded to obtain an image B. By subtracting the image B from the depth map A, a gradient image C is obtained, making the edges of the workpiece in the image more prominent. By traversing the gradient image C and setting the pixel points that meet the requirements to 0, the overall edges of the high-level workpieces are made more prominent, so as to avoid the situation of registration errors caused by the same features of the low-level workpieces and the high-level workpieces. Since in the application scenario of workpiece stacking, the workpieces at the bottom are blocked, the connected area of the point cloud of the topmost workpiece is usually the largest. Therefore, in order to extract the point cloud of the topmost workpiece, the largest connected area of the image D is screened to obtain the point cloud image E of the topmost layer. The contour features of the point cloud image E are extracted as a feature template. Subsequent workpieces also extract contour features through the above method and perform ICP registration and positioning with the feature template, so as to facilitate the robot to perform loading and unloading operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0037] Figure 1 is a flowchart of a method for extracting image features based on stereo vision disclosed in the embodiments of the present invention;

[0038] Figure 2 is a structural diagram of an image feature extraction system disclosed in the embodiments of the present invention;

[0039] Figure 3 is a structural diagram of a computer storage medium disclosed in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0041] In the description, claims and the above drawings of the present invention, terms such as "first" and "second" are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or terminal comprising a series of steps or units is not limited to the listed steps or units, but may optionally further include steps or units not listed, or may optionally further include other steps or units inherent to these processes, methods, products or terminals.

[0042] Reference to "embodiment" herein means that a particular feature, structure or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0043] The present invention discloses a method, system and storage medium for extracting image features based on stereo vision. In the embodiments of the present invention, first, a stereo vision device is used to take a picture of a newly arrived workpiece to collect a point cloud map of the image. After data separation processing of the point cloud map, a depth map A is obtained. After dilation processing of the depth map A, the overall pixel value of the depth map A is expanded to obtain an image B. By subtracting the image B from the depth map A, a gradient image C is obtained, making the edges of the workpiece in the image more prominent. By traversing the gradient image C and setting the pixel points that meet the requirements to 0, the overall edge of the high-level workpiece is made more prominent; since in the application scenario of workpiece stacking, the bottom workpieces are blocked, the connected area of the point cloud of the topmost workpiece is usually the largest. Therefore, in order to extract the point cloud of the topmost workpiece, the largest connected area of the image D is screened out to obtain the point cloud image E of the topmost layer; the contour features of the point cloud image E are extracted as a feature template, and subsequent workpieces are also processed through the above method to extract contour features and perform ICP registration and positioning with the feature template, so as to facilitate the robot for loading and unloading operations. The following will be described in detail respectively.

[0044] Embodiment 1

[0045] Please refer to Figure 1 , Figure 1 which is a schematic flow chart of a method for extracting image features based on stereo vision disclosed in the embodiments of the present invention. As Figure 1 shown, the method for extracting image features based on stereo vision may include the following operations:

[0046] 101 Image acquisition: A stereo vision device is used to take a picture of a workpiece to collect a point cloud map, and the point cloud map is separated to obtain a depth map A;

[0047] 102 Dilation processing: Perform dilation processing on the depth map A to obtain an image B, and subtract the dilated image B from the depth map A to obtain a gradient image C;

[0048] Dilation processing is essentially to expand the boundary points of the binary object, merge all background points in contact with the object into the object, and expand the boundary outwards. In the embodiment of the present invention, a kernel matrix K is further included. During the dilation processing, the following steps are included: obtaining a source matrix S of the depth map A, where the source matrix S includes the height value of the depth map; taking the maximum value other than 0 in the source matrix S as the kernel anchor point; the kernel matrix K traverses from left to right and from top to bottom starting from the kernel anchor point, and updates the source matrix S to obtain an image matrix DD of the image B.

[0049] 103 Hierarchical edge extraction: Traverse the gradient image C, and set the pixel points in the gradient image C with a height gradient value less than a set threshold θ to 0 to obtain an image D with contour features;

[0050] Among them, the following steps are further included in the hierarchical edge extraction: subtracting the source matrix S from the image matrix DD to obtain a result matrix E of the gradient image C; traversing the pixel values of the result matrix E, and setting the pixel values with a height value greater than the set threshold θ to 0. The reasons for the above dilation processing and hierarchical edge extraction are that when the depth of field between the stereo vision device and the workpiece exceeds 1000 mm in height and the thickness of each layer of the workpiece is only 10 - 20 mm, the hierarchical boundary reflected by the height difference can hardly be reflected in the depth map. At the same time, since the Z direction of the camera and the height direction of the workpiece are not strictly in the same direction, that is, the Z direction may be inclined, it is also impossible to directly extract the hierarchical boundary of the highest layer through the height Z. Therefore, it is necessary to first highlight the hierarchical boundary of the workpieces in the stacked workpieces through dilation processing, and then highlight the edges of the workpieces belonging to the upper layer through the steps of hierarchical edge extraction, realizing the highlighting of the point cloud edges of the upper-layer workpieces in the stacked workpieces for subsequent screening of the highest layer.

[0051] 104 Highest-level screening: Perform connected region screening on the image D, and extract the largest connected region therefrom to obtain a point cloud image E. Further, when performing the highest-level screening, the algorithm used for connected region screening of the image D includes one of Two-Pass and Seed Filling. Since the layered area of stacked sheet metal or stamped parts at the highest level is definitely the largest, the focus of this step is to achieve the screening of the largest connected region. Therefore, the method of applying other connected region screening algorithms should also be within the scope of protection of this patent. The reason for extracting the point cloud contour feature image of the highest-level workpiece is that if all contour feature images are extracted, the lower-layer edge contour features will be the same as those of the top layer. In the case of non-overlap, multiple contour features will cause deviation in the final fine matching (ICP registration).

[0052] 105 Contour feature registration: Perform contour extraction on the point cloud image E to obtain contour features, and call the ICP registration of the contour features with the saved feature template.

[0053] Further, when performing contour extraction on the point cloud image E, the boundary point cloud of the point cloud image E is extracted by the edge normal vector mutation algorithm, including the following steps:

[0054] Calculate the normal of the point cloud image E;

[0055] Based on the normal and data, calculate the boundary of the point cloud image E to obtain the boundary point cloud.

[0056] Among them, in the contour feature registration, the saved feature template called is the point cloud image feature obtained by performing image acquisition, dilation processing, layered edge extraction, and highest-level screening on the workpiece in an offline state.

[0057] The embodiments of the present invention have at least the following beneficial effects:

[0058] For the three-dimensional workpiece stacking scenario without surface curvature features or height difference features, the point cloud recognition and registration of the highest-level workpiece are proposed, making the robot more accurate in positioning the workpiece during processing.

[0059] It can be seen that the implementation Figure 1A method for extracting image features based on stereo vision is described. First, a stereo vision device takes pictures of newly arrived workpieces to collect the point cloud map of the image. After performing data separation processing on the point cloud map, a depth map A is obtained. After performing dilation processing on the depth map A, the overall pixel values of the depth map A are expanded to obtain an image B. By subtracting the image B from the depth map A, a gradient image C is obtained, making the edges of the workpiece in the image more prominent. By traversing the gradient image C and setting the pixel points that meet the requirements to 0, the overall edges of the high-level workpiece are made more prominent. Since in the application scenario of workpiece stacking, the workpieces at the bottom are blocked, the connected region of the point cloud of the topmost workpiece is usually the largest. Therefore, in order to extract the point cloud of the topmost workpiece, the largest connected region of the image D is selected to obtain the point cloud image E of the topmost layer. The contour features of the point cloud image E are extracted as the feature template. Subsequent workpieces also extract contour features through the above method and perform ICP registration and positioning with the feature template to facilitate the loading and unloading operations of the robot.

[0060] Embodiment 2

[0061] Please refer to Figure 2 , Figure 2 which is a schematic structural diagram of the image feature extraction system disclosed in the embodiment of the present invention. As Figure 2 shown, the method for extracting image features based on stereo vision may include:

[0062] A stereo vision device 201, configured to take pictures of workpieces to collect a point cloud map, and separate the point cloud map to obtain a depth map A;

[0063] An expansion processing module 202, configured to perform expansion processing on the depth map A to obtain an image B, and subtract the expanded image B from the depth map A to obtain a gradient image C;

[0064] A hierarchical edge extraction module 203, configured to traverse the gradient image C, and set the pixel points in the gradient image C with a height gradient value less than a set threshold θ to 0 to obtain an image D with contour features;

[0065] A screening module 204, configured to perform connected region screening on the image D, and extract the largest connected region therein to obtain a point cloud image E;

[0066] An extraction module 205, configured to perform contour extraction on the point cloud image E to obtain contour features;

[0067] A storage module 206, configured to store the feature template;

[0068] A registration module 207, configured to obtain the feature template from the stored template and perform ICP registration with the contour features.

[0069] For the specific description of the above image feature extraction system, reference may be made to the specific description of the above image feature extraction method based on stereo vision, which will not be elaborated here one by one.

[0070] Embodiment III

[0071] Please refer to Figure 3 , Figure 3 which is a schematic structural diagram of a computer storage medium disclosed in an embodiment of the present invention. As Figure 3 shown, an embodiment of the present invention discloses a computer storage medium 301. The computer storage medium 301 stores computer instructions, which are used to execute the steps in the image feature extraction method based on stereo vision disclosed in Embodiment I of the present invention when the computer instructions are called.

[0072] The device embodiments described above are only illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0073] Through the specific descriptions of the above embodiments, those skilled in the art can clearly understand that each implementation mode can be realized by means of software plus a necessary general hardware platform, and of course, it can also be realized by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, and the storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0074] Finally, it should be noted that: The disclosed method, system, and storage medium for image feature extraction based on stereo vision according to the embodiments of the present invention are only the preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An image feature extraction method based on stereo vision, characterized in that, The method includes: Image acquisition: Taking pictures of the workpiece through a stereo vision device to collect a point cloud map, separating the point cloud map to obtain a depth map A; Dilation processing: Performing dilation processing on the depth map A to obtain an image B, subtracting the dilated image B from the depth map A to obtain a gradient image C; Hierarchical edge extraction: Traversing the gradient image C, setting the pixel points with a height gradient value less than a set threshold θ in the gradient image C to 0 to obtain an image D with contour features; Highest layer screening: Performing connected region screening on the image D, extracting the largest connected region therein to obtain a point cloud image E; Contour feature registration: Performing contour extraction on the point cloud image E to obtain contour features, and calling the contour features to perform ICP registration with a saved feature template; It further includes a kernel matrix K. During the dilation processing, the following steps are included: Obtaining a source matrix S of the depth map A, where the source matrix S includes the height value of the depth map; Taking the non-zero maximum value in the source matrix S as the kernel anchor point; The kernel matrix K traverses from left to right and from top to bottom starting from the kernel anchor point, updating the source matrix S to obtain an image matrix DD of the image B; The following steps are further included in the hierarchical edge extraction: Subtracting the source matrix S from the image matrix DD to obtain a result matrix E of the gradient image C; Traversing the pixel values of the result matrix E, and setting them to 0 if the pixel value height value is greater than the set threshold θ.

2. The method for extracting image features based on stereo vision according to claim 1, characterized in that In the contour feature registration, the saved feature template called is the point cloud image feature obtained by performing the image acquisition, dilation processing, hierarchical edge extraction, and highest layer screening on the workpiece in an offline state.

3. The method for extracting image features based on stereo vision according to claim 1, characterized in that During the highest layer screening, the algorithm for performing connected region screening on the image D includes one of Two-Pass and Seed Filling.

4. The method for extracting image features based on stereo vision according to claim 1, wherein When performing contour extraction on the point cloud image E, the boundary point cloud of the point cloud image E is extracted through an edge normal vector mutation algorithm, including the following steps: Calculating the normal of the point cloud image E; Calculating the boundary of the point cloud image E based on the normal and data to obtain the boundary point cloud.

5. The method for extracting image features based on stereo vision according to claim 4, characterized in that After obtaining the boundary point cloud, the burr points of the boundary point cloud are forcibly smoothed in the height direction through a smoothing algorithm.

6. An image feature extraction device for implementing the stereo vision-based image feature extraction method according to any one of claims 1-5, characterized in that, The device includes: A stereo vision device for taking pictures of the workpiece to collect a point cloud map, separating the point cloud map to obtain a depth map A; A dilation processing module for performing dilation processing on the depth map A to obtain an image B, subtracting the dilated image B from the depth map A to obtain a gradient image C; A hierarchical edge extraction module for traversing the gradient image C, setting the pixel points with height gradient values less than a set threshold θ in the gradient image C to 0 to obtain an image D with contour features; A screening module for performing connected region screening on the image D and extracting the largest connected region therein to obtain a point cloud image E; An extraction module for performing contour extraction on the point cloud image E to obtain contour features; A saving module for saving feature templates; A registration module for obtaining the feature template from the saved template and performing ICP registration with the contour features.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for extracting image features based on stereo vision according to any one of claims 1-5.

Images