Part surface positioning method based on machine vision

Abstract

The invention belongs to the field of part positioning, and particularly relates to a part surface positioning method based on machine vision, which comprises the steps of multi-modal data fusion, template production, local feature extraction, template clustering and hierarchical matching and exception handling. According to the scheme, the infrared sensor and the laser scanner are introduced, multi-modal data fusion is achieved, high-temperature abnormal areas and surface geometric anomalies are extracted, the advantages of low power consumption of a common camera and high imaging quality of a high-precision macro camera are utilized, whether the high-precision macro camera is triggered to carry out high-definition supplementary shooting or not is judged according to the confidence coefficient, and high-precision supplementary shooting is realized. The technical effects of intelligent positioning of parts, high precision and low cost are achieved; layered matching of part models and quantification of defect probability are realized through the template library, template matching is performed only by using local key features, computing resources are optimized through a clustering mode, detection efficiency is improved, and the technical effect of part model positioning in a complex industrial environment is realized.

Description

Technical Field

[0001] The present invention relates to the field of component positioning, and in particular to a component surface positioning method based on machine vision. Background Art

[0002] The component surface positioning method based on machine vision is a method for surface positioning of components based on machine vision.

[0003] Among the existing approximate solutions, for example, CN114972355B is a method for locating and measuring surface defects of components based on visual perception. This solution addresses the technical problems of high label data labeling costs, insufficient network generalization capabilities, and low network reasoning accuracy in existing technologies. It uses a density clustering method to obtain multiple angle sets, calculates the description vector of each angle set based on edge structure information, dynamically adjusts the translation step size, and trains the defect classification network to make the step size more adaptable to the shape and size of the abnormal area. This achieves the technical effect of reducing the cost of labeling data, eliminating the need for complex network reasoning, and improving the accuracy of defect detection. However, the existing technology mainly targets components with obvious surface defects that can be captured by cameras, and has difficulty identifying subtle defects and defects in complex backgrounds.

[0004] In addition, for example, CN117291985B discloses an image positioning method for part punching. This method addresses the technical problem that the Canny algorithm connects unrelated edges together in the final edge connection stage, resulting in low accuracy of the edge results. The method calculates the pixel feature indicators of each area in the grayscale image of the part surface, extracts the edges to be connected, evaluates the similarity of the edge structure, and matches the connected edges with the standard positioning template to achieve high-precision punching positioning, thereby achieving the technical effect of improving the accuracy of punching position positioning. However, when there are different part materials, lighting conditions or surface texture changes, the fixed threshold may lead to false detection or missed detection, and the parameters are highly dependent and the high-noise environment such as metal surface reflection and scratches affects the screening of similar edge areas. Summary of the Invention

[0005] In response to the above situation and to overcome the shortcomings of the prior art, the present invention provides a method for component surface positioning based on machine vision. The prior art mainly targets components with obvious surface defects that can be captured by cameras, but has difficulty identifying subtle defects or defects in complex backgrounds. This solution introduces infrared sensors and laser scanners to achieve multimodal data fusion, extract high-temperature anomaly areas and surface geometric anomalies, and leverages the advantages of low power consumption of ordinary cameras and high imaging quality of high-precision macro cameras. Based on the confidence level, it determines whether to trigger the high-precision macro camera for high-definition re-shooting, thereby achieving the technical effect of intelligent component positioning with both high precision and low cost. Furthermore, the prior art has the technical problems that fixed thresholds may lead to false detection or missed detection when different part materials, lighting conditions, or surface textures vary, and that parameters are highly dependent, and that high-noise environments such as metal surface reflections and scratches affect the screening of similar edge areas. This solution uses a template library to achieve hierarchical matching of component models, quantify defect probabilities, and only use local key features for template matching. Clustering is used to optimize computing resources and improve detection efficiency, thereby achieving the technical effect of component model positioning in complex industrial environments.

[0006] The technical solution adopted by the present invention is as follows: The present invention provides a component surface positioning method based on machine vision, which includes the following steps:

[0007] Step S1: multimodal data fusion;

[0008] Step S2: Template production, obtaining variant templates, and recording the variant templates of all component models as a template library;

[0009] Step S3: extracting local features, specifically, for each model variant template, extracting local key features that are different from other models' variant templates and storing them as structured data;

[0010] Step S4: Template clustering and hierarchical matching: clustering the template library using a clustering algorithm. The variant template closest to the cluster center in each cluster is used as the representative template, and the matching score is obtained through hierarchical matching.

[0011] Step S5: Exception handling, specifically, calculating the dispersion of the matching scores corresponding to all component models, presetting a matching threshold, and if the dispersion exceeds the matching threshold, the matching fails. For each component to be located that fails to match, a warning is triggered and it is automatically added to the template library as a new template for manual review and judgment. If the dispersion does not exceed the matching threshold, the matching is successful, and the component model with the highest matching score is output.

[0012] Furthermore, in step S1, the multimodal data fusion specifically includes the following steps:

[0013] Step S11: Initial defect screening, specifically, using an infrared sensor to detect fine cracks on parts and using a laser scanner to supplement surface scratches and concave-convex information on parts, to obtain scanning point cloud data and thermal radiation distribution maps respectively;

[0014] Step S12: rough modeling, specifically, using a low-power wide-angle camera to shoot the parts, and generating a rough mold based on the shooting effect. The rough mold represents a rough three-dimensional virtual model, and the scanning point cloud data and thermal radiation distribution map are mapped to the rough mold;

[0015] Step S13: Temperature anomaly extraction, used to quantify the probability of component defects. Specifically, the high temperature abnormal area of ​​the component is extracted through threshold segmentation, and the confidence of the thermal radiation distribution map is calculated. The formula used is as follows:

[0016]

[0017] Where IRs represents the confidence level of the thermal radiation distribution map, {map} represents the total surface area of ​​the component, {map|T_p>T_avg+2σ} represents the area of ​​the abnormal region of the component, σ represents the temperature standard deviation of the component, T_avg represents the average temperature of the component, T_p represents the component temperature of any pixel in the thermal radiation distribution map, σ_d represents the temperature standard deviation in the abnormal region, and max(ΔT_d) represents the maximum temperature difference in the abnormal region.

[0018] Step S14: Geometric anomaly extraction is used to quantify the probability of geometric anomalies on the component surface and calculate the laser confidence. The formula used is as follows:

[0019] Ls=avg(sigmoid(Δz_j)+C_j);

[0020] Where Ls represents the laser confidence, avg() represents the mean function, sigmoid() represents the activation function, C_j represents the curvature of any point in the point cloud, and Δz_j represents the difference between the depth of any point in the point cloud and the average depth of the neighboring points;

[0021] Step S15: Analyze the confidence of each item, preset the retake threshold, assign weights to the thermal radiation distribution map confidence and laser confidence, and calculate the verification confidence. The formula used is as follows:

[0022] Cf=μ·IRs+(1-μ)·Ls;

[0023] Where Cf represents the verification confidence, IRs represents the thermal radiation distribution map confidence, Ls represents the laser confidence, and μ represents the contribution coefficient;

[0024] Step S16: High-definition review, used to trigger macro reshooting. Specifically, when a suspected defect is detected, that is, when the review confidence is greater than the reshoot threshold, the angle and intensity of the light source irradiating the component are adjusted, and the high-precision macro camera is forced to start to perform high-definition reshooting of the component. If no suspected defect is detected, that is, when the review confidence is less than or equal to the reshoot threshold, step S11 is repeated.

[0025] Furthermore, in step S2, the template production specifically includes the following steps:

[0026] Step S21: Angle variation, collecting standard part edge templates of all component models, rotating the standard part edge templates, and generating multi-angle templates;

[0027] Step S22: Dimensional variants are used to adapt to dimensional differences caused by stamping and thermal deformation, scaling the edge template of the standard part in proportion to generate a multi-size template;

[0028] Step S23: Deformation template: Repeat step S1 to create a rough mold of the standard part edge template with marked material parameters. Through finite element analysis, simulate the mechanical deformation of the rough mold caused by the punching pressure and clamping force, and generate the edge deformation template of the component after being compressed. According to the temperature field distribution of the component, the thermal deformation template caused by thermal expansion is obtained;

[0029] Step S24: Lighting variation: for reflection and shadow scenes, synthesize light and shadow variation templates under different lighting conditions, and record the combination of multi-angle template, multi-size template, edge deformation template, thermal deformation template and light and shadow variation template as a variation template.

[0030] Furthermore, in step S4, the hierarchical matching specifically includes the following steps:

[0031] Step S41: pre-processing the template library, reducing the resolution of all variant templates in the template library, and downsampling layer by layer;

[0032] Step S42: Coarse matching, used to filter out obvious mismatches. Specifically, when matching, the component to be located is first compared with each representative template, and binary matching is performed on the structured data of the component to be located. Irrelevant representative templates are quickly filtered out at low resolution to determine the two most similar representative templates.

[0033] Step S43: Fine matching is used for fine matching within the cluster. Specifically, the component to be located is finely matched with other variant templates in the cluster to which the two representative templates closest to it belong in turn at high resolution. The point cloud registration algorithm is used to align the scanned point cloud data of the component to be located with the variant template before downsampling with high precision, and the component model is located to obtain a matching score. The matching score refers to the possibility that the component to be located belongs to any component model, and is expressed using the Hausdorff distance.

[0034] The present invention provides a component surface positioning method based on machine vision. The beneficial effects achieved by the present invention using the above scheme are as follows:

[0035] (1) In view of the technical problem that the existing technology mainly targets parts with obvious surface defects that can be captured by cameras, and it is difficult to identify subtle defects and defects in complex backgrounds, this solution introduces infrared sensors and laser scanners to achieve multimodal data fusion, extract high-temperature abnormal areas and surface geometric abnormalities, and utilizes the advantages of low power consumption of ordinary cameras and high imaging quality of high-precision macro cameras. It determines whether to trigger the high-precision macro camera for high-definition reshooting based on the confidence level, thus achieving the technical effect of intelligent positioning of parts with both high precision and low cost;

[0036] (2) In view of the technical problems in the existing technology, when different part materials, lighting conditions or surface textures change, fixed thresholds may lead to false detection or missed detection, and the parameters are highly dependent. In addition, high noise environments such as metal surface reflections and scratches affect the screening of similar edge areas. This solution uses a template library to achieve hierarchical matching of parts models, quantify the defect probability, and only use local key features for template matching. It optimizes computing resources through clustering, improves detection efficiency, and achieves the technical effect of part model positioning in complex industrial environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic flow chart of a component surface positioning method based on machine vision provided by the present invention;

[0038] Figure 2 is a schematic diagram of step S1;

[0039] Figure 3 is a schematic diagram of step S2;

[0040] Figure 4 is a schematic diagram of step S4.

[0041] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] In the description of the present invention, it should be understood that terms such as "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, they should not be understood as limiting the present invention.

[0044] Example 1, see Figures 1 to 3 The present invention provides a component surface positioning method based on machine vision, which includes the following steps:

[0045] Step S1: multimodal data fusion;

[0046] Step S2: Template production, obtaining variant templates, and recording the variant templates of all component models as a template library;

[0047] Step S3: extracting local features, specifically, for each model variant template, extracting local key features that are different from other models' variant templates and storing them as structured data;

[0048] Step S4: Template clustering and hierarchical matching: clustering the template library using a clustering algorithm. The variant template closest to the cluster center in each cluster is used as the representative template, and the matching score is obtained through hierarchical matching.

[0049] Step S5: Exception handling, specifically, calculating the dispersion of the matching scores corresponding to all component models, presetting a matching threshold, and if the dispersion exceeds the matching threshold, the matching fails. For each component to be located that fails to match, a warning is triggered and it is automatically added to the template library as a new template for manual review and judgment. If the dispersion does not exceed the matching threshold, the matching is successful, and the component model with the highest matching score is output.

[0050] Example 2, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In step S3, the hypergraph is simplified, specifically including the following steps:

[0051] In step S1, the multimodal data fusion specifically includes the following steps:

[0052] Step S11: Initial defect screening, specifically, using an infrared sensor to detect fine cracks on parts and using a laser scanner to supplement surface scratches and concave-convex information on parts, to obtain scanning point cloud data and thermal radiation distribution maps respectively;

[0053] Step S12: rough modeling, specifically, using a low-power wide-angle camera to shoot the parts, and generating a rough mold based on the shooting effect. The rough mold represents a rough three-dimensional virtual model, and the scanning point cloud data and thermal radiation distribution map are mapped to the rough mold;

[0054] Step S13: Temperature anomaly extraction, used to quantify the probability of component defects. Specifically, the high temperature abnormal area of ​​the component is extracted through threshold segmentation, and the confidence of the thermal radiation distribution map is calculated. The formula used is as follows:

[0055]

[0056] Where IRs represents the confidence level of the thermal radiation distribution map, {map} represents the total surface area of ​​the component, {map|T_p>T_avg+2σ} represents the area of ​​the abnormal region of the component, σ represents the temperature standard deviation of the component, T_avg represents the average temperature of the component, T_p represents the component temperature of any pixel in the thermal radiation distribution map, σ_d represents the temperature standard deviation in the abnormal region, and max(ΔT_d) represents the maximum temperature difference in the abnormal region.

[0057] Step S14: Geometric anomaly extraction is used to quantify the probability of geometric anomalies on the component surface and calculate the laser confidence. The formula used is as follows:

[0058] Ls=avg(sigmoid(Δz_j)+C_j);

[0059] Where Ls represents the laser confidence, avg() represents the mean function, sigmoid() represents the activation function, C_j represents the curvature of any point in the point cloud, and Δz_j represents the difference between the depth of any point in the point cloud and the average depth of the neighboring points;

[0060] Step S15: Analyze the confidence of each item, preset the retake threshold, assign weights to the thermal radiation distribution map confidence and laser confidence, and calculate the verification confidence. The formula used is as follows:

[0061] Cf=μ·IRs+(1-μ)·Ls;

[0062] Where Cf represents the verification confidence, IRs represents the thermal radiation distribution map confidence, Ls represents the laser confidence, and μ represents the contribution coefficient;

[0063] Step S16: High-definition review, used to trigger macro reshooting. Specifically, when a suspected defect is detected, that is, when the review confidence is greater than the reshoot threshold, the angle and intensity of the light source irradiating the component are adjusted, and the high-precision macro camera is forced to start to perform high-definition reshooting of the component. If no suspected defect is detected, that is, when the review confidence is less than or equal to the reshoot threshold, step S11 is repeated.

[0064] Example 3, see Figures 1 to 3 This embodiment is based on the above embodiment. In step S2, the template production specifically includes the following steps:

[0065] Step S21: Angle variation, collecting standard part edge templates of all component models, rotating the standard part edge templates, and generating multi-angle templates;

[0066] Step S22: Dimensional variants are used to adapt to dimensional differences caused by stamping and thermal deformation, scaling the edge template of the standard part in proportion to generate a multi-size template;

[0067] Step S23: Deformation template: Repeat step S1 to create a rough mold of the standard part edge template with marked material parameters. Through finite element analysis, simulate the mechanical deformation of the rough mold caused by the punching pressure and clamping force, and generate the edge deformation template of the component after being compressed. According to the temperature field distribution of the component, the thermal deformation template caused by thermal expansion is obtained;

[0068] Step S24: Lighting variation: for reflection and shadow scenes, synthesize light and shadow variation templates under different lighting conditions, and record the combination of multi-angle template, multi-size template, edge deformation template, thermal deformation template and light and shadow variation template as a variation template.

[0069] Example 4, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S4, the hierarchical matching specifically includes the following steps:

[0070] Step S41: pre-processing the template library, reducing the resolution of all variant templates in the template library, and downsampling layer by layer;

[0071] Step S42: Coarse matching, used to filter out obvious mismatches. Specifically, when matching, the component to be located is first compared with each representative template, and binary matching is performed on the structured data of the component to be located. Irrelevant representative templates are quickly filtered out at low resolution to determine the two most similar representative templates.

[0072] Step S43: Fine matching is used for fine matching within the cluster. Specifically, the component to be located is finely matched with other variant templates in the cluster to which the two representative templates closest to it belong in turn at high resolution. The point cloud registration algorithm is used to align the scanned point cloud data of the component to be located with the variant template before downsampling with high precision, and the component model is located to obtain a matching score. The matching score refers to the possibility that the component to be located belongs to any component model, and is expressed using the Hausdorff distance.

[0073] Example 5, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S12, the low-power wide-angle camera is a camera with a resolution lower than 1280x720.

[0074] Example 6, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S13, the high-precision macro camera is a CCD industrial camera.

[0075] Example 7, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S23, the material parameters include elastic modulus and Poisson's ratio.

[0076] Example 8, see Figures 1 to 4 ,This embodiment is based on the above embodiment. In step S4, the K-means clustering algorithm is used to cluster the template library into 20 groups.

[0077] Example 9, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S15, μ represents the contribution coefficient and is set to 0.6.

[0078] Example 10, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S4, the standard part edge template is rotated 5° clockwise and 5° counterclockwise in steps of 18°.

[0079] Example 11, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S4, the matching score operation code is as follows:

[0080] defedge_matching_score(edges_test,edges_template):

[0081] #Calculate the matching score between the edge of the part to be tested and the edge of the template through the Hausdorff distance

[0082] #edges_test(numpy.ndarray): represents the binary edge image of the part to be tested (non-zero pixels represent edges)

[0083] #edges_template(numpy.ndarray): Represents the binary edge image of the template (non-zero pixels represent edges)

[0084] pts_test=np.argwhere(edges_test>0).astype(np.float32)#represents the point cloud of the part to be tested

[0085] pts_template = np.argwhere(edges_template>0).astype(np.float32) # template point cloud

[0086] dist_test_to_template=directed_hausdorff(pts_test,pts_template)

[0087] dist_template_to_test = directed_hausdorff(pts_template, pts_test) #Hausdorff distance is the maximum of the two-way distance to ensure symmetry

[0088] max_dist=max(dist_test_to_template,dist_template_to_test)

[0089] #Normalization

[0090] normalization_factor=np.sqrt(edges_template.shape[0]**2+edges_template.shape[1]**2)

[0091] #Calculate score

[0092] score=1-min(max_dist / normalization_factor,1.0).

[0093] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0094] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0095] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. A component surface positioning method based on machine vision, characterized by: The method comprises the following steps: Step S1: multimodal data fusion; Step S2: Template production, obtaining variant templates, and recording the variant templates of all component models as a template library; Step S3: extracting local features, specifically, for each model variant template, extracting local key features that are different from other models' variant templates and storing them as structured data; Step S4: Template clustering and hierarchical matching: clustering the template library using a clustering algorithm. The variant template closest to the cluster center in each cluster is used as the representative template, and the matching score is obtained through hierarchical matching. Step S5: Exception handling, specifically, calculating the dispersion of the matching scores corresponding to all component models, presetting a matching threshold, and if the dispersion exceeds the matching threshold, the matching fails. For each component to be located that fails to match, a warning is triggered and it is automatically added to the template library as a new template for manual review and judgment. If the dispersion does not exceed the matching threshold, the matching is successful, and the component model with the highest matching score is output.

2. The component surface positioning method based on machine vision according to claim 1, characterized in that: In step S1, the multimodal data fusion specifically includes the following steps: Step S11: Initial defect screening, specifically, using an infrared sensor to detect fine cracks on parts and using a laser scanner to supplement surface scratches and concave-convex information on parts, to obtain scanning point cloud data and thermal radiation distribution maps respectively; Step S12: rough modeling, specifically, using a low-power wide-angle camera to shoot the parts, generating a rough model based on the shooting effect, and mapping the scanned point cloud data and thermal radiation distribution map to the rough model; Step S13: Temperature anomaly extraction, specifically, extracting abnormally high temperature areas of components through threshold segmentation and calculating the confidence of the thermal radiation distribution map; Step S14: extracting geometric anomalies and calculating laser confidence; Step S15: Analyze the confidence of each item, preset the retake threshold, assign weights to the thermal radiation distribution map confidence and the laser confidence, and calculate the verification confidence; Step S16: High-definition review, used to trigger macro reshooting. Specifically, when a suspected defect is detected, that is, when the review confidence is greater than the reshoot threshold, the angle and intensity of the light source irradiating the component are adjusted, and the high-precision macro camera is forced to start to perform high-definition reshooting of the component. If no suspected defect is detected, that is, when the review confidence is less than or equal to the reshoot threshold, step S11 is repeated.

3. The component surface positioning method based on machine vision according to claim 2, characterized in that: In step S2, the template production specifically includes the following steps: Step S21: Angle variation, collecting standard part edge templates of all component models, rotating the standard part edge templates, and generating multi-angle templates; Step S22: Dimensional variants are used to adapt to dimensional differences caused by stamping and thermal deformation, scaling the edge template of the standard part in proportion to generate a multi-size template; Step S23: Deformation template: Repeat step S1 to create a rough mold of the standard part edge template with marked material parameters. Through finite element analysis, simulate the mechanical deformation of the rough mold caused by the punching pressure and clamping force, and generate the edge deformation template of the component after being compressed. According to the temperature field distribution of the component, the thermal deformation template caused by thermal expansion is obtained; Step S24: Lighting variation: for reflection and shadow scenes, synthesize light and shadow variation templates under different lighting conditions, and record the combination of multi-angle template, multi-size template, edge deformation template, thermal deformation template and light and shadow variation template as a variation template.

4. The component surface positioning method based on machine vision according to claim 3, characterized in that: In step S4, the hierarchical matching specifically includes the following steps: Step S41: pre-processing the template library, reducing the resolution of all variant templates in the template library, and downsampling layer by layer; Step S42: Coarse matching, used to filter out obvious mismatches. Specifically, when matching, the component to be located is first compared with each representative template, and binary matching is performed on the structured data of the component to be located. Irrelevant representative templates are quickly filtered out at low resolution to determine the two most similar representative templates. Step S43: Fine matching is used for fine matching within the cluster. Specifically, the component to be located is finely matched with other variant templates in the cluster to which the two representative templates closest to it belong in turn at high resolution. The point cloud registration algorithm is used to align the scanned point cloud data of the component to be located with the variant template before downsampling with high precision, and the component model is located to obtain a matching score. The matching score refers to the possibility that the component to be located belongs to any component model, and is expressed using the Hausdorff distance.

Images