A Method for Surface Defect Recognition of Mechanical Parts Based on Multimodal Vision Fusion

By fusing visible light, polarization, and structured light data, a reference surface and processing texture are constructed. An improved CMX model is used for feature correction, which solves the problems of false detection and missed detection in the detection of surface defects of mechanical parts using multimodal data, and improves the accuracy and stability of identification.

CN123118640APending Publication Date: 2026-11-17重庆精渝田科技股份有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611619570.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing methods for detecting surface defects in mechanical parts, the spatial correspondence of multimodal data, the reference for normal machining textures, and the normal surface morphology are not fully utilized. This leads to normal machining textures being easily misidentified as linear defects, shallow scratches and micro-cracks being missed, and the identification boundaries between different defect types being unclear.

Method used

By fusing visible light, polarization, and structured light data, a reference surface and reference processing texture are constructed. An improved CMX model is used for physical deviation-driven feature correction and defect discrimination. Visual features are extracted and correction features are generated by spatially aligned multimodal data to identify surface defects.

Benefits of technology

It improves the accuracy and stability of surface defect identification, reduces the false detection rate, enhances the ability to identify shallow defects, and can effectively distinguish between normal machining marks, shallow scratches, and micro-cracks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118640A_ABST
    Figure CN123118640A_ABST
Patent Text Reader

Abstract

This invention discloses a method for identifying surface defects in mechanical parts based on multimodal visual fusion, belonging to the field of mechanical surface inspection technology. The method includes the following steps: acquiring visible light images, polarization images, and structured light data of the mechanical part to be inspected and qualified mechanical parts; spatially aligning these to generate aligned multimodal data; establishing a reference surface, reference machining texture, and normal polarization state based on the qualified data; calculating height deviation and polarization deviation; extracting linear abnormal regions and generating machining texture change results; constructing an improved CMX; correcting visual features based on height deviation, polarization deviation, and machining texture change results; fusing the corrected features to complete defect identification and secondary discrimination of linear abnormalities; determining the defect region, category, and location based on the correction results, and generating surface defect identification results. This invention achieves accurate identification of multimodal mechanical part defects and improves the detection accuracy of shallow defects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of mechanical surface inspection technology, and in particular to a method for identifying surface defects in mechanical parts based on multimodal visual fusion. Background Technology

[0002] During processing, assembly, and use, mechanical parts can develop defects such as scratches, cracks, pits, indentations, burrs, and protrusions. Current surface defect detection methods mostly use visible light images for texture and edge recognition. Some technologies also incorporate polarization imaging and structured light 3D measurement to assist in judging surface reflection status and geometric shape, and use deep learning models to complete multimodal feature fusion and defect classification.

[0003] Existing methods fail to adequately utilize the spatial correspondence of multi-source data, the benchmark of normal processing textures, and normal surface morphology. Different modal features often rely on direct fusion based on internal network correlations. Normal processing textures are easily identified as linear defects, while shallow scratches and microcracks exhibit weak height variations and inconspicuous structured light responses. Visible light texture anomalies, polarization state changes, and surface height deviations lack a unified positional correlation and pre-fusion correction mechanism, which can easily lead to false detection of normal processing textures, missed detection of shallow defects, and unclear identification boundaries between different defect types.

[0004] Therefore, how to provide a method for identifying surface defects in mechanical parts based on multimodal visual fusion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a method for identifying surface defects in mechanical parts based on multimodal visual fusion. This invention integrates visible light, polarization, and structured light data to construct a reference surface and reference machining texture, and uses an improved CMX model for feature correction and defect discrimination driven by physical deviation. It has the advantages of low false detection rate, strong ability to identify shallow defects, and high detection accuracy.

[0006] A method for identifying surface defects in mechanical parts based on multimodal visual fusion according to an embodiment of the present invention includes the following steps:

[0007] Visible light images, polarization images, and structured light data of the mechanical parts to be inspected and qualified mechanical parts are collected. Data corresponding to the same surface position are spatially aligned to generate aligned multimodal data.

[0008] Based on the alignment multimodal data of qualified mechanical parts, a reference surface and reference machining texture are established, the normal of the reference surface is determined, and the correspondence between the normal of the reference surface and the normal polarization state is established.

[0009] Structured light data and polarization images are read from the aligned multimodal data of the mechanical part to be inspected. The height deviation relative to the reference surface is calculated based on the reference surface normal, and the polarization deviation is calculated based on the normal polarization state.

[0010] Visible light images are read from the aligned multimodal data of the mechanical part to be inspected, linear abnormal regions are extracted, and the linear abnormal regions are compared with reference machining textures to generate machining texture variation results.

[0011] An improved CMX model is constructed, visual features of aligned multimodal data are extracted, feature correction coefficients are generated based on height deviation, polarization deviation and processing texture changes, and visual features are corrected using the feature correction coefficients to generate corrected features;

[0012] The surface defect identification is performed by integrating correction features. The identified linear abnormal areas are then distinguished as normal machining marks, shallow scratches, and micro-cracks to correct the surface defect identification results.

[0013] Based on the corrected surface defect identification results, the defect area, defect type, and defect location are determined, and the surface defect identification results of the mechanical parts are generated.

[0014] Optionally, the spatial alignment process includes:

[0015] The coordinate transformation relationship is determined based on the calibration parameters of each acquisition device, and the surface coordinates of the mechanical parts are established based on the structured light data.

[0016] By utilizing coordinate transformation relationships, the pixel positions in visible light images and polarized images are converted to the surface coordinates of mechanical parts, thereby determining the surface position corresponding to each pixel position;

[0017] Based on the surface coordinates of the mechanical parts, data corresponding to the same surface position in visible light images, polarization images, and structured light data are correlated to generate aligned multimodal data.

[0018] Optionally, the process of establishing the reference surface and reference machining texture includes:

[0019] Select several mechanical parts with the same model and surface finish that have passed inspection;

[0020] Based on the surface coordinates of the mechanical parts, outlier values ​​and median values ​​are removed from the structured light 3D coordinates of the same surface positions of each qualified mechanical part, and a reference surface is established based on the median coordinates of each surface position.

[0021] Based on the visible light images corresponding to each qualified mechanical part, the edges of the machining texture are extracted. According to the extension direction of the machining texture edges and the arrangement position of adjacent machining texture edges, the edges of the machining texture that appear repeatedly are filtered to generate reference machining textures.

[0022] Based on the surface coordinates of the mechanical part, the reference machining texture is associated with the corresponding surface position in the reference surface.

[0023] Optionally, the process of establishing the correspondence between the reference surface normal and the normal polarization state includes:

[0024] Based on the surface coordinates of the mechanical part, select the target surface position on the reference surface, and extract the median value of the coordinates around the target surface position as the neighborhood point;

[0025] Perform local plane fitting on the neighborhood points, determine the reference surface normal of the target surface position based on the normal direction of the fitted plane, and unify the orientation of the reference surface normal according to the surface coordinates of the mechanical parts.

[0026] The polarization image corresponding to the target surface position is read from the alignment multimodal data of each qualified mechanical part, and the light intensity values ​​corresponding to the polarization directions of 0°, 45°, 90° and 135° are extracted. The linear polarization degree and polarization angle of each qualified mechanical part at the target surface position are calculated.

[0027] Outlier values ​​were removed and median values ​​were calculated for the linear polarization degree and polarization angle corresponding to each qualified mechanical part. The normal polarization state of the target surface position was determined based on the calculation results.

[0028] Using the reference surface normal at the target surface position as an index, a correspondence is established between the reference surface normal and the normal polarization state of the target surface position, and each surface position of the reference surface is processed sequentially.

[0029] Optionally, the process of calculating the height deviation relative to the reference surface includes:

[0030] Find the reference surface point corresponding to the current surface point in the reference surface, and read the reference surface normal corresponding to the reference surface point;

[0031] Calculate the position difference vector between the current surface point and the reference surface point, project the position difference vector onto the reference surface normal, and generate the initial height deviation of the current surface point.

[0032] The initial height deviation of the current surface point and the adjacent surface position are read according to the surface coordinates of the mechanical part. The initial height deviation that is continuous with the adjacent surface position and has the same sign is retained, and the isolated initial height deviation is discarded.

[0033] The retained initial height deviation is defined as the height deviation, the height deviation along the normal direction of the reference surface is defined as the positive height deviation, and the height deviation along the opposite direction of the normal direction of the reference surface is defined as the negative height deviation.

[0034] Optionally, the process of calculating the polarization deviation based on the normal polarization state includes:

[0035] Obtain the current degree of linear polarization and the current polarization angle at each surface position as the current polarization state;

[0036] Match the current polarization state with the normal polarization state according to the surface coordinates of the mechanical part, calculate the difference between the current linear polarization degree and the normal linear polarization degree, and determine the minimum angle difference between the current polarization angle and the normal polarization angle according to the 180° cycle.

[0037] The difference and the minimum angular difference are correlated with the corresponding surface positions to generate polarization deviation.

[0038] Optionally, the process of generating the processing texture variation result includes:

[0039] The edge connected regions are extracted from the visible light image of the mechanical part to be inspected, and the edge connected regions whose length-to-width ratio reaches a preset condition are identified as linear abnormal regions.

[0040] Based on the surface coordinates of the mechanical part, read the linear abnormal area and the edge of the reference machining texture corresponding to the adjacent surface position from the reference machining texture;

[0041] Calculate the difference in the extension direction between the edge of the machining texture and the corresponding reference machining texture within the linear anomaly region, and generate the direction comparison result;

[0042] Calculate the spacing between the edges of adjacent machining marks on both sides of the linear anomaly region, compare it with the spacing between the corresponding reference machining mark edges, and generate the spacing comparison result.

[0043] The connection status of the machining texture edges within the linear abnormal area is checked along the extension direction of the reference machining texture edge to determine the interruption, misalignment, and missing positions of the machining texture edges, and to generate continuous state comparison results.

[0044] The direction comparison results, spacing comparison results, and continuous state comparison results are correlated with the corresponding linear abnormal areas according to the surface coordinates of the mechanical part to generate machining texture change results.

[0045] Optionally, the improved CMX model includes a visible light feature extraction branch, a polarization feature extraction branch, a structured light feature extraction branch, a feature correction unit, a feature fusion unit, a defect identification branch, and a linear defect branch;

[0046] The visible light feature extraction branch reads the visible light image from the aligned multimodal data and extracts visible light features;

[0047] The polarization feature extraction branch reads the polarization image from the aligned multimodal data and extracts polarization features;

[0048] The structured light feature extraction branch reads structured light data from the aligned multimodal data and extracts structured light features;

[0049] The feature correction unit receives visible light features, polarization features, and structured light features, and receives height deviation, polarization deviation, and processing texture variation results to generate correction features;

[0050] The feature fusion unit fuses correction features to generate fusion defect features;

[0051] The defect identification branch generates surface defect identification results based on the fused defect features;

[0052] The linear defect branch reads the fusion defect features corresponding to the linear abnormal area, distinguishes normal processing marks, shallow scratches and fine cracks in the linear abnormal area, and corrects the surface defect identification results based on the discrimination results.

[0053] Optionally, the process of generating the correction features includes:

[0054] When the result of the change in the processing texture does not reach the corresponding preset change threshold, and the polarization deviation and height deviation do not reach the corresponding preset threshold, a visible light feature correction coefficient of less than 1 is set according to the degree of change in the processing texture corresponding to the visible light feature.

[0055] When the change result of the processing texture reaches the corresponding preset change threshold and the polarization deviation reaches the preset polarization threshold, a visible light feature correction coefficient and a polarization feature correction coefficient greater than 1 are set according to the degree of processing texture change and polarization deviation, respectively.

[0056] When the height deviation reaches the preset height threshold, a structured light feature correction coefficient greater than 1 is set according to the height deviation.

[0057] The visible light feature, polarization feature, and structured light feature are multiplied by their respective feature correction coefficients to generate the correction feature.

[0058] Optionally, the process of correcting the surface defect identification results includes:

[0059] Linear abnormal areas where neither the processing texture change result nor the polarization deviation reaches the corresponding preset threshold are judged as normal processing textures.

[0060] The linear abnormal regions where the processing texture change results and polarization deviation reach the corresponding preset threshold are input into the linear defect branch. If the height deviation does not reach the preset height threshold, the linear abnormal regions are retained for further discrimination.

[0061] Determine whether the edges within the linear anomaly area are continuously connected, and determine whether there are two or more edges extending in different directions at the same surface location. Combine the results of processing texture changes, polarization deviation, and height deviation to generate shallow scratch discrimination results and micro-crack discrimination results.

[0062] The preliminary identification results of the corresponding surface locations are replaced with the identification results of normal machining marks, shallow scratches and fine cracks to generate corrected surface defect identification results.

[0063] The beneficial effects of this invention are:

[0064] (1) The present invention maps visible light images, polarization images and structured light data to the unified surface coordinates of mechanical parts, establishes reference surface, reference machining texture and normal polarization state, so that the detection information of different modes corresponds accurately at the same surface position, reducing false detection caused by data misalignment and normal machining texture.

[0065] (2) The present invention utilizes height deviation, polarization deviation and processing texture change results to directly generate feature correction coefficients, suppress normal processing texture interference before feature fusion, enhance the visual features corresponding to shallow anomalies and morphological anomalies, and improve the recognition ability of shallow scratches, fine cracks, pits, indentations, burrs and protrusions.

[0066] (3) The present invention sets up linear defect branches to perform secondary discrimination on linear abnormal areas. When the height change is not obvious, linear areas with processing texture changes and polarization abnormalities are retained, thus completing the distinction between normal processing textures, shallow scratches and fine cracks, and improving the accuracy and stability of surface defect identification. Attached Figure Description

[0067] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0068] Figure 1 This is an overall flowchart of a method for identifying surface defects in mechanical parts based on multimodal visual fusion proposed in this invention.

[0069] Figure 2 This is a pixel back projection diagram of a method for identifying surface defects in mechanical parts based on multimodal visual fusion proposed in this invention.

[0070] Figure 3 This is a schematic diagram of multimodal data alignment for a method for identifying surface defects in mechanical parts based on multimodal visual fusion proposed in this invention.

[0071] Figure 4 This is a schematic diagram of multi-sensor coordinate mapping for a method for identifying surface defects in mechanical parts based on multimodal visual fusion proposed in this invention.

[0072] Figure 5 This is the original CMX structure diagram of a method for identifying surface defects in mechanical parts based on multimodal visual fusion proposed in this invention;

[0073] Figure 6This is an improved CMX structure diagram of a method for identifying surface defects in mechanical parts based on multimodal visual fusion proposed in this invention.

[0074] Figure 7 This is a fine crack detection image of a method for identifying surface defects in mechanical parts based on multimodal visual fusion proposed in this invention.

[0075] Figure 8 This is a pit defect detection image of a surface defect recognition method for mechanical parts based on multimodal visual fusion proposed in this invention.

[0076] Figure 9 This is a normal machining texture detection image of a method for identifying surface defects in mechanical parts based on multimodal visual fusion proposed in this invention.

[0077] Figure 10 This is a shallow scratch detection image of a surface defect identification method for mechanical parts based on multimodal visual fusion proposed in this invention.

[0078] Figure 11 This is a linear crack detection image for a method for identifying surface defects in mechanical parts based on multimodal visual fusion proposed in this invention. Detailed Implementation

[0079] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0080] Example 1: Reference Figures 1-11 A method for identifying surface defects in mechanical parts based on multimodal visual fusion includes the following steps:

[0081] Visible light images, polarization images, and structured light data of the mechanical parts to be inspected and qualified mechanical parts are collected. Data corresponding to the same surface position are spatially aligned to generate aligned multimodal data.

[0082] Based on the alignment multimodal data of qualified mechanical parts, a reference surface and reference machining texture are established, the normal of the reference surface is determined, and the correspondence between the normal of the reference surface and the normal polarization state is established.

[0083] Structured light data and polarization images are read from the aligned multimodal data of the mechanical part to be inspected. The height deviation relative to the reference surface is calculated based on the reference surface normal, and the polarization deviation is calculated based on the normal polarization state.

[0084] Visible light images are read from the aligned multimodal data of the mechanical part to be inspected, linear abnormal regions are extracted, and the linear abnormal regions are compared with reference machining textures to generate machining texture variation results.

[0085] An improved CMX model is constructed, visual features of aligned multimodal data are extracted, feature correction coefficients are generated based on height deviation, polarization deviation and processing texture changes, and visual features are corrected using the feature correction coefficients to generate corrected features;

[0086] The surface defect identification is performed by integrating correction features. The identified linear abnormal areas are then distinguished as normal machining marks, shallow scratches, and micro-cracks to correct the surface defect identification results.

[0087] Based on the corrected surface defect identification results, the defect area, defect type, and defect location are determined, and the surface defect identification results of the mechanical parts are generated.

[0088] In this embodiment, visible light images, polarization images, and structured light data of the mechanical parts to be inspected and qualified mechanical parts are acquired, specifically as follows:

[0089] A positioning fixture is set up to fix the mechanical parts. The placement position of the mechanical parts in the positioning fixture is determined by the machining reference surface and the positioning reference edge of the mechanical parts, so that the mechanical parts to be inspected and the surfaces to be inspected of qualified mechanical parts are at the same acquisition position. A visible light camera, a polarization camera and a structured light acquisition device are fixedly set above the positioning fixture, and the installation position and acquisition direction of each acquisition device relative to the positioning fixture are kept unchanged.

[0090] Visible light images are acquired using an industrial camera in conjunction with a fixed illumination source. The imaging area of ​​the industrial camera covers the surface of the mechanical part to be inspected. After the mechanical part is positioned, the illumination direction of the light source, the working distance of the camera, the exposure time, and the focal length remain unchanged to acquire a visible light image that can characterize the grayscale changes, edge distribution, and processing texture of the surface to be inspected. The mechanical part to be inspected and qualified mechanical parts use the same visible light image acquisition parameters.

[0091] The polarization images are acquired using a polarization camera. The polarization camera obtains light intensity images of the surface to be inspected at polarization directions of 0°, 45°, 90°, and 135°. The light intensity images of the four polarization directions are matched according to pixel positions to generate polarization images. Based on the polarization images, the linear polarization degree and polarization angle corresponding to each surface position are determined to determine the normal polarization state and calculate the polarization deviation. The position of the polarization camera, the illumination direction, and the acquisition parameters are consistent between the mechanical parts to be inspected and qualified mechanical parts.

[0092] Structured light data is acquired using a structured light projector and a structured light camera. A preset structured light pattern is projected onto the surface of the mechanical part to be inspected. The deformation image formed by the structured light pattern on the surface to be inspected is obtained by the structured light camera. The three-dimensional coordinates of each sampling position are determined according to the calibration parameters of the structured light projector and the structured light camera, and structured light data characterizing the spatial morphology of the surface of the mechanical part to be inspected is generated. The mechanical part to be inspected and qualified mechanical parts use the same structured light projection method, acquisition position and reconstruction parameters.

[0093] For qualified mechanical parts, select mechanical parts with the same model, surface and processing technology as the mechanical parts to be tested, and whose surface inspection results are qualified. Install them on the positioning fixture according to the same positioning reference as the mechanical parts to be tested, and collect visible light images, polarization images and structured light data in sequence. Record the three types of data according to the mechanical part number, so that the visible light images, polarization images and structured light data corresponding to the same mechanical part can be associated for subsequent spatial alignment, reference surface establishment, reference processing texture extraction and normal polarization state determination.

[0094] In this embodiment, the spatial alignment process includes:

[0095] The coordinate transformation relationship is determined based on the calibration parameters of each acquisition device, and the surface coordinates of the mechanical parts are established based on the structured light data. Figure 4 middle, , and These represent the coordinate systems of the visible light camera, polarization camera, and structured light camera, respectively. Represents the coordinate system of the mechanical part surface. This indicates example sampling locations on the surface of the mechanical part. Each acquisition device determines its coordinate transformation relationship relative to the coordinate system of the mechanical part surface through calibration parameters. Figure 4 The projection beam and imaging beam in the diagram are used to represent the structured light projection and imaging process.

[0096] The calibration parameters for each acquisition device include the intrinsic and extrinsic parameters of the visible light camera, the intrinsic and extrinsic parameters of the polarization camera, and the calibration parameters of the structured light acquisition device.

[0097] The intrinsic parameters of visible light cameras and polarization cameras include focal length, principal point coordinates, and lens distortion parameters. The focal length includes the horizontal focal length and the vertical focal length. The principal point coordinates include the horizontal and vertical coordinates of the principal point in the imaging plane. The lens distortion parameters include radial distortion parameters and tangential distortion parameters. The extrinsic parameters of visible light cameras and polarization cameras include the rotation matrix and translation vector used to represent the camera coordinate system relative to the calibration reference coordinate system.

[0098] The calibration parameters of the structured light acquisition device include the intrinsic parameters of the structured light camera, the relative rotation matrix and relative translation vector between the structured light camera and the structured light projector, the projection intrinsic parameters of the structured light projector, and the rotation matrix and translation vector used to transform the 3D points in the structured light camera coordinate system to the calibration reference coordinate system or the coordinate system of the mechanical part surface. During calibration, a planar calibration plate with known feature point spacing is placed sequentially in the acquisition area jointly covered by the visible light camera, polarization camera, and structured light acquisition device. Images of the calibration plate are acquired at multiple different positions and orientations, and corresponding feature points in the calibration plate are extracted. Based on the known spatial positions and corresponding pixel positions of the corresponding feature points, the focal length, principal point position, and distortion parameters of the visible light camera and polarization camera are calculated respectively. The 3D coordinates corresponding to the structured light data are determined based on the calibration results between the structured light camera and the projector of the structured light acquisition device.

[0099] In one specific implementation, a 9×6 checkerboard calibration board with internal corner points and an adjacent corner point spacing of 10 mm is used. At least 10 sets of calibration board data with different orientations are collected within a common acquisition area. The intrinsic parameters and distortion parameters of the visible light camera and polarization camera are determined using the Zhang Zhengyou camera calibration method. The coordinate system of the structured light camera is used as the intermediate reference coordinate system for the calibration process. Based on the coordinate correspondence of the same set of calibration board feature points in the visible light camera, polarization camera, and structured light camera, rigid body transformation is used to solve for the rotation matrix and translation vector from the visible light camera coordinate system to the structured light camera coordinate system, as well as the rotation matrix and translation vector from the polarization camera coordinate system to the structured light camera coordinate system, thereby determining the coordinate transformation relationship. After calibration, the reprojection error of the calibration board feature points is calculated, and the corresponding calibration parameters are saved when the average reprojection error is no greater than 0.3 pixels.

[0100] When establishing the surface coordinates of mechanical parts based on structured light data, a three-dimensional point cloud of the surface to be inspected is reconstructed using a structured light acquisition device. The machining reference surface and the positioning reference intersecting with it are extracted from the three-dimensional point cloud. Plane fitting is performed based on the point cloud within the machining reference surface. The normal direction of the fitted plane is set as the Z-axis direction of the mechanical part's surface coordinates, and the extension direction of the positioning reference within the machining reference surface is set as the X-axis direction. The Y-axis direction is determined based on the cross product of the Z-axis and X-axis directions. The preset intersection point of the machining reference surface and the positioning reference is set as the origin of the mechanical part's surface coordinates. For both the mechanical part to be inspected and qualified mechanical parts, the same machining reference surface and positioning reference are used to establish the surface coordinates, ensuring consistent coordinate representation for the same designed surface positions of different mechanical parts.

[0101] The three-dimensional point cloud output by the structured light acquisition device is initially located in the structured light camera coordinate system. Based on the origin of the mechanical part surface coordinates and the three coordinate axis directions, the rotation matrix and translation vector from the structured light camera coordinate system to the mechanical part surface coordinates are obtained. Each three-dimensional point in the structured light data is transformed to the mechanical part surface coordinates, and the transformed three-dimensional points represent the surface position corresponding to the structured light data.

[0102] By utilizing coordinate transformation relationships, the pixel positions in visible light images and polarized images are converted to the surface coordinates of mechanical parts, thereby determining the surface position corresponding to each pixel position; Figure 2 middle, This represents the visible light camera coordinate system. Indicates the coordinate system of the polarization camera. and These represent the pixel coordinates in the visible light image and the polarization image, respectively. and This indicates the intersection point between the corresponding back-projected ray and the reference surface. Indicates transformation to the coordinate system of the mechanical part surface. The symbols used to represent the three-dimensional surface positions are used to illustrate the process of converting two-dimensional pixels into three-dimensional surface coordinates through back projection and surface intersection.

[0103] The pixel position transformation process is as follows: distortion correction is performed on the pixel position in the visible light image; the corrected two-dimensional pixel position is back-projected into a spatial ray in the visible light camera coordinate system according to the intrinsic parameters of the visible light camera; the spatial ray is transformed into the structured light camera coordinate system according to the rotation matrix and translation vector from the visible light camera coordinate system to the structured light camera coordinate system; and the spatial ray is transformed into the surface coordinate of the mechanical part according to the rotation matrix and translation vector from the structured light camera coordinate system to the surface coordinate of the mechanical part.

[0104] Based on the 3D point cloud formed from structured light data, a neighborhood search is performed on the local point cloud traversed by the current spatial ray. The local surface is then fitted with neighborhood points to determine the intersection points of the spatial ray and the local surface. The 3D coordinates of these intersection points in the surface coordinate system of the mechanical part are used to determine the surface position corresponding to the visible light pixel location. If a triangular mesh has already been constructed from the 3D point cloud, the intersection points of the spatial ray and the triangular mesh are directly calculated, and the positive intersection point closest to the camera's optical center is determined as the corresponding surface position.

[0105] The polarized image is transformed using the same method. Distortion correction is applied to the pixel positions in the polarized image. A spatial ray is formed based on the intrinsic parameters of the polarization camera. Then, using the coordinate transformation relationships from the polarization camera coordinate system to the structured light camera coordinate system, and from the structured light camera coordinate system to the surface coordinates of the mechanical part, the spatial ray is transformed to the surface coordinates of the mechanical part. Finally, the surface position corresponding to each pixel position in the polarized image is determined based on the intersection of the spatial ray and the corresponding surface in the structured light data. Therefore, the pixel positions in the visible light image, the pixel positions in the polarized image, and the three-dimensional points in the structured light data are all represented using the surface coordinates of the mechanical part.

[0106] Based on the surface coordinates of the mechanical parts, data corresponding to the same surface position in visible light images, polarization images, and structured light data are correlated to generate aligned multimodal data. Figure 3 middle, and These represent the visible light pixel coordinates and the polarization pixel coordinates, respectively. This represents the three-dimensional surface points corresponding to the structured light data. Indicates the target matching position on the surface of the mechanical part. Indicates the location of other surfaces. The symbol indicates the positional tolerance used when performing multimodal data association. Circles, triangles, and squares represent visible light pixel candidates, polarized pixel candidates, and structured light point candidates, respectively. Each symbol is used to illustrate the multimodal data matching relationship at the same surface location.

[0107] Using the three-dimensional points in the structured light data as the surface position reference, the surface positions obtained by converting the visible light image and the polarization image are searched in the surface coordinates of the mechanical part, respectively. The spatial distance between the surface position corresponding to the structured light data and the surface position corresponding to the visible light image, as well as the spatial distance between the surface position corresponding to the structured light data and the surface position corresponding to the polarization image are calculated.

[0108] When the spatial distance between the surface position corresponding to the visible light image and the surface position corresponding to the structured light data is not greater than the preset position tolerance, the corresponding visible light pixel data is associated with the structured light data; when the spatial distance between the surface position corresponding to the polarization image and the surface position corresponding to the structured light data is not greater than the preset position tolerance, the corresponding polarization pixel data is associated with the structured light data; when there are multiple candidate surface positions, the surface position with the smallest spatial distance is selected to establish the association.

[0109] In one specific implementation, a preset position tolerance is set to 0.2 mm based on the detection size of the surface defects of the mechanical part and the spacing of the structured light point cloud. When the spacing between adjacent points of the structured light data is less than 0.2 mm, the corresponding surface positions of visible light pixels and polarized pixels are searched within a 0.2 mm range centered on the structured light 3D point, and the data with the smallest spatial distance is selected to complete the matching. For visible light data and polarized data that do not directly fall into the position of the structured light 3D point, linear interpolation is performed using the data of adjacent surface positions, and the interpolation result is associated with the corresponding structured light 3D point.

[0110] After the association is completed, the corresponding visible light pixel data, polarization pixel data and structured light three-dimensional coordinates are recorded together at the same mechanical part surface position, so that each surface position has three types of data that correspond to each other. The three types of data are organized according to the arrangement order of the mechanical part surface coordinates to generate aligned multimodal data. The aligned multimodal data is used for subsequent reference surface establishment, height deviation calculation, polarization deviation calculation and linear anomaly region extraction.

[0111] In this embodiment, the process of establishing the reference surface and reference machining texture includes:

[0112] Select several mechanical parts with the same model and surface processing method that have passed inspection, and establish reference surfaces and reference processing textures based on aligned multimodal data;

[0113] Based on the surface coordinates of the mechanical parts, outlier values ​​and median values ​​are removed from the structured light 3D coordinates of the same surface positions of each qualified mechanical part, and a reference surface is established based on the median coordinates of each surface position.

[0114] Specifically, each qualified mechanical part is divided into the same surface position according to the surface coordinates of the mechanical parts. The structured light three-dimensional coordinates corresponding to each qualified mechanical part are read at each surface position. For the i-th surface position, the coordinate components of each structured light three-dimensional coordinate in the X-axis, Y-axis and Z-axis directions are arranged according to their numerical values. The median of the three coordinate components is taken to form the initial median coordinates of the i-th surface position.

[0115] Calculate the Euclidean distance between the structured light 3D coordinates and the initial median coordinates of each qualified mechanical part at the i-th surface position. Calculate the median of the Euclidean distances. Calculate the median of the absolute differences between each Euclidean distance and the median distance to obtain the median absolute deviation. Structured light 3D coordinates with Euclidean distances greater than the sum of the median distance and three times the median absolute deviation are identified as outliers and removed. When the median absolute deviation is zero, structured light 3D coordinates with Euclidean distances greater than 0.05 mm are identified as outliers and removed.

[0116] After outlier removal, the medians of the X, Y, and Z axis coordinate components of the remaining structured light 3D coordinates are recalculated. The three recalculated coordinate components are used to form the median value of the i-th surface position. All surface positions of the mechanical part to be inspected are processed in sequence to obtain a set of reference points composed of multiple coordinate medians.

[0117] Based on the spatial adjacency relationship between the median values ​​of each coordinate in the coordinate system of the mechanical part, adjacent median values ​​of the reference point set are connected. Triangular patches are constructed between adjacent median values, allowing them to share the corresponding boundary median values. All these triangular patches are then continuously connected to form the reference surface. For locations lacking median values, supplementary points are generated through linear interpolation based on the surrounding median values. These supplementary points are then connected to the reference surface along with their adjacent median values. The resulting reference surface maintains the actual curved surface shape of the mechanical part's normal surface, avoiding the fitting of the entire surface to be inspected as a single plane.

[0118] Based on the visible light images corresponding to each qualified mechanical part, the edges of the machining texture are extracted. According to the extension direction of the machining texture edges and the arrangement position of adjacent machining texture edges, the edges of the machining texture that appear repeatedly are filtered to generate reference machining textures.

[0119] Read the visible light images of each qualified mechanical part at the same surface position, calculate the gray-level gradient of the image, extract edge pixels based on the local maxima of the gray-level gradient, connect the edge pixels that are continuous in direction and spatially adjacent to form the edge of the processing texture; record the position, extension direction, length and spacing between adjacent processing texture edges according to the surface coordinates of the mechanical part.

[0120] Matching is performed on the edges of machining marks extracted from the same surface position of different qualified mechanical parts. Machining marks edges with an average positional distance between them not greater than 0.15 mm, an angle between their extension directions not greater than 5°, a relative difference between the spacing of adjacent machining marks not greater than 10%, and an overlap length of not less than 70% of the shorter machining mark edge length are identified as corresponding machining mark edges.

[0121] Count the number of times corresponding machining edge appears in all qualified mechanical parts. When corresponding machining edge appears repeatedly in no less than 70% of qualified mechanical parts, the corresponding machining edge is retained; when the occurrence rate is less than 70%, the corresponding machining edge is removed from the normal machining edge.

[0122] For the retained machining texture edges, the corresponding machining texture edges in different qualified mechanical parts are sampled at equal intervals according to the surface coordinates of the mechanical parts. The median value of the corresponding edge point coordinates is calculated at each sampling position. The median values ​​of each edge point coordinate are connected according to the extension order of the machining texture to form the corresponding reference machining texture edge. All reference machining texture edges are organized according to the arrangement order and adjacent position relationship of each reference machining texture edge to generate the reference machining texture.

[0123] Based on the surface coordinates of the mechanical part, the reference machining pattern is associated with the corresponding surface position on the reference surface. Specifically, the median value of the coordinates or the triangular facet containing the same surface coordinates is found on the reference surface, and each edge point in the edge of the reference machining pattern is associated with the corresponding surface position on the reference surface, so that the position, extension direction, and arrangement of the reference machining pattern all have a corresponding reference surface position.

[0124] In this embodiment, the process of establishing the correspondence between the reference surface normal and the normal polarization state includes:

[0125] Based on the surface coordinates of the mechanical part, select the target surface position on the reference surface, and extract the median value of the coordinates around the target surface position as the neighborhood point;

[0126] The target surface location is any location on the reference surface to be processed, represented by three-dimensional coordinates in the surface coordinate system of the mechanical part. Centered on the target surface location, the median value of adjacent coordinates is searched on the reference surface according to a preset neighborhood radius. The median value falling within the preset neighborhood radius is taken as a neighborhood point. The preset neighborhood radius is determined based on the spacing between adjacent median values ​​on the reference surface. For example, if the average spacing between adjacent median values ​​on the reference surface is 0.1 mm, the preset neighborhood radius is set to 0.5 mm, ensuring that each target surface location has at least 12 neighborhood points. If the number of neighborhood points is insufficient, neighborhood points are added according to their spatial distance from the target surface location, from closest to furthest.

[0127] Perform local plane fitting on the neighborhood points, determine the reference surface normal of the target surface position based on the normal direction of the fitted plane, and unify the orientation of the reference surface normal according to the surface coordinates of the mechanical parts.

[0128] In this embodiment, local plane fitting is performed using the least squares plane fitting method. For N neighborhood points corresponding to the target surface position, the three-dimensional coordinates of each neighborhood point in the coordinate system of the mechanical part surface are read, and the mean of the three-dimensional coordinates of the N neighborhood points is calculated to obtain the neighborhood point center. The coordinate difference of each neighborhood point relative to the neighborhood point center is calculated, and a three-dimensional covariance matrix is ​​constructed based on all coordinate differences. Eigenvalue decomposition is performed on the three-dimensional covariance matrix, and the unit eigenvector corresponding to the smallest eigenvalue is selected as the normal vector of the fitting plane. The local fitting plane passing through the neighborhood point center and the normal vector is determined. The local fitting plane is used as an approximation of the local tangent plane of the reference surface near the target surface position, and the normal vector of the local fitting plane is determined as the initial reference surface normal of the target surface position.

[0129] In this method, local plane fitting is used instead of global plane fitting of the complete reference surface, so that different surface positions in the arc surface, cylindrical surface and continuous curved surface can obtain the corresponding local normal direction.

[0130] Since two opposite unit vectors of the same fitting plane can both serve as normal vectors in a mathematical sense, it is necessary to unify the orientation of the initial reference surface normal. When establishing the surface coordinates of the mechanical part, the positive direction of the Z-axis is set as the reference direction pointing from the mechanical part entity to the outside of the surface to be inspected. The Z-axis unit vector of the mechanical part surface coordinates is denoted as... Calculate the initial reference surface normal and The dot product is calculated; when the dot product is greater than or equal to 0, the initial reference surface normal remains unchanged; when the dot product is less than 0, all three components of the initial reference surface normal are inverted. The unit normal vector after unifying the orientation is used as the reference surface normal for the target surface position. Therefore, the normals at each surface position of the reference surface all point to the outer side of the mechanical part's surface to be inspected, and adjacent positions will not have opposite normal directions.

[0131] The polarization image corresponding to the target surface position is read from the alignment multimodal data of each qualified mechanical part, and the light intensity values ​​corresponding to the polarization directions of 0°, 45°, 90° and 135° are extracted. The linear polarization degree and polarization angle of each qualified mechanical part at the target surface position are calculated.

[0132] The light intensity value is the digital response value obtained by a polarization camera after photoelectric conversion of the incident light irradiance intensity in the corresponding polarization direction. When using a 12-bit polarization camera, the original response range is 0–4095. Before calculation, a dark-field image under shading conditions is acquired and the dark-field response is subtracted from the original response value. Then, a flat-field image acquired under uniform illumination conditions is used to correct the response differences of different pixels. The pixel response value after dark-field and flat-field correction is used as the light intensity value. The light intensity values ​​in the four polarization directions have a unified response benchmark.

[0133] The polarization image consists of polarization data generated by a polarization camera in four polarization directions: 0°, 45°, 90°, and 135°. For a qualified mechanical part that has already undergone spatial alignment, the corresponding polarization pixel position is read from the alignment multimodal data based on the coordinates of the target surface position in the surface coordinates of the mechanical part. The pixel response values ​​of the polarization pixel positions in the 0°, 45°, 90°, and 135° polarization directions are read respectively and denoted as I0, I45, I90, and I135.

[0134] Stokes parameters are calculated based on the light intensity values ​​in four polarization directions, where S0 = I0 + I90, S1 = I0 - I90, and S2 = I45 - I135. The degree of linear polarization ρ and the polarization angle φ are calculated based on S0, S1, and S2. The degree of linear polarization characterizes the proportion of the polarized component at the current surface position relative to the total light intensity, and the polarization angle characterizes the direction of linear polarization. The degree of linear polarization ranges from 0 to 1, and the polarization angle is uniformly expressed within the range of 0° to 180° according to a 180° period. For each qualified mechanical part, the above calculations are performed at the same target surface position to obtain the corresponding degree of linear polarization and polarization angle for each qualified mechanical part.

[0135] Outlier values ​​were removed and median values ​​were calculated for the linear polarization degree and polarization angle corresponding to each qualified mechanical part. The normal polarization state of the target surface position was determined based on the calculation results.

[0136] For the same target surface location, the linear polarization degrees obtained from each qualified mechanical part are used to form a linear polarization degree sample set. The median and median absolute deviation of the linear polarization degree sample set are calculated. For any linear polarization degree sample, the absolute difference between the linear polarization degree sample and the median of the linear polarization degree is calculated. When the absolute difference is greater than three times the median absolute deviation, the corresponding linear polarization degree sample is removed as an outlier. The median of the remaining linear polarization degree samples is recalculated, and the recalculated median is determined as the normal linear polarization degree of the target surface location.

[0137] Polarization angles have a 180° periodicity, and ordinary angle differences cannot be directly used to handle polarization angles close to 0° and 180°. When performing outlier removal and median calculation for polarization angles, all polarization angles are uniformly converted to the range of 0° to 180°. The angle difference between any two polarization angles is defined as the smaller of their direct angle difference and 180° minus the direct angle difference. The deviation of each polarization angle from its position in the sample set is calculated based on the angle difference, and polarization angles with deviations exceeding a preset outlier range are removed. For the retained polarization angles, the circumferential median is used to determine the normal polarization angle, that is, the angle that minimizes the sum of the periodic angle differences with all retained polarization angles within the range of 0° to 180° is selected as the normal polarization angle.

[0138] In one specific implementation, when the polarization angles of qualified mechanical parts corresponding to a certain target surface position are 178°, 179°, 1°, 2° and 0° respectively, the polarization angles are not considered to be distributed in two far apart angle intervals of 178° and 1°, but are determined to be concentrated around 0° based on the periodicity of 180°, thereby avoiding the polarization direction error caused by ordinary arithmetic position values.

[0139] The normal linear polarization degree and normal polarization angle at the target surface location are combined to form the normal polarization state of the target surface location. The normal polarization state is recorded as binary data, where the first data item is the normal linear polarization degree and the second data item is the normal polarization angle. The normal polarization state represents the normal polarization response of a qualified mechanical part at the target surface location, assuming that the mechanical part model, surface processing method, acquisition location, and lighting conditions remain consistent.

[0140] Using the reference surface normal at the target surface position as an index, a correspondence is established between the reference surface normal and the normal polarization state of the target surface position, and each surface position of the reference surface is processed sequentially.

[0141] The specific steps for establishing the correspondence are as follows: for each surface position in the reference surface, a corresponding record is created. The corresponding record shall at least record the three-dimensional coordinates of the surface position in the surface coordinates of the mechanical part, the three directional components of the reference surface normal, the normal linear polarization degree, and the normal polarization angle; the three directional components of the reference surface normal shall be used as the index field of the normal polarization state, and the three-dimensional coordinates of the surface position shall be used as the position association field.

[0142] For the i-th surface position, the corresponding record can be represented as "surface position, reference surface normal, normal linear polarization degree, and normal polarization angle". The same processing is performed on all surface positions of the reference surface sequentially to obtain the correspondence between the reference surface normal and the normal polarization state, consisting of multiple corresponding records.

[0143] When processing the mechanical parts to be inspected, the current position of the surface to be inspected is determined according to the surface coordinates of the mechanical parts. The normal of the reference surface at the same position is read. The normal linear polarization degree and normal polarization angle corresponding to the reference surface normal are obtained according to the corresponding records. These are used to compare with the current linear polarization degree and current polarization angle at the same position of the mechanical parts to be inspected.

[0144] In actual storage, the surface coordinates of the mechanical part and the normal of the reference surface are saved simultaneously to avoid index confusion when different positions of the reference surface have the same or similar reference surface normals. The surface position is used to locate the corresponding record, the reference surface normal is used to characterize the normal geometric orientation of the surface position, and the normal polarization state is used as the normal optical state corresponding to the reference surface normal.

[0145] In this embodiment, the process of calculating the height deviation relative to the reference surface includes:

[0146] Structured light data is read from the aligned multimodal data of the mechanical part to be inspected, and the current surface point is determined according to the surface coordinates of the mechanical part. The current surface point is a three-dimensional point represented in the surface coordinates of the mechanical part after spatial alignment of the structured light data of the mechanical part to be inspected. The current surface point is represented by X-axis, Y-axis and Z-axis coordinates. The position of the current surface point is recorded according to the surface coordinates of the mechanical part so that the current surface point can be matched with the corresponding position in the reference surface.

[0147] Find the reference surface point corresponding to the current surface point in the reference surface, and read the reference surface normal corresponding to the reference surface point;

[0148] The reference surface point is determined based on the current surface point's position in the coordinate system of the mechanical part. When the current surface point falls on the median value of the reference surface's coordinates, that median value is used as the reference surface point. When the current surface point lies between adjacent median values ​​of the reference surface's coordinates, the triangular facet containing the current surface point is determined. Based on the projection position of the current surface point within the triangular facet, linear interpolation is performed on the coordinates of the three vertices of the triangular facet to obtain the reference surface point. This ensures that the reference surface point and the current surface point correspond to the same surface position on the mechanical part.

[0149] Calculate the position difference vector between the current surface point and the reference surface point, project the position difference vector onto the reference surface normal, and generate the initial height deviation of the current surface point.

[0150] Specifically, the X-axis, Y-axis, and Z-axis coordinates of the current surface point and the reference surface point are read from the coordinate system of the mechanical part. The three coordinate components of the current surface point are subtracted from the corresponding three coordinate components of the reference surface point to obtain the position difference in the X-axis, Y-axis, and Z-axis directions. These position difference vectors are then combined according to the order of the X, Y, and Z axes to form a position difference vector. The position difference vector starts at the reference surface point and ends at the current surface point, representing the spatial offset of the current surface of the mechanical part under test relative to the corresponding position on the reference surface.

[0151] In one specific embodiment, the coordinates of the mechanical part surface at the reference surface point are (20.000 mm, 15.000 mm, 6.000 mm), and the corresponding coordinates of the current surface point are (20.012 mm, 14.994 mm, 5.965 mm). The differences in these three coordinate directions are then calculated to obtain the three-dimensional position difference between the current position and the corresponding position on the reference surface. This three-dimensional position difference includes both the tangential offset along the reference surface and the normal offset perpendicular to the reference surface.

[0152] When projecting the position difference vector onto the reference surface normal, the reference surface normal is treated as a unit vector of length 1. The components of the position difference vector in the X, Y, and Z axes are multiplied by the components of the reference surface normal in the corresponding directions. The products are then added together, and the resulting signed value is used as the projection length of the position difference vector along the reference surface normal direction. This projection length is determined as the initial height deviation of the current surface point.

[0153] The above projection processing only retains the normal deviation component of the current surface point relative to the reference surface, and does not include the positional offset generated along the tangential direction of the reference surface in the initial height deviation. For cylindrical, curved, and other curved mechanical parts, different surface positions are projected using the corresponding reference surface normal, so that the initial height deviation of each position represents the local height change relative to the normal surface at the current position.

[0154] The reference surface normal is uniformly defined as the direction pointing from the inside of the mechanical part to the outside of the surface to be inspected. When the projection length is positive, it indicates that the current surface point is located outside the reference surface along the reference surface normal direction; when the projection length is negative, it indicates that the current surface point is located inside the reference surface along the opposite direction of the reference surface normal direction; when the projection length is close to zero, it indicates that the current surface point does not have a significant deviation in the reference surface normal direction.

[0155] In one specific implementation, the reference surface normal is close to the positive Z-axis direction of the mechanical part's surface coordinates. When the current surface point is offset inward by 0.035 mm relative to the reference surface point along the reference surface normal, the initial height deviation is approximately -0.035 mm; when the current surface point is offset outward by 0.042 mm along the reference surface normal, the initial height deviation is approximately 0.042 mm. For curved surfaces with an inclined reference surface normal, the same projection operation is performed according to the actual reference surface normal, and the Z-axis coordinate difference between the current surface point and the reference surface point is not directly used as the height deviation.

[0156] The initial height deviation of the current surface point and the adjacent surface position are read according to the surface coordinates of the mechanical part. The initial height deviation that is continuous with the adjacent surface position and has the same sign is retained, and the isolated initial height deviation is discarded.

[0157] The positions of adjacent surfaces are determined based on the adjacency relationship of triangular facets in the reference surface. For the current surface point, the initial height deviations corresponding to the adjacent triangular facets that share the boundary with the triangular facet containing the current surface point are read. When the initial height deviations of the current surface point and multiple adjacent surface positions have the same positive or negative sign, and the change between adjacent initial height deviations does not exceed a preset continuity threshold, the initial height deviation of the current surface point is retained.

[0158] When the initial height deviation of a current surface point reaches a preset abnormal range, and no initial height deviation in the same direction occurs at the adjacent surface positions, the current surface point is identified as an isolated abnormal point and removed. In one specific embodiment, when the spacing between adjacent structured light surface points is 0.1 mm, at least three spatially continuous surface points having the same height deviation sign are used as a continuity criterion to reduce the impact of isolated measurement noise on height deviation during structured light reconstruction.

[0159] The retained initial height deviation is defined as the height deviation, the height deviation along the normal direction of the reference surface is defined as the positive height deviation, and the height deviation along the opposite direction of the normal direction of the reference surface is defined as the negative height deviation.

[0160] Positive height deviation is used to characterize the local morphological changes of the surface of the mechanical part under test, which protrudes outward relative to the reference surface, while negative height deviation is used to characterize the local morphological changes of the surface of the mechanical part under test, which inward relative to the reference surface. The height deviation corresponding to each surface position is recorded according to the surface coordinates of the mechanical part and used for feature correction and surface defect identification in the improved CMX model.

[0161] In this embodiment, the process of calculating the polarization deviation based on the normal polarization state includes:

[0162] The current linear polarization degree and current polarization angle of each surface position are obtained from the aligned multimodal data of the mechanical part to be tested, and are used as the current polarization state.

[0163] Match the current polarization state with the normal polarization state according to the surface coordinates of the mechanical part, calculate the difference between the current linear polarization degree and the normal linear polarization degree, and determine the minimum angle difference between the current polarization angle and the normal polarization angle according to the 180° cycle.

[0164] The difference and minimum angle difference are associated with the corresponding surface position. Using the surface coordinates of the mechanical part as an index, the linear polarization degree difference and minimum angle difference corresponding to the same surface position are written into the data record of that surface position to generate polarization deviation.

[0165] In this embodiment, the process of generating the machining texture variation result includes:

[0166] The edge connected regions are extracted from the visible light image of the mechanical part to be inspected, and the edge connected regions whose length-to-width ratio reaches a preset condition are identified as linear abnormal regions.

[0167] An edge-connected region is a continuous region composed of edge pixels that are connected to each other within the pixel neighborhood after edge detection in a visible light image. Adjacent edge pixels are connected using an 8-neighborhood connectivity method, and edge pixels that can be reached from each other through consecutive adjacent edge pixels belong to the same edge-connected region.

[0168] For each edge-connected region, the main extension direction is determined based on the coordinates of all edge pixels within the region. The distance between the two farthest boundary points of the edge-connected region is calculated along this main extension direction as its length. The maximum distance between the two boundaries of the edge-connected region perpendicular to the main extension direction is calculated as its width. The ratio of length to width is then calculated. Specifically, a preset condition is set that the length-to-width ratio is not less than 3. In one specific implementation, the length of the edge-connected region is also required to be not less than 0.5 mm to eliminate short edge regions formed by isolated texture noise. Edge-connected regions that meet the preset conditions exhibit obvious linear extension characteristics, and these regions are identified as linear anomalous regions.

[0169] Based on the surface coordinates of the mechanical part, read the linear abnormal area and the edge of the reference machining texture corresponding to the adjacent surface position from the reference machining texture;

[0170] Calculate the difference in the extension direction between the edge of the machining texture and the corresponding reference machining texture within the linear anomaly region, and generate the direction comparison result;

[0171] The extension direction is determined by the principal direction analysis of edge coordinates. Each edge point of the machining texture edge in the linear abnormal area is transformed to the surface coordinates of the mechanical part. Linear fitting or principal component analysis is performed on each edge point, and the direction of the fitted line or the first principal component is determined as the extension direction of the current machining texture edge. The extension direction of the corresponding reference machining texture edge is determined in the same way.

[0172] Since the direction of the machining texture does not distinguish between forward and reverse, the angle between the current machining texture edge extension direction and the reference machining texture edge extension direction is uniformly converted to a range of 0° to 90°, and the minimum angle between the two is taken as the extension direction difference. When the extension direction difference is not greater than 5°, it is recorded as a direction-maintained state; when the extension direction difference is greater than 5°, the corresponding angular deviation is recorded, forming the direction comparison result. 5° is used to allow for small-range texture direction fluctuations that occur during normal processing, and in practical applications, it can be determined based on the discrete range of the machining texture direction of qualified mechanical parts.

[0173] Calculate the spacing between the edges of adjacent machining marks on both sides of the linear anomaly region, compare it with the spacing between the corresponding reference machining mark edges, and generate the spacing comparison result.

[0174] The spacing between the edges of the machining marks is calculated in a direction perpendicular to the extension direction of the reference machining mark edge. For the target surface location of a linear anomaly region, the two nearest machining mark edges on both sides of the target surface location are determined. The distances from the two machining mark edges to the target surface location are calculated in a direction perpendicular to the extension direction of the reference machining mark edge, and the sum of the two distances is taken as the current machining mark edge spacing.

[0175] Read the corresponding reference machining texture edges on both sides of the same surface position from the reference machining texture, and calculate the reference machining texture edge spacing in the same way. Subtract the reference machining texture edge spacing from the current machining texture edge spacing to obtain the spacing difference, and use the ratio of the absolute value of the spacing difference to the reference machining texture edge spacing as the spacing change ratio.

[0176] In one specific implementation, when the reference machining edge spacing is 0.20 mm and the current machining edge spacing is 0.24 mm, the spacing change ratio is 20%. When the spacing change ratio is no greater than 10%, it is recorded as a spacing-maintained state; when the spacing change ratio is greater than 10%, the corresponding spacing deviation is recorded, forming a spacing comparison result. Specifically, 10% can be determined based on the normal machining edge spacing fluctuation range of multiple qualified mechanical parts.

[0177] The connection status of the machining texture edges within the linear abnormal area is checked along the extension direction of the reference machining texture edge to determine the interruption, misalignment, and missing positions of the machining texture edges, and to generate continuous state comparison results.

[0178] The connection status check uses the edge of the reference processed texture as the position reference. Sampling positions are set sequentially along the extension direction of the edge of the reference processed texture. At each sampling position, a search section perpendicular to the edge of the reference processed texture is established. The current processed texture edge corresponding to the edge of the reference processed texture is searched in the visible light edge of the linear abnormal region.

[0179] When there is a continuous gap where no corresponding edge is detected between two adjacent matched processing texture edge segments, and the length of the gap segment along the extension direction of the reference processing texture is greater than the preset interruption length, the gap segment is determined as the interruption position; specifically, the preset interruption length can be set to 0.3 mm, or set to 3 times the average width of the current processing texture, whichever is larger.

[0180] If the edge of the current processed texture remains continuous, but the edge of the current processed texture is offset relative to the edge of the corresponding reference processed texture in a direction perpendicular to the extension of the reference processed texture, and the offset distance is greater than the preset position tolerance, the corresponding segment is determined as a misaligned position; the preset position tolerance is adapted to the spatial alignment accuracy of multimodal data, and is set to 0.15 mm in one specific embodiment.

[0181] When the reference machining texture edge exists at multiple consecutive sampling positions, but the current machining texture edge is not detected within the corresponding search section, and the length of the unmatched segment reaches the preset missing length, the corresponding segment is determined as a missing position; in one specific embodiment, the preset missing length is set to 0.5 mm. An interruption indicates a local break within the same machining texture edge, while a missing position indicates that the corresponding reference machining texture edge is not detected within a continuous surface segment, and the two are distinguished based on the continuous gap range.

[0182] Record each interrupted, misaligned, and missing position according to the surface coordinates of the mechanical part, and count the number, continuous length, and spatial distribution of each type of position in the linear abnormal area to generate a continuous state comparison result.

[0183] The direction comparison results, spacing comparison results, and continuous state comparison results are correlated with the corresponding linear abnormal areas according to the surface coordinates of the mechanical part to generate machining texture change results.

[0184] In this embodiment, the improved CMX model includes a visible light feature extraction branch, a polarization feature extraction branch, a structured light feature extraction branch, a feature correction unit, a feature fusion unit, a defect identification branch, and a linear defect branch;

[0185] The visible light feature extraction branch reads the visible light image from the aligned multimodal data and extracts visible light features;

[0186] The polarization feature extraction branch reads the polarization image from the aligned multimodal data and extracts polarization features;

[0187] The structured light feature extraction branch reads structured light data from the aligned multimodal data and extracts structured light features;

[0188] The feature correction unit receives visible light features, polarization features, and structured light features, and receives height deviation, polarization deviation, and processing texture variation results to generate correction features;

[0189] The feature fusion unit fuses correction features to generate fusion defect features;

[0190] Specifically, three types of correction features corresponding to the surface position of the same mechanical part are mapped to the same channel dimension. After alignment according to the surface position, the channels are stitched together. The correlation response between the three types of correction features is calculated through cross-attention. Then, the cross-attention output is channel mapped and feature compressed to generate fused defect features corresponding to the surface position. In one embodiment, the feature fusion unit is based on the Feature Fusion Module in the CMX model and extends the original dual-modal feature fusion to the fusion of three types of correction features: visible light, polarization, and structured light.

[0191] The defect identification branch generates surface defect identification results based on the fused defect features;

[0192] The fused defect features are channel-mapped and spatial resolution restored to correspond to the detection positions in the surface coordinates of the mechanical parts. For each surface position, the recognition score of each preset surface defect category is calculated. The category with the highest recognition score is determined as the preliminary recognition category of that surface position. Surface positions with the same defect category are merged to form corresponding defect regions, generating surface defect recognition results.

[0193] The linear defect branch reads the fused defect features corresponding to the linear anomaly region, and distinguishes between normal machining marks, shallow scratches, and microcracks in the linear anomaly region. Based on the discrimination results, the surface defect identification results are corrected. The linear defect branch is used to perform independent secondary discrimination on the identified linear anomaly regions. Specifically, fused defect features corresponding to the coverage position of the linear anomaly region are extracted according to the surface coordinates of the mechanical part. The fused defect features within the linear anomaly region are converged to generate linear region features. The linear region features, along with the corresponding machining mark changes, polarization deviation, and height deviation, are input into the linear defect branch. Discrimination scores for normal machining marks, shallow scratches, and microcracks are calculated respectively. The secondary discrimination result of the linear anomaly region is determined based on the discrimination scores.

[0194] In this embodiment, the process of generating correction features includes:

[0195] Based on the surface coordinates of the mechanical parts, establish a positional correspondence between visible light features, polarization features, and structured light features and the height deviation, polarization deviation, and machining texture changes of the corresponding surface positions;

[0196] When the result of the change in the processing texture does not reach the corresponding preset change threshold, and the polarization deviation and height deviation do not reach the corresponding preset threshold, a visible light feature correction coefficient of less than 1 is set according to the degree of change in the processing texture corresponding to the visible light feature.

[0197] Preset variation thresholds are used to limit the normal fluctuation range of machining marks on qualified mechanical parts. These preset variation thresholds include direction variation thresholds, spacing variation thresholds, interruption length thresholds, misalignment distance thresholds, and missing length thresholds. Direction comparison results, spacing comparison results, and continuous state comparison results are collected from the corresponding surface positions of multiple qualified mechanical parts. Normal sample distributions are established for each type of comparison quantity, and the corresponding preset variation thresholds are determined using the 95th percentile value of the comparison quantity corresponding to the normal sample.

[0198] In one specific implementation, the direction change threshold is set to... The spacing change threshold is set to 10%, and the interruption length threshold is set to... The misalignment distance threshold is set to The missing length threshold is set to The above values ​​serve as specific implementation parameters, used to distinguish between texture fluctuations caused by normal processing and texture changes that exceed normal processing conditions.

[0199] To ensure that variations in processing texture with different dimensions can be uniformly included in feature correction, each variation in processing texture is divided by its corresponding preset variation threshold, and the maximum value of the resulting ratio is taken as the degree of variation in processing texture. :

[0200] ;

[0201] in, To extend the difference in direction, This represents the percentage change in spacing. For the interrupt length, The misalignment distance. The length of the missing data; , , , , These are the corresponding preset change thresholds. This indicates that the variation in each processing texture is within the normal range. This indicates that at least one of the processing marks is outside the normal range.

[0202] For example, the difference in the direction of extension is When the spacing change ratio is 15% and all other continuous state comparison values ​​are 0, the corresponding normalized results are 0.6, 1.5, 0, 0, and 0, respectively, indicating the degree of change in the machining texture. We take 1.5. Therefore, even if the linear abnormal area is close to the direction of the reference machining texture, as long as the spacing or continuity changes significantly, it can still be determined that the machining texture has changed abnormally.

[0203] When the degree of change in processing texture, polarization, and height are all less than 1, the visible light characteristic correction coefficient corresponding to the normal processing texture is limited to between 0.7 and 0.9, and the specific value is determined according to the degree of change in processing texture.

[0204] ;

[0205] in, This is the visible light characteristic correction coefficient. When... hour, ;when As it gradually approaches 1, It gradually approaches 0.9. This process reduces the linear visible light response formed by normal machining textures while preserving the grayscale, edge, and local texture information at the corresponding locations.

[0206] When the change result of the processing texture reaches the corresponding preset change threshold and the polarization deviation reaches the preset polarization threshold, a visible light feature correction coefficient and a polarization feature correction coefficient greater than 1 are set according to the degree of processing texture change and polarization deviation, respectively.

[0207] The preset polarization thresholds include a preset linear polarization degree threshold and a preset polarization angle threshold. Based on the normal polarization states of the same surface positions of multiple qualified mechanical parts, the difference in linear polarization degree and the minimum angle difference are statistically analyzed. The 95th percentile of the normal linear polarization degree difference distribution is determined as the preset linear polarization degree threshold, and the 95th percentile of the normal minimum angle difference distribution is determined as the preset polarization angle threshold.

[0208] In one specific embodiment, the preset linear polarization degree threshold is set to 0.08, and the preset polarization angle threshold is set to... Divide the current linear polarization degree difference by a preset linear polarization degree threshold, and divide the minimum angle difference by a preset polarization angle threshold. Take the larger of the two ratios as the degree of polarization change. . This indicates that the current polarization state is within the normal polarization fluctuation range. This indicates that the current polarization state has changed abnormally compared to the normal polarization state.

[0209] For example, the current linear polarization degree difference is 0.12, and the minimum angular difference is... At that time, the corresponding ratios were 1.5 and 0.75, respectively, indicating the degree of polarization change. The value is set at 1.5.

[0210] For visible light features, polarization features, and structured light features that require enhancement, a unified method for determining the enhancement feature correction coefficients is adopted. Let the corresponding degree of change be... ,when When, the corresponding characteristic correction coefficient Determined according to the following formula:

[0211] ;

[0212] Among them, the visible light features correspond to Degree of variation in machining texture Polarization characteristics corresponding to Take the degree of polarization change Structured light features corresponding to Take the degree of height change Therefore, when the degree of change is 1.2, the corresponding feature correction coefficient is 1.10; when the degree of change is 1.5, the corresponding feature correction coefficient is 1.25; when the degree of change reaches 2, the corresponding feature correction coefficient is 1.50; and when the degree of change exceeds 2, the corresponding feature correction coefficient remains at 1.50.

[0213] After adopting the above calculation method, the feature correction coefficient greater than 1 is uniquely determined by the degree to which the actual detection result exceeds the normal threshold; the greater the degree of change, the higher the degree of visual feature enhancement. The upper limit of the feature correction coefficient is set to 1.50 to limit the excessive influence of a single modality feature on the subsequent fusion result.

[0214] When the degree of change in the machining texture And the degree of polarization change At that time, respectively and Substituting the above-described method for calculating enhanced feature correction coefficients, we obtain the visible light feature correction coefficients and polarization feature correction coefficients. This process enhances linear anomalous regions supported by both processing texture anomalies and polarization anomalies before feature fusion, ensuring that visible light and polarization features are preserved and enhanced even if no significant height change occurs at the corresponding location.

[0215] When the height deviation reaches the preset height threshold, a structured light feature correction coefficient greater than 1 is set according to the height deviation.

[0216] A preset height threshold is used to distinguish between normal surface undulations and actual surface morphology changes of qualified mechanical parts. The absolute values ​​of height deviations at the same surface position of multiple qualified mechanical parts are collected, and the 95th percentile of the absolute value of the normal height deviation is calculated. The axial measurement accuracy of the structured light acquisition device is read, and the 95th percentile of the absolute value of the normal height deviation is compared with twice the axial measurement accuracy; the larger value is taken as the preset height threshold.

[0217] In one specific embodiment, the axial measurement accuracy of the structured light acquisition device is: The 95th percentile of the absolute value of the normal height deviation of qualified mechanical parts is Twice the axial measurement accuracy Then The preset height threshold has been determined.

[0218] Divide the absolute value of the height deviation of the current surface position by the preset height threshold to obtain the degree of height change. ;in, This represents the height deviation of the current surface position. This is a preset height threshold.

[0219] This indicates that the current surface height change is within the normal surface fluctuation range. This indicates that the current surface location has undergone morphological changes that exceed the normal range. The sign of the height deviation is retained to distinguish between positive and negative height deviations, and the magnitude of the characteristic correction coefficient is determined based on the absolute value of the height deviation.

[0220] when At that time, the degree of height change Substituting the values ​​into the above-described method for calculating enhanced feature correction coefficients, we obtain the structured light feature correction coefficients. For example, the preset height threshold is... The absolute value of the height deviation is hour, The structured light feature correction coefficient is 1.10; the absolute value of the height deviation is... hour, The structured light feature correction coefficient is 1.25; the absolute value of the height deviation reaches... hour, The structured light feature correction factor is 1.50.

[0221] Therefore, when the changes in the processing texture are not obvious but the structured light data shows that there are continuous changes in surface morphology, the structured light features at the corresponding positions can still be enhanced, so that the geometric features corresponding to pits, indentations, burrs and protrusions will not be weakened in the subsequent fusion process due to the weak visible light texture response.

[0222] The visible light feature, polarization feature, and structured light feature are multiplied by their respective feature correction coefficients to generate the correction feature.

[0223] In this embodiment, the process of correcting the surface defect identification results includes:

[0224] Linear abnormal areas where neither the processing texture change result nor the polarization deviation reaches the corresponding preset threshold are judged as normal processing textures.

[0225] The linear abnormal regions where the processing texture change results and polarization deviation reach the corresponding preset threshold are input into the linear defect branch. If the height deviation does not reach the preset height threshold, the linear abnormal regions are retained for further discrimination.

[0226] Based on the characteristics of fusion defects, determine whether the edges in the linear abnormal area are continuously connected, determine whether there are two or more edges extending in different directions at the same surface location, and combine the results of processing texture changes, polarization deviation and height deviation to generate shallow scratch discrimination results and fine crack discrimination results;

[0227] Specifically, an eight-neighbor connectivity search is performed on the edge pixels within the linear anomaly region, and edges that can be continuously reached through adjacent edge pixels are grouped into the same edge connectivity region; the interval between adjacent edge segments is checked along the edge extension direction, and adjacent edge segments are connected when the interval is not greater than the preset connection distance.

[0228] For each edge segment passing through the same local neighborhood, fit the extension direction and calculate the direction angle between any two edge segments; when at least two edge segments intersect or connect in the same local neighborhood and the direction angle reaches the preset direction difference, it is determined that there are two or more edges extending in different directions.

[0229] When generating shallow scratch and fine crack discrimination results, the shallow scratch discrimination result is determined when the processing texture change result reaches the preset change threshold, the polarization deviation reaches the preset polarization threshold, and the linear abnormal area mainly forms a unidirectional continuous edge; the fine crack discrimination result is determined when the processing texture change result has interrupted or missing processing texture edges, the polarization deviation reaches the preset polarization threshold, and the linear abnormal area has two or more edges extending in different directions; the height deviation is used to supplement the surface morphology change information. If the height deviation does not reach the preset height threshold, the shallow scratch discrimination result or fine crack discrimination result already supported by the processing texture change and polarization anomaly is not cancelled.

[0230] The preliminary identification results of the corresponding surface locations are replaced with the identification results of normal machining marks, shallow scratches and fine cracks to generate corrected surface defect identification results.

[0231] Example 2: To verify the feasibility of this invention in practice, it was applied to the surface quality inspection of a batch of precision-machined mechanical parts. The parts to be inspected included bushings and flanges, and the surfaces included planes, cylindrical surfaces, and partially transitional surfaces. After machining, the parts retained relatively stable turning and grinding marks. Defect types included shallow scratches, microcracks, pits, indentations, burrs, and protrusions. In actual inspection, normal machining marks also appeared as continuous lines in visible light images. Some machining marks were close to the direction of shallow scratches, making it easy to identify normal machining marks as scratches based solely on grayscale and edge information. Some shallow scratches had depths below the range of significant morphological changes, and the height changes in structured light data were not prominent, making it easy to miss detection based solely on three-dimensional morphology. This example focuses on verifying the invention's ability to distinguish between normal machining marks, shallow scratches, and microcracks, and also verifies its ability to identify the morphology of pits, indentations, burrs, and protrusions.

[0232] The testing setup includes a visible light camera, a polarization camera, and a structured light acquisition device. The mechanical part to be tested is mounted on a fixed positioning fixture. The visible light camera acquires surface images with a resolution of 2048×2048 pixels. The polarization camera acquires images corresponding to polarization directions of 0°, 45°, 90°, and 135°. The structured light acquisition device generates a 3D point cloud of the mechanical part's surface, with a point spacing of 0.1 mm and an axial measurement accuracy of 0.01 mm. All three acquisition devices maintain consistent installation positions, working distances, exposure parameters, and light source directions during testing. Qualified mechanical parts and the mechanical parts to be tested are clamped using the same positioning reference to ensure a unified surface position reference for each batch of data.

[0233] Before testing, a calibration board was used to jointly calibrate the visible light camera, polarization camera, and structured light acquisition device. Based on the calibration results, the coordinate transformation relationships between each acquisition device were determined. The structured light 3D point cloud was used to establish the surface coordinates of the mechanical part based on the processing reference surface and the positioning reference. Pixel positions in the visible light and polarization images were transformed to the surface coordinates of the mechanical part after distortion correction and ray back projection. A correlation was established between visible light pixels, polarization pixels, and structured light 3D points within the same surface position tolerance range. In this embodiment, the position matching tolerance was set to 0.2 mm. When the distance met the tolerance condition, the data with the smallest 3D distance was selected to establish a correspondence, generating aligned multimodal data.

[0234] Fifty qualified mechanical parts of the same model, material, and surface processing method were selected to establish a normal surface reference. For the surface coordinate positions of the same mechanical parts, the structured light 3D coordinates corresponding to the 50 qualified mechanical parts were read. Coordinates that significantly deviated from the normal distribution were removed, and the median value of the remaining coordinates was calculated. The median values ​​of the coordinates at each position were connected to form a reference surface. After edge extraction, the visible light image was used to record the position, extension direction, spacing, and continuous length of the processing texture edges. Repeated processing texture edges appearing in multiple qualified mechanical parts were retained to form reference processing textures. The tested normal processing texture direction fluctuations were mainly concentrated within 5°, and the spacing variation ratio was mainly concentrated within 10%. In this embodiment, 5° was set as the direction change threshold, 10% as the spacing change threshold, 0.3 mm as the interruption length threshold, 0.15 mm as the misalignment distance threshold, and 0.5 mm as the missing length threshold.

[0235] The reference surface is divided into local neighborhoods according to the surface coordinates of the mechanical parts. Local plane fitting is performed on the neighborhood points, and the normal of the fitted plane is determined as the normal of the reference surface at the corresponding position. Polarization images of the same position of 50 qualified mechanical parts are used to calculate the degree of linear polarization and the polarization angle. After outlier removal and median calculation, the normal polarization state is obtained. The statistical results of the normal samples show that the 95th percentile of the degree of linear polarization difference is 0.08, and the 95th percentile of the minimum angle difference is close to 8°. In this embodiment, 0.08 and 8° are set as preset thresholds for degree of linear polarization and preset thresholds for polarization angle, respectively. The statistical analysis of the normal height fluctuation of structured light shows that the 95th percentile of the absolute value of the height deviation is 0.025 mm, which is higher than twice the axial measurement accuracy of 0.02 mm of the structured light acquisition device. In this embodiment, 0.025 mm is set as the preset height threshold.

[0236] After the mechanical part to be inspected enters the inspection position, the current surface point in the structured light data is matched with the corresponding point on the reference surface. The position difference vector between the current surface point and the reference surface point is calculated, and then the position difference vector is projected onto the normal of the reference surface to obtain the height deviation. Continuous height deviations that shift outward from the reference surface are recorded as positive height deviations, and continuous height deviations that shift inward from the reference surface are recorded as negative height deviations. Isolated single-point height changes are not directly used as topographic anomalies; only when consecutive adjacent positions show height changes of the same sign do they enter the structured light feature correction process. This processing can reduce the interference of local noise in the structured light point cloud on the identification of pits, indentations, burrs, and protrusions.

[0237] The polarization image is used to read the current linear polarization degree and current polarization angle according to the surface coordinates of the mechanical part. This is compared with the normal polarization state at the same location to obtain the difference in linear polarization degree and the minimum angle difference. After edge extraction, the visible light image forms edge-connected regions. Edge-connected regions whose length-to-width ratio meets the conditions for linear regions are recorded as linear anomalous regions. These linear anomalous regions are compared with reference machining marks at the same surface location to calculate the difference in extension direction and the percentage change in spacing, and to check for interruptions, misalignments, and missing machining marks. In a certain shallow scratch test area, the difference in extension direction is only 3.2°, very close to the direction of normal machining marks. The percentage change in spacing reaches 17.6%, the difference in linear polarization degree is 0.116, the minimum angle difference is 10.4°, and the absolute value of the height deviation is only 0.018 mm. Based solely on direction and height information, this area would easily be classified as a normal machining mark. However, this invention retains this linear anomalous region for shallow scratch and micro-crack differentiation based on the spacing change and polarization anomaly.

[0238] The improved CMX model extracts visible light features, polarization features, and structured light features from aligned multimodal data. The degree of variation in processing texture, polarization, and height is normalized according to the ratio of the detected value to the corresponding threshold. When all three detection results for normal processing texture locations are within the normal range, the visible light feature correction coefficient is set between 0.7 and 0.9 to reduce the strong linear response formed by regular processing textures. When the degree of variation in processing texture and polarization exceeds 1, the visible light and polarization features are enhanced using correction coefficients greater than 1. When the degree of height variation exceeds 1, the structured light features are enhanced using correction coefficients greater than 1. The enhanced correction coefficient is linearly mapped from 1 to 2 to 1 to 1.5 according to the degree of variation, and remains at 1.5 after the degree of variation reaches 2. Thus, the physical detection results directly change the feature expression intensity of the corresponding mode before feature fusion.

[0239] The processing texture variation in a shallow scratch region is 1.42, the polarization variation is 1.36, and the height variation is 0.72. The calculated visible light feature correction coefficient is 1.21, the polarization feature correction coefficient is 1.18, and the structured light feature correction coefficient remains at 1. The model did not delete the shallow scratch region due to the weak height variation. The processing texture variation in a fine crack region is 1.78, the polarization variation is 1.64, and the height variation is 1.12, with corresponding visible light, polarization, and structured light feature correction coefficients of 1.39, 1.32, and 1.06, respectively. The crack location simultaneously exhibits processing texture interruption, polarization anomaly, and local morphological changes, and all three types of features participate in subsequent fusion. The processing texture variation in a normal processing texture region is 0.31, the polarization variation is 0.44, and the height variation is 0.36. The visible light feature correction coefficient is 0.762, the polarization and structured light features remain at their original values, and the visible light ray response generated by the normal processing texture is suppressed.

[0240] The three types of visual features, after correction, are input into the feature fusion unit. Channel mapping and cross-feature fusion are performed according to the corresponding surface positions to generate fused defect features. The defect recognition branch generates preliminary surface defect recognition results based on the fused defect features. The fused defect features corresponding to the linear anomaly region are then input into the linear defect branch, which reads the processing texture changes, polarization deviation, and height deviation at the same surface position. Linear regions where both processing texture changes and polarization deviations are within the normal range are identified as normal processing textures. Linear regions where processing texture changes and polarization deviations exceed the corresponding thresholds continue to be processed for shallow scratches and fine cracks. Linear anomaly regions are retained when the height deviation is less than 0.025 mm. When the processing texture edge shows obvious interruption or loss, and forms edges extending in different directions at local locations, a fine crack discrimination result is generated by combining polarization deviation and height deviation. Linear regions that do not meet the fine crack discrimination conditions but have processing texture changes and polarization anomalies generate shallow scratch discrimination results. The secondary discrimination result replaces the preliminary recognition result at the same surface position. Figure 9 The end detection result is a binary defect mask, where black pixels represent non-defect areas and white pixels represent defect areas; normal processing textures do not form defect areas after being identified by this invention, and the pixel values ​​in the corresponding binary defect mask are all 0, displayed as black.

[0241] The validation data consisted of independent training and testing data. The training data contained 3600 surface regions, the validation data contained 600 surface regions, and the testing data contained 1200 surface regions. The mechanical parts corresponding to these three sets of data were not repeated. The testing data included 300 normal machining texture regions, 220 shallow scratch regions, 180 microcrack regions, 140 pit regions, 120 indentation regions, 120 burr regions, and 120 protrusion regions. The results of manual microscopic observation and structured light 3D contour verification were used together as test labels. To observe the actual changes resulting from the improved structure, tests were conducted on the same test data using a single visible light recognition method, the original CMX visible light and polarization dual-mode method, a three-mode direct fusion method, and the method of this invention.

[0242] Single visible light recognition methods generate numerous linear false positives in areas with normal machining textures, with a false positive rate of 12.7% for normal machining textures, a recall rate of 82.3% for shallow scratches, and a recall rate of 84.4% for microcracks. After adding polarization information to the original CMX, the false positive rate for normal machining textures decreased to 8.4%, the recall rate for shallow scratches increased to 88.6%, and the recall rate for microcracks increased to 90.0%. The three-modal direct fusion method, after incorporating structured light data, improves the recognition ability of pits, indentations, burrs, and protrusions, but confusion still exists between normal machining textures and shallow linear defects. Using the method of this invention, the false positive rate for normal machining textures decreased to 2.8%, the recall rate for shallow scratches reached 95.9%, the recall rate for microcracks reached 96.1%, the average recall rate for morphological defects reached 97.0%, and the overall recognition accuracy reached 96.4%. Test results show that referencing machining marks and normal polarization states can reduce misjudgments of normal machining marks, feature correction driven by physical deviations can enhance the corresponding features of real defects, and secondary discrimination of linear defects can retain shallow scratches and fine cracks with weak height changes but abnormal machining marks and polarization states.

[0243] Table 1: Comparison of Surface Defect Identification Results for Mechanical Parts

[0244] The comparison results in Table 1 show that the overall accuracy of the single visible light recognition method is 88.2%, the false detection rate of normal processing textures is 12.7%, and the recall rates for shallow scratches and fine cracks are 82.3% and 84.4%, respectively. This indicates that when relying solely on visible light texture and edge information, normal processing textures are easily confused with real linear defects. After introducing polarization information into the original CMX dual-modal method, the overall accuracy improved to 91.6%, the false detection rate of normal processing textures decreased to 8.4%, and the recall rates for shallow scratches and fine cracks increased to 88.6% and 90.0%, respectively. This demonstrates that polarization information can supplement the description of surface reflection state changes in visible light images.

[0245] After further incorporating structured light data, the three-modal direct fusion method achieved an overall accuracy of 93.2%, with the average recall rate for morphology-related defects increasing to 95.1%, indicating that structured light data can enhance the surface morphology information corresponding to pits, indentations, burrs, and protrusions. However, the method still had a false positive rate of 6.1% for normal machining marks and a recall rate of 90.9% for shallow scratches, suggesting that simply increasing the number of modalities and directly fusing them is insufficient to fully address the interference from normal machining marks and the lack of significant height variation in shallow linear defects.

[0246] After adopting the method of this invention, the overall accuracy was improved to 96.4%, the overall precision and overall recall reached 96.1% and 96.3%, respectively, the false detection rate of normal machining marks was reduced to 2.8%, the recall rates of shallow scratches and fine cracks reached 95.9% and 96.1%, respectively, and the average recall rate of morphology-related defects reached 97.0%. These results demonstrate that this invention establishes a normal state benchmark by referencing machining marks, normal polarization states, and a reference surface, and uses machining mark variations, polarization deviations, and height deviations to correct different modal features before feature fusion. This can suppress interference features corresponding to normal machining marks, retain shallow scratches and fine cracks with weak height variations, and enhance structured light features with significant morphological changes. After secondary discrimination of linear defect branches in linear abnormal regions, false alarms of normal machining marks are further reduced, improving the identification ability of shallow linear defects.

[0247] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-modal visual fusion based mechanical part surface defect recognition method, characterized in that, Includes the following steps: Visible light images, polarization images, and structured light data of the mechanical parts to be inspected and qualified mechanical parts are collected. Data corresponding to the same surface position are spatially aligned to generate aligned multimodal data. Based on the alignment multimodal data of qualified mechanical parts, a reference surface and reference machining texture are established, the normal of the reference surface is determined, and the correspondence between the normal of the reference surface and the normal polarization state is established. Structured light data and polarization images are read from the aligned multimodal data of the mechanical part to be inspected. The height deviation relative to the reference surface is calculated based on the reference surface normal, and the polarization deviation is calculated based on the normal polarization state. Visible light images are read from the aligned multimodal data of the mechanical part to be inspected, linear abnormal regions are extracted, and the linear abnormal regions are compared with reference machining textures to generate machining texture variation results. An improved CMX model is constructed, including a visible light feature extraction branch, a polarization feature extraction branch, a structured light feature extraction branch, a feature correction unit, a feature fusion unit, a defect recognition branch, and a linear defect branch; Visible light features, polarization features, and structured light features are extracted from aligned multimodal data. Feature correction coefficients are generated based on height deviation, polarization deviation, and processing texture variations. These coefficients are then used to correct the visible light features, polarization features, and structured light features, generating corrected features. Specifically, this includes: When the result of the change in the processing texture does not reach the corresponding preset change threshold, and the polarization deviation and height deviation do not reach the corresponding preset threshold, a visible light feature correction coefficient of less than 1 is set according to the degree of change in the processing texture corresponding to the visible light feature. When the change result of the processing texture reaches the corresponding preset change threshold and the polarization deviation reaches the preset polarization threshold, a visible light feature correction coefficient and a polarization feature correction coefficient greater than 1 are set according to the degree of processing texture change and polarization deviation, respectively. When the height deviation reaches the preset height threshold, a structured light feature correction coefficient greater than 1 is set according to the height deviation. The visible light feature, polarization feature, and structured light feature are multiplied by their respective feature correction coefficients to generate the correction feature; The surface defect identification is performed by integrating correction features. The identified linear abnormal areas are then distinguished as normal machining marks, shallow scratches, and micro-cracks to correct the surface defect identification results. Based on the corrected surface defect identification results, the defect area, defect type, and defect location are determined, and the surface defect identification results of the mechanical parts are generated.

2. The method for identifying surface defects of mechanical parts based on multimodal visual fusion according to claim 1, characterized in that, The spatial alignment process includes: The coordinate transformation relationship is determined based on the calibration parameters of each acquisition device, and the surface coordinates of the mechanical parts are established based on the structured light data. By utilizing coordinate transformation relationships, the pixel positions in visible light images and polarized images are converted to the surface coordinates of mechanical parts, thereby determining the surface position corresponding to each pixel position; Based on the surface coordinates of the mechanical parts, data corresponding to the same surface position in visible light images, polarization images, and structured light data are correlated to generate aligned multimodal data.

3. The method for identifying surface defects of mechanical parts based on multimodal visual fusion according to claim 2, characterized in that, The process of establishing the reference surface and reference machining texture includes: Select several mechanical parts with the same model and surface finish that have passed inspection; Based on the surface coordinates of the mechanical parts, outlier values ​​and median values ​​are removed from the structured light 3D coordinates of the same surface positions of each qualified mechanical part, and a reference surface is established based on the median coordinates of each surface position. Based on the visible light images corresponding to each qualified mechanical part, the edges of the machining texture are extracted. According to the extension direction of the machining texture edges and the arrangement position of adjacent machining texture edges, the edges of the machining texture that appear repeatedly are filtered to generate reference machining textures. Based on the surface coordinates of the mechanical part, the reference machining texture is associated with the corresponding surface position in the reference surface.

4. The method for identifying surface defects of mechanical parts based on multimodal visual fusion according to claim 3, characterized in that, The process of establishing the correspondence between the reference surface normal and the normal polarization state includes: Based on the surface coordinates of the mechanical part, select the target surface position on the reference surface, and extract the median value of the coordinates around the target surface position as the neighborhood point; Perform local plane fitting on the neighborhood points, determine the reference surface normal of the target surface position based on the normal direction of the fitted plane, and unify the orientation of the reference surface normal according to the surface coordinates of the mechanical parts. The polarization image corresponding to the target surface position is read from the alignment multimodal data of each qualified mechanical part, and the light intensity values ​​corresponding to the polarization directions of 0°, 45°, 90° and 135° are extracted. The linear polarization degree and polarization angle of each qualified mechanical part at the target surface position are calculated. Outlier values ​​were removed and median values ​​were calculated for the linear polarization degree and polarization angle corresponding to each qualified mechanical part. The normal polarization state of the target surface position was determined based on the calculation results. Using the reference surface normal at the target surface position as an index, a correspondence is established between the reference surface normal and the normal polarization state of the target surface position, and each surface position of the reference surface is processed sequentially.

5. The method for identifying surface defects of mechanical parts based on multimodal visual fusion according to claim 4, characterized in that, The process of calculating the height deviation relative to the reference surface includes: Find the reference surface point corresponding to the current surface point in the reference surface, and read the reference surface normal corresponding to the reference surface point; Calculate the position difference vector between the current surface point and the reference surface point, project the position difference vector onto the reference surface normal, and generate the initial height deviation of the current surface point. The initial height deviation of the current surface point and the adjacent surface position are read according to the surface coordinates of the mechanical part. The initial height deviation that is continuous with the adjacent surface position and has the same sign is retained, and the isolated initial height deviation is discarded. The retained initial height deviation is defined as the height deviation, the height deviation along the normal direction of the reference surface is defined as the positive height deviation, and the height deviation along the opposite direction of the normal direction of the reference surface is defined as the negative height deviation.

6. The method for identifying surface defects of mechanical parts based on multimodal visual fusion according to claim 5, characterized in that, The process of calculating polarization deviation based on normal polarization state includes: Obtain the current degree of linear polarization and the current polarization angle at each surface position as the current polarization state; Match the current polarization state with the normal polarization state according to the surface coordinates of the mechanical part, calculate the difference between the current linear polarization degree and the normal linear polarization degree, and determine the minimum angle difference between the current polarization angle and the normal polarization angle according to the 180° cycle. The difference and the minimum angular difference are correlated with the corresponding surface positions to generate polarization deviation.

7. The method for identifying surface defects of mechanical parts based on multimodal visual fusion according to claim 6, characterized in that, The process of generating the processing texture change result includes: The edge connected regions are extracted from the visible light image of the mechanical part to be inspected, and the edge connected regions whose length-to-width ratio reaches a preset condition are identified as linear abnormal regions. Based on the surface coordinates of the mechanical part, read the linear abnormal area and the edge of the reference machining texture corresponding to the adjacent surface position from the reference machining texture; Calculate the difference in the extension direction between the edge of the machining texture and the corresponding reference machining texture within the linear anomaly region, and generate the direction comparison result; Calculate the spacing between the edges of adjacent machining marks on both sides of the linear anomaly region, compare it with the spacing between the corresponding reference machining mark edges, and generate the spacing comparison result. The connection status of the machining texture edges within the linear abnormal area is checked along the extension direction of the reference machining texture edge to determine the interruption, misalignment, and missing positions of the machining texture edges, and to generate continuous state comparison results. The direction comparison results, spacing comparison results, and continuous state comparison results are correlated with the corresponding linear abnormal areas according to the surface coordinates of the mechanical part to generate machining texture change results.

8. The method for identifying surface defects of mechanical parts based on multimodal visual fusion according to claim 7, characterized in that, The improved CMX model includes a visible light feature extraction branch, a polarization feature extraction branch, a structured light feature extraction branch, a feature correction unit, a feature fusion unit, a defect identification branch, and a linear defect branch; The visible light feature extraction branch reads the visible light image from the aligned multimodal data and extracts visible light features; The polarization feature extraction branch reads the polarization image from the aligned multimodal data and extracts polarization features; The structured light feature extraction branch reads structured light data from the aligned multimodal data and extracts structured light features; The feature correction unit receives visible light features, polarization features, and structured light features, and receives height deviation, polarization deviation, and processing texture variation results to generate correction features; The feature fusion unit fuses correction features to generate fusion defect features; The defect identification branch generates surface defect identification results based on the fused defect features; The linear defect branch reads the fusion defect features corresponding to the linear abnormal area, distinguishes normal processing marks, shallow scratches and fine cracks in the linear abnormal area, and corrects the surface defect identification results based on the discrimination results.

9. The method for identifying surface defects of mechanical parts based on multimodal visual fusion according to claim 8, characterized in that, The process of correcting the surface defect identification results includes: Linear abnormal areas where neither the processing texture change result nor the polarization deviation reaches the corresponding preset threshold are judged as normal processing textures. The linear abnormal regions where the processing texture change results and polarization deviation reach the corresponding preset threshold are input into the linear defect branch. If the height deviation does not reach the preset height threshold, the linear abnormal regions are retained for further discrimination. Determine whether the edges within the linear anomaly area are continuously connected, and determine whether there are two or more edges extending in different directions at the same surface location. Combine the results of processing texture changes, polarization deviation, and height deviation to generate shallow scratch discrimination results and micro-crack discrimination results. The preliminary identification results of the corresponding surface locations are replaced with the identification results of normal machining marks, shallow scratches and fine cracks to generate corrected surface defect identification results.