Intelligent sorting system and method based on machine vision

Abstract

The present application relates to the field of visual detection, and more particularly to an intelligent sorting system and method based on machine vision. Image sequences of a target are collected under multiple polarization directions and multiple spectral bands; a polarization-reflection physical constraint model is constructed to decompose pixels into diffuse reflection components and specular reflection components; a highlight mask is calculated according to the spatial distribution characteristics of the specular reflection components; a multispectral intrinsic image mapping relationship is established based on the diffuse reflection components and spectral band reflectivity, and the diffuse reflection intrinsic images under different polarization states are spliced and reconstructed through an adaptive weighted fusion strategy to generate highlight-free intrinsic images; texture and edge features are extracted on the highlight-free intrinsic images for visual detection and sorting positioning. The present application removes highlight interference from the optical mechanism, restores texture and edge features in the highlight-occluded area, overcomes the problem of feature loss on complex curvature reflective surfaces, and guarantees the recognition stability and positioning reliability in strong light reflection scenes.

Description

Technical Field

[0001] This invention relates to the field of visual inspection, and more specifically to an intelligent sorting system and method based on machine vision. Background Technology

[0002] In industrial automated production lines, machine vision-based intelligent sorting systems are widely used in target recognition and positioning. When sorting targets contain highly reflective or transparent materials, the target surface experiences strong specular reflection under light illumination, creating highlight areas that cover the original texture and defect features of the surface. Conventional vision inspection systems typically suppress reflection by adding a fixed-angle polarizing filter in front of the camera lens. The optical principle behind this is to utilize the directional transmission characteristics of the polarizer, allowing only light waves with specific vibration directions to pass through, preventing light waves with polarization directions perpendicular to the transmission axis generated by surface specular reflection from entering the sensor. This method can filter out some unidirectional reflected light and obtain the target surface texture features when processing targets with flat surfaces and unidirectional polarization of reflected light.

[0003] When the target surface has complex curvature variations, the polarization state of specularly reflected light at different spatial locations differs with changes in the incident and reflection angles. A fixed-angle polarizing filter cannot simultaneously adapt to the differences in polarization state across different regions of the field of view, leading to localized highlight retention or loss of target outline. Another conventional approach is to adjust the camera's global exposure parameters, either by shortening the exposure time to reduce the brightness of highlight areas or by switching between multiple-angle polarizers to find the optimal filtering angle. While shortening the exposure time can reduce pixel saturation in highlight areas, it also causes brightness decay and detail loss in non-reflective areas, introducing image noise and failing to balance highlight suppression and shadow detail preservation. The speed of mechanically switching multiple-angle polarizers is difficult to match the pace of high-speed production lines, and it still relies on a global filtering solution, failing to address the problem of spatially uneven polarization state differences.

[0004] Existing fixed-polarization filtering and exposure adjustment methods cannot preserve the original texture and edge details of the target surface while eliminating specular reflection highlights. Feature loss on complex curvature reflective surfaces constitutes a core technical obstacle to accurate visual inspection, identification, and sorting. Fixed-polarization filtering only works on reflected light with a single polarization state. When facing multi-directional polarized reflected beams from curved targets, the local highlights generated by filter failure directly cover defect features, preventing feature extraction algorithms from obtaining complete edge contours and texture topology. Global exposure adjustment suppresses highlights at the expense of details in non-reflective areas, disrupting the overall grayscale dynamic range of the image and causing the fine features of the target surface to be submerged in noise. Feature loss results in a lack of reliable input data for subsequent defect identification and sorting coordinate positioning, leading to frequent missed detections and positioning misalignments in sorting systems under highly reflective and transparent material conditions. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent sorting system and method based on machine vision, which can solve the problems in the background art mentioned above.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A machine vision-based intelligent sorting system includes: a multi-polarization spectral image acquisition array for acquiring image sequences of targets in multiple polarization directions and multiple spectral bands; a physical constraint modeler for constructing a polarization-reflection physical constraint model, decomposing pixels in the image sequence into diffuse reflection components and specular reflection components; a specular mask calculator for calculating specular masks based on the spatial distribution characteristics of the specular reflection components; an intrinsic image reconstructor for establishing a multi-spectral intrinsic image mapping relationship based on the diffuse reflection components and spectral band reflectance, and stitching and reconstructing diffuse reflection intrinsic images under different polarization states through an adaptive weighted fusion strategy to generate a specular-free intrinsic image; and a visual inspection and sorting locator for extracting texture and edge features from the specular-free intrinsic image for defect identification and sorting coordinate positioning.

[0007] Preferably, the multi-polarization spectral image acquisition array includes a polarization state control subarray and a spectral filtering subarray. The polarization state control subarray and the spectral filtering subarray work together to synchronously acquire sub-image sets with different polarization angles and different wavelength combinations under the same spatial coordinates. The physical constraint modeler constructs the polarization-reflection physical constraint model based on Fresnel's law of reflection and polarization Stokes parameters, mapping the intensity information of the sub-image sets into Stokes parameters containing polarization degree and polarization angle. By solving the difference between the orthogonal polarization components in the Stokes parameters, the diffuse reflection component and the specular reflection component are separated.

[0008] Preferably, the specular mask calculator performs binarization thresholding on the specular reflection component based on the gradient magnitude and local curvature change of the specular reflection component to generate an initial specular region; performs morphological dilation and hole filling operations on the initial specular region to determine a continuous specular mask boundary; and determines a weight allocation factor for the specular mask based on the coverage ratio of the specular mask boundary in the image sequence. The weight allocation factor is used to suppress the interference of pixels inside the specular mask on the subsequent feature extraction process.

[0009] Preferably, the intrinsic image reconstructor constructs a spectral reflectance matrix based on the spectral absorption and transmission characteristics of different materials in the multispectral band; multiplies the diffuse reflection component with the inverse of the spectral reflectance matrix to obtain the diffuse reflectance intrinsic image after eliminating the influence of ambient light; and establishes a wavelength-spatial frequency joint mapping table based on the center wavelength of the multispectral band and the frequency domain response characteristics of the diffuse reflectance intrinsic image. The wavelength-spatial frequency joint mapping table is used to guide the calculation of fusion weights for images in different bands in the adaptive weighted fusion strategy.

[0010] Preferably, when executing the adaptive weighted fusion strategy, the intrinsic image reconstructor calculates the structural similarity of the diffuse intrinsic images under different polarization states in the gradient direction; using the structural similarity as a constraint, it performs Laplacian pyramid decomposition on local pixel blocks in the diffuse intrinsic images; and applies a fusion rule based on spatial frequency energy to the decomposed high-frequency sub-bands and low-frequency sub-bands respectively, and performs pyramid reconstruction on the fused high-frequency sub-bands and low-frequency sub-bands to generate the intrinsic image without specular highlights.

[0011] Preferably, for transparent material regions where the linear polarization degree in the Stokes parameter exceeds a preset threshold, a polarization degree attenuation factor is introduced to suppress residual polarization crosstalk in the diffuse reflection component; by analyzing the phase difference change of the orthogonal polarization components at different wavelengths, a multispectral polarization phase consistency constraint function is established, and the edge contour of the diffuse reflection component is corrected using the multispectral polarization phase consistency constraint function to eliminate the edge diffusion effect caused by multiple refractions.

[0012] Preferably, the intrinsic image reconstructor extracts the boundary transition region between high-frequency texture features and low-frequency hue features in the diffuse intrinsic image according to the wavelength-spatial frequency joint mapping table; within the boundary transition region, it calculates the local variance of adjacent pixels and constructs an edge-preserving smoothing filter kernel based on the local variance; and uses the edge-preserving smoothing filter kernel to perform convolution smoothing processing on the splicing gap between the high-frequency subband and the low-frequency subband during the pyramid reconstruction process to eliminate spectral splicing artifacts in the intrinsic image without highlights.

[0013] Preferably, the system further includes an ambient light dynamic compensator, which collects the ambient light spectral distribution and illuminance fluctuation information of the current working environment in real time; calculates an illumination bias matrix based on the ambient light spectral distribution and illuminance fluctuation information; and superimposes the illumination bias matrix onto the polarization-reflection physical constraint model to dynamically correct the decomposition coefficients of the diffuse reflection component and the specular reflection component, thereby compensating for the impact of sudden changes in ambient light on the pixel intensity of the image sequence.

[0014] Preferably, it also includes a sorting coordinate verifier, which maps the sorting coordinates output by the visual detection and sorting locator to the robot's execution space; extracts the three-dimensional normal vector information of the target in the specular-free intrinsic image, performs alignment calculations between the three-dimensional normal vector information and the preset grasping posture in the execution space to generate a posture correction matrix; and performs a spatial affine transformation on the sorting coordinates according to the posture correction matrix to output the target grasping posture after posture compensation.

[0015] A machine vision-based intelligent sorting method includes: acquiring image sequences of targets in multiple polarization directions and multiple spectral bands; constructing a polarization-reflection physical constraint model to decompose pixels in the image sequence into diffuse reflection components and specular reflection components; calculating a specular mask based on the spatial distribution characteristics of the specular reflection components; establishing a multispectral intrinsic image mapping relationship based on the diffuse reflection components and spectral band reflectance, and stitching and reconstructing diffuse reflection intrinsic images under different polarization states through an adaptive weighted fusion strategy to generate a specular-free intrinsic image; and extracting texture and edge features from the specular-free intrinsic image for visual detection and sorting coordinate positioning.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention acquires image sequences of targets under multiple polarization directions and multiple spectral bands, constructs a polarization-reflection physical constraint model, decomposes the pixels in the image sequence into diffuse reflection components and specular reflection components, calculates a specular mask based on the spatial distribution characteristics of the specular reflection component, establishes a multispectral intrinsic image mapping relationship based on the diffuse reflection component and the reflectivity of the spectral band, and uses an adaptive weighted fusion strategy to stitch together and reconstruct a specular-free intrinsic image from diffuse reflection intrinsic images under different polarization states. Texture and edge features are extracted from the specular-free intrinsic image for visual detection and sorting coordinate positioning. By physically decoupling image pixels into diffuse reflection and specular reflection components, specular interference is removed from the optical mechanism level, and a specular-free intrinsic image containing only the intrinsic information of the target is reconstructed. This overcomes the defect that fixed polarization filters cannot adapt to the polarization state differences caused by changes in the curvature of the target surface, restores the texture and edge features of the specular-occluded area, solves the problem of detection failure caused by the loss of features on complex curvature reflective surfaces, and ensures the recognition stability and positioning reliability of the sorting system in strong reflective scenes. By using Fresnel's law of reflection and polarization Stokes parameters to solve for the difference between orthogonal polarization components, and combining the spectral reflectance matrix to obtain the diffuse reflectance intrinsic image that eliminates the influence of ambient light, the subbands of the Laplacian pyramid decomposition are fused based on spatial frequency energy according to structural similarity, ensuring that the reconstructed image has smooth edge transitions and complete texture details.

[0017] 2. For areas with excessive linear polarization in transparent materials, a polarization attenuation factor is introduced to suppress residual polarization crosstalk in the diffuse reflection component. A multispectral polarization phase consistency constraint function is then used to correct the edge contour of the diffuse reflection component, eliminating edge blurring caused by multiple refractions and improving the clarity of the transparent target contour. When reconstructing an image without specular highlights, the boundary transition region between high-frequency texture features and low-frequency hue features is extracted based on a wavelength-spatial frequency joint mapping table. Local variance is calculated to construct an edge-preserving smoothing filter kernel. Convolutional smoothing is applied to the stitching gaps during pyramid reconstruction to eliminate spectral stitching artifacts and improve image visual consistency. Real-time acquisition of ambient light spectral distribution and illuminance fluctuation information is used to calculate the illumination bias matrix and superimpose it onto the physical constraint model to dynamically correct the decomposition coefficients, compensating for the impact of sudden ambient light changes on pixel intensity and enhancing the system's resistance to illumination interference. The sorting coordinates are mapped to the execution space, and a posture correction matrix is ​​generated by combining the 3D normal vector information with the pre-set grasping posture alignment. A spatial affine transformation is performed on the sorting coordinates to output the posture-compensated target grasping pose, correcting the grasping pose deviation caused by target tilt. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the overall data processing of the machine vision-based intelligent sorting system of the present invention. Figure 2 This is a flowchart of the step-by-step processing of the sub-image set acquired by the dual-path sub-array collaborative acquisition of the multi-polarization spectral acquisition array of the present invention; Figure 3 The flowchart of the specular mask calculator of the present invention is shown below for solving the specular mask and weight allocation factor based on specular reflection components; Figure 4 The flowchart for constructing the spectral reflectance matrix and solving the diffuse reflectance intrinsic image processing for the intrinsic image reconstructor of this invention is as follows: Figure 5 This is a flowchart of the Laplacian pyramid hierarchical adaptive image fusion method for generating images without highlights based on structural similarity constraints, as described in this invention. Figure 6 This is a flowchart of the entire process of combined correction of ambient light dynamic compensation and sorting coordinate attitude correction in this invention. Detailed Implementation

[0019] refer to Figure 1In one embodiment, a machine vision-based intelligent sorting system is deployed at the sorting station of an industrial automated production line. The target object moves at a constant speed along a conveyor belt to the image acquisition area. A multi-polarization spectral image acquisition array is installed directly above the conveyor belt, maintaining a perpendicular distance from the conveyor belt plane, and its field of view covers the effective sorting width of the conveyor belt. The multi-polarization spectral image acquisition array acquires image sequences of the target object in multiple polarization directions and multiple spectral bands. Each sub-image in the image sequence corresponds to a specific combination of a specific polarization angle and a specific spectral band, and all sub-images have the same spatial resolution and pixel coordinate mapping relationship. A physical constraint modeler receives the image sequence output by the multi-polarization spectral image acquisition array and constructs a polarization-reflection physical constraint model, decomposing each pixel in the image sequence into diffuse reflection and specular reflection components. A specular mask calculator receives the specular reflection component output by the physical constraint modeler and calculates a specular mask based on the spatial distribution characteristics of the specular reflection component. The specular mask is a binary image, where the pixel value in the specular region is 1, and the pixel value in the non-spectral region is 0. The intrinsic image reconstructor receives the diffuse reflection component output by the physical constraint modeler, establishes a multispectral intrinsic image mapping relationship based on the diffuse reflection component and spectral band reflectance, and stitches together the diffuse reflection intrinsic images under different polarization states through an adaptive weighted fusion strategy to generate a specular-free intrinsic image. The visual inspection and sorting locator receives the specular-free intrinsic image output by the intrinsic image reconstructor, extracts texture and edge features from the specular-free intrinsic image for defect identification and sorting coordinate localization, and outputs the target object's category information, defect location information, and sorting coordinate information.

[0020] In this embodiment, the multi-polarization spectral image acquisition array consists of multiple image acquisition units, each including an image sensor, a polarization state controller, and a spectral filter. The polarization state controller can adjust the polarization direction of the incident light, and the spectral filter can select light within a specific wavelength range to pass through. The multi-polarization spectral image acquisition array sequentially switches between different polarization states and spectral bands according to a preset acquisition sequence, synchronously acquiring images of the target object under different polarization directions and spectral bands. The acquisition sequence setting ensures that image acquisition of all combinations of polarization states and spectral bands is completed within the time it takes for the target object to pass through the acquisition field of view. The physical constraint modeler establishes a mathematical relationship between pixel intensity and polarization parameters based on the polarization and reflection characteristics of light, converting the intensity information in the image sequence into a polarization parameter matrix containing physical quantities such as polarization degree and polarization angle. Based on the difference in polarization characteristics between diffuse and specular light, the physical constraint modeler decomposes the total light intensity of each pixel into diffuse light intensity and specular light intensity. The specular mask calculator analyzes the spatial distribution of the specular reflection component, identifies specular regions where pixel intensity is significantly higher than the surrounding area, and generates specular masks to mark the specular positions. The intrinsic image reconstructor utilizes the differences in reflectance characteristics of different materials across different spectral bands to eliminate the influence of ambient light on diffuse reflection components and extract the intrinsic reflection information of the target object. By fusing diffuse reflection intrinsic images under different polarization states, the intrinsic image reconstructor integrates effective information from each polarization state to generate a complete intrinsic image without specular highlights. The visual inspection and sorting locator uses an edge detection operator to extract edge features from the intrinsic image without specular highlights and a texture analysis operator to extract texture features. The extracted features are matched with a preset defect feature template to identify whether the target object has defects and the type and location of those defects. Simultaneously, the visual inspection and sorting locator calculates the center coordinates and rotation angle of the target object based on its edge contour to generate sorting coordinate information.

[0021] refer to Figure 2In one optional embodiment, the multi-polarization spectral image acquisition array includes a polarization state control subarray and a spectral filtering subarray. The polarization state control subarray and the spectral filtering subarray work together to synchronously acquire sub-image sets with different polarization angles and different wavelength combinations under the same spatial coordinates. The polarization state control subarray consists of multiple liquid crystal polarizers, each corresponding to an image acquisition unit. By applying different voltages, the alignment direction of the liquid crystal molecules is controlled, thus changing the polarization direction of the incident light. The spectral filtering subarray consists of multiple narrowband filters, each corresponding to a specific center wavelength. Different narrowband filters are selected to enter the optical path through mechanical switching or electrical tuning. The physical constraint modeler constructs a polarization-reflection physical constraint model based on Fresnel's law of reflection and polarization Stokes parameters. It maps the intensity information of the sub-image sets into Stokes parameters containing polarization degree and polarization angle. By solving for the difference between the orthogonal polarization components in the Stokes parameters, the diffuse reflection component and the specular reflection component are separated.

[0022] In this embodiment, the polarization state control subarray can provide at least four different polarization directions, namely 0°, 45°, 90°, and 135°. The spectral filtering subarray can provide at least four different spectral bands with center wavelengths of 450nm, 550nm, 650nm, and 850nm, respectively. When acquiring images, the multi-polarization spectral image acquisition array sequentially acquires images in four polarization directions for each spectral band, forming a sub-image set containing 16 sub-images. Each sub-image in the sub-image set has the same spatial size and corresponding pixel coordinates. The physical constraint modeler first converts the intensity information in the sub-image set into Stokes parameters. The Stokes parameters consist of four parameters, namely... , , and ,in Indicates total light intensity. This represents the difference between the horizontal and vertical polarization components. This represents the difference in polarization components between the 45° and 135° directions. This represents the circularly polarized component. For linearly polarized light, The value is approximately 0, therefore only 0 is considered in this embodiment. , and Three parameters.

[0023] The formula for calculating the Stokes parameter is as follows:

[0024] in, , , and These represent the image intensities when the polarization direction is 0°, 45°, 90°, and 135°, respectively.

[0025] Linear polarization degree P and polarization angle The calculation formula is as follows:

[0026]

[0027] Where P takes values ​​in the range [0,1]. The value range is [-90°, 90°].

[0028] According to Fresnel's law of reflection, specularly reflected light has a high degree of polarization, typically close to 1, while diffusely reflected light has a low degree of polarization, typically close to 0. Therefore, the physical constraint modeler will consider the total light intensity... Decomposed into diffuse reflection components With specular reflection component :

[0029] Specular reflection component The relationship with the linear polarization degree P is as follows:

[0030] Where k is a proportionality coefficient, with a value ranging from [0,1], which is adjusted according to the reflectivity of the target material. For highly reflective metallic materials, k takes a larger value; for low-reflective non-metallic materials, k takes a smaller value.

[0031] Diffuse reflection component The calculation formula is:

[0032] Using the above formula, the physical constraint modeler decomposes each pixel in the sub-image set into diffuse reflection and specular reflection components. For each spectral band, the corresponding diffuse reflection and specular reflection components are calculated separately to form a multispectral diffuse reflection matrix and a multispectral specular reflection matrix.

[0033] refer to Figure 3 In one optional embodiment, the specular mask calculator performs binarization thresholding on the specular reflection component based on the gradient magnitude and local curvature changes of the specular reflection component to generate an initial specular region. Morphological dilation and hole-filling operations are performed on the initial specular region to determine continuous specular mask boundaries. Based on the coverage ratio of the specular mask boundaries in the image sequence, a weighting factor for the specular mask is determined. This weighting factor is used to suppress interference from pixels within the specular mask on subsequent feature extraction processes.

[0034] In this embodiment, the specular mask calculator first calculates the gradient magnitude of the specular reflection component. The gradient magnitude reflects the rate of change of pixel intensity in the image; edges of specular regions typically have larger gradient magnitudes. The gradient magnitude is calculated using the Sobel operator, calculating the gradient components in the horizontal and vertical directions separately, and then synthesizing the gradient magnitude. For the specular reflection component image... Horizontal gradient component gradient components in the vertical direction The calculation formula is:

[0035]

[0036] gradient magnitude The calculation formula is:

[0037] The specular mask calculator simultaneously calculates the local curvature of the specular reflection component. Local curvature reflects the degree of bending of pixel intensity in an image; the center of the specular region typically has a larger local curvature. The calculation of local curvature uses a second-derivative operator to calculate the Laplacian value of each pixel as a measure of local curvature. (Laplacian value) The calculation formula is:

[0038] The specular mask calculator determines the binarization threshold based on the joint distribution of gradient magnitude and local curvature. Pixels with both gradient magnitude and local curvature greater than the gradient threshold are marked as initial specular pixels. The gradient threshold and curvature threshold are adaptively determined based on the statistical characteristics of the specular reflection components; the gradient threshold is taken as the 95th quantile of the gradient magnitude histogram, and the curvature threshold is taken as the 95th quantile of the Laplacian value histogram.

[0039] After generating the initial specular region, the specular mask calculator performs a morphological dilation operation on it. The dilation structuring element is a 3×3 rectangle, and the dilation is performed twice. This morphological dilation operation connects adjacent specular regions, forming a continuous specular region. Subsequently, the specular mask calculator performs a hole-filling operation on the dilated specular region, filling the small holes inside the specular region to ensure its integrity.

[0040] The specular mask calculator calculates the coverage ratio of the specular mask in an image sequence, which is the ratio of the number of pixels in the specular region to the total number of pixels. Based on the coverage ratio, the weighting factor w of the specular mask is determined. The formula for calculating the weighting factor w is:

[0041] Where r is the coverage ratio of the specular mask, The value of w is an adjustment coefficient, ranging from [0,1]. When the coverage ratio r is small, the weight allocation factor w is close to 1, resulting in a weaker suppression effect on pixels in the highlight area; when the coverage ratio r is large, the weight allocation factor w decreases, resulting in a stronger suppression effect on pixels in the highlight area.

[0042] In the subsequent feature extraction process, for pixels inside the highlight mask, their feature weights are multiplied by a weight allocation factor w to reduce the impact of highlight region pixels on the feature extraction results. For pixels outside the highlight region, their feature weights remain unchanged.

[0043] refer to Figure 4 In one optional embodiment, the intrinsic image reconstructor constructs a spectral reflectance matrix based on the spectral absorption and transmission characteristics of different materials in multispectral bands. The diffuse reflectance component is multiplied by the inverse of the spectral reflectance matrix to obtain a diffuse reflectance intrinsic image free from the influence of ambient light. Based on the center wavelength of the multispectral bands and the frequency domain response characteristics of the diffuse reflectance intrinsic image, a wavelength-spatial frequency joint mapping table is established. This table guides the calculation of fusion weights for different band images in the adaptive weighted fusion strategy.

[0044] In this embodiment, different materials exhibit different reflectivities in different spectral bands. This difference in reflectivity is an inherent property of the materials and is independent of ambient lighting. The intrinsic image reconstructor first constructs a spectral reflectivity matrix R, where the number of rows equals the number of spectral bands, and the number of columns equals the number of material types. Matrix element R(i,j) represents the reflectivity of the j-th material in the i-th spectral band.

[0045] In this embodiment, the spectral reflectance data of different materials are shown in Table 1.

[0046] Table 1. Spectral reflectance data for different materials

[0047] Ambient light intensity varies across different spectral bands. Let the ambient light intensity vector be L, with its length equal to the number of spectral bands. The vector element L(i) represents the ambient light intensity in the i-th spectral band. (Diffuse reflection component vector) The relationship between the spectral reflectance matrix R and the ambient light intensity vector L is as follows:

[0048] The intrinsic image reconstructor obtains the ambient light intensity vector L by solving the above equation. Since the spectral reflectance matrix R is usually not a square matrix, the least squares method is used to find the optimal solution to the equation. After obtaining the ambient light intensity vector L, the diffuse intrinsic image is obtained. The calculation formula is:

[0049] in, It is the pseudo-inverse of the spectral reflectance matrix R.

[0050] Based on the above calculations, the diffuse reflection intrinsic image The influence of ambient light is eliminated, and only the inherent reflective properties of the target material are included. For the diffuse reflection component under each polarization state, the corresponding diffuse reflection eigenimage is calculated to form a multi-polarization state diffuse reflection eigenimage set.

[0051] The intrinsic image reconstructor establishes a wavelength-spatial frequency joint mapping table based on the frequency domain response characteristics of the center wavelength of the multispectral band and the diffuse reflection intrinsic image. This table records the optimal spatial frequency range corresponding to different center wavelengths, as well as the weighting coefficients at different spatial frequencies. For shorter wavelengths, the corresponding optimal spatial frequency is higher, suitable for extracting detailed texture features from the image; for longer wavelengths, the corresponding optimal spatial frequency is lower, suitable for extracting overall contour features from the image.

[0052] In this embodiment, the wavelength-spatial frequency joint mapping relationship is shown in Table 2.

[0053] Table 2 Wavelength-Spatial Frequency Joint Mapping Table

[0054] In Table 2, This represents the Nyquist frequency of the image, which is half the image sampling frequency. The wavelength-spatial frequency joint mapping table is used to guide the calculation of fusion weights for images in different bands in the adaptive weighted fusion strategy, ensuring that the most suitable spectral band information is selected for fusion within different spatial frequency ranges.

[0055] refer to Figure 5 In one optional embodiment, the intrinsic image reconstructor calculates the structural similarity of diffuse intrinsic images under different polarization states along the gradient direction when executing an adaptive weighted fusion strategy. Using structural similarity as a constraint, Laplacian pyramid decomposition is performed on local pixel blocks in the diffuse intrinsic image. A fusion rule based on spatial frequency energy is applied to the decomposed high-frequency and low-frequency sub-bands respectively, and pyramid reconstruction is performed on the fused high-frequency and low-frequency sub-bands to generate an intrinsic image without specular highlights.

[0056] In this embodiment, the intrinsic image reconstructor first divides the multi-polarization state diffuse intrinsic image set into multiple non-overlapping local pixel blocks, each pixel block having a size of 16×16. For each pixel block location, the structural similarity between pixel blocks of diffuse intrinsic images under different polarization states at that location is calculated. The structural similarity calculation is based on the consistency of gradient directions. For two pixel blocks A and B, the structural similarity is... The calculation formula is:

[0057] in, and These represent the gradient magnitudes of pixel blocks A and B at position (x, y), respectively. C is a constant used to avoid zero in the denominator, and its value is 0.01.

[0058] Structural similarity The value of is in the range of [0,1], and the larger the value, the more similar the structures of the two pixel blocks are. The intrinsic image reconstructor selects the two diffuse intrinsic image pixel blocks with the highest structural similarity under the two polarization states as the basis for fusion to ensure that the fused pixel blocks retain the clearest structural information.

[0059] The intrinsic image reconstructor performs Laplacian pyramid decomposition on the two selected pixel blocks, with a decomposition layer of 4. Laplacian pyramid decomposition divides the image into a low-frequency sub-band and multiple high-frequency sub-bands. The low-frequency sub-band contains the overall contour information of the image, while the high-frequency sub-bands contain the detailed texture information. For each decomposed sub-band, a fusion rule based on spatial frequency energy is applied.

[0060] In this embodiment, the fusion rules for each layer of the Laplace pyramid are shown in Table 3.

[0061] Table 3. Merging Rules of Each Layer of the Pyramid of Laplace

[0062] The formula for calculating spatial frequency energy is:

[0063] in, and These represent the horizontal and vertical gradient components of the subband at position (x, y), respectively.

[0064] For the low-frequency subbands, a weighted average fusion method is used, with the weight proportional to the average gray value of the subband to ensure that the fused low-frequency subband retains the correct overall brightness information. For the first and second layer high-frequency subbands, coefficients with larger absolute values ​​are selected for fusion to retain the clearest edge and detail information. For the third layer high-frequency subband, a weighted average fusion method is used, with the weight proportional to the spatial frequency energy of the subband to balance the detail information under different polarization states.

[0065] After fusion, the intrinsic image reconstructor performs Laplacian pyramid reconstruction on the fused sub-bands to generate fused diffuse intrinsic image pixel blocks. All fused pixel blocks are then stitched together to form a complete intrinsic image without highlights.

[0066] In one optional embodiment, for transparent material regions where the linear polarization degree in the Stokes parameter exceeds a preset threshold, a polarization degree attenuation factor is introduced to suppress residual polarization crosstalk in the diffuse reflection component. By analyzing the phase difference changes of orthogonal polarization components at different wavelengths, a multispectral polarization phase consistency constraint function is established. This function is then used to correct the edge contour of the diffuse reflection component, eliminating the edge diffusion effect caused by multiple refractions.

[0067] In this embodiment, transparent materials such as glass and plastics not only exhibit specular reflection but also multiple refractions on their surfaces. Multiple refractions cause some refracted light to escape from within the transparent material, forming a diffuse reflection component with a certain degree of polarization; this phenomenon is called polarization crosstalk. Polarization crosstalk causes the diffuse reflection component to contain partial polarization information, affecting the accuracy of separating the diffuse reflection component from the specular reflection component.

[0068] The intrinsic image reconstructor first calculates the linear polarization degree P of each pixel based on the Stokes parameter. When the linear polarization degree P exceeds a preset threshold... When this occurs, the pixel is determined to belong to a transparent material area. (Preset threshold) The value range is [0.3, 0.5], and it is adjusted according to the actual application scenario. For pixels in transparent material areas, a polarization attenuation factor is introduced. Correction is made for the diffuse reflection component:

[0069] in, This is the corrected diffuse reflection component. The attenuation coefficient, with a value range of [0,1]. When the degree of linear polarization P equals... When the attenuation factor is 1, the diffuse reflection component remains unchanged; when the linear polarization degree P is greater than 1, the diffuse reflection component remains unchanged. When the attenuation factor is less than 1, the diffuse reflection component is suppressed, reducing the influence of residual polarization crosstalk.

[0070] The intrinsic image reconstructor analyzes the phase difference variation of orthogonal polarization components at different wavelengths. Orthogonal polarization components refer to those in the 0° and 90° polarization directions, and those in the 45° and 135° polarization directions. For the edge regions of transparent materials, the phase difference of the orthogonal polarization components changes at different wavelengths due to multiple refractions. The intrinsic image reconstructor establishes a multispectral polarization phase consistency constraint function, which takes the following form:

[0071] in, Indicates wavelength as Phase difference of orthogonal polarization components, This represents the reference phase difference, which is the average of the phase differences across all wavelengths.

[0072] Multispectral polarization phase consistency constraint function The smaller the value, the more consistent the phase difference at that wavelength is with the reference phase difference, and the clearer the edge contour. The intrinsic image reconstructor is based on the constraint function. The value is used to perform weighted fusion of the diffuse reflection component edges at different wavelengths, with the weights being... The value is inversely proportional. In this way, the fused edge profile retains the edge information at the wavelength with the best phase consistency, eliminating the edge dispersion effect caused by multiple refractions.

[0073] In one optional embodiment, the intrinsic image reconstructor extracts the boundary transition region between high-frequency texture features and low-frequency hue features in the diffuse intrinsic image based on a wavelength-spatial frequency joint mapping table. Within the boundary transition region, the local variance of adjacent pixels is calculated, and an edge-preserving smoothing filter kernel is constructed based on the local variance. This edge-preserving smoothing filter kernel is used to perform convolutional smoothing on the seams between the high-frequency and low-frequency subbands during the pyramid reconstruction process, eliminating spectral stitching artifacts in the spectrophotometric intrinsic image.

[0074] In this embodiment, spectral stitching artifacts typically appear in the transition region between high-frequency texture features and low-frequency hue features. This is because different spectral bands have different fusion weights at different spatial frequencies, leading to discontinuities at the boundary. The intrinsic image reconstructor determines the frequency boundary point between high-frequency texture features and low-frequency hue features based on the wavelength-spatial frequency joint mapping table. For regions with frequencies higher than the boundary point, information from short-wavelength spectral bands is mainly composed of information from short-wavelength spectral bands; for regions with frequencies lower than the boundary point, information from long-wavelength spectral bands is mainly composed of information from long-wavelength spectral bands. The boundary transition region is the area near the frequency boundary point, with a width of ±10% of the frequency boundary point.

[0075] The intrinsic image reconstructor calculates the local variance within a 3×3 neighborhood of each pixel in the boundary transition region. Local variance reflects the degree of variation in pixel intensity within the neighborhood; a larger local variance indicates richer texture in the region, while a smaller local variance indicates more uniform tone. Based on the local variance, the intrinsic image reconstructor constructs an edge-preserving smoothing filter kernel with a size of 5×5. The weight of the filter kernel is inversely proportional to the local variance. For regions with large local variance, the weight of the filter kernel is smaller, resulting in a weaker smoothing effect to preserve edge information; for regions with small local variance, the weight of the filter kernel is larger, resulting in a stronger smoothing effect to eliminate stitching artifacts.

[0076] During the Laplacian pyramid reconstruction process, the intrinsic image reconstructor applies a convolutional smoothing kernel to the seam between the high-frequency and low-frequency subbands. The seam refers to the overlapping portion of the high-frequency and low-frequency subbands at the boundary transition region. Through convolutional smoothing, the pixel intensity transition at the seam is more natural, eliminating spectral stitching artifacts and improving the visual consistency of the intrinsic image without highlights.

[0077] refer to Figure 6 In one optional embodiment, the machine vision-based intelligent sorting system further includes an ambient light dynamic compensator, which collects real-time information on the ambient light spectral distribution and illuminance fluctuations of the current working environment. Based on the ambient light spectral distribution and illuminance fluctuation information, an illumination bias matrix is ​​calculated. This illumination bias matrix is ​​then superimposed onto a polarization-reflection physical constraint model to dynamically correct the decomposition coefficients of the diffuse reflection and specular reflection components, compensating for the impact of sudden changes in ambient light on the pixel intensity of the image sequence.

[0078] In this embodiment, the ambient light dynamic compensator includes a spectral sensor and an illuminance sensor. The spectral sensor is used to collect the spectral distribution information of ambient light, and the illuminance sensor is used to collect the illuminance fluctuation information of ambient light. The spectral sensor and the illuminance sensor are installed near the multi-polarization spectral image acquisition array and work synchronously with the multi-polarization spectral image acquisition array to collect ambient light information in real time.

[0079] The ambient light dynamic compensator calculates the ambient light intensity in each spectral band based on the ambient light spectral distribution information output by the spectral sensor. Based on the illuminance fluctuation information output by the illuminance sensor, the average illuminance of the ambient light is calculated. The ambient light dynamic compensator constructs an illumination bias matrix B, the size of which is the same as the size of the image sequence, and the matrix elements... The wavelength at position (x, y) is Illumination bias value.

[0080] The formula for calculating the illumination bias matrix B is:

[0081] in, For reference, ambient light spectral distribution, For reference average illuminance, The compensation coefficient has a value range of [0,1]. The reference ambient light spectral distribution and reference average illuminance are standard ambient light parameters obtained during the system calibration process.

[0082] The ambient light dynamic compensator superimposes the illumination bias matrix B onto the polarization-reflection physical constraint model to correct the pixel intensity of the image sequence:

[0083] in, The pixel intensity of the original image sequence. The pixel intensity of the corrected image sequence.

[0084] The corrected image sequence is input into the physical constraint modeler, which recalculates the Stokes parameters based on the corrected pixel intensities and decomposes the diffuse and specular components. By dynamically correcting the decomposition coefficients, the ambient light dynamic compensator can compensate for the impact of sudden changes in ambient light on the pixel intensity of the image sequence, ensuring that the separation accuracy of the diffuse and specular components is not affected by changes in ambient light.

[0085] In one optional embodiment, the machine vision-based intelligent sorting system further includes a sorting coordinate verifier, which maps the sorting coordinates output by the visual detection and sorting locator to the robot's execution space. The system extracts the 3D normal vector information of the target from the specular-free intrinsic image, aligns this information with a preset grasping posture in the execution space, and generates a posture correction matrix. Based on the posture correction matrix, a spatial affine transformation is performed on the sorting coordinates to output the posture-compensated target grasping pose.

[0086] In this embodiment, the sorting coordinates output by the visual detection and sorting locator are two-dimensional coordinates in image space, including the x-coordinate and y-coordinate of the target center and the rotation angle of the target. The sorting coordinate calibrator first maps the two-dimensional coordinates in the image space to the robot's execution space using a hand-eye calibration matrix, thus obtaining the initial grasping pose in the execution space. The hand-eye calibration matrix is ​​the coordinate transformation matrix between the image space and the execution space obtained during the system calibration process.

[0087] The sorting coordinate verifier extracts the 3D normal vector information of the target from the specular-free intrinsic image. This 3D normal vector information is acquired using photometric stereo vision technology. By utilizing specular-free intrinsic images under different lighting directions, the normal vector of each point on the target surface is calculated. For each target, the average of the normal vectors of all points on its surface is calculated as the overall normal vector of the target.

[0088] The sorting coordinate verifier aligns the target's global normal vector with the preset grasping posture in the execution space. The preset grasping posture is the standard grasping posture of the robot's end effector, with its normal vector direction perpendicular to the conveyor belt plane. The goal of the alignment operation is to find a rotation matrix that rotates the target's global normal vector to the normal vector direction of the preset grasping posture. This rotation matrix is ​​the posture correction matrix R.

[0089] The attitude correction matrix R is calculated using the Rodriguez rotation formula:

[0090] Where I is the identity matrix, Let be the angle between the target normal vector and the preset grasping posture normal vector, and K be an antisymmetric matrix composed of rotation axis vectors.

[0091] The sorting coordinate verifier performs a spatial affine transformation on the initial grasping pose based on the attitude correction matrix R, obtaining the attitude-compensated target grasping pose. The spatial affine transformation includes rotation and translation transformations, ensuring that the robot's end effector can grasp the target object with the correct attitude. Through attitude compensation, the sorting coordinate verifier can correct grasping pose deviations caused by target tilt, improving the grasping success rate of the sorting system.

Claims

1. A machine vision-based intelligent sorting system, characterized in that, include: Multipolar spectral image acquisition array, used to acquire image sequences of targets in multiple polarization directions and multiple spectral bands; A physical constraint modeler is used to construct a polarization-reflection physical constraint model, which decomposes the pixels in the image sequence into diffuse reflection components and specular reflection components. A specular mask calculator is used to calculate a specular mask based on the spatial distribution characteristics of the specular reflection components. The intrinsic image reconstructor is used to establish a multispectral intrinsic image mapping relationship based on the diffuse reflection component and the reflectance of the spectral band, and to stitch together and reconstruct the diffuse reflection intrinsic images under different polarization states through an adaptive weighted fusion strategy to generate an intrinsic image without specular highlights. A visual inspection and sorting locator is used to extract texture and edge features from the non-highlight intrinsic image for defect identification and sorting coordinate positioning. The specular mask calculator performs binarization thresholding on the specular reflection component based on the gradient amplitude and local curvature change of the specular reflection component, thereby generating an initial specular region. Perform morphological dilation and void filling operations on the initial specular region to determine the continuous specular mask boundary; Based on the coverage ratio of the highlight mask boundary in the image sequence, a weight allocation factor for the highlight mask is determined. The weight allocation factor is used to suppress the interference of pixels inside the highlight mask on the subsequent feature extraction process. The intrinsic image reconstructor constructs a spectral reflectance matrix based on the spectral absorption and transmission characteristics of different materials in the multispectral band. Multiply the diffuse reflectance component by the inverse of the spectral reflectance matrix to obtain the diffuse reflectance intrinsic image that eliminates the influence of ambient light. Based on the center wavelength of the multispectral band and the frequency domain response characteristics of the diffuse reflection intrinsic image, a wavelength-spatial frequency joint mapping table is established. The wavelength-spatial frequency joint mapping table is used to guide the calculation of fusion weights for images of different bands in the adaptive weighted fusion strategy. When the intrinsic image reconstructor executes the adaptive weighted fusion strategy, it calculates the structural similarity of the diffuse intrinsic images under different polarization states in the gradient direction. Using the structural similarity as a constraint, Laplacian pyramid decomposition is performed on local pixel blocks in the diffuse intrinsic image; The high-frequency subband and low-frequency subband after decomposition are respectively applied to the fusion rule based on spatial frequency energy, and the fused high-frequency subband and low-frequency subband are reconstructed by pyramid to generate the intrinsic image without specular highlights.

2. The intelligent sorting system based on machine vision according to claim 1, characterized in that, The multi-polarization spectral image acquisition array includes a polarization state control subarray and a spectral filtering subarray. The polarization state control subarray and the spectral filtering subarray work together to synchronously acquire sub-image sets with different polarization angles and different wavelength combinations under the same spatial coordinates. The physical constraint modeler constructs the polarization-reflection physical constraint model based on Fresnel's law of reflection and polarization Stokes parameters, maps the intensity information of the sub-image set into Stokes parameters containing polarization degree and polarization angle, and separates the diffuse reflection component and the specular reflection component by solving the difference between the orthogonal polarization components in the Stokes parameters.

3. The intelligent sorting system based on machine vision according to claim 2, characterized in that, For transparent material regions where the linear polarization degree in the Stokes parameter exceeds a preset threshold, a polarization degree attenuation factor is introduced to suppress residual polarization crosstalk in the diffuse reflection component. By analyzing the phase difference variation of the orthogonal polarization components at different wavelengths, a multispectral polarization phase consistency constraint function is established. The edge profile of the diffuse reflection component is corrected using the multispectral polarization phase consistency constraint function, thereby eliminating the edge dispersion effect caused by multiple refractions.

4. The intelligent sorting system based on machine vision according to claim 3, characterized in that, The intrinsic image reconstructor extracts the boundary transition region between high-frequency texture features and low-frequency tone features in the diffuse intrinsic image based on the wavelength-spatial frequency joint mapping table. Within the boundary transition region, the local variance of adjacent pixels is calculated, and an edge-preserving smoothing filter kernel is constructed based on the local variance. The edge-preserving smoothing filter kernel is used to perform convolution smoothing on the splicing gap between the high-frequency subband and the low-frequency subband during the pyramid reconstruction process, thereby eliminating spectral splicing artifacts in the intrinsic image without specular highlights.

5. The intelligent sorting system based on machine vision according to claim 4, characterized in that, It also includes an ambient light dynamic compensator, which collects the ambient light spectral distribution and illuminance fluctuation information of the current working environment in real time; Calculate the illumination bias matrix based on the ambient light spectral distribution and illuminance fluctuation information; The illumination bias matrix is ​​superimposed on the polarization-reflection physical constraint model to dynamically correct the decomposition coefficients of the diffuse reflection component and the specular reflection component, thereby compensating for the impact of sudden changes in ambient light on the pixel intensity of the image sequence.

6. The intelligent sorting system based on machine vision according to claim 5, characterized in that, It also includes a sorting coordinate verifier, which maps the sorting coordinates output by the visual detection and sorting locator to the robot's execution space; Extract the three-dimensional normal vector information of the target in the non-highlight intrinsic image, and perform alignment operation between the three-dimensional normal vector information and the preset grasping posture in the execution space to generate a posture correction matrix; Based on the attitude correction matrix, a spatial affine transformation is performed on the sorting coordinates to output the target grasping pose after attitude compensation.

7. A machine vision-based intelligent sorting method, characterized in that, The system described in any one of claims 1 to 6 is used to acquire image sequences of the target in multiple polarization directions and multiple spectral bands. A polarization-reflection physical constraint model is constructed to decompose the pixels in the image sequence into diffuse reflection components and specular reflection components. Calculate the specular mask based on the spatial distribution characteristics of the specular reflection components; Based on the diffuse reflection component and the reflectance of the spectral band, a multispectral intrinsic image mapping relationship is established, and the diffuse reflection intrinsic images under different polarization states are stitched together and reconstructed through an adaptive weighted fusion strategy to generate an intrinsic image without specular highlights. Texture and edge features are extracted from the non-highlight intrinsic image for visual detection and sorting coordinate positioning; The specular mask calculator performs binarization thresholding on the specular reflection component based on the gradient magnitude and local curvature change of the specular reflection component, thereby generating an initial specular region. Perform morphological dilation and void filling operations on the initial specular region to determine the continuous specular mask boundary; Based on the coverage ratio of the highlight mask boundary in the image sequence, a weight allocation factor for the highlight mask is determined. The weight allocation factor is used to suppress the interference of pixels inside the highlight mask on the subsequent feature extraction process. The intrinsic image reconstructor constructs a spectral reflectance matrix based on the spectral absorption and transmission characteristics of different materials in the multispectral bands. Multiply the diffuse reflectance component by the inverse of the spectral reflectance matrix to obtain the diffuse reflectance intrinsic image that eliminates the influence of ambient light. Based on the center wavelength of the multispectral band and the frequency domain response characteristics of the diffuse reflection intrinsic image, a wavelength-spatial frequency joint mapping table is established. The wavelength-spatial frequency joint mapping table is used to guide the calculation of fusion weights for images of different bands in the adaptive weighted fusion strategy. When the intrinsic image reconstructor executes the adaptive weighted fusion strategy, it calculates the structural similarity of the diffuse intrinsic images under different polarization states in the gradient direction. Using the structural similarity as a constraint, Laplacian pyramid decomposition is performed on local pixel blocks in the diffuse intrinsic image; The high-frequency subband and low-frequency subband after decomposition are respectively applied to the fusion rule based on spatial frequency energy, and the fused high-frequency subband and low-frequency subband are reconstructed by pyramid to generate the intrinsic image without specular highlights.

Images