Pitching side type material taking machine material positioning method based on image recognition

Abstract

The invention discloses a pitching side type material taking machine material positioning method based on image recognition. The method comprises the following steps that a structured light sequence is projected to a material stacking area according to a preset coding grating mode; synchronously acquiring multi-view image data of the surface of the stockpile; executing preprocessing; spatial domain processing is carried out, geometric morphology information of the surface of the stockpile is extracted, and height distribution of the surface of the stockpile is analyzed; frequency domain processing is carried out, and light intensity distribution change rule information is extracted; performing fusion processing; time sequence three-dimensional reconstruction operation is executed, and a three-dimensional stockpile surface model is constructed; and determining three-dimensional coordinates of the material target area, and generating material positioning result data. Structured light reconstruction and multi-modal feature fusion are fused, high-precision three-dimensional positioning of the stacked materials is achieved, and the method has the advantages of being precise, robust and high in real-time performance.

Description

Technical Field

[0001] The present invention relates to the field of target positioning, and in particular to a material positioning method of a pitch side reclaimer based on image recognition. Background Art

[0002] At present, pitch-side reclaimers are widely used in material handling and automated operations in stockpiling yards. Related positioning technologies mostly rely on a single visual sensor or laser ranging system to perform two-dimensional or low-precision three-dimensional positioning of material targets in the stockpiling area. Existing technologies generally use traditional structured light or stereo vision methods to achieve surface reconstruction, and usually only focus on the spatial geometric features of point clouds. They lack effective processing for fine-grained texture features under complex surface morphology, motion parameter synchronization under dynamic working conditions, and multimodal information fusion. They are easily affected by ambient light interference, occlusion, surface irregularities and equipment movement, resulting in unstable and inaccurate three-dimensional material positioning results, making it difficult to meet the actual needs of efficient automated material reclaiming and intelligent control.

[0003] To address the above problems, existing technologies lack systematic and engineered innovative solutions in terms of structured light coding patterns, fusion of spatial and frequency domain features, participation of mechanical motion parameters in three-dimensional reconstruction, and voxel grid incremental fusion. High-precision positioning of complex material targets in the stockpiling area is impossible to achieve. Existing solutions are difficult to balance data real-time performance, positioning robustness, and the ability to adapt to dynamic scenarios. New three-dimensional material positioning methods are urgently needed to improve the automation level and operational safety of material retrieving operations. Summary of the Invention

[0004] One purpose of the present invention is to propose a material positioning method for a pitch side reclaimer based on image recognition. The present invention integrates structured light reconstruction and multimodal feature fusion to achieve high-precision three-dimensional positioning of piled materials, and has the advantages of accuracy, robustness and strong real-time performance.

[0005] According to an embodiment of the present invention, a method for positioning materials of a pitch side reclaimer based on image recognition includes the following steps: During the synchronous operation of the pitching mechanism and the lateral traveling mechanism of the pitching side reclaimer, a structured light sequence is projected onto the stockpiling area according to a preset coded grating pattern; The multi-view imaging unit is used to synchronously collect multi-view image data of the pile surface containing structured light stripes to generate a dynamic structured light scanning image sequence; performing preprocessing on the dynamic structured light scanning image sequence to generate a preprocessed structured light scanning image sequence; Perform spatial domain processing on the pre-processed structured light scanning image sequence to extract the geometric morphology information of the pile surface, and analyze the height distribution of the pile surface based on the structured light fringe coding to generate a spatial domain geometric feature data set; Perform frequency domain processing on the pre-processed structured light scanning image sequence to extract the information on the change law of light intensity distribution and generate a frequency domain feature data set; The pitch angle sequence, lateral displacement sequence and rotation angle sequence of the pitch side reclaimer are collected and used as external constraints to be fused with the spatial domain geometric feature data set and the frequency domain feature data set; Based on the fusion processing results, a time-series 3D reconstruction operation is performed to construct a 3D pile surface model; In the three-dimensional pile surface model, the three-dimensional coordinates of the material target area are determined based on the spatial domain geometric feature data set and the frequency domain feature data set, and the material positioning result data is generated.

[0006] Optionally, the projected structured light sequence specifically includes: When the pitching mechanism and the lateral travel mechanism of the pitching side reclaimer enter the synchronous operation state, the control logic of the structured light projection unit is started to monitor the pitching angular velocity and the lateral travel speed. When both are within the preset allowable range, the structured light projection unit is triggered to work; Loading a preset coded grating pattern, which consists of a phase-shifted fringe sequence and a Gray code sequence. The phase-shifted fringe sequence is used to obtain continuous phase information, and the Gray code sequence is used to perform fringe period disambiguation. The two are combined in a fixed order to form a complete round of structured light projection sequence; Set the projection time plan, specify the projection start time and inter-frame interval, send the projection frames in sequence, and record the timestamp information of each frame during the projection process; Calibrate the installation position and attitude parameters of the structured light projection unit in the mechanical coordinate system of the pitch side reclaimer, record the rotation angle and translation parameters, and define the stockpile area as the effective projection coverage range; Set the projection power, brightness threshold, and contrast parameters of the structured light projection unit, and allocate projection power ratios for the phase-shifted fringe sequence and the Gray code sequence respectively; Generate a projection window based on the boundary of the stacking area, and mask irrelevant areas outside the stacking area to ensure that the structured light stripes and Gray code patterns only act on the stacking area within the effective projection coverage range; After a complete round of projection sequence is completed, whether to enter the next round of projection is determined based on the real-time operating status of the pitch side reclaimer. When the pitch angular velocity and the lateral walking speed continue to meet the trigger conditions, a new round of projection is automatically started. Otherwise, the structured light projection unit remains on standby until the start conditions are met again.

[0007] Optionally, the generation of the pre-processed structured light scanning image sequence includes: performing distortion correction on a dynamic structured light scan image sequence; Perform brightness normalization on the corrected image to unify the brightness range of each image frame to a preset standard range; Identify stripe candidate regions in the normalized image, determine the valid area containing structured light stripes by analyzing the local brightness changes, generate stripe candidate regions, and eliminate background regions and noise interference regions; Perform denoising on the candidate stripe regions, smooth the local neighborhood of the image frame, reduce random noise and abnormal pixels, and maintain the edge clarity and overall continuity of the structured light stripes; Extract the center line of the structured light stripes from the denoised image frame, scan the stripe direction point by point, determine the brightness peak and connect them to form a complete stripe center line set; Validity is determined for each frame. If an image is overexposed, underexposed, or has motion blur exceeding a threshold, it is marked as an invalid frame. If the structured light fringes are clear, have complete coverage, and are at normal brightness, it is marked as a valid frame, and a corresponding validity mask is generated. All valid image frames, stripe centerline sets and validity masks are summarized and the preprocessed dynamic structured light scanning image sequence is output.

[0008] Optionally, the generation of the spatial domain geometric feature data set includes: Perform fringe code analysis in the vicinity of the structured light fringe centerline to obtain the package phase distribution, combine it with the Gray code sequence to determine the fringe order, and combine the package phase with the fringe order to obtain the absolute phase distribution; Using the linear mapping relationship between phase and parallax, the absolute phase value is converted into a parallax value, which is used to reflect the relative position difference of the pixel point between the structured light projection unit and the multi-view imaging unit; According to the equivalent binocular geometric relationship, the parallax value is converted into depth information to obtain the height distribution of the pile surface. The height value is related to the projection baseline length and focal length parameters. The camera's intrinsic and extrinsic parameters are used to map the height information of each pixel into a three-dimensional coordinate point, obtaining three-dimensional point cloud data represented by a multi-view imaging unit coordinate system. Establish a local neighborhood in the three-dimensional point cloud data and calculate the normal vector based on the adjacent difference method. The normal vector is used to describe the spatial direction characteristics of the pile surface at that point; The horizontal and vertical rates of change of the height distribution are calculated to obtain the slope index. The standard deviation of the height values ​​in the local neighborhood is calculated as the geometric relief index, and the surface curvature value is approximated by the second-order rate of change. Perform hole filling and boundary clipping on 3D point cloud data, and use validity masks to remove invalid area points caused by overexposure, underexposure or motion blur to obtain a continuous and complete height distribution and normal vector field; Height distribution, three-dimensional point cloud data, normal vector, slope index, geometric relief index and curvature features are integrated to construct a spatial domain geometric feature data set.

[0009] Optionally, the generation of the frequency domain feature data set specifically includes: Receive the pre-processed structured light scanning image sequence and the validity mask, divide the local analysis window with fixed size and fixed step length within the stacking area as the input of frequency domain processing; Perform a weighting operation on the grayscale intensity distribution in each local analysis window; Performing a two-dimensional discrete Fourier transform on the weighted local image to obtain the energy distribution of the grayscale intensity distribution at different frequencies and generate a power spectrum; Perform radial averaging on the power spectrum to obtain a one-dimensional power spectrum, and calculate the energy proportions on the one-dimensional power spectrum according to the preset low-frequency interval, medium-frequency interval, and high-frequency interval to obtain the low-frequency energy proportion, medium-frequency energy proportion, and high-frequency energy proportion, respectively; Determine the main peak frequency corresponding to the energy peak on the power spectrum, and use the calibration constant to convert the main peak frequency into the particle size scale of the pile surface; The power spectrum is linearly fitted in logarithmic coordinates to obtain the spectrum slope, which is used to describe the attenuation law of the surface roughness of the pile as the observation scale changes; Statistically analyze the energy distribution of the power spectrum at different directions and angles, calculate the degree of difference in directional energy, and obtain the directional variance; The power spectrum is divided into multiple frequency bands, and the energy ratio of each frequency band is regarded as a probability distribution to further calculate the texture entropy; Smoothing the frequency domain features in the time series, weighting the frequency domain feature value of the current frame and the smoothing result of the previous frame proportionally, to obtain an updated smoothed feature sequence; Invalid windows caused by overexposure, underexposure or motion blur are eliminated based on the validity mask, and only the features of valid windows are retained. The main peak frequency, particle size, spectral slope, low-frequency energy ratio, medium-frequency energy ratio, high-frequency energy ratio, directional variance, texture entropy and total energy are used as a set of complete frequency domain feature vectors. The frequency domain feature vectors of all valid windows are aggregated according to spatial position and time sequence to construct a frequency domain feature data set.

[0010] Optionally, the fusion processing specifically includes: Receive spatial domain geometric feature data sets and frequency domain feature data sets, and simultaneously receive the pitch angle sequence, lateral displacement sequence, and rotation angle sequence of the pitch side reclaimer. Synchronize the sequences along a unified time axis and align them with a fixed sampling period to form the data input set to be fused. Combining the pitch angle sequence, the lateral displacement sequence and the rotation angle sequence into a motion state vector; In the local area corresponding to the 3D point, the spatial domain geometric feature vector and the frequency domain feature vector are extracted, and the motion state vector aligned with the local area time is used as the external constraint feature to form a comprehensive observation vector containing spatial domain geometric features, frequency features and motion features. Normalization and scale mapping are performed on the spatial domain geometric features, frequency features and motion state vectors respectively, and the three types of features are unified into the same fusion representation space. They are then combined according to preset weight coefficients in the fusion representation space to generate a local fusion vector, which is accumulated region by region and moment by moment to obtain a fusion feature data set consistent with the mechanical coordinate system of the pitch side reclaimer.

[0011] Optionally, the construction of the three-dimensional pile surface model specifically includes: Initialize the voxel grid to cover the pile area, initialize all voxel distances to 0, initialize voxel weights to 0, and form a plane reference voxel field as the initial three-dimensional pile surface model; At each time index, receiving a fused feature data set; The rotation matrix and translation vector at the current moment are concatenated and defined as the pose variable at the current moment; For each observation point in the fused feature data set, find the model point with the closest Euclidean distance in the 3D stockpile surface model at the previous moment, obtain the normal vector corresponding to the model point, and extract the frequency domain features of the observation point and the frequency domain features of the model point at the same time; For each pair of observation points and model points, define the registration cost function; The optimal pose variable is obtained by minimizing the registration cost function, and the registered observation point cloud data is generated; The voxel grid fusion method is used to incrementally fuse the registered observation point cloud data of each frame, and the voxel distance and voxel weight are updated to obtain the updated voxel grid; The zero isosurface is extracted from the updated voxel grid to generate a three-dimensional pile surface model at the current moment, and the normal vector of the three-dimensional pile surface model is calculated based on the gradient of the voxel grid.

[0012] Optionally, the generation of the material positioning result data specifically includes: Perform voxel grid traversal on the three-dimensional pile surface model generated at the current moment, and extract the spatial domain geometric feature data set and frequency domain feature data set of all voxels; According to the spatial domain geometric feature data set and the frequency domain feature data set, threshold judgment is performed on the voxel set to obtain the material target area voxel set; For each target area in the material target area voxel set, extract the three-dimensional coordinate set of all voxels; Determine the three-dimensional coordinates of the weighted centroid of each target area; The three-dimensional coordinates of the weighted centroid of each material target area are used to perform a pitch side reclaimer motion constraint check, eliminating areas that do not meet the motion range and structural constraints to obtain a valid material target area voxel set; The weighted centroid three-dimensional coordinates of all material target areas in the valid material target area voxel set are output as the material target area three-dimensional coordinates to generate material positioning result data.

[0013] The beneficial effects of the present invention are: The image recognition-based material positioning method for a pitch-side reclaimer proposed in this invention breaks through the limitations of traditional two-dimensional or low-precision three-dimensional positioning methods. It fully integrates structured light three-dimensional reconstruction, multi-perspective imaging, spatial domain geometric feature and frequency domain feature extraction, mechanical motion state parameter synchronization, voxel grid three-dimensional modeling, and multi-dimensional feature fusion judgment, achieving high-precision three-dimensional positioning of material targets in complex stockpiling scenarios and dynamic reclaiming conditions. By combining a multi-perspective imaging unit and a structured light projection unit during the synchronous operation of the pitch-side reclaimer, the system can obtain high-quality dynamic structured light scanning image data covering the stockpiling area. Further preprocessing and distortion correction effectively improve the stability of the structured light fringe information and image clarity, providing a high-confidence data foundation for subsequent three-dimensional reconstruction.

[0014] The present invention targets the complex morphological features of the stockpile surface, not only extracting spatial domain geometric feature data sets such as height distribution, normal vector, slope index, geometric undulation and curvature, but also fully mining frequency domain feature information such as main peak frequency, particle size scale, spectral slope, energy distribution, directional variance and texture entropy, realizing the joint expression of multi-dimensional features and local difference discrimination. Through external constraints and data synchronization of mechanical motion parameters, it ensures that the feature data is highly consistent with the actual equipment motion state. In the process of constructing and aligning the three-dimensional stockpile surface model, the incremental fusion and dynamic optimization of voxel grids are adopted to realize continuous and robust modeling of complex material surfaces. The extraction of material target areas is not only based on the judgment and weighting of multi-dimensional features, but also comprehensively considers the physical constraints of the reclaimer motion space, thereby improving the practical usability and engineering adaptability of material positioning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a flow chart of a material positioning method for a pitch side reclaimer based on image recognition proposed by the present invention; Figure 2 This is a schematic diagram of the incremental fusion and reconstruction principle of the voxel grid three-dimensional pile surface model of the material positioning method of the pitch side reclaimer based on image recognition proposed by the present invention. DETAILED DESCRIPTION

[0016] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0017] refer to Figure 1-2 A material positioning method for a pitch side reclaimer based on image recognition comprises the following steps: During the synchronous operation of the pitching mechanism and the lateral traveling mechanism of the pitching side reclaimer, a structured light sequence is projected onto the stockpiling area according to a preset coded grating pattern; The multi-view imaging unit is used to synchronously collect multi-view image data of the pile surface containing structured light stripes to generate a dynamic structured light scanning image sequence; performing preprocessing on the dynamic structured light scanning image sequence to generate a preprocessed structured light scanning image sequence; Perform spatial domain processing on the pre-processed structured light scanning image sequence to extract the geometric morphology information of the pile surface, and analyze the height distribution of the pile surface based on the structured light fringe coding to generate a spatial domain geometric feature data set; Perform frequency domain processing on the pre-processed structured light scanning image sequence to extract the information on the change pattern of light intensity distribution and generate a frequency domain feature data set reflecting the particle size distribution and surface roughness characteristics of the pile surface; The pitch angle sequence, lateral displacement sequence and rotation angle sequence of the pitch side reclaimer are collected and used as external constraints to be fused with the spatial domain geometric feature data set and the frequency domain feature data set; Based on the fusion processing results, a time-series 3D reconstruction operation is performed to construct a 3D pile surface model; In the three-dimensional pile surface model, the three-dimensional coordinates of the material target area are determined based on the spatial domain geometric feature data set and the frequency domain feature data set, and the material positioning result data is generated.

[0018] In this embodiment, the projected structured light sequence specifically includes: When the pitching mechanism and the lateral travel mechanism of the pitching side reclaimer enter the synchronous operation state, the control logic of the structured light projection unit is started to monitor the pitching angular velocity and the lateral travel speed. When both are within the preset allowable range, the structured light projection unit is triggered to work; Loading a preset coded grating pattern, which consists of a phase-shifted fringe sequence and a Gray code sequence. The phase-shifted fringe sequence is used to obtain continuous phase information, and the Gray code sequence is used to perform fringe period disambiguation. The two are combined in a fixed order to form a complete round of structured light projection sequence; Set the projection time plan, specify the projection start time and inter-frame interval, send the projection frames in sequence, and record the timestamp information of each frame during the projection process to ensure consistency with the image acquisition data of the multi-view imaging unit; The installation position and attitude parameters of the structured light projection unit are calibrated in the mechanical coordinate system of the pitch side reclaimer. The rotation angle and translation parameters are recorded to ensure a stable mapping relationship between the structured light stripes and the mechanical coordinate system. The stockpile area is defined as the effective projection coverage range. Set the projection power, brightness threshold, and contrast parameters of the structured light projection unit, and allocate projection power ratios for the phase-shifted fringe sequence and Gray code sequence respectively, so that the projection frame maintains clear contrast in the stacking area and avoids overexposure or insufficient brightness; Generate a projection window based on the boundary of the stacking area, and mask irrelevant areas outside the stacking area to ensure that the structured light stripes and Gray code patterns only act on the stacking area within the effective projection coverage range; During the structured light projection process, the frame number, timestamp, posture parameters of the structured light projection unit, projection power configuration, and projection window information are recorded for each frame to form a projection-side metadata set, which is used to perform temporal alignment and spatial registration with the data collected by the subsequent multi-view imaging unit; After a complete round of projection sequence is completed, whether to enter the next round of projection is determined based on the real-time operating status of the pitch side reclaimer. When the pitch angular velocity and the lateral walking speed continue to meet the trigger conditions, a new round of projection is automatically started. Otherwise, the structured light projection unit remains on standby until the start conditions are met again.

[0019] In this embodiment, the generation of the pre-processed structured light scanning image sequence includes: performing distortion correction on a dynamic structured light scan image sequence; Perform brightness normalization on the corrected image to unify the brightness range of each image frame to a preset standard range. This avoids contrast fluctuations in structured light fringes caused by exposure differences at different imaging nodes and improves fringing clarity through grayscale equalization. Identify stripe candidate regions in the normalized image, determine the valid area containing structured light stripes by analyzing the local brightness changes, generate stripe candidate regions, and eliminate background regions and noise interference regions; Perform denoising on the candidate stripe regions, smooth the local neighborhood of the image frame, reduce random noise and abnormal pixels, and maintain the edge clarity and overall continuity of the structured light stripes; Extract the center line of the structured light stripes on the denoised image frame, scan the stripe direction point by point, determine the brightness peak and connect them to form a complete set of stripe center lines to describe the continuous distribution characteristics of the stripes in the image frame; Validity is determined for each frame. If an image is overexposed, underexposed, or has motion blur exceeding a threshold, it is marked as an invalid frame. If the structured light fringes are clear, have complete coverage, and are at normal brightness, it is marked as a valid frame, and a corresponding validity mask is generated. All valid image frames, stripe centerline sets and validity masks are summarized and the preprocessed dynamic structured light scanning image sequence is output.

[0020] In this embodiment, the generation of the spatial domain geometric feature data set includes: Perform fringe coding analysis in the vicinity of the structured light fringe centerline to obtain the package phase distribution, combine the Gray code sequence to determine the fringe order, and combine the package phase and fringe order to obtain the absolute phase distribution. The absolute phase is used to uniquely determine the height information of each pixel position on the pile surface; Using the linear mapping relationship between phase and parallax, the absolute phase value is converted into a parallax value, which is used to reflect the relative position difference of the pixel point between the structured light projection unit and the multi-view imaging unit; According to the equivalent binocular geometric relationship, the parallax value is converted into depth information to obtain the height distribution of the pile surface. The height value is related to the projection baseline length and focal length parameters. The camera's intrinsic and extrinsic parameters are used to map the height information of each pixel into a three-dimensional coordinate point, obtaining three-dimensional point cloud data represented by a multi-view imaging unit coordinate system. Establish a local neighborhood in the three-dimensional point cloud data and calculate the normal vector based on the adjacent difference method. The normal vector is used to describe the spatial direction characteristics of the pile surface at that point; The rates of change in the horizontal and vertical directions of the height distribution are calculated to obtain a slope index based on this. At the same time, the standard deviation of the height values ​​in the local neighborhood is calculated as a geometric relief index, and the surface curvature value is approximately calculated using the second-order rate of change. The slope index is used to describe the degree of surface inclination, the geometric relief index is used to reflect local unevenness, and the surface curvature value is used to characterize the curved characteristics of the surface. Perform hole filling and boundary clipping on 3D point cloud data, and use validity masks to remove invalid area points caused by overexposure, underexposure or motion blur to obtain a continuous and complete height distribution and normal vector field; Height distribution, three-dimensional point cloud data, normal vector, slope index, geometric relief index and curvature characteristics are integrated to construct a spatial domain geometric feature data set, which is used to fully characterize the geometric morphology of the stockpile surface.

[0021] In this embodiment, the generation of the frequency domain feature data set specifically includes: Receive the pre-processed structured light scanning image sequence and the validity mask, divide the local analysis window with fixed size and fixed step length within the stacking area as the input of frequency domain processing; A weighted operation is performed on the grayscale intensity distribution in each local analysis window, so that the weight of the central area is higher and the weight of the edge area is lower, so as to ensure the stability of the local fringe energy and reduce spectrum leakage; Performing a two-dimensional discrete Fourier transform on the weighted local image to obtain the energy distribution of the grayscale intensity distribution at different frequencies and generate a power spectrum, which is used to characterize the energy characteristics of the pile surface texture at different spatial frequencies; Perform radial averaging on the power spectrum to obtain a one-dimensional power spectrum. Calculate the energy ratios based on the preset low-frequency interval, medium-frequency interval, and high-frequency interval on the one-dimensional power spectrum to obtain the low-frequency energy ratio, medium-frequency energy ratio, and high-frequency energy ratio, respectively, to reflect the texture changes of the pile surface at different scales. Determine the main peak frequency corresponding to the energy peak on the power spectrum, and use a calibration constant to convert the main peak frequency into the particle size scale of the pile surface. The particle size scale is used to reflect the average size of the particle distribution on the pile surface. The calibration constant is determined by the pixel scale and imaging geometry. The power spectrum is linearly fitted in logarithmic coordinates to obtain the spectrum slope, which is used to describe the attenuation law of the surface roughness of the pile as the observation scale changes; Statistically analyzing the energy distribution of the power spectrum at different directional angles, calculating the degree of difference in directional energy, and obtaining directional variance, which characterizes the degree of anisotropy of the surface texture of the pile; The power spectrum is divided into multiple frequency bands, and the energy ratio of each frequency band is regarded as a probability distribution. The texture entropy is further calculated. The texture entropy reflects the complexity and discreteness of the texture energy distribution on the pile surface. The frequency domain features are smoothed in the time series, and the frequency domain feature value of the current frame is proportionally weighted with the smoothing result of the previous frame to obtain an updated smoothed feature sequence, which is used to weaken the fluctuation of a single frame and maintain the continuity of the time series features; Invalid windows caused by overexposure, underexposure or motion blur are eliminated based on the validity mask, and only the features of valid windows are retained. The main peak frequency, particle size, spectral slope, low-frequency energy ratio, medium-frequency energy ratio, high-frequency energy ratio, directional variance, texture entropy and total energy are used as a set of complete frequency domain feature vectors. The frequency domain feature vectors of all valid windows are aggregated according to spatial position and time sequence to construct a frequency domain feature data set, which reflects the particle size distribution characteristics and surface roughness characteristics of the pile surface.

[0022] In this embodiment, the fusion process specifically includes: Receive spatial domain geometric feature data sets and frequency domain feature data sets, and simultaneously receive the pitch angle sequence, lateral displacement sequence, and rotation angle sequence of the pitch side reclaimer. Synchronize the sequences along a unified time axis and align them with a fixed sampling period to form the data input set to be fused. The pitch angle sequence, lateral displacement sequence and rotation angle sequence are combined into a motion state vector, wherein the pitch angle represents the angle of the reclaiming arm relative to the base, the lateral displacement represents the displacement of the reclaimer along the lateral trajectory, and the rotation angle represents the rotation angle of the rotary platform; In the local area corresponding to the 3D point, the spatial domain geometric feature vector and the frequency domain feature vector are extracted, and the motion state vector aligned with the local area time is used as the external constraint feature to form a comprehensive observation vector containing spatial domain geometric features, frequency features and motion features. Normalization and rescaling are performed on the spatial domain geometric features, frequency features, and motion state vectors, respectively, to unify the three types of features into the same fusion representation space. Within the fusion representation space, these features are combined according to preset weight coefficients to generate a local fusion vector. This is then accumulated region by region and moment by moment to obtain a fusion feature data set consistent with the mechanical coordinate system of the pitch side reclaimer. A consistency constraint function is established during the fusion process. The consistency constraint function consists of geometric consistency residuals, normal consistency residuals and motion consistency residuals. The geometric consistency residuals are used to describe the average deviation of three-dimensional points to the local fitting plane, the normal consistency residuals are used to describe the angular difference between the normal vectors of adjacent points, and the motion consistency residuals describe the difference between the real motion state vector and the estimated motion state vector in the fused feature data. By weighted minimization of the three types of residuals, the fused features are kept consistent at the geometric level, normal level and motion level.

[0023] In this embodiment, the construction of the three-dimensional pile surface model specifically includes: Initialize the voxel grid to cover the pile area, initialize all voxel distances to 0, initialize voxel weights to 0, and form a plane reference voxel field as the initial three-dimensional pile surface model; At each time index, receiving a fused feature data set; The rotation matrix and translation vector at the current moment are spliced ​​together to define the pose variable at the current moment. The pose variable describes the spatial transformation relationship between the fused feature data set and the three-dimensional pile surface model. The rotation matrix represents the spatial direction transformation relationship of the point cloud under the action of pitch angle and rotation angle, and the translation vector represents the spatial position offset of the point cloud under the action of lateral displacement; For each observation point in the fused feature data set, find the model point with the closest Euclidean distance in the 3D stockpile surface model at the previous moment, obtain the normal vector corresponding to the model point, and extract the frequency domain features of the observation point and the frequency domain features of the model point at the same time; For each pair of observation points and model points, a registration cost function is defined, which measures the alignment error: ; ; in, Represents the registration cost function, describing the moment The total registration cost is represents the corresponding set of observation points and model points, represents the geometric residual from point to surface, represents the frequency characteristic residual, represents the motion consistency residual, Represents the normal vector corresponding to the model point, represents the increment of the currently estimated pose variable, represents the pose increment corresponding to the true motion state vector, represents the geometric residual weight, represents the frequency residual weight, represents the motion residual weight, represents the frequency domain characteristics of the observation point, Represents the frequency domain characteristics of the corresponding points, represents the rotation matrix, P represents the observation point coordinate vector, represents the coordinate vector of the model point, Represents the pose variable at the current moment; The optimal pose variables are obtained by minimizing the registration cost function, and the registered observation point cloud data is generated to accurately align the fused feature data set with the 3D stockpile surface model; The voxel grid fusion method is used to incrementally fuse the registered observation point cloud data of each frame, and the voxel distance and voxel weight are updated to obtain the updated voxel grid: ; ; in, represents the updated voxel distance, Represents the historical voxel distance, describing the voxel distance accumulated in the previous frame. Represents the historical weight, describing the accumulated weight of the previous frame, Indicates the current observation distance. represents the current weight, represents the updated voxel weight, represents the maximum weight; The zero isosurface is extracted from the updated voxel grid to generate a three-dimensional pile surface model at the current moment, and the normal vector of the three-dimensional pile surface model is calculated based on the gradient of the voxel grid.

[0024] In this embodiment, the generation of the material positioning result data specifically includes: Perform voxel grid traversal on the three-dimensional pile surface model generated at the current moment, and extract the spatial domain geometric feature data set and frequency domain feature data set of all voxels; According to the spatial domain geometric feature data set and the frequency domain feature data set, threshold judgment is performed on the voxel set to obtain the material target area voxel set; The three-dimensional stockpile surface model is traversed by voxel grid. For each voxel, a spatial domain geometric feature data set is extracted, and the above geometric features are statistically analyzed to record the structural features of each voxel in the spatial domain. Simultaneously, a frequency domain feature data set is extracted for each voxel. According to the set spatial domain geometric feature threshold and frequency domain feature threshold, each type of feature of each voxel is judged in turn. If the spatial domain geometric feature and frequency domain feature of the voxel meet the corresponding judgment conditions at the same time, the voxel is classified as a material target area voxel set, otherwise it is classified as a non-target area. Through the above multi-dimensional feature fusion judgment, accurate regional classification of the voxel set is achieved, laying the foundation for the subsequent three-dimensional coordinate extraction of the material target area and the output of the material positioning results.

[0025] For each target area in the material target area voxel set, extract the three-dimensional coordinate set of all voxels; Determine the three-dimensional coordinates of the weighted centroid of each target area: ; in, Indicates the The three-dimensional coordinates of the weighted centroid of the target area of ​​each material, represents the three-dimensional coordinates of the voxel, represents a voxel weighting coefficient, wherein the voxel weighting coefficient is comprehensively determined based on a spatial domain geometric feature data set and a frequency domain feature data set; The three-dimensional coordinates of the weighted centroid of each material target area are used to perform a pitch side reclaimer motion constraint check, eliminating areas that do not meet the motion range and structural constraints to obtain a valid material target area voxel set; The weighted centroid three-dimensional coordinates of all material target areas in the valid material target area voxel set are output as the material target area three-dimensional coordinates to generate material positioning result data.

[0026] Example 1: In order to verify the practical application effect of the present invention, the material positioning method of the pitch side reclaimer based on image recognition proposed in the present invention is applied to the automatic reclaiming system of the raw material yard of a steel enterprise. The yard is large in scale and stores various types of raw materials such as ore and coal. The material stacking pattern is complex and changes frequently. Traditional reclaiming methods generally rely on manual remote control or simple visual positioning, which is difficult to adapt to changeable working conditions and complex environments, resulting in limited positioning accuracy, and the reclaiming efficiency and safety cannot be guaranteed.

[0027] In actual on-site applications, the pitch side reclaimer is arranged in the main channel of the stockpile. The system is equipped with a structured light projection unit and a multi-view imaging unit. The data is synchronized through the control system and the pitch, lateral and rotational movements of the pitch side reclaimer. Whenever the equipment starts the automatic operation process, the structured light projection unit automatically loads the encoding grating according to the real-time movement status of the pitch mechanism and the lateral walking mechanism, and projects a sequence of structured light stripes onto the surface of the stockpile according to the established plan. The multi-view imaging unit simultaneously captures dynamic images of the stockpile surface with structured light stripes at multiple positions, and transmits the original data to the data processing center in real time through the system's high-speed bus. In order to overcome the influence of ambient light fluctuations and dust obstruction in the stockpile, the system adopts image preprocessing algorithms such as multi-frame image fusion, stripe centerline extraction, and automatic elimination of invalid frames, which effectively improves the accuracy of structured light stripe recognition and provides high-quality input for subsequent three-dimensional reconstruction.

[0028] The pre-processed data is input into the spatial domain geometric feature and frequency domain feature extraction module. The system performs multi-scale, multi-region joint statistics and analysis on spatial domain features such as height distribution, normal vector, slope, undulation and curvature, as well as frequency domain features such as main peak frequency, particle size scale, spectral slope, directional variance, and texture entropy. This processing method not only achieves a fine restoration of the spatial structure of the material surface, but also fully captures micro-texture features such as particle distribution and surface roughness. At the same time, the motion state data of the reclaimer body - including pitch angle, lateral displacement and rotation angle - is collected on a unified time axis, and external constraints and synchronization are achieved through motion state vectors and spatial-frequency features, providing time-space consistent input for three-dimensional modeling and positioning.

[0029] The system uses a voxel grid method to construct a 3D stockpile surface model. Initially, a standard voxel grid is established for the stockpile area, and all voxel distances and weights are initialized. During continuous operation, the processing center spatially aligns the fused feature data set with the previous 3D stockpile surface model. Combined with the current equipment's position variables, the processing center performs voxel-level incremental fusion on each frame of registered observation point cloud data, dynamically updating voxel distances and weights. Every few seconds, the system automatically extracts the updated zero-isosurface as the new 3D stockpile surface model, continuously improving the model's accuracy and surface normal information accordingly.

[0030] To achieve precise positioning of the target material area, the system performs a voxel grid traversal on the three-dimensional stockpile surface model, jointly determining the spatial domain geometric features and frequency domain features of all voxels. After multi-threshold fusion screening of spatial-frequency features, it can reliably distinguish between areas of different physical states and particle sizes on the stockpile surface. Furthermore, through three-dimensional connectivity analysis and physical space constraints, it automatically eliminates areas that are unreachable, structurally abnormal, or misidentified. For each valid material target area, the system calculates the three-dimensional coordinates using the voxel weighted centroid method and maps them directly to the mechanical coordinate system of the pitch-side reclaimer, achieving high-precision three-dimensional positioning of the material target area. The final material positioning result data, as the basic input for automatic material reclaiming operation path planning and execution, is pushed to the control unit in real time, driving the pitch-side reclaimer to complete automatic alignment and material grabbing.

[0031] In order to verify the performance of the present invention in practice, a comparison was made with the traditional method, and the results are shown in Table 1.

[0032] Table 1 Performance comparison test results of material positioning methods for pitch side reclaimer

[0033] As can be seen intuitively from Table 1, the method of the present invention outperforms traditional methods in all key performance indicators. In terms of positioning accuracy, traditional stereo vision relies on pixel matching and is severely affected by the irregularities of the pile surface and lighting, with an average error of more than 15 cm. Single structured light reconstruction can improve some of the errors, but there are still large fluctuations under dynamic conditions. The present invention, through the joint analysis of spatial domain geometric features and frequency domain features, combined with the motion parameter constraints of the pitch side reclaimer, makes the alignment of the model in three-dimensional space more rigorous, reducing the average error to within 6 cm, showing a significant improvement.

[0034] In terms of positioning success rate, the present invention achieved nearly 91%, which is more than 7 percentage points higher than traditional stereo vision. The reason for this improvement is that the stripe centerline extraction and invalid frame elimination mechanism are used to make the input data more effective. At the same time, multimodal feature fusion avoids misidentification caused by single-dimensional discrimination.

[0035] In terms of data processing frame rate, the present invention achieves 24fps, which is higher than the traditional 15 to 18fps, indicating that it can meet the continuous processing requirements of real-time dynamic stacking scenarios. The fundamental reason for the improvement lies in the use of an efficient voxel incremental fusion strategy, which reduces the amount of repeated calculations while ensuring model accuracy, thereby speeding up the overall computing speed.

[0036] The stability and automatic material reclaim completion rate data also reflect the advantages of this invention. Traditional methods have low stability and are significantly affected by noise and equipment jitter. However, this invention uses a consistency constraint function to jointly optimize the geometric residual, normal residual, and motion residual, enabling the system to maintain continuous and stable performance under dynamic conditions. The final automatic material reclaim completion rate exceeded 94%, an increase of approximately 14 percentage points over the traditional method. This means that in actual operations, this invention can not only more accurately identify the target material area, but also effectively support subsequent material reclaiming actions, reducing operational risks and delays.

[0037] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A material positioning method for a pitch side reclaimer based on image recognition, characterized in that: The steps include: During the synchronous operation of the pitching mechanism and the lateral traveling mechanism of the pitching side reclaimer, a structured light sequence is projected onto the stockpiling area according to a preset coded grating pattern; The multi-view imaging unit is used to synchronously collect multi-view image data of the pile surface containing structured light stripes to generate a dynamic structured light scanning image sequence; performing preprocessing on the dynamic structured light scanning image sequence to generate a preprocessed structured light scanning image sequence; Perform spatial domain processing on the pre-processed structured light scanning image sequence to extract the geometric morphology information of the pile surface, and analyze the height distribution of the pile surface based on the structured light fringe coding to generate a spatial domain geometric feature data set; Perform frequency domain processing on the pre-processed structured light scanning image sequence to extract the information on the change law of light intensity distribution and generate a frequency domain feature data set; The pitch angle sequence, lateral displacement sequence and rotation angle sequence of the pitch side reclaimer are collected and used as external constraints to be fused with the spatial domain geometric feature data set and the frequency domain feature data set; Based on the fusion processing results, a time-series 3D reconstruction operation is performed to construct a 3D pile surface model; In the three-dimensional pile surface model, the three-dimensional coordinates of the material target area are determined based on the spatial domain geometric feature data set and the frequency domain feature data set, and the material positioning result data is generated.

2. The material positioning method of a pitch side reclaimer based on image recognition according to claim 1, characterized in that: The projected structured light sequence specifically includes: When the pitching mechanism and the lateral travel mechanism of the pitching side reclaimer enter the synchronous operation state, the control logic of the structured light projection unit is started to monitor the pitching angular velocity and the lateral travel speed. When both are within the preset allowable range, the structured light projection unit is triggered to work; Loading a preset coded grating pattern, which consists of a phase-shifted fringe sequence and a Gray code sequence. The phase-shifted fringe sequence is used to obtain continuous phase information, and the Gray code sequence is used to perform fringe period disambiguation. The two are combined in a fixed order to form a complete round of structured light projection sequence; Set the projection time plan, specify the projection start time and inter-frame interval, send the projection frames in sequence, and record the timestamp information of each frame during the projection process; Calibrate the installation position and attitude parameters of the structured light projection unit in the mechanical coordinate system of the pitch side reclaimer, record the rotation angle and translation parameters, and define the stockpile area as the effective projection coverage range; Set the projection power, brightness threshold, and contrast parameters of the structured light projection unit, and allocate projection power ratios for the phase-shifted fringe sequence and the Gray code sequence respectively; Generate a projection window based on the boundary of the stacking area, and mask irrelevant areas outside the stacking area to ensure that the structured light stripes and Gray code patterns only act on the stacking area within the effective projection coverage range; After a complete round of projection sequence is completed, whether to enter the next round of projection is determined based on the real-time operating status of the pitch side reclaimer. When the pitch angular velocity and the lateral walking speed continue to meet the trigger conditions, a new round of projection is automatically started. Otherwise, the structured light projection unit remains on standby until the start conditions are met again.

3. The material positioning method of a pitch side reclaimer based on image recognition according to claim 1, characterized in that: The generation of the pre-processed structured light scanning image sequence includes: performing distortion correction on a dynamic structured light scan image sequence; Perform brightness normalization on the corrected image to unify the brightness range of each image frame to a preset standard range; Identify stripe candidate regions in the normalized image, determine the valid area containing structured light stripes by analyzing the local brightness changes, generate stripe candidate regions, and eliminate background regions and noise interference regions; Perform denoising on the candidate stripe regions, smooth the local neighborhood of the image frame, reduce random noise and abnormal pixels, and maintain the edge clarity and overall continuity of the structured light stripes; Extract the center line of the structured light stripes from the denoised image frame, scan the stripe direction point by point, determine the brightness peak and connect them to form a complete stripe center line set; Validity is determined for each frame. If an image is overexposed, underexposed, or has motion blur exceeding a threshold, it is marked as an invalid frame. If the structured light fringes are clear, have complete coverage, and are at normal brightness, it is marked as a valid frame, and a corresponding validity mask is generated. All valid image frames, stripe centerline sets and validity masks are summarized and the preprocessed dynamic structured light scanning image sequence is output.

4. The material positioning method of a pitch side reclaimer based on image recognition according to claim 1, characterized in that: The generation of the spatial domain geometric feature data set includes: Perform fringe code analysis in the vicinity of the structured light fringe centerline to obtain the package phase distribution, combine it with the Gray code sequence to determine the fringe order, and combine the package phase with the fringe order to obtain the absolute phase distribution; Using the linear mapping relationship between phase and parallax, the absolute phase value is converted into a parallax value, which is used to reflect the relative position difference of the pixel point between the structured light projection unit and the multi-view imaging unit; According to the equivalent binocular geometric relationship, the parallax value is converted into depth information to obtain the height distribution of the pile surface. The height value is related to the projection baseline length and focal length parameters. The camera's intrinsic and extrinsic parameters are used to map the height information of each pixel into a three-dimensional coordinate point, obtaining three-dimensional point cloud data represented by a multi-view imaging unit coordinate system. Establishing a local neighborhood in the three-dimensional point cloud data and calculating a normal vector based on the adjacent difference method. The normal vector is used to describe the spatial direction characteristics of the pile surface at that point; The horizontal and vertical rates of change of the height distribution are calculated to obtain the slope index. The standard deviation of the height values ​​in the local neighborhood is calculated as the geometric relief index, and the surface curvature value is approximated by the second-order rate of change. Perform hole filling and boundary clipping on 3D point cloud data, and use validity masks to remove invalid area points caused by overexposure, underexposure or motion blur to obtain a continuous and complete height distribution and normal vector field; Height distribution, three-dimensional point cloud data, normal vector, slope index, geometric relief index and curvature features are integrated to construct a spatial domain geometric feature data set.

5. The material positioning method of a pitch side reclaimer based on image recognition according to claim 1, characterized in that: The generation of the frequency domain feature data set specifically includes: Receive the pre-processed structured light scanning image sequence and the validity mask, divide the local analysis window with fixed size and fixed step length within the stacking area as the input of frequency domain processing; Perform a weighting operation on the grayscale intensity distribution in each local analysis window; Perform a two-dimensional discrete Fourier transform on the weighted local image to obtain the energy distribution of the grayscale intensity distribution at different frequencies and generate a power spectrum; Perform radial averaging on the power spectrum to obtain a one-dimensional power spectrum, and calculate the energy proportions on the one-dimensional power spectrum according to the preset low-frequency interval, medium-frequency interval, and high-frequency interval to obtain the low-frequency energy proportion, medium-frequency energy proportion, and high-frequency energy proportion, respectively; Determine the main peak frequency corresponding to the energy peak on the power spectrum, and use the calibration constant to convert the main peak frequency into the particle size scale of the pile surface; The power spectrum is linearly fitted in logarithmic coordinates to obtain the spectrum slope, which is used to describe the attenuation law of the surface roughness of the pile as the observation scale changes; Statistically analyze the energy distribution of the power spectrum at different directions and angles, calculate the degree of difference in directional energy, and obtain the directional variance; The power spectrum is divided into multiple frequency bands, and the energy ratio of each frequency band is regarded as a probability distribution to further calculate the texture entropy; Smoothing the frequency domain features in the time series, weighting the frequency domain feature value of the current frame and the smoothing result of the previous frame proportionally, to obtain an updated smoothed feature sequence; Invalid windows caused by overexposure, underexposure or motion blur are eliminated based on the validity mask, and only the features of valid windows are retained. The main peak frequency, particle size, spectral slope, low-frequency energy ratio, medium-frequency energy ratio, high-frequency energy ratio, directional variance, texture entropy and total energy are used as a set of complete frequency domain feature vectors. The frequency domain feature vectors of all valid windows are aggregated according to spatial position and time sequence to construct a frequency domain feature data set.

6. The material positioning method of a pitch side reclaimer based on image recognition according to claim 1, characterized in that: The fusion processing specifically includes: Receive spatial domain geometric feature data sets and frequency domain feature data sets, and simultaneously receive the pitch angle sequence, lateral displacement sequence, and rotation angle sequence of the pitch side reclaimer. Synchronize the sequences along a unified time axis and align them with a fixed sampling period to form the data input set to be fused. Combining the pitch angle sequence, the lateral displacement sequence and the rotation angle sequence into a motion state vector; In the local area corresponding to the 3D point, the spatial domain geometric feature vector and the frequency domain feature vector are extracted, and the motion state vector aligned with the local area time is used as the external constraint feature to form a comprehensive observation vector containing spatial domain geometric features, frequency features and motion features. Normalization and scale mapping are performed on the spatial domain geometric features, frequency features and motion state vectors respectively, and the three types of features are unified into the same fusion representation space. They are then combined according to preset weight coefficients in the fusion representation space to generate a local fusion vector, which is accumulated region by region and moment by moment to obtain a fusion feature data set consistent with the mechanical coordinate system of the pitch side reclaimer.

7. The material positioning method of a pitch side reclaimer based on image recognition according to claim 1, characterized in that: The construction of the three-dimensional pile surface model specifically includes: Initialize the voxel grid to cover the pile area, initialize all voxel distances to 0, initialize voxel weights to 0, and form a plane reference voxel field as the initial three-dimensional pile surface model; At each time index, receiving a fused feature data set; The rotation matrix and translation vector at the current moment are concatenated and defined as the pose variable at the current moment; For each observation point in the fused feature data set, find the model point with the closest Euclidean distance in the 3D stockpile surface model at the previous moment, obtain the normal vector corresponding to the model point, and extract the frequency domain features of the observation point and the frequency domain features of the model point at the same time; For each pair of observation points and model points, define the registration cost function; The optimal pose variable is obtained by minimizing the registration cost function, and the registered observation point cloud data is generated; The voxel grid fusion method is used to incrementally fuse the registered observation point cloud data of each frame, and the voxel distance and voxel weight are updated to obtain the updated voxel grid; The zero isosurface is extracted from the updated voxel grid to generate a three-dimensional pile surface model at the current moment, and the normal vector of the three-dimensional pile surface model is calculated based on the gradient of the voxel grid.

8. The material positioning method of a pitch side reclaimer based on image recognition according to claim 1, characterized in that: The generation of the material positioning result data specifically includes: Perform voxel grid traversal on the three-dimensional pile surface model generated at the current moment, and extract the spatial domain geometric feature data set and frequency domain feature data set of all voxels; According to the spatial domain geometric feature data set and the frequency domain feature data set, threshold judgment is performed on the voxel set to obtain the material target area voxel set; For each target area in the material target area voxel set, extract the three-dimensional coordinate set of all voxels; Determine the three-dimensional coordinates of the weighted centroid of each target area; The three-dimensional coordinates of the weighted centroid of each material target area are used to perform a pitch side reclaimer motion constraint check, eliminating areas that do not meet the motion range and structural constraints to obtain a valid material target area voxel set; The weighted centroid three-dimensional coordinates of all material target areas in the valid material target area voxel set are output as the material target area three-dimensional coordinates to generate material positioning result data.

Images