Image recognition based material positioning method for luffing side-type material handler

Abstract

The application discloses a kind of based on image recognition's pitch side formula material positioning method of material handling machine, comprising the following steps: according to preset encoding grating mode, structure light sequence is projected to stacking area;Synchronous acquisition multi-view image data of stacking surface;Execute pre-processing;Carry out spatial domain processing, extract stacking surface geometry information, and analyze stacking surface height distribution;Carry out frequency domain processing, extract light intensity distribution variation law information;Carry out fusion processing;Execute timing three-dimensional reconstruction operation, construct three-dimensional stacking surface model;Determine the three-dimensional coordinates of material target area, generate material positioning result data.The application fuses structure light reconstruction and multimodal feature fusion, realizes stacking material high-precision three-dimensional positioning, with the advantages of accurate, robust and strong real-time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target positioning, and in particular to a pitch-lateral type material positioning method based on image recognition. BACKGROUND

[0002] Currently, the pitch-lateral type material positioning method is widely used in material handling and automatic operation in the stockyard, and the related positioning technology mainly depends on a single visual sensor or laser ranging system to perform two-dimensional or low-precision three-dimensional positioning on the material target in the stockyard area. The prior art generally uses a traditional structured light or stereo vision method to realize surface reconstruction, and usually only focuses on the spatial geometric features of the point cloud. However, for the fine-grained texture features under complex surface morphology, the synchronization of motion parameters under dynamic working conditions, and the multi-modal information fusion, there is a lack of effective processing, which is easily affected by environmental light interference, occlusion, irregular surface and equipment motion, resulting in unstable and inaccurate three-dimensional positioning results of the material, and it is difficult to meet the actual needs of efficient automatic material taking and intelligent control.

[0003] In view of the above problems, the prior art lacks systematic and engineering innovative solutions in terms of structured light encoding mode, spatial domain and frequency domain feature fusion, mechanical motion parameter participation in three-dimensional reconstruction, and voxel grid incremental fusion, and cannot realize high-precision positioning of complex material targets in the stockyard area. The existing scheme cannot balance the data real-time performance, positioning robustness and adaptability to dynamic scenes, and a new three-dimensional material positioning method is urgently needed to improve the automation level and operation safety of the material taking operation. SUMMARY

[0004] An object of the present application is to provide a pitch-lateral type material positioning method based on image recognition. The present application combines structured light reconstruction and multi-modal feature fusion to realize high-precision three-dimensional positioning of the stockyard material, and has the advantages of precision, robustness and real-time performance.

[0005] According to the pitch-lateral type material positioning method based on image recognition of the present application, the following steps are included:

[0006] During the synchronous operation of the pitch mechanism and the lateral walking mechanism of the pitch-lateral type material positioning method, a structured light sequence is projected onto the stockyard area according to a preset coded grating pattern;

[0007] The multi-view imaging unit is used to synchronously collect multi-view image data of the stockyard surface containing structured light stripes, and generate a dynamic structured light scanning image sequence;

[0008] The dynamic structured light scanning image sequence is preprocessed to generate a preprocessed structured light scanning image sequence;

[0009] The pre-processed structured light scanning image sequence is subjected to spatial domain processing to extract the geometric morphology information of the stockpile surface, and the stockpile surface height distribution is analyzed based on the structured light fringe coding to generate a set of spatial domain geometric feature data;

[0010] The pre-processed structured light scanning image sequence is subjected to frequency domain processing to extract the light intensity distribution variation information to generate a set of frequency domain feature data;

[0011] The pitch angle sequence, lateral displacement sequence and rotation angle sequence of the pitch-lateral type reclaimer are collected as external constraints, and are subjected to fusion processing with the set of spatial domain geometric feature data and the set of frequency domain feature data;

[0012] Based on the fusion processing result, a time sequence three-dimensional reconstruction operation is performed to construct a three-dimensional stockpile surface model;

[0013] In the three-dimensional stockpile surface model, the three-dimensional coordinates of the material target region are determined based on the set of spatial domain geometric feature data and the set of frequency domain feature data to generate material positioning result data.

[0014] Optionally, the sequence of projected structured light specifically includes:

[0015] When the pitch mechanism and the lateral walking mechanism of the pitch-lateral type reclaimer enter a synchronous running state, the control logic of the structured light projection unit is started to monitor the pitch angle speed and the lateral walking speed, and when both are within a preset allowable range, the structured light projection unit is triggered to work;

[0016] A preset coded grating pattern is loaded, which is composed of a phase shift fringe sequence and a Gray code sequence, the phase shift fringe sequence is used to obtain continuous phase information, and the Gray code sequence is used to perform fringe period disambiguation, and both are combined in a fixed order to form a complete structured light projection sequence;

[0017] A projection time plan is set to specify the projection start time and the inter-frame interval, and the projection frames are sequentially issued, and the timestamp information of each frame is recorded during the projection process;

[0018] The installation position and attitude parameters of the structured light projection unit are calibrated in the mechanical coordinate system of the pitch-lateral type reclaimer, the rotation angle and translation parameters are recorded, and the stockpile area is limited as the effective projection coverage range;

[0019] The projection power, brightness threshold and contrast parameters of the structured light projection unit are set, and the projection power proportion is allocated to the phase shift fringe sequence and the Gray code sequence respectively;

[0020] A projection window is generated based on the boundary of the stockpile area, the irrelevant areas outside the stockpile area are subjected to shielding processing, and it is ensured that the structured light fringe and the Gray code pattern only act on the stockpile area within the effective projection coverage range.

[0021] After a complete sequence of projection is finished, it is determined whether to enter the next sequence of projection according to the real-time running state of the luffing side type material taking machine, when the luffing angle velocity and the lateral walking velocity continue to satisfy the triggering condition, a new sequence of projection is automatically started, otherwise the structured light projection unit is kept standby until the starting condition is satisfied again.

[0022] Optionally, the generation of the pre-processed structured light scanning image sequence comprises:

[0023] Performing distortion correction on the dynamic structured light scanning image sequence;

[0024] Performing brightness normalization processing on the corrected image, and unifying the brightness range of each image frame to a preset standard interval;

[0025] Identifying a fringe candidate region in the normalized image, determining an effective region containing the structured light fringe by analyzing the local brightness change, generating the fringe candidate region, and eliminating the background region and noise interference region;

[0026] Performing denoising processing on the fringe candidate region, smoothing the local neighborhood of the image frame, weakening random noise and abnormal pixel points, and maintaining the edge definition and overall continuity of the structured light fringe;

[0027] Extracting a structured light fringe center line on the denoised image frame, scanning the fringe direction point by point, determining the brightness peak value and connecting to form a complete set of fringe center lines;

[0028] Performing effectiveness determination on each image frame, marking the image as an invalid frame when overexposure, underexposure or motion blur exceeds the threshold, marking the image as a valid frame when the structured light fringe is clear, complete and normal in brightness, and generating a corresponding effectiveness mask;

[0029] Summarizing all valid image frames, fringe center line sets and effectiveness masks, and outputting the pre-processed dynamic structured light scanning image sequence.

[0030] Optionally, the generation of the spatial domain geometric feature data set comprises:

[0031] Performing fringe code analysis in the neighborhood range of the structured light fringe center line, obtaining wrapped phase distribution, combining the Gray code sequence to determine the fringe order, and combining the wrapped phase and the fringe order to obtain absolute phase distribution;

[0032] Using the linear mapping relationship between the phase and the disparity, converting the absolute phase value into a disparity value, and the disparity value is used to reflect the relative position difference of the pixel points between the structured light projection unit and the multi-view imaging unit;

[0033] According to the equivalent binocular geometry relationship, the parallax value is converted into depth information to obtain the height distribution of the stockpile surface, and the height value is related to the projection baseline length and the focal length parameter;

[0034] The height information of each pixel point is mapped into a three-dimensional coordinate point by using the camera intrinsic and extrinsic parameters to obtain three-dimensional point cloud data represented in the multi-view imaging unit coordinate system;

[0035] A local neighborhood is established in the three-dimensional point cloud data, and a normal vector is calculated based on the adjacent difference method, the normal vector being used to describe the spatial direction characteristics of the stockpile surface at the point position;

[0036] The change rates in the horizontal and vertical directions are calculated on the height distribution, and the slope index is obtained accordingly, and the standard deviation of the height value in the local neighborhood is calculated as the geometric undulation index, and the surface curvature value is calculated by the second-order change rate approximation;

[0037] The three-dimensional point cloud data is executed for hole repair and boundary clipping, and the invalid area points generated due to overexposure, underexposure or motion blur are removed by using the validity mask to obtain a continuous and complete height distribution and normal vector field;

[0038] The height distribution, three-dimensional point cloud data, normal vector, slope index, geometric undulation index and curvature characteristics are integrated to construct a spatial domain geometric feature data set.

[0039] Optionally, the generation of the frequency domain feature data set specifically includes:

[0040] The pre-processed structured light scanning image sequence and the validity mask are received, and a local analysis window of a fixed size and a fixed step is divided in the stockpile area range as an input for frequency domain processing;

[0041] A weighted operation is performed on the gray intensity distribution in each local analysis window;

[0042] A two-dimensional discrete Fourier transform is performed on the weighted local image to obtain the energy distribution of the gray intensity distribution at different frequencies, and a power spectrum is generated;

[0043] A radial average is performed on the power spectrum to obtain a one-dimensional power spectrum, and the energy proportion is calculated on the one-dimensional power spectrum according to a preset low-frequency interval, a medium-frequency interval and a high-frequency interval to obtain a low-frequency energy proportion, a medium-frequency energy proportion and a high-frequency energy proportion, respectively;

[0044] The main peak frequency corresponding to the energy peak value is determined on the power spectrum, and the main peak frequency is converted into the particle size scale of the stockpile surface by using a calibration constant;

[0045] The power spectrum is linearly fitted in a logarithmic coordinate to obtain a spectral slope, the spectral slope being used to describe the attenuation law of the roughness of the stockpile surface when the observation scale changes.

[0046] The statistical power spectrum is used to calculate the energy distribution in different direction angles, the difference degree of the direction energy, and the direction variance;

[0047] The power spectrum is divided into multiple frequency bands, and the energy proportion of each frequency band is regarded as a probability distribution, and the texture entropy is further calculated;

[0048] The frequency domain feature values of the current frame and the smoothing result of the previous frame are proportionally weighted to obtain an updated smoothing feature sequence.

[0049] According to the validity mask, the invalid window caused by overexposure, underexposure or motion blur is removed, only the features of the valid window are retained, and the main peak frequency, particle size scale, spectral slope, low-frequency energy proportion, medium-frequency energy proportion, high-frequency energy proportion, direction variance, texture entropy and total energy are taken as a complete frequency domain feature vector;

[0050] The frequency domain feature vectors of all valid windows are summarized according to the spatial position and time sequence to construct a frequency domain feature data set.

[0051] Optionally, the fusion processing specifically includes:

[0052] The spatial domain geometric feature data set and the frequency domain feature data set are received, and the pitch angle sequence, the lateral displacement sequence and the rotation angle sequence of the pitch-lateral type reclaimer are received, the sequences are synchronized according to a unified time axis, and the alignment is completed at a fixed sampling period to form a data input set to be fused;

[0053] The pitch angle sequence, the lateral displacement sequence and the rotation angle sequence are combined into a motion state vector;

[0054] In the local area corresponding to the three-dimensional point, the spatial domain geometric feature vector and the frequency domain feature vector are extracted, and the motion state vector time-aligned with the local area is taken as an external constraint feature to form a comprehensive observation vector containing spatial domain geometric features, frequency features and motion features;

[0055] The spatial domain geometric features, the frequency features and the motion state vector are respectively normalized and scaled to unify the three types of features to the same fusion representation space, and the three types of features are combined in the fusion representation space according to the preset weight coefficients to generate a local fusion vector, which is accumulated region by region and time by time to obtain a fusion feature data set consistent with the mechanical coordinate system of the pitch-lateral type reclaimer.

[0056] Optionally, the construction of the three-dimensional stacking surface model specifically includes:

[0057] Initialize the voxel grid to cover the stockyard area, all voxel distances are initialized as 0, and voxel weights are initialized as 0, form a plane reference voxel field as an initial three-dimensional stockyard surface model;

[0058] At each time index, a set of fused feature data is received;

[0059] Concatenate the rotation matrix and translation vector at the current time to define a pose variable at the current time;

[0060] For each observation point in the set of fused feature data, find the model point with the closest Euclidean distance in the three-dimensional stockyard surface model at the previous time, and obtain the normal vector corresponding to the model point, while extracting the frequency domain features of the observation point and the frequency domain features of the model point;

[0061] For each pair of observation points and model points, define a registration cost function;

[0062] Obtain the optimal pose variable by minimizing the registration cost function to generate the registered observation point cloud data;

[0063] Using the voxel grid fusion method, incrementally fuse the registered observation point cloud data of each frame, update the voxel distance and voxel weight, and obtain the updated voxel grid;

[0064] Extract the zero-equal surface from the updated voxel grid to generate the three-dimensional stockyard surface model at the current time, and calculate the normal vector of the three-dimensional stockyard surface model according to the gradient of the voxel grid.

[0065] Optionally, the generation of the material positioning result data specifically includes:

[0066] Perform voxel grid traversal on the three-dimensional stockyard surface model generated at the current time to extract the set of spatial domain geometric feature data and the set of frequency domain feature data of all voxels;

[0067] According to the set of spatial domain geometric feature data and the set of frequency domain feature data, perform threshold judgment on the voxel set to obtain a set of material target region voxels;

[0068] For each target region in the set of material target region voxels, extract a set of three-dimensional coordinates of all voxels;

[0069] Determine the weighted centroid three-dimensional coordinates of each target region;

[0070] Perform pitch-side type reclaimer motion space constraint verification on the weighted centroid three-dimensional coordinates of each material target region, eliminate regions that do not meet the motion range and structure constraints, and obtain a set of effective material target region voxels;

[0071] Output the weighted centroid three-dimensional coordinates of all material target regions in the effective material target region voxel set as the material target region three-dimensional coordinates to generate the material positioning result data.

[0072] The beneficial effects of the present application are:

[0073] The luffing side type material positioning method based on image recognition provided by the present application breaks through the limitations of traditional two-dimensional or low-precision three-dimensional positioning methods, fully integrates structured light three-dimensional reconstruction, multi-view imaging, spatial domain geometric feature and frequency domain feature extraction, mechanical motion state parameter synchronization, voxel grid three-dimensional modeling and multi-dimensional feature fusion determination, and realizes high-precision three-dimensional positioning of the material target in a complex stacking scene and a dynamic material taking working condition.

[0074] The present application not only extracts spatial domain geometric feature data sets such as height distribution, normal vector, slope index, geometric relief and curvature, but also fully excavates frequency domain feature information such as main peak frequency, particle size scale, spectral slope, energy distribution, direction variance and texture entropy, realizes joint expression and local difference discrimination of multi-dimensional features, ensures high consistency of feature data and actual equipment motion state through external constraint and data synchronization of mechanical motion parameters, and realizes continuous and robust modeling of the complex material surface by using voxel grid incremental fusion and dynamic optimization in the three-dimensional stacking surface model construction and registration process, so that the extraction of the material target region is not only based on the determination and weighting of multi-dimensional features, but also considers the physical constraints of the material taking machine motion space, and improves the actual availability and engineering adaptability of the material positioning result. BRIEF DESCRIPTION OF DRAWINGS

[0075] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the present application and are used to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0076] Fig. 1 A flowchart of a luffing side type material positioning method based on image recognition provided by the present application is shown in the figure.

[0077] Fig. 2 A voxel grid three-dimensional stacking surface model incremental fusion and reconstruction principle diagram of a luffing side type material positioning method based on image recognition provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0078] The application will be described in further detail below with reference to the drawings. These drawings are simplified schematic diagrams and only show the basic structure of the application in a schematic manner, and thus only show the components relevant to the application.

[0079] Reference Figs. 1-2 An image recognition-based material positioning method for a luffing-side type material handler, comprising the following steps:

[0080] During synchronous operation of the luffing mechanism and the lateral traveling mechanism of the luffing-side type material handler, a structured light sequence is projected to the stacking area according to a preset coded grating pattern;

[0081] Multi-view imaging units are used to synchronously collect multi-view image data of the stacking surface containing structured light fringes, to generate a dynamic structured light scanning image sequence;

[0082] The dynamic structured light scanning image sequence is preprocessed to generate a preprocessed structured light scanning image sequence;

[0083] The preprocessed structured light scanning image sequence is subjected to spatial domain processing, to extract the geometric morphological information of the stacking surface and analyze the height distribution of the stacking surface based on the structured light fringe coding, to generate a spatial domain geometric feature data set;

[0084] The preprocessed structured light scanning image sequence is subjected to frequency domain processing, to extract the light intensity distribution variation law information, to generate a frequency domain feature data set reflecting the particle size distribution and surface roughness characteristics of the stacking surface;

[0085] The luffing angle sequence, lateral displacement sequence and rotation angle sequence of the luffing-side type material handler are collected and used as external constraints, and are fused with the spatial domain geometric feature data set and the frequency domain feature data set;

[0086] Based on the fusion processing result, a time sequence three-dimensional reconstruction operation is performed to construct a three-dimensional stacking surface model;

[0087] In the three-dimensional stacking surface model, the three-dimensional coordinates of the material target area are determined based on the spatial domain geometric feature data set and the frequency domain feature data set, to generate material positioning result data.

[0088] In this embodiment, the projected structured light sequence specifically comprises:

[0089] When the luffing mechanism and the lateral traveling mechanism of the luffing-side type material handler enter a synchronous operation state, the control logic of the structured light projection unit is started, the luffing angle velocity and the lateral traveling velocity are monitored, and the structured light projection unit is triggered to work when both are within a preset allowable range;

[0090] Load a preset coded fringe pattern, which is composed of a phase-shift fringe sequence and a Gray code sequence, the phase-shift fringe sequence is used to obtain continuous phase information, and the Gray code sequence is used to perform fringe period disambiguation, and the two are combined in a fixed order to form a complete structured light projection sequence;

[0091] Set a projection time plan, specify the projection start time and the interframe interval, sequentially issue the projection frames, and record the timestamp information of each frame during the projection process, which is used to keep consistent with the image acquisition data of the multi-view imaging unit subsequently;

[0092] Calibrate the installation position and attitude parameters of the structured light projection unit in the pitch-side type reclaimer mechanical coordinate system, record the rotation angle and translation parameters, ensure the establishment of a stable mapping relationship between the structured light fringe and the mechanical coordinate system, and limit the stockpile area as the effective projection coverage range;

[0093] Set the projection power, brightness threshold and contrast parameters of the structured light projection unit, respectively allocate the projection power proportion for the phase-shift fringe sequence and the Gray code sequence, so that the projection frame maintains clear contrast in the stockpile area, and overexposure or insufficient brightness is avoided;

[0094] Generate a projection window based on the boundary of the stockpile area, implement shielding processing on irrelevant areas outside the stockpile area, and ensure that the structured light fringe and the Gray code pattern only act on the stockpile area within the effective projection coverage range;

[0095] During the structured light projection process, record the frame number, timestamp, attitude parameters of the structured light projection unit, projection power configuration and projection window information for each frame to form a projection side metadata set, which is used for time sequence alignment and spatial registration with the data acquired subsequently by the multi-view imaging unit;

[0096] After a complete projection sequence is completed, determine whether to enter the next round of projection according to the real-time running state of the pitch-side type reclaimer, when the pitch angle speed and the lateral walking speed continue to meet the triggering conditions, automatically start a new round of projection, otherwise keep the structured light projection unit standby until the starting conditions are met again.

[0097] In the embodiment, the generation of the preprocessed structured light scanning image sequence includes:

[0098] Perform distortion correction on the dynamic structured light scanning image sequence;

[0099] Perform brightness normalization processing on the corrected images, unify the brightness range of each image frame to a preset standard interval, avoid the contrast fluctuation of the structured light fringe caused by the exposure difference of different imaging nodes, and improve the fringe clarity through gray scale equalization;

[0100] Identify the fringe candidate region in the normalized image, determine the effective region containing the structured light fringe by analyzing the local brightness change, generate the fringe candidate region, and eliminate the background region and noise interference region;

[0101] Perform denoising processing on the fringe candidate region, smooth the local neighborhood of the image frame, weaken random noise and abnormal pixel points, and maintain the edge definition and overall continuity of the structured light fringe;

[0102] Extract the structured light fringe center line on the denoised image frame, scan the fringe direction point by point, determine the brightness peak value and connect to form a complete fringe center line set, and describe the continuous distribution characteristics of the fringe in the image frame;

[0103] Perform validity determination on each image frame, mark it as an invalid frame when the image appears overexposure, underexposure or motion blur exceeds the threshold, mark it as an effective frame when the structured light fringe is clear, complete and normal in brightness, and generate the corresponding validity mask;

[0104] Summarize all effective image frames, fringe center line sets and validity masks, and output the preprocessed dynamic structured light scanning image sequence.

[0105] In this embodiment, the generation of the spatial domain geometric feature data set includes:

[0106] Perform fringe encoding analysis in the neighborhood range of the structured light fringe center line, obtain the wrapped phase distribution, determine the fringe order combining the Gray code sequence, and obtain the absolute phase distribution by combining the wrapped phase and the fringe order, the absolute phase is used to uniquely determine the height information of each pixel position on the stacking surface;

[0107] Use the linear mapping relationship between phase and disparity to convert the absolute phase value to the disparity value, the disparity value is used to reflect the relative position difference of the pixel points between the structured light projection unit and the multi-view imaging unit;

[0108] According to the equivalent binocular geometric relationship, convert the disparity value to depth information to obtain the height distribution of the stacking surface, the height value is related to the projection baseline length and the focal length parameter;

[0109] Map the height information of each pixel point to a three-dimensional coordinate point using the camera intrinsic and extrinsic parameters to obtain the three-dimensional point cloud data represented in the multi-view imaging unit coordinate system;

[0110] Establish a local neighborhood in the three-dimensional point cloud data, and calculate the normal vector based on the adjacent difference method, the normal vector is used to describe the spatial direction characteristics of the stacking surface at the point position;

[0111] The rate of change in the horizontal direction and the vertical direction is calculated on the height distribution, and a slope index is obtained therefrom, a standard deviation of the height values in the local neighborhood is calculated as a geometric relief index, and a surface curvature value is calculated by a second-order rate of change approximation, the slope index is used to describe the surface inclination degree, the geometric relief index is used to reflect the local unevenness, and the surface curvature value is used to represent the bending characteristics of the surface;

[0112] The three-dimensional point cloud data is subjected to hole repair and boundary clipping, and the invalid area points generated due to overexposure, underexposure or motion blur are removed by using an effectiveness mask, so as to obtain a continuous and complete height distribution and a normal vector field;

[0113] The height distribution, three-dimensional point cloud data, normal vector, slope index, geometric relief index and curvature characteristics are integrated to construct a spatial domain geometric feature data set, which is used to completely represent the geometric morphology of the stock surface.

[0114] In the embodiment, the generation of the frequency domain feature data set specifically comprises:

[0115] The pre-processed structured light scanning image sequence and the effectiveness mask are received, and a local analysis window with a fixed size and a fixed step is divided in the stock area range as the input of the frequency domain processing;

[0116] A weighted operation is performed on the gray intensity distribution in each local analysis window, so that the weight value of the central region is higher and the weight value of the edge region is lower, so as to ensure the stability of the local stripe energy and reduce the spectral leakage;

[0117] A two-dimensional discrete Fourier transform is performed on the weighted local image to obtain the energy distribution of the gray intensity distribution at different frequencies, and a power spectrum is generated, the power spectrum being used to represent the energy characteristics of the stock surface texture at different spatial frequencies;

[0118] A radial average is performed on the power spectrum to obtain a one-dimensional power spectrum, and an energy proportion is calculated on the one-dimensional power spectrum according to a preset low-frequency interval, a medium-frequency interval and a high-frequency interval to obtain a low-frequency energy proportion, a medium-frequency energy proportion and a high-frequency energy proportion, respectively, which are used to reflect the texture change of the stock surface at different scales;

[0119] A main peak frequency corresponding to the energy peak value is determined on the power spectrum, and a calibration constant is used to convert the main peak frequency into a particle size scale of the stock surface, the particle size scale being used to reflect the average size of the particle distribution of the stock surface, and the calibration constant being determined by the pixel scale and the imaging geometry;

[0120] The power spectrum is linearly fitted in the logarithmic coordinates to obtain a spectral slope, the spectral slope being used to describe the attenuation law of the stock surface roughness when the observation scale changes;

[0121] The statistical power spectrum is used to calculate the energy distribution in different direction angles, the difference degree of the direction energy, and the direction variance, which represents the anisotropy degree of the surface texture of the stockpile;

[0122] The power spectrum is divided into multiple frequency bands, and the energy proportion of each frequency band is regarded as a probability distribution. The texture entropy is further calculated, which reflects the complexity and dispersion degree of the energy distribution of the surface texture of the stockpile;

[0123] The frequency domain features are smoothed in the time sequence. The frequency domain feature values of the current frame and the smoothing results of the previous frame are proportionally weighted to obtain an updated smoothing feature sequence, which is used to weaken the single-frame fluctuation and maintain the continuity of the time sequence features;

[0124] According to the validity mask, the invalid windows caused by overexposure, underexposure or motion blur are removed, and only the features of the valid windows are retained. The main peak frequency, particle size scale, spectral slope, low-frequency energy proportion, medium-frequency energy proportion, high-frequency energy proportion, direction variance, texture entropy and total energy are taken as a complete frequency domain feature vector;

[0125] The frequency domain feature vectors of all valid windows are summarized according to the spatial position and time sequence to construct a frequency domain feature data set, which reflects the particle size distribution characteristics and surface roughness characteristics of the stockpile surface.

[0126] In the embodiment, the fusion processing specifically includes:

[0127] The spatial domain geometric feature data set and the frequency domain feature data set are received, and the pitch angle sequence, the lateral displacement sequence and the rotation angle sequence of the pitch-side reclaimer are received simultaneously. The sequences are synchronized according to a unified time axis, and are aligned at a fixed sampling period to form a data input set to be fused;

[0128] The pitch angle sequence, the lateral displacement sequence and the rotation angle sequence are combined into a motion state vector. The pitch angle represents the angle of the reclaimer arm relative to the base, the lateral displacement represents the displacement of the reclaimer along the lateral trajectory, and the rotation angle represents the rotation angle of the rotation platform;

[0129] In the local area corresponding to the three-dimensional point, the spatial domain geometric feature vector and the frequency domain feature vector are extracted, and the motion state vector time-aligned with the local area is taken as an external constraint feature to form a comprehensive observation vector containing spatial domain geometric features, frequency features and motion features;

[0130] The normalization and scale mapping are respectively performed on the spatial domain geometric features, the frequency features and the motion state vectors, the three types of features are unified to the same fusion representation space, and the local fusion vectors are generated according to the preset weight coefficients in the fusion representation space, and the fusion feature data set consistent with the pitch side type reclaimer mechanical coordinate system is obtained by accumulating region by region and time by time.

[0131] A consistency constraint function is established in the fusion process, the consistency constraint function is composed of a geometric consistency residual, a normal consistency residual and a motion consistency residual, the geometric consistency residual is used to describe the average deviation of three-dimensional points to the local fitting plane, the normal consistency residual is used to describe the angle difference between adjacent point normal vectors, and the motion consistency residual describes the difference between the real motion state vector in the fusion feature data and the estimated motion state vector, and the fusion feature is simultaneously kept consistent in the geometric level, the normal level and the motion level by weighted minimization of the three types of residuals.

[0132] In the embodiment, the construction of the three-dimensional stacking surface model specifically includes:

[0133] The voxel grid covers the stacking area, the distance of all voxels is initialized as 0, the voxel weight is initialized as 0, the plane reference voxel field is formed as the initial three-dimensional stacking surface model;

[0134] At each time index, the fusion feature data set is received;

[0135] The rotation matrix and the translation vector at the current time are spliced to define the pose variable at the current time, the pose variable describes the spatial transformation relationship between the fusion feature data set and the three-dimensional stacking surface model, the rotation matrix represents the spatial direction transformation relationship of the point cloud under the action of the pitch angle and the rotation angle, and the translation vector represents the spatial position offset amount of the point cloud under the action of the lateral displacement;

[0136] For each observation point in the fusion feature data set, the Euclidean distance closest model point in the three-dimensional stacking surface model at the last time is found, and the normal vector corresponding to the model point is obtained, and the frequency domain features of the observation point and the model point are extracted;

[0137] For each pair of observation points and model points, a registration cost function is defined, the registration cost function measures the alignment error:

[0138] ;

[0139] ;

[0140] wherein, represents the registration cost function, and describes the registration total cost at time , represents a set of correspondences between the observation points and the model points, represents a point-to-plane geometric residual, represents a frequency feature residual, represents a motion consistency residual, represents a normal vector corresponding to the model point, represents an increment of the current estimated pose variable, represents a pose increment corresponding to the real motion state vector, represents a geometric residual weight, represents a frequency residual weight, represents a motion residual weight, represents a frequency domain feature of the observation point, represents a frequency domain feature of the corresponding point, represents a rotation matrix, P represents an observation point coordinate vector, represents a model point coordinate vector, represents a pose variable at the current time;

[0141] An optimal pose variable is obtained by minimizing a registration cost function, and a registered observation point cloud data is generated, which is used to accurately align the fused feature data set with the three-dimensional stockpile surface model;

[0142] A voxel grid fusion method is used to incrementally fuse the registered observation point cloud data of each frame, and voxel distance and voxel weight are updated to obtain an updated voxel grid:

[0143] ;

[0144] ;

[0145] wherein, represents an updated voxel distance, represents a historical voxel distance, describing the cumulative voxel distance of the previous frame, represents a historical weight, describing the cumulative weight of the previous frame, represents a current observation distance, represents a current weight, represents an updated voxel weight, represents a maximum weight;

[0146] A zero-equal surface is extracted from the updated voxel grid to generate a three-dimensional stockpile surface model at the current time, and a normal vector of the three-dimensional stockpile surface model is calculated according to the gradient of the voxel grid.

[0147] In the embodiment, the generation of the material positioning result data specifically includes:

[0148] Perform a voxel mesh traversal on the three-dimensional material pile surface model generated at the current moment, and extract the spatial domain geometric feature data set and frequency domain feature data set of all voxels;

[0149] Based on the spatial domain geometric feature data set and the frequency domain feature data set, a threshold determination is performed on the voxel set to obtain the voxel set of the material target area;

[0150] A voxel mesh is traversed on the 3D material pile surface model. For each voxel, a set of spatial domain geometric feature data is extracted. Statistical analysis is performed on the above geometric features to record the structural features of each voxel in the spatial domain. Simultaneously, a set of frequency domain feature data is extracted for each voxel. Based on the set spatial domain geometric feature threshold and frequency domain feature threshold, various features of each voxel are judged in turn. If the spatial domain geometric features and frequency domain features of a voxel simultaneously meet the corresponding judgment conditions, the voxel is classified into the material target area voxel set; otherwise, it is classified into a non-target area. Through the above multi-dimensional feature fusion judgment, the accurate regional classification of the voxel set is achieved, laying the foundation for the subsequent extraction of the 3D coordinates of the material target area and the output of material positioning results.

[0151] For each target region in the set of voxels of the material target region, extract the set of three-dimensional coordinates of all voxels;

[0152] Determine the weighted centroid 3D coordinates of each target region:

[0153] ;

[0154] in, Indicates the first The weighted centroid three-dimensional coordinates of the target area of ​​each material Represents the three-dimensional coordinates of a voxel. The voxel weighting coefficients are determined based on a combination of spatial domain geometric feature data sets and frequency domain feature data sets.

[0155] For each material target area, the weighted centroid three-dimensional coordinates are used to perform a pitch and side-mounted material handling machine motion space constraint verification, and areas that do not meet the motion range and structural constraints are eliminated to obtain the effective material target area voxel set;

[0156] The weighted centroid 3D coordinates of all material target regions in the effective material target region voxel set are output as the material target region 3D coordinates, generating material positioning result data.

[0157] Example 1:

[0158] In order to verify the practical application effect of the present application, the material positioning method of the pitch-yaw type material taking machine based on image recognition is applied to the automatic material taking system of a raw material yard of a certain steel enterprise. The yard is large in scale and stores various types of raw materials such as ores and coal. The material accumulation form is complex and changes frequently. The traditional material taking method generally relies on manual remote control or simple visual positioning, which is difficult to adapt to variable working conditions and complex environments, resulting in limited positioning accuracy and cannot guarantee the material taking efficiency and safety.

[0159] In actual field application, the pitch-yaw type material taking machine is arranged in the main passageway of the yard. The system is equipped with a structured light projection unit and a multi-view imaging unit. The data is synchronized through the control system and the pitch, lateral and rotary movements of the pitch-yaw type material taking machine. Whenever the device starts the automatic operation process, the structured light projection unit automatically loads the coded grating according to the real-time movement state of the pitch mechanism and the lateral walking mechanism, and projects a sequence of structured light stripes onto the material surface according to the predetermined scheme. The multi-view imaging unit simultaneously captures dynamic images of the material surface with structured light stripes at multiple positions. The original data is transmitted in real time to the data processing center through the high-speed bus of the system. In order to overcome the influence of environmental light fluctuation, dust shielding and other factors, the system uses multi-frame image fusion, stripe center line extraction, invalid frame automatic removal and other image preprocessing algorithms, which effectively improves the structured light stripe recognition accuracy and provides high-quality input for subsequent three-dimensional reconstruction.

[0160] The preprocessed data is input into the spatial domain geometric feature and frequency domain feature extraction module. The system performs multi-scale and multi-region joint statistics and analysis on the spatial domain features such as height distribution, normal vector, slope, fluctuation and curvature, and the frequency domain features such as main peak frequency, particle size scale, spectral slope, direction variance and texture entropy. This processing method not only realizes the fine restoration of the spatial structure of the material surface, but also fully captures the microscopic texture features such as particle distribution and surface roughness. At the same time, the motion state data of the material taking machine, including the pitch angle, lateral displacement and rotation angle, are collected on a unified time axis, and the motion state vector and spatial-frequency features are realized. The external constraint and synchronization provide time and space consistent input for three-dimensional modeling and positioning.

[0161] The system uses the voxel grid method to construct the three-dimensional material surface model. Initially, a standard voxel grid is established for the material stacking area, and all voxel distances and weights are initialized. During continuous operation of the device, the processing center performs spatial registration on the fused feature data set and the previous three-dimensional material surface model, and combines the current device pose variables to perform voxel-level incremental fusion on each frame of registered observation point cloud data. The voxel distance and weight are dynamically updated. Every few seconds, the system automatically extracts the updated zero-equal surface as a new three-dimensional material surface model, and continuously improves the model accuracy and surface normal information.

[0162] To realize the accurate positioning of the target material area, the system performs voxel grid traversal on the three-dimensional stacking surface model, jointly determines the spatial domain geometric features and frequency domain features of all voxels, and after multi-threshold fusion screening of the spatial-frequency features, can reliably distinguish the areas of different physical states and particle sizes of the stacking surface. Further, through three-dimensional connectivity analysis and physical space constraints, the motion-unreachable, structurally abnormal or misidentified areas are automatically removed. For each effective material target area, the system calculates the three-dimensional coordinates according to the voxel weighted centroid method, and directly maps them to the pitch-lateral type material taking machine coordinate system, realizing high-precision three-dimensional positioning of the material target area. The final material positioning result data is pushed to the control unit in real time as the basis for automatic material taking operation path planning and execution, driving the pitch-lateral type material taking machine to complete automatic positioning and material grabbing and other actions.

[0163] To verify the performance of the present application in implementation, a comparison with the traditional method was made, and the results are shown in Table 1.

[0164] Table 1 Performance comparison experiment results of pitch-lateral type material taking machine material positioning method

[0165]

[0166] From Table 1, it can be seen that the present application method is superior to the traditional method in all key performance indicators. In terms of positioning accuracy, the traditional stereo vision is severely affected by the irregularity of the stacking surface and the light due to its dependence on pixel matching, with an average error of more than 15 cm. The single structured light reconstruction can improve some of the errors, but there is still a large fluctuation under dynamic working conditions. However, the present application, through the joint analysis of spatial domain geometric features and frequency domain features, in combination with the motion parameter constraints of the pitch-lateral type material taking machine, makes the alignment of the model in three-dimensional space more rigorous, with an average error of less than 6 cm, showing a significant improvement.

[0167] In terms of positioning success rate, the present application reaches nearly 91%, which is more than 7 percentage points higher than the traditional stereo vision. The reason for this improvement is that the center line extraction and invalid frame removal mechanism are used, making the effectiveness of the input data higher, and the multi-modal feature fusion avoids misidentification caused by single-dimensional discrimination.

[0168] In terms of data processing frame rate, the present application reaches 24fps, which is higher than the traditional 15 to 18fps, indicating that it can meet the continuous processing needs of real-time dynamic stacking scenes. The fundamental reason for the improvement is that an efficient voxel incremental fusion strategy is adopted, which reduces the repeated operation amount while ensuring the model accuracy, thereby speeding up the overall operation speed.

[0169] The data of stability and automatic picking completion rate also reflect the advantages of the present application. The stability of the traditional method is low and is greatly affected by noise and equipment jitter. The present application jointly optimizes the geometric residual, normal residual and motion residual through the consistency constraint function, so that the system maintains continuous and stable performance under dynamic working conditions. The final automatic picking completion rate is more than 94%, which is about 14 percentage points higher than that of the traditional method, which means that in actual operation, the present application can not only more accurately identify the target material area, but also effectively support the subsequent picking action, reduce the operation risk and operation delay.

[0170] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can make equivalent replacements or changes within the technical range disclosed by the present application according to the technical solution and inventive concept of the present application, which should be covered within the protection scope of the present application.

Claims

1. An image recognition based material positioning method for a luffing side type material handler, characterized by, The method comprises the following steps: During synchronous operation of the luffing mechanism and the lateral traveling mechanism of the luffing-lateral type reclaimer, a structured light sequence is projected to the stockpile area according to a preset coded grating pattern; Multi-view imaging units are used to synchronously collect multi-view image data of the stockpile surface containing structured light stripes, and a dynamic structured light scanning image sequence is generated; Preprocessing is performed on the dynamic structured light scanning image sequence to generate a preprocessed structured light scanning image sequence; Spatial domain processing is performed on the preprocessed structured light scanning image sequence to extract the geometric morphological information of the stockpile surface, and the height distribution of the stockpile surface is analyzed based on the structured light stripe coding to generate a spatial domain geometric feature data set; Frequency domain processing is performed on the preprocessed structured light scanning image sequence to extract the light intensity distribution variation law information, and a frequency domain feature data set is generated; The luffing angle sequence, lateral displacement sequence and rotation angle sequence of the luffing-lateral type reclaimer are collected and used as external constraints for fusion processing with the spatial domain geometric feature data set and the frequency domain feature data set; Based on the fusion processing result, time sequence three-dimensional reconstruction operation is performed to construct a three-dimensional stockpile surface model; The three-dimensional coordinates of the material target area are determined in the three-dimensional stockpile surface model based on the spatial domain geometric feature data set and the frequency domain feature data set, and material positioning result data is generated.

2. A method of material positioning for a luffing side planer according to claim 1, characterized in that, The projected structured light sequence specifically includes: When the luffing mechanism and the lateral traveling mechanism of the luffing-lateral type reclaimer enter a synchronous operation state, the control logic of the structured light projection unit is started, the luffing angle velocity and the lateral traveling velocity are monitored, and the structured light projection unit is triggered to work when both are within a preset allowable range; A preset coded grating pattern is loaded, which is composed of a phase shift stripe sequence and a Gray code sequence, the phase shift stripe sequence is used to obtain continuous phase information, and the Gray code sequence is used to perform stripe period disambiguation, both of which are combined in a fixed order to form a complete structured light projection sequence; The projection time plan is set to clearly define the projection start time and the frame interval, sequentially issue the projection frames, and record the timestamp information of each frame during the projection process; The installation position and attitude parameters of the structured light projection unit are calibrated in the mechanical coordinate system of the luffing-lateral type reclaimer, the rotation angle and translation parameters are recorded, and the stockpile area is defined as the effective projection coverage range; The projection power, brightness threshold and contrast parameters of the structured light projection unit are set, and the projection power proportion is allocated to the phase shift stripe sequence and the Gray code sequence respectively; Based on the boundary of the stockpile area, a projection window is generated to implement shielding processing on irrelevant areas outside the stockpile area, so as to ensure that the structured light stripes and Gray code patterns only act on the stockpile area within the effective projection coverage range; After a complete projection sequence is completed, it is judged whether to enter the next projection according to the real-time running state of the luffing-lateral type reclaimer, when the luffing angle velocity and the lateral traveling velocity continue to meet the triggering conditions, a new round of projection is automatically started, otherwise the structured light projection unit remains standby until the start condition is met again.

3. A method of material positioning for a luffing side planar material handler based on image recognition as claimed in claim 1, characterized in that, The generation of the preprocessed structured light scanning image sequence includes: Distortion correction is performed on the dynamic structured light scanning image sequence; Performing brightness normalization on the corrected images, the brightness range of each image frame is unified to a preset standard interval; Identifying a fringe candidate region in the normalized image, determining an effective region containing the structured light fringe by analyzing the local brightness variation, generating the fringe candidate region, and eliminating the background region and noise interference region; Performing denoising processing on the fringe candidate region, smoothing the local neighborhood of the image frame, weakening random noise and abnormal pixel points, and maintaining the edge definition and overall continuity of the structured light fringe; Extracting the structured light fringe center line on the denoised image frame, scanning the fringe direction point by point, determining the brightness peak value and connecting to form a complete fringe center line set; Performing effectiveness determination on each image frame, marking the image as an invalid frame when overexposure, underexposure or motion blur exceeds the threshold, marking the image as a valid frame when the structured light fringe is clear, complete and normal in brightness, and generating a corresponding effectiveness mask; Summarizing all valid image frames, fringe center line sets and effectiveness masks, and outputting the preprocessed dynamic structured light scanning image sequence.

4. The image recognition based material positioning method for a luffing side type material handler according to claim 1, characterized in that, The generation of the spatial domain geometric feature data set includes: Performing fringe coding analysis in the neighborhood range of the structured light fringe center line, obtaining wrapped phase distribution, combining Gray code sequence to determine fringe order, and combining wrapped phase and fringe order to obtain absolute phase distribution; Using the linear mapping relationship between phase and disparity, converting the absolute phase value into a disparity value, which is used to reflect the relative position difference between the structured light projection unit and the multi-view imaging unit; According to the equivalent binocular geometric relationship, the disparity value is converted into depth information to obtain the height distribution of the stock surface, and the height value is related to the projection baseline length and focal length parameters; Using the camera intrinsic and extrinsic parameters, the height information of each pixel point is mapped into a three-dimensional coordinate point to obtain three-dimensional point cloud data represented in the multi-view imaging unit coordinate system; Local neighborhood is established in the three-dimensional point cloud data, and the normal vector is calculated based on the adjacent difference method, which is used to describe the spatial direction characteristics of the stock surface at this point; The change rates in the horizontal and vertical directions are calculated on the height distribution, and the slope index is obtained accordingly. At the same time, the standard deviation of the height value in the local neighborhood is calculated as the geometric relief index, and the surface curvature value is calculated by the second order change rate approximation; Performing hole repair and boundary clipping on the three-dimensional point cloud data, using the effectiveness mask to eliminate invalid area points caused by overexposure, underexposure or motion blur, to obtain continuous and complete height distribution and normal vector field; Integrate the height distribution, three-dimensional point cloud data, normal vector, slope index, geometric relief index and curvature feature to construct the spatial domain geometric feature data set.

5. The image recognition based material positioning method for a luffing side type material handler according to claim 1, characterized in that, The generation of the frequency domain feature data set specifically includes: Receiving the preprocessed structured light scanning image sequence and the effectiveness mask, dividing local analysis windows of fixed size and fixed step in the stock area range as the input of frequency domain processing; Performing weighted operation on the gray intensity distribution in each local analysis window; Performing two-dimensional discrete Fourier transform on the weighted local image to obtain the energy distribution of the gray intensity distribution at different frequencies, and generating the power spectrum; Performing radial averaging on the power spectrum to obtain a one-dimensional power spectrum, and calculating the energy proportion in the preset low-frequency interval, medium-frequency interval and high-frequency interval on the one-dimensional power spectrum to obtain the low-frequency energy proportion, medium-frequency energy proportion and high-frequency energy proportion respectively; Determining the main peak frequency corresponding to the energy peak on the power spectrum, and converting the main peak frequency into the particle size scale of the stock surface by using the calibration constant; Linear fitting of the power spectrum in logarithmic coordinates to obtain the spectral slope, which is used to describe the attenuation law of the roughness of the stock surface with the change of the observation scale; Statistical energy distribution of the power spectrum at different direction angles to calculate the difference degree of the direction energy and obtain the direction variance; Dividing the power spectrum into multiple frequency bands, and regarding the energy proportion of each frequency band as a probability distribution to further calculate the texture entropy; Smooth processing of the frequency domain features in the time sequence, proportionally weighting the frequency domain feature values of the current frame and the smoothing results of the previous frame to obtain the updated smoothing feature sequence; According to the effectiveness mask, invalid windows caused by overexposure, underexposure or motion blur are removed, only the features of the valid windows are retained, and the main peak frequency, particle size scale, spectral slope, low-frequency energy proportion, medium-frequency energy proportion, high-frequency energy proportion, direction variance, texture entropy and total energy are taken as a complete frequency domain feature vector; The frequency domain feature vectors of all valid windows are summarized according to the spatial position and time sequence to construct a frequency domain feature data set.

6. A method of material positioning for a luffing side planar material handler based on image recognition as claimed in claim 1, characterized in that, The fusion processing specifically includes: Receiving the spatial domain geometric feature data set and the frequency domain feature data set, simultaneously receiving the pitch angle sequence, lateral displacement sequence and rotation angle sequence of the pitch lateral type reclaimer, synchronizing the sequences according to the unified time axis, and aligning them at a fixed sampling period to form a data input set to be fused; Combining the pitch angle sequence, lateral displacement sequence and rotation angle sequence into a motion state vector; In the local area corresponding to the three-dimensional point, the spatial domain geometric feature vector and the frequency domain feature vector are extracted, and the motion state vector time-aligned with the local area is taken as an external constraint feature to form a comprehensive observation vector containing spatial domain geometric features, frequency features and motion features; Performing normalization and scale mapping on the spatial domain geometric features, frequency features and motion state vectors respectively to unify the three types of features to the same fusion representation space, combining them in the fusion representation space according to the preset weight coefficient to generate a local fusion vector, and accumulating it region by region and time by time to obtain a fusion feature data set consistent with the mechanical coordinate system of the pitch lateral type reclaimer.

7. A method of material positioning for a luffing side planar material handler based on image recognition as claimed in claim 1, characterized in that, The construction of the three-dimensional stock surface model specifically includes: Initializing the voxel grid to cover the stock area, setting the distance of all voxels to 0 and the weight of all voxels to 0 to form a plane reference voxel field as the initial three-dimensional stock surface model; At each time index, the fusion feature data set is received; Splicing the rotation matrix and translation vector at the current time to define the pose variable at the current time; For each observation point in the fusion feature data set, find the nearest model point in the three-dimensional stockpile surface model at the previous time, and obtain the normal vector corresponding to the model point, while extracting the frequency domain features of the observation point and the frequency domain features of the model point; For each pair of observation points and model points, define a registration cost function; By minimizing the registration cost function, the optimal pose variable is obtained, and the registered observation point cloud data is generated; Using the voxel grid fusion method, the registered observation point cloud data of each frame is incrementally fused, and the voxel distance and voxel weight are updated to obtain the updated voxel grid; The zero-equal surface is extracted from the updated voxel grid to generate the three-dimensional stockpile surface model at the current time, and the normal vector of the three-dimensional stockpile surface model is calculated according to the gradient of the voxel grid.

8. The image recognition based material positioning method for a luffing side type material handler according to claim 1, characterized in that, The generation of the material positioning result data specifically includes: Perform voxel grid traversal on the three-dimensional stockpile surface model generated at the current time to extract the spatial domain geometric feature data set and the frequency domain feature data set of all voxels; According to the spatial domain geometric feature data set and the frequency domain feature data set, perform threshold judgment on the voxel set to obtain the material target region voxel set; For each target region in the material target region voxel set, extract the three-dimensional coordinate set of all voxels; Determine the weighted centroid three-dimensional coordinates of each target region; Perform pitch-side type reclaimer motion space constraint verification on the weighted centroid three-dimensional coordinates of each material target region, and remove regions that do not meet the motion range and structure constraints to obtain the effective material target region voxel set; Output the weighted centroid three-dimensional coordinates of all material target regions in the effective material target region voxel set as the material target region three-dimensional coordinates to generate the material positioning result data.

Images