Leaf sorting and packaging method based on multi-source vision and sorting algorithm

Abstract

The present application relates to the technical field of visual detection, in particular to a leaf sorting and packaging method based on multi-source vision and sorting algorithm. A line array camera and a surface array camera are used to obtain a leaf image sequence, a nonlinear mapping relationship between pixels and a physical conveyor coordinate system is established through a trigger time difference and a conveyor belt encoder pulse, hyperspectral texture and red-green-blue morphological features are spatiotemporally aligned and fused in the physical coordinate system to generate an overlapping leaf separation feature map; the contour center point coordinates are extracted and input into a dynamic interpolation sorting model, the motion displacement compensation amount is calculated according to the real-time pulse frequency, the leaf coordinates with the compensation amount are reorganized into a time sequence sorting queue according to the mechanical arm cycle, and the sorting queue is executed. The present application eliminates the misplacement of multi-source vision features, improves the resolution accuracy of overlapping leaves, overcomes the position prediction error caused by variable speed operation, ensures the synchronization of the grabbing action and the actual position of the leaves, and reduces the sorting misplacement rate.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology, specifically to a leaf sorting and packaging method based on multi-source vision and sorting algorithms. Background Technology

[0002] In current automated blade sorting and packaging lines, a single type of optical camera is typically installed above the conveyor belt to capture images of the moving blades. When processing these images, the conventional approach directly performs thresholding and edge extraction on the two-dimensional pixel coordinates acquired by the camera to identify the center position of the individual blade's contour. During the sorting execution phase, existing technology assumes the conveyor belt operates at a constant speed. By reading the fixed set speed at the start and stop of the conveyor belt and combining this with the fixed physical distance between the camera detection position and the downstream robotic arm's gripping position, a fixed coordinate offset is calculated using a uniform linear motion formula. The system adds the fixed coordinate offset to the identified blade pixel coordinates, directly mapping it to the robotic arm's gripping space coordinates, and generates a static sorting queue according to the image processing time sequence, which is then sent to the robotic arm to execute the gripping action.

[0003] In actual industrial production environments, conveyor belts frequently operate at varying speeds (accelerating or decelerating) to match the rhythm of upstream and downstream equipment, and adjacent blades often overlap. Current technologies still calculate offsets using a fixed speed setting under variable-speed conditions, failing to consider the nonlinear displacement changes caused by speed fluctuations. This leads to a misalignment between the calculated robotic arm grasping coordinates and the actual physical coordinates of the blade when it reaches its execution position. Furthermore, the use of a single type of camera and the lack of an alignment mechanism for multi-source data within the physical conveyor belt coordinate system prevent the accurate separation of individual blade contours when blades overlap. This results in inherent deviations in the coordinates within the generated sorting queue, ultimately causing positioning errors and grasping failures during robotic arm operations. Summary of the Invention

[0004] The purpose of this invention is to provide a leaf sorting and packaging method based on multi-source vision and sorting algorithms, which can solve the problems in the background art mentioned above.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The blade sorting and packaging method based on multi-source vision and sorting algorithm uses a linear RGB camera and a planar hyperspectral camera to form a multi-source vision acquisition node to obtain a continuous image sequence of moving blades on the conveyor belt. By using the trigger time difference between the linear RGB camera and the area hyperspectral camera and the conveyor belt encoder pulse, a nonlinear mapping relationship between the pixel coordinate system and the physical conveyor belt coordinate system is established. The hyperspectral texture features of the area hyperspectral camera and the RGB morphological features of the linear RGB camera are fused together at the feature level in the physical conveyor belt coordinate system to generate a separation feature map of the overlapping blades. The contours of individual blades in the separated feature map are extracted using connected component analysis. The coordinates of the center point of the individual blade contours are input into a dynamic interpolation sorting model. The dynamic interpolation sorting model receives the real-time pulse frequency of the conveyor belt encoder and calculates the motion displacement compensation amount of the blade from the detection position to the execution position of the sorting robot arm based on the real-time pulse frequency. The blade coordinates with the motion displacement compensation amount are reorganized into a time-series sorting queue according to the motion cycle of the sorting robot arm, and the time-series sorting queue is sent to the sorting robot arm to perform the gripping and packaging action.

[0006] Preferably, the step of establishing a nonlinear mapping relationship between the pixel coordinate system and the physical conveyor belt coordinate system by using the trigger time difference between the linear RGB camera and the area array hyperspectral camera and the conveyor belt encoder pulse includes: recording the first timestamp of the first row of pixels of the same blade acquired by the linear RGB camera, recording the second timestamp of the center pixel of the same blade acquired by the area array hyperspectral camera, and calculating the time difference between the first timestamp and the second timestamp. Extract the total number of pulses output by the conveyor belt encoder within the time difference, multiply the total number of pulses by the single-pulse equivalent displacement of the conveyor belt, and obtain the physical conveyor belt displacement distance corresponding to the time difference. Using the displacement distance of the physical conveyor belt as the horizontal axis offset and the height difference between the principal optical axis of the linear RGB camera and the principal optical axis of the area array hyperspectral camera in the physical conveyor belt coordinate system as the vertical axis offset, the nonlinear mapping relationship from the pixel coordinate system to the physical conveyor belt coordinate system is constructed.

[0007] Preferably, the step of performing feature-level spatiotemporal alignment and fusion of the hyperspectral texture features and the RGB morphological features in the physical conveyor belt coordinate system to generate a separation feature map of overlapping blades includes: inputting the hyperspectral image acquired by the area array hyperspectral camera into a three-dimensional tensor decomposition network, and extracting the spectral reflectance curve of the hyperspectral image in a set band range as the hyperspectral texture feature; The RGB image captured by the linear RGB camera is input into the edge gradient calculation network, and the morphological boundary gradient vector of the RGB image is extracted as the RGB morphological feature; Using the nonlinear mapping relationship, the spectral reflectance curve and the morphological boundary gradient vector are spatially aligned in the physical conveyor belt coordinate system. The spatially aligned spectral reflectance curve and the morphological boundary gradient vector are then concatenated at the channel level and input into a convolutional network to generate the separation feature map of the overlapping blades.

[0008] Preferably, the step of extracting the single-blade outline from the separated feature map using connected component analysis includes: performing binarization processing on the separated feature map to obtain a binarized mask map; An eight-neighbor connected component search is performed on the binarized mask image to mark each independent connected region in the binarized mask image; Calculate the area of ​​each independent connected region and the aspect ratio of its circumscribed rectangle. Eliminate independent connected regions whose area is less than a preset area threshold and whose aspect ratio of the circumscribed rectangle exceeds a preset ratio range. The set of boundary pixels of the remaining independent connected regions constitutes the outline of the single leaf. Extract the average coordinates of all pixels inside the single blade outline, and use the average coordinates as the coordinates of the center point of the single blade outline.

[0009] Preferably, the step of calculating the motion displacement compensation of the blade from the detection position to the sorting robot arm execution position based on the real-time pulse frequency includes: inputting the real-time pulse frequency into the cubic spline interpolation calculation layer in the dynamic interpolation sorting model, using the fixed physical distance between the detection position and the sorting robot arm execution position as the interpolation interval, and converting the real-time pulse frequency into an instantaneous velocity sequence within the interpolation interval; The instantaneous velocity sequence is integrated over time within the interpolation interval to obtain the predicted value of the actual motion displacement. The motion displacement compensation amount is obtained by subtracting the fixed physical distance from the actual predicted motion displacement value.

[0010] Preferably, the step of reorganizing the blade coordinates with the motion displacement compensation amount into a time-series sorting queue according to the motion cycle of the sorting robot arm includes: obtaining the standard motion cycle duration required for the sorting robot arm to complete a single grasping action; Using the standard motion cycle duration as a time window, the blade coordinates that fall within the same time window when the timestamp of the sorting robot arm's execution position are classified into the same batch. Within the same batch, the blade coordinates are sorted from smallest to largest according to the ordinate values ​​of the blade coordinates with the motion displacement compensation, and the sorted blade coordinates are combined in time sequence to generate the time sequence sorting queue.

[0011] Preferably, before recording the first timestamp of the first row of pixels of the same blade acquired by the linear RGB camera, the method further includes: obtaining the physical horizontal distance between the linear RGB camera and the area array hyperspectral camera in the conveyor belt conveying direction; Divide the physical horizontal spacing by the conveyor belt speed corresponding to the previous historical pulse cycle of the conveyor belt encoder to obtain the static trigger delay value. The static trigger delay value is written as a basic offset into the hardware trigger controller of the linear RGB camera, so that the physical overlap field of view between the linear RGB camera and the area array hyperspectral camera is maximized when acquiring the same leaf.

[0012] Preferably, the step of aligning the spectral reflectance curve and the morphological boundary gradient vector in the physical conveyor belt coordinate system using the nonlinear mapping relationship includes: extracting the horizontal coordinate offset and the vertical coordinate offset in the nonlinear mapping relationship and constructing a two-dimensional affine transformation matrix; The spatial feature grid corresponding to the spectral reflectance curve is translated and scaled using the two-dimensional affine transformation matrix, so that the origin of the coordinate system of the transformed spatial feature grid coincides with the origin of the coordinate system of the feature grid where the morphological boundary gradient vector is located.

[0013] Preferably, before inputting the real-time pulse frequency into the cubic spline interpolation calculation layer of the dynamic interpolation sorting model, the method further includes: extracting several adjacent historical pulse frequencies of the real-time pulse frequency and calculating the frequency change rate of the historical pulse frequencies; Determine whether the absolute value of the frequency change rate is greater than a set mutation threshold. If it is greater than the mutation threshold, lock the real-time pulse frequency to the average of the historical pulse frequencies and input the locked real-time pulse frequency into the cubic spline interpolation calculation layer.

[0014] Preferably, the step of generating the time-series sorting queue by combining the sorted blade coordinates in a time sequence includes: extracting two adjacent blade coordinates after sorting and calculating the Euclidean distance between the two adjacent blade coordinates; The Euclidean distance is compared with the maximum physical size of the end effector of the sorting robot arm. If the Euclidean distance is less than the maximum physical size, the blade coordinates that are sorted later in the time-series sorting queue are deleted, and the deleted blade coordinates are carried over to the next batch corresponding to the time window for reordering.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention establishes a nonlinear mapping relationship between the pixel coordinate system and the physical conveyor belt coordinate system. It performs feature-level spatiotemporal alignment and fusion of hyperspectral texture features and red-green-blue morphological features in the physical conveyor belt coordinate system to generate a separated feature map. This overcomes the target feature misalignment phenomenon caused by differences in acquisition angle and frame rate in multi-source vision, and achieves the analysis of blade edge features in overlapping states. The dynamic interpolation sorting model receives the real-time pulse frequency of the conveyor belt encoder, calculates the displacement compensation of the blade from the detection position to the execution position of the sorting robot arm based on the real-time pulse frequency, and reorganizes the temporal sorting queue. This changes the method of calculating the offset based on a fixed set speed, eliminates the position prediction error of the blade reaching the sorting node under the condition of conveyor belt acceleration and deceleration fluctuations, ensures the synchronization of the sorting robot arm's grasping action and the actual physical position of the blade under variable speed motion, and reduces sorting misalignment.

[0016] 2. By recording the trigger time difference between the linear scan camera and the area scan camera and extracting the total number of pulses output by the conveyor belt encoder within the time difference, the physical conveyor belt displacement distance is calculated. This physical conveyor belt displacement distance is used as the horizontal axis offset to construct a nonlinear mapping relationship, eliminating the interference of camera spatial layout differences on feature fusion. When calculating the motion displacement compensation, the real-time pulse frequency is converted into an instantaneous velocity sequence within the interpolation interval, and time integration is performed to calculate the predicted actual motion displacement value, matching the true physical displacement trajectory of the conveyor belt's variable speed motion. Based on the sorting robot arm's motion cycle, the blade coordinates with motion displacement compensation are divided into batches and sorted according to the vertical axis value, standardizing the generation logic of the sorting queue and avoiding disordered jumps in the robot arm's motion trajectory. Attached Figure Description

[0017] Figure 1 This is a flowchart of the overall process for a leaf sorting and packaging method based on multi-source vision and sorting algorithms. Figure 2 A flowchart is constructed to illustrate the relationship between multi-source visual acquisition and nonlinear mapping. Figure 3 Flowchart for feature extraction and spatiotemporal alignment fusion; Figure 4 Flowchart for connected component analysis and single blade contour extraction; Figure 5 Flowchart for calculating dynamic displacement compensation; Figure 6 Generate a flowchart for the time-series sorting queue. Detailed Implementation

[0018] Please refer to the attached document. Figure 1In one embodiment, a leaf sorting and packaging method based on multi-source vision and sorting algorithms is applied to an automated sorting and packaging production line for leafy agricultural products. A linear RGB camera and an area-array hyperspectral camera are sequentially mounted on a fixed bracket above the conveyor belt along the conveyor belt's transport direction. The field of view of the linear RGB camera covers the entire width of the conveyor belt, while the field of view of the area-array hyperspectral camera partially overlaps with that of the linear RGB camera in the conveyor belt's transport direction. An incremental encoder is mounted on the drive roller of the conveyor belt. The encoder's output is electrically connected to the input port of a data processing unit, and the output of the data processing unit is electrically connected to the controller of the sorting robotic arm.

[0019] refer to Figure 2 The data processing unit receives a continuous row scan image sequence from the linear RGB camera, where each row corresponds to a physical cross-section of the conveyor belt perpendicular to the conveying direction. The row scan frequency of the linear RGB camera is dynamically adjusted according to the conveyor belt's running speed to ensure that the physical displacement between adjacent rows is less than the physical distance corresponding to a single pixel. Simultaneously, the data processing unit receives a sequence of frame images from the area array hyperspectral camera, where each frame contains two-dimensional spatial information across multiple spectral bands. The exposure time of the area array hyperspectral camera is automatically adjusted according to the ambient light intensity to ensure that the acquired hyperspectral images have a suitable signal-to-noise ratio. When a blade enters the field of view of the linear RGB camera, the camera begins acquiring row pixel data for that blade, and the data processing unit records the first timestamp of the first row of pixels acquired. When the blade moves with the conveyor belt to the center of the area array hyperspectral camera's field of view, the camera triggers the acquisition of a hyperspectral image containing the blade, and the data processing unit records the second timestamp of the center pixel of that frame.

[0020] The data processing unit extracts the time difference between the first and second timestamps and simultaneously reads the total number of pulses output by the conveyor belt encoder within this time difference. Multiplying the total number of pulses by the equivalent single-pulse displacement of the conveyor belt yields the physical conveyor belt displacement distance corresponding to this time difference. The equivalent single-pulse displacement is the circumference of the conveyor belt's drive roller divided by the number of pulses output per encoder revolution; this parameter is predetermined during system calibration using a standard length calibration block. Using the physical conveyor belt displacement distance as the horizontal axis offset and the height difference between the principal optical axes of the linear RGB camera and the area array hyperspectral camera in the physical conveyor belt coordinate system as the vertical axis offset, a nonlinear mapping relationship is constructed from the pixel coordinate system to the physical conveyor belt coordinate system. For any pixel in the RGB image acquired by the linear RGB camera... Its coordinates (X,Y) in the physical conveyor belt coordinate system satisfy:

[0021] in This represents the pixel equivalent in the x-direction of a linear RGB camera. This represents the pixel equivalent in the y-direction of a linear RGB camera. This is the horizontal axis offset. This represents the offset of the vertical axis.

[0022] refer to Figure 3 The hyperspectral images acquired by the area array hyperspectral camera are input into a three-dimensional tensor decomposition network, and the spectral reflectance curves of the hyperspectral images in a set band range are extracted as hyperspectral texture features. The hyperspectral images are represented as three-dimensional tensors. Where H is the image height, W is the image width, and B is the number of spectral bands. The 3D tensor decomposition network uses the CP decomposition algorithm to decompose the 3D tensor into the sum of the outer products of the spatial factor matrix and the spectral factor matrix:

[0023] Where R is the rank of the decomposition. Let be the first spatial factor vector. For the second space factor vector, For spectral factor vectors, This represents the outer product operation. The column vectors corresponding to the defined band intervals are extracted from the spectral factor matrix to form the hyperspectral texture feature tensor.

[0024] The RGB images captured by the linear RGB camera are converted to grayscale images and input into an edge gradient calculation network to extract the gradient vectors of the morphological boundaries. The edge gradient calculation network uses a 3×3 Sobel operator to calculate the gradient components of the grayscale image in the x and y directions respectively.

[0025]

[0026] Where I is a grayscale image. This represents a convolution operation. The gradient magnitude and direction are calculated based on the gradient components.

[0027]

[0028] Gradient magnitude G and gradient direction Combined into a shape boundary gradient vector, which serves as the RGB shape feature.

[0029] Using the constructed nonlinear mapping relationship, the spectral reflectance curve and the morphological boundary gradient vector are spatially aligned in the physical conveyor belt coordinate system. The horizontal and vertical coordinate offsets in the nonlinear mapping relationship are extracted to construct a two-dimensional affine transformation matrix.

[0030] For any feature point (x, y) in the spatial feature grid corresponding to the hyperspectral texture feature, its transformed coordinates (x', y') satisfy:

[0031] This transformation translates the spatial feature grid of the hyperspectral texture features to the same coordinate system as the spatial feature grid of the RGB morphological features, and performs a scaling transformation based on the pixel equivalent difference between the two cameras to make the resolution of the two feature grids consistent.

[0032] The spatially aligned hyperspectral texture features and RGB morphological features are concatenated at the channel level to obtain a fused feature tensor. The number of channels in the fused feature tensor is the sum of the number of channels in the hyperspectral texture features and the number of channels in the RGB morphological features. This fused feature tensor is then input into a convolutional network containing two 3×3 convolutional layers and one 2×2 max-pooling layer. The first convolutional layer outputs 64 channels with ReLU activation; the second convolutional layer outputs 32 channels with ReLU activation; and the pooling layer has a stride of 2. The output of the convolutional network is the separation feature map of the overlapping leaves. In the separation feature map, the boundary regions between overlapping leaves have higher response values, while the internal regions of the leaves have lower response values.

[0033] refer to Figure 4 The separated feature map is binarized, and the Otsu method is used to automatically determine the binarization threshold. Let L be the gray level of the separated feature map, and let the number of pixels with gray value i be... If the total number of pixels is N, then the probability of grayscale value i appearing is: Divide the image into two classes: foreground and background. The foreground grayscale value ranges from 0 to T, and the background grayscale value ranges from T+1 to L-1. Then the inter-class variance is:

[0034] in Foreground probability, For background probability, The average gray level of the foreground. The average gray level of the background. Iterating through all possible T values ​​will make... The largest T value is used as the binarization threshold to generate a binarized mask.

[0035] An eight-neighbor connected component search is performed on the binarized mask image, and a seed-filling algorithm is used to mark each independent connected region. Each pixel in the binarized mask image is traversed. When an unmarked foreground pixel is encountered, it is used as a seed point and marked as the current connected region number. Then, the eight neighboring pixels of the seed point are checked, and unmarked foreground pixels are added to the seed queue. This process is repeated until the seed queue is empty, completing the marking of one connected region.

[0036] Calculate the area and aspect ratio of the bounding rectangle for each independent connected region. The area is the total number of pixels contained within the connected region, the bounding rectangle is the smallest rectangle that can completely enclose the connected region, and the aspect ratio is the ratio of the rectangle's length to its width. The preset area threshold is determined based on the average pixel area of ​​a single blade, and the preset aspect ratio range is determined based on the shape characteristics of the single blade. Independent connected regions with areas smaller than the preset area threshold and bounding rectangle aspect ratios exceeding the preset aspect ratio range are removed. The set of boundary pixels of the remaining independent connected regions constitutes the outline of a single blade. The average coordinates of all pixels within the single blade outline are extracted and used as the coordinates of the center point of the single blade outline.

[0037] Where N is the total number of pixels inside the contour. Let be the coordinates of the i-th pixel.

[0038] refer to Figure 5 The coordinates of the center point of a single blade profile are input into the dynamic interpolation sorting model, which simultaneously receives the real-time pulse frequency output from the conveyor belt encoder. The cubic spline interpolation calculation layer in the dynamic interpolation sorting model uses the fixed physical distance S between the detection position and the execution position of the sorting robot arm as the interpolation interval, converting the real-time pulse frequency into an instantaneous velocity sequence v(t) within the interpolation interval. The cubic spline interpolation function S(t) in each sub-interval... The above is a cubic polynomial with continuous first and second derivatives over the entire interpolation interval. After obtaining the instantaneous velocity sequence by solving the cubic spline interpolation equations, time integration is performed over the interpolation interval to obtain the predicted actual displacement.

[0039] in The moment when the blade leaves the detection position. The predicted time when the blade arrives at the sorting robot arm's execution position. Subtracting the fixed physical distance from the predicted actual displacement value yields the displacement compensation amount:

[0040] The motion displacement compensation is superimposed onto the coordinates of the center point of the single blade profile to obtain the compensated blade coordinates.

[0041] Obtain the standard motion cycle time required for a sorting robot arm to complete a single grasping action. Using this duration as a time window, blade coordinates whose timestamps arrive at the sorting robot's execution position fall within the same time window are grouped into the same batch. Within the same batch, blade coordinates are sorted from smallest to largest based on their ordinate values ​​with motion displacement compensation; a smaller ordinate value indicates that the blade is positioned further forward in the conveyor belt's transport direction. The sorted blade coordinates are then combined in time sequence to generate a time-series sorting queue. The data processing unit sends this time-series sorting queue to the sorting robot's controller, which in turn controls the sorting robot to perform the grasping action sequentially according to the coordinates in the queue, placing the grasped blades into the corresponding packaging containers.

[0042] Table 1. Correspondence between timestamps and encoder pulse counts when acquiring the same leaf image using a linear RGB camera and a hyperspectral camera. Leaf number First timestamp (ms) Second timestamp (ms) Time difference (ms) Total number of pulses Physical displacement distance (mm) 1 125632 125687 55 110 27.5 2 125741 125798 57 114 28.5 3 125853 125912 59 118 29.5 4 125967 126028 61 122 30.5 5 126082 126145 63 126 31.5 Table 1 shows the relationship between the acquisition timestamps of five consecutive blades and the corresponding encoder pulse counts. The time differences between different blades correspond to different physical displacement distances, which is due to slight fluctuations in the conveyor belt speed. By constructing a nonlinear mapping relationship using the physical displacement distance as the horizontal axis offset, the acquisition time differences between different blades can be accurately compensated, ensuring the spatial alignment accuracy of multi-source features.

[0043] This embodiment fully realizes the entire process of multi-source visual data acquisition, spatiotemporal alignment and fusion, blade contour extraction, dynamic displacement compensation, and time-series sorting queue generation. By establishing a nonlinear mapping relationship between the pixel coordinate system and the physical conveyor belt coordinate system, the feature misalignment problem caused by the difference in the multi-source visual acquisition perspective is solved. The motion displacement compensation amount is calculated through a dynamic interpolation sorting model, which adapts to the variable speed operation state of the conveyor belt.

[0044] In a preferred embodiment, before recording the first timestamp of the first row of pixels on the same blade captured by the linear RGB camera, the data processing unit first obtains the physical horizontal distance between the linear RGB camera and the area array hyperspectral camera in the conveyor belt transport direction. This physical horizontal distance is the straight-line distance between the projection points of the two cameras' principal optical axes on the conveyor belt plane, measured by a laser rangefinder. The static trigger delay value is obtained by dividing the physical horizontal distance by the conveyor belt speed corresponding to the previous historical pulse cycle of the conveyor belt encoder.

[0045] Where D is the physical horizontal spacing. The speed of the conveyor belt corresponds to the previous historical pulse cycle. The static trigger delay value is written as the basic offset into the hardware trigger controller of the linear RGB camera. The hardware trigger controller adjusts the trigger time of the linear RGB camera according to this basic offset, so as to maximize the physical overlap field of view between the linear RGB camera and the area array hyperspectral camera when acquiring the same blade.

[0046] When the conveyor belt speed changes, the data processing unit updates the conveyor belt speed corresponding to the historical pulse cycle in real time and recalculates the static trigger delay value, writing the updated static trigger delay value into the hardware trigger controller. When the hardware trigger controller of the line scan RGB camera receives the trigger pulse from the encoder, it initiates line scanning acquisition. For each row of pixels acquired, it sends a line synchronization signal to the data processing unit, which records the timestamp corresponding to each line synchronization signal. When the data processing unit detects that the sum of the grayscale values ​​of multiple consecutive rows of pixels exceeds a preset blade detection threshold, it determines that a blade has entered the field of view of the line scan RGB camera and uses the timestamp corresponding to the first row of pixels exceeding the threshold as the first timestamp.

[0047] The trigger signal for the area array hyperspectral camera is generated by the data processing unit based on the first timestamp and static trigger delay value of the linear array RGB camera. When the trigger time is reached, the data processing unit sends the trigger signal to the area array hyperspectral camera, which acquires a frame of hyperspectral image and sends the acquisition completion timestamp as the second timestamp to the data processing unit. The data processing unit calculates the time difference between the first and second timestamps and simultaneously records the total number of pulses output by the conveyor belt encoder within this time difference using a counter module. The counter module is reset to zero at the first timestamp and latched at the second timestamp; this count value represents the total number of pulses within the time difference.

[0048] The physical conveyor belt displacement distance is obtained by multiplying the total number of pulses by the equivalent displacement of a single pulse on the conveyor belt. The equivalent displacement of a single pulse is determined during the system calibration phase by placing a standard-length calibration block on the conveyor belt, recording the total number of pulses as the calibration block passes through the field of view of the linear RGB camera, and then dividing the length of the calibration block by the total number of pulses. A nonlinear mapping relationship is constructed from the pixel coordinate system to the physical conveyor belt coordinate system, with the physical conveyor belt displacement distance as the horizontal axis offset and the height difference between the principal optical axes of the linear RGB camera and the area array hyperspectral camera in the physical conveyor belt coordinate system as the vertical axis offset. For any pixel in the hyperspectral image acquired by the area array hyperspectral camera... Its coordinates (X,Y) in the physical conveyor belt coordinate system satisfy:

[0049] in This represents the pixel equivalent in the x-direction of an area array hyperspectral camera. The pixel equivalent in the y-direction of the area array hyperspectral camera. This represents the abscissa offset of the area array hyperspectral camera. This represents the ordinate offset of the area array hyperspectral camera.

[0050] Table 2. Correspondence between static trigger delay value and the proportion of physically overlapping field of view. Static trigger delay value (ms) Physical overlap field of view percentage (%) Leaf feature integrity (%) 40 62 78 45 75 86 50 88 94 55 92 97 60 85 91 65 71 82 Table 2 shows the percentage of physically overlapping fields of view and the completeness of blade features for different static trigger delay values. When the static trigger delay value is set to 55ms, the percentage of physically overlapping fields of view reaches 92%, and the completeness of blade features reaches 97%, which is the optimal setting. When the static trigger delay value deviates from the optimal value, the percentage of physically overlapping fields of view decreases, and the completeness of blade features decreases, which will affect the effect of subsequent feature fusion.

[0051] This embodiment maximizes the physical overlap of the two cameras' fields of view by pre-setting a static trigger delay value, thereby improving the integrity of the same blade's feature data. By accurately recording the trigger timestamps of the two cameras and combining them with the encoder pulse count to calculate the physical displacement distance, an accurate nonlinear mapping relationship between the pixel coordinate system and the physical conveyor belt coordinate system is constructed, providing a reliable coordinate reference for subsequent feature spatiotemporal alignment and fusion.

[0052] In a preferred embodiment, hyperspectral images acquired by an area array hyperspectral camera are input into a three-dimensional tensor decomposition network (TDN) to extract hyperspectral texture features. During the training phase, the TDN uses labeled hyperspectral leaf images as the training set, training the network parameters with the goal of minimizing the reconstruction error. The reconstruction error is defined as the mean square error between the original hyperspectral tensor and the reconstructed tensor.

[0053] in To reconstruct the tensor, the network parameters are updated using the backpropagation algorithm until the reconstruction error is less than a preset threshold.

[0054] During the inference phase, the acquired hyperspectral images are input into a trained 3D tensor decomposition network to obtain spatial factor matrices and spectral factor matrices. Column vectors corresponding to the 450nm to 950nm wavelength range in the spectral factor matrix are extracted as hyperspectral texture features. This wavelength range contains characteristic absorption peaks of substances such as chlorophyll, carotenoids, and tea polyphenols in the leaves, effectively reflecting the texture and composition information of the leaves.

[0055] After converting the RGB images captured by the linear RGB camera into grayscale images, they are input into the edge gradient calculation network. The edge gradient calculation network first performs Gaussian filtering on the grayscale image to remove noise, and then uses the Sobel operator to calculate the gradient components. The Gaussian filter uses a 5×5 Gaussian kernel with a standard deviation of 1.0. The filtered grayscale image reduces the impact of noise on edge detection and improves the accuracy of gradient calculation.

[0056] By utilizing a constructed nonlinear mapping relationship, the spectral reflectance curve and the morphological boundary gradient vector are spatially aligned in the physical conveyor belt coordinate system. The horizontal and vertical coordinate offsets in the nonlinear mapping relationship are extracted to construct a two-dimensional affine transformation matrix. This matrix is ​​used to perform a translation transformation on the spatial feature grid corresponding to the hyperspectral texture features, ensuring that the origin of the transformed spatial feature grid's coordinate system coincides with the origin of the feature grid containing the morphological boundary gradient vector. Simultaneously, based on the pixel equivalent ratio of the linear RGB camera and the area-array hyperspectral camera, the spatial feature grid of the hyperspectral texture features is scaled to ensure that the physical distance between each pixel in both feature grids is consistent.

[0057] The spatially aligned hyperspectral texture features and RGB morphological features are concatenated at the channel level to obtain a fused feature tensor. The size of the fused feature tensor is... Where H' and W' are the height and width of the feature grid, The number of channels for hyperspectral texture features. The number of channels represents the RGB morphological features. The fused feature tensor is input into a convolutional network. The first convolutional layer performs preliminary feature extraction from the fused features, the second convolutional layer further extracts high-level features, and pooling layers reduce the resolution of the feature map, thus reducing computational cost. The output of the convolutional network is a separation feature map of overlapping leaves, which clearly distinguishes the boundaries between overlapping leaves.

[0058] Table 3. Contribution of spectral reflectance at different wavelengths to blade edge separation Wavelength range (nm) Mean spectral reflectance Edge separation contribution (%) 450-500 0.23 12 500-550 0.31 18 550-600 0.45 22 600-650 0.38 15 650-700 0.16 28 700-750 0.52 5 Table 3 shows the contribution of spectral reflectance in different wavelength bands to leaf edge separation. The 650nm to 700nm band has the highest contribution, reaching 28%. This band corresponds to the red light absorption peak of chlorophyll, and the spectral reflectance difference between the leaf and the background, as well as between overlapping leaves, is greatest in this band, effectively distinguishing leaf boundaries. The 550nm to 600nm band has the second highest contribution, at 22%, corresponding to the green light reflection peak of the leaf.

[0059] This embodiment extracts texture features from hyperspectral images using a three-dimensional tensor decomposition network and extracts morphological features from RGB images using an edge gradient calculation network. It achieves precise spatiotemporal alignment of the two features in the physical conveyor belt coordinate system using a two-dimensional affine transformation matrix and generates a separation feature map that can effectively separate overlapping blades through a convolutional network, thus solving the problem that a single camera cannot accurately identify the contours of overlapping blades.

[0060] In a preferred embodiment, after binarizing the separated feature map, a morphological opening operation is performed on the binarized mask image to remove small noise points. The morphological opening operation first performs an erosion operation and then a dilation operation, using a 3×3 rectangular structuring element. The erosion operation can remove small foreground regions, and the dilation operation can restore the shape of the remaining foreground regions.

[0061] An eight-neighbor connected component search is performed on the processed binarized mask image, and an optimized seed-filling algorithm is used to mark each independent connected region. The optimized seed-filling algorithm uses a queue to store seed points, avoiding the stack overflow problem caused by recursive algorithms. Simultaneously, the algorithm calculates the area and bounding rectangle parameters of the connected regions in real time while marking them, reducing the time overhead of subsequent calculations.

[0062] Calculate the area of ​​each independent connected region and the aspect ratio of its bounding rectangle. A preset area threshold is set to 60% of the average pixel area of ​​a single leaf, and a preset aspect ratio range is set to 1.2 to 3.0. Connected regions with an area smaller than the preset area threshold are typically leaf fragments or noise, while connected regions with an aspect ratio exceeding the preset range are typically irregular regions formed by multiple leaves adhering together. After removing these unsuitable connected regions, the set of boundary pixels of the remaining independent connected regions constitutes the outline of a single leaf.

[0063] The mean coordinates of all pixels within the contour of a single blade are extracted and used as the coordinates of the blade's center point. These center point coordinates are then input into a dynamic interpolation sorting model, which simultaneously receives the real-time pulse frequency output from the conveyor belt encoder. Before inputting the real-time pulse frequency into the cubic spline interpolation layer, the five adjacent historical pulse frequencies of the real-time pulse frequency are extracted, and the frequency change rate of these historical pulse frequencies is calculated.

[0064] in For real-time pulse frequency, This represents the historical pulse frequency from 5 cycles ago. The time interval between adjacent pulses is defined as follows: The absolute value of the frequency change rate is checked against a set abrupt change threshold. If the absolute value exceeds the threshold, a sudden change in conveyor belt speed is determined, the real-time pulse frequency is locked to the average of historical pulse frequencies, and the locked real-time pulse frequency is input into the cubic spline interpolation layer. If the absolute value does not exceed the threshold, the real-time pulse frequency is directly input into the cubic spline interpolation layer.

[0065] The cubic spline interpolation calculation layer employs natural boundary conditions, meaning the second derivative at both ends of the interpolation interval is zero. The coefficients of the cubic spline interpolation function are obtained by solving a tridiagonal system of equations, leading to the instantaneous velocity sequence within the interpolation interval. Time integration of the instantaneous velocity sequence yields the predicted actual displacement. Subtracting a fixed physical distance from the predicted actual displacement gives the displacement compensation. This compensation is then superimposed onto the blade center point coordinates to obtain the compensated blade coordinates.

[0066] refer to Figure 6 The standard motion cycle time required for the sorting robot arm to complete a single grasping action is obtained. Using this time as a time window, the blade coordinates whose timestamps arriving at the robot arm's execution position fall within the same time window are grouped into the same batch. Within the same batch, the blade coordinates are sorted from smallest to largest based on their ordinate values ​​with motion displacement compensation. The Euclidean distance between two adjacent blade coordinates after sorting is calculated.

[0067] in The coordinates of the previous leaf are given. The coordinates of the next blade are used. The Euclidean distance is compared to the maximum physical size of the end effector of the sorting robot arm. If the Euclidean distance is less than the maximum physical size, it is determined that the distance between the two blades is too close, and the sorting robot arm cannot grasp them simultaneously. The blade coordinates that appear later in the sorting queue are deleted, and the deleted blade coordinates are carried over to the next batch in the corresponding time window for reordering. This process is repeated until the Euclidean distance between all adjacent blade coordinates within the same batch is greater than or equal to the maximum physical size. The finally sorted blade coordinates are combined in a time sequence to generate a sorting queue, which is then sent to the sorting robot arm to perform the grasping action.

[0068] Table 4. Calculation results of displacement compensation under different conveyor belt speed fluctuations. Conveyor belt acceleration (m / s²) Real-time pulse frequency (Hz) Predicted actual displacement (mm) Fixed physical distance (mm) Motion displacement compensation (mm) 0 2000 500.0 500.0 0.0 0.2 2050 512.3 500.0 12.3 0.4 2100 525.1 500.0 25.1 -0.2 1950 487.6 500.0 -12.4 -0.4 1900 474.8 500.0 -25.2 Table 4 shows the calculated displacement compensation results under different conveyor belt accelerations. When the conveyor belt is running at a constant speed, the displacement compensation is 0; when the conveyor belt is accelerating, the displacement compensation is positive; and when the conveyor belt is decelerating, the displacement compensation is negative. As the conveyor belt acceleration increases, the absolute value of the displacement compensation also increases, demonstrating that the dynamic interpolation sorting model can adapt to the variable speed operation of the conveyor belt and accurately calculate the displacement compensation.

[0069] This embodiment accurately extracts the contour and center point coordinates of a single blade through connected component analysis, calculates accurate motion displacement compensation through frequency mutation detection and cubic spline interpolation, and generates a reasonable temporal sorting queue by batch division and adjacent blade distance judgment, ensuring stable grasping of the sorting robot arm under the conditions of conveyor belt speed change and dense blade distribution.

Claims

1. A leaf sorting and packaging method based on multi-source vision and sorting algorithms, characterized in that, A multi-source vision acquisition node is constructed using a linear RGB camera and a planar hyperspectral camera to acquire a continuous image sequence of moving blades on a conveyor belt. By using the trigger time difference between the linear RGB camera and the area hyperspectral camera and the conveyor belt encoder pulse, a nonlinear mapping relationship between the pixel coordinate system and the physical conveyor belt coordinate system is established. The hyperspectral texture features of the area hyperspectral camera and the RGB morphological features of the linear RGB camera are fused together at the feature level in the physical conveyor belt coordinate system to generate a separation feature map of the overlapping blades. The contours of individual blades in the separated feature map are extracted using connected component analysis. The coordinates of the center point of the individual blade contours are input into a dynamic interpolation sorting model. The dynamic interpolation sorting model receives the real-time pulse frequency of the conveyor belt encoder and calculates the motion displacement compensation amount of the blade from the detection position to the execution position of the sorting robot arm based on the real-time pulse frequency. The blade coordinates with the motion displacement compensation amount are reorganized into a time-series sorting queue according to the motion cycle of the sorting robot arm, and the time-series sorting queue is sent to the sorting robot arm to perform the gripping and packaging action.

2. The leaf sorting and packaging method based on multi-source vision and sorting algorithm according to claim 1, characterized in that, The step of establishing a nonlinear mapping relationship between the pixel coordinate system and the physical conveyor belt coordinate system by using the trigger time difference between the linear RGB camera and the area hyperspectral camera and the conveyor belt encoder pulse includes: recording the first timestamp of the first row of pixels of the same blade acquired by the linear RGB camera, recording the second timestamp of the center pixel of the same blade acquired by the area hyperspectral camera, and calculating the time difference between the first timestamp and the second timestamp. Extract the total number of pulses output by the conveyor belt encoder within the time difference, multiply the total number of pulses by the single-pulse equivalent displacement of the conveyor belt, and obtain the physical conveyor belt displacement distance corresponding to the time difference. Using the displacement distance of the physical conveyor belt as the horizontal axis offset and the height difference between the principal optical axis of the linear RGB camera and the principal optical axis of the area array hyperspectral camera in the physical conveyor belt coordinate system as the vertical axis offset, the nonlinear mapping relationship from the pixel coordinate system to the physical conveyor belt coordinate system is constructed.

3. The leaf sorting and packaging method based on multi-source vision and sorting algorithm according to claim 2, characterized in that, The step of performing feature-level spatiotemporal alignment and fusion of the hyperspectral texture features and the RGB morphological features in the physical conveyor belt coordinate system to generate a separation feature map of overlapping blades includes: inputting the hyperspectral image acquired by the area array hyperspectral camera into a three-dimensional tensor decomposition network, and extracting the spectral reflectance curve of the hyperspectral image in a set band range as the hyperspectral texture features. The RGB image captured by the linear RGB camera is input into the edge gradient calculation network, and the morphological boundary gradient vector of the RGB image is extracted as the RGB morphological feature; Using the nonlinear mapping relationship, the spectral reflectance curve and the morphological boundary gradient vector are spatially aligned in the physical conveyor belt coordinate system. The spatially aligned spectral reflectance curve and the morphological boundary gradient vector are then concatenated at the channel level and input into a convolutional network to generate the separation feature map of the overlapping blades.

4. The leaf sorting and packaging method based on multi-source vision and sorting algorithm according to claim 3, characterized in that, The step of extracting the single-blade outline from the separated feature map using connected component analysis includes: performing binarization processing on the separated feature map to obtain a binarized mask map; An eight-neighbor connected component search is performed on the binarized mask image to mark each independent connected region in the binarized mask image; Calculate the area of ​​each independent connected region and the aspect ratio of its circumscribed rectangle. Eliminate independent connected regions whose area is less than a preset area threshold and whose aspect ratio of the circumscribed rectangle exceeds a preset ratio range. The set of boundary pixels of the remaining independent connected regions constitutes the outline of the single leaf. Extract the average coordinates of all pixels inside the single blade outline, and use the average coordinates as the coordinates of the center point of the single blade outline.

5. The leaf sorting and packaging method based on multi-source vision and sorting algorithm according to claim 4, characterized in that, The calculation of the displacement compensation of the blade from the detection position to the execution position of the sorting robot arm based on the real-time pulse frequency includes: inputting the real-time pulse frequency into the cubic spline interpolation calculation layer in the dynamic interpolation sorting model, taking the fixed physical distance between the detection position and the execution position of the sorting robot arm as the interpolation interval, and converting the real-time pulse frequency into an instantaneous velocity sequence within the interpolation interval; The instantaneous velocity sequence is integrated over time within the interpolation interval to obtain the predicted value of the actual motion displacement. The motion displacement compensation amount is obtained by subtracting the fixed physical distance from the actual predicted motion displacement value.

6. The leaf sorting and packaging method based on multi-source vision and sorting algorithm according to claim 5, characterized in that, The step of reorganizing the blade coordinates with the motion displacement compensation amount into a time-series sorting queue according to the motion cycle of the sorting robot arm includes: obtaining the standard motion cycle duration required for the sorting robot arm to complete a single grasping action; Using the standard motion cycle duration as a time window, the blade coordinates that fall within the same time window when the timestamp of the sorting robot arm's execution position are classified into the same batch. Within the same batch, the blade coordinates are sorted from smallest to largest according to the ordinate values ​​of the blade coordinates with the motion displacement compensation, and the sorted blade coordinates are combined in time sequence to generate the time sequence sorting queue.

7. The leaf sorting and packaging method based on multi-source vision and sorting algorithm according to claim 6, characterized in that, Before recording the first timestamp of the first row of pixels of the same blade acquired by the linear RGB camera, the method further includes: obtaining the physical horizontal distance between the linear RGB camera and the area array hyperspectral camera in the conveyor belt conveying direction; Divide the physical horizontal spacing by the conveyor belt speed corresponding to the previous historical pulse cycle of the conveyor belt encoder to obtain the static trigger delay value. The static trigger delay value is written as a basic offset into the hardware trigger controller of the linear RGB camera, so that the physical overlap field of view between the linear RGB camera and the area array hyperspectral camera is maximized when acquiring the same leaf.

8. The leaf sorting and packaging method based on multi-source vision and sorting algorithm according to claim 7, characterized in that, The step of aligning the spectral reflectance curve and the morphological boundary gradient vector in the physical conveyor belt coordinate system using the nonlinear mapping relationship includes: extracting the horizontal coordinate offset and the vertical coordinate offset in the nonlinear mapping relationship and constructing a two-dimensional affine transformation matrix; The spatial feature grid corresponding to the spectral reflectance curve is translated and scaled using the two-dimensional affine transformation matrix, so that the origin of the coordinate system of the transformed spatial feature grid coincides with the origin of the coordinate system of the feature grid where the morphological boundary gradient vector is located.

9. The leaf sorting and packaging method based on multi-source vision and sorting algorithm according to claim 8, characterized in that, Before inputting the real-time pulse frequency into the cubic spline interpolation calculation layer of the dynamic interpolation sorting model, the method further includes: extracting several adjacent historical pulse frequencies of the real-time pulse frequency and calculating the frequency change rate of the historical pulse frequencies. Determine whether the absolute value of the frequency change rate is greater than a set mutation threshold. If it is greater than the mutation threshold, lock the real-time pulse frequency to the average of the historical pulse frequencies and input the locked real-time pulse frequency into the cubic spline interpolation calculation layer.

10. The leaf sorting and packaging method based on multi-source vision and sorting algorithm according to claim 9, characterized in that, The step of generating the time-series sorting queue by combining the sorted blade coordinates in a time sequence includes: extracting two adjacent blade coordinates after sorting and calculating the Euclidean distance between the two adjacent blade coordinates; The Euclidean distance is compared with the maximum physical size of the end effector of the sorting robot arm. If the Euclidean distance is less than the maximum physical size, the blade coordinates that are sorted later in the time-series sorting queue are deleted, and the deleted blade coordinates are carried over to the next batch corresponding to the time window for reordering.

Images