Berry lossless sorting method and system based on heterogeneous visual information space-time registration

Abstract

The invention provides a berry lossless sorting method and system based on heterogeneous visual information space-time registration, and belongs to the technical field of agricultural product automatic lossless detection and sorting. According to the method, when berries move along with a conveying belt, image sequences are collected through a linear array hyperspectral camera and a visible light camera, and internal quality features and center coordinates are extracted. The fixed linear edge of the conveyor belt is used as a unique space reference datum, the projection distances from the feature information and the coordinates to the datum are calculated, matching is conducted through the sequence of the projection distances, and the corresponding relation between the quality and the position is established. And then, the escape distance of the berries relative to the mechanical arm movement boundary (namely the downstream boundary of the working area of the parallel robot) in the conveying direction is calculated, a grabbing priority queue is dynamically generated from near to far according to the escape distance, and the parallel robot is controlled to complete sorting according to the quality. According to the method, the problem of dislocation of asynchronous collection of heterogeneous visual information is solved through a software algorithm, and the sorting omission rate is remarkably reduced through dynamic priority scheduling.

Description

Technical Field

[0001] This invention belongs to the field of automated non-destructive testing and sorting technology for agricultural products, and specifically relates to a non-destructive sorting method and system for berries based on spatiotemporal registration of heterogeneous visual information. Background Technology

[0002] Blueberries are a general term for plants in the genus *Vaccinium* of the Ericaceae family that produce blue or bluish-purple fruits. When ripe, the fruits are mostly deep blue or violet. Globally, the main blueberry producing areas are in North America, while in my country, wild blueberry resources are mainly distributed in forest areas such as the Changbai Mountains, the Greater Khingan Mountains, and the Lesser Khingan Mountains. In recent years, with the continuous expansion of blueberry cultivation, the harvesting process has faced significant challenges: manual harvesting is time-consuming, labor-intensive, and costly; while mechanized harvesting is limited by the lack of suitable agricultural machinery and difficulties in determining ripeness. Furthermore, regardless of whether harvesting is manual or mechanical, subsequent fruit sorting generally suffers from low efficiency and poor uniformity. Therefore, how to achieve efficient and non-destructive automated sorting of blueberries has become a pressing technical problem that needs to be solved to improve industry efficiency.

[0003] A review of publicly available technical solutions reveals several key aspects. For instance, invention CN116060309A uses a visible light camera to capture images and analyze blueberry diameters, then sorts them using an air-blowing device based on size. Invention CN118635144A employs a dual-camera system combined with a target detection model to classify blueberries based on appearance (size, color, shape) and has them removed by a robotic arm. The core problems of existing technologies can be summarized in two points: First, in disordered and dynamic conveying scenarios, it is difficult to reliably and with low latency bind asynchronously or displaced internal quality information to the precise spatial location of the fruit. Second, sorting decisions are static or singular, unable to simultaneously respond to internal quality grading requirements and dynamically changing spatial constraints (such as the fruit needing to be moved out of the work area), leading to missed or incorrect sorting, or low efficiency. Therefore, how to achieve a low-latency, highly robust correlation between the internal quality and spatial location of berries under continuous movement and random, disordered feeding conditions, and how to combine this with dynamic priority scheduling to improve sorting success rates, is a pressing technical problem to be solved in this field. Summary of the Invention

[0004] In view of this, the present invention proposes a non-destructive sorting method and system for berries based on spatiotemporal registration of heterogeneous visual information, so as to provide a non-destructive sorting method and system that can achieve real-time and accurate binding of internal quality information and spatial location information of berries under continuous movement and random disordered feeding conditions, and dynamically schedule the grasping order based on escape risk.

[0005] A non-destructive berry sorting method based on heterogeneous visual spatiotemporal registration is applied to a sorting system including a conveyor belt, a hyperspectral imaging unit, a visible light imaging unit, and a parallel robot. The method includes:

[0006] Step 1: Heterogeneous visual information acquisition. During the continuous movement of the berries along the conveyor belt, hyperspectral image sequences and visible light image sequences of the same batch of berries flowing through the detection area are acquired separately.

[0007] Step 2: Quality and Location Feature Extraction. Internal quality feature information of each berry is extracted from the hyperspectral image sequence; the center coordinates of each berry in the pixel coordinate system are detected and extracted from the visible light image sequence.

[0008] Step 3: Spatiotemporal registration based on spatial topological sorting. Using a fixed straight edge extending along the length of the conveyor belt as the sole spatial reference, calculate the first projection distance from the quality feature information of each berry in the hyperspectral image to this reference, and the second projection distance from the center coordinates of each berry in the visible light image to this reference. Ignoring the image acquisition timestamp information, sort the quality feature information based solely on the magnitude of the first projection distance to generate a first ordered sequence, and sort the center coordinates based on the magnitude of the second projection distance to generate a second ordered sequence. Match the data items with the same sequence number in the first and second ordered sequences one-to-one to establish the correspondence between the quality feature information and the center coordinates.

[0009] Step 4: Dynamic Escape Priority Sorting. The quality characteristics of each associated berry and its corresponding center coordinates are used to construct a sorting task set; the conveyor belt speed is acquired in real time, and the escape distance of each berry's center coordinates relative to the downstream boundary of the parallel robot's working area along the conveying direction is calculated; the grasping priority is determined according to the escape distance from near to far; when the escape distance of a berry is less than a preset safe distance threshold, the berry is marked as the highest priority; the parallel robot is controlled to grasp the berries sequentially according to the priority queue, and the berries are distributed to different areas based on the quality characteristics.

[0010] Step 5: Coordinate Transformation (Preferred Step). Based on the affine transformation matrix pre-established using the nine-point calibration method, the center coordinates are transformed from the pixel coordinate system to spatial coordinates in the parallel robot base coordinate system. The affine transformation matrix is ​​obtained by fitting the pixel coordinates and robot coordinates of the nine calibration points using the least squares method.

[0011] The present invention also provides a non-destructive berry sorting system for implementing the above method, comprising:

[0012] The conveyor belt has guide plates on both sides, and the inner wall of one of the guide plates forms the fixed straight edge reference datum.

[0013] The hyperspectral imaging unit, located above the conveyor belt, includes a linear array hyperspectral camera and a halogen lamp light source, and is used to acquire hyperspectral image sequences of berries;

[0014] A visible light imaging unit, positioned above the conveyor belt and downstream of the hyperspectral imaging unit along the conveying direction, includes an area array visible light camera for acquiring visible light image sequences of berries.

[0015] Parallel robots are positioned above the sorting area at the end of the conveyor belt, and their end effectors are vacuum suction cups.

[0016] The control and processing system, which is communicatively connected to the hyperspectral imaging unit, the visible light imaging unit, and the parallel robot, is configured to execute the above-described method.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] This invention solves the technical challenge of synchronous correlation in asynchronous acquisition. Instead of relying on high-precision hardware synchronization signals, it utilizes a software algorithm—"spatial topology sorting and matching based on a common physical reference"—to achieve robust spatiotemporal registration of two heterogeneous visual information sources: hyperspectral linear array scanning and visible light area array imaging. This effectively overcomes information misalignment caused by differences in acquisition timing and conveyor belt movement.

[0019] Significantly improves dynamic sorting efficiency. By introducing "escape distance" as the basis for dynamic priority arbitration, the grasping order of parallel robots responds to the conveyor belt movement status in real time, prioritizing the grasping of high-risk targets that are about to be moved out of the work area, fundamentally reducing missed picking caused by target escape.

[0020] Achieving integrated, non-destructive sorting based on both internal and external quality. By using hyperspectral technology to retrieve internal quality information such as sugar content, combined with visible light positioning and data registration, it overcomes the limitations of sorting based solely on external features, achieving true "quality-based sorting".

[0021] It supports flexible and automated production. The system is compatible with random and disordered feeding, eliminating the need for additional sorting mechanisms, simplifying the structure and reducing costs. At the same time, the parallel robot, in conjunction with the vacuum suction cup end effector, achieves compliant gripping, resulting in a high fruit integrity rate. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart of a non-destructive berry sorting method based on heterogeneous visual spatiotemporal registration provided in an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of the overall system structure layout provided in an embodiment of the present invention;

[0025] Figure 3 A detailed flowchart of the "spatiotemporal registration" mechanism provided in this embodiment of the invention;

[0026] Figure 4 This is a schematic diagram illustrating the principle of the "escape priority" dynamic capture strategy provided in this embodiment of the invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment uses blueberries as an example, but the invention is equally applicable to other small berries such as cherries, strawberries, and mulberries.

[0028] I. System Overall Structure

[0029] Please see Figure 2 The blueberry non-destructive sorting system provided in this embodiment mainly includes: a conveyor belt 1, a hyperspectral camera 2, a halogen lamp light source 3, a visible light camera 4, a parallel robot body 5, and a sorting frame 6.

[0030] Conveyor belt 1: Horizontally arranged, with a matte black surface to reduce glare interference, and guide panels on both sides. The inner walls of the guide panels form a fixed physical edge reference.

[0031] Hyperspectral camera 2: Installed directly above conveyor belt 1, it uses a linear array push-broom imaging method and has a spectral range of 400-1000nm.

[0032] Halogen lamp light source 3: symmetrically arranged on both sides of hyperspectral camera 2 to provide a uniform and stable lighting environment for hyperspectral imaging. The entire hyperspectral imaging area is placed in a darkroom.

[0033] Visible light camera 4: Installed directly above conveyor belt 1, downstream of hyperspectral camera 2 along the conveying direction, with a spacing of approximately 30-50cm. It uses a CMOS sensor with a resolution of no less than 2 megapixels. Parallel robot 5: Adopting a Delta configuration, it is installed above the sorting area at the end of conveyor belt 1. The end effector is a vacuum suction cup with a diameter of 8-12mm and a working negative pressure of -50kPa.

[0034] Sorting box 6: Placed within the working radius of parallel robot 5, divided according to blueberry sugar content grade.

[0035] The control and processing system is a host computer (industrial control computer) that communicates with the hyperspectral camera 2, visible light camera 4, and parallel robot 5 via Ethernet or USB interface. It runs a Windows or Linux operating system and has a built-in image acquisition card, GPU accelerator card, and motion control card. Those skilled in the art should understand that the specific configuration of this industrial control computer is merely an example, and the functions of the control and processing system can also be implemented collaboratively by multiple distributed computing devices, and are not limited to a single industrial control computer.

[0036] Furthermore, this embodiment also provides a non-destructive berry sorting method based on spatiotemporal registration of heterogeneous visual information, including:

[0037] Step S1: Collect heterogeneous visual information: Blueberries are placed at the beginning of conveyor belt 1 by manual or automatic feeding device in a natural scattered manner. This invention does not set a forced sorting mechanism to achieve random and disordered feeding. Conveyor belt 1 runs continuously at a constant speed v (e.g., 0.1-0.3 m / s).

[0038] When the blueberries move to a position below the hyperspectral camera 2, the control and processing system triggers the hyperspectral camera 2 to perform linear push-broom imaging and acquire hyperspectral image data. Reflectance calibration (using a standard white board and dark current images) is performed before and after acquisition to eliminate the influence of ambient light and equipment noise. Subsequently, as the blueberries continue to move along the conveyor belt to a position below the visible light camera 4, the control and processing system triggers the visible light camera 4 to acquire area array images.

[0039] Step S2: Extraction of quality and location features

[0040] (1) Internal Quality Feature Extraction: The control and processing system performs image segmentation on the hyperspectral image and extracts the average spectral curve of each blueberry region of interest (ROI). The spectral curve is preprocessed: first, multivariate scattering correction (MSC) is used to eliminate scattering effects, and then Savitzky-Golay smoothing filtering (window width 11, polynomial order 2) is used for noise reduction. The preprocessed spectral data is input into a pre-trained neural network model (e.g., a BP neural network with the input layer dimension matching the number of spectral bands, 64 hidden neurons, and 1 output neuron, outputting a sugar content prediction value) to obtain the predicted sugar content value of the blueberry, which serves as the internal quality feature information.

[0041] (2) Location Feature Extraction: The control and processing system inputs the visible light image into a pre-trained target detection network (e.g., a deep learning-based detection model, already trained on a labeled blueberry image dataset), and outputs the coordinates of the detection bounding box for each blueberry:

[0042] ( , )=( )

[0043] in,( , ) represents the minimum pixel coordinates of the blueberry detection box. , The maximum pixel coordinates of the blueberry detection bounding box are given. The pixel-level center coordinates of each blueberry are calculated. , ).

[0044] Step S3: Spatiotemporal registration based on spatial topological sorting, please refer to [link / reference]. Figure 3 This step is used to address the spatiotemporal differences between hyperspectral linear array scanning and visible light area array imaging.

[0045] Specifically, the control and processing system uses the inner wall of the left guide vane of conveyor belt 1 (presented as a fixed vertical edge line in the image) as a common reference datum L. For each blueberry i detected in the hyperspectral image (with a sugar content H...) i ), calculate the distance d from the center of its bounding box to the reference datum L. Hi For each blueberry j detected in the visible light image (with center coordinate x), cj , y cj Calculate the distance d from its center to the reference datum L. Vj .

[0046] Since the width between the guide panels on both sides of the conveyor belt is fixed and less than twice the diameter of the blueberry, only 2-3 blueberries can pass side by side on the same cross section. Therefore, the sorting along the conveying direction (the row direction in the image) has a clear physical meaning: the blueberries that enter the detection area first have a smaller d value (or a larger d value, depending on whether the reference datum is located on the left or right side of the image, which is determined by calibration).

[0047] The control and processing system respectively converts H in the hyperspectral image i Press d Hi Arrange in ascending order to generate the first ordered sequence H1, H2, ..., H n ; to extract x from a visible light image cj , y cj Press d Vj Sort in ascending order to generate a second ordered sequence P1, P2, ..., P n Then, data items with the same sequence number are matched: H1 corresponds to P1, H2 corresponds to P2, and so on, thereby accurately assigning the sugar content value to the corresponding spatial coordinates.

[0048] Robustness Enhancement: When the difference in d-values ​​between two or more blueberries is less than a preset threshold (e.g., 5 pixels), the control and processing system introduces a secondary sorting criterion—the distance of each blueberry to another fixed boundary in the image perpendicular to the direction of conveyor belt movement (e.g., the upper or lower boundary of the image)—to eliminate matching ambiguity.

[0049] Step S4: Dynamic Escape Priority Sorting: Please refer to Figure 4 After the spatiotemporal registration in step S3, the control and processing system obtains the coordinate data (H1, P1), (H2, P2), ..., (Hn, Pn) of the bound sugar content values.

[0050] The control and processing system acquires the conveyor belt speed v in real time (which can be read through encoder feedback or a set value). For each blueberry P k Calculate the escape distance D from its current position along the conveying direction to the downstream boundary B of the parallel robot's working area. remain The downstream boundary B of the work area is a pre-defined fixed position (e.g., a certain distance behind the center of the parallel robot along the conveying direction).

[0051] Control and processing system according to D remain A priority queue is dynamically generated in ascending order, with the blueberry having the shortest escape distance (most urgent) receiving the highest priority. When a blueberry's D... remain When the distance is less than a preset safe distance threshold (e.g., 5cm), the control and processing system marks it as an emergency task, inserts it at the head of the queue, and executes it immediately.

[0052] Parallel robot 5, based on the highest priority instruction, plans a smooth trajectory (typically using a fifth-order polynomial or trapezoidal velocity planning), drives the vacuum suction cup to move above the target blueberry, descends to make contact, and applies vacuum suction (the contact time between the suction cup and the blueberry surface is no more than 0.2 seconds, negative pressure -50kPa), then transports the blueberry above the corresponding sorting box 6, releasing the vacuum to complete the sorting. After one grab is completed, the control and processing system removes the blueberry from the task set, recalculates the priority of the remaining blueberries, and updates the queue until all blueberries in the current batch have been processed.

[0053] Step S5: Coordinate Transformation: Before controlling the parallel robot to grasp, the pixel coordinates need to be converted into spatial coordinates in the robot's base coordinate system. This embodiment uses the nine-point calibration method: the calibration board is placed at different positions on the conveyor belt's working plane, and the coordinates (u, u) of the nine feature points on the calibration board in the pixel coordinate system are recorded. i ,v i ) and its corresponding coordinates (X) in the parallel robot base coordinate system. i Y iThe affine transformation matrix M is solved using the least squares method. During actual operation, the pixel coordinates obtained in step S3 are substituted into matrix M to obtain the target coordinates X that the robot can directly use. target Y target Since the blueberries are basically at the same height on the conveyor belt, the Z coordinate can be set to a fixed value (which needs to be calibrated according to the working distance of the suction cups).

[0054] The preferred parameter ranges for the above steps are as follows: conveyor belt speed: 0.1 ~ 0.5 m / s, 0.2 m / s recommended; hyperspectral camera acquisition frame rate: 100 ~ 200 fps (linear array); visible light camera acquisition frame rate: 30 ~ 60 fps; parallel robot grasping cycle: 0.8 ~ 1.5 seconds / cycle (including movement, adsorption, and release); vacuum suction cup negative pressure: -50 kPa (recommended value); suction cup diameter: 8 ~ 12 mm (blueberry average diameter is 12-16 mm, suction cup is slightly smaller than fruit diameter); installation distance between the hyperspectral imaging unit and the visible light imaging unit: 30 cm to 50 cm, this distance is determined based on the conveyor belt speed and image processing delay, so that the same berry is still within the same batch association window when collected by the two units successively.

[0055] A blueberry sorting experiment was conducted using the above system, and compared with a conventional sorting system with no escape priority ranking (all other conditions being the same). The results are as follows:

[0056]

[0057] Experimental data show that the present invention is significantly superior to the existing technology in key indicators such as sorting success rate, missed sorting rate, and fruit integrity rate.

[0058] Those skilled in the art should understand that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any adjustments, modifications, or equivalent substitutions made to the following aspects without departing from the concept of the present invention should be included within the scope of protection of the present invention:

[0059] Replace blueberries with other small berries (cherries, strawberries, mulberries, raspberries, etc.); replace sugar content with other internal quality indicators (acidity, firmness, ripeness, etc.); replace the BP neural network with other machine learning or deep learning models (CNN, random forest, support vector machine, etc.); replace the nine-point calibration method with other calibration methods (four-point calibration, direct linear transformation, etc.); replace the parallel robot with other robot configurations (SCARA, six-axis articulated robots, etc.), as long as its end effector can achieve suction cup grasping.

[0060] The above embodiments are merely preferred embodiments of the present invention, used to clearly demonstrate the principles and implementation of the present invention. Any adjustments or modifications made by those skilled in the art based on these embodiments, such as changes to camera models, installation distances, light source configurations, neural network structures, or sorting algorithm details, as long as they employ the same core technical concept of the present invention, should be included within the scope of protection of the present invention.

Claims

1. A non-destructive berry sorting method and system based on spatiotemporal registration of heterogeneous visual information, characterized in that, The method, applied to a sorting system including a conveyor belt, a hyperspectral imaging unit, a visible light imaging unit, and a parallel robot, comprises: Step 1: During the continuous movement of the berries along the conveyor belt, collect hyperspectral image sequences and visible light image sequences of the same batch of berries flowing through the detection area; Step 2: Extract the internal quality feature information of each berry from the hyperspectral image sequence; detect and extract the center coordinates of each berry in the pixel coordinate system from the visible light image sequence; Step 3: Using a fixed straight edge extending along the length of the conveyor belt as the sole spatial reference, calculate the first projection distance of the quality feature information of each berry in the hyperspectral image to the reference, and the second projection distance of the center coordinates of each berry in the visible light image to the reference; disregarding the image acquisition timestamp information, sort the quality feature information according to the magnitude of the first projection distance to generate a first ordered sequence, and sort the center coordinates according to the magnitude of the second projection distance to generate a second ordered sequence; match the data items with the same sequence number in the first ordered sequence and the second ordered sequence one by one to establish the correspondence between the quality feature information and the center coordinates; Step 4: Construct a sorting task set by associating the quality characteristic information of each berry and its corresponding center coordinates; acquire the conveyor belt speed in real time, calculate the escape time of the center coordinate of each berry relative to the downstream boundary of the parallel robot's working area along the conveying direction; dynamically generate a grasping priority queue in order of escape time from shortest to longest; control the parallel robot to grasp the berries in sequence according to the priority queue, and distribute the berries to different areas according to the quality characteristic information.

2. The method according to claim 1, characterized in that, The extraction of internal quality feature information in step 2 specifically includes: extracting the average spectral curve of each region of interest in the hyperspectral image, performing multivariate scattering correction and Savitzky-Golay smoothing filtering preprocessing on the average spectral curve in sequence, and inputting the preprocessed spectral data into a pre-trained neural network model to obtain the sugar content prediction value of the berry.

3. The method according to claim 1, characterized in that, Step 2, extracting the center coordinates, specifically includes: inputting the visible light image into a pre-trained target detection network, outputting the detection bounding boxes for each berry, and calculating the center coordinates using a formula: ( , )=( ) in( , ) and( , ) are the minimum and maximum pixel coordinates of the detected bounding box, respectively.

4. The method according to claim 1, characterized in that, In step 3, when the absolute value of the difference between the first projection distance or the second projection distance of two or more berries to the reference datum is less than a preset pixel threshold, a secondary sorting is performed based on the distance of each berry to another fixed boundary in the image perpendicular to the direction of conveyor belt movement, in order to eliminate matching ambiguity.

5. The method according to claim 1, characterized in that, The dynamic generation of the grasping priority queue in step 4 specifically includes: acquiring the conveyor belt speed in real time, calculating the escape distance of the center coordinates of each berry along the conveying direction relative to the downstream boundary of the working area of ​​the parallel robot; determining the grasping priority in order of escape distance from near to far; and marking the berry as the highest priority when the escape distance of a certain berry is less than the preset safe distance threshold.

6. The method according to claim 1, characterized in that, Step 4 is followed by a coordinate transformation step: based on the affine transformation matrix established in advance by the nine-point calibration method, the center coordinates are transformed from the pixel coordinate system to the spatial coordinates under the parallel robot base coordinate system. The affine transformation matrix is ​​obtained by fitting the pixel coordinates and robot coordinates of the nine calibration points by the least squares method.

7. A non-destructive berry sorting system for implementing the method according to any one of claims 1-6, characterized in that, include: The conveyor belt has guide plates on both sides, and the inner wall of one of the guide plates forms the fixed straight edge reference datum. The hyperspectral imaging unit, located above the conveyor belt, includes a linear array hyperspectral camera and a halogen lamp light source, and is used to acquire hyperspectral image sequences of berries; A visible light imaging unit, positioned above the conveyor belt and downstream of the hyperspectral imaging unit along the conveying direction, includes an area array visible light camera for acquiring visible light image sequences of berries. Parallel robots are positioned above the sorting area at the end of the conveyor belt, and their end effectors are vacuum suction cups. The control and processing system is communicatively connected to the hyperspectral imaging unit, the visible light imaging unit, and the parallel robot, and is configured to perform the method described in any one of claims 1-6.

8. The system according to claim 7, characterized in that, The installation distance between the hyperspectral imaging unit and the visible light imaging unit is 30cm to 50cm. This distance is determined based on the conveyor belt speed and image processing delay, so that the same berry is still within the same batch association window when it is collected by the two units successively.

9. The system according to claim 7, characterized in that, The parallel robot is of Delta configuration, and the working negative pressure of its vacuum suction cup is set to -40kPa to -60kPa. The contact time between the suction cup and the berry surface is no more than 0.2 seconds.

Images