Machine vision assisted food sorting operation method and system

Abstract

The application discloses a kind of machine vision assisted food sorting operation method and system.The method comprises: food raw material single layer is transported to detection station;Through position encoder trigger visible light camera, hyperspectral camera and near infrared spectrum sensor synchronous acquisition appearance and spectral data;Temporal and spatial registration and feature fusion are carried out, appearance and internal nutritional quality characteristics are extracted, and quality category is divided based on terminal processing adaptability, comprehensive score and sorting instruction are generated;Dynamic scheduling module adopts load balancing distribution and collision avoidance path planning, controls multiple parallel robots cooperative capture delivery;Self-learning optimization module uses incremental learning and reinforcement learning to continuously update quality evaluation model and sorting strategy.Realize internal and external quality synchronous nondestructive testing, raw material differentiation terminal matching and multi-robot efficient cooperation, with comprehensive detection dimension, high sorting precision, strong self-adaptive ability and other advantages, significantly improve the processing adaptation value and sorting automation level of food raw materials.

Description

Technical Field

[0001] This invention relates to the field of food processing automation and intelligent manufacturing technology, and more specifically, to a machine vision-assisted food sorting method and system. Background Technology

[0002] Food sorting is a crucial step in agricultural product processing and food manufacturing, and its efficiency and accuracy directly impact product quality and economic benefits. In recent years, machine vision technology has been widely applied in food sorting due to its advantages such as non-contact operation, speed, and objectivity. For example, existing technologies include visible light image-based surface defect detection devices that achieve primary sorting by identifying features such as color, size, shape, and damage on the food surface; there are also methods using near-infrared spectroscopy for non-destructive testing of internal food quality (such as sugar content, acidity, moisture content, and protein content); and some systems are even exploring the use of hyperspectral imaging to simultaneously acquire spatial and spectral information. Furthermore, to improve sorting efficiency, high-speed dynamic sorting systems based on parallel robots have been used on food production lines, capable of hundreds of grasping actions per minute.

[0003] However, existing technologies still have the following shortcomings: First, most systems rely on only a single detection dimension (such as appearance or internal quality), making it difficult to simultaneously acquire internal and external quality information of food under high-speed conveying conditions, resulting in an incomplete comprehensive evaluation of the processing value of raw materials. Second, existing sorting decisions are mainly based on simple pass / fail or a few grade classifications, lacking the ability to establish a correlation between food quality characteristics and end-processing suitability (such as the differentiated requirements of different end products for the nutritional components of raw materials), and thus failing to achieve precise raw material orientation based on intended use. Third, in multi-robot collaborative sorting scenarios, existing task allocation and path planning methods are mostly statically set, making it difficult to adaptively handle dynamic changes such as fluctuations in conveying speed, uneven robot load, and grasping failures, affecting sorting success rate and system stability. Finally, existing systems generally lack self-learning capabilities, failing to continuously optimize detection models and sorting strategies using accumulated data in actual production, resulting in poor adaptability.

[0004] Therefore, how to achieve high-speed synchronous detection and integrated evaluation of the internal and external quality of food, establish accurate sorting logic based on terminal adaptability, improve the intelligent level of dynamic collaboration of multiple robots, and endow the system with self-learning optimization capabilities are technical problems that urgently need to be solved in this field. Summary of the Invention

[0005] The technical problem to be solved by this invention is how to achieve high-speed synchronous detection and integrated evaluation of the internal and external quality of food, and to establish a precise sorting logic based on terminal adaptability. In view of the above-mentioned defects of the prior art, this invention provides a machine vision-assisted food sorting operation method and system.

[0006] The technical solution adopted by this invention to solve its technical problem is: on the one hand A machine vision-assisted food sorting method includes the following steps: S1. After pretreatment, the food raw materials to be sorted are transported to the testing station in a single-layer uniform arrangement. S2. At the detection station, multimodal data of food raw materials are collected synchronously through a multimodal sensing module. The multimodal data includes appearance image data, hyperspectral image data, and near-infrared spectral data. S3. The collected multimodal data is transmitted to the multi-source data fusion and analysis module for spatiotemporal registration and feature fusion. Appearance quality features and internal nutritional quality features are extracted. Based on the appearance quality features and internal nutritional quality features, a comprehensive quality evaluation result is generated through the quality evaluation module. The corresponding sorting instruction is determined based on the comprehensive quality evaluation result. S4. The sorting instruction is sent to the dynamic scheduling module. The dynamic scheduling module performs adaptive task allocation and grasping path planning based on the real-time working status of the multiple parallel robots, the dynamic location information of the food to be sorted, and the priority in the sorting instruction. S5. The intelligent sorting execution module controls the multiple parallel robots to cooperate in grabbing food raw materials according to the pre-planned grabbing path, and delivers the food raw materials to the designated graded collection channel. S6. By learning and optimizing the actual sorting results of the mobile phone and downstream feedback data, the quality evaluation module and task allocation and grasping path planning strategies are continuously updated using incremental learning and reinforcement learning algorithms.

[0007] Preferably, the synchronous acquisition in S2 is based on a location encoding-triggered three-synchronization mechanism, which specifically includes: Time-domain synchronization is achieved by using a position encoder installed on the conveyor belt to detect the speed of the conveyor belt in real time. When the food raw material arrives at the detection station, the encoder generates a trigger signal, which simultaneously triggers the visible light camera, hyperspectral camera, and near-infrared spectral sensor to perform sampling. Spatial registration uses the fixed edge line of the conveyor belt as a unified reference benchmark to establish a spatial coordinate mapping matrix of visible light image, hyperspectral image and near-infrared spectral sensor points to achieve pixel-level alignment. Image fusion combines color, size, and defect features extracted from aligned visible light images with protein, fat, and water content features retrieved from hyperspectral images, and uses dot matrix data from near-infrared spectral sensors to locally correct the fusion result.

[0008] Preferably, the generation of a comprehensive quality evaluation result in step S3, and the determination of the corresponding sorting instruction based on the comprehensive quality evaluation result, specifically includes: Construct a set of multimodal feature vectors, including at least color features, size features, defect features, protein content, fat content, and water content; Based on the multimodal feature vectors, food raw materials are divided into multiple pre-set quality categories, and the quality categories correspond to the end-processing adaptability. For raw materials within the same quality category, the overall quality score is calculated using the following formula. : ; in For dynamic weighting coefficients, t represents the appearance scoring function, nutrition scoring function, and defect penalty function, respectively; Based on the quality category and comprehensive quality score, a sorting instruction set is generated, which includes the dynamic location coordinates of the target material, the target collection channel number, and the grabbing priority.

[0009] Preferably, the adaptive task allocation and crawling path planning in S4 specifically includes: Dynamic target tracking uses the pulse value of the position encoder to calculate the instantaneous coordinates of food raw materials on the conveyor belt in real time and predict the time domain window for them to enter each robot's work area. The load balancing strategy, with the objective function of minimizing the total sorting time and balancing the load of each robot, dynamically distributes the raw materials to be sorted to multiple parallel robots. The objective function expression is as follows: ; in Total sorting time, For the first The current load of the robot For average load, These are the weighting coefficients; Collision avoidance path planning: For each assigned grasping task, a multi-objective optimization algorithm is used to generate the shortest, collision-free, and energy-efficient grasping trajectory from the robot's current position to the target grasping point. An abnormal recovery mechanism is in place. When a robot fails to grasp a material or is overloaded, the dynamic scheduling module automatically triggers a neighboring robot to preemptively recover the material.

[0010] Preferably, the incremental learning and reinforcement learning algorithms in S6 specifically include: Set incremental learning cycle Within each cycle, the actual test results are compared with data from manual review or downstream processing feedback, the deviation is calculated, and the dynamic weighting coefficient is updated on a rolling basis. ; Record the actual grasping success rate, grasping time and robot energy consumption for each sorting task. Construct a reward function based on the actual grasping success rate, grasping time and robot energy consumption. Continuously optimize the load balancing allocation strategy and collision avoidance path planning parameters through reinforcement learning algorithms. The quality distribution characteristics and corresponding optimal end uses of different batches and varieties of food raw materials are stored in a knowledge base to form a quality and use knowledge graph, which is used for the initial quality category classification of new batches of raw materials.

[0011] on the other hand A machine vision-assisted food sorting system includes: The raw material pretreatment module is used to evenly spread bulk food raw materials and transport them to the testing station in a single-layer arrangement. The multimodal sensing module includes at least one visible light camera, at least one near-infrared hyperspectral camera, at least one near-infrared spectral sensor, a synchronous light source system, and a position encoder, used to synchronously acquire appearance image data, hyperspectral image data, and near-infrared spectral data of food raw materials; The multi-source data fusion and analysis module, including edge computing units and a central server, is used to perform spatiotemporal registration, feature extraction and fusion analysis on multimodal data, and generate comprehensive quality evaluation results and corresponding sorting instructions. The intelligent sorting execution module includes multiple parallel Delta robots, flexible end effectors, and multiple graded collection channels, which are used to perform gripping and delivery actions according to sorting instructions; The dynamic scheduling module is communicatively connected to the multi-source data fusion analysis module and the intelligent sorting execution module, respectively, and is used to perform adaptive task allocation and grasping path planning based on the real-time working status of each robot, the dynamic position of food raw materials and the priority of sorting instructions. The self-learning optimization module includes a model training unit, an incremental learning unit, and a knowledge base management unit. It is used to continuously collect actual sorting results and downstream feedback data, and to update the evaluation model in the multi-source data fusion analysis module and the allocation and planning strategy in the dynamic scheduling module online. The knowledge base management unit stores a knowledge graph of quality characteristics and end-processing suitability; the self-learning optimization module is also used to: when a new batch of food raw materials enters the system, based on prior knowledge of historically similar varieties in the knowledge graph, preset the initial quality category classification threshold and the initial weight coefficient of the comprehensive quality score in the multi-source data fusion analysis module. The beneficial effects of this invention are as follows: 1. By employing a three-synchronous acquisition mechanism, this invention organically integrates visible light imaging, near-infrared hyperspectral imaging, and near-infrared spectral sensing. This enables the simultaneous acquisition of the appearance characteristics (color, size, surface defects) and internal nutritional indicators (protein, fat, moisture content, etc.) of food raw materials under high-speed transport conditions. Compared to existing technologies that rely on only a single detection dimension (either appearance or internal), this invention achieves non-destructive, real-time, and collaborative detection of both internal and external quality, significantly improving the comprehensiveness and accuracy of raw material quality evaluation.

[0012] 2. This application differs from fixed-grade sorting models by constructing a three-level mapping model from quality characteristics and quality categories to end-product adaptation. Based on the nutritional composition and appearance characteristics of food raw materials, they are automatically categorized into quality categories suitable for different end products (such as tofu, soy milk, soybean oil, etc.), and differentiated sorting instructions are generated. This solution enables targeted diversion of raw materials according to downstream processing needs, maximizing the economic value and processing adaptability of raw materials, and avoiding resource waste caused by "misuse of good materials and abuse of inferior materials."

[0013] 3. The dynamic scheduling module incorporates a load-balancing-based adaptive task allocation algorithm and a multi-objective collision avoidance path planning method. This allows for real-time tracking of the dynamic position of food ingredients on the conveyor belt, intelligent task allocation to multiple parallel robots, and dynamic generation of the shortest, collision-free, and energy-efficient grasping trajectory. Simultaneously, an abnormal material recovery mechanism is designed; when a robot fails to grasp an ingredient or is overloaded, a neighboring robot can automatically take over the recovery. This system maintains a high grasping success rate and system stability even under high-speed conveying conditions, effectively improving the overall efficiency of the equipment.

[0014] 4. Integrating incremental learning and reinforcement learning algorithms, a closed-loop self-learning optimization module was constructed. The system can collect actual sorting results and downstream processing feedback data, automatically update the weight parameters in the quality evaluation model, and continuously optimize task allocation and path planning strategies. As the running time increases, the system's detection accuracy and sorting efficiency continuously improve. It can adapt to the quality distribution changes of different varieties, batches, and origins of food raw materials, and has strong generalization ability and industrial practical value. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a flowchart illustrating the steps of a machine vision-assisted food sorting method according to an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention. Example 1

[0017] Preferred embodiments of the present invention, for example Figure 1 As shown, a machine vision-assisted food sorting method includes the following steps: S1. After pre-processing such as washing, impurity removal, and cutting (if necessary), the food raw materials to be sorted are conveyed to the detection station in a single-layer, uniform, non-overlapping, and non-adhesive arrangement via a vibrating feeder and belt shaping device. The purpose is to ensure that the subsequent multimodal sensors can collect complete information of each individual raw material without obstruction, avoiding false detections and missed detections caused by stacking.

[0018] S2. At the inspection station, multimodal data of food raw materials are synchronously collected through a multimodal sensing module. This multimodal data includes appearance image data, hyperspectral image data, and near-infrared spectral data. Synchronous acquisition is based on a position-encoded trigger-triggered three-synchronization mechanism. This mechanism specifically addresses the spatiotemporal consistency problem of multi-source heterogeneous sensors in high-speed motion scenarios. The three-synchronization mechanism specifically includes: With time-domain synchronization, a high-precision incremental position encoder mounted on the conveyor belt drive shaft outputs pulse signals in real time that are proportional to the conveyor belt displacement. The system presets a detection trigger threshold (pulse count). When the encoder's accumulated pulse count indicates that the leading edge of the food material has just reached the preset detection optical axis, a hardware trigger signal is immediately generated, and the encoder count value at that moment is latched. This signal triggers the visible light camera, hyperspectral camera, and near-infrared spectral sensor in parallel with a microsecond-level delay, ensuring that all sensors capture the same material's transient physical properties in the same space. This fundamentally eliminates image ghosting and data misalignment caused by fluctuations in conveyor belt speed or asynchronous sensor triggering.

[0019] Spatial registration, using the fixed edge line of the conveyor belt as a unified reference, establishes a spatial coordinate mapping matrix for the visible light image, hyperspectral image, and near-infrared spectral sensor points, achieving pixel-level alignment. Specifically, through a joint calibration method based on an infrared reflection calibration plate, the homography matrix and spatial coordinate mapping matrix between the three coordinate systems (visible light image pixel coordinate system, hyperspectral image pixel coordinate system, and near-infrared spectral sensor array physical coordinate system) are pre-calculated and stored. During sorting, the system aligns the component content retrieved from each pixel in the hyperspectral image with the appearance features of the corresponding pixel in the visible light image at the sub-pixel level according to the mapping matrix, and maps the measured values ​​of the discrete array of the near-infrared spectral sensor to the same spatial grid, achieving pixel-level three-modal data registration.

[0020] Image fusion involves fusing color (e.g., Lab color space values), size (length, width, and area of ​​the minimum bounding rectangle), and defect features (scratches, black spots, mold areas) extracted from aligned visible light images with protein, fat, and moisture content features retrieved from hyperspectral images using the band ratio method and partial least squares regression (PLSR) model. This generates a multidimensional feature tensor of appearance and endogenous matter for each raw material, and the fusion result is locally corrected using lattice data from a near-infrared spectral sensor. The lattice data from the near-infrared spectral sensor (which has a higher signal-to-noise ratio and faster response speed) is used to locally weight and correct the predicted component content values ​​at corresponding spatial locations in the fused feature tensor, particularly for edge regions of hyperspectral images that are susceptible to uneven illumination and surface curvature, significantly improving the robustness and accuracy of endogenous matter detection.

[0021] S3. The collected multimodal data is transmitted to the multi-source data fusion and analysis module for spatiotemporal registration and feature fusion. Appearance quality features and internal nutritional quality features are extracted. Based on these features, a comprehensive quality evaluation result is generated through the quality evaluation module. The corresponding sorting instructions are then determined based on the comprehensive quality evaluation result. Specifically, this includes: Construct a multimodal feature vector set, including at least the color mean / variance, size parameter, defect area ratio / type encoding, mean protein content, mean fat content, and mean moisture content.

[0022] Based on multimodal feature vectors, and using support vector machines (SVM) or lightweight gradient boosting machines (LightGBM) classifiers, food raw materials are divided into multiple pre-set quality categories (e.g., Category A - for fresh cutting, Category B - for juicing, Category C - for deep processing, Category D - for rejecting substandard products), with each quality category corresponding to the suitability for the final processing.

[0023] For raw materials within the same quality category, the overall quality score is calculated using the following formula. (A higher value indicates better suitability for the terminal's intended use): ,in The weighting coefficients are dynamic, with initial values ​​set by the knowledge graph based on the raw material type and preset end-use. They are then continuously optimized during the incremental learning process in S6 based on downstream feedback (such as yield and customer satisfaction). For example, if downstream feedback indicates "low juice yield," the system automatically increases the weighting coefficients. (Nutritional weight); if the feedback is "high cutting and preparation loss", then increase the weight. (Appearance and size weights) and adjust the defect penalty coefficient. .

[0024] These are the appearance scoring function, nutrition scoring function, and defect penalty function, respectively. Cosine similarity calculation based on color, size, and pre-stored standard appearance templates. The calculation is based on the weighted Euclidean distance between the internal component content and the standard requirements of the end use (such as the need for high sugar and high acid in juicing). It is a non-linear function, such as g = log(1 + k·defect area ratio), to avoid a sharp drop in score due to minor defects.

[0025] Based on the quality category and overall quality score, a sorting instruction set is generated, including the dynamic location coordinates of the target material, the target collection channel number, and the grasping priority. As an optional embodiment, the calculation expression for the sorting instruction of each food ingredient is: ,in, The dynamic position coordinates of the raw material center in the current world coordinate system (calculated and updated in real time from the encoder value latched in S2); The estimated timestamp for arrival at the robot's work area; Collect channel numbers for target hierarchical classification; According to The priority of data collection is calculated based on factors such as raw material shelf life requirements (high Q value or perishable raw materials have higher priority).

[0026] S4. The sorting instruction is sent to the dynamic scheduling module. The dynamic scheduling module performs adaptive task allocation and grasping path planning based on the real-time working status of each of the parallel robots, the dynamic location information of the food to be sorted, and the priority in the sorting instruction. The adaptive task allocation and grasping path planning specifically include: Dynamic target tracking utilizes the pulse values ​​of the position encoder to calculate the instantaneous coordinates of food ingredients on the conveyor belt in real time and predict their entry into the working areas of each robot. Specifically, the module monitors the pulse increments of the position encoder in real time and, combined with the initial coordinates of each ingredient output by S3, continuously updates its instantaneous coordinates. Simultaneously, based on a constant velocity motion model and an extended Kalman filter (EKF), the future time window of each raw material is predicted. The precise trajectory within the area determines the time-domain window and entry / exit points for each robot to enter its work area. This allows the robot to plan its actions in advance, even before the raw materials arrive, achieving "predictive grasping."

[0027] The load balancing strategy, with the objective function of minimizing the total sorting time and balancing the load of each robot, dynamically distributes the raw materials to be sorted to multiple parallel robots. The objective function expression is as follows: ,in Total sorting time, For the first The current load of the robot is the number of... The estimated total execution time of all tasks in the robot's task queue (grabbing + throwing + resetting). This is to average the load and prevent individual robots from becoming system bottlenecks due to excessive workloads. These are the weighting coefficients. (Time of the robot completer task queue) - Current time, used to maximize system throughput.

[0028] For collision avoidance path planning, a multi-objective optimization algorithm is used to generate the shortest, collision-free, and energy-efficient grasping trajectory from the robot's current position to the target grasping point for each assigned grasping task. Specifically, for each assigned grasping task (dynamic position of raw materials, designated robot), an improved fast random search tree (FSB) is used. A smooth trajectory is planned in the robot's joint space or Cartesian space, either using a fifth-order polynomial curve or a quintic polynomial curve, from the robot's current position, through an avoidance midpoint, to the target grasping point. Multi-robot collision avoidance constraints are embedded as hard constraints in the planner to ensure that the workspaces of adjacent robots do not overlap and that the grasping sequence does not conflict.

[0029] An abnormal recovery mechanism is implemented. When a robot fails to grasp a material or is overloaded, the dynamic scheduling module automatically triggers a neighboring robot to preemptively recover the material. Specifically, when a robot determines a "grabbing failure" (e.g., empty grasp, material slippage) through vacuum sensors, tactile sensors, or grasping force feedback on its end effector, it immediately reports the failure event (with the predicted coordinates of the material at the time of failure) to the dynamic scheduling module. The scheduling module determines whether the material is still within the reachable working area of ​​a neighboring robot based on a preset time threshold. If so, it automatically elevates the grasping priority of the material to the highest level and triggers the neighboring robot to interrupt its current low-priority sub-actions and immediately perform preemptive recovery, adding the material to its task queue and replanning its interception trajectory. This significantly improves the system's fault tolerance for grasping failures and prevents material loss.

[0030] S5. The intelligent sorting execution module, based on the grasping trajectory planned in S4, sends interpolation motion commands to the designated parallel Delta robot via a high-performance real-time industrial Ethernet (such as EtherCAT). This controls the robot's end effector (such as an adaptive suction cup or soft gripper) to precisely grasp food ingredients at the predicted spatiotemporal intersection point and accurately deliver the ingredients to the corresponding graded collection channel according to the channel number in the command.

[0031] S6. By learning from the actual sorting results of the mobile phone and downstream feedback data, the quality evaluation module and task allocation and grasping path planning strategies are continuously updated using incremental learning and reinforcement learning algorithms. The incremental learning and reinforcement learning algorithms specifically include: Set incremental learning cycle (For example, after processing 1000 raw materials), within each cycle, the actual test results are compared with manual verification or downstream processing feedback data, the deviation is calculated, and the dynamic weighting coefficient is updated on a rolling basis. Specifically, within each cycle, the quality evaluation and sorting results generated by the system are compared with high-confidence feedback data (such as manual sampling verification results, real-time juice yield / finished product yield measured by downstream processing lines, and customer return records), and the prediction deviation between the feature vector and the score is calculated. An online gradient descent or mini-batch update algorithm is used to continuously update the dynamic weight coefficients in S3. This includes the decision boundaries of the SVM / LightGBM classifier. This allows the quality assessment model to quickly adapt to seasonal raw material differences, the introduction of new varieties, and changes in downstream demand.

[0032] The system records the actual success rate, time, and robot energy consumption for each sorting task. A reward function is constructed using these metrics, and a reinforcement learning algorithm is used to continuously optimize the load balancing strategy and collision avoidance path planning parameters. Specifically, the entire sorting process is modeled as a Markov Decision Process (MDP). The state space includes: robot load, conveyor belt material density distribution, priority queue of materials to be picked, and average system energy consumption. The action space consists of: task allocation decisions (which robot to assign to) and trajectory planning parameters (grabbing speed, acceleration, and safety distance threshold). The reward function... The load balancing strategy in S4 is optimized through continuous iteration during the "exploration-exploitation" process in both the simulation environment and the real production line, employing Deep Q-Network (DQN) or Proximal Policy Optimization (PPO) algorithms. The adjustment rules) and the parameters of the collision avoidance path planner (such as the collision avoidance path planner parameters) (Step size, maximum search time).

[0033] The quality characteristic distribution (multidimensional vector center, variance) and corresponding optimal end-use (verified by downstream feedback) of food raw materials from different batches, varieties, origins, and seasons are stored as a structured knowledge graph. Nodes in the graph represent "variety," "origin / batch," "quality characteristic cluster," and "end-use," while edges represent relationships such as "possessing characteristics," "suitable for," and "feedback benefits." When a new batch of raw materials enters the system, the self-learning optimization module first retrieves the 3-5 most similar historical varieties / batches from the knowledge graph and uses their prior knowledge to: preset the initial quality category classification threshold in S3; and set a comprehensive quality score. Dynamic weighting coefficients Initialize the value of the initial value; initialize the reinforcement learning strategy to shorten the cold start time.

[0034] Example 2 A machine vision-assisted food sorting system is provided for executing any of the method steps in Embodiment 1. The system includes a raw material pretreatment module, a multimodal perception module, a multi-source data fusion and analysis module, an intelligent sorting execution module, a dynamic scheduling module, and a self-learning optimization module. The system modules are interconnected via a Time-Sensitive Network (TSN) based industrial switch to ensure low-latency, highly deterministic transmission of control commands and data synchronization.

[0035] The raw material pretreatment module includes a vibrating feeder, a belt shaping baffle, and brush rollers, which are used to evenly spread bulk food raw materials and transport them to the testing station in a single layer and directional (such as the direction of the long axis of the conveyor belt) arrangement.

[0036] The multimodal sensing module includes at least one visible light camera, at least one near-infrared hyperspectral camera, at least one near-infrared spectral sensor, a synchronous light source system, and a position encoder, used to synchronously acquire appearance image data, hyperspectral image data, and near-infrared spectral data of food raw materials. The visible light camera, near-infrared hyperspectral camera, and near-infrared spectral sensor are arranged sequentially above and / or to the side of the same detection station along the direction of conveyor belt movement; the position encoder is mounted on the conveyor belt drive roller, and its output pulse signal serves as an external trigger source for the visible light camera, hyperspectral camera, and near-infrared spectral sensor; the synchronous light source system includes a visible light LED light source and a near-infrared light source, and the emission timing of the two is synchronously controlled with the camera exposure timing by the trigger signal of the position encoder.

[0037] The multi-source data fusion analysis module includes an edge computing unit (located on the production line side, responsible for real-time spatiotemporal registration, feature extraction, and lightweight classification) and a central server (responsible for complex fusion models, global optimization, and knowledge graph management). Together, they generate comprehensive quality evaluation results and corresponding sorting instructions, used for spatiotemporal registration, feature extraction, and fusion analysis of multimodal data to generate comprehensive quality evaluation results and corresponding sorting instructions.

[0038] The intelligent sorting execution module includes multiple parallel Delta robots, flexible end effectors, and multiple graded collection channels, which are used to perform gripping and delivery actions according to sorting instructions.

[0039] The dynamic scheduling module, either as independent software middleware or a high real-time process running on the central server, communicates with the multi-source data fusion and analysis module (receiving sorting instructions) and the intelligent sorting execution module (reading robot status and issuing motion instructions) via a publish-subscribe (Pub-Sub) model. Internally, it includes a task queue manager and a trajectory planner library, used to perform adaptive task allocation and grasping path planning based on the real-time working status of each robot, the dynamic position of food ingredients, and the priority of sorting instructions.

[0040] The self-learning optimization module includes a model training unit (GPU server), an incremental learning unit, and a knowledge base management unit (graph database). It asynchronously collects real-time production line logs and feedback data from downstream information systems (such as MES and ERP) via message queues, and updates the evaluation model (SVM, weight coefficients) in the multi-source data fusion analysis module and the allocation and planning strategy (reinforcement learning policy network) in the dynamic scheduling module online. After the updated model is verified to be effective through A / B testing, it is deployed to the production environment via hot loading, achieving non-stop upgrades.

[0041] The knowledge base management unit stores a knowledge graph of quality characteristics and end-processing adaptability; the self-learning optimization module is also used to perform the following: when a new batch of food raw materials enters the system, based on the prior knowledge of similar historical varieties in the knowledge graph, the initial quality category classification threshold and the initial weight coefficient of the comprehensive quality score are preset in the multi-source data fusion analysis module.

[0042] The implementation principle of the machine vision-assisted food sorting method in this application is as follows: First, a three-synchronization mechanism solves the problem of simultaneously acquiring high-precision appearance images, internal component distribution maps, and point spectral data of food raw materials on a high-speed conveyor belt. Using a position encoder as the global clock and spatial anchor point, microsecond-level time synchronization and sub-pixel-level spatial registration of multi-source heterogeneous sensors are achieved. Furthermore, high signal-to-noise ratio near-infrared point spectral data is used to locally correct the hyperspectral inversion results, thereby establishing a spatiotemporally consistent, internally and externally compatible multimodal feature vector for each raw material.

[0043] Secondly, based on multidimensional features, the system adopts a dynamically weighted comprehensive quality evaluation model. This model uses an incremental learning algorithm to continuously receive feedback from downstream processing or consumer ends (such as juice yield, finished product rate, and customer satisfaction), dynamically adjusting the weight coefficients of indicators such as appearance, nutrition, and defects. This ensures that the definition of quality is no longer a static laboratory standard, but is dynamically linked to specific end-uses and actual commercial value. It can adapt to changes in different varieties, seasons, and market demands, achieving a leap from uniform grading to on-demand sorting.

[0044] Furthermore, addressing the efficiency and robustness challenges of multi-robot collaborative sorting, the system designs a closed-loop scheduling strategy based on prediction, allocation, planning, and recycling. By predicting the dynamic trajectory of raw materials using an extended Kalman filter, the "grabbing" task is transformed into a "spatiotemporal intersection" problem. By establishing a multi-objective optimization model that includes total sorting time and load balancing terms, real-time optimal allocation of tasks among multiple robots is achieved. Through collision-free trajectory planning and a "task preemptive anomaly recycling" mechanism, the system's fault tolerance for anomalies such as grabbing failures is significantly improved, ensuring the continuity and high throughput of the sorting process.

[0045] Finally, the entire system achieves self-evolution through reinforcement learning and incremental learning. Reinforcement learning continuously explores optimal task allocation and path planning strategies in simulated and real environments to improve success rates, reduce energy consumption, and shorten time. Incremental learning and knowledge graphs allow the quality assessment model to learn from each sorting and downstream feedback, continuously accumulating knowledge about the correlation between different raw material varieties, batches, and optimal uses. This makes the system smarter with use, possessing rapid adaptability across varieties, production lines, and seasons, fundamentally solving the pain points of traditional machine vision sorting systems where models are fixed and difficult to transfer.

[0046] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A machine vision-assisted food sorting method, characterized in that, Includes the following steps: S1. After pretreatment, the food raw materials to be sorted are transported to the testing station in a single-layer uniform arrangement. S2. At the detection station, multimodal data of food raw materials are collected synchronously through a multimodal sensing module. The multimodal data includes appearance image data, hyperspectral image data, and near-infrared spectral data. S3. The collected multimodal data is transmitted to the multi-source data fusion and analysis module for spatiotemporal registration and feature fusion. Appearance quality features and internal nutritional quality features are extracted. Based on the appearance quality features and internal nutritional quality features, a comprehensive quality evaluation result is generated through the quality evaluation module. The corresponding sorting instruction is determined based on the comprehensive quality evaluation result. S4. The sorting instruction is sent to the dynamic scheduling module. The dynamic scheduling module performs adaptive task allocation and grasping path planning based on the real-time working status of the multiple parallel robots, the dynamic location information of the food to be sorted, and the priority in the sorting instruction. S5. The intelligent sorting execution module controls the multiple parallel robots to cooperate in grabbing food raw materials according to the pre-planned grabbing path, and delivers the food raw materials to the designated graded collection channel. S6. By learning and optimizing the actual sorting results of the mobile phone and downstream feedback data, the quality evaluation module and task allocation and grasping path planning strategies are continuously updated using incremental learning and reinforcement learning algorithms.

2. The machine vision-assisted food sorting method according to claim 1, characterized in that, The synchronous acquisition in S2 is based on a location encoding-triggered three-synchronization mechanism, which specifically includes: Time-domain synchronization is achieved by using a position encoder installed on the conveyor belt to detect the speed of the conveyor belt in real time. When the food raw material arrives at the detection station, the encoder generates a trigger signal, which simultaneously triggers the visible light camera, hyperspectral camera, and near-infrared spectral sensor to perform sampling. Spatial registration uses the fixed edge line of the conveyor belt as a unified reference benchmark to establish a spatial coordinate mapping matrix of visible light image, hyperspectral image and near-infrared spectral sensor points to achieve pixel-level alignment. Image fusion combines color, size, and defect features extracted from aligned visible light images with protein, fat, and water content features retrieved from hyperspectral images, and uses dot matrix data from near-infrared spectral sensors to locally correct the fusion result.

3. The machine vision-assisted food sorting method according to claim 1, characterized in that, The process of generating a comprehensive quality evaluation result in S3, and determining the corresponding sorting instruction based on the comprehensive quality evaluation result, specifically includes: Construct a set of multimodal feature vectors, including at least color features, size features, defect features, protein content, fat content, and water content; Based on the multimodal feature vectors, food raw materials are divided into multiple pre-set quality categories, and the quality categories correspond to the end-processing adaptability. For raw materials within the same quality category, the overall quality score is calculated using the following formula. : ， in For dynamic weighting coefficients, t represents the appearance scoring function, nutrition scoring function, and defect penalty function, respectively; Based on the quality category and comprehensive quality score, a sorting instruction set is generated, which includes the dynamic location coordinates of the target material, the target collection channel number, and the grabbing priority.

4. The machine vision-assisted food sorting method according to claim 3, characterized in that, The adaptive task allocation and crawling path planning in S4 specifically includes: Dynamic target tracking uses the pulse value of the position encoder to calculate the instantaneous coordinates of food raw materials on the conveyor belt in real time and predict the time domain window for them to enter each robot's work area. The load balancing strategy, with the objective function of minimizing the total sorting time and balancing the load of each robot, dynamically distributes the raw materials to be sorted to multiple parallel robots. The objective function expression is as follows: ， in Total sorting time, For the first The current load of the robot For average load, These are the weighting coefficients; Collision avoidance path planning: For each assigned grasping task, a multi-objective optimization algorithm is used to generate the shortest, collision-free, and energy-efficient grasping trajectory from the robot's current position to the target grasping point. An abnormal recovery mechanism is in place. When a robot fails to grasp a material or is overloaded, the dynamic scheduling module automatically triggers a neighboring robot to preemptively recover the material.

5. A machine vision-assisted food sorting method according to claim 4, characterized in that, The incremental learning and reinforcement learning algorithms in S6 specifically include: Set incremental learning cycle Within each cycle, the actual test results are compared with data from manual review or downstream processing feedback, the deviation is calculated, and the dynamic weighting coefficient is updated on a rolling basis. ; Record the actual grasping success rate, grasping time and robot energy consumption for each sorting task. Construct a reward function based on the actual grasping success rate, grasping time and robot energy consumption. Continuously optimize the load balancing allocation strategy and collision avoidance path planning parameters through reinforcement learning algorithms. The quality distribution characteristics and corresponding optimal end uses of different batches and varieties of food raw materials are stored in a knowledge base to form a quality and use knowledge graph, which is used for the initial quality category classification of new batches of raw materials.

6. A machine vision-assisted food sorting system, characterized in that, include: The raw material pretreatment module is used to evenly spread bulk food raw materials and transport them to the testing station in a single-layer arrangement. The multimodal sensing module includes at least one visible light camera, at least one near-infrared hyperspectral camera, at least one near-infrared spectral sensor, a synchronous light source system, and a position encoder, used to synchronously acquire appearance image data, hyperspectral image data, and near-infrared spectral data of food raw materials; The multi-source data fusion and analysis module, including edge computing units and a central server, is used to perform spatiotemporal registration, feature extraction and fusion analysis on multimodal data, and generate comprehensive quality evaluation results and corresponding sorting instructions. The intelligent sorting execution module includes multiple parallel Delta robots, flexible end effectors, and multiple graded collection channels, which are used to perform gripping and delivery actions according to sorting instructions; The dynamic scheduling module is communicatively connected to the multi-source data fusion analysis module and the intelligent sorting execution module, respectively, and is used to perform adaptive task allocation and grasping path planning based on the real-time working status of each robot, the dynamic position of food raw materials and the priority of sorting instructions. The self-learning optimization module includes a model training unit, an incremental learning unit, and a knowledge base management unit. It is used to continuously collect actual sorting results and downstream feedback data, and to update the evaluation model in the multi-source data fusion analysis module and the allocation and planning strategy in the dynamic scheduling module online. The knowledge base management unit stores a knowledge graph of quality characteristics and end-processing adaptability; the self-learning optimization module is also used to perform the following: when a new batch of food raw materials enters the system, based on the prior knowledge of historical similar varieties in the knowledge graph, the initial quality category classification threshold and the initial weight coefficient of the comprehensive quality score are preset in the multi-source data fusion analysis module.

Images