A picking robot end-cloud collaborative vision model self-evolution system and method

Abstract

The application discloses a picking robot end-cloud collaborative visual model self-evolution system and method, which adopts an end-cloud collaborative architecture and comprises a picking robot carrying an image acquisition unit, an airborne computing unit, a wireless communication unit and a work execution unit, and a cloud server unit; the airborne computing unit is internally provided with a fruit and vegetable recognition and segmentation-positioning module, a difficult example sample active sampling module, a visual reinspection feedback module, a local cache and update management module, and the cloud server unit is internally provided with a standardized data management module and a cross-scenario anti-forgetting incremental training module. The airborne end actively screens difficult example samples and completes authenticity determination based on target point cloud three-dimensional geometric features, the cloud realizes anti-forgetting incremental training through historical sample playback, hierarchical network freezing and joint loss constraint, and drives the visual model to continuously and autonomously iterate with work. The application effectively alleviates the catastrophic forgetting problem of incremental learning, and significantly improves the robustness, cross-scenario generalization and long-term work stability of the model in a complex field environment.

Description

Technical Field

[0001] This invention relates to the fields of agricultural robots, computer vision and smart agricultural equipment technology, and in particular to a self-evolving system and method for a cloud-edge collaborative visual model for harvesting robots. Background Technology

[0002] With the rapid development of the smart agriculture industry, fruit and vegetable harvesting robots have become core equipment to replace manual labor in repetitive and high-intensity harvesting operations. Currently, the fruit and vegetable recognition and segmentation models of mainstream harvesting robots are pre-trained based on labeled datasets from laboratory environments or specific planting scenarios and varieties. However, in actual open-air fields or greenhouses, these models are highly susceptible to complex environmental factors such as fluctuations in light intensity, overlapping and shading of branches and leaves, differences in crop varieties and phenotypic characteristics, weather changes, and uneven fruit ripening. This leads to a rapid decline in model inference accuracy and a continuous decrease in harvesting success rate, severely hindering the large-scale commercial deployment of harvesting robots.

[0003] Chinese invention patent application (publication number CN120167229A) discloses a fruit and vegetable picking robot system based on data augmentation and recognition accuracy optimization. It uses the YOLOv5 target detection algorithm to identify fruits and vegetables and uses image augmentation algorithms to improve environmental adaptability. However, this solution can only achieve passive adaptation of the algorithm and lacks a closed loop of label verification based on the physical results of the picking operation. The model cannot be automatically repaired in time after its accuracy decays during the operation. Chinese invention patent application (publication number CN116363505A) discloses a target picking method based on the vision system of the picking robot. It improves the target recognition accuracy in complex environments through multi-stage visual detection and model optimization. However, this solution does not propose a solution to the "learning new and forgetting old" problem in the incremental training process of the model. After the model adapts to new scenarios and new varieties, the recognition accuracy of the original scenarios is prone to a significant decrease. It has insufficient cross-scenario generalization and the existing network optimization strategies rely on specific model architectures, which limit the generalization and applicability. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned defects of the prior art and provide a self-evolving system and method for a cloud-edge collaborative visual model for harvesting robots. Through the cloud-edge collaborative architecture, a complete process of "inference-sampling-feedback-correction-training-update" is constructed during the continuous operation of the harvesting robot, realizing the model's on-the-spot optimization, self-iteration, and self-evolution. This completely solves the core problems of poor field adaptability, rapid accuracy decay, easy "learning new and forgetting old" in incremental training, and lack of self-evolution ability in the prior art, and significantly improves the robustness of harvesting robots in field operations, cross-scenario generalization, and long-term operational stability.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution.

[0006] A self-evolving visual model system for a harvesting robot with cloud-edge collaboration includes a harvesting robot body, an image acquisition unit, an onboard computing unit, a wireless communication unit, and a work execution unit for performing harvesting operations, all mounted on the harvesting robot body; it also includes a cloud server unit located in a remote computer room, which is connected to the onboard computing unit via a bidirectional wireless link through the wireless communication unit.

[0007] The image acquisition unit is connected to the airborne computing unit and is used to acquire color images and depth images of fruits and vegetables before harvesting, calculate a three-dimensional point cloud before harvesting based on the depth image before harvesting, and send the color image, depth image, and three-dimensional point cloud before harvesting to the airborne computing unit.

[0008] The airborne computing unit is equipped with a fruit and vegetable recognition, segmentation and localization module, a difficult sample active sampling module, a visual re-inspection feedback module and a local cache and update management module.

[0009] The cloud server unit is equipped with a standardized data management module and a cross-scenario anti-forgetting incremental training module.

[0010] In the above technical solution, the pre-harvest 3D point cloud is calculated based on the pre-harvest depth image. Specifically, the image acquisition unit uses a depth camera, and for any valid pixel in the pre-harvest depth image... Corresponding depth value The effective pixels are back-projected onto the camera's three-dimensional coordinate system to obtain three-dimensional spatial points. ,in , , ,in, The x-axis coordinates of the pixel in the image coordinate system. This represents the ordinate of the pixel in the image coordinate system. In the camera image coordinate system Axial focal length, In the camera image coordinate system Axial focal length, In the camera image coordinate system Axis distortion parameters, In the camera image coordinate system Axial distortion parameters; a 3D point cloud before harvesting is constructed from the 3D spatial points corresponding to all valid pixels. .

[0011] In the above technical solution, the fruit and vegetable recognition, segmentation, and localization module performs real-time inference on the color image before harvesting and outputs a structured inference field with a unique identifier, including category, model confidence, polygon segmentation, segmentation mask, and anchor box, wherein: category Integer encoding is used to achieve multi-variety classification management; model confidence level. The floating-point values ​​output by the classification branch of the base model network characterize the reliability of the predicted fruit and vegetable target categories; polygon segmentation A pixel-level coordinate array stores the outlines of fruits and vegetables, representing the precise contour boundaries of the target fruits and vegetables; segmentation mask. It is a binary mask matrix aligned pixel-level with the input image, marking the foreground region of the fruit and vegetable targets; anchor boxes The top-left pixel coordinates, width, and height of the smallest bounding rectangle for storing fruits and vegetables are given. The coordinates of the top-left pixel of the smallest bounding rectangle. The width and height are the minimum bounding rectangle; the fruit and vegetable recognition, segmentation, and positioning module uses the segmentation mask field. Extracting target point clouds of fruits and vegetables for corresponding regions from the pre-harvest 3D point cloud. By combining the pre-calibrated hand-eye transformation matrix, the centroid coordinates of the fruit and vegetable target point cloud are transformed from the camera coordinate system to the robot arm base coordinate system. The three-dimensional pose field of the fruit and vegetable is calculated and sent to the robot arm controller for picking path planning and operation execution.

[0012] In the above technical solution, the active sampling module for difficult examples is used to filter difficult examples with model optimization value from the structured inference field, wherein the difficult examples are model confidence scores. Within the preset middle confidence interval, and the segmentation mask Fruit and vegetable target samples that meet the effective mask area and effective depth point conditions in the corresponding region; the preset intermediate confidence interval is composed of a low confidence threshold. and high confidence threshold Determined, that is: The mask area effective depth point ratio Number of valid points in the target point cloud , For target point cloud The number of points in three-dimensional space; when and At that time, fruits and vegetables were identified as difficult samples, among which, The mask area threshold, The effective depth point ratio threshold.

[0013] In the above technical solution, the visual re-inspection feedback module determines the authenticity of difficult sample based on the three-dimensional geometric features of the target point cloud, specifically as follows:

[0014] Statistical outlier removal is performed on the target point cloud corresponding to the difficult example samples, and the eigenvalues ​​of the neighborhood covariance matrix of the target point cloud are calculated point by point. And obtain the local curvature at a single point. Furthermore, the overall average curvature of the point cloud of the difficult example sample was obtained. , Indicates the number of points in the target point cloud;

[0015] The global covariance matrix of the overall target points is used to obtain the global eigenvalues. And calculate the linearity of the point cloud of the difficult example sample. Flatness of point clouds compared to difficult sample samples ;

[0016] When the target point cloud corresponding to the difficult example sample simultaneously satisfies , , If the sample is a genuine fruit or vegetable target, it is identified as such and marked as a valid positive difficult sample. For curvature threshold, The linearity threshold, This is the flatness threshold; samples that do not meet the criteria are marked as misidentified samples or samples requiring manual review.

[0017] The visual re-inspection feedback module is also equipped with a manual review web interface. Maintenance personnel can view the color images, point cloud visualization results, geometric feature parameters and automatic judgment results of difficult samples through a remote web terminal, and can correct the judgment results.

[0018] In the above technical solution, the local caching and update management module is used to complete the local storage of difficult sample and re-examination data, as well as the verification and iterative update of the cloud-optimized model.

[0019] In the above technical solution, the standardized data management module includes a perception data repository, a re-inspection feedback data repository, a labeled data repository, and a model and configuration repository. The perception data repository stores the pre-harvest color images, pre-harvest depth images, and pre-harvest 3D point cloud data of all samples. The re-inspection feedback data repository stores the category, confidence level, mask, segmentation field, anchor box, point cloud features, 3D pose, re-inspection results, manual review status, timestamp, and globally unique sample UUID. The labeled data repository stores the labels of valid positive samples determined by re-inspection, records of misidentified samples, and manually corrected labels, as well as the labeled data of the historical benchmark training set. The model and configuration repository is used to store the model weight parameter files, network structure configuration files, and dataset index information associated with each incremental training session.

[0020] In the above technical solution, the cross-scenario anti-forgetting incremental training module performs incremental training based on both new scenario samples and historical benchmark samples, specifically as follows:

[0021] Extract effective hard positive samples from the new scenario sample set to form the new scenario training set. Historical samples of fruits and vegetables are randomly selected proportionally from the historical baseline sample set to form the historical playback sample set. ;

[0022] The basic model of the fruit and vegetable recognition, segmentation, and localization module is divided into a feature extraction layer. Feature fusion layer and task output layer In the initial stage of incremental training, the feature extraction layer is frozen. The network parameters for the feature fusion layer With task output layer The network parameters are updated; after the initial training phase is completed, the feature extraction layer is selectively unfrozen. Some high-level network parameters are used to enable them to participate in subsequent training.

[0023] During training, an anti-forgetting objective function convergence constraint is introduced. This convergence constraint is specifically implemented by constructing a joint loss function that integrates historical features. The overall loss of the joint loss function is... ,in, The sum of classification loss, segmentation loss, and confidence loss corresponding to the training set for the new scene. This is the sum of the classification loss, segmentation loss, and confidence loss corresponding to the historical playback sample set. This is the anti-forgetting weighting coefficient;

[0024] After the model is trained until the loss converges and the accuracy of the validation set meets the target, the model is lightweighted to generate an optimized model that can be deployed on the airborne end, and then distributed to each harvesting robot through the wireless communication unit.

[0025] A self-evolution method for a cloud-edge collaborative visual model of a harvesting robot, implemented based on the aforementioned self-evolution system for a cloud-edge collaborative visual model of a harvesting robot, includes:

[0026] The system collects color images and depth images of the target fruit and vegetable area before harvesting, and generates three-dimensional point cloud data before harvesting based on the depth images. The color images, depth images, and three-dimensional point cloud data before harvesting are then transmitted to the onboard computing unit.

[0027] The fruit and vegetable recognition, segmentation, and localization module performs target recognition and instance segmentation on the pre-harvest color image, outputting target category field, category confidence score, anchor box field, polygon segmentation field, segmentation mask, and target point cloud. And 3D pose fields;

[0028] The difficult example active sampling module is based on the category confidence level. Segmentation mask and target point cloud The target samples are screened, and the difficult samples selected are transmitted to the visual re-inspection feedback module.

[0029] The visual re-inspection feedback module is based on the target point cloud. Extract the average curvature of the target point cloud linearity and flatness The authenticity of difficult samples is determined, and the final re-inspection result is generated by combining the manual review interface. A structured re-inspection feedback frame is formed, which is cached locally and uploaded to the cloud server unit through the local caching and update management module.

[0030] The standardized data management module standardizes the uploaded difficult sample data and structured re-examination feedback frames to construct a new scenario sample set and a historical benchmark sample set.

[0031] The cross-scenario anti-forgetting incremental training module performs incremental training based on the new scenario sample set and the historical benchmark sample set, including constructing a new scenario training set and a historical replay sample set, performing network freezing and hierarchical updates, and model training to generate an optimized model.

[0032] Furthermore, the standardized data management module performs standardized processing on the uploaded difficult case sample data and structured review feedback frames, specifically as follows:

[0033] The standardized data management module establishes a unified structured sample representation.

[0034]

[0035] in, Color images before harvesting. For depth images before harvesting, For 3D pose fields, For the final re-inspection result label, This is a field for manual review status.

[0036] Compared with the prior art, the present invention has the following significant advantages:

[0037] 1. A closed-loop self-supervised learning mechanism based on the physical results of harvesting can be implemented to determine the authenticity of the recognition results without manual annotation, realizing a closed loop of "perception-execution-feedback-correction" and significantly reducing the cost of data annotation;

[0038] 2. Significantly improves the robustness of the model in complex field environments. Through the active sampling mechanism of difficult examples, it focuses on learning targets with low confidence and high boundary uncertainty, enabling the model to continuously adapt to complex scenarios such as changes in light intensity, shading, and variety differences.

[0039] 3. Effectively solve the catastrophic forgetting problem in incremental learning by introducing a cross-scene anti-forgetting incremental training strategy. Through historical sample playback and loss function constraints, the model can maintain its ability to recognize historical scenes while learning new scene features.

[0040] 4. Achieve continuous self-evolution capability of edge-cloud collaboration. Data is filtered and initially judged through cloud servers, and training and model optimization are completed in the cloud, realizing lightweight and efficient iteration, which is suitable for long-term unmanned operation scenarios. Attached Figure Description

[0041] Figure 1 This is a block diagram of the overall architecture of the self-evolving vision model system for the harvesting robot in this invention.

[0042] Figure 2 This is a schematic diagram illustrating the module composition and data interaction of the airborne computing unit of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating the modular composition and data interaction of the cloud server unit of the present invention;

[0044] Figure 4 This is a flowchart illustrating the process of active sampling and visual re-inspection feedback for difficult-to-sample cases in this invention.

[0045] Figure 5 This is a flowchart illustrating the overall steps of the self-evolution method for the edge-cloud collaborative visual model of the harvesting robot according to the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto. This embodiment discloses a self-evolving system and method for a cloud-edge collaborative visual model for harvesting robots, which is applied to automated fruit and vegetable harvesting operations in complex field scenarios such as open-air orchards and greenhouses. It can be adapted to harvesting tasks of multiple types of fruits and vegetables such as apples, citrus, tomatoes, and strawberries (apples are selected as the harvesting object in this embodiment). Through the cloud-edge collaborative architecture, the visual recognition model achieves unmanned annotation, continuous self-iteration, and anti-forgetting evolution, effectively solving the industry pain points of poor field adaptability, rapid accuracy decay, and easy forgetting of incremental learning in traditional harvesting robot visual models.

[0047] Overall architecture as follows Figure 1As shown, this system includes a harvesting robot body, an image acquisition unit, an onboard computing unit, a wireless communication unit, a work execution unit, and a remote cloud server unit. The harvesting robot body adopts an autonomous mobile platform, and the end effector of the robotic arm is equipped with a gripper or suction cup type end effector, which constitutes the work execution unit, used to complete the tasks of grasping apples and separating the fruit stems. Its motion control is executed by the onboard controller issuing path planning instructions.

[0048] The image acquisition unit uses an Intel RealSense D435i depth camera. This unit works in sync with the robotic arm's operation controller and onboard computing unit, matching the acquisition cycle to the duration of a single harvesting operation, and triggering image acquisition before the robotic arm performs the harvesting action.

[0049] Specifically, the depth camera outputs a 640×480 resolution RGB color image before harvesting and a pixel-aligned single-channel depth image before harvesting. The camera intrinsic parameters are preset as follows: in the camera image coordinate system... Axial focal length Camera image coordinate system Axial focal length Camera image coordinate system Axis distortion parameters =320.0, in the camera image coordinate system Axis distortion parameters Prior to each harvest, color images of the target area are simultaneously acquired. Pre-harvest depth images The three-dimensional point cloud before harvesting was calculated based on the pre-harvest depth image and camera intrinsic parameters through back projection. The calculation method is as follows:

[0050] For any valid pixel in the depth image before harvesting Corresponding depth value The three-dimensional spatial points are obtained by back-projecting onto the camera's three-dimensional coordinate system. ,in , , , The x-axis coordinates of the pixel in the image coordinate system. The vertical axis coordinates of the pixels in the image coordinate system; the three-dimensional point cloud before harvesting is composed of the three-dimensional spatial points corresponding to all valid pixels. .

[0051] The RGB color images, depth images, and pre-harvest 3D point cloud data are synchronously transmitted to the onboard computing unit as the raw input for subsequent identification and re-inspection. The onboard computing unit, using an embedded computing platform deployed within the electrical control cabinet of the harvesting robot, is responsible for real-time visual inference, sample screening, and re-inspection judgment. The wireless communication unit uses a 5G industrial module to achieve bidirectional data transmission between the onboard computing unit and the cloud server unit, supporting stable interaction of sample data and model files. The cloud server unit, deployed in a remote data center and equipped with multi-GPU computing nodes, is responsible for sample data management and incremental model training. It connects simultaneously to multiple harvesting robots via the wireless communication unit, enabling multi-terminal sample aggregation and unified model optimization.

[0052] The modular composition and data interaction of the airborne computing unit, such as Figure 2 As shown, the module internally includes a fruit and vegetable recognition, segmentation, and localization module; a difficult sample active sampling module; a visual re-inspection feedback module; and a local caching and update management module. The fruit and vegetable recognition, segmentation, and localization module has a pre-set basic model for apple recognition, segmentation, and localization. This basic model can use the YOLOv8-seg instance segmentation network, or it can be replaced with a MaskR-CNN network depending on the computing power and accuracy requirements. The initial training set of the model covers standard orchard labeled samples of three apple varieties: Forbris, Ascentek, and Golden Delicious, enabling simultaneous recognition and instance segmentation of multiple apple varieties.

[0053] During the operation, the fruit and vegetable recognition, segmentation, and localization module processes the input pre-harvest color image. Perform real-time inference and output structured inference fields with unique identifiers, including category, model confidence, segmentation field, segmentation mask, and anchor boxes. Integer encoding is used, with 1 representing Forbris, 2 representing Arsentech, and 3 representing Golden Marshal, to achieve multi-variety classification management; model confidence level The floating-point value output by the classification branch of the base model network represents the confidence level of the target category prediction; anchor box field Store the top-left pixel coordinates, width, and height of the smallest bounding rectangle of the target. The coordinates of the top left corner of the rectangle are in pixels. Width and height of the rectangle; polygon segmentation field An array of pixel-level coordinates storing the apple's outline, i.e. The target's precise contour boundary is represented; segmentation mask results. Mark the apple foreground region as a binary mask matrix aligned pixel-level with the input image.

[0054] Furthermore, the fruit and vegetable recognition, segmentation, and localization module uses the segmentation mask results... 3D point cloud before harvesting Extract the target point cloud of the corresponding region That is, satisfying A set of three-dimensional spatial points is used to represent the three-dimensional spatial shape of a single apple target.

[0055] Finally, the fruit and vegetable recognition, segmentation, and localization module, combined with a pre-calibrated hand-eye transformation matrix, transforms the centroid coordinates of the target point cloud from the camera coordinate system to the robotic arm's base coordinate system, and calculates the three-dimensional pose field of the apple. The data is sent to the robotic arm controller for picking path planning and operation execution; all the above inference fields are simultaneously output to the difficult sample active sampling module and the visual re-inspection feedback module.

[0056] The processing flow of the difficult sample active sampling module is as follows: Figure 4 As shown, this method is used to filter difficult examples with model optimization value from all recognition results (i.e., structured inference fields). In this embodiment, a low confidence threshold is preset. and high confidence threshold satisfy , specifically , Simultaneously, a preset mask area threshold is established. Pixel, effective depth point ratio threshold .

[0057] The specific filtering logic is as follows: If Samples deemed low-confidence and suspected misidentifications, often consisting of misidentified leaves, branches, or background debris, are directly discarded and not proceed to subsequent optimization processes; if If the sample is identified as a high-confidence familiar sample, a regular apple target that has been stably identified by the basic model, it has no significant optimization value and will not proceed to the subsequent optimization process; Samples deemed to have intermediate confidence levels are mostly targets with high identification uncertainty due to changes in lighting, foliage obstruction, differences in maturity, or phenotypic differences in new scenes, and will proceed to subsequent verification.

[0058] For samples with intermediate confidence levels, their mask area is further calculated. Number of valid points in the target point cloud ( For target point cloud (Number of points in 3D space), and calculate the proportion of effective depth points. .

[0059] When both conditions are met and When this happens, the sample is classified as a difficult sample.

[0060] After the judgment is completed, the difficult sample is bound to the corresponding pre-harvest color image, pre-harvest depth image, all inference fields, mask area, effective depth point ratio, collection timestamp and globally unique sample UUID, temporarily stored in the local cache, and synchronously transmitted to the visual re-inspection feedback module.

[0061] The visual re-inspection feedback module determines the authenticity of difficult examples based on the three-dimensional geometric features (average curvature, linearity, flatness) of the target point cloud. The process is as follows: Figure 4 As shown, firstly, the target point cloud... Perform statistical outlier removal to eliminate noise and marginal outliers, then calculate the target point cloud point cloud point by point. eigenvalues ​​of the neighborhood covariance matrix And obtain the local curvature at a single point. Further statistical analysis yielded the overall average curvature of the point cloud of the difficult-to-observe sample. .in, , Represents the target point cloud The first in A three-dimensional point, Represents the target point cloud The number of point clouds in the data.

[0062] Simultaneously, the visual re-inspection feedback module calculates the global covariance matrix of the overall target points using cloud computing to obtain global eigenvalues. And calculate the linearity of the point cloud of the difficult example sample. Flatness of point clouds compared to difficult sample samples .

[0063] The average curvature characterizes the degree of bending of the target surface, linearity is used to distinguish linear structures such as branches, and flatness is used to distinguish leaf-like structures.

[0064] In this embodiment, a preset curvature threshold is used. Linearity threshold Flatness threshold When the difficult sample simultaneously satisfies , , When a target is determined to have a three-dimensional shape that is approximately spherical or ellipsoidal, characteristic of apples, it is considered a real fruit or vegetable target and is marked as a valid positive sample; otherwise, it is marked as a misidentified sample or a sample to be manually verified.

[0065] Furthermore, the visual re-inspection feedback module is equipped with a non-blocking manual review web interface. Operation and maintenance personnel can view the color images, point cloud visualization results, geometric feature parameters and automatic judgment results of difficult cases through remote web terminals, and can correct the judgment results. When no one is reviewing, the automatic process continues to run without being blocked. The final re-inspection result is a difficult case of a real apple, forming a structured re-inspection feedback frame, which is included in the upload queue.

[0066] The local caching and update management module utilizes the onboard 128GB solid-state storage to persistently store difficult sample data and re-examination data locally, and monitors the bandwidth status of the 5G communication link in real time: when bandwidth is sufficient, all valid samples are uploaded in batches; when bandwidth is limited or network fluctuations occur, high-value difficult sample data with confidence levels closer to the threshold boundary are uploaded first. Simultaneously, the local caching and update management module receives optimized model files from the cloud server, completes MD5 integrity verification, and then, during idle windows such as robot relocation and robotic arm reset, performs iterative updates of the weights of the onboard fruit and vegetable identification, segmentation, and positioning module, without interrupting normal harvesting operations.

[0067] The modular composition and data interaction of a cloud server unit, such as Figure 3 As shown, the system internally deploys a standardized data management module and a cross-scenario anti-forgetting incremental training module. The standardized data management module receives difficult sample data and re-inspection feedback data uploaded by multiple harvesting robots, completes format unification, field mapping, and classification storage, and constructs four types of storage units: a perception data repository, a re-inspection feedback data repository, a labeled data repository, and a model and configuration repository. The perception data repository stores the pre-harvest color image, pre-harvest depth image, and pre-harvest 3D point cloud data of all samples. The re-inspection feedback data repository stores all structured fields such as category, confidence level, mask, segmentation field, anchor box, point cloud features, pose, re-inspection result, manual review status, timestamp, and UUID. The labeled data repository stores the labels of valid positive samples determined by re-inspection, records of misidentified samples, and manually corrected labels, as well as the labeled data of the historical benchmark training set. The model and configuration repository is used to store the model weight parameter files, network structure configuration files, and dataset index information associated with each incremental training session.

[0068] The standardized data management module establishes a unified structured sample representation.

[0069]

[0070] in, For the final re-inspection result label, This is for manually reviewed status fields. Based on standardized samples (i.e., unified structured samples), the module divides the sample set according to the work scenario and product category, constructing a new scenario sample set for the current work and a historical benchmark sample set, providing standardized data input for incremental training.

[0071] The new scenario sample set refers to the high-value data set that the harvesting robot collects in real time in new operating scenarios that are currently unknown or have undergone environmental changes, such as light fluctuations or new variety phenotypes, and is determined to be valid through re-examination. It represents the sample set of new visual feature distributions that lead to the current inference accuracy of the basic model decay and urgently need to be learned by the model.

[0072] The historical benchmark sample set refers to the standard fruit and vegetable dataset stored in the cloud for the initial training of the basic model, such as standard orchard samples covering three varieties: Forbris, Arsentech, and Golden Delicious, as well as archived sample data that have been incrementally trained and fully learned by the model in previous historical operations. It represents the sample set of old visual feature distributions that the model has stably mastered.

[0073] The cross-scenario anti-forgetting incremental training module performs incremental training based on both new scenario samples and historical baseline samples to solve the catastrophic forgetting problem of "learning new things and forgetting old ones". The specific process is as follows:

[0074] First, an incremental training dataset is constructed by extracting effective hard positive samples from the new scene sample set to form the new scene training set. Historical samples of three apple varieties—Forbris, Ascentet, and Golden Delicious—were randomly selected proportionally from the historical baseline sample set to form the historical playback sample set. .

[0075] In this embodiment, the ratio of new to old samples is set to 2:1, and the sample weights are set to... , The overall weighting relationship is as follows: .

[0076] Then, network freezing and layered updates are performed. Taking the YOLOv8-seg network as an example, the model is divided into three parts:

[0077] The backbone network CSPDarknet serves as the feature extraction layer. The neck PAFPN feature pyramid serves as a feature fusion layer. The detection head and segmentation head serve as the task output layer. .

[0078] In the initial stage of incremental training, the feature extraction layer is frozen. Update all parameters, only the feature fusion layer. With task output layer The parameters are adjusted to quickly adapt to high-level semantic features in new scenarios; after the initial 10 rounds of training, the feature extraction layer is selectively unfrozen. The last two residual module parameters participate in subsequent gradient updates, adapting to the detailed features of new scenes while preserving the underlying general visual features.

[0079] During training, a convergence constraint for the anti-forgetting objective function is introduced. This convergence constraint is specifically implemented by constructing a joint loss function that incorporates historical features. The overall loss of the joint loss function is... ,in, The sum of classification loss, segmentation loss, and confidence loss corresponding to the training set for the new scene. The sum of classification loss, segmentation loss, and confidence loss for the historical playback sample set, and the anti-forgetting weight coefficient. We set it to 0.6 to balance the weights of learning new scenarios and memorizing old scenarios.

[0080] After the model is trained until the loss converges and the accuracy of the validation set meets the target, the model is lightweighted through INT8 quantization and structured pruning to generate an optimized model that can be deployed on the airborne end. The optimized model is then distributed to each harvesting robot through the wireless communication unit. At the same time, the parameters of the new generation model, the training configuration and the corresponding dataset are associated and stored in the standardized data management module to complete the model iteration and knowledge accumulation.

[0081] The overall steps of the self-evolution method for the edge-cloud collaborative visual model of the harvesting robot described in this embodiment are as follows: Figure 5 As shown, the specific steps include the following:

[0082] Step S1, Image Acquisition and Point Cloud Generation: After the harvesting robot moves to the target fruit tree work position, the robotic arm moves to the canopy observation position, and the image acquisition unit simultaneously acquires color images of the target area before harvesting. Depth image before harvesting The airborne computing unit generates a 3D point cloud before harvesting based on camera intrinsic parameters. And transmit all data to the onboard functional modules;

[0083] Step S2, Fruit and Vegetable Recognition and Pose Calculation: The fruit and vegetable recognition, segmentation and localization module calls the YOLOv8-seg network to infer the color image before harvesting, and outputs the category, confidence, anchor box, segmentation field and segmentation mask of three varieties: Forbris, Arsentech and Golden Delicious. It extracts the corresponding target point cloud and calculates the three-dimensional pose in the robot arm base coordinate system, and sends it to the operation execution unit to prepare for harvesting.

[0084] Step S3, Active screening of difficult samples: The active sampling module for difficult samples selects effective difficult samples with intermediate confidence from all recognition results based on the confidence interval, mask area and effective depth point ratio, temporarily stores them locally and sends them to the visual re-inspection feedback module.

[0085] Step S4, Visual Re-inspection and Result Upload: The visual re-inspection feedback module extracts the average curvature, linearity, and flatness of the point cloud of difficult sample cases, determines whether the target is a real apple, and generates the final re-inspection result and structured feedback frame; the local cache and update management module uploads the valid samples to the cloud server unit according to the network status.

[0086] Step S5, Cloud-based standardized data management: The cloud-based standardized data management module performs format verification, field mapping, and classification storage on uploaded samples, builds a standardized sample library, and divides the new scenario training set and the historical playback sample set.

[0087] Step S6, Incremental Training and Iterative Model Update: The cross-scene anti-forgetting incremental training module employs a layered freezing and joint loss constraint strategy to perform incremental training. After generating a lightweight optimized model, it is distributed to each harvesting robot. Each harvesting robot completes model updates during its idle periods, completing a self-evolving closed loop of job-sampling-feedback-training-update. As the harvesting robots continue to operate in different orchards and with different varieties, the above closed loop continues to cycle, and the basic model continuously optimizes autonomously, gradually improving the recognition accuracy and robustness under complex lighting, foliage occlusion, cross-variety, and cross-terrain scenarios, achieving self-evolving capabilities during long-term operation.

[0088] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A self-evolving visual model system for a harvesting robot with edge-cloud collaboration, characterized in that, It includes a harvesting robot body, and an image acquisition unit, an onboard computing unit, a wireless communication unit, and a work execution unit for performing harvesting operations, all mounted on the harvesting robot body; it also includes a cloud server unit located in a remote computer room, which is connected to the onboard computing unit via a two-way wireless link through the wireless communication unit; The image acquisition unit is connected to the airborne computing unit and is used to acquire color images and depth images of fruits and vegetables before harvesting, calculate a three-dimensional point cloud before harvesting based on the depth image before harvesting, and send the color image, depth image, and three-dimensional point cloud before harvesting to the airborne computing unit. The airborne computing unit is equipped with a fruit and vegetable recognition, segmentation and localization module, a difficult sample active sampling module, a visual re-inspection feedback module and a local cache and update management module. The cloud server unit is equipped with a standardized data management module and a cross-scenario anti-forgetting incremental training module.

2. The self-evolving visual model system for the harvesting robot with cloud-edge collaboration according to claim 1, characterized in that, The pre-harvest depth image is used to calculate the pre-harvest 3D point cloud. Specifically, the image acquisition unit uses a depth camera, and for any valid pixel in the pre-harvest depth image... Corresponding depth value The effective pixels are back-projected onto the camera's three-dimensional coordinate system to obtain three-dimensional spatial points. ,in , , ,in, The x-axis coordinates of the pixel in the image coordinate system. This represents the ordinate of the pixel in the image coordinate system. In the camera image coordinate system Axial focal length, In the camera image coordinate system Axial focal length, In the camera image coordinate system Axis distortion parameters, In the camera image coordinate system Axial distortion parameters; a 3D point cloud before harvesting is constructed from the 3D spatial points corresponding to all valid pixels. .

3. The self-evolving visual model system for the harvesting robot with cloud-edge collaboration according to claim 2, characterized in that, The fruit and vegetable recognition, segmentation, and localization module performs real-time inference on the color image before harvesting, outputting structured inference fields with unique identifiers, including category, model confidence, polygon segmentation, segmentation mask, and anchor boxes, where: category Integer encoding is used to achieve multi-variety classification management; model confidence level. The floating-point values ​​output by the classification branch of the base model network characterize the reliability of the predicted fruit and vegetable target categories; polygon segmentation A pixel-level coordinate array stores the outlines of fruits and vegetables, representing the precise contour boundaries of the target fruits and vegetables; segmentation mask. It is a binary mask matrix aligned pixel-level with the input image, marking the foreground region of the fruit and vegetable targets; anchor boxes The top-left pixel coordinates, width, and height of the smallest bounding rectangle for storing fruits and vegetables are given. The coordinates of the top-left pixel of the smallest bounding rectangle. The width and height are the minimum bounding rectangle; the fruit and vegetable recognition, segmentation, and positioning module uses the segmentation mask field. Extracting target point clouds of fruits and vegetables for corresponding regions from the pre-harvest 3D point cloud. By combining the pre-calibrated hand-eye transformation matrix, the centroid coordinates of the fruit and vegetable target point cloud are transformed from the camera coordinate system to the robot arm base coordinate system. The three-dimensional pose field of the fruit and vegetable is calculated and sent to the robot arm controller for picking path planning and operation execution.

4. The self-evolving visual model system for the harvesting robot with cloud-edge collaboration according to claim 3, characterized in that, The active sampling module for difficult examples is used to filter difficult examples with model optimization value from the structured inference fields. These difficult examples represent model confidence scores. Within the preset middle confidence interval, and the segmentation mask Fruit and vegetable target samples that meet the effective mask area and effective depth point conditions in the corresponding region; the preset intermediate confidence interval is composed of a low confidence threshold. and high confidence threshold Determined, that is: The mask area effective depth point ratio Number of valid points in the target point cloud , For target point cloud The number of points in three-dimensional space; when and At that time, fruits and vegetables were identified as difficult samples, among which, The mask area threshold, The effective depth point ratio threshold.

5. The self-evolving visual model system for the harvesting robot with cloud-edge collaboration according to claim 4, characterized in that, The visual re-inspection feedback module determines the authenticity of difficult sample data based on the three-dimensional geometric features of the target point cloud, specifically as follows: Statistical outlier removal is performed on the target point cloud corresponding to the difficult example samples, and the eigenvalues ​​of the neighborhood covariance matrix of the target point cloud are calculated point by point. And obtain the local curvature at a single point. Furthermore, the overall average curvature of the point cloud of the difficult example sample was obtained. , Indicates the number of points in the target point cloud; The global covariance matrix of the overall target points is used to obtain the global eigenvalues. And calculate the linearity of the point cloud of the difficult example sample. Flatness of point clouds compared to difficult sample samples ; When the target point cloud corresponding to the difficult example sample simultaneously satisfies , , If the sample is a genuine fruit or vegetable target, it is identified as such and marked as a valid positive difficult sample. For curvature threshold, The linearity threshold, This is the flatness threshold; samples that do not meet the criteria are marked as misidentified samples or samples requiring manual review. The visual re-inspection feedback module is also equipped with a manual review web interface. Maintenance personnel can view the color images, point cloud visualization results, geometric feature parameters and automatic judgment results of difficult samples through a remote web terminal, and can correct the judgment results.

6. The self-evolving visual model system for the harvesting robot with cloud-edge collaboration according to claim 5, characterized in that, The local caching and update management module is used to complete the local storage of difficult case samples and re-examination data, as well as the verification and iterative updates of the cloud-optimized model.

7. The self-evolving visual model system for the harvesting robot with cloud-edge collaboration according to claim 6, characterized in that, The standardized data management module includes a perception data repository, a re-inspection feedback data repository, a labeling data repository, and a model and configuration repository. The perception data repository stores the pre-harvest color images, pre-harvest depth images, and pre-harvest 3D point cloud data of all samples. The re-inspection feedback data repository stores the category, confidence level, mask, segmentation field, anchor box, point cloud features, 3D pose, re-inspection results, manual review status, timestamp, and globally unique sample UUID. The labeled data repository stores valid positive sample labels that have been reviewed and determined, records of misidentified samples and manually corrected labels, as well as historical benchmark training set labeled data; The model and configuration repository is used to store the model weight parameter files, network structure configuration files, and dataset index information associated with each incremental training session.

8. The self-evolving visual model system for the harvesting robot with cloud-edge collaboration according to claim 7, characterized in that, The cross-scenario anti-forgetting incremental training module performs incremental training based on both new scenario samples and historical baseline samples, specifically as follows: Extract effective hard positive samples from the new scenario sample set to form the new scenario training set. Historical samples of fruits and vegetables are randomly selected proportionally from the historical baseline sample set to form the historical playback sample set. ; The basic model of the fruit and vegetable recognition, segmentation, and localization module is divided into a feature extraction layer. Feature fusion layer and task output layer In the initial stage of incremental training, the feature extraction layer is frozen. The network parameters for the feature fusion layer With task output layer The network parameters are updated; after the initial training phase is completed, the feature extraction layer is selectively unfrozen. Some high-level network parameters are used to enable them to participate in subsequent training. During training, an anti-forgetting objective function convergence constraint is introduced. This convergence constraint is specifically implemented by constructing a joint loss function that integrates historical features. The overall loss of the joint loss function is... ,in, The sum of classification loss, segmentation loss, and confidence loss corresponding to the training set for the new scene. This is the sum of the classification loss, segmentation loss, and confidence loss corresponding to the historical playback sample set. This is the anti-forgetting weighting coefficient; After the model is trained until the loss converges and the accuracy of the validation set meets the target, the model is lightweighted to generate an optimized model that can be deployed on the airborne end, and then distributed to each harvesting robot through the wireless communication unit.

9. A self-evolution method for a cloud-edge collaborative visual model of a harvesting robot, implemented based on the cloud-edge collaborative visual model self-evolution system of a harvesting robot as described in any one of claims 1-8, characterized in that: The system collects color images and depth images of the target fruit and vegetable area before harvesting, and generates three-dimensional point cloud data before harvesting based on the depth images. The color images, depth images, and three-dimensional point cloud data before harvesting are then transmitted to the onboard computing unit. The fruit and vegetable recognition, segmentation, and localization module performs target recognition and instance segmentation on the pre-harvest color image, outputting target category field, category confidence score, anchor box field, polygon segmentation field, segmentation mask, and target point cloud. And 3D pose fields; The difficult example active sampling module is based on the category confidence level. Segmentation mask and target point cloud The target samples are screened, and the difficult samples selected are transmitted to the visual re-inspection feedback module. The visual re-inspection feedback module is based on the target point cloud. Extract the average curvature of the target point cloud linearity and flatness The authenticity of difficult samples is determined, and the final re-inspection result is generated by combining the manual review interface. A structured re-inspection feedback frame is formed, which is cached locally and uploaded to the cloud server unit through the local caching and update management module. The standardized data management module standardizes the uploaded difficult sample data and structured re-examination feedback frames to construct a new scenario sample set and a historical benchmark sample set. The cross-scenario anti-forgetting incremental training module performs incremental training based on the new scenario sample set and the historical benchmark sample set, including constructing a new scenario training set and a historical replay sample set, performing network freezing and hierarchical updates, and model training to generate an optimized model.

10. The self-evolution method for the edge-cloud collaborative visual model of the harvesting robot according to claim 9, characterized in that, The standardized data management module performs standardization processing on the uploaded difficult sample data and structured review feedback frames, specifically as follows: The standardized data management module establishes a unified structured sample representation. in, Color images before harvesting. For depth images before harvesting, For 3D pose fields, For the final re-inspection result label, This is a field for manual review status.

Images