Image recognition-based eggplant picking control method and system

Abstract

The application discloses a kind of based on image recognition's eggplant picking control method and system, the present application relates to the technical field of image recognition, according to the morphological characteristics of the eggplant and the surface abnormal characteristics of each part Determine the multiple surface picking positions of eggplant, according to the spatial position of each eggplant, maturity and the distribution diagram of eggplant picking site Determine the pickable area, further identify the sub-image of each part in eggplant, improve the accuracy of pickable area.Therefore, according to the area form of pickable area, the flight space of eggplant picking site and the current position of picking mechanical arm Determine the corresponding picking path;Based on picking path and the spatial position of each eggplant Determine multiple picking nodes, according to the peripheral environment of multiple picking nodes, the multiple surface picking positions of each eggplant and the picking form of picking mechanical arm Determine the picking action of each eggplant, improve the accuracy of the picking action of each eggplant, and further improve the accuracy of picking dynamic diagram.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a tomato picking control method and system based on image recognition. BACKGROUND

[0002] With the development of science and technology, agricultural picking gradually enters the intelligent era, and the tomato picking ground covers tomatoes with different maturity, and each tomato is distributed at different positions of the tomato picking ground and has a spatial difference. In the prior art, the distribution positions of each tomato are collected, and the corresponding pickable area is determined according to the distribution positions of each tomato, without considering the maturity of each tomato and the spatial position of each tomato, which affects the accuracy of the pickable area. Meanwhile, the picking action of each tomato is designed along a preset dimension, without fully considering the multiple surface picking positions of each tomato and the picking form of the picking mechanical arm. SUMMARY

[0003] The present application aims to overcome the shortcomings of the prior art, and provides a tomato picking control method and system based on image recognition.

[0004] The present application provides a tomato picking control method based on image recognition, which comprises the following steps: marking the spatial position of each tomato and collecting multiple current images of each tomato when a UAV inspects a tomato picking ground, determining the form feature of the tomato according to the recognition of the multiple current images of each tomato; determining the sub-image of each part of the tomato based on the recognition of the multiple current images, determining the surface abnormal feature of each part according to the recognition of the sub-image of each part of the tomato; determining multiple surface picking positions of the tomato according to the form feature of the tomato and the surface abnormal feature of each part, determining a pickable area according to the spatial position, maturity of each tomato and the distribution map of the tomato picking ground; the UAV is equipped with a picking mechanical arm, the flight space of the tomato picking ground is collected, and the corresponding picking path is determined according to the area form of the pickable area, the flight space of the tomato picking ground and the current position of the picking mechanical arm; determining multiple picking nodes based on the picking path and the spatial position of each tomato, determining the picking action of each tomato according to the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each tomato and the picking form of the picking mechanical arm, and constructing a picking dynamic map corresponding to the picking path.

[0005] The present application provides a tomato picking control system based on image recognition, which is applied to the tomato picking control method based on image recognition described above, and comprises the following components:

[0006] An image recognition module is configured to mark the spatial positions of the eggplants when the unmanned aerial vehicle inspects the eggplant picking area, and to collect a plurality of current images of the eggplants, and to determine the shape features of the eggplants according to the recognition of the plurality of current images of the eggplants.

[0007] A surface abnormal feature module is configured to determine the sub-images of the parts of the eggplants based on the recognition of the plurality of current images, and to determine the surface abnormal features of the parts of the eggplants according to the recognition of the sub-images of the parts of the eggplants.

[0008] A pickable area module is configured to determine the plurality of surface picking positions of the eggplants according to the shape features of the eggplants and the surface abnormal features of the parts of the eggplants, and to determine the pickable area according to the spatial positions of the eggplants, the ripeness of the eggplants and the distribution map of the eggplant picking area.

[0009] A picking path module is configured to determine the corresponding picking paths according to the area shape of the pickable area, the flight space of the eggplant picking area and the current position of the picking mechanical arm, when the unmanned aerial vehicle is equipped with the picking mechanical arm and collects the flight space of the eggplant picking area.

[0010] A picking action module is configured to determine the plurality of picking nodes based on the picking paths and the spatial positions of the eggplants, to determine the picking actions of the eggplants according to the surrounding environment of the plurality of picking nodes, the plurality of surface picking positions of the eggplants and the picking shape of the picking mechanical arm, and to construct the picking dynamic map corresponding to the picking paths.

[0011] Compared with the prior art, the present application has the following advantages:

[0012] In the embodiment of the present application, the surface abnormal features of the parts of the eggplants are determined according to the recognition of the sub-images of the parts of the eggplants, the plurality of surface picking positions of the eggplants are determined according to the shape features of the eggplants and the surface abnormal features of the parts of the eggplants, the pickable area is determined according to the spatial positions of the eggplants, the ripeness of the eggplants and the distribution map of the eggplant picking area, the sub-images of the parts of the eggplants are introduced, the sub-images of the parts of the eggplants are further recognized, the overall consideration of the spatial positions of the eggplants, the ripeness of the eggplants and the distribution map of the eggplant picking area is compatible, and the precision of the pickable area is improved.

[0013] Therefore, the unmanned aerial vehicle is provided with a picking mechanical arm, a corresponding picking path is determined according to the region form of the pickable region, the flight space of the eggplant picking ground and the current position of the picking mechanical arm, a plurality of picking nodes are determined based on the picking path and the spatial positions of the eggplants, the picking action of each eggplant is determined according to the surrounding environment of the plurality of picking nodes, the plurality of surface picking positions of each eggplant and the picking form of the picking mechanical arm, so as to construct the picking dynamic graph corresponding to the picking path, the plurality of picking nodes are introduced, the overall consideration of the surrounding environment of the plurality of picking nodes, the plurality of surface picking positions of each eggplant and the picking form of the picking mechanical arm is realized, the precision of the picking action of each eggplant is improved, and the picking action of each eggplant and the picking path are fully utilized, and the precision of the picking dynamic graph is improved. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of the eggplant picking control method based on image recognition in the embodiment of the application;

[0015] Figure 2 is a flowchart of step S11 in the eggplant picking control method based on image recognition in the embodiment of the application;

[0016] Figure 3 is a flowchart of step S12 in the eggplant picking control method based on image recognition in the embodiment of the application;

[0017] Figure 4 is a flowchart of step S13 in the eggplant picking control method based on image recognition in the embodiment of the application;

[0018] Figure 5 is a flowchart of step S14 in the eggplant picking control method based on image recognition in the embodiment of the application;

[0019] Figure 6 is a flowchart of step S15 in the eggplant picking control method based on image recognition in the embodiment of the application;

[0020] Figure 7 is a structural composition diagram of the eggplant picking control system based on image recognition in the embodiment of the application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application.

[0022] Please refer to Figures 1 to 7 An eggplant picking control method based on image recognition is applied to an image recognition scene; the eggplant picking control method based on image recognition comprises the following steps:

[0023] Step S11: The unmanned aerial vehicle marks the spatial position of each eggplant when patrolling the eggplant picking site and collects a plurality of current images of each eggplant, and determines the shape feature of the eggplant according to the recognition of the plurality of current images of each eggplant;

[0024] Step S12: Determine the sub-image of each part of the eggplant based on the recognition of the plurality of current images, and determine the surface abnormal feature of each part according to the recognition of the sub-image of each part of the eggplant;

[0025] Step S13: Determine a plurality of surface picking positions of the eggplant according to the shape feature of the eggplant and the surface abnormal feature of each part, and determine a pickable area according to the spatial position of each eggplant, the maturity and the distribution map of the eggplant picking site;

[0026] Step S14: The unmanned aerial vehicle is equipped with a picking mechanical arm, and the flight space of the eggplant picking site is collected, and the corresponding picking path is determined according to the area shape of the pickable area, the flight space of the eggplant picking site and the current position of the picking mechanical arm;

[0027] Step S15: Determine a plurality of picking nodes based on the picking path and the spatial position of each eggplant, determine the picking action of each eggplant according to the surrounding environment of the plurality of picking nodes, the plurality of surface picking positions of each eggplant and the picking shape of the picking mechanical arm, and construct a picking dynamic map corresponding to the picking path.

[0028] Reference Figure 2 In step S11, the specific steps are:

[0029] S111: Collect the distribution map of the eggplant picking site, determine the eggplant distribution area and the flight area according to the recognition of the distribution map of the eggplant picking site, determine the patrol route of the unmanned aerial vehicle to the eggplant picking site based on the eggplant distribution area, the flight area and the flight shape of the unmanned aerial vehicle, and the unmanned aerial vehicle flies along the patrol route. The unmanned aerial vehicle performs dynamic shooting during flight and marks the spatial position of each eggplant;

[0030] S112: Collect the image set of each eggplant based on the dynamic shooting of the unmanned aerial vehicle, determine a plurality of current images of each eggplant according to the screening of the image set of each eggplant, construct a three-dimensional model of each eggplant according to the plurality of current images of each eggplant and the growth contour of the eggplant, and determine the shape feature of the eggplant based on the recognition of the three-dimensional model of each eggplant.

[0031] In the embodiments of the present application, the system collects the distribution map of the eggplant picking site through satellite remote sensing, unmanned aerial vehicle aerial photography or ground surveying equipment. The distribution map of the eggplant picking site not only contains two-dimensional plane information, but also contains elevation data. Based on the distribution map of the eggplant picking site, the system automatically identifies the eggplant distribution area and the flight area through image segmentation and region recognition. The eggplant distribution area refers to the range of the land where eggplants are actually planted, and the system identifies the distribution of each row and each eggplant. The flight area is the spatial range where the unmanned aerial vehicle can safely fly, and needs to avoid dangerous areas such as obstacles and no-fly zones.

[0032] The system comprehensively considers the geometric characteristics of the eggplant distribution area, the spatial constraints of the flight area and the flight performance parameters of the unmanned aerial vehicle (such as maximum flight speed, minimum turning radius, hovering accuracy, etc.), and designs the optimal inspection route by using an optimization algorithm. The route design needs to meet the full coverage principle to ensure that each eggplant can be photographed. At the same time, flight efficiency needs to be considered to minimize repeated flight and invalid flight. At this time, the unmanned aerial vehicle flies autonomously according to the inspection route. During the flight, the flight control system of the unmanned aerial vehicle adjusts the flight attitude and speed in real time to ensure the accuracy of the flight path. At the same time, the system will dynamically adjust according to the real-time environmental changes (such as sudden gusts, temporary obstacles, etc.) to ensure flight safety.

[0033] During the flight, the camera carried by the unmanned aerial vehicle continuously photographs according to the preset parameters. The photographing system automatically adjusts the photographing frequency according to the flight speed and height to ensure that the image overlap degree meets the subsequent processing requirements. At the same time, the system calculates the camera position and attitude corresponding to each image in real time through multi-sensor fusion technology (such as visual odometry, IMU, GPS, etc.).

[0034] Further, the camera carried by the unmanned aerial vehicle continuously collects images at a preset photographing frequency and angle during the inspection flight. For each detected eggplant, the system collects images taken from different angles and distances to form an image set of the eggplant. These images include multiple angles such as top view, side view and oblique view to ensure coverage of all surfaces of the eggplant.

[0035] From the collected image set, the system determines multiple high-quality current images of each eggplant through a series of screening algorithms. Based on image clarity evaluation, images that are blurred, shaky or out of focus are removed. According to the light condition screening, images with moderate brightness and good contrast are selected. Considering the photographing angle, the selected images can cover all key parts of the eggplant.

[0036] The screened multi-angle images are combined with the growth contour features of eggplants to construct a three-dimensional model of each eggplant; the system identifies and matches the same feature points in images at different angles through a feature point matching algorithm; the three-dimensional spatial coordinates of these feature points are calculated through a structure from motion (SfM) algorithm using camera calibration parameters and pose information at the time of shooting; based on prior knowledge of the growth contour of eggplants (such as the characteristics that eggplants are usually long and thin at both ends and thick in the middle), the reconstructed point cloud data is optimized and corrected; a three-dimensional mesh model of the eggplant is generated through a surface reconstruction algorithm (such as Poisson reconstruction or Delaunay triangulation); the texture information of the multi-angle images is mapped onto the three-dimensional model to form a three-dimensional model with real texture; during the construction process, the system also considers the natural growth morphology of the eggplant, such as bending and twisting, to ensure that the three-dimensional model accurately reflects the morphological characteristics of the real eggplant.

[0037] Based on the constructed three-dimensional model, the system identifies various morphological features of the eggplant through a three-dimensional feature extraction algorithm; performs size measurement to calculate basic parameters such as the length, maximum diameter, and volume of the eggplant; analyzes shape features such as curvature, taper, and symmetry; identifies surface features including color distribution, glossiness, and texture characteristics; detects morphological feature abnormalities such as deformities, depressions, and bulges; evaluates maturity characteristics by comprehensively judging the maturity state of the eggplant through color, size, and shape indicators; during the identification process, the system compares the extracted features with standard eggplant models to calculate the deviation values of various features, and finally forms a complete morphological feature description of the eggplant, including size parameters, shape features, surface state, and maturity evaluation, and other key information.

[0038] Reference Figure 3 In step S12, the specific steps are as follows:

[0039] S121: For each eggplant, determine the sub-image of each part of the eggplant based on the plurality of current images, the position of each part of the eggplant, and the corresponding part morphology, determine a plurality of surface regions according to the identification of the sub-image of each part, and determine each abnormal part according to the detection of the plurality of surface regions;

[0040] S122: Collect the relative positions of each abnormal part, determine the concentration distribution map of each abnormal part according to the relative positions of each abnormal part, the corresponding morphology, and the region morphology of the surface region, and determine the surface abnormal feature of each part based on the concentration distribution map of each abnormal part, the corresponding abnormal type, and the corresponding part.

[0041] In an embodiment of the present application, the system divides the eggplant into three main parts based on the three-dimensional model of the eggplant and botanical knowledge: the top (calyx region), the middle (main body region), and the bottom (fruit stalk region); each part has its specific morphological features and position parameters.

[0042] For the top region, the system identifies the morphological features (star-shaped structure, green or brown color) and location parameters (the topmost end of the eggplant); for the middle region, the system identifies the morphological features (long oval shape, purple surface) and location parameters (the main length part of the eggplant); for the bottom region, the system identifies the morphological features (woodiness, cylindrical shape) and location parameters (the end point of the eggplant connected to the plant); after determining the location and morphology of each part, the system selects the image that best displays each part from multiple current images and extracts the corresponding sub-image using image segmentation algorithms.

[0043] The system uses multi-level image segmentation technology, based on color and texture features for preliminary segmentation, and then combines edge detection and region growing algorithm for fine segmentation; for the top sub-image, the system divides it into calyx region, calyx surrounding region and top pericarp region; for the middle sub-image, the system divides it into three pericarp regions according to the longitudinal characteristics of the eggplant: upper, middle and lower; for the bottom sub-image, the system divides it into peduncle region, peduncle connection region and bottom pericarp region; the division of each surface region takes into account the physiological characteristics and abnormal distribution rules of eggplant growth; during the segmentation process, the system applies a deep learning model to identify the region boundaries, improving the segmentation accuracy; at the same time, the system also records the relative position relationship, area size and morphological features of each surface region, establishing a topological relationship diagram between surface regions, providing spatial reference for subsequent anomaly detection.

[0044] The system uses a multi-modal detection method, combining color analysis, texture analysis, shape analysis and deep learning classifiers to comprehensively detect each surface region; in terms of color analysis, the system calculates the color histogram of each surface region and compares it with the color model of normal eggplants to identify color abnormal regions; in terms of texture analysis, the system uses methods such as gray level co-occurrence matrix and wavelet transform to extract surface texture features and identify texture abnormal regions; in terms of shape analysis, the system identifies shape abnormalities such as concave, convex and deformation through edge detection and contour analysis; for each detected abnormal region, the system further analyzes its severity, impact range and causes.

[0045] Further, the system establishes a three-dimensional coordinate system with the geometric center of the eggplant as the origin, the X-axis along the length direction of the eggplant, the Y-axis along the width direction of the eggplant, and the Z-axis along the height direction of the eggplant; for each detected abnormal part, the system records its accurate coordinate value in the three-dimensional coordinate system, and calculates the relative distance and angle relationship between the abnormal parts; during the collection process, the system considers the curvature characteristics of the eggplant surface, projects the three-dimensional coordinates onto a two-dimensional development plane to form a plane relative position map; for each abnormal part, the system also records its position offset relative to the key feature points of the eggplant (such as the center of the calyx, the connection point of the fruit stalk, and the maximum diameter point), and these offsets are expressed in percentage form, eliminating the size differences between eggplants of different sizes.

[0046] The density-based clustering algorithm (such as DBSCAN) is used to perform spatial clustering on the abnormal parts to identify the areas of abnormal concentration; at this time, the system considers the spatial distance between the abnormal parts, the morphological similarity, and the abnormal type correlation to ensure the rationality of the clustering results; for each cluster, the system calculates the center point, coverage range, abnormal density, and shape characteristics to form the description parameters of the concentrated areas of the abnormal parts; based on these parameters, the system constructs a concentrated distribution map of the abnormal parts, which is displayed in the form of a heat map, with color depth representing abnormal density, and different colors marking different types of abnormalities; the concentrated distribution map also contains the eggplant surface grid information, which divides the eggplant surface into several equal-area grid cells, each cell displaying abnormal quantity and type statistical information; in addition, the system also calculates the directional characteristics of abnormal distribution to identify whether the abnormalities are concentrated in a specific direction (such as longitudinal, transverse, or circumferential), which helps to analyze the causes of the abnormalities.

[0047] The system performs multi-dimensional analysis on the concentrated distribution map, including spatial distribution patterns, abnormal type combinations, and severity evaluation; for each part, the system identifies its main abnormal type, secondary abnormal type, and their distribution characteristics; during the analysis process, the system considers the correlation between abnormalities, such as whether certain abnormalities often occur simultaneously or whether there is a causal relationship; the system also evaluates the impact of abnormalities on eggplant quality, considering factors such as the location, size, type, and number of abnormalities.

[0048] Reference Figure 4 In step S13, the specific steps are as follows:

[0049] S131: Collecting multiple current images of the eggplant, determining the surrounding environment space of the eggplant based on the identification of the multiple current images of the eggplant, determining multiple sub-space features based on the identification of the surrounding environment space of the eggplant, determining the surrounding space features of the eggplant based on the multiple sub-space features and the space mapping relationship, and determining the first picking distribution map of the eggplant based on the surrounding space features of the eggplant and the morphological features of the eggplant;

[0050] S132: determining a second picking distribution map of the eggplants according to the peripheral space features and the surface abnormality features of each part of the eggplants, and determining a plurality of surface picking positions of the eggplants based on the first picking distribution map and the second picking distribution map of the eggplants;

[0051] S133: determining corresponding surface color coefficients according to the detection of the plurality of surface picking positions of the eggplants, determining the ripeness of the eggplants according to the surface color coefficients, the morphological features of the eggplants and the growth time list of the eggplants, determining a plurality of eggplant ripening areas based on the ripeness of each eggplant, the corresponding spatial position and the distribution map of the eggplant picking site, and determining the pickable area according to the area position of the plurality of eggplant ripening areas, the corresponding area morphology and the flight path of the unmanned aerial vehicle relative to the eggplant picking site.

[0052] In the embodiments of the present application, a plurality of current images of the eggplants are collected, which not only contain the eggplants themselves, but also contain the environmental information around the eggplants; through image recognition technology, the system extracts the environmental space around the eggplants from the images, including the upper space (lighting conditions, obstructions such as leaves or supports), the side space (adjacent plants, support structures, other fruits) and the lower space (ground conditions, supports, other plants); the system divides the environmental space into three main areas: a picking interference area (an area that hinders picking operations), a safe operation area (an area suitable for picking operations) and an auxiliary support area (an area that can provide support or assist in operations); each area has its specific spatial parameters and attributes.

[0053] The system quantitatively analyzes the identified environmental space and extracts a plurality of subspace features, including: upper openness (0-100%, indicating the openness of the space above the eggplant), side obstacle distance (centimeters, indicating the distance from the side of the eggplant to the nearest obstacle), lower support stability (0-10 score, indicating the stability of the support below the eggplant), light intensity (lux, indicating the lighting conditions on the surface of the eggplant), wind influence coefficient (0-1, indicating the degree of influence of environmental wind on picking operations) and operation accessibility (0-100%, indicating the ease of reaching the surface of the eggplant by the robotic arm); each subspace feature has its specific measurement method and quantification standard, for example, the upper openness is determined by calculating the proportion of the unobstructed area above the eggplant, and the side obstacle distance is determined by measuring the three-dimensional distance from the surface of the eggplant to the nearest obstacle through depth images.

[0054] The system establishes a spatial mapping relationship, mapping various subspace features into a unified multi-dimensional feature space. This mapping relationship takes into account the mutual influence and weight relationship between different subspace features, for example, there is a positive correlation between the openness above and the light intensity, and there is a positive correlation between the side obstacle distance and the operation accessibility. Then, the system integrates multiple subspace features into a surrounding space feature through a weighted fusion algorithm, forming a feature vector containing multiple dimensions. This feature vector usually includes: environmental complexity (0-10, indicating the complexity of the picking environment), operation safety (0-100%, indicating the safety of picking operation), picking efficiency expectation (0-100%, indicating the expected efficiency of picking in this environment), and environmental adaptability (0-100%, indicating the adaptability of the robot arm in this environment). Each dimension is calculated by weighting the relevant subspace features, for example, the environmental complexity is the weighted average of the openness above, the side obstacle distance, and the wind influence coefficient.

[0055] The system obtains the morphological features of the eggplant, including size (length, diameter), shape (long circle, ellipse, pear shape, etc.), posture (inclination angle, bending degree), and surface characteristics (smoothness, hardness). Then, the system matches and analyzes these morphological features with the surrounding space features to evaluate the picking feasibility of each location on the eggplant surface. The system uses a gridding method to divide the eggplant surface into multiple small areas (usually 1 cm x 1 cm grids), and calculates the picking feasibility score (0-100 points) for each area. The scoring factors include: the accessibility of the area (based on the surrounding space features), the mechanical strength of the area (based on the morphological features), the operation safety of the area (based on the surrounding space features), and the picking efficiency of the area (based on the morphological features and the surrounding space features). The system visualizes these scores in the form of a heat map to generate a first picking distribution map, which intuitively displays the picking feasibility of each location on the eggplant surface.

[0056] Specifically, suppose the system is analyzing an eggplant A located in the middle of the eggplant plant; the system collects the current image of the eggplant from four angles: front, left side, 45° angle above, and 30° angle below; through image recognition, the system determines that the surrounding environment space of the eggplant includes: there are 3 leaf parts above (blocking area about 30%), there is another eggplant on the left side (distance about 5 cm), there is a support on the right side (distance about 3 cm), and there is a branch support below (support stability score 7 points); the system divides these environmental spaces into picking interference areas (upper leaf blocking area and right support area), safe operation areas (eggplant front and left side areas), and auxiliary support areas (lower branch support area).

[0057] The system determines multiple sub-space features: the openness above is 70% (30% is blocked by leaves), the left obstacle distance is 5 cm (distance to another eggplant), the right obstacle distance is 3 cm (distance to the support), the support stability below is 7 points (out of 10), the light intensity is 8000 lux (moderate light), the wind influence coefficient is 0.2 (slight wind influence), and the operation accessibility is 85% (most areas are easily accessible).

[0058] The system determines the surrounding space features based on the spatial mapping relationship: the environment complexity is 4.2 points (moderate complexity, weighted based on the openness above 70%, the minimum obstacle distance on the side 3 cm, and the wind influence coefficient 0.2), the operation safety is 88% (high safety, comprehensive evaluation based on the obstacle distance on the side, the support stability below, and the wind influence coefficient), the picking efficiency expectation is 82% (high efficiency, comprehensive evaluation based on the operation accessibility, the environment complexity, and the light intensity), and the environmental adaptability is 90% (high adaptability, comprehensive evaluation based on the operation accessibility, the operation safety, and the environment complexity).

[0059] The system generates a first picking distribution map combining the surrounding space features and the eggplant morphological features (length 12 cm, maximum diameter 6 cm, shape long circular, inclination angle 15°, surface smoothness score 8 points); the system divides the eggplant surface into 60 grid areas (10 length direction x 6 circumference direction), and calculates the picking feasibility score for each area; for example, the score of the upper area on the front of the eggplant is 92 points (high accessibility, high safety, high efficiency), the score of the area near the support on the right side of the eggplant is 45 points (low accessibility, low safety), and the score of the area near the branch below the eggplant is 88 points (high safety, moderate efficiency); the final first picking distribution map is displayed in the form of a heat map, where the red area (90-100 points) represents the best picking position, the yellow area (70-89 points) represents the good picking position, the green area (50-69 points) represents the general picking position, and the blue area (less than 50 points) represents the non-recommended picking position; the distribution map shows that the front and upper left areas of the eggplant are the best picking positions, while the area near the support on the right side is not recommended for picking.

[0060] Further, the system integrates the previously acquired surrounding space features (such as operation accessibility, environmental complexity, operation safety, etc.) and the surface abnormality features of each part (such as abnormal position, abnormal type, abnormal severity, etc.); the system establishes an abnormality influence evaluation model, and sets different influence weights for different types of abnormalities; for example, the weight of mechanical damage type of abnormality is 0.8 (greater impact on picking quality), the weight of color abnormality type of abnormality is 0.5 (moderate impact on picking quality), and the weight of shape abnormality type of abnormality is 0.6 (moderate to greater impact on picking quality); the system also considers the influence coefficient of abnormal position on picking operation, such as the influence coefficient of eggplant top abnormality is 0.7 (influencing picking point selection), the influence coefficient of middle abnormality is 0.9 (seriously affecting picking quality), and the influence coefficient of bottom abnormality is 0.4 (less affecting picking); the system comprehensively calculates these factors to generate a second picking distribution map, which specially marks the abnormal area that should be avoided and the recommended safe picking area.

[0061] The system fuses the first picking distribution map (based on environmental factors) and the second picking distribution map (based on abnormality factors) by weighting, usually with environmental factor weight of 0.6 and abnormality factor weight of 0.4; then, the system sets picking position selection criteria, including: comprehensive score not less than 75 points, distance from the nearest abnormal point not less than 1 cm, no serious abnormality within 2 cm range around the picking point, accessibility score of the area where the picking point is located not less than 70 points, etc.; the system also considers the continuity and efficiency of picking operation, usually selecting 3-5 alternative picking positions and ranking them by priority; finally, the system determines multiple optimal surface picking positions on the eggplant surface according to these criteria, and provides detailed operation parameters for each picking position, such as contact angle, pressure range, movement trajectory, etc.

[0062] Specifically, suppose the system is analyzing an eggplant A located in the middle of the eggplant plant; the surrounding space features of this eggplant include: operation accessibility 85% (high), environmental complexity 35% (low), operation safety 90% (high), light intensity 85% (good), wind influence coefficient 0.2 (slight influence); the surface abnormality features of this eggplant include: a diameter of 0.4 cm of sunburn spot (color abnormality, slight severity) in the top area; a diameter of 0.6 cm of indentation (shape abnormality, moderate severity) in the middle right side; a length of 0.5 cm of slight crack (mechanical damage, slight severity) in the bottom area.

[0063] The system determines a second picking distribution map according to the surrounding space features and surface abnormal features; the system evaluates each abnormality: the abnormal weight of the top sunburn spot is 0.5 (color abnormality), the location influence coefficient is 0.7 (top area), and the comprehensive influence value is 0.5x0.7=0.35; the abnormal weight of the middle depression is 0.6 (morphological abnormality), the location influence coefficient is 0.9 (middle area), and the comprehensive influence value is 0.6x0.9=0.54; the abnormal weight of the bottom crack is 0.8 (mechanical damage), the location influence coefficient is 0.4 (bottom area), and the comprehensive influence value is 0.8x0.4=0.32.

[0064] The system divides the eggplant surface into 60 grid areas and calculates the abnormal influence score for each area; for example, the abnormal influence score of the area around the top sunburn spot is 65 (moderate influence), the abnormal influence score of the area around the middle depression is 40 (greater influence), and the abnormal influence score of the area around the bottom crack is 70 (lesser influence); the system generates a second picking distribution map by combining the surrounding space features (such as operation accessibility 85, environmental complexity 35, etc.) and the abnormal influence score, which shows that the area around the top sunburn spot of the eggplant is a medium-risk area, the area around the middle depression is a high-risk area, the area around the bottom crack is a low-risk area, and the area on the left side and directly above the eggplant is a low-risk high-safety area.

[0065] The system performs weighted fusion of the first picking distribution map (mainly considering environmental factors) and the second picking distribution map (mainly considering abnormal factors); assuming that the score of the area directly above the front of the eggplant in the first picking distribution map is 92, the score of the left side area is 85, and the score of the right side area is 45; the score of the area directly above the front of the eggplant in the second picking distribution map is 88, the score of the left side area is 90, and the score of the right side area is 38; the system calculates according to the environmental factor weight 0.6 and the abnormal factor weight 0.4: the comprehensive score of the area directly above the front is 92x0.6+88x0.4=90.4; the comprehensive score of the left side area is 85x0.6+90x0.4=87; the comprehensive score of the right side area is 45x0.6+38x0.4=42.2.

[0066] The system determines three optimal surface picking positions on the surface of the eggplant according to the picking position selection criteria (the comprehensive score is not less than 75 points, the distance from the nearest abnormal point is not less than 1 cm, etc.): the first position is located in the upper region of the front surface of the eggplant, the comprehensive score is 90.4 points, the distance from the nearest abnormal point (top sunburn spot) is 2.3 cm, the recommended contact angle is 45 degrees, and the pressure range is 0.5-0.8 N; the second position is located in the upper middle region of the left side of the eggplant, the comprehensive score is 87 points, the distance from the nearest abnormal point (middle depression) is 1.8 cm, the recommended contact angle is 30 degrees, and the pressure range is 0.6-0.9 N; the third position is located in the middle region of the left side of the eggplant, the comprehensive score is 82 points, the distance from the nearest abnormal point (bottom crack) is 2.5 cm, the recommended contact angle is 60 degrees, and the pressure range is 0.4-0.7 N; the system prioritizes these three picking positions, with the first position being the preferred picking position, the second position being the backup picking position, and the third position being the emergency picking position, providing accurate target positions and operation parameters for subsequent picking motion control.

[0067] Therefore, according to the detection of multiple surface picking positions of eggplants, the corresponding surface color coefficients are determined, and the maturity of the eggplants is determined according to the respective surface color coefficients, the morphological characteristics of the eggplants, and the growth time list of the eggplants; based on the maturity of each eggplant, the corresponding spatial position, and the distribution map of the eggplant picking site, multiple eggplant maturity areas are determined, and the pickable area is determined according to the area position, the corresponding area morphology of the multiple eggplant maturity areas, and the flight path of the unmanned aerial vehicle relative to the eggplant picking site, improving the accuracy of the pickable area. At the same time, the sub-image of each part of the eggplant is introduced, and the sub-image of each part of the eggplant is further identified, which takes into account the overall consideration of the spatial position, maturity, and distribution map of the eggplant picking site of each eggplant, further improving the accuracy of the pickable area.

[0068] At this time, the system performs color detection on the previously determined surface picking positions, calculates the color coefficient of each picking position through RGB color space analysis and HSV color space conversion; the color coefficient is a comprehensive index, including hue coefficient (reflecting the basic color tone of the color, the mature hue coefficient of purple eggplant ranges from 0.75 to 0.85), saturation coefficient (reflecting the purity of the color, the saturation coefficient of mature eggplant ranges from 0.6 to 0.8), and brightness coefficient (reflecting the lightness of the color, the brightness coefficient of mature eggplant ranges from 0.4 to 0.6); the system measures the color coefficient of each picking position, and then takes the average value as the overall color coefficient of the eggplant; at the same time, the system extracts the morphological features of the eggplant, including length (mature eggplant is usually 15-25 cm), diameter (mature eggplant is usually 5-8 cm), shape index (ratio of length to diameter, mature eggplant is usually 2.5-4.0), and surface glossiness (mature eggplant usually has medium to high gloss); in addition, the system queries the growth time list of the eggplant to obtain the growth days of the eggplant (eggplant usually takes 25-35 days from flowering to maturity); finally, the system inputs these parameters into the maturity assessment model, which uses a weighted calculation method, with color coefficient weight of 0.5, morphological feature weight of 0.3, and growth time weight of 0.2, to calculate the maturity index of the eggplant (range from 0 to 100, above 70 indicates mature and can be picked).

[0069] The system collects the maturity data of all detected eggplants and their spatial position coordinates (X, Y, Z); then, the system applies a spatial clustering algorithm (such as DBSCAN or K-means) to group these eggplants, dividing eggplants with similar maturity and similar spatial positions into the same maturity area; during clustering, the system sets a maturity similarity threshold (such as a maturity index difference of no more than 10) and a spatial distance threshold (such as a distance between adjacent eggplants of no more than 50 cm); for each maturity area formed by clustering, the system calculates the characteristic parameters of the area, including area center coordinates, area range (length, width, height), number of eggplants in the area, average maturity, maturity standard deviation, etc.; the system also divides the maturity area into different levels according to the average maturity of the eggplants in the area: fully mature area (average maturity ≥ 85), moderate maturity area (70 ≤ average maturity < 85), and immature area (average maturity < 70); finally, the system labels these maturity areas on the distribution map of the eggplant picking site, forming an eggplant maturity area distribution map, providing spatial guidance for subsequent picking planning.

[0070] The system analyzes the regional position of each mature area, calculates the shortest distance, relative height difference and azimuth angle between each mature area and the flight path of the unmanned aerial vehicle, then evaluates the regional morphology of each mature area, including the shape complexity of the area (by calculating the irregularity of the area boundary), the compactness of the area (by calculating the ratio of the area to the perimeter) and the connectivity of the area (by calculating the connection strength between the eggplants in the area), and considers the flight path characteristics of the unmanned aerial vehicle relative to the eggplant picking site, including flight height, flight speed, turning radius and hovering stability, etc. The system establishes a pickability evaluation model, considering position factors (weight 0.4), morphology factors (weight 0.3) and flight path factors (weight 0.3), and calculates the picking feasibility index of each mature area. The system sets a picking feasibility threshold (such as an index not less than 60), and marks the mature areas that meet the conditions as pickable areas. For each pickable area, the system also determines the optimal picking order and picking strategy, including the unmanned aerial vehicle entry angle, hovering position and picking mechanical arm operation range, etc. The system generates a pickable area distribution map to provide accurate spatial navigation and operation guidance for the actual picking operation of the unmanned aerial vehicle.

[0071] Specifically, assuming that the system is analyzing an eggplant A located in the middle of the eggplant plant; the system detects the color coefficient of three surface picking positions of eggplant A: the first position (front and top) has a hue coefficient of 0.82, a saturation coefficient of 0.75 and a brightness coefficient of 0.48; the second position (left middle upper part) has a hue coefficient of 0.81, a saturation coefficient of 0.73 and a brightness coefficient of 0.47; the third position (left middle part) has a hue coefficient of 0.80, a saturation coefficient of 0.72 and a brightness coefficient of 0.46; the overall color coefficient of eggplant A is calculated to be 0.78; at the same time, the system extracts the morphological characteristics of eggplant A: length is 18.5 cm, diameter is 5.2 cm, shape index is 3.56, surface glossiness is 0.75; querying the growth time list shows that the growth days of eggplant A are 28 days; inputting these parameters into the maturity evaluation model, the maturity index of eggplant A is calculated to be 82, indicating that the eggplant has matured and can be picked.

[0072] The system detects 12 eggplants around eggplant A, 8 of which have a maturity index between 75-90 and a spatial position within 30-50 cm from eggplant A; applying a spatial clustering algorithm, the system divides these 9 eggplants (including eggplant A) into a maturity area, labeled as maturity area M1; the system calculates the characteristic parameters of M1: the area center coordinates are (X: 125.3 m, Y: 67.8 m, Z: 0.8 m), the area range is 2.5 m long, 1.8 m wide, and 0.6 m high, the number of eggplants in the area is 9, the average maturity is 78.5, and the maturity standard deviation is 4.2; according to the average maturity, the system divides M1 into a moderate maturity area; similarly, the system identifies 15 maturity areas in the entire planting base, 5 of which are fully mature areas, 8 of which are moderate maturity areas, and 2 of which are immature areas.

[0073] The system analyzes the regional position characteristics of the maturity area M1: the shortest distance from the UAV flight path is 3.2 m, the relative height difference is 0.3 m, and the azimuth angle is 45 degrees; the regional shape characteristics: shape complexity is 0.25 (low, indicating that the shape is relatively regular), compactness is 0.68 (medium), and connectivity is 0.82 (high); UAV flight path characteristics: flight height is 2.5 m, flight speed is 1.5 m / s, turning radius is 1.0 m, and hovering stability is 0.85 (high); input these parameters into the pickability evaluation model, and calculate the picking feasibility index of M1 as 78, which exceeds the set threshold of 60, so M1 is marked as a pickable area; the system further determines the optimal picking strategy for M1: the UAV should enter from a 45-degree angle and hover 1.5 m above the center of the area, and the mechanical arm's operating range is a hemispherical space with a radius of 1.2 m; finally, the system determines 12 pickable areas among the 15 maturity areas, providing precise spatial guidance for the UAV's picking task.

[0074] Reference Figure 5 In step S14, the specific steps are:

[0075] S141: The UAV is equipped with a camera for shooting, and a picking mechanical arm is mounted at the front end of the UAV, which adjusts its posture and position as the UAV flies; the eggplant distribution area and the flight area are collected, the interpenetration space is determined based on the detection of the eggplant distribution area, and the flight space of the eggplant picking site is determined according to the interpenetration space, the flight area, and the flight form of the UAV;

[0076] S142: Collect the regional shape of the pickable area, determine the overall picking space according to the regional shape of the pickable area and the flight space of the eggplant picking site, and determine the first sub-picking path according to the identification of the overall picking space;

[0077] S143: The picking mechanical arm adjusts its position as the UAV flies, collects multiple pose parameters of the picking mechanical arm, determines the spatial position of the picking mechanical arm relative to the overall picking space based on the multiple pose parameters of the picking mechanical arm, the current position of the picking mechanical arm, and the overall picking space, and determines a second sub-picking path based on the spatial position of the picking mechanical arm relative to the overall picking space and the flight path of the UAV relative to the eggplant picking site. The corresponding picking path is determined based on the first sub-picking path and the second sub-picking path.

[0078] In the embodiments of the present application, the UAV is built-in with a camera for shooting, and the UAV front end is equipped with a picking mechanical arm. The picking mechanical arm and the camera are distributed at different positions of the UAV. The picking mechanical arm adjusts its pose and position as the UAV flies. At the same time, the picking mechanical arm mounted on the front end of the UAV is usually a 6-DOF mechanical arm with a working radius of 0.5-1.2 meters, a maximum load of 2 kilograms, and a positioning accuracy of ±2 millimeters. The mechanical arm is equipped with a real-time pose adjustment system including an IMU (Inertial Measurement Unit) and a servo motor, which can automatically adjust its pose according to the flight state of the UAV to maintain stability.

[0079] When the UAV performs the tilt system, it first collects the eggplant distribution area. These data are derived from the eggplant spatial distribution information determined in the previous step (e.g., S111). The eggplant distribution area is represented in the form of three-dimensional point cloud data, with each point representing the position of an eggplant or eggplant plant, containing X, Y, Z coordinate information.

[0080] At the same time, the system collects flight area data, which are derived from the UAV safe flight area determined in step S111, represented in the form of a polygon boundary, containing height limit information; for example, the flight area is a rectangular area with a length of 22 meters and a width of 17 meters, with a height limit of 1-3 meters; based on the detection of the eggplant distribution area, the system determines the interpenetration space, i.e., the gap area between eggplant plants; the detection algorithm of the interpenetration space is based on clustering analysis and spatial segmentation of three-dimensional point cloud data.

[0081] The eggplant plant point cloud data is clustered to identify the occupied space of each plant, and then the gap area between plants is calculated; the calculation of the interpenetration space takes into account the following factors: minimum safety distance (usually not less than 0.8 meters to ensure the safety of the UAV), geometric shape of the gap (such as circular, elliptical or irregular shape), and connectivity of the gap (to ensure continuous flight of the UAV). At the same time, the system determines the flight space of the eggplant picking site based on the interpenetration space, the flight area, and the flight form of the UAV through spatial superposition analysis and path planning algorithm.

[0082] The system superimposes the insertion space and the flight area for analysis to determine the intersection area, i.e., the area that meets both the safe flight condition and the proximity to the eggplants. The system considers the flight form of the unmanned aerial vehicle, including the size of the unmanned aerial vehicle (e.g., 0.8 meters long, 0.6 meters wide, and 0.3 meters high), the maneuverability (e.g., a minimum turning radius of 1.5 meters and a maximum climbing angle of 30 degrees), and the flight speed (usually 0.5-2 meters per second). The planning of the flight space follows the following principles: safety principle (ensuring that the unmanned aerial vehicle maintains a safe distance from obstacles), efficiency principle (minimizing flight distance and time), and operability principle (considering the physical limitations of the unmanned aerial vehicle). The system uses a three-dimensional algorithm for path planning to generate a continuous three-dimensional flight space composed of a series of interconnected three-dimensional areas, each with clear boundaries and height restrictions.

[0083] Further, the region form of the harvestable region is collected, and the overall harvesting space is determined according to the region form of the harvestable region and the flight space of the eggplant harvesting site. The first sub-harvesting path is determined according to the identification of the overall harvesting space, and the first sub-harvesting path is introduced.

[0084] At this time, the region form of the harvestable region is collected, which refers to the region where the eggplant plants that meet the harvesting conditions are located. It usually needs to meet the following conditions: the maturity of the eggplants reaches the harvesting standard, there are no obstructions around, the mechanical arm can be accessed, etc. The region form of the harvestable region is fused with the previously determined flight space to construct the "overall harvesting space". The overall harvesting space is a three-dimensional space where the unmanned aerial vehicle and the mechanical arm can cooperatively perform the harvesting task. It needs to meet the following conditions: spatial continuity: the flight space and the harvestable region are spatially connected and have no obstacles; operational accessibility: the mechanical arm can enter the harvestable region with a reasonable posture and perform harvesting; safety constraints: avoid collisions between the unmanned aerial vehicle or the mechanical arm and the plants, supports, etc. At the same time, the system uses a combination of spatial mapping and Boolean operations to map the harvestable region into the flight space to generate the overall harvesting space.

[0085] The first sub-harvesting path is determined according to the identification of the overall harvesting space, which is the flight path of the unmanned aerial vehicle in the overall harvesting space. This path needs to meet the following requirements: path continuity: the path is composed of a series of waypoints, and the unmanned aerial vehicle can fly smoothly along the path; path optimization: the path length is as short as possible, and the flight time is as short as possible; task adaptability: the path needs to cover all harvestable points and ensure that the unmanned aerial vehicle hovers at the appropriate position for the mechanical arm to operate; the system uses a path planning algorithm (such as RRT) to generate an initial path and optimizes the path trajectory through smoothing processing (such as Bezier curve fitting).

[0086] Specifically, taking the A-A whole picking space as an example, in the A-A whole picking space, the system identifies 12 pickable points distributed at different heights and positions; the system generates a path containing 8 waypoints based on the algorithm, with a total length of 11.3 meters and an estimated flight time of 28 seconds; the starting point of the path is the A-A space entrance (14.5, 7.5, 1.5), and the ending point is the exit (16.2, 9.9, 1.2), passing through 6 hovering points in between, each corresponding to one or more picking positions of eggplants; for example, at waypoint 3 (15.4, 8.3, 1.0), the UAV hovers for 5 seconds, and the mechanical arm picks two eggplants numbered B2-3 and B2-4; the system records this path as "first sub-picking path-A-A" and outputs it as a trajectory instruction sequence executable by the UAV flight control system.

[0087] Therefore, the picking mechanical arm adjusts its position along with the flight of the UAV, collects multiple pose parameters of the picking mechanical arm, determines the spatial position of the picking mechanical arm relative to the whole picking space based on the multiple pose parameters of the picking mechanical arm, the current position of the picking mechanical arm, and the whole picking space; determines the second sub-picking path based on the spatial position of the picking mechanical arm relative to the whole picking space and the flight path of the UAV relative to the eggplant picking site, and determines the corresponding picking path based on the first sub-picking path and the second sub-picking path, thereby improving the accuracy of the picking path.

[0088] At this time, during the flight of the UAV, the mechanical arm needs to adjust its position in real time to adapt to different picking requirements; the system achieves this by the following means: the mechanical arm is equipped with multiple sensors (such as joint encoders, IMUs, gyroscopes), which collect parameters such as joint angles, end effector positions, rotation angles, and extension lengths in real time; typical pose parameters include base rotation angle (a), large arm swing angle (b), small arm pitch angle (g), end effector rotation angle (d), and extension length (L); combined with the current absolute position of the mechanical arm (determined by the UAV GPS positioning) and the relative position (calculated by the joint angle of the mechanical arm), the system calculates the precise coordinates (x, y, z) of the end effector of the mechanical arm in the whole picking space through forward kinematics; mapping the end effector position of the mechanical arm to the whole picking space coordinate system ensures that the operating range of the mechanical arm matches the picking space; the system will real-time calibrate the relative relationship between the mechanical arm workspace and the whole picking space to avoid exceeding the operating range.

[0089] Specifically, assume that the UAV flies to waypoint 3 (15.4, 8.3, 1.0) in the A-A whole picking space, at this time the robot arm starts to adjust the posture to prepare for picking the B2-3 eggplant; the system collects the current posture parameters of the robot arm: base rotation angle a = 45°, large arm swing angle b = 30°, small arm pitch angle g = -15°, end effector rotation angle d = 0°, and extension length L = 0.6 meters; through forward kinematics calculation, the position of the robot arm end effector in the whole picking space is (15.42, 8.32, 0.85); the system marks this position as "robot arm working point-A-A-3", and confirms that the point is located in the effective operation area of the whole picking space (more than 0.3 meters away from the boundary).

[0090] According to the spatial position of the picking robot arm relative to the whole picking space and the flight path of the UAV relative to the eggplant picking site, a second sub-picking path is determined, and the picking action path of the robot arm is planned in combination with the spatial position of the robot arm and the flight path of the UAV; the system realizes it in the following way: the second sub-picking path of the robot arm needs to meet the following conditions: the starting point of the path is the current position of the robot arm, and the end point is the target picking point; the path needs to avoid obstacles (such as other eggplants, branches, supports, etc.); the path needs to consider the kinematic constraints of the robot arm (such as joint angle limit, speed limit); the path needs to be coordinated with the flight path of the UAV to ensure that the UAV hovers in the appropriate position; the system uses a robot arm path planning algorithm (such as a sampling-based RRT algorithm or an optimization-based gradient descent method) to generate a smooth path composed of multiple intermediate points; the path points include the angle sequence of each joint of the robot arm and the spatial coordinate sequence of the end effector; the system performs smoothing processing and collision detection on the generated path to ensure the feasibility and safety of the path.

[0091] Specifically, in the A-A overall picking space, the current position of the mechanical arm is (15.42, 8.32, 0.85), and the position of the target picking point B2-3 is (15.45, 8.35, 0.82); the system plans a second sub-picking path containing 5 intermediate points: starting point: (15.42, 8.32, 0.85), joint angle (45°, 30°, -15°, 0°); intermediate point 1: (15.43, 8.33, 0.84), joint angle (47°, 32°, -14°, 5°); intermediate point 2: (15.44, 8.34, 0.83), joint angle (49°, 34°, -13°, 10°); intermediate point 3: (15.445, 8.345, 0.825), joint angle (50°, 35°, -12°, 15°); end point: (15.45, 8.35, 0.82), joint angle (51°, 36°, -10°, 20°); the total length of the path is 0.05 meters, and the estimated execution time is 2.5 seconds; the system records this path as "second sub-picking path-A-A-3-B2-3", and synchronizes it with the flight path of the unmanned aerial vehicle (hovering at waypoint 3 for 5 seconds) in time, ensuring that the mechanical arm action is completed during the hovering of the unmanned aerial vehicle.

[0092] Based on the first sub-picking path and the second sub-picking path, the corresponding picking path is determined, and the unmanned aerial vehicle flight path (first sub-picking path) and the mechanical arm movement path (second sub-picking path) are fused into a complete picking path; the system realizes it in the following way: path fusion principle: time synchronization: ensure that the mechanical arm action and the unmanned aerial vehicle flight are coordinated in time; spatial coordination: ensure that the unmanned aerial vehicle is in the right position when the mechanical arm operates; task continuity: ensure seamless connection of the picking task between different waypoints; the system uses a space-time synchronization algorithm to align the two paths according to the time axis, generating a composite path containing unmanned aerial vehicle flight and mechanical arm action; the composite path consists of a series of "state-action" pairs, each state including the position and attitude of the unmanned aerial vehicle, and each action including the motion parameters of the mechanical arm; the fused picking path is output in time sequence form, containing the unmanned aerial vehicle state and mechanical arm action at each time point.

[0093] Specifically, in the A-A whole picking space, the system fuses the first sub-picking path (drone flight path) and the second sub-picking path (mechanical arm movement path) into a complete picking path; taking the picking task at waypoint 3 as an example, the time sequence of the fused picking path is as follows: t=0 seconds: the drone is located at waypoint 3 (15.4, 8.3, 1.0) and is in a hovering state; the mechanical arm is located at the starting position (15.42, 8.32, 0.85); t=0.5 seconds: the drone remains hovering; the mechanical arm moves to intermediate point 1 (15.43, 8.33, 0.84); t=1.0 seconds: the drone remains hovering; the mechanical arm moves to intermediate point 2 (15.44, 8.34, 0.83); t=1.5 seconds: the drone remains hovering; the mechanical arm moves to intermediate point 3 (15.445, 8.345, 0.825); t=2.0 seconds: the drone remains hovering; the mechanical arm moves to the end point (15.45, 8.35, 0.82) and performs the picking action; t=2.5 seconds: the drone remains hovering; the mechanical arm completes picking and is retracted to a safe position; t=3.0 seconds: the drone starts moving to waypoint 4; the mechanical arm remains in standby state; the system records this fused path as "picking path-A-A-3-B2-3" and outputs it as a sequence of cooperative control instructions for the drone and the mechanical arm; this path ensures accurate coordination of the drone and the mechanical arm in time and space, achieving efficient and accurate picking operation.

[0094] Reference Figure 6 In step S15, the specific steps are as follows:

[0095] S151: Collect the picking path; determine the corresponding mature areas according to the picking path and the maturity of each eggplant, and mark the maturity grades of each mature area; determine the picking order of each mature area according to the picking quantity of eggplants in each mature area, the maturity grade of each mature area, and the relative position of each mature area; determine a plurality of picking nodes according to the picking order of each mature area, the corresponding area morphology, and the spatial position of each eggplant;

[0096] S152: Determine a plurality of environmental parameters based on the environmental detection of the picking nodes; determine the surrounding environment of the picking nodes according to the plurality of environmental parameters, the positions of the picking nodes, and the corresponding node states; determine the first action trajectory according to the plurality of surface picking positions of each eggplant and the surrounding environment of the corresponding picking nodes;

[0097] S153: Collect the picking morphology of the picking mechanical arm; determine the second action trajectory according to the plurality of surface picking positions of each eggplant and the picking morphology of the picking mechanical arm; determine the picking action of each eggplant according to the first action trajectory, the second action trajectory, and the surface abnormal features of each part of the eggplant; mark the picking action of each eggplant at the corresponding position of the picking path, and construct the picking dynamic diagram corresponding to the picking path.

[0098] In the embodiments of the present application, the system first acquires the complete picking path generated in S143, including the flight trajectory of the unmanned aerial vehicle and the motion trajectory of the mechanical arm, and stores it in a time sequence manner; the system performs spatial superposition analysis on the picking path and the eggplant maturity data (determined by S133), identifies a cluster of eggplants with similar maturity near the path, and divides it into a "mature area"; each area is assigned a maturity level, for example: A level: maturity ≥ 90%, uniform color and luster of the skin, no damage; B level: maturity 70%~90%, slightly uneven color and luster, no serious damage; C level: maturity 50%~70%, with slight damage or not fully mature.

[0099] The system calculates the picking priority of each mature area according to the following three factors: picking quantity: the more eggplants that can be picked in the area, the higher the priority; maturity level: A level area is preferred to B level, and B level is preferred to C level; relative position: adjacent areas are picked first to reduce the flight time and energy consumption of the unmanned aerial vehicle; in each mature area, the system determines a number of "picking nodes" according to the area morphology (such as area shape, size, and eggplant distribution density) and the spatial position of each eggplant.

[0100] Further, a plurality of environmental parameters are determined based on the environment detection of the picking nodes, the surrounding environment of the picking nodes is determined according to the plurality of environmental parameters, the positions of the picking nodes and the corresponding node states; a first action trajectory is determined according to the plurality of surface picking positions of each eggplant and the surrounding environment of the corresponding picking nodes, and the first action trajectory is introduced.

[0101] At this time, the environment of each picking node is perceived and parameters are extracted, providing environmental basis for subsequent action trajectory planning; the environmental parameters include but are not limited to: light intensity (unit: lux): the light intensity of the current node is measured by the light sensor carried by the unmanned aerial vehicle, which affects the image recognition accuracy; wind speed (unit: m / s): detected by the wind speed sensor, which affects the stability of the mechanical arm; air humidity (unit: %RH): detected by the humidity sensor, which is used to judge whether the surface of the eggplant is wet, which affects the clamping strategy; obstacle distribution (type / distance / angle): detected by laser radar or depth camera, such as support, leaf, other eggplants, etc.; temperature (unit: ℃): detected by the temperature sensor, which affects the working state of the mechanical arm motor.

[0102] The environmental parameters are combined with the node position and state to construct a surrounding environment model of the node; environmental safety evaluation: whether it is suitable for picking operation according to the wind speed and obstacle distribution; visibility evaluation: whether the image recognition is reliable according to the light intensity; surface state evaluation: whether the eggplant surface is smooth according to the humidity, which affects the clamping force; the node state includes "standby", "detection in progress", "picking in progress", "completed" and the like, which are used for controlling the process; the system constructs an "environmental feature vector" of the node according to these information, which is used for generating the action trajectory.

[0103] The first action trajectory of the mechanical arm is planned in combination with the eggplant surface picking position (which has been determined in S131) and the surrounding environment of the node; the first action trajectory mainly refers to the spatial path of the mechanical arm moving from the standby position to the target picking position, including: starting point: the default standby position of the mechanical arm; intermediate path point: dynamically generating an obstacle avoidance path according to the obstacle distribution; target point: the best picking position on the eggplant surface; movement mode: including straight line motion, circular motion, segmented approximation and the like; speed control: adjusting the movement speed according to the environmental safety; the system generates the optimal trajectory by using the path planning algorithm (such as RRT), to ensure safety, efficiency and precision; the system realizes the complete process from environmental parameter collection, surrounding environment modeling to first action trajectory planning.

[0104] Therefore, the picking form of the picking mechanical arm is collected, the second action trajectory is determined according to the multiple surface picking positions of each eggplant and the picking form of the picking mechanical arm, the picking action of each eggplant is determined according to the first action trajectory, the second action trajectory and the surface abnormal features of each part of the eggplant, the picking action of each eggplant is marked at the corresponding position of the picking path, and the picking dynamic graph corresponding to the picking path is constructed, the picking dynamic graph is introduced, at the same time, multiple picking nodes are introduced, the overall consideration of the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each eggplant and the picking form of the picking mechanical arm is realized, the precision of the picking action of each eggplant is improved, and the picking dynamic graph is improved.

[0105] At this time, the physical structure, movement ability and working state of the current mechanical arm are obtained, which are used for subsequent trajectory planning and action generation; the picking form includes: mechanical arm degrees of freedom (DOF): for example, a 6-DOF mechanical arm has multiple rotary joints such as shoulder, elbow and wrist; end effector type: such as gripper type, shearing type, suction cup type and the like; clamping force range: such as 0.1N~5N, which can be adjusted; maximum extension length: such as 1.2 meters; movement speed range: such as 0.05m / s~0.5m / s; current posture (joint angle): such as θ1=30°, θ2=-45°, θ3=60° and the like, these parameters determine whether the mechanical arm can accurately reach the target position and execute the picking action.

[0106] According to the plurality of surface picking positions of each eggplant and the picking posture of the picking mechanical arm, a second action trajectory is determined, combining the surface picking position of the eggplant (such as the plurality of contact points determined in S132) and the current picking posture of the mechanical arm, a fine motion trajectory of the end effector of the mechanical arm is planned; The trajectory needs to meet: reachability: ensure that all target points are within the workspace of the mechanical arm; smoothness: the trajectory should be continuous and without abrupt changes; obstacle avoidance: avoid stems, leaves, supports and other obstacles; time efficiency: shorten the motion time as much as possible under the premise of ensuring accuracy; The trajectory is usually composed of a series of spatial points, each point including coordinates, attitude angles, velocities, clamping forces, etc.

[0107] Specifically, assuming that the currently used picking mechanical arm is a 6-DOF gripper type mechanical arm, a schematic table of picking postures is collected, and the schematic table of picking postures is shown in Table 1:

[0108] Table 1 Schematic table of picking postures

[0109]

[0110] The system records this posture as "Posture-ARM07" and uses it as the basis for subsequent trajectory planning; Collect the surface picking position schematic table of eggplant A, and the surface picking position schematic table is shown in Table 2:

[0111] Table 2 Schematic table of surface picking positions

[0112]

[0113] The system generates a second action trajectory according to "Posture-ARM07", and the schematic table of the second action trajectory is shown in Table 3:

[0114] Table 3 Schematic table of second action trajectory

[0115]

[0116] The system records this trajectory as "Second Action Trajectory-A"; according to the first action trajectory, the second action trajectory, and the surface abnormal features of each part of the eggplant, the picking action of each eggplant is determined, the first action trajectory (drone / mechanical arm approach path) is combined with the second action trajectory (mechanical arm fine operation path), and the surface abnormal features (such as rot, damage, wormhole, etc.) are considered to generate the final picking action sequence; The picking action includes: the mechanical arm moves from the standby position to the picking position; fine-tune the attitude to align with the picking position; execute the picking operation; collect the eggplant back to the collection area; Surface abnormal features will affect the selection of picking points and clamping strategies, such as avoiding rot areas and adjusting clamping force to prevent damage.

[0117] Mark each eggplant picking action (time, location, action type) on the picking path to form a spatio-temporal corresponding picking dynamic graph; the picking dynamic graph is a visualization tool, which usually includes: a UAV flight path (blue line); a mechanical arm movement path (green line); picking action markers (red dots, indicating action type and time); a time axis (the horizontal axis represents time, and the vertical axis represents spatial position); the picking dynamic graph is used to guide the actual picking operation and support post-event analysis and optimization.

[0118] Specifically, the schematic table of surface abnormal features of eggplant A is shown in Table 4:

[0119] Table 4 Schematic table of surface abnormal features

[0120]

[0121] The system combines the first action trajectory (the "first action trajectory-N2" generated in S152) and the second action trajectory (the "second action trajectory-A" generated in this step), and avoids the abnormal area, to generate a final picking action sequence table, which is shown in Table 5:

[0122] Table 5 Final picking action sequence table

[0123]

[0124] The system records the action sequence as "picking action-A" and generates control information; it is assumed that in the picking path of F7 plot, the system has generated the picking action of eggplant A and marked it on the picking path; the picking dynamic graph is shown in Table 6:

[0125] Table 6 Picking dynamic graph

[0126]

[0127] Please refer to Figure 7 , Figure 7 is a structural composition schematic diagram of the image recognition-based eggplant picking control system in the embodiment of the present application; the image recognition-based eggplant picking control system comprises:

[0128] An image recognition module 21 is configured to mark the spatial position of each eggplant when a UAV inspects the eggplant picking area, and collect a plurality of current images of each eggplant, and determine the morphological features of the eggplant according to the recognition of the plurality of current images of each eggplant;

[0129] A surface abnormal feature module 22 is configured to determine the sub-image of each part of the eggplant based on the recognition of the plurality of current images, and determine the surface abnormal features of each part according to the recognition of the sub-image of each part of the eggplant;

[0130] The pickable area module 23 is configured to determine a plurality of surface picking positions of the eggplants according to the morphological features and the surface abnormal features of the respective parts of the eggplants, and determine a pickable area according to the spatial positions of the respective eggplants and the distribution map of the eggplant picking area;

[0131] The picking path module 24 is configured to acquire a flight space of the eggplant picking area when the unmanned aerial vehicle is equipped with a picking mechanical arm, and determine a corresponding picking path according to the area morphology of the pickable area, the flight space of the eggplant picking area, and the current position of the picking mechanical arm;

[0132] The picking action module 25 is configured to determine a plurality of picking nodes based on the picking path and the spatial positions of the respective eggplants, determine a picking action of each eggplant according to the surrounding environment of the plurality of picking nodes, the plurality of surface picking positions of the respective eggplants, and the picking morphology of the picking mechanical arm, and construct a picking dynamic map corresponding to the picking path.

[0133] Any combination of the technical features of the above embodiments is possible. In order to make the description concise, not all combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist, they should be considered as the scope of the present disclosure.

Claims

1. An image recognition-based picking control method for eggplants, characterized by, The method comprises the following steps: The unmanned aerial vehicle marks the spatial position of each eggplant when patrolling the eggplant picking area and collects multiple current images of each eggplant, and determines the shape feature of the eggplant according to the recognition of the multiple current images of each eggplant; The method comprises the following steps: determining the sub-image of each part of the eggplant based on the recognition of the multiple current images, determining the surface abnormal feature of each part according to the recognition of the sub-image of each part of the eggplant, including: for each eggplant, determining the sub-image of each part of the eggplant based on the multiple current images, the position of each part of the eggplant, and the corresponding part shape, determining the multiple surface regions according to the recognition of the sub-image of each part, and determining each abnormal part according to the detection of the multiple surface regions; collecting the relative position of each abnormal part, determining the concentration distribution map of each abnormal part according to the relative position of each abnormal part, the corresponding shape, and the region shape of the surface region, and determining the surface abnormal feature of each part based on the concentration distribution map of each abnormal part, the corresponding abnormal type, and the corresponding part; The method comprises the following steps: determining the multiple surface picking positions of the eggplant according to the shape feature of the eggplant and the surface abnormal feature of each part, and determining the pickable region according to the spatial position of each eggplant, the maturity, and the distribution map of the eggplant picking area, including: collecting the multiple current images of the eggplant, determining the surrounding environment space of the eggplant based on the recognition of the multiple current images of the eggplant, determining the multiple sub-space features according to the recognition of the surrounding environment space of the eggplant, determining the surrounding space feature of the eggplant according to the multiple sub-space features and the spatial mapping relationship, and determining the first picking distribution map of the eggplant based on the surrounding space feature of the eggplant and the shape feature of the eggplant; determining the second picking distribution map of the eggplant according to the surrounding space feature of the eggplant and the surface abnormal feature of each part, and determining the multiple surface picking positions of the eggplant based on the first picking distribution map and the second picking distribution map of the eggplant; determining the corresponding surface color coefficient according to the detection of the multiple surface picking positions of the eggplant, determining the maturity of the eggplant according to the multiple surface color coefficients, the shape feature of the eggplant, and the growth time list of the eggplant; determining the multiple eggplant maturity regions based on the maturity of each eggplant, the corresponding spatial position, and the distribution map of the eggplant picking area, and determining the pickable region according to the region position of the multiple eggplant maturity regions, the corresponding region shape, and the flight path of the unmanned aerial vehicle relative to the eggplant picking area. The unmanned aerial vehicle is provided with a picking mechanical arm, and a flight space of an eggplant picking site is collected, and a corresponding picking path is determined according to a region form of a pickable region, the flight space of the eggplant picking site and a current position of the picking mechanical arm, comprising: the unmanned aerial vehicle is provided with a camera for shooting, and the picking mechanical arm is mounted at the front end of the unmanned aerial vehicle and adjusts its posture and position along with the flight of the unmanned aerial vehicle; a distribution region of eggplants and a flight region are collected, a penetration space is determined based on the detection of the distribution region of eggplants, and the flight space of the eggplant picking site is determined according to the penetration space, the flight region and the flight form of the unmanned aerial vehicle; the region form of the pickable region is collected, the overall picking space is determined according to the region form of the pickable region and the flight space of the eggplant picking site, and the first sub-picking path is determined according to the identification of the overall picking space; the picking mechanical arm adjusts its position along with the flight of the unmanned aerial vehicle, a plurality of posture parameters of the picking mechanical arm are collected, the spatial position of the picking mechanical arm relative to the overall picking space is determined according to the plurality of posture parameters of the picking mechanical arm, the current position of the picking mechanical arm and the overall picking space; the second sub-picking path is determined according to the spatial position of the picking mechanical arm relative to the overall picking space and the flight path of the unmanned aerial vehicle relative to the eggplant picking site, and the corresponding picking path is determined based on the first sub-picking path and the second sub-picking path; A plurality of picking nodes are determined based on the picking path and the spatial position of each eggplant, and the picking action of each eggplant is determined according to the surrounding environment of the plurality of picking nodes, a plurality of surface picking positions of each eggplant and the picking form of the picking mechanical arm, so as to construct a picking dynamic graph corresponding to the picking path.

2. The image recognition-based picking control method of eggplants according to claim 1, characterized in that, The unmanned aerial vehicle marks the spatial position of each eggplant when it inspects the eggplant picking site, and collects a plurality of current images of each eggplant, and the form feature of the eggplant is determined according to the identification of the plurality of current images of each eggplant, comprising: A distribution map of the eggplant picking site is collected, the distribution region of eggplants and the flight region are determined according to the identification of the distribution map of the eggplant picking site, the inspection route of the unmanned aerial vehicle to the eggplant picking site is determined based on the distribution region of eggplants, the flight region and the flight form of the unmanned aerial vehicle, the unmanned aerial vehicle flies along the inspection route, the unmanned aerial vehicle dynamically shoots in the flight process, and the spatial position of each eggplant is marked; An image set of each eggplant is collected based on the dynamic shooting of the unmanned aerial vehicle, a plurality of current images of each eggplant are determined according to the screening of the image set of each eggplant, a three-dimensional model of each eggplant is constructed according to the plurality of current images of each eggplant and the growth contour of the eggplant, and the form feature of the eggplant is determined based on the identification of the three-dimensional model of each eggplant.

3. The image recognition-based picking control method of eggplants according to claim 1, characterized in that, The plurality of picking nodes are determined based on the picking path and the spatial position of each eggplant, and the picking action of each eggplant is determined according to the surrounding environment of the plurality of picking nodes, a plurality of surface picking positions of each eggplant and the picking form of the picking mechanical arm, so as to construct a picking dynamic graph corresponding to the picking path, comprising: The picking path is collected; a corresponding ripening area is determined according to the picking path and the maturity of each eggplant, and a ripening grade of each ripening area is marked, a picking order of each ripening area is determined according to the picking quantity of the eggplants in each ripening area, the ripening grade of each ripening area and the relative position of each ripening area; and a plurality of picking nodes are determined according to the picking order of each ripening area, the corresponding area morphology and the spatial position of each eggplant.

4. The image recognition-based picking control method of eggplants according to claim 3, characterized in that, The plurality of picking nodes are determined based on the picking path and the spatial position of each eggplant, the picking action of each eggplant is determined according to the surrounding environment of the plurality of picking nodes, the plurality of surface picking positions of each eggplant and the picking morphology of the picking mechanical arm, and a picking dynamic map corresponding to the picking path is constructed, and the picking control system further comprises: A plurality of environment parameters are determined based on the environment detection of the picking nodes, the surrounding environment of the picking nodes is determined according to the plurality of environment parameters, the position of the picking nodes and the corresponding node state, a first action track is determined according to the plurality of surface picking positions of each eggplant and the surrounding environment of the corresponding picking node; The picking morphology of the picking mechanical arm is collected, a second action track is determined according to the plurality of surface picking positions of each eggplant and the picking morphology of the picking mechanical arm, the picking action of each eggplant is determined according to the first action track, the second action track and the surface abnormal features of each part of the eggplant, the picking action of each eggplant is marked at the corresponding position of the picking path, and a picking dynamic map corresponding to the picking path is constructed.

5. A harvesting control system for eggplants based on image recognition, characterized in that, The eggplant picking control system based on image recognition is applied to the eggplant picking control method based on image recognition as claimed in any one of claims 1-4, and the eggplant picking control system based on image recognition comprises: An image recognition module is configured to mark the spatial position of each eggplant when the unmanned aerial vehicle inspects the eggplant picking area, collect a plurality of current images of each eggplant, and determine the morphological features of the eggplant according to the recognition of the plurality of current images of each eggplant; A surface abnormal feature module is configured to determine the sub-image of each part of the eggplant based on the recognition of the plurality of current images, determine the surface abnormal features of each part according to the recognition of the sub-image of each part of the eggplant, and comprises: for each eggplant, determining the sub-image of each part of the eggplant based on the plurality of current images, the position of each part of the eggplant and the corresponding part morphology, determining a plurality of surface regions according to the recognition of the sub-image of each part, and determining each abnormal part according to the detection of the plurality of surface regions; collecting the relative position of each abnormal part, determining the concentrated distribution map of each abnormal part according to the relative position of each abnormal part, the corresponding morphology and the area morphology of the surface region, and determining the surface abnormal features of each part based on the concentrated distribution map of each abnormal part, the corresponding abnormal type and the corresponding part. The picking area module is used to determine the multiple surface picking positions of the eggplants according to the morphological features and the surface abnormal features of the parts of the eggplants, and to determine the pickable area according to the spatial positions of the eggplants, the ripeness and the distribution map of the eggplant picking ground, and comprises: collecting multiple current images of the eggplants, determining the surrounding environment space of the eggplants based on the recognition of the multiple current images of the eggplants, determining multiple subspace features according to the recognition of the surrounding environment space of the eggplants, determining the surrounding space features of the eggplants according to the multiple subspace features and the spatial mapping relationship, and determining the first picking distribution map of the eggplants based on the surrounding space features of the eggplants and the morphological features of the eggplants; determining the second picking distribution map of the eggplants according to the surrounding space features of the eggplants and the surface abnormal features of the parts, and determining the multiple surface picking positions of the eggplants based on the first picking distribution map and the second picking distribution map of the eggplants; determining the corresponding surface color coefficients according to the detection of the multiple surface picking positions of the eggplants, determining the ripeness of the eggplants according to the surface color coefficients, the morphological features of the eggplants and the growth time list of the eggplants, and determining the multiple eggplant ripening areas based on the ripeness of each eggplant, the corresponding spatial position and the distribution map of the eggplant picking ground, and determining the pickable area according to the area position of the multiple eggplant ripening areas, the corresponding area morphology and the flight path of the unmanned aerial vehicle relative to the eggplant picking ground. The picking path module is used for the unmanned aerial vehicle to carry the picking mechanical arm, to collect the flight space of the eggplant picking ground, and to determine the corresponding picking path according to the area morphology of the pickable area, the flight space of the eggplant picking ground and the current position of the picking mechanical arm, and comprises: the unmanned aerial vehicle is built-in with a camera for shooting, and the picking mechanical arm is carried at the front end of the unmanned aerial vehicle and adjusts the posture and position along with the flight of the unmanned aerial vehicle; collecting the eggplant distribution area and the flight area, determining the interspersed space based on the detection of the eggplant distribution area, and determining the flight space of the eggplant picking ground according to the interspersed space, the flight area and the flight morphology of the unmanned aerial vehicle; collecting the area morphology of the pickable area, determining the overall picking space according to the area morphology of the pickable area and the flight space of the eggplant picking ground, and determining the first sub-picking path according to the recognition of the overall picking space; the picking mechanical arm adjusts the position along with the flight of the unmanned aerial vehicle, collects multiple posture parameters of the picking mechanical arm, determines the spatial position of the picking mechanical arm relative to the overall picking space according to the multiple posture parameters of the picking mechanical arm, the current position of the picking mechanical arm and the overall picking space, determines the second sub-picking path according to the spatial position of the picking mechanical arm relative to the overall picking space and the flight path of the unmanned aerial vehicle relative to the eggplant picking ground, and determines the corresponding picking path based on the first sub-picking path and the second sub-picking path. The picking action module is used to determine multiple picking nodes based on the picking path and the spatial position of each eggplant, to determine the picking action of each eggplant according to the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each eggplant and the picking morphology of the picking mechanical arm, and to construct the picking dynamic map corresponding to the picking path.

Images