Eggplant picking control method and system based on image recognition

Abstract

The invention discloses an eggplant picking control method and system based on image recognition, and relates to the technical field of image recognition, and the method comprises the steps: determining a plurality of surface picking positions of an eggplant according to the morphological characteristics of the eggplant and the surface abnormal characteristics of each part; according to the spatial position and the maturity of each eggplant and the distribution diagram of the eggplant picking place, the pickable area is determined, the sub-image of each part in the eggplant is further identified, and the accuracy of the pickable area is improved. Therefore, a corresponding picking path is determined according to the area form of the picking area, the flying space of the eggplant picking place and the current position of the picking mechanical arm; a plurality of picking nodes are determined based on the picking path and the spatial position of each eggplant, and the picking action of each eggplant is determined according to the surrounding environment of the plurality of picking nodes, the plurality of surface picking positions of each eggplant and the picking form of the picking mechanical arm, so that the accuracy of the picking action of each eggplant is improved; and thus, the accuracy of the picking dynamic graph is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to an eggplant picking control method and system based on image recognition. Background Art

[0002] With the development of science and technology, agricultural picking has gradually entered the era of intelligence. Eggplant picking areas cover eggplants of different maturity levels. Each eggplant is distributed in different locations of the eggplant picking area and has spatial differences. In the existing technology, the distribution positions of each eggplant are collected, and the corresponding picking areas are determined according to the distribution positions of each eggplant. The maturity of each eggplant and the spatial position of each eggplant are not taken into account, which affects the accuracy of the picking area. At the same time, the picking action of each eggplant is designed along the preset dimensions, and the multiple surface picking positions of each eggplant and the picking form of the picking robot arm are not fully considered. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art. The present invention provides an eggplant picking control method and system based on image recognition.

[0004] The embodiment of the present invention provides an eggplant picking control method based on image recognition, comprising: when inspecting the eggplant picking site, a drone marks the spatial position of each eggplant, collects multiple current images of each eggplant, and determines the morphological characteristics of the eggplant based on the recognition of the multiple current images; determines the sub-image of each part of the eggplant based on the recognition of the multiple current images, and determines the surface abnormality characteristics of each part based on the recognition of the sub-image of each part of the eggplant; determines multiple surface picking positions of the eggplant based on the morphological characteristics of the eggplant and the surface abnormality characteristics of each part, and determines the surface abnormality characteristics of each part based on the recognition of the sub-image of each part of the eggplant. The picking area is determined according to the spatial position, maturity and distribution map of the eggplant picking sites of each eggplant; the drone is equipped with a picking robotic arm to collect the flight space of the eggplant picking site, and the corresponding picking path is determined according to the regional morphology of the picking area, the flight space of the eggplant picking site and the current position of the picking robotic arm; based on the picking path and the spatial position of each eggplant, multiple picking nodes are determined, and the picking action of each eggplant is determined according to the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each eggplant and the picking morphology of the picking robotic arm, so as to construct a picking dynamic graph corresponding to the picking path.

[0005] An embodiment of the present invention provides an eggplant picking control system based on image recognition. The eggplant picking control system based on image recognition is applied to the above-mentioned eggplant picking control method based on image recognition. The eggplant picking control system based on image recognition includes: An image recognition module is used by the drone to mark the spatial position of each eggplant when inspecting the eggplant picking site, collect multiple current images of each eggplant, and determine the morphological characteristics of each eggplant based on the recognition of the multiple current images of each eggplant; a surface abnormality feature module, configured to determine sub-images of various parts of the eggplant based on the recognition of multiple current images, and determine surface abnormality features of various parts according to the recognition of the sub-images of various parts of the eggplant; The picking area module is used to determine multiple surface picking locations of the eggplant based on the morphological characteristics of the eggplant and the surface abnormalities of each part, and to determine the picking area based on the spatial position, maturity and distribution map of the eggplant picking areas; The picking path module is used for the drone equipped with a picking robot arm to collect the flight space of the eggplant picking area and determine the corresponding picking path according to the regional morphology of the picking area, the flight space of the eggplant picking area and the current position of the picking robot arm; The picking action module is used to determine multiple picking nodes based on the picking path and the spatial position of each eggplant, and to determine the picking action of each eggplant according to the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each eggplant and the picking form of the picking robot arm, so as to construct a picking dynamic graph corresponding to the picking path.

[0006] Compared with the prior art, the present invention has the following beneficial effects: In an embodiment of the present invention, through the method in the embodiment of the present invention, the surface abnormality features of each part of the eggplant are determined based on the identification of sub-images of each part; multiple surface picking positions of the eggplant are determined based on the morphological characteristics of the eggplant and the surface abnormality features of each part, and the picking area is determined based on the spatial position, maturity and distribution map of the eggplant picking sites of each eggplant. The sub-images of each part of the eggplant are introduced, and the sub-images of each part of the eggplant are further identified, which is compatible with the overall consideration of the spatial position, maturity and distribution map of the eggplant picking sites of each eggplant, thereby improving the accuracy of the picking area.

[0007] Therefore, the drone is equipped with a picking robotic arm, which determines the corresponding picking path according to the regional morphology of the picking area, the flight space of the eggplant picking site and the current position of the picking robotic arm; multiple picking nodes are determined based on the picking path and the spatial position of each eggplant, and the picking action of each eggplant is determined according to the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each eggplant and the picking form of the picking robotic arm to construct a picking dynamic graph corresponding to the picking path. Multiple picking nodes are introduced to achieve overall consideration of the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each eggplant and the picking form of the picking robotic arm, thereby improving the accuracy of the picking action of each eggplant, and making full use of the picking action and picking path of each eggplant, thereby improving the accuracy of the picking dynamic graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 1 is a flow chart of an eggplant picking control method based on image recognition in an embodiment of the present invention; Figure 2 1 is a flow chart of step S11 in the eggplant picking control method based on image recognition in an embodiment of the present invention; Figure 3 1 is a flow chart of step S12 in the eggplant picking control method based on image recognition in an embodiment of the present invention; Figure 4 1 is a flow chart of step S13 in the eggplant picking control method based on image recognition in an embodiment of the present invention; Figure 5 1 is a flow chart of step S14 in the eggplant picking control method based on image recognition in an embodiment of the present invention; Figure 6 1 is a flow chart of step S15 in the eggplant picking control method based on image recognition in an embodiment of the present invention; Figure 7 Schematic diagram of the structure of an eggplant picking control system based on image recognition in an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0010] See also Figures 1 to 7 , an eggplant picking control method based on image recognition is applied to image recognition scenarios; the eggplant picking control method based on image recognition includes: Step S11: The drone marks the spatial position of each eggplant while inspecting the eggplant picking site, collects multiple current images of each eggplant, and determines the morphological characteristics of the eggplant based on the recognition of the multiple current images of each eggplant; Step S12: determining sub-images of various parts of the eggplant based on the recognition of the multiple current images, and determining surface abnormality features of the various parts based on the recognition of the sub-images of the various parts of the eggplant; Step S13: determining multiple surface picking positions of the eggplant based on the morphological characteristics of the eggplant and surface abnormalities of various parts, and determining a picking area based on the spatial position and maturity of each eggplant and a distribution map of the eggplant picking areas; Step S14: The drone is equipped with a picking robot arm, which collects the flight space of the eggplant picking area and determines the corresponding picking path according to the regional morphology of the picking area, the flight space of the eggplant picking area, and the current position of the picking robot arm; Step S15: Determine multiple picking nodes based on the picking path and the spatial position of each eggplant, and determine the picking action of each eggplant according to the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each eggplant and the picking form of the picking robot arm to construct a picking dynamic graph corresponding to the picking path.

[0011] refer to Figure 2 In step S11, the specific steps are: S111: Collecting a distribution map of eggplant picking areas, identifying the eggplant distribution area and flight area based on the distribution map of the eggplant picking areas, determining a drone inspection route for the eggplant picking areas based on the eggplant distribution area, the flight area, and the flight pattern of the drone, and flying the drone along the inspection route. The drone takes dynamic images during flight and marks the spatial position of each eggplant. S112: Based on the dynamic shooting of the drone, an image set of each eggplant is collected, multiple current images of each eggplant are determined based on the screening of the image set of each eggplant, a three-dimensional model of each eggplant is constructed based on the multiple current images of each eggplant and the growth outline of the eggplant, and the morphological characteristics of the eggplant are determined based on the recognition of the three-dimensional model of each eggplant.

[0012] In an embodiment of the present application, the system collects a distribution map of eggplant picking sites through satellite remote sensing, drone aerial photography, or ground mapping equipment. The distribution map of the eggplant picking sites contains not only two-dimensional plane information but also elevation data. Based on the distribution map of the eggplant picking sites, the system automatically identifies the eggplant distribution area and flight area through image segmentation and region recognition. The eggplant distribution area refers to the actual range of land where eggplants are planted, and the system will identify the distribution of each row and each eggplant plant. The flight area is the spatial range in which the drone can safely fly, and it is necessary to avoid obstacles, no-fly zones, and other dangerous areas.

[0013] The system comprehensively considers the geometric characteristics of the eggplant distribution area, the spatial constraints of the flight area, and the flight performance parameters of the UAV (such as maximum flight speed, minimum turning radius, hovering accuracy, etc.), and uses an optimization algorithm to design the optimal inspection route; the route design needs to meet the principle of full coverage to ensure that every eggplant can be photographed; at the same time, flight efficiency must be considered to minimize repeated flights and invalid flights; at this time, the UAV flies autonomously according to the inspection route; during the flight, the UAV's flight control system adjusts the flight attitude and speed in real time to ensure route accuracy; at the same time, the system will make dynamic adjustments based on real-time environmental changes (such as sudden gusts of wind, temporary obstacles, etc.) to ensure flight safety.

[0014] During the flight, the camera onboard the drone takes continuous photos according to preset parameters. The shooting system automatically adjusts the shooting frequency according to the flight speed and altitude to ensure that the image overlap meets the requirements of subsequent processing. At the same time, the system uses multi-sensor fusion technology (such as visual odometry, IMU, GPS, etc.) to calculate the camera position and posture corresponding to each image in real time.

[0015] Furthermore, during the inspection flight, the camera on board the drone will continuously capture images at a preset shooting frequency and angle. For each eggplant detected, the system will collect images taken from different perspectives and distances to form an image set of the eggplant. These images include top-down, side-view, oblique and other angles to ensure that every surface of the eggplant is covered.

[0016] From the collected image collection, the system uses a series of screening algorithms to determine multiple high-quality current images of each eggplant; based on image clarity assessment, it eliminates blurred, jittery or out-of-focus images; based on lighting conditions, it selects images with moderate brightness and good contrast; and considering the shooting angle, it ensures that the screened images can cover all key parts of the eggplant.

[0017] The screened multi-angle images are combined with the growth contour features of the eggplant to construct a three-dimensional stereo model of each eggplant; the system uses a feature point matching algorithm to identify and match the same feature points in images at different angles; the three-dimensional spatial coordinates of these feature points are calculated using the structure-from-motion (SfM) algorithm using the camera calibration parameters and the posture information at the time of shooting; based on the prior knowledge of the eggplant's growth contour (such as the fact that eggplants are usually long and thin at both ends and thicker in the middle), the reconstructed point cloud data is optimized and corrected; a three-dimensional mesh model of the eggplant is generated using a surface reconstruction algorithm (such as Poisson reconstruction or Delaunay triangulation); the texture information of the multi-angle images is mapped onto the three-dimensional model to form a three-dimensional model with realistic texture; during the construction process, the system also considers the natural growth forms of the eggplant, such as bending and twisting, to ensure that the three-dimensional model can accurately reflect the morphological characteristics of the real eggplant.

[0018] Based on the constructed 3D model, the system uses a 3D feature extraction algorithm to identify the morphological characteristics of the eggplant; performs dimensional measurements and calculates basic parameters such as the length, maximum diameter, and volume of the eggplant; analyzes shape characteristics such as curvature, taper, and symmetry; identifies surface features, including color distribution, glossiness, and texture characteristics; detects morphological abnormalities such as deformities, depressions, and enlargements; evaluates maturity characteristics and comprehensively judges the maturity status of the eggplant through indicators such as color, size, and shape; during the recognition process, the system compares the extracted features with the standard eggplant model, calculates the deviation values ​​of each feature, and ultimately forms a complete morphological feature description of the eggplant, including key information such as size parameters, shape characteristics, surface status, and maturity assessment.

[0019] refer to Figure 3 In step S12, the specific steps are: S121: For each eggplant, a sub-image of each part of the eggplant is determined based on multiple current images, the position of each part of the eggplant, and the corresponding part morphology; multiple surface areas are determined based on the recognition of the sub-image of each part; and each abnormal part is determined based on the detection of the multiple surface areas; S122: Collect the relative positions of each abnormal part, determine the concentration distribution map of each abnormal part according to the relative positions of each abnormal part, the corresponding morphology and the regional morphology of the surface area, and determine the surface abnormality characteristics of each part based on the concentration distribution map of each abnormal part, the corresponding abnormal type and the corresponding part.

[0020] In an embodiment of the present application, the system divides the eggplant into three main parts based on the three-dimensional model of the eggplant and botanical knowledge: the top (calyx area), the middle (main body area) and the bottom (pedicel area); each part has its own specific morphological characteristics and position parameters.

[0021] For the top area, the system identifies the morphological characteristics of the calyx (star-shaped structure, green or brown) and position parameters (the top of the eggplant); for the middle area, the system identifies the morphological characteristics of the main body (long oval, purple surface) and position parameters (the main length part of the eggplant); for the bottom area, the system identifies the morphological characteristics of the stalk (lignified, cylindrical) and position parameters (the end point where the eggplant connects to the plant); after determining the position and morphology of each part, the system selects the image that can most clearly show each part from multiple current images, and uses image segmentation algorithms to extract the corresponding sub-images.

[0022] The system uses multi-level image segmentation technology to perform preliminary segmentation based on color and texture features, and then combines edge detection and region growing algorithms for fine segmentation; for the top sub-image, the system divides it into the calyx area, the area around the calyx, and the top peel area; for the middle sub-image, the system divides it into three peel areas: the upper section, the middle section, and the lower section according to the longitudinal characteristics of the eggplant; for the bottom sub-image, the system divides it into the stalk area, the stalk connection area, and the bottom peel area; the division of each surface area takes into account the physiological characteristics of eggplant growth and the abnormal distribution law; during the segmentation process, the system applies a deep learning model to identify regional boundaries and improve segmentation accuracy; at the same time, the system also records the relative position relationship, area size, and morphological characteristics of each surface area, establishes a topological relationship map between surface areas, and provides a spatial reference for subsequent anomaly detection.

[0023] The system uses a multimodal detection method, combining color analysis, texture analysis, shape analysis and deep learning classifiers to conduct comprehensive detection of each surface area; in terms of color analysis, the system calculates the color histogram of each surface area and compares it with the color model of normal eggplant to identify areas with abnormal color; in terms of texture analysis, the system uses methods such as gray-level co-occurrence matrix and wavelet transform to extract surface texture features and identify areas with abnormal texture; in terms of shape analysis, the system uses edge detection and contour analysis to identify shape abnormalities such as depressions, protrusions, and deformations; for each detected abnormal area, the system will further analyze its severity, scope of impact and cause.

[0024] Furthermore, the system establishes a three-dimensional coordinate system with the geometric center of the eggplant as the origin, with the X-axis along the length of the eggplant, the Y-axis along the width of the eggplant, and the Z-axis along the height of the eggplant; for each abnormal part detected, the system records its precise coordinate value in the three-dimensional coordinate system, and calculates the relative distance and angular relationship between the abnormal parts; during the acquisition process, the system considers the curvature characteristics of the eggplant surface, projects the three-dimensional coordinates onto a two-dimensional unfolded plane, and forms a plane relative position map; for each abnormal part, the system also records its position offset relative to the key feature points of the eggplant (such as the center of the calyx, the connection point of the stalk, and the maximum diameter point). These offsets are expressed in percentage to eliminate the size differences between eggplants of different sizes.

[0025] A density-based clustering algorithm (such as DBSCAN) is used to spatially cluster the abnormal parts and identify areas where abnormalities are concentrated. At this time, the system considers the spatial distance, morphological similarity and correlation of abnormal types between abnormal parts to ensure the rationality of the clustering results. For each cluster, the system calculates its center point, coverage, abnormal density and shape characteristics to form descriptive parameters of the abnormal concentrated area. Based on these parameters, the system constructs a concentrated distribution map of the abnormal parts, which is displayed in the form of a heat map. The depth of color indicates the high and low density of abnormalities, and different types of abnormalities are marked with different colors. The concentrated distribution map also contains eggplant surface gridding information, dividing the eggplant surface into several equal-area grid units, each unit showing statistical information on the number and type of abnormalities. In addition, the system also calculates the directional characteristics of the abnormal distribution to identify whether the anomaly is concentrated along a specific direction (such as longitudinal, transverse or circumferential), which helps to analyze the cause of the anomaly.

[0026] The system performs multi-dimensional analysis on the concentrated distribution map, including spatial distribution patterns, abnormality type combinations and severity assessments; for each part, the system identifies its main abnormality type, secondary abnormality type and their distribution characteristics; during the analysis process, the system considers the correlation between abnormalities, such as whether certain abnormalities often occur at the same time, or whether there is a causal relationship; the system also evaluates the impact of abnormalities on eggplant quality, taking into account factors such as the location, size, type and number of abnormalities.

[0027] refer to Figure 4 In step S13, the specific steps are: S131: Acquire multiple current images of the eggplant, determine a surrounding environment space of the eggplant based on recognition of the multiple current images of the eggplant, determine multiple subspace features based on the recognition of the surrounding environment space of the eggplant, determine a surrounding space feature of the eggplant based on the multiple subspace features and a spatial mapping relationship, and determine a first picking distribution map of the eggplant based on the surrounding space feature of the eggplant and a morphological feature of the eggplant; S132: determining a second picking distribution map of the eggplant based on the surrounding spatial features and the surface abnormal features of each part of the eggplant, and determining multiple surface picking positions of the eggplant based on the first picking distribution map and the second picking distribution map; S133: Determine the corresponding surface color coefficient based on the detection of multiple surface picking positions of the eggplant, and determine the maturity of the eggplant based on each surface color coefficient, the morphological characteristics of the eggplant and the list of the growth time of the eggplant; determine multiple eggplant maturity areas based on the maturity of each eggplant, the corresponding spatial position and the distribution map of the eggplant picking site, and determine the picking area based on the regional positions of the multiple eggplant maturity areas, the corresponding regional morphology and the flight path of the drone relative to the eggplant picking site.

[0028] In an embodiment of the present application, multiple current images of the eggplant are collected, and these images contain not only the eggplant itself, but also the environmental information around the eggplant; through image recognition technology, the system extracts the environmental space around the eggplant from these images, including the upper space (lighting conditions, obstructions such as leaves or brackets), the side space (neighboring plants, supporting structures, other fruits) and the lower space (ground conditions, supports, other plants); the system divides the environmental space into three main areas: picking interference area (area that hinders picking operations), safe operation area (area suitable for picking operations) and auxiliary support area (area that can provide support or auxiliary operations); each area has its own specific spatial parameters and attributes.

[0029] The system performs a quantitative analysis of the identified environmental space and extracts multiple subspace features, including: upper openness (0-100%), indicating the degree of openness of the space above the eggplant, side obstacle distance (cm, indicating the distance from the side of the eggplant to the nearest obstacle), lower support stability (0-10 score, indicating the stability of the support below the eggplant), light intensity (lux, indicating the lighting conditions on the eggplant surface), wind influence coefficient (0-1, indicating the degree of influence of environmental wind on the picking operation) and operational accessibility (0-100%, indicating the difficulty of the robotic arm reaching the eggplant surface). Each subspace feature has its own specific measurement method and quantitative standard. For example, upper openness is determined by calculating the proportion of unobstructed area above the eggplant, and side obstacle distance is determined by measuring the three-dimensional distance from the eggplant surface to the nearest obstacle using depth images.

[0030] The system establishes a spatial mapping relationship and maps each subspace feature into a unified multidimensional feature space. This mapping relationship takes into account the mutual influence and weight relationship between different subspace features. For example, there is a positive correlation between the upper openness and the light intensity, and there is a positive correlation between the side obstacle distance and the operational accessibility. Then, the system integrates multiple subspace features into the surrounding space features through a weighted fusion algorithm to form a feature vector containing multiple dimensions. This feature vector usually includes: environmental complexity (0-10, indicating the complexity of the picking environment), operational safety (0-100%, indicating the safety of the picking operation), expected picking efficiency (0-100%, indicating the expected efficiency of picking in this environment) and environmental adaptability (0-100%, indicating the adaptability of the robotic arm in this environment). Each dimension is obtained by weighted calculation of the relevant subspace features. For example, environmental complexity is the weighted average of the upper openness, side obstacle distance and wind influence coefficient.

[0031] The system obtains the morphological characteristics of eggplants, including size (length, diameter), shape (oblong, oval, pear-shaped, etc.), posture (tilt angle, degree of curvature) and surface characteristics (smoothness, hardness); then, the system matches and analyzes these morphological characteristics with the surrounding spatial characteristics to evaluate the picking feasibility of various locations on the eggplant surface; the system uses a gridding method to divide the eggplant surface into multiple small areas (usually 1cm×1cm grids), and calculates a picking feasibility score (0-100 points) for each area; scoring factors include: the accessibility of the area (based on the surrounding spatial characteristics), the mechanical strength of the area (based on the morphological characteristics), the operational safety of the area (based on the surrounding spatial characteristics) and the picking efficiency of the area (based on the morphological characteristics and the surrounding spatial characteristics); the system visualizes these scores in the form of a heat map to generate a first picking distribution map, which intuitively shows the picking feasibility of various locations on the eggplant surface.

[0032] Specifically, assume that the system is analyzing an eggplant A located in the middle of the eggplant plant; the system collects the current image of the eggplant from four angles: the front, the left, a 45° angle above, and a 30° angle below; through image recognition, the system determines that the surrounding environment space of the eggplant includes: 3 leaves partially blocking the upper part (the blocking area is about 30%), another eggplant on the left (about 5 cm away), a bracket on the right (about 3 cm away), and branch support below (support stability score 7 points); the system divides these environmental spaces into a picking interference area (the upper leaf blocking area and the area near the right bracket), a safe operation area (the front and left side areas of the eggplant), and an auxiliary support area (the lower branch support area).

[0033] The system determined several subspace characteristics: the upper openness was 70% (30% blocked by leaves), the obstacle distance on the left was 5 cm (the distance to another eggplant), the obstacle distance on the right was 3 cm (the distance to the support), the lower support stability was 7 points (out of 10 points), the light intensity was 8000 lux (medium light), the wind influence coefficient was 0.2 (slight wind influence), and the operational accessibility was 85% (most areas were easy to reach).

[0034] The system determines the surrounding space characteristics based on the spatial mapping relationship: the environmental complexity is 4.2 points (medium complexity, based on a weighted calculation of 70% upper openness, 3cm minimum side obstacle distance, and 0.2 wind influence coefficient), the operational safety is 88% (high safety, based on a comprehensive assessment of side obstacle distance, lower support stability, and wind influence coefficient), the picking efficiency is expected to be 82% (high efficiency, based on a comprehensive assessment of operational accessibility, environmental complexity, and light intensity), and the environmental adaptability is 90% (high adaptability, based on a comprehensive assessment of operational accessibility, operational safety, and environmental complexity).

[0035] The system generates the first picking distribution map by combining the surrounding spatial characteristics and the eggplant's morphological characteristics (length 12 cm, maximum diameter 6 cm, oblong shape, tilt angle 15°, surface smoothness score 8 points); the system divides the eggplant surface into 60 grid areas (10 length directions × 6 perimeter directions) and calculates the picking feasibility score for each area; for example, the area above the front of the eggplant scores 92 points (high accessibility, high safety, and high efficiency), the area on the right side of the eggplant near the bracket scores 45 points (low accessibility, low safety), and the eggplant is easy to pick. The score of the area below the eggplant near the branches is 88 points (high safety, medium efficiency); the final first picking distribution map is displayed in the form of a heat map, where the red area (90-100 points) indicates the best picking position, the yellow area (70-89 points) indicates a good picking position, the green area (50-69 points) indicates a general picking position, and the blue area (below 50 points) indicates a not recommended picking position; the distribution map shows that the front and upper left areas of the eggplant are the best picking positions, while the area on the right side close to the bracket is a not recommended picking position.

[0036] Furthermore, the system integrates the previously acquired surrounding spatial features (such as operational accessibility, environmental complexity, operational safety, etc.) and the surface abnormality features of each part (such as abnormality location, abnormality type, abnormality severity, etc.); the system establishes an abnormality impact assessment model and sets different impact weights for different types of abnormalities; for example, the weight of mechanical damage type abnormalities is 0.8 (has a greater impact on picking quality), the weight of color abnormality type abnormalities is 0.5 (has a moderate impact on picking quality), and the weight of morphological abnormalities is 0.6 (has a moderate to large impact on picking quality); the system also considers the impact coefficient of the abnormal position on the picking operation, such as the impact coefficient of the abnormality on the top of the eggplant is 0.7 (affecting the selection of picking points), the impact coefficient of the abnormality in the middle is 0.9 (seriously affecting the picking quality), and the impact coefficient of the abnormality at the bottom is 0.4 (has a small impact on picking); the system calculates these factors comprehensively to generate a second picking distribution map, which specifically marks the abnormal areas that should be avoided and the recommended safe picking areas.

[0037] The system performs weighted fusion of the first picking distribution map (based on environmental factors) and the second picking distribution map (based on abnormal factors). Usually, the weight of environmental factors is 0.6, and the weight of abnormal factors is 0.4. Then, the system sets the picking location selection criteria, including: a comprehensive score of not less than 75 points, a distance from the nearest abnormal point of not less than 1 cm, no serious abnormalities within 2 cm around the picking point, and an accessibility score of not less than 70 points in the area where the picking point is located. The system also considers the continuity and efficiency of the picking operation, usually selecting 3-5 alternative picking locations and sorting them by priority. Finally, the system determines multiple optimal surface picking positions on the eggplant surface based on these criteria, and provides detailed operating parameters for each picking position, such as contact angle, pressure range, movement trajectory, etc.

[0038] Specifically, assume that the system is analyzing eggplant A located in the middle of the eggplant plant; the surrounding spatial characteristics of the eggplant include: operational accessibility 85% (high), environmental complexity 35% (low), operational safety 90% (high), light intensity 85% (good), and wind influence coefficient 0.2 (slight impact); the surface abnormalities of the eggplant include: a sunburn spot with a diameter of 0.4 cm in the top area (abnormal color, mild severity); a depression with a diameter of 0.6 cm in the middle right side (abnormal morphology, medium severity); and a slight crack with a length of 0.5 cm in the bottom area (mechanical damage, mild severity).

[0039] The system determines the second picking distribution map based on the surrounding spatial characteristics and surface abnormality characteristics; the system evaluates each abnormality: the abnormality weight of the top sunburn is 0.5 (color abnormality), the position influence coefficient is 0.7 (top area), and the comprehensive impact value is 0.5×0.7=0.35; the abnormality weight of the middle depression is 0.6 (morphological abnormality), the position influence coefficient is 0.9 (middle area), and the comprehensive impact value is 0.6×0.9=0.54; the abnormality weight of the bottom crack is 0.8 (mechanical damage), the position influence coefficient is 0.4 (bottom area), and the comprehensive impact value is 0.8×0.4=0.32.

[0040] The system divides the eggplant surface into 60 grid areas and calculates an abnormal impact score for each area. For example, the abnormal impact score of the area around the sunburn on the top is 65 points (medium impact), the abnormal impact score of the area around the middle depression is 40 points (large impact), and the abnormal impact score of the area around the bottom crack is 70 points (small impact). The system combines the surrounding spatial characteristics (such as operational accessibility 85 points, environmental complexity 35 points, etc.) and the abnormal impact score to generate a second picking distribution map. The distribution map shows that the area around the sunburn on the top of the eggplant is a medium-risk area, the area around the middle depression is a high-risk area, the area around the bottom crack is a low-risk area, and the area on the left side and directly above the eggplant is a low-risk and high-safety area.

[0041] The system performs weighted fusion on the first picking distribution map (mainly considering environmental factors) and the second picking distribution map (mainly considering abnormal factors); assuming that the score of the upper front area of ​​the eggplant in the first picking distribution map is 92 points, the score of the left area is 85 points, and the score of the right area is 45 points; the score of the upper front area of ​​the eggplant in the second picking distribution map is 88 points, the score of the left area is 90 points, and the score of the right area is 38 points; the system calculates according to the environmental factor weight of 0.6 and the abnormal factor weight of 0.4: the comprehensive score of the upper front area is 92×0.6+88×0.4=90.4 points; the comprehensive score of the left area is 85×0.6+90×0.4=87 points; and the comprehensive score of the right area is 45×0.6+38×0.4=42.2 points.

[0042] Based on the picking position selection criteria (comprehensive score of at least 75 points, distance from the nearest abnormal point of at least 1 cm, etc.), the system determines the three optimal surface picking positions on the eggplant surface: the first position is located in the upper front area of ​​the eggplant, with an overall score of 90.4 points, 2.3 cm away from the nearest abnormal point (the sunburn on the top), a recommended contact angle of 45 degrees, and a pressure range of 0.5-0.8N; the second position is located in the upper-middle area of ​​the left side of the eggplant, with an overall score of 87 points, 1.8 cm away from the nearest abnormal point (the middle depression), a recommended contact angle of 30 degrees, and a pressure range of 0.6-0.9N; the third position is located in the middle area of ​​the left side of the eggplant, with an overall score of 82 points, 2.5 cm away from the nearest abnormal point (the bottom crack), a recommended contact angle of 60 degrees, and a pressure range of 0.4-0.7N. The system prioritizes these three picking positions, with the first position as the preferred picking position, the second position as the backup picking position, and the third position as the emergency picking position, providing precise target positions and operating parameters for subsequent picking action control.

[0043] Therefore, the corresponding surface color coefficient is determined based on the detection of multiple surface picking positions of the eggplant, and the maturity of the eggplant is determined based on each surface color coefficient, the morphological characteristics of the eggplant and the list of the eggplant's growth time; multiple eggplant maturity areas are determined based on the maturity of each eggplant, the corresponding spatial position and the distribution map of the eggplant picking site, and the picking area is determined based on the regional positions of the multiple eggplant maturity areas, the corresponding regional morphology and the flight path of the drone relative to the eggplant picking site, thereby improving the accuracy of the picking area. At the same time, sub-images of various parts of the eggplant are introduced to further identify the sub-images of various parts of the eggplant, which is compatible with the overall consideration of the spatial position, maturity and distribution map of the eggplant picking site of each eggplant, thereby further improving the accuracy of the picking area.

[0044] At this point, the system performs color detection on each previously determined surface picking position, and calculates the color coefficient of each picking position through RGB color space analysis and HSV color space conversion; the color coefficient is a comprehensive indicator, including the hue coefficient (reflecting the basic hue of the color, the mature hue coefficient of the purple eggplant ranges from 0.75 to 0.85), the saturation coefficient (reflecting the purity of the color, the saturation coefficient of the mature eggplant ranges from 0.6 to 0.8) and the brightness coefficient (reflecting the brightness of the color, the brightness coefficient of the mature eggplant ranges from 0.4 to 0.6); the system measures the color coefficient of each picking position and then takes the average value as the overall color coefficient of the eggplant; at the same time, the system extracts the morphological characteristics of the eggplant , including length (mature eggplant is usually 15-25cm), diameter (mature eggplant is usually 5-8cm), shape index (ratio of length to diameter, mature eggplant is usually 2.5-4.0) and surface gloss (mature eggplant is usually medium to high gloss); in addition, the system queries the eggplant growth time list to obtain the number of days the eggplant grows (eggplant usually takes 25-35 days from flowering to maturity); finally, the system inputs these parameters into the maturity assessment model, which uses a weighted calculation method, in which the color coefficient weight is 0.5, the morphological feature weight is 0.3, and the growth time weight is 0.2 to calculate the maturity index of the eggplant (ranging from 0-100, and above 70 means it is mature and can be picked).

[0045] The system collects the maturity data and spatial location coordinates (X, Y, Z) of all detected eggplants. It then applies a spatial clustering algorithm (such as DBSCAN or K-means) to group these eggplants, assigning eggplants with similar spatial locations and maturity levels to the same maturity zone. During the clustering process, the system sets a maturity similarity threshold (e.g., a maturity index difference of no more than 10) and a spatial distance threshold (e.g., the distance between adjacent eggplants is no more than 50 cm). For each mature zone formed by a cluster, the system calculates the characteristic parameters of the zone, including the coordinates of the zone center, the zone range (length, width, height), the number of eggplants in the zone, the average maturity, and the standard deviation of maturity. The system also divides the mature zones into different levels based on the average maturity of the eggplants in the zone: fully mature (average maturity ≥ 85), moderately mature (70 ≤ average maturity < 85), and immature (average maturity < 70). Finally, the system marks these mature zones on a map of eggplant picking areas to form an eggplant maturity zone distribution map, providing spatial guidance for subsequent picking planning.

[0046] The system analyzes the location of each ripening area and calculates the shortest distance, relative height difference, and azimuth between each ripening area and the drone's flight path. It then evaluates the regional morphology of each ripening area, including its shape complexity (by calculating the irregularity of its boundaries), its compactness (by calculating the ratio of its area to its perimeter), and its connectivity (by calculating the strength of connections between eggplants within the area). The system also considers the characteristics of the drone's flight path relative to the eggplant picking area, including flight altitude, speed, turning radius, and hovering stability. The system establishes a harvestability assessment model, comprehensively considering location (weighted 0.4), morphology (weighted 0.3), and flight path (weighted 0.3) to calculate the harvestability index for each ripening area. The system sets a harvestability threshold (e.g., an index of at least 60) and marks ripening areas that meet these criteria as harvestable. For each harvestable area, the system also determines the optimal harvesting sequence and strategy, including the drone's entry angle, hovering position, and the operating range of the harvesting robot arm. The system generates a harvestable area distribution map, providing precise spatial navigation and operational guidance for the drone's actual harvesting operations.

[0047] Specifically, suppose the system is analyzing an eggplant A located in the middle of the eggplant plant; the system detects the color coefficients of the three surface picking positions of eggplant A: the hue coefficient of the first position (top front) is 0.82, the saturation coefficient is 0.75, and the brightness coefficient is 0.48; the hue coefficient of the second position (upper middle left) is 0.81, the saturation coefficient is 0.73, and the brightness coefficient is 0.47; the hue coefficient of the third position (middle left) is 0.80, the saturation coefficient is 0.72, and the brightness coefficient is 0.46; the overall color coefficient of eggplant A is calculated to be 0.78; at the same time, the system extracts the morphological characteristics of eggplant A: length 18.5 cm, diameter 5.2 cm, shape index 3.56, surface gloss 0.75; by querying the growth time list, it is learned that the growth days of eggplant A are 28 days; these parameters are input into the maturity assessment model, and the maturity index of eggplant A is calculated to be 82, indicating that the eggplant is mature and can be picked.

[0048] The system detected 12 eggplants around eggplant A, of which 8 had a maturity index between 75 and 90, and their spatial distance from eggplant A was within 30-50 cm. Applying the spatial clustering algorithm, the system divided these 9 eggplants (including eggplant A) into a mature area, marked as mature area M1. The system calculated the characteristic parameters of M1: the coordinates of the area center were (X:125.3m, Y:67.8m, Z:0.8m), the area range was 2.5m long, 1.8m wide, and 0.6m high, the number of eggplants in the area was 9, the average maturity was 78.5, and the standard deviation of maturity was 4.2. Based on the average maturity, the system divided M1 into a moderately mature area. Similarly, the system identified 15 mature areas in the entire planting base, of which 5 were fully mature areas, 8 were moderately mature areas, and 2 were immature areas.

[0049] The system analyzed the regional location characteristics of mature area M1: the shortest distance to the drone's flight path was 3.2m, the relative height difference was 0.3m, and the azimuth angle was 45 degrees; the regional morphological characteristics: shape complexity was 0.25 (low, indicating a relatively regular shape), compactness was 0.68 (medium), and connectivity was 0.82 (high); the drone's flight path characteristics: flight altitude was 2.5m, flight speed was 1.5m / s, turning radius was 1.0m, and hovering stability was 0.85 (high); these parameters were input into the harvestability assessment model, and the harvesting feasibility index of M1 was calculated to be 78, exceeding the set threshold of 60, so M1 was marked as a harvestable area; the system further determined the optimal harvesting strategy for M1: the drone should enter from a 45-degree angle, hover 1.5m above the center of the area, and the operating range of the robotic arm should be a hemispherical space with a radius of 1.2m. Ultimately, the system identified 12 harvestable areas among the 15 mature areas, providing precise spatial guidance for the drone's harvesting mission.

[0050] refer to Figure 5 In step S14, the specific steps are: S141: The drone has a built-in camera for filming. The front end of the drone is equipped with a harvesting robotic arm, which adjusts its attitude and position as the drone flies. The drone collects information about the eggplant distribution area and the flight area. Based on the detected eggplant distribution area, an interlaced space is determined. The flight space for the eggplant harvesting area is determined based on the interlaced space, the flight area, and the drone's flight pattern. S142: Collecting the regional morphology of the picking area, determining the overall picking space based on the regional morphology of the picking area and the flight space of the eggplant picking area, and determining the first sub-picking path based on the identification of the overall picking space; S143: The picking robot arm adjusts its position as the drone flies, collects multiple posture parameters of the picking robot arm, and determines the spatial position of the picking robot arm relative to the overall picking space based on the multiple posture parameters of the picking robot arm, the current position of the picking robot arm, and the overall picking space; determines the second sub-picking path based on the spatial position of the picking robot arm relative to the overall picking space and the flight path of the drone relative to the eggplant picking site, and determines the corresponding picking path based on the first sub-picking path and the second sub-picking path.

[0051] In an embodiment of the present application, the drone is equipped with a built-in camera for shooting, and the front end of the drone is equipped with a picking robotic arm. The picking robotic arm and the camera are distributed at different positions of the drone. The picking robotic arm adjusts its attitude and position as the drone flies. At the same time, the picking robotic arm carried on the front end of the drone is usually a 6-degree-of-freedom robotic arm with a working radius of 0.5-1.2 meters, a maximum load of 2 kilograms, and a positioning accuracy of ±2 mm; the robotic arm is equipped with a real-time attitude adjustment system, including an IMU (inertial measurement unit) and a servo motor, which can automatically adjust its own attitude according to the flight status of the drone to maintain stability.

[0052] When the drone is tilted, the system first collects the eggplant distribution area. This data comes from the eggplant spatial distribution information determined in the previous step (such as S111); the eggplant distribution area is represented in the form of three-dimensional point cloud data, and each point represents the position of an eggplant or eggplant plant, including X, Y, and Z coordinate information.

[0053] At the same time, the system collects flight area data. This data comes from the safe flight area of ​​the drone determined in step S111, is represented by a polygonal boundary, and contains height restriction information. For example, the flight area is a rectangular area 22 meters long and 17 meters wide, with a height restriction of 1-3 meters. Based on the detection of the eggplant distribution area, the system determines the interspersed space, that is, the gap area between the eggplant plants. The detection algorithm of the interspersed space is based on cluster analysis and spatial segmentation of three-dimensional point cloud data.

[0054] The eggplant plant point cloud data is clustered to identify the space occupied by each plant, and then the interspace between plants is calculated. The interspace calculation takes into account the following factors: minimum safety distance (typically no less than 0.8 meters to ensure safe passage of drones), the geometry of the interspace (such as circular, elliptical, or irregular), and the connectivity of the interspace (to ensure continuous drone flight). Based on the interspace, flight area, and drone flight pattern, the system determines the flight space for the eggplant picking area through spatial overlay analysis and path planning algorithms.

[0055] The system overlays and analyzes the interspersed space and the flight area to determine the intersection of the two, that is, the area that meets both safe flight conditions and can approach the eggplant; the system takes into account the flight form of the drone, including the drone's size (for example, 0.8 meters long, 0.6 meters wide, and 0.3 meters high), maneuverability (such as a minimum turning radius of 1.5 meters and a maximum climb angle of 30 degrees) and flight speed (usually 0.5-2 meters per second); the planning of the flight space follows the following principles: safety principle (ensuring that the drone maintains a safe distance from obstacles), efficiency principle (minimizing flight distance and time) and operability principle (taking into account the physical limitations of the drone); the system uses a three-dimensional algorithm for path planning to generate a continuous three-dimensional flight space, which consists of a series of interconnected three-dimensional areas, each with clear boundaries and height restrictions.

[0056] Furthermore, the regional morphology of the picking area is collected, the overall picking space is determined based on the regional morphology of the picking area and the flight space of the eggplant picking site, and the first sub-picking path is determined based on the identification of the overall picking space, and the first sub-picking path is introduced.

[0057] At this time, the regional morphology of the picking area is collected. The picking area refers to the area where the eggplant plants that meet the picking conditions are located. Usually, the following conditions must be met: the maturity of the eggplant reaches the picking standard, there are no obstructions around, and the robotic arm is accessible. The regional morphology of the picking area is integrated with the previously determined flight space to construct an "overall picking space". The overall picking space is a three-dimensional space in which the drone and the robotic arm can collaboratively perform picking tasks. The following conditions must be met: spatial continuity: the flight space and the picking area are spatially connected and there are no obstacles blocking them. Operational accessibility: the robotic arm can enter the picking area with a reasonable posture and perform picking. Safety constraints: avoid collisions between the drone or robotic arm and plants, brackets, etc. At the same time, the system uses a method that combines spatial mapping and Boolean operations to map the picking area to the flight space to generate an overall picking space.

[0058] The first sub-picking path is determined based on the identification of the overall picking space. The first sub-picking path is the flight path of the drone in the overall picking space. The path must meet the following requirements: path continuity: the path consists of a series of waypoints, and the drone can fly smoothly along the path; path optimization: the path length is as short as possible and the flight time is as short as possible; task adaptability: the path must cover all picking points and ensure that the drone hovers in a suitable position for robotic arm operation; the system uses a path planning algorithm (such as RRT) to generate an initial path and optimizes the path trajectory through smoothing processing (such as Bezier curve fitting).

[0059] Specifically, taking the AA overall picking space as an example, in the AA overall picking space, the system identifies 12 picking points, distributed at different heights and positions; based on the algorithm, the system generates a path containing 8 waypoints, with a total length of 11.3 meters and an estimated flight time of 28 seconds; the starting point of the path is the entrance of the AA space (14.5, 7.5, 1.5), and the end point is the exit (16.2, 9.9, 1.2), passing through 6 hovering points in the middle, each hovering point corresponding to the picking position of one or more eggplants; for example, at waypoint 3 (15.4, 8.3, 1.0), the drone hovers for 5 seconds, and the robotic arm picks two eggplants numbered B2-3 and B2-4; the system records the path as "first sub-picking path-AA" and outputs it as a trajectory instruction sequence executable by the drone flight control system.

[0060] Therefore, the picking robot arm adjusts its position as the drone flies, collects multiple posture parameters of the picking robot arm, and determines the spatial position of the picking robot arm relative to the overall picking space based on the multiple posture parameters of the picking robot arm, the current position of the picking robot arm, and the overall picking space; determines the second sub-picking path based on the spatial position of the picking robot arm relative to the overall picking space and the flight path of the drone relative to the eggplant picking site, and determines the corresponding picking path based on the first sub-picking path and the second sub-picking path, thereby improving the accuracy of the picking path.

[0061] At this time, during the flight of the drone, the robotic arm needs to adjust its position in real time to adapt to different picking needs; the system achieves this in the following ways: the robotic arm is equipped with multiple sensors (such as joint encoders, IMUs, and gyroscopes) to collect parameters such as joint angles, end effector positions, rotation angles, and telescopic lengths in real time; typical posture parameters include: base rotation angle (α), upper arm swing angle (β), lower arm pitch angle (γ), end effector rotation angle (δ), telescopic length (L), etc.; combining the current absolute position (through the drone GPS positioning) and relative position (calculated through the robotic arm joint angles) of the robotic arm, the system uses forward kinematics to solve the precise coordinates (x, y, z) of the robotic arm's end effector in the overall picking space; the end position of the robotic arm is mapped to the overall picking space coordinate system to ensure that the operating range of the robotic arm matches the picking space; the system will calibrate the relative relationship between the robotic arm's workspace and the overall picking space in real time to avoid exceeding the operating range.

[0062] Specifically, assume that the drone flies to waypoint 3 (15.4, 8.3, 1.0) in the AA overall picking space. At this time, the robotic arm begins to adjust its posture to prepare for picking eggplant B2-3. The system collects the following current posture parameters of the robotic arm: base rotation angle α = 45°, arm swing angle β = 30°, arm pitch angle γ = -15°, end effector rotation angle δ = 0°, and telescopic length L = 0.6 meters. Through forward kinematics calculation, the position of the robotic arm end effector in the overall picking space is (15.42, 8.32, 0.85). The system marks this position as "robotic arm working point - AA-3" and confirms that it is within the effective operating area of ​​the overall picking space (more than 0.3 meters from the boundary).

[0063] The second sub-picking path is determined based on the spatial position of the picking robot arm relative to the overall picking space and the flight path of the drone relative to the eggplant picking site. The picking action path of the robot arm is planned in combination with the spatial position of the robot arm and the flight path of the drone. The system is implemented in the following ways: the second sub-picking path of the robot arm must meet the following conditions: the starting point of the path is the current position of the robot arm, and the end point is the target picking point; the path must avoid obstacles (such as other eggplants, branches and leaves, brackets, etc.); the path must consider the kinematic constraints of the robot arm (such as joint angle limits, speed limits); the path must be coordinated with the flight path of the drone to ensure that the drone hovers in the appropriate position; the system uses a robot arm path planning algorithm (such as a sampling-based RRT algorithm or an optimization-based gradient descent method) to generate a smooth path consisting of multiple intermediate points; the path points include the angle sequence of each joint of the robot arm and the spatial coordinate sequence of the end effector; the system smoothes and performs collision detection on the generated path to ensure the feasibility and safety of the path.

[0064] Specifically, in the AA overall picking space, the current position of the robot arm is (15.42, 8.32, 0.85), and the position of the target picking point B2-3 is (15.45, 8.35, 0.82); the system plans a second sub-picking path containing 5 intermediate points: starting point: (15.42, 8.32, 0.85), joint angle (45°, 30°, -15°, 0°); intermediate point 1: (15.43, 8.33, 0.84), joint angle (47°, 32°, -14°, 5°); intermediate point 2: (15.44, 8.34, 0.83), joint angle (49°, 34°, -13°, 10°); intermediate point 3: (15.445, 8.345, 0.825), joint angle (50°, 35°, -12°, 15°); end point: (15.45, 8.35, 0.82), joint angle (51°, 36°, -10°, 20°); the total length of the path is 0.05 meters, and the estimated execution time is 2.5 seconds; the system records this path as "Second Sub-Picking Path - AA-3 - B2-3" and synchronizes it with the drone's flight path (hovering at waypoint 3 for 5 seconds) to ensure that the robotic arm's movements are completed while the drone is hovering.

[0065] Based on the first and second sub-picking paths, the corresponding picking path is determined, and the UAV flight path (first sub-picking path) and the robotic arm motion path (second sub-picking path) are fused into a complete picking path. The system is implemented in the following ways: Path fusion principle: Time synchronization: ensuring that the robotic arm movement and UAV flight are coordinated in time; Spatial coordination: ensuring that the UAV is in the appropriate position when the robotic arm operates; Task continuity: ensuring that the picking task is seamlessly connected between different waypoints; The system uses a spatiotemporal synchronization algorithm to align the two paths along the time axis to generate a composite path that includes UAV flight and robotic arm movement; The composite path consists of a series of "state-action" pairs, each state includes the position and posture of the UAV, and each action includes the motion parameters of the robotic arm; The fused picking path is output in the form of a time series, including the UAV state and robotic arm movement at each time point.

[0066] Specifically, in the AA overall picking space, the system merges the first sub-picking path (UAV flight path) and the second sub-picking path (robotic arm motion path) into a complete picking path; taking the picking task at waypoint 3 as an example, the time series of the fused picking path is as follows: t=0 seconds: the UAV is at waypoint 3 (15.4, 8.3, 1.0) and is in a hovering state; the robotic arm is at the starting position (15.42, 8.32, 0.85); t=0.5 seconds: the UAV remains in a hovering state; the robotic arm moves to the middle point 1 (15.43, 8.33, 0.84); t=1.0 seconds: the UAV remains in a hovering state; the robotic arm moves to the middle point 2 (15.44, 8.34, 0.83); t=1 0.5 seconds: The UAV remains in hovering; the robotic arm moves to the middle point 3 (15.445, 8.345, 0.825); t=2.0 seconds: The UAV remains in hovering; the robotic arm moves to the end point (15.45, 8.35, 0.82) and performs the picking action; t=2.5 seconds: The UAV remains in hovering; the robotic arm completes the picking and retracts to a safe position; t=3.0 seconds: The UAV begins to move toward waypoint 4; the robotic arm remains in standby mode; the system records this fused path as "picking path-AA-3-B2-3" and outputs it as a coordinated control command sequence for the UAV and robotic arm; this path ensures precise coordination of the UAV and robotic arm in time and space, achieving efficient and accurate picking operations.

[0067] refer to Figure 6 In step S15, the specific steps are: S151: Collecting a picking path; determining corresponding mature areas based on the picking path and the maturity of each eggplant, marking the maturity level of each mature area, and determining a picking order for each mature area based on the number of eggplants picked in each mature area, the maturity level of each mature area, and the relative position of each mature area; determining multiple picking nodes based on the picking order of each mature area, the corresponding area morphology, and the spatial position of each eggplant; S152: Determine multiple environmental parameters based on environmental detection of the picking node, determine the surrounding environment of the picking node based on the multiple environmental parameters, the position of the picking node, and the corresponding node state; and determine a first motion trajectory based on multiple surface picking positions of each eggplant and the surrounding environment of the corresponding picking node. S153: Collect the picking shape of the picking robot arm, determine the second motion trajectory according to the multiple surface picking positions of each eggplant and the picking shape of the picking robot arm, determine the picking action of each eggplant according to the first motion trajectory, the second motion trajectory and the surface abnormal features of each part of the eggplant, mark the picking action of each eggplant at the corresponding position of the picking path, and construct a picking dynamic graph corresponding to the picking path.

[0068] In an embodiment of the present application, the system first obtains the complete picking path generated in S143, including the flight trajectory of the drone and the motion trajectory of the robotic arm, and stores it in a time series manner; the system performs spatial overlay analysis on the picking path and the eggplant maturity data (determined by S133), identifies clusters of eggplants with similar maturity near the path, and divides them into a "maturity area"; each area is assigned a maturity level, for example: Grade A: maturity ≥90%, uniform skin color, no damage; Grade B: maturity 70%~90%, slightly uneven color, no serious damage; Grade C: maturity 50%~70%, slight damage or not fully mature.

[0069] The system calculates the picking priority of each mature area and generates a picking order based on the following three factors: Picking quantity: the more eggplants that can be picked in the area, the higher the priority; Maturity level: Grade A areas take precedence over Grade B, and Grade B takes precedence over Grade C; Relative position: Adjacent areas are picked first to reduce drone flight time and energy consumption; within each mature area, the system determines several "picking nodes" based on the area morphology (such as area shape, size, eggplant distribution density) and the spatial position of each eggplant; each picking node represents the specific location where the drone hovers in the area and performs robotic arm picking.

[0070] Furthermore, multiple environmental parameters are determined based on environmental detection of the picking node, and the surrounding environment of the picking node is determined according to the multiple environmental parameters, the position of the picking node and the corresponding node status; the first action trajectory is determined according to the multiple surface picking positions of each eggplant and the surrounding environment of the corresponding picking node, and the first action trajectory is introduced.

[0071] At this time, environmental perception and parameter extraction are performed on each picking node to provide an environmental basis for subsequent action trajectory planning; environmental parameters include but are not limited to: light intensity (unit: lux): the lighting conditions of the current node are measured by the photosensor carried by the drone, which affects the image recognition accuracy; wind speed (unit: m / s): detected by the wind speed sensor, which affects the stability of the robotic arm; air humidity (unit: %RH): detected by the humidity sensor, used to determine whether the eggplant surface is moist, affecting the clamping strategy; obstacle distribution (type / distance / angle): detected by lidar or depth camera, such as brackets, leaves, other eggplants, etc.; temperature (unit: ℃): detected by the temperature sensor, which affects the working state of the robotic arm motor.

[0072] Environmental parameters are combined with node position and status to construct a surrounding environment model of the node; environmental safety assessment: determines whether picking operations are suitable based on wind speed and obstacle distribution; visibility assessment: determines whether image recognition is reliable based on light intensity; surface condition assessment: determines whether the eggplant surface is smooth based on humidity, which affects the clamping force; node status includes "standby", "detecting", "picking", "completed" and other states, which are used to control the process; the system constructs the "environmental feature vector" of the node based on this information, which is used to generate the action trajectory.

[0073] The first motion trajectory of the robot arm is planned based on the picking position on the eggplant surface (determined in S131) ​​and the surrounding environment of the node. The first motion trajectory mainly refers to the spatial path of the robot arm moving from the standby position to the target picking position, including: starting point: the default standby position of the robot arm; intermediate path points: dynamically generating obstacle avoidance paths based on obstacle distribution; target point: the optimal picking position on the eggplant surface; movement mode: including linear movement, circular movement, segmented approximation, etc.; speed control: adjusting the movement speed according to environmental safety; the system uses a path planning algorithm (such as RRT) to generate the optimal trajectory to ensure safety, efficiency, and accuracy; the system realizes a complete process from environmental parameter collection, surrounding environment modeling to first motion trajectory planning.

[0074] Therefore, the picking form of the picking robot arm is collected, and the second action trajectory is determined according to the multiple surface picking positions of each eggplant and the picking form of the picking robot arm. The picking action of each eggplant is determined according to the first action trajectory, the second action trajectory and the surface abnormal characteristics of each part of the eggplant. The picking action of each eggplant is marked at the corresponding position of the picking path, and a picking dynamic graph corresponding to the picking path is constructed. The picking dynamic graph is introduced. At the same time, multiple picking nodes are introduced, and the surrounding environment of multiple picking nodes, the multiple surface picking positions of each eggplant and the picking form of the picking robot arm are taken into overall consideration, which improves the accuracy of the picking action of each eggplant, makes full use of the picking action and picking path of each eggplant, and improves the accuracy of the picking dynamic graph.

[0075] At this point, the physical structure, motion capability, and working status of the current robotic arm are obtained for subsequent trajectory planning and action generation. The picking form includes: robotic arm degrees of freedom (DOF): for example, a 6-DOF robotic arm with multiple rotational joints such as shoulder, elbow, and wrist; end effector type: such as gripper, shear, suction cup, etc.; clamping force range: such as 0.1N~5N, adjustable; maximum extension length: such as 1.2 meters; motion speed range: such as 0.05m / s~0.5m / s; current posture (joint angle): such as θ1=30°, θ2=-45°, θ3=60°, etc. These parameters determine whether the robotic arm can accurately reach the target position and perform the picking action.

[0076] The second motion trajectory is determined based on the multiple surface picking positions of each eggplant and the picking form of the picking robot arm. Combined with the surface picking positions of the eggplant (such as the multiple contact points determined in S132) and the current picking form of the robot arm, the fine motion trajectory of the robot arm end effector is planned; the trajectory must meet the following requirements: accessibility: ensure that all target points are within the working space of the robot arm; smoothness: the trajectory should be continuous and without sudden changes; obstacle avoidance: avoid obstacles such as stems, leaves, and brackets; time efficiency: shorten the motion time as much as possible while ensuring accuracy; the trajectory is usually composed of a series of spatial points, each point including coordinates, attitude angles, speed, clamping force, etc.

[0077] Specifically, assuming that the currently used picking robot arm is a 6-DOF gripper-type robot arm, a schematic table of the picking form is shown in Table 1: Table 1 Schematic diagram of picking patterns

[0078] The system records this shape as "Shape-ARM07" and uses it as the basis for subsequent trajectory planning. A schematic diagram of the surface picking position of eggplant A is collected. The schematic diagram of the surface picking position is shown in Table 2: Table 2 Schematic diagram of surface picking positions

[0079] The system generates the second motion trajectory based on "Morphology-ARM07". The schematic diagram of the second motion trajectory is shown in Table 3: Table 3 Schematic diagram of the second motion trajectory

[0080] The system records this trajectory as "second action trajectory-A"; based on the first action trajectory, the second action trajectory and the surface abnormalities of each part of the eggplant, the picking action of each eggplant is determined, and the first action trajectory (drone / robotic arm approach path) is combined with the second action trajectory (robotic arm fine operation path), and the abnormal surface features of the eggplant (such as rot, damage, wormholes, etc.) are taken into account to generate the final picking action sequence; the picking action includes: the robotic arm moves from the standby position to the picking position; fine-tunes the posture to align with the picking position; performs the picking operation; and returns the eggplant to the collection area; surface abnormalities will affect the selection of picking points and the clamping strategy, such as avoiding rotten areas and adjusting the clamping force to prevent damage.

[0081] The picking action (time, location, and action type) of each eggplant is marked on the picking path to form a picking dynamic graph corresponding to time and space. The picking dynamic graph is a visualization tool that usually includes: the drone flight path (blue line); the robotic arm motion path (green line); the picking action marker (red dot, indicating the action type and time); and the timeline (the horizontal axis represents time, and the vertical axis represents spatial position). The picking dynamic graph is used to guide actual picking operations and support subsequent analysis and optimization.

[0082] Specifically, a schematic diagram of abnormal surface characteristics of eggplant A is shown in Table 4: Table 4 Schematic diagram of surface abnormality characteristics

[0083] The system combines the first action trajectory ("first action trajectory-N2" generated in S152) and the second action trajectory ("second action trajectory-A" generated in this step), avoids abnormal areas, and generates a final picking action sequence table. The final picking action sequence table is shown in Table 5: Table 5 Final picking action sequence

[0084] The system records this action sequence as "picking action-A" and generates control information. Assume that in the picking path of plot F7, the system has generated the picking action of eggplant A and marked it on the picking path. The picking dynamic diagram is shown in Table 6: Table 6 Picking dynamic diagram

[0085] See also Figure 7 , Figure 7 : is a schematic diagram of the structure of an eggplant picking control system based on image recognition in an embodiment of the present invention; the eggplant picking control system based on image recognition includes: Image recognition module 21, used by the drone to mark the spatial position of each eggplant when inspecting the eggplant picking site, collect multiple current images of each eggplant, and determine the morphological characteristics of each eggplant based on the recognition of the multiple current images of each eggplant; a surface abnormality feature module 22 for determining sub-images of various parts of the eggplant based on the recognition of the multiple current images, and determining surface abnormality features of the various parts according to the recognition of the sub-images of the various parts of the eggplant; The picking area module 23 is used to determine multiple surface picking locations of the eggplant based on the morphological characteristics of the eggplant and the surface abnormalities of various parts, and to determine the picking area based on the spatial position and maturity of each eggplant and the distribution map of the eggplant picking areas; The picking path module 24 is used for the drone equipped with a picking robot arm to collect the flight space of the eggplant picking area and determine the corresponding picking path according to the regional morphology of the picking area, the flight space of the eggplant picking area and the current position of the picking robot arm; The picking action module 25 is used to determine multiple picking nodes based on the picking path and the spatial position of each eggplant, and determine the picking action of each eggplant according to the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each eggplant and the picking form of the picking robot arm, so as to construct a picking dynamic graph corresponding to the picking path.

[0086] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. An eggplant picking control method based on image recognition, characterized in that: include: When inspecting the eggplant picking area, the drone marks the spatial position of each eggplant and collects multiple current images of each eggplant. Based on the recognition of the multiple current images of each eggplant, the morphological characteristics of the eggplant are determined. Determining sub-images of various parts of the eggplant based on the recognition of the multiple current images, and determining surface abnormality features of the various parts based on the recognition of the sub-images of the various parts of the eggplant; Determining multiple surface picking locations of the eggplant based on the morphological characteristics of the eggplant and surface abnormalities of various parts, and determining a picking area based on the spatial location, maturity, and distribution map of the eggplant picking areas of each eggplant; The drone is equipped with a picking robot arm, which collects the flight space of the eggplant picking area and determines the corresponding picking path based on the regional morphology of the picking area, the flight space of the eggplant picking area, and the current position of the picking robot arm; Based on the picking path and the spatial position of each eggplant, multiple picking nodes are determined. According to the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each eggplant and the picking form of the picking robot arm, the picking action of each eggplant is determined to construct a picking dynamic graph corresponding to the picking path.

2. The eggplant picking control method based on image recognition according to claim 1 is characterized in that: The drone marks the spatial position of each eggplant when inspecting the eggplant picking site, collects multiple current images of each eggplant, and determines the morphological characteristics of the eggplant based on the recognition of the multiple current images of each eggplant, including: A distribution map of eggplant picking sites is collected. Based on this map, the eggplant distribution area and flight zone are determined. Based on the eggplant distribution area, flight zone, and the drone's flight pattern, a drone inspection route for the eggplant picking sites is determined. The drone flies along this inspection route, taking dynamic images during flight and marking the spatial location of each eggplant. Based on the dynamic shooting of the drone, an image set of each eggplant is collected, and multiple current images of each eggplant are determined based on the screening of the image set of each eggplant. A three-dimensional model of each eggplant is constructed based on the multiple current images of each eggplant and the growth outline of the eggplant, and the morphological characteristics of the eggplant are determined based on the recognition of the three-dimensional model of each eggplant.

3. The eggplant picking control method based on image recognition according to claim 1 is characterized in that: The determining of sub-images of various parts of the eggplant based on the recognition of the multiple current images, and determining surface abnormality features of various parts of the eggplant based on the recognition of the sub-images of various parts of the eggplant, include: For each eggplant, a sub-image of each part of the eggplant is determined based on multiple current images, the position of each part of the eggplant, and the corresponding part morphology. Multiple surface areas are determined based on the recognition of the sub-image of each part. Each abnormal part is determined based on the detection of the multiple surface areas. The relative positions of each abnormal part are collected, and the concentrated distribution map of each abnormal part is determined according to the relative positions of each abnormal part, the corresponding morphology and the regional morphology of the surface area. The surface abnormality characteristics of each part are determined based on the concentrated distribution map of each abnormal part, the corresponding abnormal type and the corresponding part.

4. The eggplant picking control method based on image recognition according to claim 1, characterized in that: The method of determining multiple surface picking positions of the eggplant according to the morphological characteristics of the eggplant and the surface abnormalities of various parts thereof, and determining the picking area according to the spatial position, maturity and distribution map of the eggplant picking areas of each eggplant, includes: capturing multiple current images of the eggplant, determining a surrounding environment space of the eggplant based on recognition of the multiple current images of the eggplant, determining multiple subspace features based on the recognition of the surrounding environment space of the eggplant, determining surrounding space features of the eggplant based on the multiple subspace features and a spatial mapping relationship, and determining a first picking distribution map of the eggplant based on the surrounding space features of the eggplant and morphological features of the eggplant; A second picking distribution map of the eggplant is determined according to the peripheral spatial characteristics and surface abnormal characteristics of each part of the eggplant, and multiple surface picking positions of the eggplant are determined based on the first picking distribution map and the second picking distribution map of the eggplant.

5. The eggplant picking control method based on image recognition according to claim 4 is characterized in that: The method further includes: determining multiple surface picking positions of the eggplant according to the morphological characteristics of the eggplant and the surface abnormalities of various parts of the eggplant, and determining the picking area according to the spatial position, maturity and distribution map of the eggplant picking areas of each eggplant. The corresponding surface color coefficient is determined by detecting multiple surface picking positions of the eggplant, and the maturity of the eggplant is determined based on each surface color coefficient, the morphological characteristics of the eggplant and a list of the eggplant's growth time. Multiple eggplant maturity areas are determined based on the maturity of each eggplant, the corresponding spatial position and the distribution map of the eggplant picking site, and the picking area is determined based on the regional positions of the multiple eggplant maturity areas, the corresponding regional morphology and the flight path of the drone relative to the eggplant picking site.

6. The eggplant picking control method based on image recognition according to claim 1, characterized in that: The drone is equipped with a picking robot arm, which collects the flight space of the eggplant picking area and determines the corresponding picking path according to the regional morphology of the picking area, the flight space of the eggplant picking area and the current position of the picking robot arm, including: The drone has a built-in camera for taking pictures, and a picking robot arm is mounted on the front end of the drone. The picking robot arm adjusts its posture and position as the drone flies. The eggplant distribution area and flight area are collected, and the interlaced space is determined based on the detection of the eggplant distribution area. The flight space of the eggplant picking area is determined according to the interlaced space, flight area and flight form of the drone.

7. The eggplant picking control method based on image recognition according to claim 6, characterized in that: The drone is equipped with a picking robot arm, collects the flight space of the eggplant picking area, and determines the corresponding picking path according to the regional shape of the picking area, the flight space of the eggplant picking area, and the current position of the picking robot arm, and further includes: Collecting the regional morphology of the picking area, determining the overall picking space based on the regional morphology of the picking area and the flight space of the eggplant picking area, and determining the first sub-picking path based on the identification of the overall picking space; The picking robot arm adjusts its position as the drone flies, collects multiple posture parameters of the picking robot arm, and determines the spatial position of the picking robot arm relative to the overall picking space based on the multiple posture parameters of the picking robot arm, the current position of the picking robot arm, and the overall picking space; determines the second sub-picking path based on the spatial position of the picking robot arm relative to the overall picking space and the flight path of the drone relative to the eggplant picking site, and determines the corresponding picking path based on the first sub-picking path and the second sub-picking path.

8. The eggplant picking control method based on image recognition according to claim 1, characterized in that: The method comprises determining a plurality of picking nodes based on the picking path and the spatial position of each eggplant, determining the picking action of each eggplant according to the surrounding environment of the plurality of picking nodes, the plurality of surface picking positions of each eggplant, and the picking form of the picking robot arm, so as to construct a picking dynamic graph corresponding to the picking path, including: Collect picking paths; determine corresponding mature areas according to the picking paths and the maturity of each eggplant, mark the maturity level of each mature area, and determine the picking order of each mature area according to the number of eggplants picked in each mature area, the maturity level of each mature area and the relative position of each mature area; determine multiple picking nodes according to the picking order of each mature area, the corresponding area morphology and the spatial position of each eggplant.

9. The eggplant picking control method based on image recognition according to claim 8, characterized in that: The method further includes determining a plurality of picking nodes based on the picking path and the spatial position of each eggplant, determining the picking action of each eggplant according to the surrounding environment of the plurality of picking nodes, the plurality of surface picking positions of each eggplant, and the picking form of the picking robot arm, so as to construct a picking dynamic graph corresponding to the picking path, and further including: Determine multiple environmental parameters based on environmental detection of the picking node, determine the surrounding environment of the picking node based on the multiple environmental parameters, the position of the picking node, and the corresponding node state; determine a first action trajectory based on multiple surface picking positions of each eggplant and the surrounding environment of the corresponding picking node; The picking patterns of the picking robot arm are collected, and a second motion trajectory is determined based on multiple surface picking positions of each eggplant and the picking patterns of the picking robot arm. The picking action of each eggplant is determined based on the first motion trajectory, the second motion trajectory, and surface abnormality features of each part of the eggplant. The picking action of each eggplant is marked at the corresponding position of the picking path, and a picking dynamic graph corresponding to the picking path is constructed.

10. An eggplant picking control system based on image recognition, characterized in that: The eggplant picking control system based on image recognition is applied to the eggplant picking control method based on image recognition as claimed in any one of claims 1 to 9. The eggplant picking control system based on image recognition includes: An image recognition module is used by the drone to mark the spatial position of each eggplant when inspecting the eggplant picking site, collect multiple current images of each eggplant, and determine the morphological characteristics of each eggplant based on the recognition of the multiple current images of each eggplant; a surface abnormality feature module, configured to determine sub-images of various parts of the eggplant based on the recognition of multiple current images, and determine surface abnormality features of various parts according to the recognition of the sub-images of various parts of the eggplant; The picking area module is used to determine multiple surface picking locations of the eggplant based on the morphological characteristics of the eggplant and the surface abnormalities of each part, and to determine the picking area based on the spatial position, maturity and distribution map of the eggplant picking areas; The picking path module is used for the drone equipped with a picking robot arm to collect the flight space of the eggplant picking area and determine the corresponding picking path according to the regional morphology of the picking area, the flight space of the eggplant picking area and the current position of the picking robot arm; The picking action module is used to determine multiple picking nodes based on the picking path and the spatial position of each eggplant, and to determine the picking action of each eggplant according to the surrounding environment of the multiple picking nodes, the multiple surface picking positions of each eggplant and the picking form of the picking robot arm, so as to construct a picking dynamic graph corresponding to the picking path.

Images